From f5de197536358c4fcda375df1bb421bad968c41a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 11:30:10 -0600 Subject: [PATCH 01/11] fix(ui): keep a busy-workspace wait as a quiet status, not a notice (#19507) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The task page tells the user what an agent is doing and what needs their attention > - When another task uses the same workspace, the agent waits and Paperclip starts the work automatically > - A tester reported that the "Waiting for workspace" notice is not helpful, because the user cannot do anything about it > - On `master`, the task page shows this wait four times: a pinned top banner, a conversation banner, the composer strip and the Workspace property row > - This pull request removes the two banners and keeps the two quiet status surfaces > - The benefit is that a transient system wait no longer looks like something that needs the user ## Linked Issues or Issue Description Refs #19369 **What happened?** During new-user onboarding tests on Paperclip Cloud staging (Claude subscription path), the tester saw a "Waiting for workspace" notice and reported: "this isn't helpful and the user can't do anything about it." **Steps to reproduce** 1. Start work on a task while another task uses the same workspace. 2. Open the waiting task. 3. See the pinned "Waiting for workspace" banner under the title and a second "Waiting for workspace" banner in the conversation. **Expected behavior** A busy workspace is a transient system wait with no user action. It shows only as a quiet status on the task. It does not show as a banner or notice. **Investigation** - No server inbox, Needs-you item or activity entry is created for a workspace wait. The live-updates toast already skips `workspace_busy` (`LiveUpdatesProvider.tsx`), and the chat thread already hides the cancelled admission attempt. - The remaining notices are UI banners: `IssueMonitorBanner` (pinned top banner) and `IssueScheduledRetryCard` (conversation banner). - #19369 stopped showing resource waits as failed attempts, but it kept these banners. ## What Changed - `IssueMonitorBanner`: render nothing for a workspace wait. The composer strip still shows "Waiting for workspace" and "You can keep sending instructions while the agent waits." - `IssueScheduledRetryCard`: render nothing for `workspace_busy`. The Workspace property row still shows "Waiting for workspace". - `DESIGN.md`: add the rule that a busy-workspace wait is a quiet status only, never a pinned banner, conversation notice or toast. - Tests: replace the old "shows the banner" assertions with N-14 regression tests. ## Verification Red on `master` (`819420a60a`), test change only: ``` cd ui && npx vitest run src/components/IssueMonitorBanner.test.tsx src/components/IssueScheduledRetryCard.test.tsx × does not raise a pinned banner for a workspace wait the user cannot act on (N-14) 7ms ⎯⎯⎯⎯⎯⎯⎯ Failed Tests 2 ⎯⎯⎯⎯⎯⎯⎯ FAIL src/components/IssueMonitorBanner.test.tsx > IssueMonitorBanner / IssueMonitorComposerStrip rendering > does not raise a pinned banner for a workspace wait the user cannot act on (N-14) AssertionError: expected 'Waiting for workspaceAnother task is …' to be '' // Object.is equality FAIL src/components/IssueScheduledRetryCard.test.tsx > IssueScheduledRetryCard > raises no conversation notice for workspace contention the user cannot act on (N-14) AssertionError: expected 'Waiting for workspaceAnother task is …' to be '' // Object.is equality Tests 2 failed | 32 passed (34) ``` Green after the fix, same command: ``` Test Files 2 passed (2) Tests 34 passed (34) ``` Neighbouring suites: ``` cd ui && npx vitest run src/components/TaskChatThread.test.tsx src/pages/IssueDetail src/components/issue-properties src/lib/issue-monitor Test Files 7 passed (7) Tests 687 passed | 1 skipped (688) ``` - `pnpm check:token-gates`: all gates clean. - `tsc --noEmit -p ui`: no errors in the changed files. Local errors come only from unbuilt workspace packages (`@paperclipai/plugin-sdk/ui`, Storybook feature map). CI typecheck covers the full build. ## Risks - Low risk. UI-only change. Two banners no longer render for `workspace_busy`. Other scheduled-retry reasons (failures, pool exhaustion, continuations) are unchanged. - A long workspace wait now has no prominent notice. No actionable next step exists in the current data (the API does not name the task that holds the workspace). A persist-past-threshold escalation with a link to that task can follow when the data exists. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, browser for read-only staging evidence). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- DESIGN.md | 2 +- ui/src/components/IssueMonitorBanner.test.tsx | 14 ++++++++++++-- ui/src/components/IssueMonitorBanner.tsx | 4 ++++ ui/src/components/IssueScheduledRetryCard.test.tsx | 6 ++---- ui/src/components/IssueScheduledRetryCard.tsx | 11 +++-------- 5 files changed, 22 insertions(+), 15 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index cff31c9a04c..7b860708c8e 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -153,7 +153,7 @@ results in place; keep failures actionable inline. Notifications for other work remain useful. Repeated delivery of the same run outcome must refresh cached state without repeating its toast, including after reconnecting. A terminal outcome delivered more than five minutes after the run finished is historical and should -refresh state silently. Expected cancellation is neutral gray, not an error. The composer's Stop action stops the current response and leaves the composer available for a new message. Pause work is a separate explicit task or subtree action. A paused task replaces the composer with an amber takeover. It says “Task is +refresh state silently. Expected cancellation is neutral gray, not an error. A busy workspace is a transient system wait with no user action. Show it only as a quiet status in the composer strip and the Workspace property row, never as a pinned banner, conversation notice or toast. The composer's Stop action stops the current response and leaves the composer available for a new message. Pause work is a separate explicit task or subtree action. A paused task replaces the composer with an amber takeover. It says “Task is paused.” and “Resume this task to send a message.” with a “Resume task” action. Subtrees use “Subtree is paused.” and “Resume subtree.” The takeover cannot be dismissed, retains drafts, and hides message inputs until the pause is released. diff --git a/ui/src/components/IssueMonitorBanner.test.tsx b/ui/src/components/IssueMonitorBanner.test.tsx index 9156408fc95..d117e87c1d6 100644 --- a/ui/src/components/IssueMonitorBanner.test.tsx +++ b/ui/src/components/IssueMonitorBanner.test.tsx @@ -206,13 +206,23 @@ describe("IssueMonitorBanner / IssueMonitorComposerStrip rendering", () => { flushSync(() => root.unmount()); }); - it("explains automatic workspace waiting without promising that a reply bypasses the lock", () => { + it("does not raise a pinned banner for a workspace wait the user cannot act on (N-14)", () => { + const issue = { + status: "todo", scheduledRetry: { status: "scheduled_retry", scheduledRetryReason: "workspace_busy", scheduledRetryAt: NOW.toISOString() }, + } as Issue; + const root = createRoot(container); + flushSync(() => root.render()); + expect(container.textContent).toBe(""); + flushSync(() => root.unmount()); + }); + + it("keeps the workspace wait as a quiet composer status without promising that a reply bypasses the lock", () => { const issue = { status: "todo", scheduledRetry: { status: "scheduled_retry", scheduledRetryReason: "workspace_busy", scheduledRetryAt: NOW.toISOString() }, } as Issue; const root = createRoot(container); flushSync(() => root.render(<>)); - expect(container.textContent).toContain("Waiting for workspace"); + expect(container.querySelector('[data-testid="issue-monitor-composer-strip"]')?.textContent).toContain("Waiting for workspace"); expect(container.textContent).toContain("You can keep sending instructions while the agent waits."); expect(container.textContent).not.toContain("wakes the agent now"); expect(container.querySelector("button")).toBeNull(); diff --git a/ui/src/components/IssueMonitorBanner.tsx b/ui/src/components/IssueMonitorBanner.tsx index c0382a51f1d..21868393bad 100644 --- a/ui/src/components/IssueMonitorBanner.tsx +++ b/ui/src/components/IssueMonitorBanner.tsx @@ -278,6 +278,10 @@ export function IssueMonitorBanner({ const copy = useMonitorSurfaceCopy(issue, reviews.length); if (!copy) return null; + // A busy workspace is a transient system wait the user cannot act on. It stays + // a quiet status (composer strip, properties row), never a pinned notice. + if (copy.workspaceWait) return null; + const canCancel = onCancelMonitor && deriveMonitorState(issue).source === "monitor"; return ( diff --git a/ui/src/components/IssueScheduledRetryCard.test.tsx b/ui/src/components/IssueScheduledRetryCard.test.tsx index 8b6fdaea874..8099576fe17 100644 --- a/ui/src/components/IssueScheduledRetryCard.test.tsx +++ b/ui/src/components/IssueScheduledRetryCard.test.tsx @@ -121,11 +121,9 @@ function getRetryNowButton() { } describe("IssueScheduledRetryCard", () => { - it("shows workspace contention as an automatic wait without failure or retry controls", () => { + it("raises no conversation notice for workspace contention the user cannot act on (N-14)", () => { renderWithProviders(); - expect(container.textContent).toContain("Waiting for workspace"); - expect(container.textContent).toContain("Work starts automatically"); - expect(container.textContent).not.toMatch(/failed|Retry|Attempt|Replaces run/); + expect(container.textContent).toBe(""); expect(getRetryNowButton()).toBeNull(); }); it("surfaces model capacity and the automatic retry without a generic failure", () => { diff --git a/ui/src/components/IssueScheduledRetryCard.tsx b/ui/src/components/IssueScheduledRetryCard.tsx index 383fce3be87..dd6325bad5f 100644 --- a/ui/src/components/IssueScheduledRetryCard.tsx +++ b/ui/src/components/IssueScheduledRetryCard.tsx @@ -7,7 +7,6 @@ import { formatRetryReason } from "@/lib/runRetryState"; import type { IssueScheduledRetry } from "@paperclipai/shared"; import { useRetryNowMutation, type RetryNowError } from "../hooks/useRetryNowMutation"; import { Badge } from "@/components/ui/badge"; -import { InlineBanner } from "@/components/InlineBanner"; import { describeError } from "@/api/errors"; const MAX_TURN_CONTINUATION = "max_turns_continuation"; @@ -34,13 +33,9 @@ export function IssueScheduledRetryCard({ if (!scheduledRetry || !issueId) return null; if (scheduledRetry.status !== "scheduled_retry") return null; - if (scheduledRetry.scheduledRetryReason === "workspace_busy") { - return ( - - Another task is using this workspace. Work starts automatically when it is available. - - ); - } + // Workspace contention is a transient wait with nothing for the user to do. + // The composer strip and properties row carry the quiet status instead. + if (scheduledRetry.scheduledRetryReason === "workspace_busy") return null; const continuation = isContinuationReason(scheduledRetry.scheduledRetryReason); const dueAtIso = scheduledRetry.scheduledRetryAt From ff1d1dea5d3d2556caa89e6b5c548d92b5222728 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Fri, 6 Nov 2026 09:30:24 -0800 Subject: [PATCH 02/11] fix(tools): reject non-tool connections before any write (#19492) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Apps and tools connect to Paperclip through tool connections > - `POST /api/companies/:companyId/tools/connections` creates a tool connection through `createConnection` > - The request schema accepts a channel purpose and the chat and voice transports, but `createConnection` cannot store them > - The database then rejects the row, so the caller gets a 500 and a stray tool application stays behind > - This pull request rejects those requests with a 422 before any write, and stores `config` as `transportConfig` when the caller leaves `transportConfig` out > - The benefit is a clear error, no leftover rows, and consistent connection rows ## Linked Issues or Issue Description No issue exists. The new test typecheck in #18739 found this problem. #18739 listed it as a possible product problem and did not change it. **What happened?** 1. `createConnection` never writes `input.connectionPurpose`, so every row gets the database default `tool`. 2. The schema allows `transport: "chat_sdk"` and `transport: "voice"`. The database check `tool_connections_channel_transport_check` does not allow these transports on a `tool` row. 3. No error mapping exists for a check violation, so the route returns 500. 4. When the request has no `applicationId`, the service inserts a `tool_applications` row first, outside the connection transaction. That row stays after the failure. 5. The schema gave `transportConfig` a default of `{}`. The service stores `input.transportConfig ?? config`, so it stored `{}` instead of a copy of `config`. **Expected behavior** The route returns 422 for a purpose or transport that it cannot create, and writes no rows. When the request leaves `transportConfig` out, the row stores `config` there. **Steps to reproduce** 1. Send `POST /api/companies/:companyId/tools/connections` with `{ "name": "x", "transport": "chat_sdk" }`. 2. See the 500 response. 3. See the new `tool_applications` row for the company. ## What Changed - `server/src/services/tool-access.ts`: `createConnection` rejects a purpose other than `tool`, and the `chat_sdk` and `voice` transports, with 422. The check runs before any lookup or write. Channel connections still come from their own services. - `packages/shared/src/validators/tool-access.ts`: `transportConfig` is optional and has no default. The update schema already removed defaults, so it does not change. - `server/src/__tests__/tool-access-service.test.ts`: added tests for the three rejected inputs (no rows written), and for a request that sends only `config`. ## Verification - `pnpm exec vitest run server/src/__tests__/tool-access-service.test.ts server/src/__tests__/generic-mcp-connection.test.ts server/src/__tests__/tool-connection-health-work.test.ts server/src/__tests__/openapi-routes.test.ts server/src/__tests__/tool-access-routes.test.ts packages/shared/src/validators`: 27 files, 995 tests pass. - `cd server && pnpm typecheck`: passes. This includes the source layers and the three test-tree shards. - Not run locally: the full `pnpm test:run`, the root `pnpm build`, and the UI typecheck. Required CI covers them. ## Risks - Low risk. A client that sends a channel purpose or a chat or voice transport to this route now gets 422 instead of 500. The request failed before this change too. - A client that sends only `config` now gets the same value in `transportConfig`. Every reader checks `config` first, so this changes no behavior. - The OpenAPI output for this body no longer shows a default for `transportConfig`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- packages/shared/src/validators/tool-access.ts | 3 +- .../src/__tests__/tool-access-service.test.ts | 37 +++++++++++++++++++ server/src/services/tool-access.ts | 7 ++++ 3 files changed, 46 insertions(+), 1 deletion(-) diff --git a/packages/shared/src/validators/tool-access.ts b/packages/shared/src/validators/tool-access.ts index 914af74e7ab..f55163c22eb 100644 --- a/packages/shared/src/validators/tool-access.ts +++ b/packages/shared/src/validators/tool-access.ts @@ -191,7 +191,8 @@ export const createToolConnectionSchema = z.object({ status: toolConnectionStatusSchema.optional(), connectionKind: toolConnectionKindSchema.default("managed"), config: toolTransportConfigSchema.optional(), - transportConfig: toolTransportConfigSchema.default({}), + // No default: when the caller omits it, the service stores `config` here. + transportConfig: toolTransportConfigSchema.optional(), credentialRefs: z.array(mcpConnectionCredentialRefSchema).optional(), credentialSecretRefs: z.array(toolCredentialSecretRefSchema).default([]), enabled: z.boolean().optional(), diff --git a/server/src/__tests__/tool-access-service.test.ts b/server/src/__tests__/tool-access-service.test.ts index 186b20e3042..69525cf08e0 100644 --- a/server/src/__tests__/tool-access-service.test.ts +++ b/server/src/__tests__/tool-access-service.test.ts @@ -68,6 +68,7 @@ import { GOOGLE_WORKSPACE_DEFAULT_SELECTION, getAvailableConnectionMethod, getConnectableAppDefinition, + createToolConnectionSchema, type CreateToolConnection, type GoogleWorkspaceConnectorProfileId, } from "@paperclipai/shared"; @@ -3363,6 +3364,42 @@ describeEmbeddedPostgres("tool access service", () => { .where(eq(companySecrets.companyId, company.id))).toEqual([]); }); + it.each([ + { name: "channel purpose", input: { connectionPurpose: "channel" as const, transport: "local_stdio" as const } }, + { name: "chat transport", input: { transport: "chat_sdk" as const } }, + { name: "voice transport", input: { transport: "voice" as const } }, + ])("rejects a $name before it writes any row", async ({ input }) => { + const company = await createCompany(db); + const service = createTestToolAccessService(db); + + await expect(service.createConnection(company.id, connectionInput({ + name: "Not a tool", + config: { templateId: "paperclip.echo-calculator-time" }, + ...input, + }))).rejects.toMatchObject({ status: 422 }); + + expect(await db.select().from(toolConnections) + .where(eq(toolConnections.companyId, company.id))).toEqual([]); + expect(await db.select().from(toolApplications) + .where(eq(toolApplications.companyId, company.id))).toEqual([]); + }); + + it("stores config as transportConfig when the request leaves transportConfig out", async () => { + const company = await createCompany(db); + const service = createTestToolAccessService(db); + const body = createToolConnectionSchema.parse({ + name: "Local echo config only", + transport: "local_stdio", + config: { templateId: "paperclip.echo-calculator-time" }, + }); + + const connection = await service.createConnection(company.id, body); + + const [stored] = await db.select().from(toolConnections).where(eq(toolConnections.id, connection.id)); + expect(stored?.config).toMatchObject({ templateId: "paperclip.echo-calculator-time" }); + expect(stored?.transportConfig).toEqual(stored?.config); + }); + it("registers an approved local stdio template and exposes its runtime slot", async () => { const company = await createCompany(db); const service = createTestToolAccessService(db); diff --git a/server/src/services/tool-access.ts b/server/src/services/tool-access.ts index 7441bcee493..ae1682df28c 100644 --- a/server/src/services/tool-access.ts +++ b/server/src/services/tool-access.ts @@ -4530,6 +4530,13 @@ export function toolAccessService( let applicationNamespace = input.applicationName ?? input.name; const transport = input.transport; if (!transport) throw badRequest("Tool connection transport is required"); + // This path writes only tool connections. Channel setup has its own + // services. Reject other purposes here, before any row is written, so the + // database purpose/transport check never turns them into a 500. + if (input.connectionPurpose !== undefined && input.connectionPurpose !== "tool") + throw unprocessable("Only tool connections can be created here"); + if (transport === "chat_sdk" || transport === "voice") + throw unprocessable("Chat and voice connections cannot be created here"); const config = normalizeGoogleSheetsConnectionConfig( input.config ?? input.transportConfig ?? {}, ); From 543aa4d331a95500a7fe1514451f0a490eb958d3 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 11:33:46 -0600 Subject: [PATCH 03/11] fix(tasks): show a pending native same-run resume instead of a competing Retry (#19503) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native run cannot reattach to its provider after a control-plane restart, the native coordinator resumes the same run by itself after about 30 seconds, up to three attempts > - The run projection shows this correctly as "Retry scheduled" with only `inspect_run` permitted > - The task interruption shown on the issue read only system-recovery evidence and `scheduledRetryAt`. It never read the coordinator's `nextAttemptAt`, so it showed `state: failed, canRetry: true` and offered a manual Retry while the automatic resume was still pending > - The two surfaces disagree, and a user click can race the automatic resume. That is one way a Retry gets parked behind "execution checks" > - This pull request makes the task interruption read the coordinator's pending resume time > - The benefit is one consistent story: while the automatic resume is pending, the task shows it with its time and no competing Retry. If recovery ends, exactly one Retry returns ## Linked Issues or Issue Description Refs #19487, #19497, #19426. **What happened?** A native run failed with `native_session_interrupted` after a restart. Its finalization row had `phase: retryable_failure` and `nextAttemptAt` 30 seconds later. The run projection said "Retry scheduled". The issue interruption said `state: failed, canRetry: true, nextRetryAt: null`. **What did you expect?** While the automatic resume is pending, the task shows that it is resuming, with the time, and no Retry. When the resume fails for good or never runs, exactly one Retry that works on the first click. **Steps to reproduce** 1. Run a native task and restart the control plane so the native attach fails. 2. Read `GET /api/issues/:id` while the coordinator's resume is pending. 3. Observe `interruption.canRetry: true` while `execution.phase` is `retry_scheduled`. ## What Changed - `task-interruption-presentation.ts`: the batched run read selects the native coordinator's `next_attempt_at` when the native run failed, has no result, and its finalization phase is `retryable_failure`. - A pending resume in the future shows `retry_scheduled` with its time and `canRetry: false`. - A due resume shows `working` for up to two minutes while its dispatch runs. After that, or once recovery is terminal, the task falls through to the existing single Retry. ## Verification - `task-interruption-presentation.test.ts`: new case for a future resume, a due resume, a stale resume, and no resume. It fails before the change (the task offered Retry) and passes after. - `task-interruption-query.test.ts` (embedded Postgres): the real batched query shows the pending resume for both list and detail reads, and returns one Retry after the coordinator becomes `terminal_failure`. - All 40 presentation tests and 17 query tests pass. Server typecheck passes. ## Risks - Low risk. This changes only the read model for tasks whose native coordinator holds a pending same-run resume. - If a resume never dispatches, Retry is hidden for up to two minutes after its due time before it returns. - The `POST /issues/:id/retry` endpoint is unchanged. A client that ignores `canRetry` can still submit a Retry during the resume. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../__tests__/task-interruption-query.test.ts | 24 ++++++++++++++++++- .../task-interruption-presentation.test.ts | 18 ++++++++++++++ .../task-interruption-presentation.ts | 24 +++++++++++++++---- 3 files changed, 61 insertions(+), 5 deletions(-) diff --git a/server/src/__tests__/task-interruption-query.test.ts b/server/src/__tests__/task-interruption-query.test.ts index 70cb7cd8ec9..df854e62152 100644 --- a/server/src/__tests__/task-interruption-query.test.ts +++ b/server/src/__tests__/task-interruption-query.test.ts @@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto"; import { eq } from "drizzle-orm"; import { createRunSecretRedactionRegistry } from "../services/run-secret-redaction.js"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; -import { agents, agentWakeupRequests, companies, createDb, applyPendingMigrations, heartbeatRuns, issues, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; +import { agents, agentWakeupRequests, companies, createDb, applyPendingMigrations, heartbeatRuns, issues, nativeRunFinalizations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; import { taskInterruptionPresentation, taskInterruptionPresentations } from "../services/task-interruption-presentation.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; @@ -24,6 +24,28 @@ describe.skipIf(!support.supported)("task interruption latest run query", () => }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); afterAll(async () => { await db?.$client.end(); await database?.cleanup(); }); + it("shows a pending native same-run resume instead of a competing Retry", async () => { + const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Resuming task", issuePrefix: `R${companyId.slice(0, 7)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Resuming agent", adapterType: "paperclip_runner" }); + const [issue] = await db.insert(issues).values({ companyId, title: "Resume after attach failure", + status: "in_progress", assigneeAgentId: agentId }).returning(); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue", + status: "failed", runtimeMode: "native", errorCode: "native_session_interrupted", finishedAt: new Date() }); + const resumeAt = new Date(Date.now() + 30_000); + await db.insert(nativeRunFinalizations).values({ companyId, issueId: issue.id, runId, phase: "retryable_failure", + attempt: 1, nextAttemptAt: resumeAt }); + for (const presentation of [(await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id), + await taskInterruptionPresentation(db, issue)]) { + expect(presentation).toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: resumeAt.toISOString() }); + } + // Exhausted recovery is terminal: exactly one Retry returns. + await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", nextAttemptAt: null }) + .where(eq(nativeRunFinalizations.runId, runId)); + expect((await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id)) + .toMatchObject({ state: "failed", canRetry: true }); + }); + it("redacts workspace preparation errors on task detail and list reads", async () => { vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64)); try { diff --git a/server/src/services/task-interruption-presentation.test.ts b/server/src/services/task-interruption-presentation.test.ts index adbc79479a5..fab317dc824 100644 --- a/server/src/services/task-interruption-presentation.test.ts +++ b/server/src/services/task-interruption-presentation.test.ts @@ -80,6 +80,24 @@ describe("unadmitted chat Retry presentation", () => { }); }); +// The native coordinator resumes the same run itself after an attach failure. +// A manual Retry offered meanwhile would race that automatic resume. +it("shows a pending native same-run resume instead of a competing Retry", () => { + const now = Date.parse("2026-11-06T15:25:50.000Z"); + const failed = { ...run, runtimeMode: "native" as const, errorCode: "native_session_interrupted", resultJson: null }; + const resumeAt = new Date("2026-11-06T15:26:18.687Z"); + expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: resumeAt }, agent, paused: false, approval: false }, now)) + .toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: resumeAt.toISOString() }); + // A due resume is being dispatched; it is still not the user's turn. + expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: new Date(now - 30_000) }, agent, paused: false, approval: false }, now)) + .toMatchObject({ state: "working", canRetry: false }); + // A resume that never ran, or no resume at all, leaves exactly one Retry. + expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: new Date(now - 10 * 60_000) }, agent, paused: false, approval: false }, now)) + .toMatchObject({ state: "failed", canRetry: true }); + expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false }, now)) + .toMatchObject({ state: "failed", canRetry: true }); +}); + it("explains a persisted Retry wait without exposing the saved payload", () => { const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null }; const wait = { id: "retry", requestedAt: new Date(), sourceRunId: failed.id, diff --git a/server/src/services/task-interruption-presentation.ts b/server/src/services/task-interruption-presentation.ts index 59797d3958e..dec56770f76 100644 --- a/server/src/services/task-interruption-presentation.ts +++ b/server/src/services/task-interruption-presentation.ts @@ -1,6 +1,6 @@ import { and, desc, eq, inArray, isNull, sql, type SQL } from "drizzle-orm"; import { agents, agentWakeupRequests, approvals, chatActions, companies, heartbeatRuns, issueApprovals, issueRecoveryActions, - issues, issueTreeHoldMembers, issueTreeHolds, projects, type Db } from "@paperclipai/db"; + issues, issueTreeHoldMembers, issueTreeHolds, nativeRunFinalizations, projects, type Db } from "@paperclipai/db"; import { unionAll } from "drizzle-orm/pg-core"; import { HEARTBEAT_RUN_STATUSES, readProviderFailure, type TaskInterruptionPresentation } from "@paperclipai/shared"; import { captureTaskInterruptionFailure } from "../sentry.js"; @@ -26,11 +26,17 @@ import { AI_CONNECTION_BUSY_RETRY_REASON, AI_CONNECTION_POOL_WAIT_RETRY_REASON, const resourceWaitRetryReasons = new Set([WORKSPACE_BUSY_RETRY_REASON, AI_CONNECTION_BUSY_RETRY_REASON, AI_CONNECTION_POOL_WAIT_RETRY_REASON, RUNNER_REVISION_WAIT_RETRY_REASON]); +// A due same-run resume dispatches within seconds; past this, offer Retry. +const NATIVE_RESUME_DISPATCH_GRACE_MS = 2 * 60_000; + type ProjectionIssue = Pick; type ProjectionRun = Pick & Partial>; + | "scheduledRetryReason">> & { + /** Due time of the native coordinator's own same-run resume, if one is pending. */ + nativeResumeAt?: Date | null; +}; type ProjectionWait = { id: string; requestedAt: Date; sourceRunId?: string | null; executionWaitReason?: string | null; preparationStatus?: string | null; preparationRetryAt?: number | string | null }; type ProjectionState = { @@ -148,10 +154,16 @@ export function projectTaskInterruption(issue: ProjectionIssue, input: Projectio } if (run.status === "scheduled_retry" && resourceWaitRetryReasons.has(run.scheduledRetryReason ?? "") && !system) return { ...base, state: "working" }; - const retryAt = system?.nextAttemptAt ?? run.scheduledRetryAt?.getTime(); + // The native coordinator resumes this same run by itself. A manual Retry + // would race it, so show the resume until it is due and briefly while its + // dispatch runs. A resume that never ran falls through to one Retry. + const nativeResumeAt = run.nativeResumeAt?.getTime(); + const retryAt = system?.nextAttemptAt ?? nativeResumeAt ?? run.scheduledRetryAt?.getTime(); if (retryAt && retryAt > now) return { ...base, state: "retry_scheduled", - nextRetryAt: new Date(retryAt).toISOString(), canRetry: !preparationPending, + nextRetryAt: new Date(retryAt).toISOString(), canRetry: !preparationPending && !nativeResumeAt, ...(system?.nextAttemptAt && recoveryStep ? { recoveryStep } : {}) }; + if (nativeResumeAt && !system && now - nativeResumeAt < NATIVE_RESUME_DISPATCH_GRACE_MS) + return { ...base, state: "working" }; if (recoveryRunning) return { ...base, state: "working", ...(recoveryStep ? { recoveryStep } : {}) }; if (retryWait?.sourceRunId === run.id && retryWait.executionWaitReason) { const reason = Object.hasOwn(retryWaitMessages, retryWait.executionWaitReason) ? retryWait.executionWaitReason : "execution_recovery"; @@ -203,6 +215,10 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i or ${heartbeatRuns.errorCode} = 'workspace_validation_failed' then ${heartbeatRuns.error} else null end`, errorCode: heartbeatRuns.errorCode, livenessState: heartbeatRuns.livenessState, scheduledRetryAt: heartbeatRuns.scheduledRetryAt, scheduledRetryReason: heartbeatRuns.scheduledRetryReason, + nativeResumeAt: sql`(select ${nativeRunFinalizations.nextAttemptAt} from ${nativeRunFinalizations} + where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId} and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id} + and ${heartbeatRuns.runtimeMode} = 'native' and ${heartbeatRuns.status} = 'failed' + and ${nativeRunFinalizations.resultId} is null and ${nativeRunFinalizations.phase} = 'retryable_failure')`.mapWith(value => value == null ? null : new Date(value)), // The run whose resultJson supplied providerFailure. A scheduled retry // shows its predecessor's reason, so redaction must use that run's registry. providerFailureRunId: sql`case when ${heartbeatRuns.resultJson}->'providerFailure' is null From 98ebeefbc94a278081a48282087f8334a21d1cbd Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Fri, 6 Nov 2026 09:35:13 -0800 Subject: [PATCH 04/11] feat(environments): let managed config declare plugin environments (#19508) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A managed instance gets its instance-level environments from the `PAPERCLIP_MANAGED_CONFIG` document > - #19490 added the plugin environment rows, their marker index and write floor, but nothing creates them yet > - The managed config still accepts only one sandbox entry, so a managed instance cannot get a plugin-provided environment without manual setup > - This pull request lets the managed config declare plugin entries beside the sandbox entry, and provisions one locked row for each at boot > - The benefit is that a managed instance gets its plugin environments with no manual setup, and the rows recover by themselves when their plugin restarts ## Linked Issues or Issue Description Refs #19490 (the plugin environment rows this pull request provisions) **Subsystem affected** Server environments: the managed-config parser, the managed environment bootstrap, and the environment routes' write floor. **Problem or motivation** A managed instance cannot get a plugin environment from its managed config. The `environments` section accepts one entry, and that entry always provisions the managed sandbox. Company settings cannot create a plugin environment either. So each managed instance that needs one requires manual setup. **Proposed solution** Let an `environments` entry declare `driver: "plugin"`. Boot provisions one locked row for each plugin entry through the row layer from #19490. A later pull request adds a one-time instance default for one plugin entry, and advertises the new managed-config capability. **Alternatives considered** Manual setup on each instance, and a new control-plane endpoint. Both need a step outside the deploy. The managed config already delivers the sandbox entry. **Roadmap alignment** This extends the existing managed-config `environments` section, which already provisions the managed sandbox. It adds no new product surface. ## What Changed - `server/src/services/managed-config.ts`: an `environments` entry may set `driver` to `"sandbox"` or `"plugin"`. An entry without `driver` is still the sandbox entry, so today's documents parse unchanged. - A plugin entry has `name`, optional `description`, `pluginKey` (the plugin's manifest id), `driverKey` and optional `driverConfig`. - At most one sandbox entry. Names are unique across the section, and `pluginKey`–`driverKey` pairs are unique. Secret-like keys in `driverConfig` refuse startup, as for sandbox `config`. An unknown key or driver still refuses startup, and that includes `instanceDefault` until the follow-up pull request. - Plugin entries parse into a new `pluginEnvironments` list. The `environments` list keeps only the sandbox entry, so its consumers do not change. - `MANAGED_ENVIRONMENT_DRIVERS` lists the accepted drivers. - `server/src/services/managed-environments.ts` (`applyManagedEnvironments`): - Boot refuses to start unless an auto-installed catalog entry installs each plugin entry's `pluginKey`. `plugins.autoInstall` holds catalog keys, not manifest ids, so the parser cannot check this. Boot uses the manifest ids `createApp` already resolves from the bundled and distribution catalogs (`app.locals.managedBundledPluginKeys`, passed in by `server/src/index.ts`). Without that list it resolves the catalog itself. - Plugin entries run after the sandbox entry and reuse its readiness handling. Boot awaits `pluginsReady`, then requires the plugin `ready`, declaring the entry's driver, and running a worker. Otherwise boot archives the row and re-ensures it on the worker's `ready`. - Recovery re-ensures on every worker `ready` until one ensure succeeds or the plugin rejects the config (HTTP 422). - If the ensure fails because the worker died, boot archives the row and waits for `ready`. If the worker is still running (for example the call timed out), boot keeps the row as it is, retries once at once and on the next `ready`, and otherwise at the next boot. No timer is added. - Removing an entry never archives its row. - A plugin-only document does not conflict with `PAPERCLIP_EXECUTION_MODE`, because plugin rows do not use the sandbox slot. - Plugin entries never change the instance default. The sandbox stamp still fills only an unset or local default, so it never replaces a plugin default. - `server/src/routes/environments.ts`: `isPlatformSlotEnvironment` treats a marked plugin row as a live slot while the document declares its entry. So the marker-clear escape hatch is refused then, and allowed again after the entry leaves the document. - `doc/DEVELOPING.md`: new "Managed environments" section. ## Verification - `cd server && npx vitest run src/__tests__/managed-config.test.ts`: old documents parse unchanged. Sandbox plus plugin parses. Each of these refuses startup: two sandbox entries, `driver: null`, duplicate names, a duplicate pair, a secret-like key, an unknown key (including `instanceDefault`), an unknown driver, a malformed `driverKey` or `driverConfig`. - `npx vitest run src/services/managed-environments-plugin-entries.test.ts` (the sandbox-entry tests stay in `managed-environments.test.ts`; shared fixtures are in `managed-environments.test-support.ts`): - A plugin entry is ensured with the validator's worker manager, after the sandbox entry. - Boot refuses a plugin entry whose `pluginKey` no auto-installed catalog entry installs, both from the passed list and from its own catalog lookup. - A plugin that is down archives the row, and worker recovery re-ensures it. A missing plugin, a plugin without the driver, or a worker manager without `call` counts as unavailable. - Recovery stays armed after a failed attempt and stops after one success or a permanent config rejection (422). A `ready` during an attempt queues one more. - A worker that dies during the boot ensure archives the row and arms recovery. A failure while the worker keeps running leaves the row as it is and retries. - Removing an entry does not archive. Plugin entries and the sandbox stamp never change a plugin default. - An ensure failure is counted, not thrown. - `npx vitest run src/__tests__/managed-plugin-environments.test.ts` (embedded Postgres, real `applyManagedEnvironments`): the first boot creates the row and the second changes nothing. A plugin that is down archives the row, and recovery brings it back. Removing the entry does not archive the row. The instance default stays unset. The row tests from #19490 still pass. - `npx vitest run src/__tests__/environment-routes.test.ts`: the marker clear is refused while the document declares the entry, and allowed after it leaves. - `npx vitest run src/__tests__/instance-settings-service.test.ts src/__tests__/instance-settings-cloud-defaults.test.ts`: pass with the new `pluginEnvironments` field. - `pnpm --filter @paperclipai/server typecheck` passes. - I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build`. CI covers them. ## Risks - No schema change. - A managed-config document with a plugin entry now provisions a row at boot. A document without one behaves as before. - Adoption renames an unmarked row for the same plugin and driver to the declared name and resets its driver config to the declared one. The row keeps its id, agents and other metadata, and it is then locked like the managed sandbox. - Boot now refuses to start when a plugin entry names a plugin that no auto-installed catalog entry provides. This is the same fail-closed posture as an unknown sandbox provider. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (file edits, shell, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/DEVELOPING.md | 31 ++ .../src/__tests__/environment-routes.test.ts | 64 ++- .../instance-settings-cloud-defaults.test.ts | 1 + .../instance-settings-service.test.ts | 4 +- server/src/__tests__/managed-config.test.ts | 100 ++++- .../managed-plugin-environments.test.ts | 79 +++- server/src/app.ts | 3 + server/src/index.ts | 14 +- server/src/routes/environments.ts | 26 +- server/src/services/managed-config.ts | 283 +++++++++---- ...anaged-environments-plugin-entries.test.ts | 385 ++++++++++++++++++ .../managed-environments.test-support.ts | 139 +++++++ .../src/services/managed-environments.test.ts | 130 +----- server/src/services/managed-environments.ts | 330 +++++++++++++-- 14 files changed, 1335 insertions(+), 254 deletions(-) create mode 100644 server/src/services/managed-environments-plugin-entries.test.ts create mode 100644 server/src/services/managed-environments.test-support.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 5704355c9fc..5a7659c4ee9 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1919,6 +1919,37 @@ that the whole web server is disposable or that all execution and recovery paths survive web-server replacement. Run-log/trace storage and producer retention are separate changes. +### Managed environments + +The `environments` list in `PAPERCLIP_MANAGED_CONFIG` declares instance-level +environments that boot provisions and tenants cannot edit. It holds at most one +sandbox entry (no `driver`, or `driver: "sandbox"`) and any number of +`driver: "plugin"` entries. A plugin entry names a plugin by its manifest id +(`pluginKey`) and one of its environment drivers: + +```json +{ + "name": "Remote Host", + "description": "Runs the agent on a remote host", + "driver": "plugin", + "pluginKey": "example.remote-host-plugin", + "driverKey": "tasks", + "driverConfig": {} +} +``` + +Names and `pluginKey`–`driverKey` pairs must be unique, and `driverConfig` must +not contain secret-like keys. Any other key refuses startup. `plugins.autoInstall` +holds catalog keys, not manifest ids, so boot also refuses to start unless an +auto-installed bundled or distribution catalog entry installs the entry's +`pluginKey`. + +Boot validates the driver config through the plugin, marks the row +`managedPluginEnvironment: ":"`, and adopts an existing +unmarked row for the same plugin and driver. While the plugin is down, the row is +archived; it comes back when the worker is ready. Removing an entry never archives +its row. Plugin entries do not change the instance default. + ### Experimental native remote workspace storage `experimental.remoteWorkspaceStorageEnabled` is a managed feature, on by default diff --git a/server/src/__tests__/environment-routes.test.ts b/server/src/__tests__/environment-routes.test.ts index 786e97d6ea2..0274007ac2d 100644 --- a/server/src/__tests__/environment-routes.test.ts +++ b/server/src/__tests__/environment-routes.test.ts @@ -1022,6 +1022,20 @@ describe("environment routes", () => { describe("managed plugin environment rows", () => { const PLUGIN_KEY = "example.remote-host-plugin"; + const MANAGED_CONFIG_WITH_PLUGIN_ENTRY = JSON.stringify({ + v: 1, + mode: "cloud", + catalogVersion: "2026.1106.0", + features: {}, + plugins: { autoInstall: ["remote-host"] }, + environments: [{ + name: "Remote Host", + driver: "plugin", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + driverConfig: {}, + }], + }); const pluginRow = () => ({ ...createPlatformSandboxEnvironment(), id: "env-plugin-1", @@ -1058,20 +1072,46 @@ describe("environment routes", () => { expect(res.body.details).toMatchObject({ code: "environment_platform_managed" }); }); - it("allows the marker clear while no managed-config entry can own the row", async () => { - // Nothing provisions marked plugin rows yet, so a marker is stale and - // stays recoverable. The managed-config plugin entry PR refuses this - // while the document declares the row's entry. - const row = pluginRow(); - mockEnvironmentService.getById.mockResolvedValue(row); - mockEnvironmentService.update.mockResolvedValue({ ...row, metadata: {} }); + it("refuses a marker clear while the document declares the entry", async () => { + process.env.PAPERCLIP_MANAGED_CONFIG = MANAGED_CONFIG_WITH_PLUGIN_ENTRY; + try { + mockEnvironmentService.getById.mockResolvedValue(pluginRow()); + const app = createApp(ownerAdminActor); - const res = await request(createApp(ownerAdminActor)) - .patch("/api/environments/env-plugin-1") - .send({ metadata: { managedByPaperclip: null, managedPluginEnvironment: null } }); + for (const metadata of [ + { managedByPaperclip: false }, + { managedPluginEnvironment: null }, + { managedByPaperclip: null, managedPluginEnvironment: null }, + ]) { + const res = await request(app).patch("/api/environments/env-plugin-1").send({ metadata }); + expect(res.status).toBe(403); + expect(res.body.details).toMatchObject({ code: "environment_platform_managed" }); + } + expect(mockEnvironmentService.update).not.toHaveBeenCalled(); + } finally { + delete process.env.PAPERCLIP_MANAGED_CONFIG; + } + }); - expect(res.status).toBe(200); - expect(mockEnvironmentService.update).toHaveBeenCalled(); + it("allows the marker clear once the entry left the document", async () => { + process.env.PAPERCLIP_MANAGED_CONFIG = JSON.stringify({ + ...JSON.parse(MANAGED_CONFIG_WITH_PLUGIN_ENTRY), + environments: [], + }); + try { + const row = pluginRow(); + mockEnvironmentService.getById.mockResolvedValue(row); + mockEnvironmentService.update.mockResolvedValue({ ...row, metadata: {} }); + + const res = await request(createApp(ownerAdminActor)) + .patch("/api/environments/env-plugin-1") + .send({ metadata: { managedByPaperclip: null, managedPluginEnvironment: null } }); + + expect(res.status).toBe(200); + expect(mockEnvironmentService.update).toHaveBeenCalled(); + } finally { + delete process.env.PAPERCLIP_MANAGED_CONFIG; + } }); it("refuses a client that stamps the plugin marker on its own row", async () => { diff --git a/server/src/__tests__/instance-settings-cloud-defaults.test.ts b/server/src/__tests__/instance-settings-cloud-defaults.test.ts index 76ff39b0074..6a9a550032f 100644 --- a/server/src/__tests__/instance-settings-cloud-defaults.test.ts +++ b/server/src/__tests__/instance-settings-cloud-defaults.test.ts @@ -17,6 +17,7 @@ function managedConfig(features: ManagedInstanceConfig["features"] = {}): Manage features, plugins: { autoInstall: [] }, environments: [], + pluginEnvironments: [], }; } diff --git a/server/src/__tests__/instance-settings-service.test.ts b/server/src/__tests__/instance-settings-service.test.ts index 6c2847b6eb9..4ced4dcc4c1 100644 --- a/server/src/__tests__/instance-settings-service.test.ts +++ b/server/src/__tests__/instance-settings-service.test.ts @@ -26,7 +26,7 @@ describe("instance settings service", () => { } expect(applyExperimentalSettingsPatch({}, { enableAgentChat: false }).enableAgentChat).toBe(true); const config: ManagedInstanceConfig = { - v: 1, mode: "cloud", catalogVersion: "test", plugins: { autoInstall: [] }, environments: [], + v: 1, mode: "cloud", catalogVersion: "test", plugins: { autoInstall: [] }, environments: [], pluginEnvironments: [], features: { enableAgentChat: false }, }; const result = applyManagedExperimentalOverlay(normalizeExperimentalSettings({ enableAgentChat: false }), config); @@ -54,7 +54,7 @@ describe("instance settings service", () => { { stored: { remoteWorkspaceStorageEnabled: true }, managed: false, expected: false }, ])("respects Cloud enrollment defaults and managed choices: $expected", ({ stored, managed, expected }) => { const config: ManagedInstanceConfig = { - v: 1, mode: "cloud", catalogVersion: "test", plugins: { autoInstall: [] }, environments: [], + v: 1, mode: "cloud", catalogVersion: "test", plugins: { autoInstall: [] }, environments: [], pluginEnvironments: [], features: managed === undefined ? {} : { remoteWorkspaceStorageEnabled: managed }, }; const normalized = normalizeExperimentalSettings(stored); diff --git a/server/src/__tests__/managed-config.test.ts b/server/src/__tests__/managed-config.test.ts index 226ef9bf401..e01bdbb31d0 100644 --- a/server/src/__tests__/managed-config.test.ts +++ b/server/src/__tests__/managed-config.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from "vitest"; import { MANAGED_CONFIG_ENV_KEY, + MANAGED_ENVIRONMENT_DRIVERS, getManagedInstanceConfig, managedFeatureKeySet, parseManagedConfigEnv, @@ -110,6 +111,7 @@ describe("parseManagedConfigEnv", () => { features: { enableApps: false, enableStatusCards: true }, plugins: { autoInstall: ["daytona", "kubernetes"] }, environments: [], + pluginEnvironments: [], }); }); @@ -132,6 +134,7 @@ describe("parseManagedConfigEnv", () => { features: {}, plugins: { autoInstall: [] }, environments: [], + pluginEnvironments: [], }); }); @@ -342,12 +345,17 @@ describe("parseManagedConfigEnv environments section", () => { ).toThrow(/"environments\[0\]" must be an object/); }); - it("rejects more than one entry (single managed sandbox slot)", () => { + it("rejects more than one sandbox entry (single managed sandbox slot)", () => { expect(() => parseManagedConfigEnv( envWith(validDoc({ environments: [entry(), entry({ name: "Other", provider: "kubernetes" })] })), ), - ).toThrow(/at most one entry/); + ).toThrow(/at most one sandbox entry/); + expect(() => + parseManagedConfigEnv( + envWith(validDoc({ environments: [entry(), entry({ name: "Other", driver: "sandbox", provider: "kubernetes" })] })), + ), + ).toThrow(/at most one sandbox entry/); }); it("rejects unknown entry keys", () => { @@ -406,6 +414,94 @@ describe("parseManagedConfigEnv environments section", () => { }); }); +describe("parseManagedConfigEnv plugin environment entries", () => { + const PLUGIN_KEY = "example.remote-host-plugin"; + const remoteHost = (overrides: Record = {}) => ({ + name: "Remote Host", + description: "Runs the agent on a remote host", + driver: "plugin", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + driverConfig: {}, + ...overrides, + }); + const sandbox = { name: "Managed Sandbox", provider: "daytona", config: { target: "us" } }; + // `plugins.autoInstall` holds catalog keys ("remote-host"); the entry's + // `pluginKey` is the plugin's manifest id, which boot maps through the catalog. + const docWith = (environments: unknown[], autoInstall = ["daytona", "remote-host"]) => + envWith(validDoc({ plugins: { autoInstall }, environments })); + + it("lists the drivers an entry may declare", () => { + expect(MANAGED_ENVIRONMENT_DRIVERS).toEqual(["sandbox", "plugin"]); + }); + + it("parses an old document's sandbox entry unchanged", () => { + const config = parseManagedConfigEnv(docWith([sandbox])); + expect(config?.environments).toEqual([{ name: "Managed Sandbox", provider: "daytona", config: { target: "us" } }]); + expect(config?.pluginEnvironments).toEqual([]); + }); + + it("parses the sandbox entry beside a plugin entry", () => { + const config = parseManagedConfigEnv(docWith([sandbox, remoteHost()])); + expect(config?.environments).toEqual([{ name: "Managed Sandbox", provider: "daytona", config: { target: "us" } }]); + expect(config?.pluginEnvironments).toEqual([{ + driver: "plugin", + name: "Remote Host", + description: "Runs the agent on a remote host", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + driverConfig: {}, + }]); + expect(Object.isFrozen(config?.pluginEnvironments)).toBe(true); + expect(Object.isFrozen(config?.pluginEnvironments[0])).toBe(true); + }); + + it("parses a plugin entry alone, with optional fields omitted", () => { + const config = parseManagedConfigEnv(docWith([{ name: "Remote", driver: "plugin", pluginKey: PLUGIN_KEY, driverKey: "tasks" }])); + expect(config?.environments).toEqual([]); + expect(config?.pluginEnvironments[0]).toMatchObject({ driverConfig: {} }); + expect(config?.pluginEnvironments[0]?.description).toBeUndefined(); + }); + + it("does not compare the manifest id with autoInstall catalog keys (boot checks the catalog)", () => { + expect(parseManagedConfigEnv(docWith([remoteHost()]))?.pluginEnvironments[0]?.pluginKey).toBe(PLUGIN_KEY); + }); + + it("refuses duplicate names, including across drivers", () => { + expect(() => parseManagedConfigEnv(docWith([sandbox, remoteHost({ name: "Managed Sandbox" })]))) + .toThrow(/"environments\[1\].name" duplicates another entry's name/); + }); + + it("refuses a duplicate pluginKey and driverKey pair", () => { + expect(() => parseManagedConfigEnv(docWith([remoteHost(), remoteHost({ name: "Second" })]))) + .toThrow(/duplicates another plugin entry's pluginKey and driverKey/); + }); + + it("refuses secret-like driverConfig keys at any depth", () => { + expect(() => parseManagedConfigEnv(docWith([remoteHost({ driverConfig: { apiToken: "t" } })]))) + .toThrow(/"environments\[0\].driverConfig" key "apiToken" looks secret-bearing/); + expect(() => parseManagedConfigEnv(docWith([remoteHost({ driverConfig: { auth: [{ password: "p" }] } })]))) + .toThrow(/auth\[0\].password/); + }); + + it("refuses unknown keys, including sandbox keys on a plugin entry and the reverse", () => { + expect(() => parseManagedConfigEnv(docWith([remoteHost({ envVars: {} })]))).toThrow(/unknown key "envVars"/); + expect(() => parseManagedConfigEnv(docWith([remoteHost({ provider: "daytona" })]))).toThrow(/unknown key "provider"/); + expect(() => parseManagedConfigEnv(docWith([{ ...sandbox, pluginKey: PLUGIN_KEY }]))).toThrow(/unknown key "pluginKey"/); + // Not supported by this build on either entry. + expect(() => parseManagedConfigEnv(docWith([remoteHost({ instanceDefault: true })]))).toThrow(/unknown key "instanceDefault"/); + expect(() => parseManagedConfigEnv(docWith([{ ...sandbox, instanceDefault: false }]))).toThrow(/unknown key "instanceDefault"/); + }); + + it("refuses an unknown driver and malformed plugin fields", () => { + expect(() => parseManagedConfigEnv(docWith([remoteHost({ driver: "ssh" })]))).toThrow(/"environments\[0\].driver" must be one of "sandbox", "plugin"/); + expect(() => parseManagedConfigEnv(docWith([{ ...sandbox, driver: null }]))).toThrow(/"environments\[0\].driver" must be one of "sandbox", "plugin" when present \(got null\)/); + expect(() => parseManagedConfigEnv(docWith([remoteHost({ driverKey: "Tasks" })]))).toThrow(/"environments\[0\].driverKey" must start/); + expect(() => parseManagedConfigEnv(docWith([remoteHost({ driverKey: undefined })]))).toThrow(/"environments\[0\].driverKey" must be a non-empty string/); + expect(() => parseManagedConfigEnv(docWith([remoteHost({ driverConfig: [] })]))).toThrow(/"environments\[0\].driverConfig" must be an object/); + }); +}); + describe("getManagedInstanceConfig", () => { it("caches by raw env value and reparses when it changes", () => { const raw = validDoc(); diff --git a/server/src/__tests__/managed-plugin-environments.test.ts b/server/src/__tests__/managed-plugin-environments.test.ts index fc8496b236d..4179d238dbf 100644 --- a/server/src/__tests__/managed-plugin-environments.test.ts +++ b/server/src/__tests__/managed-plugin-environments.test.ts @@ -7,15 +7,42 @@ import { companies, createDb, environments, + instanceSettings, plugins, type Db, } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { environmentService } from "../services/environments.js"; +import { MANAGED_CONFIG_ENV_KEY, parseManagedConfigEnv } from "../services/managed-config.js"; +import { applyManagedEnvironments } from "../services/managed-environments.js"; const PLUGIN_KEY = "example.remote-host-plugin"; const MARKER = `${PLUGIN_KEY}:tasks`; +type ReadyListener = (payload: { pluginId: string }) => void; + +function managedConfig(entries: unknown[]) { + return parseManagedConfigEnv({ + [MANAGED_CONFIG_ENV_KEY]: JSON.stringify({ + v: 1, + mode: "cloud", + catalogVersion: "2026.1106.0", + features: {}, + plugins: { autoInstall: ["remote-host"] }, + environments: entries, + }), + }); +} + +const remoteHostEntry = { + name: "Remote Host", + description: "Runs the agent on a remote host", + driver: "plugin", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + driverConfig: {}, +}; + const support = await getEmbeddedPostgresTestSupport(); (support.supported ? describe : describe.skip)("managed plugin environment rows", () => { let database: Awaited>; @@ -24,7 +51,17 @@ const support = await getEmbeddedPostgresTestSupport(); let running: boolean; const call = vi.fn(async (_id: string, method: string, input: { config: Record }) => method === "environmentValidateConfig" ? { ok: true, normalizedConfig: { ...input.config, normalized: true } } : null); - const workerManager = { isRunning: vi.fn(() => running), call }; + let readyListeners: ReadyListener[] = []; + const workerManager = { + isRunning: vi.fn(() => running), + call, + getWorker: vi.fn(() => ({ + on: (_event: "ready", listener: ReadyListener) => { readyListeners.push(listener); }, + off: (_event: "ready", listener: ReadyListener) => { + readyListeners = readyListeners.filter((candidate) => candidate !== listener); + }, + })), + }; beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("paperclip-managed-plugin-environment-rows-"); @@ -33,9 +70,11 @@ const support = await getEmbeddedPostgresTestSupport(); afterAll(async () => { await database?.cleanup(); }); beforeEach(async () => { running = true; + readyListeners = []; call.mockClear(); await db.delete(activityLog); await db.delete(agents); + await db.delete(instanceSettings); await db.delete(environments); await db.delete(plugins); await db.delete(companies); @@ -175,4 +214,42 @@ const support = await getEmbeddedPostgresTestSupport(); metadata: { managedPluginEnvironment: MARKER }, })).rejects.toThrow(); }); + + describe("boot (applyManagedEnvironments)", () => { + const boot = (entries: unknown[] = [remoteHostEntry]) => + applyManagedEnvironments(db, managedConfig(entries), { env: {}, workerManager, autoInstalledPluginKeys: [PLUGIN_KEY] }); + const instanceDefault = async () => + (await db.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, "default")))[0]?.defaultEnvironmentId ?? null; + + it("creates the row on the first boot and changes nothing on the second", async () => { + const first = await boot(); + expect(first).toMatchObject({ ensured: 1, added: 1, failed: 0 }); + const [row] = await pluginRows(); + expect(row).toMatchObject({ name: "Remote Host", status: "active", metadata: { managedByPaperclip: true, managedPluginEnvironment: MARKER } }); + + const second = await boot(); + expect(second).toMatchObject({ ensured: 1, added: 0, unchanged: 1 }); + const after = await pluginRows(); + expect(after).toHaveLength(1); + expect(after[0]!.updatedAt.getTime()).toBe(row!.updatedAt.getTime()); + expect(await instanceDefault()).toBeNull(); + }); + + it("archives the row while the plugin is down and reactivates it when the worker recovers", async () => { + await boot(); + running = false; + expect(await boot()).toMatchObject({ ensured: 0, failed: 1 }); + expect((await pluginRows())[0]!.status).toBe("archived"); + + running = true; + for (const listener of [...readyListeners]) listener({ pluginId }); + await vi.waitFor(async () => expect((await pluginRows())[0]?.status).toBe("active")); + }); + + it("never archives a row whose entry left the document", async () => { + await boot(); + expect(await boot([])).toBeNull(); + expect((await pluginRows())[0]!.status).toBe("active"); + }); + }); }); diff --git a/server/src/app.ts b/server/src/app.ts index 75a3ee1c207..5a765ee0c95 100644 --- a/server/src/app.ts +++ b/server/src/app.ts @@ -1348,6 +1348,9 @@ export async function createApp( logger.error({ err }, "Failed to load ready plugins on startup"); }); app.locals.bundledPluginsStartup = trackIdleWork(bundledPluginsStartup); + // Manifest ids the managed auto-install list resolves to; the managed + // environments bootstrap checks plugin entries against them. + app.locals.managedBundledPluginKeys = managedBundledPluginKeys; void bundledPluginsStartup.then(() => { lifecyclePluginsReady = true; resourceLifecycleDelivery.wake(); agentLifecycleDelivery.wake(); }).catch(() => logger.warn("Agent lifecycle recovery failed; retrying.")); // The shutdown hook runs at most once. It caches the in-flight promise, so a // second caller (for example the `exit` handler) awaits the same completion diff --git a/server/src/index.ts b/server/src/index.ts index e36818bfdee..13c55b20807 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1175,8 +1175,8 @@ async function startServerWithDatabaseTeardown( throw err; } - // Ensure sandbox environments declared in the managed-config document - // (`environments` section) before the heartbeat resumes queued runs. The + // Ensure the sandbox and plugin environments declared in the managed-config + // document (`environments` section) before the heartbeat resumes queued runs. The // document already parsed fail-closed above; the ensure step itself is // fail-safe per entry (a degraded boot beats a fleet-wide crash loop), but // a contradictory deployment that also forces PAPERCLIP_EXECUTION_MODE @@ -1186,14 +1186,16 @@ async function startServerWithDatabaseTeardown( // additionally gates each entry on a live plugin worker (and archives the // row of a provider that did not come up). try { - const bundledPluginsStartup = (app as { locals?: { bundledPluginsStartup?: Promise } }) - .locals?.bundledPluginsStartup; + const appLocals = (app as { + locals?: { bundledPluginsStartup?: Promise; managedBundledPluginKeys?: readonly string[] }; + }).locals; const managedEnvironmentsResult = await applyManagedEnvironments(db as any, managedConfig, { - pluginsReady: bundledPluginsStartup, + pluginsReady: appLocals?.bundledPluginsStartup, + autoInstalledPluginKeys: appLocals?.managedBundledPluginKeys, workerManager: pluginWorkerManager, }); if (managedEnvironmentsResult) { - logger.warn(managedEnvironmentsResult, "managed sandbox environments ensured from managed config"); + logger.warn(managedEnvironmentsResult, "managed environments ensured from managed config"); } } catch (err) { logger.error({ err }, "failed to apply managed environments from managed config"); diff --git a/server/src/routes/environments.ts b/server/src/routes/environments.ts index 9a33f1b66f9..09b2023b7e7 100644 --- a/server/src/routes/environments.ts +++ b/server/src/routes/environments.ts @@ -19,7 +19,10 @@ import { import { conflict, forbidden, unprocessable } from "../errors.js"; import { isCloudManagedInstance } from "../services/cloud-instance.js"; import { getManagedInstanceConfig, SECRET_LIKE_CONFIG_KEY_PATTERN } from "../services/managed-config.js"; -import { MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY } from "../services/managed-plugin-environments.js"; +import { + MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY, + managedPluginEnvironmentMarker, +} from "../services/managed-plugin-environments.js"; import { parseExecutionPolicyBootstrapEnv } from "../services/execution-policy-bootstrap.js"; import { isExecutionForcedToKubernetes } from "../services/execution-allowlist.js"; import { validate } from "../middleware/validate.js"; @@ -182,6 +185,20 @@ function isManagedSandboxProvisioningConfigured(): boolean { } } +/** + * Whether the managed-config document declares the plugin entry this row's + * marker names. Fails closed like `isManagedSandboxProvisioningConfigured`. + */ +function isManagedPluginEnvironmentDeclared(marker: string): boolean { + try { + return (getManagedInstanceConfig()?.pluginEnvironments ?? []).some( + (spec) => managedPluginEnvironmentMarker(spec) === marker, + ); + } catch { + return true; + } +} + /** * Whether this row occupies a provisioner-owned environment slot whose * platform markers are LIVE state — i.e. some platform code path converges @@ -196,6 +213,9 @@ function isManagedSandboxProvisioningConfigured(): boolean { * owned only while a managed-sandbox bootstrap path is configured — * `ensureManagedSandboxEnvironment` then adopts and refreshes whichever * row holds the slot on every boot. + * - The row of a managed-config plugin entry (its + * `managedPluginEnvironment` marker) while the document declares that + * entry: `applyManagedEnvironments` converges the markers on it every boot. * - Any sandbox row bearing the legacy kubernetes marker while the * persisted instance execution policy forces kubernetes: * `findKubernetesEnvironment` selects marked rows (newest first) for @@ -214,6 +234,10 @@ async function isPlatformSlotEnvironment( options: { isForcedKubernetesExecution: () => Promise }, ): Promise { if (environment.driver === "local") return true; + if (environment.driver === "plugin") { + const pluginMarker = environment.metadata?.[MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY]; + return typeof pluginMarker === "string" && isManagedPluginEnvironmentDeclared(pluginMarker); + } if (environment.driver !== "sandbox") return false; if ( environment.metadata?.managedByPaperclip === true && diff --git a/server/src/services/managed-config.ts b/server/src/services/managed-config.ts index 0bdcb72a304..bea4e28ab6f 100644 --- a/server/src/services/managed-config.ts +++ b/server/src/services/managed-config.ts @@ -9,12 +9,23 @@ * "mode": "cloud", * "catalogVersion": "2026.720.0", * "features": { "": true | false, ... }, - * "plugins": { "autoInstall": ["daytona", "kubernetes"] }, + * "plugins": { "autoInstall": ["daytona", "remote-host"] }, * "environments": [ - * { "name": "Daytona", "provider": "daytona", "config": { "target": "us" } } + * { "name": "Daytona", "provider": "daytona", "config": { "target": "us" } }, + * { + * "name": "Remote Host", + * "driver": "plugin", + * "pluginKey": "example.remote-host-plugin", + * "driverKey": "tasks", + * "driverConfig": {} + * } * ] * } * + * An `environments` entry without `driver` is the single sandbox entry; any + * number of `driver: "plugin"` entries may sit beside it (see + * `MANAGED_ENVIRONMENT_DRIVERS`). + * * Parsing follows the `execution-policy-bootstrap.ts` doctrine: a pure * function over `Record`, strict, and fail * closed — bad JSON, an unsupported `v`, an unknown feature key, a feature @@ -54,8 +65,47 @@ export interface ManagedInstanceConfig { readonly workerEnv?: Readonly>; readonly enabledCompanies?: Readonly>; }; - /** Sandbox environments the control plane provisions at boot (empty when the section is absent). */ + /** + * The sandbox entry of the `environments` section: zero or one, since it + * provisions the single Paperclip-managed sandbox slot (empty when the + * section is absent or declares only plugin entries). + */ environments: readonly ManagedEnvironmentSpec[]; + /** The `driver: "plugin"` entries of the `environments` section (empty when none are declared). */ + pluginEnvironments: readonly ManagedPluginEnvironmentSpec[]; +} + +/** + * The environment drivers a managed-config `environments` entry may declare. + * An entry without `driver` is the sandbox entry, so documents written + * before plugin entries existed parse unchanged. + */ +export const MANAGED_ENVIRONMENT_DRIVERS = ["sandbox", "plugin"] as const; +export type ManagedEnvironmentDriver = (typeof MANAGED_ENVIRONMENT_DRIVERS)[number]; + +/** + * One declared managed plugin environment: an instance-level `plugin`-driver + * environment row served by an auto-installed plugin's environment driver. + * The row is + * identified by its `pluginKey`–`driverKey` pair, which boot stamps on the + * row as `metadata.managedPluginEnvironment`. + */ +export interface ManagedPluginEnvironmentSpec { + driver: "plugin"; + /** Display name of the instance-level environment row (unique per instance). */ + name: string; + description?: string; + /** + * The serving plugin's manifest id (for example + * `example.remote-host-plugin`), not its `plugins.autoInstall` catalog key. + * Boot requires a catalog entry + * whose `key` is in `plugins.autoInstall` to install this id. + */ + pluginKey: string; + /** The plugin's environment driver key. */ + driverKey: string; + /** Driver config, validated by the plugin at ensure time; never carries secrets. */ + driverConfig: Readonly>; } /** @@ -142,6 +192,155 @@ function describeJsonValue(value: unknown): string { return `${JSON.stringify(value)}`; } +const SANDBOX_ENTRY_KEYS = ["name", "description", "driver", "provider", "config"] as const; +const PLUGIN_ENTRY_KEYS = [ + "name", + "description", + "driver", + "pluginKey", + "driverKey", + "driverConfig", +] as const; +/** Same rule as the plugin environment config schema in `environment-config.ts`. */ +const PLUGIN_DRIVER_KEY_PATTERN = /^[a-z0-9][a-z0-9._-]*$/; + +function requireTrimmedString(value: unknown, path: string): string { + if (typeof value !== "string" || value.length === 0 || value.trim() !== value) { + fail(`"${path}" must be a non-empty string without surrounding whitespace (got ${describeJsonValue(value)})`); + } + return value; +} + +function readOptionalDescription(value: unknown, path: string): string | undefined { + if (value === undefined) return undefined; + if (typeof value !== "string" || value.trim().length === 0) { + fail(`"${path}" must be a non-empty string when present (got ${describeJsonValue(value)})`); + } + return value; +} + +function readSecretFreeObject(value: unknown, path: string): Record { + if (value === undefined) return {}; + if (!isPlainObject(value)) { + fail(`"${path}" must be an object (got ${describeJsonValue(value)})`); + } + const secretLikeKey = findSecretLikeConfigKey(value, ""); + if (secretLikeKey) { + fail( + `"${path}" key "${secretLikeKey}" looks secret-bearing; credentials are delivered to managed instances as process environment variables (the provider's documented env fallback), never in the managed-config document`, + ); + } + return { ...value }; +} + +/** + * Parse the `environments` section: at most one sandbox entry (no `driver`, + * or `driver: "sandbox"`) plus any number of `driver: "plugin"` entries. + * Names are unique across the section, and plugin entries are unique by + * their `pluginKey`–`driverKey` pair. + */ +function parseManagedEnvironmentsSection( + raw: unknown, + autoInstall: readonly string[], +): { sandbox: ManagedEnvironmentSpec[]; plugin: ManagedPluginEnvironmentSpec[] } { + const sandbox: ManagedEnvironmentSpec[] = []; + const plugin: ManagedPluginEnvironmentSpec[] = []; + if (raw === undefined) return { sandbox, plugin }; + if (!Array.isArray(raw)) { + fail(`"environments" must be an array of environment objects (got ${describeJsonValue(raw)})`); + } + const names = new Set(); + const pluginDrivers = new Set(); + for (const [index, entry] of raw.entries()) { + const at = `environments[${index}]`; + if (!isPlainObject(entry)) { + fail(`"${at}" must be an object (got ${describeJsonValue(entry)})`); + } + // Only an absent `driver` means the sandbox entry; an explicit `null` is malformed. + const driver = entry.driver === undefined ? "sandbox" : entry.driver; + if (!(MANAGED_ENVIRONMENT_DRIVERS as readonly unknown[]).includes(driver)) { + fail( + `"${at}.driver" must be one of ${MANAGED_ENVIRONMENT_DRIVERS.map((value) => JSON.stringify(value)).join(", ")} when present (got ${describeJsonValue(entry.driver)})`, + ); + } + const allowedKeys: readonly string[] = driver === "plugin" ? PLUGIN_ENTRY_KEYS : SANDBOX_ENTRY_KEYS; + for (const key of Object.keys(entry)) { + if (!allowedKeys.includes(key)) { + fail(`"${at}" has unknown key "${key}" (allowed for a ${driver} entry: ${allowedKeys.join(", ")})`); + } + } + const name = requireTrimmedString(entry.name, `${at}.name`); + if (names.has(name)) { + fail(`"${at}.name" duplicates another entry's name "${name}"; environment names are unique per instance`); + } + names.add(name); + const description = readOptionalDescription(entry.description, `${at}.description`); + + if (driver === "sandbox") { + // The DB enforces at most ONE Paperclip-managed sandbox row per instance + // (partial unique index `environments_managed_sandbox_idx`); every + // sandbox entry provisions that row, so a second one can never be + // satisfied. + if (sandbox.length > 0) { + fail( + `"environments" supports at most one sandbox entry: each sandbox entry provisions the single Paperclip-managed sandbox environment (DB invariant environments_managed_sandbox_idx)`, + ); + } + const provider = requireTrimmedString(entry.provider, `${at}.provider`); + // Coherence: on a managed instance the control plane is the only plugin + // install path, so an environment whose provider plugin is not + // auto-installed could never serve a lease. Catch the skew at parse time. + if (!autoInstall.includes(provider)) { + fail( + `"${at}.provider" is "${provider}", which is not in "plugins.autoInstall"; a managed environment requires its provider plugin to be provisioned`, + ); + } + if (isPlainObject(entry.config) && entry.config.provider !== undefined) { + fail(`"${at}.config" must not set "provider"; it is forced from the entry's provider key`); + } + const config = readSecretFreeObject(entry.config, `${at}.config`); + sandbox.push( + Object.freeze({ + name, + ...(description !== undefined ? { description } : {}), + provider, + config: Object.freeze(config), + }) as ManagedEnvironmentSpec, + ); + continue; + } + + // `pluginKey` is the plugin's manifest id (what the row's config stores), + // not a `plugins.autoInstall` catalog key, so this pure parser cannot map + // it. `applyManagedEnvironments` refuses startup when no auto-installed + // catalog entry installs this manifest id. + const pluginKey = requireTrimmedString(entry.pluginKey, `${at}.pluginKey`); + const driverKey = requireTrimmedString(entry.driverKey, `${at}.driverKey`); + if (!PLUGIN_DRIVER_KEY_PATTERN.test(driverKey)) { + fail( + `"${at}.driverKey" must start with a lowercase letter or digit and contain only lowercase letters, digits, dots, hyphens, or underscores (got ${describeJsonValue(driverKey)})`, + ); + } + const pair = `${pluginKey}:${driverKey}`; + if (pluginDrivers.has(pair)) { + fail(`"${at}" duplicates another plugin entry's pluginKey and driverKey ("${pair}"); each plugin environment driver is declared once`); + } + pluginDrivers.add(pair); + const driverConfig = readSecretFreeObject(entry.driverConfig, `${at}.driverConfig`); + plugin.push( + Object.freeze({ + driver: "plugin", + name, + ...(description !== undefined ? { description } : {}), + pluginKey, + driverKey, + driverConfig: Object.freeze(driverConfig), + }) as ManagedPluginEnvironmentSpec, + ); + } + return { sandbox, plugin }; +} + /** * Parse `PAPERCLIP_MANAGED_CONFIG` from a raw env map. Returns null only when * the variable is absent (self-hosted). Throws with a precise error on a @@ -320,79 +519,10 @@ export function parseManagedConfigEnv(env: ManagedConfigEnv): ManagedInstanceCon // absence is not a dropped security control — it simply declares no managed // environments. When present, the section is validated fail-closed like // everything else. - const environmentSpecs: ManagedEnvironmentSpec[] = []; - if (doc.environments !== undefined) { - if (!Array.isArray(doc.environments)) { - fail(`"environments" must be an array of environment objects (got ${describeJsonValue(doc.environments)})`); - } - // The DB enforces at most ONE Paperclip-managed sandbox row per instance - // (partial unique index `environments_managed_sandbox_idx`); every entry - // here provisions that row, so a longer list can never be satisfied. - if (doc.environments.length > 1) { - fail( - `"environments" supports at most one entry: each entry provisions the single Paperclip-managed sandbox environment (DB invariant environments_managed_sandbox_idx)`, - ); - } - for (const [index, entry] of doc.environments.entries()) { - if (!isPlainObject(entry)) { - fail(`"environments[${index}]" must be an object (got ${describeJsonValue(entry)})`); - } - const allowedEntryKeys = new Set(["name", "description", "provider", "config"]); - for (const key of Object.keys(entry)) { - if (!allowedEntryKeys.has(key)) { - fail(`"environments[${index}]" has unknown key "${key}" (allowed: name, description, provider, config)`); - } - } - if (typeof entry.name !== "string" || entry.name.length === 0 || entry.name.trim() !== entry.name) { - fail( - `"environments[${index}].name" must be a non-empty string without surrounding whitespace (got ${describeJsonValue(entry.name)})`, - ); - } - if ( - entry.description !== undefined - && (typeof entry.description !== "string" || entry.description.trim().length === 0) - ) { - fail(`"environments[${index}].description" must be a non-empty string when present (got ${describeJsonValue(entry.description)})`); - } - if (typeof entry.provider !== "string" || entry.provider.length === 0 || entry.provider.trim() !== entry.provider) { - fail( - `"environments[${index}].provider" must be a non-empty string without surrounding whitespace (got ${describeJsonValue(entry.provider)})`, - ); - } - // Coherence: on a managed instance the control plane is the only plugin - // install path, so an environment whose provider plugin is not - // auto-installed could never serve a lease. Catch the skew at parse time. - if (!autoInstall.includes(entry.provider)) { - fail( - `"environments[${index}].provider" is "${entry.provider}", which is not in "plugins.autoInstall"; a managed environment requires its provider plugin to be provisioned`, - ); - } - const config: Record = {}; - if (entry.config !== undefined) { - if (!isPlainObject(entry.config)) { - fail(`"environments[${index}].config" must be an object (got ${describeJsonValue(entry.config)})`); - } - if (entry.config.provider !== undefined) { - fail(`"environments[${index}].config" must not set "provider"; it is forced from the entry's provider key`); - } - const secretLikeKey = findSecretLikeConfigKey(entry.config, ""); - if (secretLikeKey) { - fail( - `"environments[${index}].config" key "${secretLikeKey}" looks secret-bearing; credentials are delivered to managed instances as process environment variables (the provider's documented env fallback), never in the managed-config document`, - ); - } - Object.assign(config, entry.config); - } - environmentSpecs.push( - Object.freeze({ - name: entry.name, - ...(entry.description !== undefined ? { description: entry.description } : {}), - provider: entry.provider, - config: Object.freeze(config), - }) as ManagedEnvironmentSpec, - ); - } - } + const { sandbox: environmentSpecs, plugin: pluginEnvironmentSpecs } = parseManagedEnvironmentsSection( + doc.environments, + autoInstall, + ); return Object.freeze({ v: SUPPORTED_MANAGED_CONFIG_VERSION, @@ -405,6 +535,7 @@ export function parseManagedConfigEnv(env: ManagedConfigEnv): ManagedInstanceCon ...(enabledCompanies ? { enabledCompanies: Object.freeze(enabledCompanies) } : {}), }), environments: Object.freeze(environmentSpecs), + pluginEnvironments: Object.freeze(pluginEnvironmentSpecs), }) as ManagedInstanceConfig; } diff --git a/server/src/services/managed-environments-plugin-entries.test.ts b/server/src/services/managed-environments-plugin-entries.test.ts new file mode 100644 index 00000000000..e9241b6728f --- /dev/null +++ b/server/src/services/managed-environments-plugin-entries.test.ts @@ -0,0 +1,385 @@ +import { describe, expect, it, vi } from "vitest"; +import { unprocessable } from "../errors.js"; +import { applyManagedEnvironments } from "./managed-environments.js"; +import { + PLUGIN_KEY, + environmentsSeam, + instanceSettingsSeam, + noDb, + parsedConfig, + pluginEnvironmentRow, + readyDriverResolver, + reconciliationResult, + recoveryHandle, + tick, + type InstanceSettingsSeam, + type RecoveryHandle, +} from "./managed-environments.test-support.js"; + +describe("applyManagedEnvironments plugin entries", () => { + const pluginEntry = (overrides: Record = {}) => ({ + name: "Remote Host", + description: "Runs the agent on a remote host", + driver: "plugin", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + driverConfig: {}, + ...overrides, + }); + const configWith = (environments: unknown[], features: Record = {}) => parsedConfig({ + plugins: { autoInstall: ["daytona", "remote-host"] }, + features, + environments, + }); + const readyPlugin = (status = "ready", declaresDriver = true) => vi.fn(async () => ({ + plugin: { id: "remote-host-plugin", pluginKey: PLUGIN_KEY, status }, + declaresDriver, + })); + const workerManager = (running = true, handle?: RecoveryHandle) => ({ + isRunning: vi.fn(() => running), + call: vi.fn(), + getWorker: vi.fn(() => handle), + }); + + it("ensures a plugin entry with the plugin validator's worker manager", async () => { + const environments = environmentsSeam(); + const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: workerManager(), + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + + expect(environments.ensureManagedPluginEnvironment).toHaveBeenCalledWith({ + name: "Remote Host", + description: "Runs the agent on a remote host", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + driverConfig: {}, + workerManager: { isRunning: expect.any(Function), call: expect.any(Function) }, + }); + expect(result).toMatchObject({ ensured: 1, added: 1, failed: 0 }); + expect(result?.pluginOutcomes).toEqual([{ + environmentId: "plugin-env", + name: "Remote Host", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + action: "added", + }]); + expect(environments.ensureManagedSandboxEnvironment).not.toHaveBeenCalled(); + }); + + it("runs the sandbox entry first", async () => { + const order: string[] = []; + const environments = environmentsSeam({ + ensureManagedSandboxEnvironment: vi.fn(async () => { + order.push("sandbox"); + return reconciliationResult() as never; + }), + ensureManagedPluginEnvironment: vi.fn(async () => { + order.push("plugin"); + return { environment: pluginEnvironmentRow(), action: "added" as const }; + }), + }); + await applyManagedEnvironments(noDb, configWith([{ name: "Daytona", provider: "daytona" }, pluginEntry()]), { + env: {}, + workerManager: workerManager(), + environments, + instanceSettings: instanceSettingsSeam(), + resolveSandboxProviderDriver: readyDriverResolver(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + expect(order).toEqual(["sandbox", "plugin"]); + }); + + it("refuses startup when no auto-installed catalog entry installs the entry's plugin", async () => { + const environments = environmentsSeam(); + await expect(applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: workerManager(), + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: ["paperclip.daytona-sandbox-provider"], + })).rejects.toThrow(/uses plugin "example.remote-host-plugin", which no "plugins.autoInstall" catalog entry installs/); + expect(environments.ensureManagedPluginEnvironment).not.toHaveBeenCalled(); + }); + + it("resolves the auto-installed manifest ids from the catalog when the caller passes none", async () => { + // "remote-host" is not in this build's bundled catalog and the test root + // has no distribution catalog, so the catalog lookup refuses. + await expect(applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: { PAPERCLIP_BUNDLED_PLUGIN_ROOT: "/nonexistent-paperclip-catalog" }, + workerManager: workerManager(), + environments: environmentsSeam(), + instanceSettings: instanceSettingsSeam(), + })).rejects.toThrow(/auto-install key "remote-host" is not in the bundled catalog/); + }); + + it("does not claim the sandbox slot, so a forced execution mode can coexist with plugin-only entries", async () => { + await expect(applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: { PAPERCLIP_EXECUTION_MODE: "kubernetes" }, + workerManager: workerManager(), + environments: environmentsSeam(), + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + })).resolves.toMatchObject({ ensured: 1 }); + }); + + it("archives the row of a plugin that is down and reactivates it on worker recovery", async () => { + const { handle, fireReady } = recoveryHandle(); + const environments = environmentsSeam({ + archiveManagedPluginEnvironment: vi.fn().mockResolvedValue(pluginEnvironmentRow({ status: "archived" })), + }); + const manager = workerManager(false, handle); + const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: manager, + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + + expect(result).toMatchObject({ ensured: 0, failed: 1 }); + expect(environments.archiveManagedPluginEnvironment).toHaveBeenCalledWith({ pluginKey: PLUGIN_KEY, driverKey: "tasks" }); + expect(environments.ensureManagedPluginEnvironment).not.toHaveBeenCalled(); + + manager.isRunning.mockReturnValue(true); + fireReady(); + fireReady(); + await tick(); + + expect(environments.ensureManagedPluginEnvironment).toHaveBeenCalledTimes(1); + expect(handle.off).toHaveBeenCalledTimes(1); + }); + + it("keeps recovery armed after a worker failure and stops after one success", async () => { + const { handle, fireReady } = recoveryHandle(); + const ensureManagedPluginEnvironment = vi.fn() + .mockRejectedValueOnce(new Error("worker crashed during environmentValidateConfig")) + .mockResolvedValue({ environment: pluginEnvironmentRow(), action: "updated" }); + const environments = environmentsSeam({ ensureManagedPluginEnvironment }); + const manager = workerManager(false, handle); + await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: manager, + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + manager.isRunning.mockReturnValue(true); + + fireReady(); + await tick(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(1); + expect(handle.off).not.toHaveBeenCalled(); + + fireReady(); + await tick(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(2); + expect(handle.off).toHaveBeenCalledTimes(1); + + fireReady(); + await tick(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(2); + }); + + it("queues one more attempt when the worker becomes ready during an attempt", async () => { + const { handle, fireReady } = recoveryHandle(); + let failFirst!: (err: Error) => void; + const ensureManagedPluginEnvironment = vi.fn() + .mockImplementationOnce(() => new Promise((_resolve, reject) => { failFirst = reject; })) + .mockResolvedValue({ environment: pluginEnvironmentRow(), action: "updated" }); + const manager = workerManager(false, handle); + await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: manager, + environments: environmentsSeam({ ensureManagedPluginEnvironment }), + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + manager.isRunning.mockReturnValue(true); + + fireReady(); + fireReady(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(1); + failFirst(new Error("worker crashed")); + await tick(); + await tick(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(2); + expect(handle.off).toHaveBeenCalledTimes(1); + }); + + it("stops recovery when the plugin permanently rejects the config", async () => { + const { handle, fireReady } = recoveryHandle(); + const ensureManagedPluginEnvironment = vi.fn().mockRejectedValue(unprocessable("driverConfig.region is not supported")); + const manager = workerManager(false, handle); + await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: manager, + environments: environmentsSeam({ ensureManagedPluginEnvironment }), + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + manager.isRunning.mockReturnValue(true); + + fireReady(); + await tick(); + fireReady(); + await tick(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(1); + expect(handle.off).toHaveBeenCalledTimes(1); + }); + + it("archives and arms recovery when the worker dies during the boot ensure", async () => { + const { handle, fireReady } = recoveryHandle(); + const manager = workerManager(true, handle); + const ensureManagedPluginEnvironment = vi.fn() + .mockImplementationOnce(async () => { + manager.isRunning.mockReturnValue(false); + throw new Error("worker crashed during environmentValidateConfig"); + }) + .mockResolvedValue({ environment: pluginEnvironmentRow(), action: "updated" }); + const environments = environmentsSeam({ ensureManagedPluginEnvironment }); + const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: manager, + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + await tick(); + + expect(result).toMatchObject({ ensured: 0, failed: 1 }); + expect(environments.archiveManagedPluginEnvironment).toHaveBeenCalledWith({ pluginKey: PLUGIN_KEY, driverKey: "tasks" }); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(1); + + manager.isRunning.mockReturnValue(true); + fireReady(); + await tick(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(2); + expect(handle.off).toHaveBeenCalledTimes(1); + }); + + it("keeps the row and retries when the ensure fails while the worker keeps running", async () => { + const { handle, fireReady } = recoveryHandle(); + const ensureManagedPluginEnvironment = vi.fn() + .mockRejectedValueOnce(new Error("environmentValidateConfig timed out")) + .mockRejectedValueOnce(new Error("environmentValidateConfig timed out")) + .mockResolvedValue({ environment: pluginEnvironmentRow(), action: "unchanged" }); + const environments = environmentsSeam({ ensureManagedPluginEnvironment }); + const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: workerManager(true, handle), + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + await tick(); + + expect(result).toMatchObject({ ensured: 0, failed: 1 }); + expect(environments.archiveManagedPluginEnvironment).not.toHaveBeenCalled(); + // The immediate retry also failed; the next `ready` succeeds. + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(2); + fireReady(); + await tick(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(3); + expect(handle.off).toHaveBeenCalledTimes(1); + }); + + it("neither archives nor retries when the boot ensure gets a permanent config rejection", async () => { + const { handle } = recoveryHandle(); + const environments = environmentsSeam({ + ensureManagedPluginEnvironment: vi.fn().mockRejectedValue(unprocessable("driverConfig.region is not supported")), + }); + const manager = workerManager(true, handle); + const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: manager, + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + expect(result).toMatchObject({ ensured: 0, failed: 1 }); + expect(environments.archiveManagedPluginEnvironment).not.toHaveBeenCalled(); + expect(manager.getWorker).not.toHaveBeenCalled(); + }); + + it("treats a missing plugin, a plugin without the driver, or a worker manager without call as unavailable", async () => { + for (const [resolveEnvironmentPlugin, manager] of [ + [vi.fn(async () => null), workerManager()], + [readyPlugin("ready", false), workerManager()], + [readyPlugin("error"), workerManager()], + [readyPlugin(), { isRunning: vi.fn(() => true), getWorker: vi.fn(() => undefined) }], + ] as const) { + const environments = environmentsSeam(); + const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: manager, + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin, + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + expect(result).toMatchObject({ ensured: 0, failed: 1 }); + expect(environments.archiveManagedPluginEnvironment).toHaveBeenCalledTimes(1); + expect(environments.ensureManagedPluginEnvironment).not.toHaveBeenCalled(); + expect(manager.getWorker).not.toHaveBeenCalled(); + } + }); + + it("never archives a row whose entry was removed from the document", async () => { + const environments = environmentsSeam(); + expect(await applyManagedEnvironments(noDb, configWith([]), { + env: {}, + workerManager: workerManager(), + environments, + instanceSettings: instanceSettingsSeam(), + })).toBeNull(); + expect(environments.archiveManagedPluginEnvironment).not.toHaveBeenCalled(); + }); + + it("never changes the instance default, and the sandbox stamp never replaces a plugin default", async () => { + const settings = instanceSettingsSeam({ + get: vi.fn().mockResolvedValue({ defaultEnvironmentId: "plugin-env" }) as InstanceSettingsSeam["get"], + }); + await applyManagedEnvironments(noDb, configWith( + [{ name: "Daytona", provider: "daytona" }, pluginEntry()], + { enableManagedSandboxOnly: true }, + ), { + env: {}, + workerManager: workerManager(), + instanceSettings: settings, + environments: environmentsSeam({ getById: vi.fn().mockResolvedValue(pluginEnvironmentRow()) }), + resolveSandboxProviderDriver: readyDriverResolver(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + expect(settings.update).not.toHaveBeenCalled(); + }); + + it("is fail-safe: an ensure failure is logged and counted, not thrown", async () => { + const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: workerManager(), + environments: environmentsSeam({ + ensureManagedPluginEnvironment: vi.fn().mockRejectedValue(new Error("validator refused")), + }), + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + expect(result).toMatchObject({ ensured: 0, failed: 1 }); + }); +}); diff --git a/server/src/services/managed-environments.test-support.ts b/server/src/services/managed-environments.test-support.ts new file mode 100644 index 00000000000..a3019692692 --- /dev/null +++ b/server/src/services/managed-environments.test-support.ts @@ -0,0 +1,139 @@ +/** + * Shared fixtures and service seams for the `applyManagedEnvironments` tests + * (`managed-environments.test.ts` for the sandbox entry and instance default, + * `managed-environments-plugin-entries.test.ts` for plugin entries). + */ +import { vi } from "vitest"; +import type { Db } from "@paperclipai/db"; +import { MANAGED_CONFIG_ENV_KEY, parseManagedConfigEnv } from "./managed-config.js"; +import type { ApplyManagedEnvironmentsOptions } from "./managed-environments.js"; + +export const noDb = null as unknown as Db; + +export function parsedConfig(overrides: Record = {}) { + const config = parseManagedConfigEnv({ + [MANAGED_CONFIG_ENV_KEY]: JSON.stringify({ + v: 1, + mode: "cloud", + catalogVersion: "2026.720.0", + features: {}, + plugins: { autoInstall: ["daytona"] }, + ...overrides, + }), + }); + if (!config) throw new Error("expected a parsed managed config"); + return config; +} + +export function environmentRow(overrides: Record = {}) { + const now = new Date(); + return { + id: "env-1", + name: "Daytona", + description: null, + driver: "sandbox", + status: "active", + config: {}, + envVars: {}, + metadata: {}, + createdAt: now, + updatedAt: now, + ...overrides, + }; +} + +export function reconciliationResult( + overrides: Record = {}, +) { + return { + environment: environmentRow(), + action: "added" as const, + stockStatus: "missing" as const, + updateAvailable: false, + stockHash: "sha256:stock", + ...overrides, + }; +} + +export function readyDriverResolver(status = "ready") { + return vi.fn(async () => ({ + plugin: { id: "plugin-1", pluginKey: "sandbox-providers/daytona", status }, + })); +} + +export function runningWorkerManager(running = true) { + return { isRunning: vi.fn(() => running), getWorker: vi.fn(() => undefined) }; +} + +export type WorkerManagerSeam = NonNullable; +export type RecoveryHandle = NonNullable>; + +/** A worker handle fake that records `ready` listeners so tests can fire them. */ +export function recoveryHandle() { + const readyListeners: Array<(payload: { pluginId: string }) => void> = []; + const handle: RecoveryHandle = { + on: vi.fn((_event, listener) => { + readyListeners.push(listener); + }), + off: vi.fn((_event, listener) => { + const at = readyListeners.indexOf(listener); + if (at !== -1) readyListeners.splice(at, 1); + }), + }; + return { + handle, + fireReady: () => { + for (const listener of [...readyListeners]) listener({ pluginId: "plugin-1" }); + }, + }; +} + +/** Lets the fire-and-forget recovery ensure settle. */ +export function tick() { + return new Promise((resolve) => setImmediate(resolve)); +} + +export type EnvironmentsSeam = NonNullable; +export type InstanceSettingsSeam = NonNullable; + +export function environmentsSeam(overrides: Partial = {}): EnvironmentsSeam { + return { + ensureManagedSandboxEnvironment: + overrides.ensureManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(reconciliationResult()), + archiveManagedSandboxEnvironment: + overrides.archiveManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(null), + getById: overrides.getById ?? vi.fn().mockResolvedValue(null), + update: overrides.update ?? vi.fn().mockResolvedValue(environmentRow()), + findManagedSandboxEnvironment: + overrides.findManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(null), + ensureManagedPluginEnvironment: + overrides.ensureManagedPluginEnvironment + ?? vi.fn().mockResolvedValue({ environment: pluginEnvironmentRow(), action: "added" }), + archiveManagedPluginEnvironment: + overrides.archiveManagedPluginEnvironment ?? vi.fn().mockResolvedValue(null), + }; +} + +export const PLUGIN_KEY = "example.remote-host-plugin"; + +export function pluginEnvironmentRow(overrides: Record = {}) { + return environmentRow({ + id: "plugin-env", + name: "Remote Host", + driver: "plugin", + config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: {} }, + metadata: { managedByPaperclip: true, managedPluginEnvironment: `${PLUGIN_KEY}:tasks` }, + ...overrides, + }) as never; +} + +export function instanceSettingsSeam(overrides: Partial = {}): InstanceSettingsSeam { + return { + get: + overrides.get ?? + (vi.fn().mockResolvedValue({ defaultEnvironmentId: null }) as InstanceSettingsSeam["get"]), + update: + overrides.update ?? + (vi.fn().mockResolvedValue({ defaultEnvironmentId: "env-1" }) as InstanceSettingsSeam["update"]), + }; +} diff --git a/server/src/services/managed-environments.test.ts b/server/src/services/managed-environments.test.ts index 2a38e6c076d..bf8ec41e6bc 100644 --- a/server/src/services/managed-environments.test.ts +++ b/server/src/services/managed-environments.test.ts @@ -1,122 +1,18 @@ import { describe, expect, it, vi } from "vitest"; -import type { Db } from "@paperclipai/db"; -import { MANAGED_CONFIG_ENV_KEY, parseManagedConfigEnv } from "./managed-config.js"; +import { applyManagedEnvironments } from "./managed-environments.js"; import { - applyManagedEnvironments, - type ApplyManagedEnvironmentsOptions, -} from "./managed-environments.js"; - -const noDb = null as unknown as Db; - -function parsedConfig(overrides: Record = {}) { - const config = parseManagedConfigEnv({ - [MANAGED_CONFIG_ENV_KEY]: JSON.stringify({ - v: 1, - mode: "cloud", - catalogVersion: "2026.720.0", - features: {}, - plugins: { autoInstall: ["daytona"] }, - ...overrides, - }), - }); - if (!config) throw new Error("expected a parsed managed config"); - return config; -} - -function environmentRow(overrides: Record = {}) { - const now = new Date(); - return { - id: "env-1", - name: "Daytona", - description: null, - driver: "sandbox", - status: "active", - config: {}, - envVars: {}, - metadata: {}, - createdAt: now, - updatedAt: now, - ...overrides, - }; -} - -function reconciliationResult( - overrides: Record = {}, -) { - return { - environment: environmentRow(), - action: "added" as const, - stockStatus: "missing" as const, - updateAvailable: false, - stockHash: "sha256:stock", - ...overrides, - }; -} - -function readyDriverResolver(status = "ready") { - return vi.fn(async () => ({ - plugin: { id: "plugin-1", pluginKey: "sandbox-providers/daytona", status }, - })); -} - -function runningWorkerManager(running = true) { - return { isRunning: vi.fn(() => running), getWorker: vi.fn(() => undefined) }; -} - -type WorkerManagerSeam = NonNullable; -type RecoveryHandle = NonNullable>; - -/** A worker handle fake that records `ready` listeners so tests can fire them. */ -function recoveryHandle() { - const readyListeners: Array<(payload: { pluginId: string }) => void> = []; - const handle: RecoveryHandle = { - on: vi.fn((_event, listener) => { - readyListeners.push(listener); - }), - off: vi.fn((_event, listener) => { - const at = readyListeners.indexOf(listener); - if (at !== -1) readyListeners.splice(at, 1); - }), - }; - return { - handle, - fireReady: () => { - for (const listener of [...readyListeners]) listener({ pluginId: "plugin-1" }); - }, - }; -} - -/** Lets the fire-and-forget recovery ensure settle. */ -function tick() { - return new Promise((resolve) => setImmediate(resolve)); -} - -type EnvironmentsSeam = NonNullable; -type InstanceSettingsSeam = NonNullable; - -function environmentsSeam(overrides: Partial = {}): EnvironmentsSeam { - return { - ensureManagedSandboxEnvironment: - overrides.ensureManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(reconciliationResult()), - archiveManagedSandboxEnvironment: - overrides.archiveManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(null), - getById: overrides.getById ?? vi.fn().mockResolvedValue(null), - update: overrides.update ?? vi.fn().mockResolvedValue(environmentRow()), - findManagedSandboxEnvironment: - overrides.findManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(null), - }; -} - -function instanceSettingsSeam(overrides: Partial = {}): InstanceSettingsSeam { - return { - get: - overrides.get ?? - (vi.fn().mockResolvedValue({ defaultEnvironmentId: null }) as InstanceSettingsSeam["get"]), - update: - overrides.update ?? - (vi.fn().mockResolvedValue({ defaultEnvironmentId: "env-1" }) as InstanceSettingsSeam["update"]), - }; -} + environmentRow, + environmentsSeam, + instanceSettingsSeam, + noDb, + parsedConfig, + readyDriverResolver, + reconciliationResult, + recoveryHandle, + runningWorkerManager, + tick, + type InstanceSettingsSeam, +} from "./managed-environments.test-support.js"; describe("applyManagedEnvironments", () => { it("no-ops for self-hosted instances and for documents without environments", async () => { diff --git a/server/src/services/managed-environments.ts b/server/src/services/managed-environments.ts index d201728cb7a..36d9c515c3a 100644 --- a/server/src/services/managed-environments.ts +++ b/server/src/services/managed-environments.ts @@ -43,9 +43,17 @@ * sandbox provider falls back to its documented process environment variable * (e.g. `DAYTONA_API_KEY`) when `config` omits the key, so the deployment * delivers secrets as env vars and the managed document stays secret-free. + * + * `driver: "plugin"` entries take the same readiness path after the sandbox + * entry: await `pluginsReady`, require the plugin `ready`, declaring the + * entry's driver, and running a worker; else archive the entry's row and + * re-ensure it on the worker's `ready`. Each entry owns one row (see + * `managed-plugin-environments.ts`). Plugin entries never touch the instance + * default. */ import type { Db } from "@paperclipai/db"; +import { HttpError } from "../errors.js"; import { logger } from "../middleware/logger.js"; import { environmentService } from "./environments.js"; import type { ManagedSandboxEnvironmentReconcileAction } from "./environments.js"; @@ -54,7 +62,10 @@ import type { ManagedInstanceConfig } from "./managed-config.js"; import type { ManagedResourceStockStatus } from "./managed-resource-drift.js"; import { parseExecutionPolicyBootstrapEnv } from "./execution-policy-bootstrap.js"; import { resolvePluginSandboxProviderDriverByKey } from "./plugin-environment-driver.js"; +import { pluginRegistryService } from "./plugin-registry.js"; +import { resolveBundledCatalogRoot, resolveBundledPluginInstalls } from "./bundled-plugins.js"; import type { PluginWorkerManager } from "./plugin-worker-manager.js"; +import type { ManagedPluginEnvironmentReconcileAction } from "./managed-plugin-environments.js"; export interface ApplyManagedEnvironmentsOptions { env?: Record; @@ -75,8 +86,11 @@ export interface ApplyManagedEnvironmentsOptions { * subscription used for post-archive recovery: a worker in crash-restart * backoff re-emits `ready` when the manager respawns it, and the listener * re-ensures the archived environment (see `applyManagedEnvironments`). + * + * Plugin entries also need `call`, to run the plugin's own config + * validation; without it every plugin entry is treated as unavailable. */ - workerManager?: Pick & { + workerManager?: Pick & Partial> & { getWorker(pluginId: string): | { on(event: "ready", listener: (payload: { pluginId: string }) => void): void; @@ -92,6 +106,8 @@ export interface ApplyManagedEnvironmentsOptions { | "getById" | "update" | "findManagedSandboxEnvironment" + | "ensureManagedPluginEnvironment" + | "archiveManagedPluginEnvironment" >; /** Test seam: overrides the instance-settings service built from `db`. */ instanceSettings?: Pick, "get" | "update">; @@ -100,6 +116,32 @@ export interface ApplyManagedEnvironmentsOptions { db: Db; driverKey: string; }) => Promise<{ plugin: { id: string; pluginKey: string; status: string } } | null>; + /** + * Test seam: overrides the lookup of a plugin entry's plugin. Returns null + * when the plugin is not installed; `declaresDriver` is false when its + * manifest does not declare the entry's environment driver. + */ + resolveEnvironmentPlugin?: (input: { + db: Db; + pluginKey: string; + driverKey: string; + }) => Promise<{ plugin: { id: string; pluginKey: string; status: string }; declaresDriver: boolean } | null>; + /** + * Manifest ids (`pluginKey`) that the `plugins.autoInstall` catalog keys + * resolve to: `app.locals.managedBundledPluginKeys`, from + * `resolveBundledPluginInstalls` over the bundled and distribution + * catalogs. A plugin entry whose `pluginKey` is not in this list refuses + * startup. When omitted, the list is resolved here the same way. + */ + autoInstalledPluginKeys?: readonly string[]; +} + +export interface ManagedPluginEnvironmentReconciliationOutcome { + environmentId: string; + name: string; + pluginKey: string; + driverKey: string; + action: ManagedPluginEnvironmentReconcileAction; } export interface ManagedEnvironmentReconciliationOutcome { @@ -123,6 +165,28 @@ export interface ApplyManagedEnvironmentsResult { /** Preserve-only reconciliation never creates replacement backups. */ backedUp: number; outcomes: ManagedEnvironmentReconciliationOutcome[]; + pluginOutcomes: ManagedPluginEnvironmentReconciliationOutcome[]; +} + +/** + * The plugin refused the entry's config (`HttpError` 422 from + * `normalizeEnvironmentConfig` or `validatePluginEnvironmentDriverConfig`). + * Retrying cannot help until the config changes. Every other failure, such + * as a worker that crashed mid-call, is treated as transient. + */ +function isPermanentConfigRejection(err: unknown): boolean { + return err instanceof HttpError && err.status === 422; +} + +async function resolveEnvironmentPluginByKey(input: { db: Db; pluginKey: string; driverKey: string }) { + const plugin = await pluginRegistryService(input.db).getByKey(input.pluginKey); + if (!plugin) return null; + return { + plugin: { id: plugin.id, pluginKey: plugin.pluginKey, status: plugin.status }, + declaresDriver: plugin.manifestJson.environmentDrivers?.some( + (driver) => driver.driverKey === input.driverKey && driver.kind !== "sandbox_provider", + ) ?? false, + }; } /** @@ -176,19 +240,38 @@ export async function applyManagedEnvironments( } } - if (managedConfig.environments.length === 0) return null; + const pluginSpecs = managedConfig.pluginEnvironments; + if (managedConfig.environments.length === 0 && pluginSpecs.length === 0) return null; // The forced-execution-mode bootstrap (`PAPERCLIP_EXECUTION_MODE=kubernetes`) // and this one both own the single Paperclip-managed sandbox row // (`environments_managed_sandbox_idx`). Configuring both is contradictory; - // refuse startup rather than let bootstrap ordering pick a winner. + // refuse startup rather than let bootstrap ordering pick a winner. Plugin + // entries do not touch that slot. const env = opts.env ?? process.env; - if (parseExecutionPolicyBootstrapEnv(env)) { + if (managedConfig.environments.length > 0 && parseExecutionPolicyBootstrapEnv(env)) { throw new Error( `PAPERCLIP_EXECUTION_MODE and the PAPERCLIP_MANAGED_CONFIG "environments" section are mutually exclusive: both manage the single instance sandbox environment`, ); } + // Coherence, fail closed like the parser: a plugin entry names its plugin by + // manifest id, and only `plugins.autoInstall` provisions plugins on a + // managed instance, so some auto-installed catalog entry must install it. + if (pluginSpecs.length > 0) { + const autoInstalledPluginKeys = opts.autoInstalledPluginKeys ?? resolveBundledPluginInstalls( + managedConfig.plugins.autoInstall, + { catalogRoot: resolveBundledCatalogRoot(env), env, enforceCatalogRoot: true }, + ).map((install) => install.pluginKey); + for (const spec of pluginSpecs) { + if (!autoInstalledPluginKeys.includes(spec.pluginKey)) { + throw new Error( + `PAPERCLIP_MANAGED_CONFIG "environments" entry "${spec.name}" uses plugin "${spec.pluginKey}", which no "plugins.autoInstall" catalog entry installs; a managed plugin environment requires its plugin to be provisioned`, + ); + } + } + } + // The heartbeat resumes queued runs right after this bootstrap step, and // lease acquisition fails hard on a provider whose plugin is missing or not // ready. Wait for the bundled-plugin startup pass to finish, then refuse to @@ -266,10 +349,7 @@ export async function applyManagedEnvironments( // worker recovered between the gate check and the subscription. No handle // means no respawn is coming (activation never started a worker), so the // degraded-until-next-boot posture stands. - const scheduleRecoveryReactivation = ( - spec: ManagedInstanceConfig["environments"][number], - pluginId: string, - ): void => { + const scheduleRecoveryReactivation = (pluginId: string, reactivateEntry: () => Promise): void => { const manager = opts.workerManager; const handle = manager?.getWorker(pluginId); if (!handle) return; @@ -278,39 +358,41 @@ export async function applyManagedEnvironments( if (reactivated) return; reactivated = true; handle.off("ready", reactivate); - void environments - .ensureManagedSandboxEnvironment({ - name: spec.name, - description: spec.description, - provider: spec.provider, - config: { ...spec.config }, - stockVersion: managedConfig.catalogVersion, - platformFullyManaged: true, - }) - .then((result) => { - logger.info( - { - environmentId: result.environment.id, - name: spec.name, - provider: spec.provider, - action: result.action, - stockStatus: result.stockStatus, - updateAvailable: result.updateAvailable, - }, - "managed sandbox environment reactivated after provider worker recovery", - ); - }) - .catch((err: unknown) => { - logger.error( - { err, name: spec.name, provider: spec.provider }, - "failed to reactivate managed sandbox environment after provider worker recovery (degraded: environment unavailable until next boot)", - ); - }); + void reactivateEntry(); }; handle.on("ready", reactivate); if (manager?.isRunning(pluginId)) reactivate(); }; + const reactivateSandboxEntry = async (spec: ManagedInstanceConfig["environments"][number]): Promise => { + try { + const result = await environments.ensureManagedSandboxEnvironment({ + name: spec.name, + description: spec.description, + provider: spec.provider, + config: { ...spec.config }, + stockVersion: managedConfig.catalogVersion, + platformFullyManaged: true, + }); + logger.info( + { + environmentId: result.environment.id, + name: spec.name, + provider: spec.provider, + action: result.action, + stockStatus: result.stockStatus, + updateAvailable: result.updateAvailable, + }, + "managed sandbox environment reactivated after provider worker recovery", + ); + } catch (err) { + logger.error( + { err, name: spec.name, provider: spec.provider }, + "failed to reactivate managed sandbox environment after provider worker recovery (degraded: environment unavailable until next boot)", + ); + } + }; + let ensured = 0; let failed = 0; let added = 0; @@ -355,7 +437,7 @@ export async function applyManagedEnvironments( // manager restarts crashed workers, but nothing (re)installs a missing // or non-ready plugin at runtime. if (resolved && resolved.plugin.status === "ready") { - scheduleRecoveryReactivation(spec, resolved.plugin.id); + scheduleRecoveryReactivation(resolved.plugin.id, () => reactivateSandboxEntry(spec)); } continue; } @@ -407,6 +489,179 @@ export async function applyManagedEnvironments( ); } } + const resolvePlugin = opts.resolveEnvironmentPlugin ?? resolveEnvironmentPluginByKey; + const pluginOutcomes: ManagedPluginEnvironmentReconciliationOutcome[] = []; + + const ensurePluginEntry = ( + spec: ManagedInstanceConfig["pluginEnvironments"][number], + workerManager: Pick, + ) => + environments.ensureManagedPluginEnvironment({ + name: spec.name, + description: spec.description, + pluginKey: spec.pluginKey, + driverKey: spec.driverKey, + driverConfig: { ...spec.driverConfig }, + workerManager, + }); + + type PluginSpec = ManagedInstanceConfig["pluginEnvironments"][number]; + type PluginWorkerAccess = Pick; + + const archivePluginEntry = (spec: PluginSpec) => + environments + .archiveManagedPluginEnvironment({ pluginKey: spec.pluginKey, driverKey: spec.driverKey }) + .catch((archiveErr: unknown) => { + logger.error( + { err: archiveErr, name: spec.name, pluginKey: spec.pluginKey, driverKey: spec.driverKey }, + "failed to archive the managed plugin environment of an unavailable plugin", + ); + return null; + }); + + /** + * Re-ensure a plugin entry each time its worker reports `ready`, until one + * ensure succeeds or the plugin permanently rejects the entry's config. A + * worker that crashes during an attempt (for example inside + * `environmentValidateConfig`) leaves the listener armed, so its next + * respawn retries. A `ready` that arrives during an attempt queues one more + * attempt. No handle means no respawn is coming (activation never started a + * worker), so the degraded-until-next-boot posture stands. + */ + const schedulePluginRecovery = (spec: PluginSpec, pluginId: string, workerAccess: PluginWorkerAccess): void => { + const manager = opts.workerManager; + const handle = manager?.getWorker(pluginId); + if (!handle) return; + const logBase = { name: spec.name, pluginKey: spec.pluginKey, driverKey: spec.driverKey }; + let settled = false; + let inFlight = false; + let queued = false; + const settle = (): void => { + settled = true; + handle.off("ready", attempt); + }; + function attempt(): void { + if (settled) return; + if (inFlight) { + queued = true; + return; + } + inFlight = true; + void ensurePluginEntry(spec, workerAccess) + .then((result) => { + settle(); + logger.info( + { ...logBase, environmentId: result.environment.id, action: result.action }, + "managed plugin environment reactivated after plugin worker recovery", + ); + }) + .catch((err: unknown) => { + if (isPermanentConfigRejection(err)) { + settle(); + logger.error( + { err, ...logBase }, + "the plugin rejected the managed plugin environment's config; not retrying (degraded: environment unavailable until the config changes)", + ); + } else { + logger.warn( + { err, ...logBase }, + "failed to reactivate managed plugin environment; retrying when the plugin worker is ready again", + ); + } + }) + .finally(() => { + inFlight = false; + if (queued && !settled) { + queued = false; + attempt(); + } + }); + } + handle.on("ready", attempt); + if (manager?.isRunning(pluginId)) attempt(); + }; + + for (const spec of pluginSpecs) { + const logBase = { name: spec.name, pluginKey: spec.pluginKey, driverKey: spec.driverKey }; + let recoverable: { pluginId: string; workerAccess: PluginWorkerAccess } | null = null; + try { + const resolved = await resolvePlugin({ db, pluginKey: spec.pluginKey, driverKey: spec.driverKey }); + const manager = opts.workerManager; + const call = manager?.call; + // Only a `ready` record that declares the driver can recover without + // another boot: the worker manager restarts crashed workers, but nothing + // (re)installs a missing or non-ready plugin at runtime. + if (resolved && resolved.plugin.status === "ready" && resolved.declaresDriver && manager && call) { + recoverable = { + pluginId: resolved.plugin.id, + workerAccess: { isRunning: manager.isRunning.bind(manager), call: call.bind(manager) }, + }; + } + const workerRunning = resolved != null && (manager?.isRunning(resolved.plugin.id) ?? false); + if (!recoverable || !workerRunning) { + failed += 1; + const archived = await archivePluginEntry(spec); + logger.error( + { + ...logBase, + pluginStatus: resolved?.plugin.status ?? null, + declaresDriver: resolved?.declaresDriver ?? null, + workerRunning, + archivedEnvironmentId: archived?.id ?? null, + }, + "managed plugin environment's plugin is not installed, ready, declaring its driver, and running; skipping ensure and archiving any previously provisioned row (degraded: environment unavailable)", + ); + if (recoverable) schedulePluginRecovery(spec, recoverable.pluginId, recoverable.workerAccess); + continue; + } + const result = await ensurePluginEntry(spec, recoverable.workerAccess); + ensured += 1; + if (result.action === "added") added += 1; + else if (result.action === "unchanged") unchanged += 1; + else updated += 1; + pluginOutcomes.push({ + environmentId: result.environment.id, + name: result.environment.name, + pluginKey: spec.pluginKey, + driverKey: spec.driverKey, + action: result.action, + }); + logger.info( + { ...logBase, environmentId: result.environment.id, action: result.action }, + "managed plugin environment reconciled", + ); + } catch (err) { + failed += 1; + if (recoverable && !isPermanentConfigRejection(err)) { + if (!recoverable.workerAccess.isRunning(recoverable.pluginId)) { + // The worker died after the readiness check (for example inside + // `environmentValidateConfig`): treat it like a plugin that is + // down, so the row is not left active, and retry on its `ready`. + const archived = await archivePluginEntry(spec); + logger.error( + { err, ...logBase, archivedEnvironmentId: archived?.id ?? null }, + "managed plugin environment's worker stopped during ensure; archived any previously provisioned row and retrying when the plugin worker is ready again (degraded: environment unavailable)", + ); + } else { + // The worker is still running (for example the call timed out). + // Keep any previously provisioned row as it is, so agents keep a + // working environment; retry once now and on the worker's next + // `ready`, and otherwise at the next boot. + logger.error( + { err, ...logBase }, + "failed to ensure managed plugin environment while its worker is running; keeping any previously provisioned row and retrying (degraded: declaration not applied)", + ); + } + schedulePluginRecovery(spec, recoverable.pluginId, recoverable.workerAccess); + } else { + logger.error( + { err, ...logBase }, + "failed to ensure managed plugin environment; continuing boot (degraded: environment unavailable)", + ); + } + } + } + return { ensured, failed, @@ -417,5 +672,6 @@ export async function applyManagedEnvironments( removed: 0, backedUp: 0, outcomes, + pluginOutcomes, }; } From a0cba70f18abe8b163e1bcc37cdf1fd9291a8dda Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 11:39:29 -0600 Subject: [PATCH 05/11] fix(runner): continue restart-interrupted ACPX and OpenCode turns (#19501) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep their provider conversation when the control plane restarts > - A local Codex turn lost in a restart already continues once in the same conversation (#15297) > - ACPX (Claude and the other ACPX profiles) and OpenCode took a different path: the runner closed the lost turn as a provider failure > - The task then blocked with "Provider turn failed", and the user's queued follow-up stayed unread > - A restart must not need a user action when continuing is safe > - This pull request gives ACPX and OpenCode the same recoverable restart-loss cause and the same single continuation as Codex > - The benefit is that work interrupted by a controller restart finishes on its own ## Linked Issues or Issue Description Refs #15297 (Codex continuation), #19399 (restart-interrupted turn hook), #19471 (harness fix used for the live reruns). **What happened?** After a control-plane restart during an active native ACPX Claude turn, the task ended `blocked` with `native_provider_terminal_failed: Provider turn failed`. The queued follow-up comment was not consumed. No continuation started. The next run then failed with `native_session_cleanup_quarantined`. A native OpenCode run killed mid-turn ended the same way (`turn.failed` with `error: null`). **Expected behavior** When the old provider process is gone and the conversation can be restored, continue the task once in the same conversation. When that is not safe, record an honest restart cause instead of a generic provider failure. **Steps to reproduce** 1. Start a native ACPX Claude task on a local runner. 2. Submit a follow-up comment while the turn is active. 3. Restart the control plane while the turn is still running. 4. Run `pnpm test:e2e:runner -- --max-automatic-retries 0 --id everyday-workflows.runner-acpx-claude.local.recover-controller`. **Root cause** - `runner-core/src/acpx_provider_backend.rs` `restore_session_if_needed`: a saved `turn_active` state always closed the session. It recorded `turn.failed` with only a top-level `code: acpx_active_turn_recovery_closed` and no `error` object. It also emitted `run.terminal`. - The runnerd transport forwards only `error`, so the controller saw `error: null` and reported generic `provider_turn_failed`. - `runner-core/src/provider_backend.rs` gave the recoverable `provider_turn_lost_on_restore` cause only to `provider_label == "codex"`. OpenCode shares that backend. - `nativeRestartInterruptedTurnId` accepted only `codex_app_server`. The runtime continuation gate accepted only `provider.kind === "codex"`. - Two follow-on defects appeared once the continuation ran live: - The restored ACPX provider rejected the controller's result for the lost turn's open semantic call. That call never settled. - The restored provider's MCP request ids restart. The controller keys semantic calls by `hash(runId, callId)`. The continuation's first Paperclip tool call reused an earlier id, was treated as already answered, and hung. After the result was accepted, `runner_settlement_incomplete` failed the run. ## What Changed - ACPX runner: for an active turn lost to a runner restart, prove that the original provider lifetime lost its fence quorum. Then restore the same conversation and record `turn.failed` with `error: { code: "provider_turn_lost_on_restore", recoverable: true }`. Keep the run open for the controller's continuation. An accepted result, an unanswered runtime request, or an active goal keeps the old close behavior. So do an unproven lifetime fence and a failed restore. - ACPX runner fallback close: add a structured `error` (`acpx_active_turn_recovery_closed`, not recoverable, with an honest restart message), so the cause survives the transport. - ACPX runner: track open semantic calls durably. On restore, retire the lost turn's open calls. Acknowledge their controller results as `retired` and do not send them to the new provider. - ACPX sidecar: when the sidecar restores an existing session, prefix forwarded semantic `callId`s with a per-process namespace (`r:`). Fresh sessions keep raw ids. - OpenCode runner: give OpenCode the same recoverable cause and open run as Codex (`restart_lost_turn_is_continuable`). Cursor Cloud stays excluded because its agent runs remotely. - Runner TS: `nativeRestartInterruptedTurnId` accepts `acpx_runtime` and `opencode_server`. The runtime continuation gate accepts `acpx` and `opencode`. All other Codex safety rules are unchanged: one continuation, admitted dead-runner restart only, no compaction, goal, semantic result, or pending runtime request. - Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`. ## Verification Red, then green, at each layer: - Rust `tests/acpx_warm_attachment.rs` `restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart`. Red: status `closed`. Then red on the lost call's result: "tool result does not match a pending provider call". Green after the fix. - Rust `tests/native_provider_backend.rs` `restart_lost_opencode_turn_records_a_recoverable_cause`, with the qualified OpenCode launch fixture. Red: `error` was `null`. Green after the fix. - Rust unit test `restart_lost_turns_continue_only_for_locally_restored_providers` (Codex, OpenCode yes; Cursor Cloud no). - TS `native-session-runtime.test.ts` "continues a restart-lost acpx_runtime / opencode_server turn once". Red: `NativeProviderTerminalFailure provider_turn_lost_on_restore`. Green: exactly one continuation, which does not resend the task. - TS `acpx-runtime-sidecar.test.ts`: a restored provider's call ids are namespaced and resolve. - Server `native-runner-restart-recovery.integration.test.ts`: a checkpointed `acpx_runtime` or `opencode_server` restart interruption is claimed as `resume_dead_runner`. Red without the driver gate. Green with it. - Two `native_provider_backend.rs` tests asserted the old close-on-restore result. They now assert the new contract. Their real invariant still holds: the delivered response is never replayed. Suites run locally on the final revision: - runner-core: `cargo test --no-fail-fast acpx`, `--test native_provider_backend` (38), `--test acpx_warm_attachment`, `--test codex_provider restart_lost` - runner TS: `native-session-runtime`, `runnerd-restart-continuation`, Codex driver recovery, contracts, and `acpx-runtime-sidecar` (all pass) - server: `native-runner-restart-recovery.integration.test.ts` (80 passed) - `pnpm --filter "@paperclipai/paperclip-runner..." build` (all generated-contract checks pass) Live runner E2E (local, real Claude via ACPX, macOS arm64): - `everyday-workflows.runner-acpx-claude.local.recover-controller`: PASS twice in a row on the final runner code. The restart crossed an active turn. The task finished `done`, the queued follow-up was consumed, the ZIP download was verified, and there were no failure codes. - `stock-harness.runner-acpx-claude.local.continuity-restart` regression: PASS. - Earlier failing reruns are kept as evidence of the follow-on defects: one harness `kill EPERM` while stopping the server (unrelated to this change), one hang on the colliding call id, and one `runner_settlement_incomplete`. ## Risks - A continuation can run more work in the restored conversation. Its prompt requires inspecting state before repeating an uncertain action, as for Codex. - The ACPX restore waits up to 5 s for the original provider lifetime fence before it falls back to the old close. - The ACPX provider state file gains two fields with defaults. Older saved states still load. A downgraded runner cannot read state that a newer runner wrote with these fields. - Restored ACPX sessions now forward prefixed call ids. Only fresh sessions keep raw ids. Live proof covers Claude. Codex-ACP, Pi, Cursor, Copilot, Grok, Kimi, and Antigravity share this path, but this PR proves them only by unit tests and code review. - OpenCode is covered by the runner fixture and gate tests. The live OpenCode restart-chaos cell was not rerun here. - `runner-core` `unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend` is flaky when run in parallel. It also fails with the new path disabled, so it predates this change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, code execution, and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 7 +- doc/run-log-events.md | 13 +- .../runner-core/src/acpx_provider_backend.rs | 226 +++++++++++++++++- .../runner-core/src/provider_backend.rs | 20 +- .../runner-core/tests/acpx_warm_attachment.rs | 165 +++++++++++++ .../tests/native_provider_backend.rs | 109 ++++++++- .../src/cli/acpx-runtime-sidecar.test.ts | 28 ++- .../src/cli/acpx-runtime-sidecar.ts | 14 +- .../src/contracts/native-session-backend.ts | 6 +- .../src/native-session-runtime.test.ts | 66 +++++ .../src/native-session-runtime.ts | 4 +- ...unner-restart-recovery.integration.test.ts | 11 +- 12 files changed, 651 insertions(+), 18 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 289b51720c8..e0e66ab3a32 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -2415,8 +2415,11 @@ run, release its environment, or start another provider. Lease expiry alone does not authorize that stale preparation to take ownership; normal recovery must establish a new claim first. -An exact local Codex restart can restore the conversation after losing its active -turn. Runnerd records this as `provider_turn_lost_on_restore`, with no invented +An exact local Codex, OpenCode, or ACPX restart can restore the conversation after +losing its active turn. ACPX first requires the original provider lifetime to lose +its fence quorum; without that proof, or with an accepted result, unanswered +runtime request, or active goal, it closes the session as before. Cursor Cloud +agents run remotely and are excluded. Runnerd records this as `provider_turn_lost_on_restore`, with no invented task result. The admitted restart attempt may send one continuation in that same conversation. It preserves the workspace and asks the agent to reconcile unfinished commands and external actions before proceeding. It does not resend the task or diff --git a/doc/run-log-events.md b/doc/run-log-events.md index 916a3a6d0a0..29e7669f97e 100644 --- a/doc/run-log-events.md +++ b/doc/run-log-events.md @@ -348,13 +348,24 @@ that ledger records `remote_execution_never_claimed` with a `bootstrap_incomplete` disposition. The provider attempt remains zero during classification; the executor increments it before starting provider work. -For a local Codex turn lost while runnerd was stopped, the canonical `turn.failed` +For a local Codex, OpenCode, or ACPX turn lost while runnerd was stopped, the canonical `turn.failed` event retains `error.code: "provider_turn_lost_on_restore"` and `error.recoverable: true`. It records the interrupted turn, not a final task outcome. There is no synthesized result for that interruption. An admitted same-session continuation emits its own `turn.submitted`, accepted turn ID, and terminal events; only the final outcome closes the run. Reconstructed thread history retains the original error so reconciliation and live delivery agree. +An ACPX session records this only after the original provider lifetime loses +its fence quorum and the same conversation is restored. Otherwise its +`turn.failed` carries `error.code: "acpx_active_turn_recovery_closed"` with +`error.recoverable: false`, and the run closes with an honest restart message. +After an ACPX restore, the controller can still deliver results for semantic +calls of the lost turn. Runnerd acknowledges each one as `retired` without +sending it to the restored provider, so the calls settle and live semantic +dispatch resumes for the continuation. A restored ACPX provider is a new process +whose MCP request ids restart, so its semantic `callId` values carry a +per-process prefix (`r:`). They never reuse an earlier call's id in +the same run. ## Native Local Process Stop Evidence diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs index 65647805628..b3ade8b5308 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs @@ -1,7 +1,7 @@ use crate::generated_acpx_profiles::{ acpx_release_profile, ACPX_DRIVER_KIND, QUALIFIED_ACPX_VERSION, }; -use std::collections::{BTreeMap, HashMap, HashSet, VecDeque}; +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque}; use std::fs::{self, DirBuilder, File}; use std::io::{Read, Write}; use std::net::TcpListener; @@ -70,6 +70,8 @@ pub const ACPX_PROVIDER_STATE_FILE: &str = "acpx-provider-state.json"; const ACPX_PROVIDER_STATE_SCHEMA: &str = "paperclip.runner.acpx-provider-state.v3"; const MAX_PROVIDER_STATE_BYTES: u64 = 16 * 1024 * 1024; const MAX_PENDING_EVENTS: usize = 8_320; +// Matches the bounded number of pending ACPX tool calls. +const MAX_OPEN_SEMANTIC_CALLS: usize = 4_096; const MAX_EVENTS_PER_POLL: usize = 128; const PROVIDER_LIFETIME_CONFIRMATION_TIMEOUT: Duration = Duration::from_secs(5); const PROVIDER_LIFETIME_CONFIRMATION_RETRY: Duration = Duration::from_millis(10); @@ -597,6 +599,13 @@ struct AcpxDurableState { pending_runtime_requests: BTreeMap, #[serde(default = "initial_event_sequence")] next_event_sequence: u64, + /// Semantic tool calls emitted to the controller without a result yet. + #[serde(default)] + open_semantic_calls: BTreeSet, + /// Open calls of a turn lost to a runner restart. The restored provider + /// never sees their results; the controller may still deliver them. + #[serde(default)] + retired_semantic_calls: BTreeSet, } impl AcpxDurableState { @@ -623,6 +632,8 @@ impl AcpxDurableState { pending_events: VecDeque::new(), pending_runtime_requests: BTreeMap::new(), next_event_sequence: initial_event_sequence(), + open_semantic_calls: BTreeSet::new(), + retired_semantic_calls: BTreeSet::new(), } } @@ -789,6 +800,31 @@ impl AcpxDurableState { if let Some(id) = event.payload.get("requestId").and_then(Value::as_str) { self.pending_runtime_requests.remove(id); } + } else if let Some(call_id) = event + .payload + .pointer("/semantic_tool/callId") + .and_then(Value::as_str) + { + // Remember semantic calls the controller may still answer. A lost + // turn's open calls become retired calls on restore, so a later + // controller result for them settles instead of failing. + match event.event_type.as_str() { + "semantic_tool.input" + if self.open_semantic_calls.len() < MAX_OPEN_SEMANTIC_CALLS => + { + self.open_semantic_calls.insert(call_id.to_owned()); + } + "semantic_tool.result" => { + self.open_semantic_calls.remove(call_id); + } + _ => {} + } + } else if matches!( + event.event_type.as_str(), + "turn.completed" | "turn.failed" | "turn.cancelled" | "turn.interrupted" + ) { + // A provider-observed terminal settles its calls in the live bridge. + self.open_semantic_calls.clear(); } self.next_event_sequence = next_sequence; self.pending_events.push_back(PolledEvent { @@ -1116,6 +1152,9 @@ impl AcpxCommandExecutor { let previous_turn = state.active_turn_id.clone(); if unsafe_active { self.context.provider_turn_id = None; + if self.restore_conversation_after_lost_turn(previous_turn.clone())? { + return Ok(()); + } let state = self .state .as_mut() @@ -1134,6 +1173,13 @@ impl AcpxCommandExecutor { "providerTerminalObserved": false, "code": "acpx_active_turn_recovery_closed", "providerShutdownFailed": true, + // Keep the cause through the controller facade, which + // forwards only the structured error of a failed turn. + "error": { + "code": "acpx_active_turn_recovery_closed", + "recoverable": false, + "message": "The runner restarted during this turn and could not restore the conversation.", + }, }), })?; state.push(NormalizedProviderEvent { @@ -1171,6 +1217,82 @@ impl AcpxCommandExecutor { self.save_state() } + /// Restores the conversation of a turn that was active when the runner + /// died, without inventing its outcome. The original provider lifetime + /// must have lost its fence quorum first, so no second provider can act on + /// the same session. The lost turn is recorded with a recoverable + /// `provider_turn_lost_on_restore` cause and the run stays open, so the + /// controller can send its single same-conversation continuation. Returns + /// false, leaving state untouched, when restoration is not safe. + fn restore_conversation_after_lost_turn( + &mut self, + previous_turn: Option, + ) -> Result { + let Some(state) = self.state.as_ref() else { + return Ok(false); + }; + // An accepted result, an unanswered runtime request, or an active goal + // needs its own reconciliation; a plain continuation cannot settle it. + let Some(identity) = state.identity.as_ref() else { + return Ok(false); + }; + if previous_turn.is_none() + || state.provider_exit_unconfirmed + || state.semantic_result.is_some() + || !state.pending_runtime_requests.is_empty() + || !(state.goal_projection.is_null() || goal_is_dormant(&state.goal_projection)) + { + return Ok(false); + } + match await_provider_lifetime_fence(identity.provider_lifetime_fence_candidates) { + // Release the quorum so the restored provider can inherit it. + Ok(listeners) => drop(listeners), + Err(_) => return Ok(false), + } + let Ok(session) = self.start_session(true) else { + return Ok(false); + }; + let identity = session.identity().clone(); + let process_id = session.process_id(); + let turn_controls = session.turn_control_capabilities(); + let state = self + .state + .as_mut() + .expect("ACPX state remains available during recovery"); + state.lifecycle = "session_open".to_owned(); + state.active_turn_id = None; + state.turn_controls = turn_controls; + // The controller may still answer the lost turn's semantic calls. + // The restored provider never issued them, so settle them on arrival. + let open = std::mem::take(&mut state.open_semantic_calls); + state.retired_semantic_calls.extend(open); + state.push(NormalizedProviderEvent { + event_type: "session.resumed".to_owned(), + priority: EventPriority::P0, + payload: session_event_payload(&state.descriptor, &identity, process_id, turn_controls), + })?; + state.push(NormalizedProviderEvent { + event_type: "turn.failed".to_owned(), + priority: EventPriority::P0, + payload: json!({ + "provider": "acpx", + "providerTurnId": previous_turn, + "status": "failed", + "providerTerminalObserved": false, + "code": "provider_turn_lost_on_restore", + "error": { + "code": "provider_turn_lost_on_restore", + "recoverable": true, + "message": "The runner restored the conversation after process loss, but the previous turn is no longer active.", + }, + }), + })?; + self.recovery_pending_effects = None; + self.session = Some(session); + self.save_state()?; + Ok(true) + } + fn start_session(&self, recovering: bool) -> Result { let state = self .state @@ -2123,6 +2245,20 @@ impl AcpxCommandExecutor { let result: ToolResult = serde_json::from_value(payload.clone()).map_err(|error| { DurableRunnerError::invalid(format!("semantic tool result is invalid: {error}")) })?; + if let Some(state) = self.state.as_ref() { + if state.retired_semantic_calls.contains(&result.call_id) { + // The call belonged to a turn lost to a runner restart. Its + // provider process is gone; acknowledge the controller result + // so the call settles, without sending it to the new provider. + // The retained receipt makes this idempotent: a delivery left + // indeterminate by a crash replays to the same acknowledgement. + return Ok(CommandExecution::result(json!({ + "status": "delivered", + "callId": result.call_id, + "retired": true, + }))); + } + } self.session .as_mut() .ok_or_else(|| DurableRunnerError::invalid("ACPX session is unavailable"))? @@ -2521,6 +2657,18 @@ fn goal_terminal_disposition<'a>( } impl CommandExecutor for AcpxCommandExecutor { + /// A restored session keeps durable receipts for the lost turn's calls. + /// Acknowledging one is idempotent and never reaches the provider or the + /// business operation, so an indeterminate delivery may replay. Live + /// calls have no durable receipt; without retired receipts this stays off. + fn can_reconcile_result_delivery(&mut self) -> Result { + self.restore()?; + Ok(self + .state + .as_ref() + .is_some_and(|state| !state.retired_semantic_calls.is_empty())) + } + fn host_park_ready(&mut self) -> Result { self.ensure_persistence_healthy()?; let Some(state) = self.state.as_ref() else { @@ -5020,6 +5168,14 @@ mod tests { assert_eq!(events[0].payload["replayAllowed"], false); assert_eq!(events[1].event_type, "turn.failed"); assert_eq!(events[1].payload["providerShutdownFailed"], true); + // An unanswered runtime request (and an unproven original lifetime) + // keeps the close path. Its cause survives the controller facade and + // is not offered for the restart continuation. + assert_eq!( + events[1].payload["error"]["code"], + "acpx_active_turn_recovery_closed" + ); + assert_eq!(events[1].payload["error"]["recoverable"], false); assert_eq!(events[2].event_type, "run.terminal"); assert_eq!(events[2].payload["schema"], "paperclip.prp.terminal.v1"); assert_eq!(events[2].payload["turnTerminalState"], "failed"); @@ -5049,6 +5205,74 @@ mod tests { fs::remove_dir_all(directory).unwrap(); } + #[test] + fn lost_turn_restoration_refuses_unreconciled_state_before_touching_the_provider() { + let directory = temporary_directory("lost-turn-guards"); + let provider_descriptor: AcpxProviderDescriptor = + serde_json::from_value(descriptor("claude")).unwrap(); + let operations = Vec::new(); + let tool_set = AuthorizedToolSet { + schema: TOOL_SET_SCHEMA.to_owned(), + schema_version: 1, + catalog_digest: authorized_tool_catalog_digest(&operations).unwrap(), + operations, + }; + let identity = AcpxProviderSessionIdentity { + kind: "acpx".to_owned(), + normalized_session_id: "session-1".to_owned(), + acpx_record_id: "record-1".to_owned(), + backend_session_id: "backend-1".to_owned(), + agent_session_id: "agent-1".to_owned(), + profile_digest: provider_descriptor.command_digest.clone(), + workspace_digest: format!("sha256:{}", "a".repeat(64)), + requested_model: provider_descriptor.model.clone(), + effective_model: provider_descriptor.model.clone(), + permission_mode: Some(provider_descriptor.permission_mode), + mode: provider_descriptor.mode.clone(), + pi_thinking_level: provider_descriptor.pi_thinking_level, + claude_effort: provider_descriptor.claude_effort.clone(), + grok_effort: provider_descriptor.grok_effort.clone(), + grok_disable_web_search: provider_descriptor.grok_disable_web_search.clone(), + provider_lifetime_fence_candidates: [60_001, 60_002, 60_003], + }; + let guards: [(&str, fn(&mut AcpxDurableState)); 4] = [ + ("provider_exit_unconfirmed", |state| { + state.provider_exit_unconfirmed = true + }), + ("accepted_semantic_result", |state| { + state.semantic_result = Some(json!({"reportedWorkDisposition": "done"})) + }), + ("unanswered_runtime_request", |state| { + state + .pending_runtime_requests + .insert("input-1".to_owned(), pending_input_request("input-1")); + }), + ("active_goal", |state| { + state.goal_projection = json!({"goal": {"status": "active"}}) + }), + ]; + for (label, seed) in guards { + let mut state = + AcpxDurableState::new(provider_descriptor.clone(), tool_set.clone(), "d".into()); + state.lifecycle = "turn_active".to_owned(); + state.identity = Some(identity.clone()); + state.active_turn_id = Some("turn-1".to_owned()); + seed(&mut state); + let mut executor = + AcpxCommandExecutor::with_runner_config(&directory, &test_config(&directory, None)); + executor.state = Some(state.clone()); + assert!( + !executor + .restore_conversation_after_lost_turn(Some("turn-1".to_owned())) + .unwrap(), + "{label} must keep the close path" + ); + assert!(executor.session.is_none(), "{label} started a provider"); + assert_eq!(executor.state, Some(state), "{label} changed saved state"); + } + fs::remove_dir_all(directory).unwrap(); + } + #[test] fn suspension_waits_for_the_original_provider_lifetime_quorum() { let (candidates, original_lifetime_fence) = reserve_provider_lifetime_fence(); diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs index 2588c4d049c..6ac3fe4dbbe 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs @@ -421,6 +421,14 @@ fn semantic_result_event( } } +/// Local Codex app-server and OpenCode server restart their own process and +/// restore the same conversation after runner loss, so a turn that vanished +/// meanwhile is a restart interruption the controller may continue once. A +/// Cursor Cloud agent runs remotely; its lost turn is not a process restart. +fn restart_lost_turn_is_continuable(provider: &str) -> bool { + matches!(provider, "codex" | "opencode") +} + fn facade_provider_name(provider: &str) -> &'static str { match provider { "opencode" => "OpenCode", @@ -2693,6 +2701,7 @@ impl CodexCommandExecutor { // trustworthy success notification to replay. Terminate it // conservatively so the controller cannot wait forever or // mistake an unknown outcome for success. + let restart_continuable = restart_lost_turn_is_continuable(&provider_label); let provider_terminal = NormalizedProviderEvent { event_type: "turn.failed".to_owned(), priority: EventPriority::P0, @@ -2701,14 +2710,14 @@ impl CodexCommandExecutor { "providerTurnId": previous_active_turn_id, "status": "failed", "providerTerminalObserved": false, - "error": if provider_label == "codex" { json!({ + "error": if restart_continuable { json!({ "code": "provider_turn_lost_on_restore", "recoverable": true, "message": "The runner restored the conversation after process loss, but the previous turn is no longer active.", }) } else { Value::Null }, }), }; - if provider_label == "codex" && !state.active_terminal_tool_rejected { + if restart_continuable && !state.active_terminal_tool_rejected { // No provider result was observed. Preserve the lost // turn as a fact, but leave the run open for the // controller's bounded same-conversation recovery. @@ -5700,6 +5709,13 @@ impl CommandExecutor for CodexCommandExecutor { #[cfg(test)] mod tests { + #[test] + fn restart_lost_turns_continue_only_for_locally_restored_providers() { + assert!(super::restart_lost_turn_is_continuable("codex")); + assert!(super::restart_lost_turn_is_continuable("opencode")); + assert!(!super::restart_lost_turn_is_continuable("cursor_cloud")); + } + #[test] fn mcp_permission_requests_use_durable_turn_and_item_identity() { let identity = ProviderEventIdentity { diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs index 608809e807e..168ba6e0a36 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs @@ -166,3 +166,168 @@ fn checkpoints_the_sidecar_and_rebinds_consecutive_warm_runs_before_accepting_wo assert_eq!(started.result["status"], "accepted"); executor.shutdown().unwrap(); } + +fn unused_fence_ports() -> [u16; 3] { + // Lifetime fence candidates must be dynamic ports (49152 and above). An + // OS-assigned port can be lower, for example on Linux. + let mut listeners = Vec::new(); + for port in (49_152..=u16::MAX).rev() { + if let Ok(listener) = std::net::TcpListener::bind(("127.0.0.1", port)) { + listeners.push(listener); + if listeners.len() == 3 { + break; + } + } + } + let ports = listeners + .iter() + .map(|listener| listener.local_addr().unwrap().port()) + .collect::>(); + [ports[0], ports[1], ports[2]] +} + +/// A runner that dies during an active ACPX turn (for example, when the +/// control plane restarts) must restore the same conversation once the old +/// provider lifetime is proven gone. It records a recoverable restart-loss +/// cause instead of a terminal provider failure. +#[test] +fn restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart() { + let fixture = + Fixture(std::env::temp_dir().join(format!("acpx-restart-{}", uuid::Uuid::new_v4()))); + fs::create_dir_all(&fixture.0).unwrap(); + fs::set_permissions(&fixture.0, fs::Permissions::from_mode(0o700)).unwrap(); + let sidecar = fixture.0.join("sidecar"); + fs::copy(env!("CARGO_BIN_EXE_fake-acpx-sidecar"), &sidecar).unwrap(); + fs::set_permissions(&sidecar, fs::Permissions::from_mode(0o700)).unwrap(); + let ports = unused_fence_ports(); + let args = vec![ + "--mode".to_owned(), + "suspend".to_owned(), + "--profile-digest".to_owned(), + PROFILE_DIGEST.to_owned(), + "--lifetime-fence-ports".to_owned(), + format!("{},{},{}", ports[0], ports[1], ports[2]), + ]; + let config = DurableRunnerConfig { + connect_url: "ws://127.0.0.1/runner".to_owned(), + ca_bundle_path: None, + state_dir: fixture.0.clone(), + runner_instance_id: "runner-1".to_owned(), + environment_lease_id: "lease-1".to_owned(), + run_id: "run-1".to_owned(), + normalized_session_id: "session-1".to_owned(), + turn_id: "turn-1".to_owned(), + item_id: "item-1".to_owned(), + runner_version: "0.0.0".to_owned(), + runner_digest: format!("sha256:{}", "a".repeat(64)), + acpx_launch_profile: Some(AcpxLaunchProfile { + candidate_admission: None, + authority_digest: format!("sha256:{}", "d".repeat(64)), + command: sidecar.clone(), + args: args.clone(), + artifacts: vec![QualifiedLaunchArtifact { + path: sidecar.clone(), + sha256: format!("sha256:{:x}", Sha256::digest(fs::read(&sidecar).unwrap())), + }], + }), + opencode_launch_profile: None, + cursor_cloud_launch_profile: None, + max_outbox_bytes: 1024 * 1024, + p0_reserve_bytes: 64 * 1024, + max_frame_bytes: 1024 * 1024, + reconnect_delay: Duration::from_millis(1), + reconnect_grace: None, + max_runtime: Duration::from_secs(30), + }; + let descriptor = json!({ + "kind": "acpx", "provider": "acpx", "driver": "acpx_runtime", + "providerVersion": "0.13.1", "agent": "codex", "model": "gpt-5.6-sol", + "acpxVersion": "0.13.1", "agentServerPackage": "@agentclientprotocol/codex-acp", + "agentServerVersion": "1.6.2", "agentRuntimePackage": "@openai/codex", + "agentRuntimeVersion": "0.160.0", "commandDigest": PROFILE_DIGEST, + "sidecarCommand": sidecar, "sidecarArgs": args, "runtimeDirectory": fixture.0, + "normalizedSessionId": "session-1", "runId": "run-1", "cwd": fixture.0, + "instructions": "Complete the supplied work.", + "permissionMode": "approve-all", "permissionModePinned": true, + }); + let mut original = AcpxCommandExecutor::with_runner_config(&fixture.0, &config); + original + .execute(&command(1, "run.prepare", json!({"provider": descriptor}))) + .unwrap(); + let opened = original + .execute(&command(2, "session.open", json!({}))) + .unwrap(); + let provider_session = opened.result["providerSessionId"].clone(); + let started = original + .execute(&command( + 3, + "turn.start", + json!({"turnId": "provider-turn-1", "text": "Do the work."}), + )) + .unwrap(); + assert_eq!(started.result["status"], "accepted"); + // The runner process dies with the turn still active, after the turn sent + // the controller a semantic call that has no result yet. + drop(original); + let state_path = fixture.0.join("acpx-provider-state.json"); + let mut saved: Value = serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap(); + saved["openSemanticCalls"] = json!(["call-lost"]); + fs::write(&state_path, serde_json::to_vec(&saved).unwrap()).unwrap(); + + let mut restored = AcpxCommandExecutor::with_runner_config(&fixture.0, &config); + let snapshot = restored + .execute(&command(4, "session.snapshot", json!({}))) + .unwrap(); + assert_ne!(snapshot.result["status"], "closed"); + assert_eq!(snapshot.result["providerSessionId"], provider_session); + let events = restored.poll_events().unwrap(); + let failed = events + .iter() + .find(|event| event.event_type == "turn.failed") + .expect("the lost turn is terminated"); + assert_eq!(failed.payload["providerTurnId"], "provider-turn-1"); + assert_eq!(failed.payload["providerTerminalObserved"], false); + assert_eq!( + failed.payload["error"]["code"], + "provider_turn_lost_on_restore" + ); + assert_eq!(failed.payload["error"]["recoverable"], true); + // The run stays open for the controller's single same-conversation continuation. + assert!(!events + .iter() + .any(|event| event.event_type == "run.terminal")); + // The controller can still answer the lost call. That result settles the + // call instead of failing against the restored provider, which never saw it. + let lost_result = json!({ + "callId": "call-lost", "operationId": "register_deliverable", + "result": {"ok": true}, "isError": false, + }); + let settled = restored + .execute(&command(5, "semantic_tool.result", lost_result.clone())) + .unwrap(); + assert_eq!(settled.result["retired"], true); + // The receipt is durable and idempotent, so a delivery left indeterminate + // by a crash may replay to the same acknowledgement. + assert!(restored.can_reconcile_result_delivery().unwrap()); + let replayed = restored + .execute(&command(6, "semantic_tool.result", lost_result)) + .unwrap(); + assert_eq!(replayed.result["retired"], true); + // Calls the lost turn never opened still fail closed. + let unknown = json!({ + "callId": "call-unknown", "operationId": "register_deliverable", + "result": {"ok": true}, "isError": false, + }); + assert!(restored + .execute(&command(7, "semantic_tool.result", unknown)) + .is_err()); + let continued = restored + .execute(&command( + 8, + "turn.start", + json!({"turnId": "provider-turn-2", "text": "Continue."}), + )) + .unwrap(); + assert_eq!(continued.result["status"], "accepted"); + restored.shutdown().unwrap(); +} diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs index 8439dd9642f..e83b70871aa 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs @@ -7,7 +7,7 @@ use std::os::unix::fs::PermissionsExt; use paperclip_runner_core::durable::{ AcpxLaunchProfile, Command, CommandExecutor, CursorCloudLaunchProfile, DurableRunnerConfig, - OpenCodeLaunchProfile, QualifiedLaunchArtifact, PROVIDER_STATE_INCOMPATIBLE, + OpenCodeLaunchProfile, PolledEvent, QualifiedLaunchArtifact, PROVIDER_STATE_INCOMPATIBLE, }; use paperclip_runner_core::native_provider_backend::NativeProviderCommandExecutor; use paperclip_runner_core::provider_bridge::{authorized_tool_catalog_digest, AuthorizedTool}; @@ -579,6 +579,30 @@ fn acpx_runtime_resolution(mode: &str, request_id: &str) -> Value { }) } +/// A turn lost to a runner restart is recorded once as a recoverable loss. +fn assert_restart_lost_turn(events: &[PolledEvent]) { + let lost = events + .iter() + .filter(|event| event.event_type == "turn.failed") + .collect::>(); + assert_eq!(lost.len(), 1); + assert_eq!( + lost[0].payload["error"]["code"], + "provider_turn_lost_on_restore" + ); + assert_eq!(lost[0].payload["error"]["recoverable"], true); + assert!(!events + .iter() + .any(|event| event.event_type == "run.terminal")); +} + +/// Restarting the restored, idle session again replays no turn or request. +fn assert_no_replayed_turn_or_request(events: &[PolledEvent]) { + assert!(events + .iter() + .all(|event| event.event_type == "session.resumed")); +} + fn shutdown_recovered_acpx_fixture(executor: &mut NativeProviderCommandExecutor, directory: &Path) { // The fake sidecar advertises the same three lifetime-fence ports for all // fixtures. Parallel recovered shutdowns hold a quorum through state fsync, @@ -679,20 +703,23 @@ fn acpx_response_delivery_survives_crash_before_journaling_without_replaying_the .execute(&command(6, "request.resolve", resolve.payload)) .is_err()); assert_eq!(recovered.poll_events().unwrap(), events); + // The lost turn's conversation is restored for one continuation. Its + // outcome is a recoverable restart loss; the response is not replayed. assert_eq!( recovered .execute(&command(7, "session.snapshot", json!({}))) .unwrap() .result["status"], - "closed" + "session_open" ); + assert_restart_lost_turn(&events); // Polling is not an ACK; only the journal's ACK retires the event. recovered.acknowledge_events(events.len()).unwrap(); assert!(recovered.poll_events().unwrap().is_empty()); drop(recovered); let mut recovered_again = NativeProviderCommandExecutor::with_runner_config(&directory, &config); - assert!(recovered_again.poll_events().unwrap().is_empty()); + assert_no_replayed_turn_or_request(&recovered_again.poll_events().unwrap()); shutdown_recovered_acpx_fixture(&mut recovered_again, &directory); fs::remove_dir_all(directory).unwrap(); } @@ -864,14 +891,15 @@ fn acpx_failed_settlement_ack_replays_the_same_delivered_event_without_expiry() .execute(&command(6, "request.resolve", response)) .is_err()); assert_eq!(recovered.poll_events().unwrap(), events); + assert_restart_lost_turn(&events); recovered.acknowledge_events(events.len()).unwrap(); assert!(recovered.poll_events().unwrap().is_empty()); shutdown_recovered_acpx_fixture(&mut recovered, &directory); drop(recovered); let mut recovered_again = NativeProviderCommandExecutor::with_runner_config(&directory, &config); - assert!(recovered_again.poll_events().unwrap().is_empty()); - recovered_again.shutdown().unwrap(); + assert_no_replayed_turn_or_request(&recovered_again.poll_events().unwrap()); + shutdown_recovered_acpx_fixture(&mut recovered_again, &directory); fs::remove_dir_all(directory).unwrap(); } } @@ -2278,6 +2306,77 @@ fn rejects_a_mutable_opencode_command_outside_the_runner_launch_profile() { } #[test] +#[cfg_attr(windows, ignore = "verified launch is not supported on Windows yet")] +fn restart_lost_opencode_turn_records_a_recoverable_cause() { + // A local OpenCode server restarts with runnerd, like Codex app-server: + // the lost turn keeps its recoverable cause and the run stays open for + // the controller's single same-conversation continuation. + let directory = temporary_directory("opencode-restart-lost-turn"); + let mut config = opencode_config(&directory); + let proxy_script = directory.join("qualified-opencode-proxy-script"); + #[cfg(unix)] + fs::set_permissions(&proxy_script, fs::Permissions::from_mode(0o700)).unwrap(); + fs::write( + &proxy_script, + format!( + "#!/bin/sh\nexec '{}' --state-file '{}' --call-log '{}' --require-completion-contract --durable-turn-ids --linger-after-turn-start\n", + env!("CARGO_BIN_EXE_fake-codex-app-server"), + directory.join("fake-opencode-state.json").display(), + directory.join("fake-opencode-calls.log").display(), + ), + ) + .unwrap(); + #[cfg(unix)] + fs::set_permissions(&proxy_script, fs::Permissions::from_mode(0o500)).unwrap(); + config + .opencode_launch_profile + .as_mut() + .unwrap() + .proxy_script = qualified_artifact(proxy_script); + let mut first = NativeProviderCommandExecutor::with_runner_config(&directory, &config); + first + .execute(&command( + 1, + "run.prepare", + opencode_prepare_payload(&directory), + )) + .unwrap(); + first + .execute(&command(2, "session.open", json!({}))) + .unwrap(); + first + .execute(&command(3, "turn.start", json!({"text": "Edit the file."}))) + .unwrap(); + drop(first); + let fake_state = directory.join("fake-opencode-state.json"); + let mut saved: Value = serde_json::from_slice(&fs::read(&fake_state).unwrap()).unwrap(); + saved["activeTurnId"] = Value::Null; + fs::write(&fake_state, serde_json::to_vec(&saved).unwrap()).unwrap(); + + let mut restored = NativeProviderCommandExecutor::with_runner_config(&directory, &config); + restored + .execute(&command(4, "session.snapshot", json!({}))) + .unwrap(); + let events = restored.poll_events().unwrap(); + let lost = events + .iter() + .find(|event| event.event_type == "turn.failed") + .expect("the lost turn is terminated"); + assert_eq!( + lost.payload["error"]["code"], + "provider_turn_lost_on_restore" + ); + assert_eq!(lost.payload["error"]["recoverable"], true); + assert!(!events + .iter() + .any(|event| event.event_type == "run.terminal")); + assert_eq!(opencode_call_count(&directory, "turn/start"), 1); + restored.shutdown().unwrap(); + fs::remove_dir_all(directory).unwrap(); +} + +#[test] +#[cfg_attr(windows, ignore = "verified launch is not supported on Windows yet")] fn rejects_opencode_launch_profile_drift_across_fresh_recovery() { let directory = temporary_directory("opencode-profile-recovery"); let config = opencode_config(&directory); diff --git a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts index 676dabac7cb..db33af5ffd6 100644 --- a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts +++ b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts @@ -467,6 +467,30 @@ describe("qualified ACPX runtime sidecar", () => { } }); + it("keeps a restored provider's restarted call ids distinct from earlier calls in the run", async () => { + // A restored session starts a new provider process whose MCP request ids + // restart. The controller keys calls by run and call id, so a reused id + // would be mistaken for an earlier, already answered call. + const raw = { reportedWorkDisposition: "done", summary: "Complete", evidence: [], verification: [], + completionClaim: { contractRevision: "1", objectiveSatisfied: true, criteria: [], remainingWork: [] } }; + const tools = new Map(), emitted: any[] = []; + const handler = loadWaitForTool({ tools, emitted, validate: validatePrpStructuredRunResult, callIdNamespace: "rgeneration1" }); + const commands = loadActualToolCommands(tools); + const bridge = await startRunnerToolBridge({ handler, captureSemanticReceipt: () => () => {} }); + try { + const response = fetch(bridge.url, { method: "POST", headers: { Authorization: `Bearer ${bridge.secret}`, "Content-Type": "application/json" }, + body: JSON.stringify({ jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "paperclip_finish", arguments: raw } }) }); + await vi.waitFor(() => expect(tools.has("rgeneration1:2")).toBe(true)); + expect(emitted).toEqual([expect.objectContaining({ callId: "rgeneration1:2", operationId: "paperclip_finish" })]); + await commands.resolve({ callId: "rgeneration1:2", turnId: "test-turn", result: { accepted: true }, error: null }); + const body = await (await response).json() as ToolCallResponseBody; + expect(body.result.isError).not.toBe(true); + } finally { + for (const pending of tools.values()) pending.cleanup(); + await bridge.close(); + } + }); + it.each([["monitor", true], ["monitor", false], ["same_agent", true], ["same_agent", false]] as const)("resolves an omitted continuation before the production sidecar validator with saved %s authority=%s", async (kind, eligible) => { const raw = { reportedWorkDisposition: "yielded", summary: "The review fix is published.", evidence: [], verification: [], completionClaim: { contractRevision: "1", objectiveSatisfied: false, criteria: [], remainingWork: [{ description: "Wait for checks.", blocksCompletion: true }] } }; @@ -1752,6 +1776,7 @@ function loadWaitForTool(input: { emitted: unknown[]; validate?: typeof validatePrpStructuredRunResult; emit?: (eventType: string, payload: unknown) => boolean; + callIdNamespace?: string | null; }): (call: RunnerToolCall) => Promise { const source = readFileSync( fileURLToPath(new URL("./acpx-runtime-sidecar.ts", import.meta.url)), @@ -1764,7 +1789,7 @@ function loadWaitForTool(input: { const factory = new Function( "boundedIdentity", "tools", "turnId", "emit", "PRP_COMPLETION_TOOL_NAME", "PRP_BLOCK_TOOL_NAME", "validatePrpStructuredRunResult", "boundedSidecarValue", "record", "MAX_PENDING_TOOLS", "stringifyAcpxSidecarFrame", - "resolveCompletionContinuation", "runId", "normalizingTools", "randomUUID", + "resolveCompletionContinuation", "runId", "normalizingTools", "randomUUID", "callIdNamespace", `return (${functionSource});`, ); return factory( @@ -1786,6 +1811,7 @@ function loadWaitForTool(input: { "test-run", new Set(), randomUUID, + input.callIdNamespace ?? null, ); } diff --git a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts index 082a9e93ed3..a95d6a4f7a6 100644 --- a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts +++ b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts @@ -176,6 +176,12 @@ let openParams: AcpxSidecarOpenParams | null = null; let claudeCompactionCount = 0; let runId: string | null = null; let turnId: string | null = null; +/** + * A restored session runs in a new provider process whose MCP request ids + * restart. The controller keys semantic calls by run and call id, so prefix + * this process's ids to keep them distinct from calls earlier in the run. + */ +let callIdNamespace: string | null = null; let activeCopilotEvidence: CopilotToolEvidence | undefined; let sequence = 0; let requestSequence = 0; @@ -374,6 +380,9 @@ async function dispatch( ); host = openedHost; openParams = params; + callIdNamespace = params.expectedIdentity === undefined + ? null + : `r${randomUUID().replaceAll("-", "").slice(0, 12)}`; emit("runtime.process", { role: "sidecar", pid: process.pid, @@ -821,7 +830,10 @@ async function waitForTool(call: RunnerToolCall): Promise { if (!activeTurnId || !activeRunId || call.signal.aborted) { throw new Error("ACPX tool call is not bound to an active turn"); } - const callId = boundedIdentity(call.callId, "callId"); + const providerCallId = boundedIdentity(call.callId, "callId"); + const callId = callIdNamespace === null + ? providerCallId + : boundedIdentity(`${callIdNamespace}:${providerCallId}`, "callId"); if (tools.has(callId) || normalizingTools.has(callId)) throw new Error("ACPX tool call is duplicated"); const operationId = boundedIdentity(call.tool, "operationId"); if (tools.size + normalizingTools.size >= MAX_PENDING_TOOLS) { diff --git a/packages/paperclip-runner/src/contracts/native-session-backend.ts b/packages/paperclip-runner/src/contracts/native-session-backend.ts index aefb325d705..fdb0f7461b7 100644 --- a/packages/paperclip-runner/src/contracts/native-session-backend.ts +++ b/packages/paperclip-runner/src/contracts/native-session-backend.ts @@ -95,11 +95,15 @@ export function isNativeRestartInterruption(error: unknown): boolean { return failure.code === NATIVE_RESTART_INTERRUPTION_CODE && failure.recoverable === true; } +const RESTART_CONTINUATION_DRIVER_KINDS: ReadonlySet = new Set(["codex_app_server", "acpx_runtime", "opencode_server"]); + /** The terminal fingerprint survives a crash between interruption and continuation. */ export function nativeRestartInterruptedTurnId(snapshot: Pick): string | null { - if (snapshot.driverKind !== "codex_app_server" || snapshot.activeTurnId != null || + // Runnerd records this cause for Codex app-server and ACPX sessions after + // proving the old provider lifetime ended and restoring the conversation. + if (!RESTART_CONTINUATION_DRIVER_KINDS.has(snapshot.driverKind ?? "") || snapshot.activeTurnId != null || snapshot.semanticResult != null || snapshot.dispositionOnlyRecoveryConsumed || snapshot.goal != null || snapshot.pendingRuntimeRequests?.length) return null; const terminal = Array.isArray(snapshot.terminalTurns) ? snapshot.terminalTurns.at(-1) : null; diff --git a/packages/paperclip-runner/src/native-session-runtime.test.ts b/packages/paperclip-runner/src/native-session-runtime.test.ts index de63070094e..7cae1d4d807 100644 --- a/packages/paperclip-runner/src/native-session-runtime.test.ts +++ b/packages/paperclip-runner/src/native-session-runtime.test.ts @@ -20,6 +20,7 @@ import { NativeSessionCloseUnrecoverableError, NativeSessionCleanupQuarantinedError, NativeSessionProtocolIntegrityError, + nativeRestartInterruptedTurnId, } from "./contracts/native-session-backend.js"; import { PROVIDER_STATE_INCOMPATIBLE } from "./contracts/harness-driver.js"; import type { @@ -508,6 +509,71 @@ describe("executeNativeSession recovery", () => { expect(checkpoints.some(snapshot => snapshot.terminalTurns?.some(turn => turn.turnId === "turn-recovery"))).toBe(true); }); + it.each(["acpx_runtime", "opencode_server"] as const)("continues a restart-lost %s turn once in the same conversation", async (driverKind) => { + const qualified = resolveQualifiedAcpxProfile("claude", "claude-sonnet-5"); + const acpxInput: NativeExecutionInputV1 = (driverKind === "acpx_runtime" ? { + ...input, + session: { ...input.session, driverKind }, + provider: { kind: "acpx", agent: "claude", model: "claude-sonnet-5", permissionPolicy: "interactive", profile: { + driverKind: qualified.driverKind, protocolVersion: qualified.protocolVersion, acpxVersion: qualified.acpxVersion, + agent: qualified.agent, agentProfileVersion: qualified.agentProfileVersion, + agentServerPackage: qualified.agentServerPackage, agentServerVersion: qualified.agentServerVersion, + agentRuntimePackage: qualified.agentRuntimePackage, agentRuntimeVersion: qualified.agentRuntimeVersion, + commandDigest: qualified.commandDigest, + } }, + } : { + ...input, + session: { ...input.session, driverKind }, + provider: { kind: "opencode", model: "openrouter/deepseek/deepseek-v4-flash-0731" }, + }) as NativeExecutionInputV1; + const failure = { code: "provider_turn_lost_on_restore", recoverable: true }; + const checkpoint: PersistedNativeSession = { + backendKind: "runner", driverKind, sessionId: "record-recovery", identity, + providerSessionId: "provider-recovery", cursor: "0", activeTurnId: "turn-recovery", + terminalTurns: [], semanticResult: null, pendingRuntimeRequests: [], lineage: [], + }; + const interrupted: PersistedNativeSession = { + ...checkpoint, activeTurnId: null, + terminalTurns: [{ turnId: "turn-recovery", fingerprint: JSON.stringify({ terminalState: "failed", error: failure, result: null }) }], + }; + const startTurn = vi.fn(async () => ({ turnId: "turn-continuation" })); + const session: NativeSession = { + identity: () => identity, + async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; }, + async *events() { + yield runnerEvent(1, "turn.failed", { status: "failed", error: failure }); + if (!startTurn.mock.calls.length) return; + yield { ...runnerEvent(2, "turn.completed", { status: "completed" }), turnId: "turn-continuation" }; + }, + startTurn, + async result() { return startTurn.mock.calls.length ? { result, terminal, turnId: "turn-continuation" } : null; }, + async snapshot() { return structuredClone(interrupted); }, + async close() {}, + }; + const backend: NativeSessionBackend = { + async descriptor() { return { kind: "runner", name: "restart", version: "1", capabilities: await session.capabilities() }; }, + async openSession() { throw new Error("must preserve the provider session"); }, + async recoverSession() { return { recovered: true, session }; }, + }; + const port: ControlPlanePort = { + async openRun() {}, + async checkpointSession() {}, + async appendEvent(event: PrpEvent) { return { cursor: event.sourceSeq, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" }; }, + async replayEvents() { return { events: [], highestContiguousSourceSeq: 0 }; }, + async completeRun() {}, + }; + await expect(executeNativeSession({ + input: acpxInput, backend, controlPlane: port, persistedSession: checkpoint, + runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery", + resumeInterruptedTurn: true, + })).resolves.toMatchObject({ result, turnId: "turn-continuation" }); + expect(startTurn).toHaveBeenCalledOnce(); + const submitted = startTurn.mock.calls[0]![0] as Parameters[0]; + expect(submitted.continuation).toBe(true); + expect(submitted.message.text).toContain("Reconcile any unfinished tool or command"); + expect(nativeRestartInterruptedTurnId(interrupted)).toBe("turn-recovery"); + }); + it("never continues task work when a restart loses a compaction turn", async () => { const failure = { code: "provider_turn_lost_on_restore", recoverable: true }; const checkpoint: PersistedNativeSession = { diff --git a/packages/paperclip-runner/src/native-session-runtime.ts b/packages/paperclip-runner/src/native-session-runtime.ts index 273f84ba33a..55fb486d92f 100644 --- a/packages/paperclip-runner/src/native-session-runtime.ts +++ b/packages/paperclip-runner/src/native-session-runtime.ts @@ -2448,7 +2448,9 @@ export async function executeNativeSession( const resumeInterruptedTurn = async (event: PrpEvent, signal: AbortSignal) => { if (restartContinuationStarted || !recovered || !options.resumeInterruptedTurn || // A compaction-only run never continues task work after a restart. - input.provider.kind !== "codex" || options.sessionCompaction || options.sessionGoalControl || options.resumeSessionGoalHeartbeat || + // Runnerd records a restart-lost turn only for these providers, after + // proving the old process ended and restoring the same conversation. + !["codex", "acpx", "opencode"].includes(input.provider.kind) || options.sessionCompaction || options.sessionGoalControl || options.resumeSessionGoalHeartbeat || recoveredSnapshot.goal || persistedSession?.semanticResult || persistedSession?.pendingRuntimeRequests?.length || event.eventType !== "turn.failed" || !isNativeRestartInterruption(event.payload.error) || event.turnId !== persistedSession?.activeTurnId) return false; diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 509213ca403..5241fe2b5b5 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -2220,10 +2220,15 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( }, ); - it.each([2, 3])("recovers a checkpointed restart interruption within provider budget %s", async (attempt) => { - const fixture = await seedRun(`INTERRUPTED-${attempt}`); + it.each([ + { attempt: 2, driverKind: "codex_app_server" }, + { attempt: 3, driverKind: "codex_app_server" }, + { attempt: 2, driverKind: "acpx_runtime" }, + { attempt: 2, driverKind: "opencode_server" }, + ])("recovers a checkpointed $driverKind restart interruption within provider budget $attempt", async ({ attempt, driverKind }) => { + const fixture = await seedRun(`INTERRUPTED-${driverKind}-${attempt}`); await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: { - driverKind: "codex_app_server", + driverKind, sessionId: fixture.native.normalizedSessionId, identity: { companyId, issueId: fixture.issueId, runId: fixture.runId, agentId, sessionId: fixture.native.normalizedSessionId }, providerSessionId: "interrupted-provider", From 63e550b338903928bb647089a1478e88941510f3 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 11:39:44 -0600 Subject: [PATCH 06/11] fix(ui): keep the audit hash out of the tool approval card (#19509) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents call connected-app tools (for example GitHub `create_repository`) through the tool gateway, and a write call waits for a human approval card > - The pending approval card showed an "Arguments and audit hash" disclosure. The disclosure printed the 64-character SHA-256 of the reviewed arguments > - A new user in onboarding testing called this "a long hash string in the review approval card" that is confusing and unapproachable > - The hash identifies the exact arguments for audit. The approver does not need to read it to decide > - This pull request removes the raw hash from the card text and keeps it behind a "Copy audit hash" action > - The benefit is a plain approval card, with the audit value still available to people who need it ## Linked Issues or Issue Description **What happened?** On the pending tool-approval card, the "Arguments and audit hash" disclosure printed the raw SHA-256 of the reviewed arguments (for example `8ca5312877235a…`, 64 hex characters) under the arguments JSON. **Expected behavior** The approval card uses plain language. Internal identifiers do not show as card text. A person who needs the audit hash can still get it. **Steps to reproduce** 1. Let an agent call a write tool on a connected app (for example GitHub `create_repository`). 2. Open the pending "Approve tool action" card ("Review request"). 3. Expand "Arguments and audit hash". The raw 64-character hash shows under the arguments. **Paperclip version or commit** Current `master` (819420a60a) and the Paperclip Cloud staging build used for onboarding tests. ## What Changed - New `ToolActionAuditHashCopy` button (`ui/src/components/ToolActionAuditHashCopy.tsx`). It copies the arguments hash to the clipboard through the shared `useCopyAction` hook, and shows "Copied" or "Couldn’t copy". - `IssueThreadInteractionCard` (`RequestToolActionCard`): the disclosure is now "Arguments". The raw hash `

` is replaced by the copy button. - `TaskChatCompactInteractionCard`: the same change in both places that printed `toolAction.argumentsHash`. - If the clipboard is blocked, the button shows "Couldn’t copy" and the hash then shows as selectable text, so the audit value stays reachable (review feedback). - Tests: a new regression test renders the review flow with the real staging payload shape (GitHub `create_repository`, raw gallery tool id, 64-hex hash) and asserts that no `/[0-9a-f]{12,}/i` text shows before or after the disclosure opens, and that the copy action writes the hash. The existing long-proposal test now asserts that the hash is not in the disclosure text. `ToolActionAuditHashCopy.test.tsx` covers copy success and the blocked-clipboard fallback. The clipboard mock is restored after the test. ## Verification Red on `master` (new test only, before the fix): ``` FAIL src/components/task-chat/TaskChatInteractionCard.test.tsx > TaskChatInteractionCard > keeps raw tool ids and the audit hash out of the tool approval card text AssertionError: expected 'Gcreate_repository · Name: brainrot-s…' not to match /[0-9a-f]{12,}/i + Received: "Gcreate_repository · Name: brainrot-storefront-prototype · Private: trueArguments and audit hash{\"name\":\"brainrot-storefront-prototype\",\"private\":true}8ca5312877235a4c0f6e9b1d2a7c3e5f9081b2c4d6e8fa1b3c5d7e9f0a2b4c6dDeclineApprove & run" Test Files 1 failed (1) Tests 1 failed | 32 skipped (33) ``` Green after the fix (same test): ``` Test Files 1 passed (1) Tests 1 passed | 32 skipped (33) ``` Neighbouring suites (`ui/src/components/task-chat`, `IssueThreadInteractionCard.test.tsx`, `AttentionQueueRow.test.tsx`, `transcript/RunTranscriptView.test.tsx`), after rebase on current `master`: ``` Test Files 62 passed (62) Tests 1047 passed (1047) ``` After the review fixes (`ToolActionAuditHashCopy.test.tsx`, `task-chat`, `IssueThreadInteractionCard.test.tsx`): ``` Test Files 61 passed (61) Tests 1004 passed (1004) ``` - `pnpm check:token-gates`: all gates clean. - `tsc --noEmit` on `ui`: no errors in the changed files. - Visual check in Storybook story `Issue Thread Interactions / Tool Action Pending Write`, disclosure open: - Before: the summary reads "Arguments and audit hash" and a mono line `sha256:9f2c1a7be4d0c8a3` shows under the JSON. - After: the summary reads "Arguments". A small ghost "Copy audit hash" button shows under the JSON. No hash text shows. ## Risks - Low risk. UI copy and layout change on one card type only. The server payload, the approval decision, and the stored hash do not change. - A person who read the hash off the card must now use the copy button. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edits, browser). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../components/IssueThreadInteractionCard.tsx | 5 +- .../ToolActionAuditHashCopy.test.tsx | 46 ++++++++++++++ ui/src/components/ToolActionAuditHashCopy.tsx | 42 +++++++++++++ .../TaskChatCompactInteractionCard.tsx | 11 ++-- .../TaskChatInteractionCard.test.tsx | 63 ++++++++++++++++++- 5 files changed, 156 insertions(+), 11 deletions(-) create mode 100644 ui/src/components/ToolActionAuditHashCopy.test.tsx create mode 100644 ui/src/components/ToolActionAuditHashCopy.tsx diff --git a/ui/src/components/IssueThreadInteractionCard.tsx b/ui/src/components/IssueThreadInteractionCard.tsx index 68c6f3c2389..6ce2763aa91 100644 --- a/ui/src/components/IssueThreadInteractionCard.tsx +++ b/ui/src/components/IssueThreadInteractionCard.tsx @@ -54,6 +54,7 @@ import { useCoordinatorWorkspaceEnabled } from "@/hooks/useCoordinatorWorkspaceE import { CoordinatorTaskCommitment } from "./coordinator/CoordinatorTaskCommitment"; import { DecisionCardFooter } from "./DecisionCardFooter"; +import { ToolActionAuditHashCopy } from "./ToolActionAuditHashCopy"; const OTHER_ANSWER_ID = "__paperclip_other__"; @@ -1847,12 +1848,12 @@ function RequestToolActionCard({ {isPending ? inspectableArguments ? (

- Arguments and audit hash + Arguments 
             {payload.argumentsSummaryJson}
-

{payload.argumentsHash}

+
) :

Request arguments are unavailable.

: null} {isPending ? ( diff --git a/ui/src/components/ToolActionAuditHashCopy.test.tsx b/ui/src/components/ToolActionAuditHashCopy.test.tsx new file mode 100644 index 00000000000..9b7e40aedcc --- /dev/null +++ b/ui/src/components/ToolActionAuditHashCopy.test.tsx @@ -0,0 +1,46 @@ +// @vitest-environment jsdom + +import { act } from "react"; +import { createRoot, type Root } from "react-dom/client"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +const copyTextToClipboard = vi.fn<(text: string) => Promise>(); +vi.mock("@/lib/clipboard", () => ({ copyTextToClipboard: (text: string) => copyTextToClipboard(text) })); + +import { ToolActionAuditHashCopy } from "./ToolActionAuditHashCopy"; + +const hash = "8ca5312877235a4c0f6e9b1d2a7c3e5f9081b2c4d6e8fa1b3c5d7e9f0a2b4c6d"; + +describe("ToolActionAuditHashCopy", () => { + let container: HTMLDivElement; + let root: Root; + + beforeEach(() => { + copyTextToClipboard.mockReset(); + container = document.createElement("div"); + document.body.appendChild(container); + root = createRoot(container); + }); + + afterEach(() => { + act(() => root.unmount()); + container.remove(); + }); + + it("copies the hash without showing it", async () => { + copyTextToClipboard.mockResolvedValue(undefined); + await act(async () => root.render()); + expect(container.textContent).toBe("Copy audit hash"); + await act(async () => container.querySelector("button")!.click()); + expect(copyTextToClipboard).toHaveBeenCalledWith(hash); + expect(container.textContent).toBe("Copied"); + }); + + it("shows the hash as selectable text when the clipboard is blocked", async () => { + copyTextToClipboard.mockRejectedValue(new Error("blocked")); + await act(async () => root.render()); + await act(async () => container.querySelector("button")!.click()); + expect(container.textContent).toContain("Couldn’t copy"); + expect(container.textContent).toContain(hash); + }); +}); diff --git a/ui/src/components/ToolActionAuditHashCopy.tsx b/ui/src/components/ToolActionAuditHashCopy.tsx new file mode 100644 index 00000000000..bf3d12680fb --- /dev/null +++ b/ui/src/components/ToolActionAuditHashCopy.tsx @@ -0,0 +1,42 @@ +import { useState } from "react"; +import { Check, Copy, X } from "lucide-react"; +import { useCopyAction } from "@/lib/use-copy-action"; +import { Button } from "./ui/button"; + +/** + * The reviewed-arguments hash identifies exactly what an approval covers, which + * matters for audits but means nothing to the person approving. Keep it off the + * card and reachable through a copy action instead of printing the raw digest. + * When the clipboard is blocked, show the hash as selectable text so the audit + * value never becomes unreachable. + */ +export function ToolActionAuditHashCopy({ hash }: { hash: string }) { + const { status, copy } = useCopyAction(); + const [copyFailed, setCopyFailed] = useState(false); + const label = status === "copied" ? "Copied" : status === "failed" ? "Couldn’t copy" : "Copy audit hash"; + const Icon = status === "copied" ? Check : status === "failed" ? X : Copy; + return ( +
+ + {copyFailed ? ( +

+ {hash} +

+ ) : null} +
+ ); +} diff --git a/ui/src/components/task-chat/TaskChatCompactInteractionCard.tsx b/ui/src/components/task-chat/TaskChatCompactInteractionCard.tsx index d4a13a54624..dbcb7487716 100644 --- a/ui/src/components/task-chat/TaskChatCompactInteractionCard.tsx +++ b/ui/src/components/task-chat/TaskChatCompactInteractionCard.tsx @@ -30,6 +30,7 @@ import type { } from "@paperclipai/adapter-utils"; import { InteractionPreparationNotice } from "@/components/InteractionPreparationNotice"; import { IssueThreadInteractionCard } from "@/components/IssueThreadInteractionCard"; +import { ToolActionAuditHashCopy } from "@/components/ToolActionAuditHashCopy"; import { ConnectionIntentInteractionBody } from "@/features/connections/ConnectionIntentInteractionBody"; import { MarkdownBody } from "@/components/MarkdownBody"; import type { MentionOption } from "@/components/MarkdownEditor"; @@ -573,9 +574,7 @@ function ReceiptDisclosure({ 
               {interaction.payload.toolAction.argumentsSummaryJson}
-

- {interaction.payload.toolAction.argumentsHash} -

+ ) : null} {interaction.payload.secretProposal ? ( @@ -934,14 +933,12 @@ function ConfirmationCard({
- Arguments and audit hash + Arguments 
               {interaction.payload.toolAction.argumentsSummaryJson}
-

- {interaction.payload.toolAction.argumentsHash} -

+
) : null} diff --git a/ui/src/components/task-chat/TaskChatInteractionCard.test.tsx b/ui/src/components/task-chat/TaskChatInteractionCard.test.tsx index 2260e4454ae..7b2763446ad 100644 --- a/ui/src/components/task-chat/TaskChatInteractionCard.test.tsx +++ b/ui/src/components/task-chat/TaskChatInteractionCard.test.tsx @@ -3,7 +3,7 @@ import { act, forwardRef, useImperativeHandle, useState, type ForwardedRef } from "react"; import { flushSync } from "react-dom"; import { createRoot, type Root } from "react-dom/client"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { afterEach, beforeEach, describe, expect, it, onTestFinished, vi } from "vitest"; vi.mock("@/hooks/useCoordinatorWorkspaceEnabled", () => ({ useCoordinatorWorkspaceEnabled: () => ({ enabled: false, loaded: true }), @@ -148,7 +148,8 @@ describe("TaskChatInteractionCard", () => { expect(container.querySelector("pre")?.textContent).toBe(interaction.payload.toolAction!.argumentsSummaryJson); expect(details.textContent).toContain("November 8, 2030"); expect(details.textContent).toContain("[redacted]"); - expect(details.textContent).toContain(interaction.payload.toolAction!.argumentsHash); + expect(details.textContent).not.toContain(interaction.payload.toolAction!.argumentsHash); + expect(details.textContent).toContain("Copy audit hash"); await act(async () => details.querySelector("summary")!.click()); expect(details.open).toBe(true); expect(accept).not.toHaveBeenCalled(); @@ -160,6 +161,64 @@ describe("TaskChatInteractionCard", () => { expect(reject).not.toHaveBeenCalled(); }); + it("keeps raw tool ids and the audit hash out of the tool approval card text", async () => { + // Shape of the GitHub create_repository approval from the Nov 5 staging run. + const rawToolName = "mcp.app-gallery-github-42e51e3f-9377-447b-ad45-333beda8a50a-b873f178:create-repository"; + const argumentsHash = "8ca5312877235a4c0f6e9b1d2a7c3e5f9081b2c4d6e8fa1b3c5d7e9f0a2b4c6d"; + const base = structuredClone(pendingToolActionLongProposalInteraction); + const interaction: RequestConfirmationInteraction = { + ...base, + title: "Approve tool action", + summary: "This action needs your approval before it runs.", + payload: { + ...base.payload, + prompt: "Approve create_repository?", + detailsMarkdown: `Tool: \`${rawToolName}\`\nRisk: \`write\``, + target: { type: "custom", key: "tool-action:2f92393c-c044-4367-8b6e-b81e2e732165", revisionId: argumentsHash, label: rawToolName }, + toolAction: { + ...base.payload.toolAction!, + toolName: rawToolName, + toolDisplayName: "create_repository", + appDisplayName: "GitHub", + previewMarkdown: "create_repository · Name: brainrot-storefront-prototype · Private: true", + argumentsSummaryJson: JSON.stringify({ name: "brainrot-storefront-prototype", private: true }), + argumentsHash, + }, + }, + }; + const writeText = vi.fn().mockResolvedValue(undefined); + const originalClipboard = Object.getOwnPropertyDescriptor(navigator, "clipboard"); + Object.defineProperty(navigator, "clipboard", { configurable: true, value: { writeText } }); + onTestFinished(() => { + if (originalClipboard) Object.defineProperty(navigator, "clipboard", originalClipboard); + else delete (navigator as { clipboard?: unknown }).clipboard; + }); + function ReviewSurface() { + const [reviewing, setReviewing] = useState(false); + return setReviewing(true)} + onAcceptInteraction={vi.fn()} onRejectInteraction={vi.fn()} />; + } + flushSync(() => root.render()); + const hashLike = /[0-9a-f]{12,}/i; + expect(container.textContent).not.toMatch(hashLike); + await act(async () => { + Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Review request")!.click(); + }); + const details = container.querySelector("details")!; + await act(async () => details.querySelector("summary")!.click()); + expect(details.open).toBe(true); + expect(container.textContent).toContain("brainrot-storefront-prototype"); + expect(container.textContent).not.toMatch(hashLike); + expect(container.textContent).not.toContain("app-gallery"); + // The audit hash stays reachable for anyone who needs it, through a copy action. + const copy = Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Copy audit hash"); + expect(copy).toBeDefined(); + await act(async () => copy!.click()); + expect(writeText).toHaveBeenCalledWith(argumentsHash); + }); + it("does not render a pending confirmation in the timeline", () => { flushSync(() => { root.render( From 19239c11d1505959f42881492d8621027c39b7f8 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 11:42:21 -0600 Subject: [PATCH 07/11] fix(onboarding): move Use a custom gateway under Advanced on the API key step (#19510) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - New users connect a model in the onboarding wizard's "Connect a model" step. > - On the Claude API and Codex API paths, the step shows an API key card. > - The "Use a custom gateway" button sat directly under the key field, inside that card, with no gap. > - A tester reported that it hugs the field and that it belongs under Advanced. > - A custom gateway is an alternative route, not the next step after the key. > - This pull request moves the button into the step's existing Advanced disclosure. > - The benefit is a cleaner key card, and the gateway route stays one click away. ## Linked Issues or Issue Description No public issue exists. A manual new-user onboarding test on a staging cloud workspace found this bug. Related: #17253 added the custom gateway route. **What happened?** On the onboarding "Connect a model" step, API key path (Claude API and Codex/OpenAI API), the "Use a custom gateway" button is shown at the top level, directly against the API key text field. **Steps to reproduce** 1. Start onboarding and reach "Connect a model". 2. Choose "Use API key instead", then pick Claude (or OpenAI). 3. See "Use a custom gateway" directly under the "Enter API key here" field. **Expected behavior** The custom gateway control is in the Advanced section of the step. It is hidden until Advanced is opened. Source: manual new-user onboarding test on Paperclip Cloud staging (item N-16, P3, Claude API path). ## What Changed - `ui/src/components/OnboardingWizard.tsx`: render `OnboardingGatewaySetup` inside the connect step's existing `Advanced` `
` (above the runner select), not inside the API key card. It keeps the same gates as the card it left: the card is live and mounted, no connect progress, API credential mode, a created company, and a Claude or Codex adapter. - `ui/src/components/onboarding/OnboardingGatewaySetup.tsx`: accept an optional `className`. The wizard passes `-ml-3` so the ghost button's label lines up with the Advanced summary. - The gateway dialog and its save/select-model/connect flow are unchanged. - `ui/src/components/OnboardingWizard.test.tsx`: new test for the Claude API and Codex API paths. It asserts the button is in a closed `Advanced` disclosure, outside the key field's card, and that the gateway dialog opens after Advanced is expanded. ## Verification Red on unchanged source (`npx vitest run src/components/OnboardingWizard.test.tsx -t "custom gateway"`): ``` ❯ src/components/OnboardingWizard.test.tsx (212 tests | 2 failed | 208 skipped) × keeps Use a custom gateway in the closed Advanced disclosure on the Claude API key step × keeps Use a custom gateway in the closed Advanced disclosure on the Codex API key step AssertionError: Use a custom gateway should live inside the Advanced disclosure: expected null to be truthy Test Files 1 failed (1) Tests 2 failed | 2 passed | 208 skipped (212) ``` Green after the fix (same command): ``` Test Files 1 passed (1) Tests 4 passed | 208 skipped (212) ``` Neighbouring suites (`OnboardingWizard.test.tsx`, `ai-connections/AiProviderSetup.test.tsx`, `components/onboarding/*`), including the existing end-to-end custom gateway save/test/hire test: ``` Test Files 8 passed (8) Tests 270 passed (270) ``` - `pnpm check:token-gates`: all gates clean. - `tsc --noEmit` for `ui`: no new errors. The only errors are the existing unbuilt `@paperclipai/plugin-sdk/ui` import in `PluginOrganizationSwitcher`. Screenshots (Storybook `Onboarding/Agent arc › Connect · Claude API key`): | Before (master) | After (Advanced closed) | After (Advanced open) | | --- | --- | --- | | ![before](https://raw.githubusercontent.com/paperclipai/paperclip/1dae9c09f9a4070c66cd313dbb64baf1636c54d0/n16-before.jpg) | ![after collapsed](https://raw.githubusercontent.com/paperclipai/paperclip/1dae9c09f9a4070c66cd313dbb64baf1636c54d0/n16-after-collapsed.jpg) | ![after expanded](https://raw.githubusercontent.com/paperclipai/paperclip/1dae9c09f9a4070c66cd313dbb64baf1636c54d0/n16-after-expanded.jpg) | The red "This item doesn't exist or was moved." line in the screenshots comes from the story's runner fixture. It is present on master too and is not related. ## Risks - Low risk. This is a layout move of one control. The gateway flow and its gates are unchanged. - The control is now one click deeper. This is the intended outcome of the report. - After a gateway is saved, "Use a different API key" stays in the card, so a user can still leave the gateway route without opening Advanced. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, browser for Storybook screenshots). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- ui/src/components/OnboardingWizard.test.tsx | 24 ++++++++++++++++++ ui/src/components/OnboardingWizard.tsx | 25 ++++++++++++------- .../onboarding/OnboardingGatewaySetup.tsx | 4 ++- 3 files changed, 43 insertions(+), 10 deletions(-) diff --git a/ui/src/components/OnboardingWizard.test.tsx b/ui/src/components/OnboardingWizard.test.tsx index f4c2dc47e84..38aac59711a 100644 --- a/ui/src/components/OnboardingWizard.test.tsx +++ b/ui/src/components/OnboardingWizard.test.tsx @@ -2138,6 +2138,30 @@ describe("OnboardingWizard restore-gate (stale localStorage across accounts)", ( return { root, queryClient }; } + // N-16: the gateway is an advanced route, not the next thing after the key. + // It sat directly under the API key field, inside the key card; it belongs + // in the step's Advanced disclosure, hidden until that is opened. + it.each([["Claude API", "claude_local", /Claude.*API/], ["Codex API", "codex_local", /OpenAI.*API/]] as const)("keeps Use a custom gateway in the closed Advanced disclosure on the %s key step", async (_path, adapterType, source) => { + const handles = await openStep4({ adapterType, runner: "paperclip", credentialMode: "api" }); + try { + await pickSource(source); + const keyField = document.body.querySelector('input[type="password"]') as HTMLInputElement; + expect(keyField, "the API key field should be showing").toBeTruthy(); + const gateway = () => [...document.body.querySelectorAll("button")].find(entry => entry.textContent?.trim() === "Use a custom gateway"); + expect(gateway(), "the custom gateway control should still exist").toBeTruthy(); + const advanced = gateway()!.closest("details"); + expect(advanced, "Use a custom gateway should live inside the Advanced disclosure").toBeTruthy(); + expect(advanced!.querySelector(":scope > summary")?.textContent?.trim()).toBe("Advanced"); + expect(advanced!.open, "Advanced should start closed").toBe(false); + expect(advanced!.contains(keyField), "the key field stays outside Advanced").toBe(false); + await act(async () => { advanced!.open = true; }); + await flushReact(); + await act(async () => gateway()!.click()); + await flushReact(); + expect(document.querySelector('input[aria-label="Provider URL"]'), "the gateway dialog should still open from Advanced").toBeTruthy(); + } finally { await act(async () => handles.root.unmount()); } + }); + it.each([["successful refresh", false], ["failed refresh and retry", true]] as const)("saves a custom gateway without probing, then explicitly tests and hires with its selected model: %s", async (_scenario, failRefresh) => { const gatewayAccounts: import("@paperclipai/shared").AiConnectionList = { currentUserId: "user-1", canManageConnections: true, connections: [], diff --git a/ui/src/components/OnboardingWizard.tsx b/ui/src/components/OnboardingWizard.tsx index 7b7297f7a31..01d2eaf323b 100644 --- a/ui/src/components/OnboardingWizard.tsx +++ b/ui/src/components/OnboardingWizard.tsx @@ -3157,15 +3157,6 @@ function OnboardingWizardInner({ onChange={setModel} open={modelOpen} onOpenChange={setModelOpen} allowDefault={false} required groupByProvider={false} creatable={connectionModels.models.length === 0} presentation="native" />} - {createdCompanyId && (adapterType === "codex_local" || adapterType === "claude_local") && { - setGatewayConnection({ companyId: createdCompanyId, adapterType, binding }); - setApiKey(""); - setModel(""); - setError(null); - }} />} {(hasGatewaySelection || connectionModels) && From 30a0b5982e03bc9a21da16da8600d88a728f886d Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 11:48:54 -0600 Subject: [PATCH 08/11] fix(runner): let Codex agents generate images with their existing provider key (#19516) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip runner launches Codex agents in an isolated app-server with a sandboxed shell > - On the Codex API path, the agent's OpenAI API key reaches only the trusted app-server process, never the shell > - The runner launched Codex with `--disable image_generation`, so Codex's built-in image tool was not available > - Codex then falls back to its `image_gen.py` script, which needs `OPENAI_API_KEY` in the shell, so the agent asked the user for a second key > - This pull request keeps Codex's built-in image tool on, so it uses the agent's existing provider auth, and lets the sandbox read only the folder where Codex saves the images > - The benefit is that a Codex agent can make images without a second key, and the provider key still does not reach the shell ## Linked Issues or Issue Description **What happened?** Manual onboarding test on Paperclip Cloud staging (Codex API path): "Open AI requests another api key for image model even though it has an active api key powering the agent." **Steps to reproduce** 1. Create a company with a Codex agent that uses an OpenAI API key. 2. Ask the agent to generate an image. 3. The agent asks for another OpenAI API key. **Expected behavior** The agent uses the OpenAI API key that already powers it. **Root cause** `createIsolatedCodexAppServerArgs` passed `--disable image_generation` and the thread config set `features.image_generation: false`. The pinned Codex 0.160.0 binary has two image paths. Its own instructions say the built-in `image_gen` tool "Does not require `OPENAI_API_KEY`". The fallback CLI path "Requires `OPENAI_API_KEY`". The runner removes `OPENAI_API_KEY` from the shell on purpose (`codex-shell-environment.ts`), so the fallback can never work and the agent asks the user. ## What Changed - `codex-security-config.ts`: remove `--disable image_generation` and `features.image_generation: false`. Codex's built-in tool runs in the trusted app-server with the agent's existing auth. - `codex-security-config.ts`: in the workspace and plan permission profiles, grant **read** on `CODEX_HOME/generated_images` only. Codex saves images there and tells the agent to copy them into the workspace. The rest of `CODEX_HOME` (for example `auth.json`) stays denied. Preconfigured-home mode is unchanged (it does not deny `CODEX_HOME`). - `eval-gateway-runtime.ts`: set `image_generation = false` in the controller-gateway Codex config. That gateway admits no hosted tools (`unqualified_hosted_tool`), so qualified evals keep the earlier behavior. - Tests: two new N-12 tests, and updates to the tests that asserted the old disable flag. ## Verification Red on `master` (819420a60a), same test file before the source change: ``` × N-12: lets an API-key Codex agent generate images with its existing provider key instead of asking for another 5ms × N-12: the pinned Codex binary keeps built-in image generation enabled under the isolated launch 4400ms ⎯⎯⎯⎯⎯⎯⎯ Failed Tests 2 ⎯⎯⎯⎯⎯⎯⎯ FAIL src/drivers/codex/codex-security-config.test.ts > Codex security configuration > N-12: lets an API-key Codex agent generate images with its existing provider key instead of asking for another AssertionError: expected [ '-c', …(22) ] to not include 'image_generation' 378| expect(args).not.toContain("image_generation"); FAIL src/drivers/codex/codex-security-config.test.ts > Codex security configuration > N-12: the pinned Codex binary keeps built-in image generation enabled under the isolated launch AssertionError: expected 'image_generation …' to match /\strue\s*$/ + Received: "image_generation stable false" Tests 2 failed | 17 skipped (19) ``` Green after the fix: ``` Tests 2 passed | 17 skipped (19) ``` Neighbouring suites (`src/drivers/codex`, `src/cli/eval-gateway-runtime.test.ts`, `src/provider-events.test.ts`): ``` Test Files 25 passed (25) Tests 429 passed (429) ``` Real sandbox check with the pinned Codex 0.160.0 binary (`codex sandbox` with the generated permission profile, macOS seatbelt): ``` generated_images/t1/img.png status=0 "PNGDATA" # agent can read the saved image auth.json status=1 "cat: ...: Operation not permitted" # rest of CODEX_HOME stays denied cp status=0 # agent can copy the image into the workspace ``` - `pnpm exec tsc --noEmit -p packages/paperclip-runner`: no errors in changed files (only pre-existing errors from the unbuilt `@paperclipai/paperclip-eval-kernel` package). - `pnpm verify` exits 2 (changes outside resolved test graphs); required CI supplies the broad coverage. - **Test scope.** The red/green tests assert the launch configuration: the disable flag, the thread config, the sandbox grant, and the effective `image_generation` state that the pinned Codex binary reports. They do not drive a model. So they do not directly observe the reported "asks for a second key" behavior. The link from the flag to that behavior comes from the pinned Codex 0.160.0 bundled instructions: the built-in `image_gen` tool "Does not require `OPENAI_API_KEY`", and the fallback `scripts/image_gen.py` CLI "Requires `OPENAI_API_KEY`". The runner removes `OPENAI_API_KEY` from the shell. - Not run: a live paid image generation with an API-key Codex agent. There is no Codex staging company. ## Risks - Image generation uses the agent's OpenAI account. Images cost money on API-key billing. This is the expected behavior when the user asks for an image. - The agent can read images it (or an earlier thread of the same Codex home) generated. It cannot read other files in `CODEX_HOME`. - Codex's built-in `view_image` tool was already enabled, so the trusted process already reads image files for the model. This change does not add a new kind of file access to the provider. - If a future Codex release changes the image save folder, the agent can still see the image inline but cannot copy it. The N-12 binary test pins the feature state for the current Codex version. ### Security policy change (needs maintainer sign-off) **Why it was disabled.** #12364 ("isolate Codex security configuration") disabled host apps, plugins, memories, multi-agent behavior, and image generation as one group. Its goal was a closed launch configuration that does not depend on ambient Codex defaults. The PR records no specific threat for image generation. This PR keeps the other features disabled. **Why re-enabling it is safe.** - The tool is a hosted tool. It runs at OpenAI inside the same model request that the trusted app-server already makes. It adds no new network endpoint, process, or shell capability. - No credential moves. `OPENAI_API_KEY` stays out of the shell, argv, and serialized config (the N-12 test asserts this). - The new sandbox grant is read-only and covers only `CODEX_HOME/generated_images`. The rest of `CODEX_HOME` (`auth.json`, config, sessions) stays denied. The real `codex sandbox` check above confirms this. In plan mode the grant is also read-only. - Codex's `view_image` tool was already enabled. So the trusted process already sends local image content to the provider. - Controller-gateway evals keep the tool off (`image_generation = false`), because that gateway admits no hosted tools. **Credentials and billing.** - API-key path: the hosted tool uses the agent's existing OpenAI API key through the app-server. OpenAI bills image generation to that API account at image-model rates. - ChatGPT-subscription path: the tool uses the agent's existing ChatGPT sign-in. Use counts against that ChatGPT plan's limits. There is no separate API bill. - These two points follow from how Codex sends hosted tools in the provider request. I did not observe them in a live paid run. - Open question: Paperclip cost tracking uses provider token usage. I did not confirm that image-generation charges appear in Paperclip's per-agent cost and budget totals. If they do not, budget hard-stops could undercount image spend. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../src/cli/eval-gateway-runtime.test.ts | 1 + .../src/cli/eval-gateway-runtime.ts | 4 +- ...-app-server-driver.context-control.test.ts | 3 - .../codex/codex-security-config.test.ts | 55 ++++++++++++++++++- .../drivers/codex/codex-security-config.ts | 19 ++++++- 5 files changed, 73 insertions(+), 9 deletions(-) diff --git a/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts b/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts index 0ef09abd57f..dfdad937057 100644 --- a/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts +++ b/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts @@ -73,6 +73,7 @@ it.each([["codex", undefined], ["acpx", "claude"]])("routes %s/%s with only its expect(config).toContain('env_key = "PAPERCLIP_AI_PROVIDER_KEY"'); expect(config).toContain("requires_openai_auth = false"); expect(config).toContain("shell_snapshot = false"); + expect(config).toContain("image_generation = false"); expect(config).toContain('web_search = "disabled"'); expect(config).not.toContain(token); expect((await stat(join(env.CODEX_HOME!, "config.toml"))).mode & 0o777).toBe(0o600); diff --git a/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts b/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts index a4e27f9c180..0dbf4d7cba6 100644 --- a/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts +++ b/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts @@ -76,7 +76,9 @@ export async function prepareEvalGatewayRuntime( 'name = "Paperclip eval controller"', `base_url = ${JSON.stringify(descriptor.baseUrl + "/v1")}`, 'wire_api = "responses"', 'requires_openai_auth = false', 'env_key = "PAPERCLIP_AI_PROVIDER_KEY"', - '[features]', 'shell_snapshot = false', '', + // The controller gateway admits no hosted tools, so qualified evals keep + // Codex's hosted image tool off even though product runs enable it. + '[features]', 'shell_snapshot = false', 'image_generation = false', '', ].join("\n"), { mode: 0o600, flag: "wx" }); } else { Object.assign(environment, { diff --git a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.context-control.test.ts b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.context-control.test.ts index 13e83500aba..f8f9d55e27d 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.context-control.test.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.context-control.test.ts @@ -129,7 +129,6 @@ describe("Codex app-server Codex driver", () => { "skills.include_instructions": false, include_apps_instructions: false, include_collaboration_mode_instructions: true, - "features.image_generation": false, }, permissions: "paperclip-runner-workspace-only", runtimeWorkspaceRoots: [WORKSPACE], @@ -160,8 +159,6 @@ describe("Codex app-server Codex driver", () => { expect.stringContaining( 'shell_environment_policy.set={PATH="/bin",LANG="C.UTF-8"}', ), - "--disable", - "image_generation", ]), ); expect( diff --git a/packages/paperclip-runner/src/drivers/codex/codex-security-config.test.ts b/packages/paperclip-runner/src/drivers/codex/codex-security-config.test.ts index b1b9f42cb3a..c8e15ed4d5c 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-security-config.test.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-security-config.test.ts @@ -1,6 +1,8 @@ import { createRequire } from "node:module"; import { execFileSync } from "node:child_process"; -import { dirname, resolve } from "node:path"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; import { resolvePinnedCodexCommand } from "./codex-command.js"; import { evalProviderTransportOptions } from "../../cli/eval-provider-runtime.js"; import { describe, expect, it } from "vitest"; @@ -358,10 +360,59 @@ describe("Codex security configuration", () => { "features.plugins": false, "features.multi_agent": false, "features.memories": false, - "features.image_generation": false, }); }); + it("N-12: lets an API-key Codex agent generate images with its existing provider key instead of asking for another", () => { + // Codex API path: the agent's OpenAI key reaches only the trusted + // app-server process. Codex's built-in image tool uses that provider auth; + // with it disabled the agent falls back to an OPENAI_API_KEY script that + // the sandboxed shell can never see, so it asks the user for a second key. + const source = { + HOME: "/host/home", CODEX_HOME: "/host/codex", PATH: "/safe/bin", + OPENAI_API_KEY: "agent-provider-key", + }; + const args = createIsolatedCodexAppServerArgs(source); + const serialized = args.join("\n"); + expect(args).not.toContain("image_generation"); + for (const mode of ["default", "plan"] as const) { + expect(createSecuredCodexThreadParams("/workspace", mode, true, false, source).config) + .not.toHaveProperty("features.image_generation"); + } + // The tool saves results under CODEX_HOME/generated_images; the agent must + // be able to read (and so copy) them into the workspace, nothing else there. + for (const profile of ["paperclip-runner-workspace-only", "paperclip-runner-workspace-read-only"]) { + const rules = args.find((arg) => arg.startsWith(`permissions.${profile}.filesystem=`))!; + expect(rules).toContain('"/host/codex"="none"'); + expect(rules).toContain('"/host/codex/generated_images"="read"'); + } + // The existing key is reused by the provider, never exposed to the shell. + expect(serialized).not.toContain("OPENAI_API_KEY"); + expect(serialized).not.toContain("agent-provider-key"); + }); + + it("N-12: grants generated images in preconfigured mode only when CODEX_HOME is outside HOME", () => { + const inside = createIsolatedCodexAppServerArgs({ HOME: "/users/executor", CODEX_HOME: "/users/executor/.codex", PAPERCLIP_RUNNER_HOME_MODE: "preconfigured" }).join("\n"); + expect(inside).not.toContain("generated_images"); + const outside = createIsolatedCodexAppServerArgs({ HOME: "/users/executor", CODEX_HOME: "/var/lib/codex-home", PAPERCLIP_RUNNER_HOME_MODE: "preconfigured" }).join("\n"); + expect(outside.match(/"\/var\/lib\/codex-home\/generated_images"="read"/g)).toHaveLength(2); + expect(outside).not.toContain('"/var/lib/codex-home"='); + }); + + it("N-12: the pinned Codex binary keeps built-in image generation enabled under the isolated launch", () => { + const codexHome = mkdtempSync(join(tmpdir(), "paperclip-n12-codex-home-")); + try { + const args = createIsolatedCodexAppServerArgs({ PATH: process.env.PATH, HOME: codexHome, CODEX_HOME: codexHome }); + const output = execFileSync(resolvePinnedCodexCommand(), [...args.slice(0, -1), "features", "list"], { + env: { PATH: process.env.PATH, HOME: codexHome, CODEX_HOME: codexHome }, + encoding: "utf8", timeout: 30_000, maxBuffer: 256 * 1024, + }); + expect(output.split("\n").find((line) => /^image_generation\s/.test(line))).toMatch(/\strue\s*$/); + } finally { + rmSync(codexHome, { recursive: true, force: true }); + } + }); + it("keeps automatic execution inside the workspace without host credentials and with normal network access", () => { const args = createIsolatedCodexAppServerArgs( { diff --git a/packages/paperclip-runner/src/drivers/codex/codex-security-config.ts b/packages/paperclip-runner/src/drivers/codex/codex-security-config.ts index 0121dce2e86..0130c7ba024 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-security-config.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-security-config.ts @@ -190,7 +190,10 @@ const SKILLLESS_BASE_CONFIG = { "features.plugins": false, "features.multi_agent": false, "features.memories": false, - "features.image_generation": false, + // Image generation stays on: Codex's built-in tool runs in the trusted + // app-server with the agent's existing provider auth. Disabling it makes the + // agent fall back to an OPENAI_API_KEY script the sandboxed shell cannot see, + // so it asks the user for a second key (N-12). } as const; export function codexCommandEnvironment( @@ -288,6 +291,16 @@ export function createIsolatedCodexAppServerArgs( .map((value) => resolve(value)), ), ]; + // Codex saves built-in image results under CODEX_HOME/generated_images and + // tells the agent to copy them into the workspace. Expose only that folder. + // A preconfigured HOME is already granted, so add the rule only when the + // Codex home sits outside it (never narrow an existing HOME write grant). + const generatedImagesRoot = typeof source.CODEX_HOME === "string" && source.CODEX_HOME.trim() + ? join(resolve(source.CODEX_HOME), "generated_images") : undefined; + const preconfiguredHome = usesPreconfiguredHome(source) && source.HOME ? resolve(source.HOME) : undefined; + const generatedImageRoots = generatedImagesRoot + && !(preconfiguredHome && codexPathInside(preconfiguredHome, generatedImagesRoot)) + ? [generatedImagesRoot] : []; const filesystemRules = [ `":root"="none"`, `":minimal"="read"`, @@ -296,6 +309,7 @@ export function createIsolatedCodexAppServerArgs( ...protectedPaths.map((path) => `${tomlString(resolve(path))}="none"`), ...(usesPreconfiguredHome(source) && source.HOME ? [`${tomlString(source.HOME)}="write"`] : []), ...readOnlyRoots.map((path) => `${tomlString(resolve(path))}="read"`), + ...generatedImageRoots.map((path) => `${tomlString(path)}="read"`), ...gitRoots.read.map((path) => `${tomlString(path)}="read"`), ...gitRoots.write.map((path) => `${tomlString(path)}="write"`), ...(instructionWorkingCopyRoot ? [`${tomlString(resolve(instructionWorkingCopyRoot))}="write"`] : []), @@ -311,6 +325,7 @@ export function createIsolatedCodexAppServerArgs( ...protectedPaths.map((path) => `${tomlString(resolve(path))}="none"`), ...(usesPreconfiguredHome(source) && source.HOME ? [`${tomlString(source.HOME)}="read"`] : []), ...readOnlyRoots.map((path) => `${tomlString(resolve(path))}="read"`), + ...generatedImageRoots.map((path) => `${tomlString(path)}="read"`), ...[...gitRoots.read, ...gitRoots.write].map((path) => `${tomlString(path)}="read"`), ...(instructionWorkingCopyRoot ? [`${tomlString(resolve(instructionWorkingCopyRoot))}="read"`] : []), ...(source.PAPERCLIP_GITHUB_BROKER_TOKEN && source.GH_CONFIG_DIR @@ -358,8 +373,6 @@ export function createIsolatedCodexAppServerArgs( // as the turn plan (`turn/plan/updated` -> `plan.updated`). "-c", CODEX_UPDATE_PLAN_TOOL_CONFIG, - "--disable", - "image_generation", ...(externalRunnerSandbox ? ["--dangerously-bypass-approvals-and-sandbox"] : []), From e3fb4bc73ae412c91eb69800c60893a8e1486cab Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 11:49:28 -0600 Subject: [PATCH 09/11] fix(native): recover saved listener startup failures (#19479) ## Thinking Path > - Paperclip manages AI agents and their work. > - Native recovery must preserve saved work and respect current task authority. > - A closed local API listener can stop startup before a provider starts. > - #19462 gives new failures a bounded bootstrap retry. > - Older saved failures still have a generic terminal code and a recovery hold. > - This pull request verifies the old producer outcome and uses the existing safe replacement path. > - Eligible tasks continue once. Unknown provider outcomes and user holds remain blocked. ## Linked Issues or Issue Description Refs: #19462 **What happened?** A restart can leave an old listener startup failure in `terminal_failure` with `native_session_interrupted`. Startup recovery does not claim it after #19462. A completed file recovery can also retain its no-replay hold. **Expected behavior** Continue automatically when durable records prove that no provider started, the source stopped, and the current files were saved. Keep the existing attempt budget and normal admission checks. **Steps to reproduce** 1. Close the actual API listener after a fresh native coordinator claim. 2. Persist the old executor's generic failure. Save the workspace and confirm source termination. 3. Restart recovery. The baseline schedules no successor. The repair schedules one successor after all receipts pass. **Paperclip version or commit** Baseline: `0252421d5c`. Old failure producer: `2f35eb1080`. **Deployment mode** Built from source. Local macOS arm64, Node 26.4.0, real embedded PostgreSQL, independent filesystem execution. No external AI service is used by the controlled acceptance. ## What Changed - Recognize the exact saved local-listener guard through one proof predicate shared by discovery and the locked transaction. - Require a fresh first attempt, matching execution bindings, no prior provider lineage, no provider events or tool calls, no checkpoint or admission receipt, and no cancellation. - Reuse current snapshot, exact stop, source block, attempt budget, and successor idempotency checks. - Wait for completed source file-recovery receipts. Retire only that same source's verified preservation hold in the successor transaction. - Bound discovery to 100 candidates per pass, keep completed history quiet, and wake through existing transactional producer notifications. - Retain retries after transient database and final ownership-proof errors; advance the bounded cursor and force a rescan so a failing page cannot hide later work. - Let only the replacement duty discharge a proved-empty notification during drain through its existing read-only readiness check, once per notification generation. Keep real work and concurrent changes outstanding without running replacement during drain. - Wake replacement at canonical local-stop proof writes and the existing dispatch eligibility boundary; these hints grant no new execution authority. - Add database regressions, scheduler quietness checks, real controller handoff coverage, and an opt-in controlled Runner continuation. ## Verification Final production head: `0df41b2fef2644195d1224d36984af3827da3e3b`. Local environment: macOS arm64, Node 26.4.0, TypeScript 7.0.2, Vitest 5.0.3, frozen-lockfile dependencies. Tests execute freshly; `--fsModuleCache` reuses only content-validated module transforms. - Native safe replacement: 62/62 PASS on isolated PostgreSQL 17 (63.18 seconds), including clean per-suite database disposal. All 62 assertions also passed on embedded PostgreSQL 18 beta, but its Darwin cleanup hook timed out; that invocation is not counted as a pass. Docker's disk was full, so the disposable PostgreSQL 17 data used a size-limited RAM mount, with no host mounts. The owned container was removed afterward. - Actual stopped-process producer suite: 31/31 PASS. A real exited child produces the stop receipt, and its replacement notification settles only after the outer transaction commits. - Workspace and separate-process drain regressions: 33/33 PASS (75.90 seconds). - Shared coordinator and execution-control lifecycle checks: 39/39 PASS (16.43 seconds). Pending work, concurrent changes, cold startup, shutdown, and passive drain boundaries remain covered. - Actual suppressed Cloud routine/drain workflow: PASS (34.64 seconds), using the unchanged test and timeouts. Its matching local RED reproduced CI's 462-query, zero-pending-query, one-idle-hold timeout after an admitted write committed during drain (69.46 seconds). - Actual historical controller handoff: PASS (48.40 seconds). A separate controller loads the original executor from `2f35eb1080`, writes its generic listener failure, and exits. Recovery at the final head admits exactly one successor. The real Rust Runner and controlled Codex process complete governed `paperclip_finish`; attempt two commits and the saved file remains intact. Only the child loads the old artifact, and both isolated artifacts are restored afterward. Rust source subtree `77961b70f177ec3c38d11abcaec92fd3d0e96ec0` matches the built binary. No external AI service is used. - Seven corrected existing fixtures pass: transactional stop-proof calls (5 cases) and notification inventory (2 cases). Their earlier CI failures exposed fixture calls outside the documented transaction and an outdated expected queue set. - Server typecheck: PASS. Red/green proofs also cover the original saved failure on baseline `0252421d5c` (zero successors instead of one), held-history quietness, bounded forward scanning through 100 parked rows and concurrent wakes, real snapshot/cleanup producer notifications, actual issue-row lock timeout, and task/history eligibility notifications. An obsolete prior-run question can be skipped to request another proof check; actual skip while the task is paused or the source owns provider tool activity schedules no successor. The failed-page regression starts with 25 persistent verifier errors and a healthy candidate 26. Baseline RED repeats the first page. Repair GREEN advances the healthy candidate, revisits the original failures after their fault clears, creates all 26 successors exactly once, and becomes quiet. Greptile is SUCCESS, 5/5 on the exact final head (2026-11-06 17:09:21 UTC). Final-head CI is GREEN: 56 successful checks, three intentional skips, one neutral result, and no pending or failing checks. [Attempt two and results](https://github.com/paperclipai/paperclip/actions/runs/42258352876/attempts/2) include passing build, typecheck, Rust, native server, package, browser E2E, serialized server, general server/chat tests, and final verification. Five jobs in the first attempt were interrupted together at 17:13:12 UTC; the native-server log records an explicit runner shutdown signal. The single infrastructure retry reran only those five jobs and dependent verification, preserving the 49 successful checks on the unchanged head. No timeout was weakened, and setup failures or interrupted invocations are not counted as passes. Focused commands from the repository root: ```sh ./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/services/remote-workspaces.integration.test.ts server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts -t 'native safe-replacement decision|reconciles the drained remote backend' --fsModuleCache --reporter=dot ./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/__tests__/execution-control-work.test.ts server/src/__tests__/delivery-work-coordinator.test.ts -t 'execution-control lifecycle|work coordinator' --fsModuleCache --reporter=dot node node_modules/typescript/bin/tsc -p server/tsconfig.json --noEmit --pretty false ``` The historical acceptance sets `PAPERCLIP_TEST_HISTORICAL_LISTENER=1` and uses an original-executor child artifact with a `try/finally` restore. Normal regressions require no Rust binary. ## Risks - Compatibility is deliberately limited to positively proved fresh remote startup failures. Other old ambiguous failures remain held. - Existing provider work, a changed task block, incomplete preservation, or revoked authority must prevent execution. Normal retry admission remains in control. - This PR does not deploy the change or mutate a hosted task. Hosted recovery still requires the repaired revision and all current proof gates. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution. The exact backend model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 28 ++ .../__tests__/execution-control-work.test.ts | 35 +++ .../historical-listener-continuation.ts | 134 ++++++++++ .../native-drain-controller.test-worker.ts | 50 ++++ .../status-card-notifications.test.ts | 2 +- server/src/index.ts | 4 +- .../src/services/delivery-work-coordinator.ts | 4 +- .../environment-cleanup-notifications.ts | 5 +- server/src/services/execution-control-work.ts | 22 +- .../explicit-native-continuation.test.ts | 4 +- .../src/services/native-local-process-stop.ts | 3 + .../native-case-handoff.test.ts | 2 +- ...er-restart-preparation.integration.test.ts | 157 ++++++++++- .../native-safe-replacement.test.ts | 112 +++++++- .../native-runtime/native-safe-replacement.ts | 217 ++++++++++++--- .../remote-workspaces.integration.test.ts | 246 ++++++++++++++++-- server/src/services/remote-workspaces.ts | 6 +- .../services/run-dispatch-notifications.ts | 1 + server/src/services/workspace-recovery.ts | 1 + 19 files changed, 961 insertions(+), 72 deletions(-) create mode 100644 server/src/__tests__/helpers/historical-listener-continuation.ts create mode 100644 server/src/__tests__/helpers/native-drain-controller.test-worker.ts diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index e0e66ab3a32..a2eebcada7c 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -3633,6 +3633,34 @@ re-read dependency readiness, since new dependencies need not change the displayed task status. Old blocked rows without a receipt remain held; no historical status backfill is performed. +Older native executors recorded the local callback-listener startup guard as +`native_session_interrupted`. A saved exact guard outcome can enter the same +bounded safe-replacement path after restart. This requires the first attempt, +an unused provider-session identity, matching saved execution bindings, no +provider events, checkpoint, admission receipt, tool call, or cancellation, and +the existing current snapshot and confirmed stop receipts. A generic error or +a pending provider-pack receipt alone grants no recovery authority. The same +transaction can retire that run's completed file-preservation hold only when +its bound recovery and finalization receipts prove completion. Unrelated holds, +later task blocks, normal dispatch admission, and the three-attempt budget remain +effective. + +Historical replacement discovery reads at most 100 candidates per pass and +keeps a cursor only for the remaining scan. A work notification preserves forward +progress and requests a final rescan, so held earlier rows cannot hide a later +eligible run. Completed scans of held history become quiet. Final snapshot +publication, verified lease cleanup, local process-stop proof, completed workspace +recovery, and dispatch eligibility changes notify the existing replacement worker +in their transactions. A cleared historical interaction only requests another +proof check; it grants no new continuation authority. Source provider effects, +task pause or cancellation, and the original authority checks still prevent replay. +Temporary decision failures retain the existing worker's retry and advance the +bounded cursor, with a forced rescan of earlier failures. A failed page cannot +hide later ready runs. During drain, the initialized replacement worker can read +readiness once per notification generation to discharge an empty notification +without starting replacement. Concrete pending work and changes arriving during +that read remain outstanding. Restart reconstructs the scan from durable records. + For legacy GitHub input without a saved App ID or its own response receipt, Task Retry may prove the endpoint's earlier non-null App claim from a processed response ledger created before that input. The proof stays within the same diff --git a/server/src/__tests__/execution-control-work.test.ts b/server/src/__tests__/execution-control-work.test.ts index 584eeafb6af..345bf605776 100644 --- a/server/src/__tests__/execution-control-work.test.ts +++ b/server/src/__tests__/execution-control-work.test.ts @@ -61,6 +61,41 @@ describe("execution-control lifecycle", () => { expect(run).toHaveBeenCalledTimes(runs + 1); expect(c.nextWakeAt()).toBeNull(); }); + it.each(["empty", "pending", "concurrent"] as const)("settles only proved-empty replacement notifications during drain (%s)", async kind => { + vi.useFakeTimers(); const owner = fakeOwner(); let draining = false; + const c = coordinator(owner, () => !draining); + const run = vi.fn(async () => {}), next = vi.fn(async (): Promise => null); + await registerExecutionControlWork(c, [{ queue: DELIVERY_QUEUES.executionReplacement, run, next, + settleEmptyWhilePaused: true }]).ready; + const before = idleWorkSnapshot().active; + draining = true; + let release: (() => void) | undefined; + if (kind === "pending") next.mockResolvedValue(Date.now() + 15_000); + if (kind === "concurrent") next.mockImplementationOnce(() => new Promise(resolve => { + release = () => resolve(null); + })); + await owner.transaction(tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement)); + await vi.advanceTimersByTimeAsync(15_000); + if (kind === "concurrent") { + expect(release).toBeTypeOf("function"); + next.mockResolvedValue(Date.now() + 15_000); + await owner.transaction(tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement)); + release!(); await vi.advanceTimersByTimeAsync(1); + } + expect(run).toHaveBeenCalledTimes(1); + expect(idleWorkSnapshot().active).toBe(before + (kind === "empty" ? 0 : 1)); + const reads = next.mock.calls.length; + await vi.advanceTimersByTimeAsync(60_000); + expect(next).toHaveBeenCalledTimes(reads); + expect(run).toHaveBeenCalledTimes(1); + if (kind !== "empty") { + draining = false; next.mockResolvedValue(null); + await vi.advanceTimersByTimeAsync(15_000); + expect(run).toHaveBeenCalledTimes(2); + } + expect(idleWorkSnapshot().active).toBe(before); + expect(c.nextWakeAt()).toBeNull(); + }); it("keeps a change arriving during a sweep and does not overlap it", async () => { vi.useFakeTimers(); const owner = fakeOwner(); const c = coordinator(owner); let release!: () => void; diff --git a/server/src/__tests__/helpers/historical-listener-continuation.ts b/server/src/__tests__/helpers/historical-listener-continuation.ts new file mode 100644 index 00000000000..332003b42ee --- /dev/null +++ b/server/src/__tests__/helpers/historical-listener-continuation.ts @@ -0,0 +1,134 @@ +import { createServer } from "node:http"; +import { spawn } from "node:child_process"; +import { createRequire } from "node:module"; +import { join, resolve } from "node:path"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { eq } from "drizzle-orm"; +import { completionContracts, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; +import { expect } from "vitest"; +import { createRunDispatch } from "../../modules/run-dispatch/index.js"; +import { prepareNativeHeartbeatRun } from "../../services/native-runtime/prepare-native-run.js"; +import { buildNativeExecutionInput } from "../../services/native-runtime/native-execution-input.js"; +import { buildNativeCompletionContract } from "../../services/native-runtime/completion-contracts.js"; +import { nativeRuntimeContextFixture } from "../../services/native-runtime/runtime-context.test-fixture.js"; +import { materializeAsset } from "../../services/native-runtime/runtime-context.js"; +import { executePaperclipNativeSession } from "../../services/native-runtime/native-session-executor.js"; +import { reconcileNativeFinalizations } from "../../services/native-runtime/native-finalization-reconciler.js"; +import { canonicalNativeRuntimeContextDigest, createNativeSessionBackend, createRunnerdCodexTransport, defaultCapabilityRunnerdBinary } from "../../vendor/paperclip-runner/index.js"; +import { registerLocalRuntimeApiServer } from "../../runtime-api.js"; +import { registerRunnerPrpAuthority, runnerPrpWebSocketInternals, setupRunnerPrpWebSocketServer } from "../../realtime/runner-prp-ws.js"; + +async function prepare(db: Db, runId: string, cwd: string, environmentLeaseId: string) { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const [issue] = await db.select().from(issues).where(eq(issues.id, run.nativeIssueId ?? String(run.contextSnapshot?.issueId))); + await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, issue.id)); + const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId }); + const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issue.id)); + const runtimeContext = nativeRuntimeContextFixture(); + runtimeContext.instructions.bundle = await materializeAsset([{ path: "AGENTS.md", mode: 0o444, + content: Buffer.from("Complete the saved task once.\n") }]); + runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(runtimeContext); + const execution = buildNativeExecutionInput({ companyId: run.companyId, agentId: run.agentId, runId, issue, + taskPrompt: "Preserve the saved file and complete the task once.", provider: "codex", + workspace: { id: issue.executionWorkspaceId!, cwd, repoUrl: null, repoRef: null, branchName: null }, + normalizedSessionId: native.normalizedSessionId, runtimeContext, + completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", + contract: buildNativeCompletionContract(issue, { revision: contract.revision }) }, + }); + const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run.runnerProfileJson, ...current.runnerProfileJson, nativeExecutionInput: execution } }) + .where(eq(heartbeatRuns.id, runId)); + return { run: current, native, execution }; +} + +export async function recordHistoricalListenerFailure(input: { + db: Db; connectionString: string; runId: string; cwd: string; environmentId: string; leaseId: string; +}) { + await input.db.update(heartbeatRuns).set({ status: "running", finishedAt: null, error: null, errorCode: null }) + .where(eq(heartbeatRuns.id, input.runId)); + const { execution, native } = await prepare(input.db, input.runId, input.cwd, input.leaseId); + await input.db.insert(nativeRunFinalizations).values({ runId: input.runId, companyId: execution.binding.companyId, + issueId: execution.binding.issueId, phase: "observed" }); + const child = spawn(process.execPath, ["--import", createRequire(import.meta.url).resolve("tsx"), + resolve(import.meta.dirname, "native-drain-controller.test-worker.ts")], { stdio: ["ignore", "pipe", "pipe", "ipc"], env: process.env }); + const logs: string[] = []; + child.stdout?.on("data", data => logs.push(String(data))); + child.stderr?.on("data", data => logs.push(String(data))); + const exited = new Promise(resolveExit => child.once("exit", resolveExit)); + try { + const result = await new Promise((resolveResult, reject) => { + const timeout = setTimeout(() => reject(new Error(logs.join("").slice(-3000))), 60_000); + child.once("error", reject); + child.on("message", (message: { ready?: boolean; done?: boolean; error?: string }) => { + if (message.ready) child.send({ connectionString: input.connectionString, options: { execution, + runnerInstanceId: native.runnerInstanceId, useRunnerd: true, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId: input.environmentId, + leaseId: input.leaseId, remoteCwd: input.cwd } } }); + if (message.done || message.error) { clearTimeout(timeout); message.error ? reject(new Error(message.error)) : resolveResult(message); } + }); + }); + expect(result).toMatchObject({ commands: 0, errorMessage: "local_runtime_api_listener_unavailable" }); + expect(await exited).toBe(0); + const [failed] = await input.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, input.runId)); + expect(failed).toMatchObject({ phase: "terminal_failure", attempt: 1, failureCode: "native_session_interrupted", + failureDetail: { message: "local_runtime_api_listener_unavailable", recoveryMode: "ambiguous_state", + providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false } }); + } finally { child.kill("SIGKILL"); await exited; } +} + +export async function completeHistoricalListenerSuccessor(input: { db: Db; runId: string; root: string; + prepareWorkspace: () => Promise<{ cwd: string; finalize: () => Promise }> }) { + const [scheduled] = await input.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)); + const dispatch = createRunDispatch(input.db); + expect(await dispatch.promoteScheduledRetry({ companyId: scheduled.companyId, runId: scheduled.id, + now: new Date(scheduled.scheduledRetryAt!.getTime() + 1) })).toMatchObject({ outcome: "promoted" }); + expect(await dispatch.cancelStaleQueuedRun({ companyId: scheduled.companyId, runId: scheduled.id, expectedStatus: "queued" })) + .toMatchObject({ outcome: "not_stale" }); + await input.db.update(heartbeatRuns).set({ status: "running", runtimeMode: "native", + nativeIssueId: String(scheduled.contextSnapshot?.issueId) }).where(eq(heartbeatRuns.id, scheduled.id)); + await input.db.update(issues).set({ executionRunId: scheduled.id }).where(eq(issues.id, String(scheduled.contextSnapshot?.issueId))); + const workspace = await input.prepareWorkspace(); + const { execution, native } = await prepare(input.db, scheduled.id, workspace.cwd, `controlled-${scheduled.id}`); + await input.db.insert(nativeRunFinalizations).values({ runId: scheduled.id, companyId: scheduled.companyId, + issueId: execution.binding.issueId, phase: "observed" }); + const home = join(input.root, "codex-home"), report = join(input.root, "result.json"), calls = join(input.root, "calls.log"); + await mkdir(home, { recursive: true }); + await writeFile(report, JSON.stringify({ schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", + summary: "Saved task completed after historical controller failure.", completionClaim: { + contractRevision: execution.completionContract.contract.revision, objectiveSatisfied: true, + criteria: execution.completionContract.contract.criteria.map(criterion => ({ criterionId: criterion.id, status: "satisfied", evidenceRefs: [] })), + remainingWork: [] }, evidence: [], verification: [], artifacts: [], attentionRequests: [] })); + const server = createServer(); + registerLocalRuntimeApiServer(server); + await new Promise(resolveListen => server.listen(0, "127.0.0.1", resolveListen)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("TCP listener required"); + setupRunnerPrpWebSocketServer(server, { apiUrl: `http://127.0.0.1:${address.port}` }); + const bundle = createRunnerdCodexTransport({ runnerBinary: defaultCapabilityRunnerdBinary(), + codexCommand: resolve(import.meta.dirname, "../../../../packages/paperclip-runner/runner/target/debug/fake-codex-app-server"), + codexArgs: ["--state-file", join(input.root, "provider.json"), "--terminal-result-file", report, "--call-log", calls, "--record-process-start"], + stateDirectory: input.root, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, + runtimeContext: execution.runtimeContext, runnerRuntimeContext: execution.runtimeContext, + sourceCodexHome: home, environment: { CODEX_HOME: home, PATH: process.env.PATH }, + prpIdentity: { runnerInstanceId: native.runnerInstanceId, environmentLeaseId: native.environmentLeaseId, + runId: scheduled.id, normalizedSessionId: native.normalizedSessionId, turnId: native.turnId, itemId: native.itemId }, + controlPlaneRegistration: authority => registerRunnerPrpAuthority({ companyId: scheduled.companyId, runId: scheduled.id, authority }), + }); + const backend = createNativeSessionBackend(execution, { runnerInstanceId: native.runnerInstanceId, + environment: { CODEX_HOME: home, PATH: process.env.PATH }, codexTransportFactory: () => bundle.transport }); + try { + expect(await executePaperclipNativeSession({ db: input.db, execution, backend, useRunnerd: true, + runnerInstanceId: native.runnerInstanceId, turnTimeoutMs: 15_000 })).toMatchObject({ exitCode: 0, timedOut: false }); + await workspace.finalize(); + await reconcileNativeFinalizations(input.db, [scheduled.id]); + const [finished] = await input.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, scheduled.id)); + expect(finished, JSON.stringify(finished.failureDetail)).toMatchObject({ phase: "committed", attempt: 2 }); + expect(await input.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, scheduled.id))) + .toMatchObject([{ schemaStatus: "accepted" }]); + expect((await readFile(calls, "utf8")).split("\n").filter(line => line === "process-start")).toHaveLength(1); + expect(await readFile(join(workspace.cwd, "saved.txt"), "utf8")).toBe("work survives a failed provider session\n"); + } finally { + await bundle.transport.close(); runnerPrpWebSocketInternals.resetForTests(); + server.closeAllConnections(); await new Promise(resolveClose => server.close(() => resolveClose())); + } +} diff --git a/server/src/__tests__/helpers/native-drain-controller.test-worker.ts b/server/src/__tests__/helpers/native-drain-controller.test-worker.ts new file mode 100644 index 00000000000..424d7598a24 --- /dev/null +++ b/server/src/__tests__/helpers/native-drain-controller.test-worker.ts @@ -0,0 +1,50 @@ +import { createServer } from "node:http"; +import { closeRegisteredClients, createDb, heartbeatRunEvents } from "@paperclipai/db"; +import { registerLocalRuntimeApiServer } from "../../runtime-api.js"; +import { setStartupRecoveryPhase } from "../../startup-recovery-state.js"; +import { heartbeatService } from "../../services/heartbeat.js"; +import { executePaperclipNativeSession } from "../../services/native-runtime/native-session-executor.js"; + +process.send?.({ ready: true }); +process.once("message", async (input: { + connectionString: string; + options: Omit[0], "db">; + providerEvidence?: boolean; +}) => { + const api = createServer((_request, response) => response.writeHead(404).end()); + registerLocalRuntimeApiServer(api); + let commands = 0; + let errorName: string | undefined; + let errorMessage: string | undefined; + try { + await new Promise(resolve => api.listen(0, "127.0.0.1", resolve)); + const db = createDb(input.connectionString); + try { + await executePaperclipNativeSession({ ...input.options, db, + runnerExecutionTarget: { ...input.options.runnerExecutionTarget!, + runner: { execute: async () => { commands++; throw new Error("Drained controller launched remote work"); } }, + } as never, + onEvent: async event => { + if (event.payload?.span !== "native.coordinator.claim") return; + if (input.providerEvidence) await db.insert(heartbeatRunEvents).values({ + companyId: input.options.execution.binding.companyId, agentId: input.options.execution.binding.agentId, + runId: input.options.execution.binding.runId, seq: 1_000_000, eventType: "turn.started", + sourceEventId: "controlled-provider-evidence" }); + setStartupRecoveryPhase("draining"); + await new Promise((resolve, reject) => api.close(error => error ? reject(error) : resolve())); + }, + }); + } catch (error) { + errorName = error instanceof Error ? error.name : String(error); + errorMessage = error instanceof Error ? error.message : String(error); + } + const heartbeat = heartbeatService(db); + try { + await heartbeat.drainRunningRunsForShutdown("SIGTERM", new Date(), [input.options.execution.binding.runId]); + } finally { await heartbeat.drainActiveRunExecutions(); } + await closeRegisteredClients(input.connectionString, { scope: "database" }); + process.send?.({ done: true, commands, errorName, errorMessage }, () => process.exit(0)); + } catch (error) { + process.send?.({ error: error instanceof Error ? error.stack : String(error) }, () => process.exit(1)); + } +}); diff --git a/server/src/__tests__/status-card-notifications.test.ts b/server/src/__tests__/status-card-notifications.test.ts index 0566ebdd394..7afd070f973 100644 --- a/server/src/__tests__/status-card-notifications.test.ts +++ b/server/src/__tests__/status-card-notifications.test.ts @@ -25,7 +25,7 @@ it.each([false, true])("keeps execution queue consumers intact and emits only a const write = vi.fn(async () => "written"); expect(await writeExecutionControl(owner as never, DELIVERY_QUEUES.workspaceRecovery, write, scope)).toBe("written"); expect(write).toHaveBeenCalledOnce(); - expect(new Set(settled.map(row => row.queue))).toEqual(new Set([DELIVERY_QUEUES.issueMonitors, DELIVERY_QUEUES.workspaceRecovery, + expect(new Set(settled.map(row => row.queue))).toEqual(new Set([DELIVERY_QUEUES.executionReplacement, DELIVERY_QUEUES.issueMonitors, DELIVERY_QUEUES.workspaceRecovery, ...(scoped ? [DELIVERY_QUEUES.statusCardEvaluation] : [])])); } finally { for (const stop of unsubscribe) stop(); } }); diff --git a/server/src/index.ts b/server/src/index.ts index 13c55b20807..0ba89a9fd11 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -81,7 +81,7 @@ import { embeddedPostgresOwnerPort } from "./embedded-postgres-owner.js"; import { deliverExecutionStatuses } from "./services/execution-status-delivery.js"; import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "./services/execution-recovery-resolution.js"; import { deliverPendingPauseResumes } from "./services/issue-tree-resume.js"; -import { reconcileSafeNativeReplacements } from "./services/native-runtime/native-safe-replacement.js"; +import { invalidateNativeReplacementDiscovery, reconcileSafeNativeReplacements } from "./services/native-runtime/native-safe-replacement.js"; import { reconcileAbandonedExecutionControl } from "./services/execution-control-reconciliation.js"; import { acknowledgeNativeStartupCleanup } from "./services/native-runtime/native-startup-cleanup.js"; import { executionControlSchedule, registerExecutionControlWork } from "./services/execution-control-work.js"; @@ -1344,7 +1344,7 @@ async function startServerWithDatabaseTeardown( { queue: DELIVERY_QUEUES.executionStatus, run: () => deliverExecutionStatuses(db), next: executionSchedule.status }, { queue: DELIVERY_QUEUES.executionDisposition, run: () => settleUnrecoverableExecutions(db), next: executionSchedule.disposition }, ...(heartbeat ? [ - { queue: DELIVERY_QUEUES.executionReplacement, run: () => reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession: run => verifyStoppedNativeSessionForReplacement(db, run) }), next: executionSchedule.replacement }, + { queue: DELIVERY_QUEUES.executionReplacement, run: () => reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession: run => verifyStoppedNativeSessionForReplacement(db, run) }), next: executionSchedule.replacement, onSettled: () => invalidateNativeReplacementDiscovery(db), settleEmptyWhilePaused: true }, { queue: DELIVERY_QUEUES.executionContinuation, run: async () => { await deliverReconciledExecutions(db, heartbeat.wakeup); await deliverPendingPauseResumes(db, heartbeat.wakeup); diff --git a/server/src/services/delivery-work-coordinator.ts b/server/src/services/delivery-work-coordinator.ts index 39df770662d..97dc7a8ea87 100644 --- a/server/src/services/delivery-work-coordinator.ts +++ b/server/src/services/delivery-work-coordinator.ts @@ -25,7 +25,7 @@ export function createDeliveryWorkCoordinator(input: { // do not postpone the obligation indefinitely. coalesceMs?: number; run: (signal: AbortSignal) => Promise; - hasPending: () => Promise; + hasPending: (runAllowed?: boolean) => Promise; // Scoped hints belong to the outer transaction, including uncertainty recovery. onSettled?: (hints?: ReadonlySet) => void; // Optional maintenance/retry deadline restored from durable state. @@ -75,7 +75,7 @@ export function createDeliveryWorkCoordinator(input: { let runFailed = false; controller = attempt; running = withDatabaseOperation(`delivery.${queue}`, () => Promise.resolve().then(() => reconcileDatabaseWork(input.owner, queue)) - .then(() => workerStopped || !(input.canRun() || (task.runDuringDrain && input.canReconcile?.())) ? undefined : task.run(attempt.signal).catch(error => { runFailed = true; throw error; })).then(() => workerStopped ? false : task.hasPending()).then(pending => { + .then(() => workerStopped || !(input.canRun() || (task.runDuringDrain && input.canReconcile?.())) ? undefined : task.run(attempt.signal).catch(error => { runFailed = true; throw error; })).then(() => workerStopped ? false : task.hasPending(input.canRun() || Boolean(task.runDuringDrain && input.canReconcile?.()))).then(pending => { pendingWork = pending || databaseWorkPending(input.owner, queue); if (pendingWork) schedule(task.retryMs); else { finishIntent?.(); finishIntent = undefined; } diff --git a/server/src/services/environment-cleanup-notifications.ts b/server/src/services/environment-cleanup-notifications.ts index fbbf3352359..6beca498981 100644 --- a/server/src/services/environment-cleanup-notifications.ts +++ b/server/src/services/environment-cleanup-notifications.ts @@ -31,7 +31,10 @@ export async function notifyEnvironmentCleanup(transaction: object, scope?: Envi await notifyNativeEnvironmentSettlement(transaction, scope); await notifyQueuedRunDispatch(transaction, scope); // A settled run lease can make that run's abandoned native owner releasable. - if (scope?.runId) await notifyDeliveryWork(transaction, DELIVERY_QUEUES.executionFinalization); + if (scope?.runId) { + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.executionFinalization); + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.executionReplacement); + } await notifyDeliveryWork(transaction, DELIVERY_QUEUES.environmentCleanup); await notifyDeliveryWork(transaction, DELIVERY_QUEUES.historicalCleanup); } diff --git a/server/src/services/execution-control-work.ts b/server/src/services/execution-control-work.ts index 7cedc0c2ea3..5a0e24f24e0 100644 --- a/server/src/services/execution-control-work.ts +++ b/server/src/services/execution-control-work.ts @@ -5,7 +5,7 @@ import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator. import { type DeliveryQueue } from "./delivery-work-notifications.js"; import { EXECUTION_RECONCILIATION_INTERVAL_MS } from "./execution-control-deadline.js"; import { obsoleteExecutionConversationHold, unresolvedExecutionDispositions } from "./execution-recovery-resolution.js"; -import { nativeReplacementPending } from "./native-runtime/native-safe-replacement.js"; +import { hasPendingNativeReplacements } from "./native-runtime/native-safe-replacement.js"; import { nextAbandonedNativeOwnerReleaseAt } from "./native-runtime/native-failed-run-owner-release.js"; import { hasPendingPauseResume } from "./issue-tree-resume.js"; @@ -16,6 +16,7 @@ export function registerExecutionControlWork( run: () => Promise; next: () => Promise; onSettled?: () => void; + settleEmptyWhilePaused?: boolean; }>, canRun: () => boolean = () => true, ) { @@ -24,6 +25,7 @@ export function registerExecutionControlWork( let reconciled = 0; let next: number | null = null; let earliestRunAt = 0; + let pausedGeneration = 0, pausedPending = true; const worker = coordinator.register(duty.queue, { retryMs: EXECUTION_RECONCILIATION_INTERVAL_MS, pausePassiveMaintenance: true, @@ -42,7 +44,18 @@ export function registerExecutionControlWork( next = deadline === null ? null : Math.max(deadline, earliestRunAt); reconciled = generation; }, - hasPending: async () => reconciled !== requested, + hasPending: async (runAllowed = true) => { + if (reconciled === requested) return false; + if (runAllowed || !duty.settleEmptyWhilePaused || !canRun() || reconciled === 0) return true; + if (pausedGeneration !== requested) { + const generation = requested; + const deadline = await duty.next(); + if (generation !== requested) return true; + pausedGeneration = generation; pausedPending = deadline !== null; next = deadline; + if (!pausedPending) reconciled = generation; + } + return pausedPending; + }, nextRunAt: () => next, }); return { ...worker, wake: () => { requested += 1; worker.wake(); } }; @@ -74,10 +87,7 @@ export function executionControlSchedule(db: Db) { return at === null ? null : deadline([new Date(at)]); }, async replacement() { - const rows = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns) - .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)) - .where(nativeReplacementPending()).limit(1); - return rows.length ? retry() : null; + return await hasPendingNativeReplacements(db) ? retry() : null; }, async continuation() { const rows = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and( diff --git a/server/src/services/explicit-native-continuation.test.ts b/server/src/services/explicit-native-continuation.test.ts index 0a9ed4725cc..756410efc6a 100644 --- a/server/src/services/explicit-native-continuation.test.ts +++ b/server/src/services/explicit-native-continuation.test.ts @@ -1586,7 +1586,7 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes await persistHeartbeatRunProcessMetadata(db, source.id, { pid: 999999999, processGroupId: null, startedAt: new Date().toISOString() }); await db.update(heartbeatRuns).set({ processPid: null }).where(eq(heartbeatRuns.id, source.id)); expect(await admit(f, true)).toBeNull(); - expect(await recordNativeLocalProcessStop(db, source)).toBe(true); + expect(await db.transaction(tx => recordNativeLocalProcessStop(tx as unknown as typeof db, source))).toBe(true); await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: source.id, agentId: f.agentId, eventType: PROCESS_START_REQUESTED }); // No PID was stored for the new launch, as when the server dies after spawn. @@ -2440,7 +2440,7 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes await db.update(issues).set({ conversationAgentId: f.agentId, conversationUserId: "board", conversationState: "active", conversationBoundaryCommentId: boundaryId }).where(eq(issues.id, f.issueId)); await db.update(issueRecoveryActions).set({ createdAt: new Date("2026-09-11T09:00:00Z") }).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)); - expect(await recordNativeLocalProcessStop(db, source)).toBe(true); + expect(await db.transaction(tx => recordNativeLocalProcessStop(tx as unknown as typeof db, source))).toBe(true); await db.update(heartbeatRuns).set({ processPid: null }).where(eq(heartbeatRuns.id, f.sourceRunId)); if (scenario === "after_reset_workspace_hold") { await db.update(heartbeatRuns).set({ processPid: 999999999, resultJson: { workspaceRestoreFailure: "restore_unsafe_archive" } }).where(eq(heartbeatRuns.id, f.sourceRunId)); diff --git a/server/src/services/native-local-process-stop.ts b/server/src/services/native-local-process-stop.ts index 37fc1569498..85d10f95359 100644 --- a/server/src/services/native-local-process-stop.ts +++ b/server/src/services/native-local-process-stop.ts @@ -6,6 +6,7 @@ import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js"; import { nativeSessionProviderEvidence } from "./native-runtime/native-session-resume.js"; import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; import os from "node:os"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; export const PROCESS_START_REQUESTED = "native.process_start_requested"; export const PROCESS_IDENTITY_RECORDED = "native.process_identity_recorded"; @@ -56,6 +57,8 @@ export async function recordNativeLocalProcessStop(db: Db, run: Pick lease.provider !== "local")) return false; if ((run.processPid && !absent(run.processPid)) || (run.processGroupId && !absent(-run.processGroupId))) return false; + await notifyDeliveryWork(db, DELIVERY_QUEUES.executionReplacement, + JSON.stringify({ companyId: run.companyId, runId: run.id })); await appendHeartbeatRunEvent(db, { companyId: run.companyId, runId: run.id, diff --git a/server/src/services/native-runtime/native-case-handoff.test.ts b/server/src/services/native-runtime/native-case-handoff.test.ts index 599b1e5584b..c13f5241e7c 100644 --- a/server/src/services/native-runtime/native-case-handoff.test.ts +++ b/server/src/services/native-runtime/native-case-handoff.test.ts @@ -84,7 +84,7 @@ describe("native persisted case handoff runtime", () => { expect(await isCaseHandoffSourceReleased(db, f.input)).toBe(false); const exited = once(child, "exit"); child.kill("SIGTERM"); await exited; const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); - expect(await recordNativeLocalProcessStop(db, run!)).toBe(true); + expect(await db.transaction(tx => recordNativeLocalProcessStop(tx as unknown as typeof db, run!))).toBe(true); // A new worker has no process-local state and reconstructs the release. expect(await isCaseHandoffSourceReleased(db, f.input)).toBe(true); expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]?.resultJson).toHaveProperty("caseHandoffExecutionReleased"); diff --git a/server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts index 20eff3138be..350eee86282 100644 --- a/server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts @@ -1,11 +1,21 @@ import { startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal } from "./native-execution-lease.js"; import { randomUUID } from "node:crypto"; -import { resolve } from "node:path"; +import { resolve, join } from "node:path"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; import { eq, sql } from "drizzle-orm"; -import { environmentLeases, environments, heartbeatRuns, nativeRunFinalizations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; +import { environmentLeases, environments, heartbeatRuns, nativeRunFinalizations, issueRecoveryActions, issues, completionContracts, nativeRunResults, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; import { beforeAll, expect, it, vi } from "vitest"; -import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js"; +import { spawn } from "node:child_process"; +import { createRequire } from "node:module"; +import { DurablePrpControlPlane, createNativeSessionBackend, createRunnerdCodexTransport, defaultCapabilityRunnerdBinary, canonicalNativeRuntimeContextDigest } from "../../vendor/paperclip-runner/index.js"; import { buildNativeExecutionInput } from "./native-execution-input.js"; +import { buildNativeCompletionContract } from "./completion-contracts.js"; +import { reconcileNativeFinalizations, claimNativeSessionResumptions } from "./native-finalization-reconciler.js"; +import { prepareNativeWorkspaceSync } from "./native-workspace-sync.js"; +import { workspaceOperationService } from "../workspace-operations.js"; +import type { EnvironmentLease } from "@paperclipai/shared"; +import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime"; +import { materializeAsset } from "./runtime-context.js"; import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; import { executePaperclipNativeSession } from "./native-session-executor.js"; import { registerNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; @@ -21,6 +31,147 @@ describeEmbeddedPostgres("native runner restart recovery: preparation", () => { let successor: typeof environment.current.successor; beforeAll(() => { ({ runtimeRoot, companyId, agentId, successor } = environment.current); }); + it.each(["recover", "provider event", "cancelled task"] as const)("reconciles the drained remote backend with %s", async mode => { + const fixture = await seedRun(`DRAINED-BACKEND-${mode.replaceAll(" ", "-")}`); + const environmentId = randomUUID(), leaseId = randomUUID(); + await fixture.db.insert(environments).values({ id: environmentId, name: `Drained backend ${environmentId}`, driver: "sandbox" }); + const [lease] = await fixture.db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId, + heartbeatRunId: fixture.runId, issueId: fixture.issueId, status: "active", + provider: "fixture", providerLeaseId: `drained-backend-${leaseId}` }).returning(); + const [issue] = await fixture.db.select().from(issues).where(eq(issues.id, fixture.issueId)); + const [contract] = await fixture.db.select().from(completionContracts).where(eq(completionContracts.issueId, fixture.issueId)); + const runtimeContext = nativeRuntimeContextFixture(); + runtimeContext.instructions.bundle = await materializeAsset([{ path: "AGENTS.md", mode: 0o444, + content: Buffer.from("Complete the controlled restart fixture task.\n") }]); + runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(runtimeContext); + const sandbox = join(runtimeRoot, `workspace-${fixture.runId}`); + await mkdir(sandbox, { recursive: true }); + const savedFile = join(sandbox, "saved-before-drain.txt"); + await writeFile(savedFile, "Preserve admitted work.\n"); + const execution = buildNativeExecutionInput({ companyId, agentId, runId: fixture.runId, + issue: { id: fixture.issueId, identifier: "NRR-DRAINED", title: "Continue after drain", description: null, workMode: "standard" }, + taskPrompt: "Continue the admitted work once.", provider: "codex", + workspace: { id: fixture.runId, cwd: sandbox, repoUrl: null, repoRef: null, branchName: null }, + normalizedSessionId: fixture.native.normalizedSessionId, runtimeContext, + completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", + contract: buildNativeCompletionContract(issue, { revision: contract.revision }) }, + }); + await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { + ...fixture.preparedProfile, nativeExecutionInput: execution, + } }).where(eq(heartbeatRuns.id, fixture.runId)); + const remoteSandbox = join(runtimeRoot, `remote-${fixture.runId}`); + const shell: CommandManagedRuntimeRunner = { execute: async input => { + const startedAt = new Date().toISOString(); + const child = spawn(input.command, input.args ?? [], { cwd: input.cwd, + env: { ...process.env, ...input.env }, stdio: ["pipe", "pipe", "pipe"] }); + let stdout = "", stderr = "", timedOut = false; + child.stdout.setEncoding("utf8").on("data", chunk => { stdout += chunk; }); + child.stderr.setEncoding("utf8").on("data", chunk => { stderr += chunk; }); + child.stdin.on("error", () => undefined); + child.stdin.end(input.stdin); + const timeout = setTimeout(() => { timedOut = true; child.kill("SIGKILL"); }, input.timeoutMs ?? 30_000); + try { return await new Promise((resolveResult, reject) => { + child.once("error", reject); + child.once("close", (exitCode, signal) => resolveResult({ stdout, stderr, exitCode, signal, timedOut, + pid: child.pid ?? null, startedAt })); + }); } finally { clearTimeout(timeout); } + } }; + const workspaceSync = await prepareNativeWorkspaceSync({ db: fixture.db, runId: fixture.runId, companyId, + workspaceId: fixture.runId, workspaceLocalDir: sandbox, lease: lease as unknown as EnvironmentLease, + target: { kind: "remote", transport: "sandbox", environmentId, leaseId, remoteCwd: remoteSandbox, runner: shell } }); + expect(workspaceSync).not.toBeNull(); + const child = spawn(process.execPath, ["--import", createRequire(import.meta.url).resolve("tsx"), + resolve(import.meta.dirname, "../../__tests__/helpers/native-drain-controller.test-worker.ts")], + { stdio: ["ignore", "pipe", "pipe", "ipc"], env: process.env }); + const logs: string[] = []; + child.stdout?.on("data", data => logs.push(String(data))); + child.stderr?.on("data", data => logs.push(String(data))); + const exited = new Promise(resolveExit => child.once("exit", resolveExit)); + try { + const result = await new Promise<{ commands: number; errorName: string; errorMessage: string }>((resolveResult, reject) => { + const deadline = setTimeout(() => reject(new Error(`Controller handoff timed out: ${logs.join("").slice(-2000)}`)), 60_000); + child.once("error", error => { clearTimeout(deadline); reject(error); }); + child.once("exit", code => { clearTimeout(deadline); reject(new Error(`Controller exited before handoff (${code}): ${logs.join("").slice(-2000)}`)); }); + child.on("message", (message: { ready?: boolean; done?: boolean; commands: number; errorName: string; errorMessage: string; error?: string }) => { + if (message.ready) child.send({ connectionString: environment.current.temporary.connectionString, providerEvidence: mode === "provider event", + options: { execution, runnerInstanceId: fixture.native.runnerInstanceId, useRunnerd: true, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId, leaseId, remoteCwd: remoteSandbox } } }); + if (message.done) { clearTimeout(deadline); resolveResult(message); } + if (message.error) { clearTimeout(deadline); reject(new Error(message.error)); } + }); + }); + expect(result).toMatchObject({ commands: 0, errorMessage: "local_runtime_api_listener_unavailable" }); + expect(await exited).toBe(0); + const [handoff] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId)); + if (mode === "provider event") { + expect(handoff).toMatchObject({ phase: "terminal_failure", nextAttemptAt: null, + failureDetail: { recoveryMode: "ambiguous_state", providerSessionEstablished: true } }); + expect(await claimNativeSessionResumptions({ db: fixture.db, runnerInstanceId: "no-provider-replay", + runIds: [fixture.runId], deferIneligible: false })).toHaveLength(0); + return; + } + expect(handoff).toMatchObject({ phase: "retryable_failure", attempt: 1, + failureDetail: { originalFailureCode: "local_runtime_api_listener_unavailable", recoveryMode: "bootstrap_retry", + providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false } }); + if (mode === "cancelled task") { + await fixture.db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, fixture.issueId)); + } + const claims = (await Promise.all(["successor-a", "successor-b"].map(runnerInstanceId => + claimNativeSessionResumptions({ db: fixture.db, runnerInstanceId, runIds: [fixture.runId], + now: new Date(handoff.nextAttemptAt!.getTime() + 1), deferIneligible: false }), + ))).flat(); + if (mode === "cancelled task") { expect(claims).toHaveLength(0); return; } + expect(claims).toHaveLength(1); + const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect(run).toMatchObject({ status: "running", nativeSessionId: fixture.native.normalizedSessionId }); + const [action] = await fixture.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId)); + expect(action).toMatchObject({ ownerType: "agent", ownerAgentId: agentId }); + if (process.env.PAPERCLIP_TEST_DRAIN_CONTINUATION === "1") { + const claim = claims[0]; + const root = join(runtimeRoot, `continued-${fixture.runId}`); + const home = join(root, "codex-home"); + await mkdir(home, { recursive: true }); + const report = join(root, "result.json"); + const calls = join(root, "calls.log"); + await writeFile(report, JSON.stringify({ schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", + summary: "Admitted work completed after controller restart.", completionClaim: { + contractRevision: execution.completionContract.contract.revision, objectiveSatisfied: true, + criteria: execution.completionContract.contract.criteria.map(criterion => ({ criterionId: criterion.id, + status: "satisfied", evidenceRefs: [] })), remainingWork: [] }, + evidence: [], verification: [], artifacts: [], attentionRequests: [] })); + const bundle = createRunnerdCodexTransport({ ...transportOptions(fixture, root), + runnerBinary: defaultCapabilityRunnerdBinary(), + codexArgs: ["--state-file", join(root, "provider.json"), "--terminal-result-file", report, + "--call-log", calls, "--record-process-start"], + runtimeContext: execution.runtimeContext, runnerRuntimeContext: execution.runtimeContext, + sourceCodexHome: home, environment: { CODEX_HOME: home, PATH: process.env.PATH }, + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, + }); + const backend = createNativeSessionBackend(execution, { runnerInstanceId: fixture.native.runnerInstanceId, + environment: { CODEX_HOME: home, PATH: process.env.PATH }, codexTransportFactory: () => bundle.transport }); + try { + expect(await executePaperclipNativeSession({ db: fixture.db, execution, backend, useRunnerd: true, + runnerInstanceId: fixture.native.runnerInstanceId, leaseOwner: claim.leaseOwner, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId, leaseId, remoteCwd: remoteSandbox, + runner: { execute: async () => { throw new Error("Unexpected remote preparation after the controlled backend owns execution"); } } as never }, + turnTimeoutMs: 15_000 })).toMatchObject({ exitCode: 0, timedOut: false }); + const recorder = workspaceOperationService(fixture.db).createRecorder({ companyId, + issueId: fixture.issueId, heartbeatRunId: fixture.runId }); + const saved = await recorder.recordOperation({ phase: "workspace_finalize", cwd: sandbox, + run: async () => { await workspaceSync!.restoreWorkspace(); return { status: "succeeded", exitCode: 0 }; } }); + expect(saved.status).toBe("succeeded"); + await reconcileNativeFinalizations(fixture.db, [fixture.runId]); + expect(await fixture.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, fixture.runId))) + .toMatchObject([{ schemaStatus: "accepted" }]); + const [finished] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId)); + expect(finished, JSON.stringify(finished.failureDetail)).toMatchObject({ phase: "committed", attempt: 2 }); + expect(await readFile(savedFile, "utf8")).toBe("Preserve admitted work.\n"); + expect((await readFile(calls, "utf8")).split("\n").filter(line => line === "process-start")).toHaveLength(1); + } finally { await bundle.transport.close(); } + } + } finally { child.kill("SIGKILL"); await exited; await workspaceSync?.cleanup(); } + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + it.each(["preparation", "claim lock"])("gives a fresh remote claim its full lease through blocked %s", async (blockedPhase) => { const fixture = await seedRun(`FRESH-SLOW-${blockedPhase === "claim lock" ? "CLAIM-LOCK" : "PREPARATION"}`); const environmentId = randomUUID(), leaseId = randomUUID(); diff --git a/server/src/services/native-runtime/native-safe-replacement.test.ts b/server/src/services/native-runtime/native-safe-replacement.test.ts index 491bae17b2b..088ae1bffc6 100644 --- a/server/src/services/native-runtime/native-safe-replacement.test.ts +++ b/server/src/services/native-runtime/native-safe-replacement.test.ts @@ -16,6 +16,11 @@ import { deliverReconciledExecutions, } from "../execution-recovery-resolution.js"; import { randomUUID } from "node:crypto"; +import { spawn } from "node:child_process"; +import { persistHeartbeatRunProcessMetadata } from "../heartbeat/run-lifecycle.js"; +import { hasNativeLocalProcessStop, releaseSettledNativeRunProcess } from "../native-local-process-stop.js"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; +import { executionControlSchedule } from "../execution-control-work.js"; import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; import { tmpdir } from "node:os"; import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; @@ -26,6 +31,7 @@ import { agents, companies, createDb, + subscribeDatabaseWork, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, @@ -41,7 +47,7 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, } from "../../__tests__/helpers/embedded-postgres.js"; -import { reconcileSafeNativeReplacements } from "./native-safe-replacement.js"; +import { invalidateNativeReplacementDiscovery, reconcileSafeNativeReplacements } from "./native-safe-replacement.js"; import { reconcileAbandonedExecutionControl } from "../execution-control-reconciliation.js"; const externalDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL; const support = externalDatabaseUrl @@ -219,6 +225,103 @@ const support = externalDatabaseUrl } }, ); + it("wakes a completed replacement scan after actual local stop proof commits", async () => { + const source = await seed(2); + const workspace = await mkdtemp(join(tmpdir(), "paperclip-late-stop-")); + const child = spawn(process.execPath, ["-e", "process.exit(0)"], { detached: true, stdio: "ignore" }); + const pid = child.pid!; + await new Promise((resolve, reject) => { child.once("exit", () => resolve()); child.once("error", reject); }); + await persistHeartbeatRunProcessMetadata(db, source.runId, { pid, processGroupId: pid, + startedAt: new Date().toISOString(), targetKind: "local" }); + await db.update(heartbeatRuns).set({ finishedAt: new Date(), runnerProfileJson: { + nativeExecutionInput: { provider: { kind: "codex" }, workspace: { cwd: workspace } }, + } }).where(eq(heartbeatRuns.id, source.runId)); + await db.update(nativeRunFinalizations).set({ failureCode: "provider_transport_failed" }) + .where(eq(nativeRunFinalizations.runId, source.runId)); + await appendHeartbeatRunEvent(db, { companyId: source.companyId, runId: source.runId, agentId: source.agentId, + eventType: "harness.ready", stream: "system", message: "Provider started before its transport failed." }); + const verifyStoppedSession = async () => await hasNativeLocalProcessStop(db, source.companyId, source.runId) + ? { evidence: {}, retire: () => true } : null; + invalidateNativeReplacementDiscovery(db); + expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 0 }); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + let wakes = 0; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.executionReplacement, event => { + if (event === "settled") { wakes++; invalidateNativeReplacementDiscovery(db); } + }); + try { + await db.transaction(async tx => { + expect(await releaseSettledNativeRunProcess(tx as unknown as typeof db, source.runId)).toBe("released"); + expect(wakes).toBe(0); + }); + expect(wakes).toBeGreaterThan(0); + expect(await executionControlSchedule(db).replacement()).not.toBeNull(); + expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 1 }); + expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 0 }); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId)); + expect(successors).toHaveLength(1); + } finally { unsubscribe(); await rm(workspace, { recursive: true, force: true }); } + }); + it("wakes completed recovery when the real task becomes eligible", async () => { + const source = await seed(); + const workspace = await mkdtemp(join(tmpdir(), "paperclip-late-eligibility-")); + await db.update(heartbeatRuns).set({ runnerProfileJson: { recoveryEventInventoryVersion: 1, + nativeExecutionInput: { provider: { kind: "codex" }, workspace: { cwd: workspace } }, + } }).where(eq(heartbeatRuns.id, source.runId)); + await issueService(db).update(source.issueId, { status: "backlog" }); + invalidateNativeReplacementDiscovery(db); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + let wakes = 0; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.executionReplacement, event => { + if (event === "settled") { wakes++; invalidateNativeReplacementDiscovery(db); } + }); + try { + await issueService(db).update(source.issueId, { status: "in_progress" }); + expect(wakes).toBeGreaterThan(0); + expect(await executionControlSchedule(db).replacement()).not.toBeNull(); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 1 }); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId))).toHaveLength(1); + } finally { unsubscribe(); await rm(workspace, { recursive: true, force: true }); } + }); + it("advances past a failed full page and retries it after later work progresses", async () => { + const workspace = await mkdtemp(join(tmpdir(), "paperclip-failed-page-")); + const sources: Awaited>[] = []; + for (let index = 0; index < 26; index++) { + const source = await seed(); sources.push(source); + await db.update(heartbeatRuns).set({ runnerProfileJson: { + nativeExecutionInput: { provider: { kind: "codex" }, workspace: { cwd: workspace } }, + } }).where(eq(heartbeatRuns.id, source.runId)); + await db.update(nativeRunFinalizations).set({ failureCode: "provider_transport_failed" }) + .where(eq(nativeRunFinalizations.runId, source.runId)); + } + sources.sort((a, b) => a.runId.localeCompare(b.runId)); + const later = sources.at(-1)!; + let failing = true; + const verifyStoppedSession = vi.fn(async (run: typeof heartbeatRuns.$inferSelect) => { + if (failing && run.id !== later.runId) throw new Error("Controlled stopped-session verifier failure"); + return { evidence: {}, retire: () => true }; + }); + invalidateNativeReplacementDiscovery(db); + try { + await expect(reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })) + .rejects.toThrow("native_replacement_decision_retry_required"); + expect(verifyStoppedSession).toHaveBeenCalledTimes(25); + expect(await executionControlSchedule(db).replacement()).not.toBeNull(); + expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 1 }); + expect(verifyStoppedSession).toHaveBeenCalledTimes(26); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, later.runId))).toHaveLength(1); + failing = false; + expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 25 }); + expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 0 }); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + const successors = await db.select().from(heartbeatRuns).where(inArray(heartbeatRuns.retryOfRunId, sources.map(source => source.runId))); + expect(successors).toHaveLength(26); + } finally { await rm(workspace, { recursive: true, force: true }); } + }, 60_000); it.each(["unproven", "changed", "verified"] as const)("requires stopped-session proof through commit (%s)", async (mode) => { const source = await seed(2); await db.update(nativeRunFinalizations).set({ failureCode: "native_session_cleanup_quarantined" }).where(eq(nativeRunFinalizations.runId, source.runId)); @@ -235,8 +338,11 @@ const support = externalDatabaseUrl eventType: "tool.execution.started", stream: "system", payload: { name: "paperclip_finish", executionId: "completion", transport: "process" }, }); - await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession }); - await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession }); + for (let sweep = 0; sweep < 2; sweep++) { + const decision = reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession }); + if (mode === "changed") await expect(decision).rejects.toThrow("native_replacement_decision_retry_required"); + else await decision; + } const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId)); expect(successors).toHaveLength(mode === "verified" ? 1 : 0); const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId)); diff --git a/server/src/services/native-runtime/native-safe-replacement.ts b/server/src/services/native-runtime/native-safe-replacement.ts index 89ce98400e5..b2ce3bef1bc 100644 --- a/server/src/services/native-runtime/native-safe-replacement.ts +++ b/server/src/services/native-runtime/native-safe-replacement.ts @@ -5,7 +5,7 @@ import { notifyExecutionRunWork } from "../execution-control-notifications.js"; import { logger } from "../../middleware/logger.js"; import { createHash, randomUUID } from "node:crypto"; import { stat } from "node:fs/promises"; -import { and, eq, inArray, isNull, sql } from "drizzle-orm"; +import { and, asc, eq, gt, inArray, isNull, or, sql } from "drizzle-orm"; import { agentWakeupRequests, environmentLeases, @@ -27,6 +27,8 @@ import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { buildExecutionContinuation } from "../execution-continuation.js"; import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js"; +import { NATIVE_PROVIDER_DURABLE_EVENT_TYPES } from "./native-retired-backup-source.js"; +import { completedWorkspaceRecoveryCondition } from "../workspace-recovery-receipt.js"; export const NATIVE_SAFE_REPLACEMENT_REASON = "native_safe_replacement"; const record = (v: unknown): Record => @@ -45,6 +47,66 @@ const processAlive = (pid: number | null) => { type ReplacementDb = Db | Parameters[0]>[0]; +const listenerUnavailable = "local_runtime_api_listener_unavailable"; +function historicalListenerGuardPending() { + const profile = heartbeatRuns.runnerProfileJson, detail = nativeRunFinalizations.failureDetail; + const providerEvents = sql.join([...NATIVE_PROVIDER_DURABLE_EVENT_TYPES].map(value => sql`${value}`), sql`, `); + return and( + eq(nativeRunFinalizations.failureCode, "native_session_interrupted"), + eq(heartbeatRuns.errorCode, "native_session_interrupted"), eq(heartbeatRuns.error, listenerUnavailable), + sql`${detail}->>'message' = ${listenerUnavailable}`, + sql`${detail}->>'originalFailureCode' = 'native_session_interrupted'`, + sql`${detail}->>'recoveryMode' = 'ambiguous_state'`, + sql`${detail}->'providerSessionEstablished' = 'false'::jsonb`, + sql`${detail}->'providerEventsExist' = 'false'::jsonb`, + sql`${detail}->'checkpointExists' = 'false'::jsonb`, + eq(nativeRunFinalizations.attempt, 1), + isNull(nativeRunFinalizations.leaseOwner), isNull(nativeRunFinalizations.leaseExpiresAt), + isNull(nativeRunFinalizations.assessmentId), isNull(nativeRunFinalizations.decisionId), + isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId), + isNull(heartbeatRuns.sessionIdBefore), isNull(heartbeatRuns.sessionIdAfter), isNull(heartbeatRuns.retryOfRunId), + sql`not (coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) ?| array['cancellation', 'nativeCancellation'])`, + sql`coalesce(${profile}->'sessionCheckpoint', 'null'::jsonb) = 'null'::jsonb`, + sql`coalesce(${profile}->'nativeProviderPackAdmission', 'null'::jsonb) = 'null'::jsonb`, + sql`coalesce(${profile}#>>'{nativeProviderPackAdmissionState,state}', 'pending') = 'pending'`, + sql`${profile}#>>'{nativeExecutionInput,binding,runId}' = ${heartbeatRuns.id}::text`, + sql`${profile}#>>'{nativeExecutionInput,binding,companyId}' = ${heartbeatRuns.companyId}::text`, + sql`${profile}#>>'{nativeExecutionInput,binding,agentId}' = ${heartbeatRuns.agentId}::text`, + sql`${profile}#>>'{nativeExecutionInput,binding,issueId}' = ${nativeRunFinalizations.issueId}::text`, + sql`${profile}#>>'{nativeExecutionInput,session,normalizedSessionId}' = ${heartbeatRuns.nativeSessionId}::text`, + sql`not exists (select 1 from ${heartbeatRuns} prior where prior.company_id = ${heartbeatRuns.companyId} + and prior.native_session_id = ${heartbeatRuns.nativeSessionId} and prior.id <> ${heartbeatRuns.id})`, + sql`not exists (select 1 from ${heartbeatRunEvents} event where event.company_id = ${heartbeatRuns.companyId} + and event.run_id = ${heartbeatRuns.id} and (event.source_event_id is not null + or event.event_type in (${providerEvents}) or event.event_type ~ '^(turn|item|tool|semantic_tool|run[.]result)[.]'))`, + sql`not exists (select 1 from ${toolInvocations} invocation where invocation.company_id = ${heartbeatRuns.companyId} + and invocation.run_id = ${heartbeatRuns.id})`, + sql`not exists (select 1 from ${issueRecoveryActions} + where ${issueRecoveryActions.companyId} = ${heartbeatRuns.companyId} + and ${issueRecoveryActions.sourceIssueId} = ${nativeRunFinalizations.issueId} + and ${issueRecoveryActions.cause} = 'native_continuation_requires_reconciliation' + and ${issueRecoveryActions.evidence}->>'runId' = ${heartbeatRuns.id}::text + and ${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked' + and not coalesce((${completedWorkspaceRecoveryCondition(heartbeatRuns.id)}), false))`, + sql`exists (select 1 from ${issues} task where task.company_id = ${heartbeatRuns.companyId} + and task.id = ${nativeRunFinalizations.issueId} and task.assignee_agent_id = ${heartbeatRuns.agentId} + and task.status in ('todo', 'in_progress', 'in_review', 'blocked') + and (task.status <> 'blocked' or exists (select 1 from ${issueRecoveryActions} hold + where hold.company_id = task.company_id and hold.source_issue_id = task.id and hold.kind = 'active_run_watchdog' + and hold.cause = ${nativeRunFinalizations.failureCode} and hold.evidence->>'runId' = ${heartbeatRuns.id}::text + and hold.evidence#>>'{nativeFailureBlock,runId}' = ${heartbeatRuns.id}::text + and hold.evidence#>'{nativeFailureBlock,statusVersion}' = to_jsonb(task.status_version))))`, + ); +} + +async function historicalListenerGuardEvidence(db: ReplacementDb, run: typeof heartbeatRuns.$inferSelect) { + const [evidence] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns) + .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)).where(and( + eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.id, run.id), historicalListenerGuardPending(), + )).limit(1); + return Boolean(evidence); +} + /** A saved remote source is proved by its current ledger, never a controller path. */ async function remoteWorkspaceReplacementEvidence( db: ReplacementDb, @@ -113,16 +175,95 @@ export function nativeReplacementPending() { return and( eq(heartbeatRuns.status, "failed"), eq(nativeRunFinalizations.phase, "terminal_failure"), - inArray( + or(inArray( nativeRunFinalizations.failureCode, ["native_provider_terminal_failed", "native_session_cleanup_quarantined", "provider_transport_failed"], - ), + ), historicalListenerGuardPending()), isNull(nativeRunFinalizations.resultId), sql`coalesce(${nativeRunFinalizations.failureDetail}->>'successorRunId', '') = ''`, sql`coalesce(${nativeRunFinalizations.failureDetail}->>'replacementDenied', '') = ''`, ); } +const REPLACEMENT_SCAN_LIMIT = 100; +const replacementDiscovery = new WeakMap(); +function replacementDiscoveryState(db: Db) { + let state = replacementDiscovery.get(db); + if (!state) { state = { after: undefined, complete: false, generation: 0, rescan: false }; replacementDiscovery.set(db, state); } + return state; +} + +export function invalidateNativeReplacementDiscovery(db: Db) { + const state = replacementDiscoveryState(db); + state.complete = false; state.rescan = true; state.generation++; +} + +async function replacementHistoryReady(db: Db, run: typeof heartbeatRuns.$inferSelect, issueId: string, retryDatabaseErrors = false) { + try { + const continuation = await buildExecutionContinuation({ db, companyId: run.companyId, issueId, + agentId: run.agentId, context: record(run.contextSnapshot), summary: null, exposeLowTrustRaw: false }); + return continuation.unresolvedInteractionIds.length === 0; + } catch (error) { + let cause: unknown = error; + for (let depth = 0; retryDatabaseErrors && depth < 4 && cause && typeof cause === "object"; depth++) { + const failure = cause as { code?: unknown; cause?: unknown }; + if (typeof failure.code === "string" && /^[A-Z0-9_]+$/.test(failure.code)) throw error; + cause = failure.cause; + } + return null; + } +} + +async function discoverNativeReplacements(db: Db, after: string | undefined, limit = 25) { + const candidates: Array<{ run: typeof heartbeatRuns.$inferSelect; coordinator: typeof nativeRunFinalizations.$inferSelect }> = []; + let scanned = 0; + for (;;) { + const rows = await db.select({ run: heartbeatRuns, coordinator: nativeRunFinalizations }).from(heartbeatRuns) + .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)) + .where(and(nativeReplacementPending(), after ? gt(heartbeatRuns.id, after) : undefined)) + .orderBy(asc(heartbeatRuns.id)).limit(Math.min(25, REPLACEMENT_SCAN_LIMIT - scanned)); + if (!rows.length) return { candidates, scanned, after: undefined }; + for (const row of rows) { + after = row.run.id; scanned++; + let ready = true; + if (row.coordinator.failureCode === "native_session_interrupted") { + ready = false; + if (row.run.runnerProfileJson?.remoteWorkspace != null) { + const leases = await db.select().from(environmentLeases).where(and( + eq(environmentLeases.companyId, row.run.companyId), eq(environmentLeases.heartbeatRunId, row.run.id), + )); + if (leases.every(lease => lease.releasedAt !== null)) { + const evidence = await remoteWorkspaceReplacementEvidence(db, row.run, row.coordinator.issueId, leases); + ready = Boolean(evidence?.workspacePreserved && evidence.providerStopped) + && await replacementHistoryReady(db, row.run, row.coordinator.issueId, true) === true; + } + } + } + if (ready) candidates.push(row); + if (candidates.length >= limit || scanned >= REPLACEMENT_SCAN_LIMIT) return { candidates, scanned, after }; + } + if (rows.length < 25) return { candidates, scanned, after: undefined }; + } +} + +function settleReplacementDiscovery(state: ReturnType, after: string | undefined, generation: number) { + const rescan = state.rescan || generation !== state.generation; + state.after = after; state.complete = !after && !rescan; + state.rescan = Boolean(after && rescan); +} + +export async function hasPendingNativeReplacements(db: Db) { + const state = replacementDiscoveryState(db); + if (state.complete) return false; + if (state.after) return true; + state.rescan = false; + const generation = state.generation; + const result = await discoverNativeReplacements(db, undefined, 1); + if (result.candidates.length) return true; + settleReplacementDiscovery(state, result.after, generation); + return !state.complete; +} + /** Retryable outbox: terminal_failure plus a new-format failure code is the durable candidate. */ export async function reconcileSafeNativeReplacements( db: Db, @@ -136,18 +277,14 @@ export async function reconcileSafeNativeReplacements( failpoint?: (phase: "successor_inserted" | "lineage_committed") => void; } = {}, ) { - const candidates = await db - .select({ run: heartbeatRuns, coordinator: nativeRunFinalizations }) - .from(heartbeatRuns) - .innerJoin( - nativeRunFinalizations, - eq(nativeRunFinalizations.runId, heartbeatRuns.id), - ) - .where( - nativeReplacementPending(), - ) - .limit(25); - let scheduled = 0; + const discovery = replacementDiscoveryState(db); + if (discovery.complete) discovery.after = undefined; + discovery.complete = false; + if (!discovery.after) discovery.rescan = false; + const generation = discovery.generation; + const selection = await discoverNativeReplacements(db, discovery.after); + const candidates = selection.candidates; + let scheduled = 0, pendingFailures = 0; for (const { run, coordinator } of candidates) { try { if ( @@ -170,11 +307,13 @@ export async function reconcileSafeNativeReplacements( ); // Teardown is still in progress. The next sweep rechecks its durable outcome. if (leases.some((lease) => lease.releasedAt === null)) continue; - const stoppedSession = coordinator.failureCode !== "native_provider_terminal_failed" + const historicalListener = coordinator.failureCode === "native_session_interrupted"; + if (historicalListener && !await historicalListenerGuardEvidence(db, run)) continue; + const stoppedSession = coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener ? await options.verifyStoppedSession?.(run) ?? null : null; // A transport label alone is not evidence. Keep inspecting these candidates // as process cleanup and the final transcript become durable. - if (coordinator.failureCode !== "native_provider_terminal_failed" && !stoppedSession) continue; + if (coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener && !stoppedSession) continue; const invocations = await db .select() .from(toolInvocations) @@ -263,23 +402,11 @@ export async function reconcileSafeNativeReplacements( ? [String(p.kind ?? p.type)] : []; }); - let historyComplete = false; - try { - const continuation = await buildExecutionContinuation({ - db, - companyId: run.companyId, - issueId: coordinator.issueId, - agentId: run.agentId, - context: record(run.contextSnapshot), - summary: null, - exposeLowTrustRaw: false, - }); - if (continuation.unresolvedInteractionIds.length > 0) continue; - historyComplete = true; - } catch { - /* The named operator outcome retains incomplete context. */ - } + const history = await replacementHistoryReady(db, run, coordinator.issueId); + if (history === false) continue; + const historyComplete = history === true; const remoteWorkspace = await remoteWorkspaceReplacementEvidence(db, run, coordinator.issueId, leases); + if (historicalListener && (!remoteWorkspace?.workspacePreserved || !remoteWorkspace.providerStopped || invocations.length)) continue; const decision = decideNativeReplacement({ failedSession: true, // A facade transport failure can conceal an authoritative protocol @@ -413,6 +540,13 @@ export async function reconcileSafeNativeReplacements( current.attempt >= 3 ) return false; + let historicalRun: typeof heartbeatRuns.$inferSelect | undefined; + if (historicalListener) { + const [currentRun] = await tx.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), + )).for("update"); + historicalRun = currentRun; + } if (task.status === "blocked") { const failureHolds = await tx.select().from(issueRecoveryActions).where(and( eq(issueRecoveryActions.companyId, run.companyId), @@ -427,6 +561,14 @@ export async function reconcileSafeNativeReplacements( }); if (!ownsBlock) return false; } + const preservationHolds = historicalListener ? await tx.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, task.id), + eq(issueRecoveryActions.kind, "active_run_watchdog"), + eq(issueRecoveryActions.cause, "native_continuation_requires_reconciliation"), + sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, + sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`, + )).for("update") : []; + if (historicalListener && (!historicalRun || !await historicalListenerGuardEvidence(tx, historicalRun))) return false; if (remoteWorkspace) { const currentWorkspace = await remoteWorkspaceReplacementEvidence(tx, run, coordinator.issueId, leases, true); if (!currentWorkspace?.workspacePreserved || !currentWorkspace.providerStopped) return false; @@ -548,6 +690,7 @@ export async function reconcileSafeNativeReplacements( eq(issueRecoveryActions.cause, current.failureCode!), sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, )).for("update"); + holds.push(...preservationHolds); if (holds.some(hold => hold.status === "active" || hold.status === "escalated")) { await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: run.companyId, issueId: task.id }); } @@ -587,11 +730,17 @@ export async function reconcileSafeNativeReplacements( } } catch (error) { if (options.failpoint) throw error; + pendingFailures++; logger.warn( { runId: run.id }, "Native replacement remains pending; another sweep will retry its durable decision", ); } } - return { scanned: candidates.length, scheduled }; + settleReplacementDiscovery(discovery, selection.after, generation); + if (pendingFailures) { + discovery.complete = false; discovery.rescan = true; + throw new Error("native_replacement_decision_retry_required"); + } + return { scanned: selection.scanned, scheduled }; } diff --git a/server/src/services/remote-workspaces.integration.test.ts b/server/src/services/remote-workspaces.integration.test.ts index 912416f1457..9e45c05470b 100644 --- a/server/src/services/remote-workspaces.integration.test.ts +++ b/server/src/services/remote-workspaces.integration.test.ts @@ -13,7 +13,7 @@ import { pathToFileURL } from "node:url"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { and, eq, sql } from "drizzle-orm"; import { activityLog, agents, agentTaskSessions, agentWakeupRequests, applyPendingMigrations, closeRegisteredClients, companies, createDb, environmentLeases, environments, - executionWorkspaceRepositories, executionWorkspaces, executionWorkspaceSnapshots, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues, livenessDb, nativeRunFinalizations, projects, projectWorkspaces, workspaceOperations } from "@paperclipai/db"; + subscribeDatabaseWork, executionWorkspaceRepositories, executionWorkspaces, executionWorkspaceSnapshots, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issueThreadInteractions, issues, livenessDb, nativeRunFinalizations, projects, projectWorkspaces, workspaceOperations } from "@paperclipai/db"; import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime"; import { buildProjectMentionHref, type Environment, type EnvironmentLease } from "@paperclipai/shared"; @@ -25,12 +25,13 @@ import { createSavedWorkspaceLoader, prepareRemoteWorkspace, readRemoteWorkspace clearRemoteWorkspaceLocationForReleasedLease, createRemoteWorkspaceGitAuth, prepareRemoteWorkspaceRepository, reserveRemoteWorkspace, saveRemoteWorkspace, shouldUseRemoteWorkspaces } from "./remote-workspaces.js"; import type { createGitRemoteAuthProvider } from "./git-credentials.js"; -import { reconcileSafeNativeReplacements } from "./native-runtime/native-safe-replacement.js"; +import { invalidateNativeReplacementDiscovery, reconcileSafeNativeReplacements } from "./native-runtime/native-safe-replacement.js"; import { remoteTerminationReceipt } from "./remote-execution-termination.js"; import { RemoteWorkspaceRestoreError } from "./remote-workspace-transport.js"; import { readRemoteRunnerState } from "./native-runtime/native-session-executor.js"; import { notFound } from "../errors.js"; import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js"; +import { issueThreadInteractionService } from "./issue-thread-interactions.js"; import { createRemoteWorkspaceFiles } from "./remote-workspace-files.js"; import { workspaceFileResourceService } from "./workspace-file-resources.js"; import { nativeWorkspaceExportPendingForLease, preserveRemoteWorkspacePreparationFailure } from "./native-runtime/native-workspace-finalization-state.js"; @@ -47,6 +48,10 @@ import { instanceSettingsService } from "./instance-settings.js"; import { resolveRemoteWorkspaceSource } from "./heartbeat/remote-workspace-source.js"; import { toExecutionWorkspace } from "./execution-workspaces.js"; import { recoveryPostgresSupport, startRecoveryPostgres } from "../__tests__/helpers/recovery-postgres.js"; +import { getExecutionBlocker } from "./execution-blocker.js"; +import { executionControlSchedule } from "./execution-control-work.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { createSystemRecovery } from "./system-recovery-state.js"; const exec = promisify(execFile); let databaseUrl = process.env.PAPERCLIP_REMOTE_WORKSPACE_TEST_DATABASE_URL; @@ -82,6 +87,7 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem await db.insert(issues).values({ id: ids.issue, companyId: ids.company, title: "Use a remote workspace", status: "in_progress", assigneeAgentId: ids.agent }); await db.insert(heartbeatRuns).values({ id: ids.run, companyId: ids.company, agentId: ids.agent, nativeIssueId: ids.issue, runtimeMode: "native", status: "running" }); + invalidateNativeReplacementDiscovery(db); await db.insert(environmentLeases).values({ id: ids.lease, companyId: ids.company, environmentId: ids.environment, heartbeatRunId: ids.run, issueId: ids.issue, provider: "daytona", providerLeaseId: `allocation-${ids.lease}`, status: "active" }); }); @@ -91,6 +97,10 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem ? (row.metadata.uploads as { objectKey: string }[]).map(item => item.objectKey) : [])].filter((value): value is string => !!value))) { await storage.deleteObject(ids.company, key); } + if (temporary && process.env.PAPERCLIP_TEST_HISTORICAL_LISTENER === "1") await db.transaction(async tx => { + await tx.execute(sql`select set_config('client_min_messages', 'warning', true)`); + await tx.execute(sql`TRUNCATE companies CASCADE`); + }); await db.delete(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.companyId, ids.company)); await db.delete(workspaceOperations).where(eq(workspaceOperations.companyId, ids.company)); await db.delete(activityLog).where(eq(activityLog.companyId, ids.company)); @@ -1712,14 +1722,20 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem } finally { await recovered.cleanup(); } }, 60_000); - it.each(["saved", "dirty", "unsaved", "foreign_snapshot", "delete_requested", "deletion_race", "controller_claim_race", "stale_epoch", "unconfirmed_stop", "unknown_action"] as const)( - "checks durable remote work at the native safe-replacement decision (%s)", async scenario => { + it.each(["saved", "dirty", "unsaved", "foreign_snapshot", "delete_requested", "deletion_race", "controller_claim_race", "stale_epoch", "unconfirmed_stop", "unknown_action", "historical_listener_guard", "historical_listener_provider_event", "historical_listener_checkpoint", "historical_listener_resumed", "historical_listener_other_message", "historical_listener_cancelled", "historical_listener_prior_session", "historical_listener_unconfirmed_stop", "historical_listener_dirty", "historical_listener_stale_block", "historical_listener_unknown_event", "historical_listener_budget", "historical_listener_bad_preservation", "historical_listener_absent", "historical_listener_later_candidate", "historical_listener_late_proof", "historical_listener_lock_timeout", "historical_listener_late_interaction", "historical_listener_interaction_paused", "historical_listener_own_question"] as const)( + "checks durable remote work at the native safe-replacement decision (%s)", async kind => { + const historicalListener = kind.startsWith("historical_listener_"); + const processAcceptance = kind === "historical_listener_guard" && process.env.PAPERCLIP_TEST_HISTORICAL_LISTENER === "1"; + const scenario = kind === "historical_listener_late_proof" ? "dirty" : kind === "historical_listener_unconfirmed_stop" ? "unconfirmed_stop" + : kind === "historical_listener_dirty" ? "dirty" : historicalListener ? "saved" : kind; + const shouldSchedule = scenario === "saved" && (!historicalListener || ["historical_listener_guard", "historical_listener_later_candidate", "historical_listener_lock_timeout"].includes(kind)); const reserved = await reserve({ kind: "empty" }), first = await target("native-replacement-sandbox"); const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: first, assertOwned: async () => undefined }); await fs.writeFile(path.join(prepared.remoteRoot, "saved.txt"), "work survives a failed provider session\n"); if (scenario !== "unsaved") await saveRemoteWorkspace({ db, storage, workspace: prepared.workspace, - runId: ids.run, lease: prepared.lease, target: first, assertOwned: async () => undefined }); + runId: ids.run, lease: prepared.lease, target: first, assertOwned: async () => undefined, + boundary: kind === "historical_listener_late_proof" ? "initial" : "final" }); const current = await workspace(reserved.id), authority = readRemoteWorkspaceAuthority(current.metadata)!; if (scenario === "dirty" || scenario === "stale_epoch") await db.update(executionWorkspaces).set({ metadata: { ...current.metadata, remoteWorkspace: { ...authority, ...(scenario === "dirty" ? { dirty: true, writerRunId: ids.run } @@ -1729,29 +1745,137 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem .where(eq(executionWorkspaceSnapshots.id, authority.snapshotId!)); if (scenario === "delete_requested") await db.update(executionWorkspaceSnapshots).set({ status: "delete_requested" }) .where(eq(executionWorkspaceSnapshots.id, authority.snapshotId!)); - const stopped = (await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success", + if (processAcceptance) { + const { recordHistoricalListenerFailure } = await import("../__tests__/helpers/historical-listener-continuation.js"); + await recordHistoricalListenerFailure({ db, connectionString: databaseUrl!, runId: ids.run, cwd: prepared.remoteRoot, + environmentId: ids.environment, leaseId: ids.lease }); + } + const stopped = kind === "historical_listener_late_proof" ? prepared.lease : (await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success", metadata: { ...prepared.lease.metadata, ...(scenario !== "unconfirmed_stop" ? { remoteExecutionTermination: remoteTerminationReceipt(prepared.lease, { providerLeaseId: prepared.lease.providerLeaseId, state: "destroyed", }), } : {}) }, }).where(eq(environmentLeases.id, ids.lease)).returning())[0]! as EnvironmentLease; - if (scenario === "saved") { + if (shouldSchedule) { await db.transaction(tx => clearRemoteWorkspaceLocationForReleasedLease({ db: tx, lease: stopped })); await fs.rm(prepared.remoteRoot, { recursive: true, force: true }); await expect(fs.stat(prepared.remoteRoot)).rejects.toThrow(); } const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, ids.run)); - await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), runnerProfileJson: { + const sessionId = randomUUID(); + await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), + ...(historicalListener && !processAcceptance ? { nativeSessionId: sessionId } : {}), + ...(historicalListener ? { error: "local_runtime_api_listener_unavailable", errorCode: "native_session_interrupted" } : {}), runnerProfileJson: { ...source!.runnerProfileJson, recoveryEventInventoryVersion: 1, - nativeExecutionInput: { provider: { kind: "codex" }, workspace: { cwd: prepared.remoteRoot } }, + nativeExecutionInput: processAcceptance ? source!.runnerProfileJson!.nativeExecutionInput : { provider: { kind: "codex" }, workspace: { cwd: prepared.remoteRoot }, + ...(historicalListener ? { binding: { runId: ids.run, companyId: ids.company, agentId: ids.agent, issueId: ids.issue }, + session: { normalizedSessionId: sessionId } } : {}) }, } }).where(eq(heartbeatRuns.id, ids.run)); - await db.insert(nativeRunFinalizations).values({ runId: ids.run, companyId: ids.company, issueId: ids.issue, - phase: "terminal_failure", attempt: 1, failureCode: "native_provider_terminal_failed", - failureDetail: { originalFailureCode: "fixture_checkpoint_unusable" } }); + if (!processAcceptance) await db.insert(nativeRunFinalizations).values({ runId: ids.run, companyId: ids.company, issueId: ids.issue, + phase: "terminal_failure", attempt: 1, failureCode: historicalListener ? "native_session_interrupted" : "native_provider_terminal_failed", + failureDetail: historicalListener ? { originalFailureCode: "native_session_interrupted", message: "local_runtime_api_listener_unavailable", + recoveryMode: "ambiguous_state", providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false } + : { originalFailureCode: "fixture_checkpoint_unusable" } }); + if (historicalListener && !processAcceptance) { + const blocked = await issueService(db).update(ids.issue, { status: "blocked" }); + await db.insert(issueRecoveryActions).values({ companyId: ids.company, sourceIssueId: ids.issue, + kind: "active_run_watchdog", cause: "native_session_interrupted", ownerType: "board", status: "active", + fingerprint: createHash("sha256").update(`${ids.run}:native_session_interrupted`).digest("hex"), + evidence: { runId: ids.run, nativeFailureBlock: { runId: ids.run, statusVersion: blocked!.statusVersion } }, + nextAction: "Reconcile the original startup failure." }); + } + if (kind === "historical_listener_guard" || kind === "historical_listener_bad_preservation") { + const actionId = randomUUID(), preservationId = randomUUID(); + await db.insert(issueRecoveryActions).values({ id: actionId, companyId: ids.company, sourceIssueId: ids.issue, + kind: "active_run_watchdog", cause: "native_continuation_requires_reconciliation", ownerType: "system", status: "resolved", outcome: "blocked", + fingerprint: `native-workspace:${ids.run}`, evidence: { runId: ids.run, + workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [ids.lease] }, + automaticRecovery: { replay: "blocked" }, continuationDelivery: "invalidated", + systemRecovery: { ...createSystemRecovery(ids.run), state: "complete", preservationOperationIds: [preservationId] }, + workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: ids.run, status: "recovered", stage: "saving_receipts", attempt: 1, requestIds: [] }, + }, nextAction: "Files were preserved without replay." }); + await db.insert(workspaceOperations).values([{ id: preservationId, companyId: ids.company, heartbeatRunId: ids.run, issueId: ids.issue, + executionWorkspaceId: reserved.id, phase: "workspace_recovery", status: kind === "historical_listener_bad_preservation" ? "failed" : "succeeded", startedAt: new Date(), + metadata: { recoveryActionId: actionId, snapshotId: authority.snapshotId } }, + { companyId: ids.company, heartbeatRunId: ids.run, issueId: ids.issue, phase: "workspace_finalize", status: "succeeded", startedAt: new Date(), + metadata: { owningService: "workspace_recovery", preservationOperationIds: [preservationId] } }]); + } + if (kind === "historical_listener_absent") await db.update(heartbeatRuns).set({ + runnerProfileJson: sql`${heartbeatRuns.runnerProfileJson} - 'remoteWorkspace'`, + }).where(eq(heartbeatRuns.id, ids.run)); + if (kind === "historical_listener_stale_block") await issueService(db).update(ids.issue, { status: "blocked", actorUserId: "operator" }); + if (kind === "historical_listener_unknown_event") await db.insert(heartbeatRunEvents).values({ companyId: ids.company, runId: ids.run, + agentId: ids.agent, seq: 100, eventType: "future.provider.activity", sourceEventId: "source-provider-event" }); + if (kind === "historical_listener_budget") await db.update(nativeRunFinalizations).set({ attempt: 3 }).where(eq(nativeRunFinalizations.runId, ids.run)); + if (kind === "historical_listener_checkpoint") await db.update(heartbeatRuns).set({ runnerProfileJson: { + ...((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, ids.run)))[0]!.runnerProfileJson), sessionCheckpoint: { providerSessionId: "already-started" }, + } }).where(eq(heartbeatRuns.id, ids.run)); + if (kind === "historical_listener_resumed") await db.update(heartbeatRuns).set({ sessionIdBefore: "prior-session" }).where(eq(heartbeatRuns.id, ids.run)); + if (kind === "historical_listener_prior_session") await db.insert(heartbeatRuns).values({ id: randomUUID(), companyId: ids.company, + agentId: ids.agent, status: "succeeded", nativeSessionId: sessionId }); + if (kind === "historical_listener_other_message") await db.update(nativeRunFinalizations).set({ failureDetail: { + message: "provider reported local_runtime_api_listener_unavailable", originalFailureCode: "native_session_interrupted", recoveryMode: "ambiguous_state", + providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false, + } }).where(eq(nativeRunFinalizations.runId, ids.run)); + if (kind === "historical_listener_cancelled") await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, ids.issue)); + if (kind === "historical_listener_provider_event") await appendHeartbeatRunEvent(db, { companyId: ids.company, runId: ids.run, + agentId: ids.agent, eventType: "turn.started", stream: "system" }); + const interactionWake = ["historical_listener_late_interaction", "historical_listener_interaction_paused", "historical_listener_own_question"].includes(kind); + const interactionId = randomUUID(); + if (interactionWake) { + const priorRunId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: priorRunId, companyId: ids.company, agentId: ids.agent, + nativeIssueId: ids.issue, status: "succeeded", contextSnapshot: { issueId: ids.issue } }); + await db.insert(issueThreadInteractions).values({ id: interactionId, companyId: ids.company, issueId: ids.issue, + kind: "request_confirmation", status: "pending", sourceRunId: kind === "historical_listener_own_question" ? ids.run : priorRunId, + payload: { version: 1, prompt: "An obsolete question from earlier work." } }); + if (kind === "historical_listener_interaction_paused") await issueService(db).update(ids.issue, { status: "backlog" }); + if (kind === "historical_listener_own_question") await appendHeartbeatRunEvent(db, { companyId: ids.company, runId: ids.run, + agentId: ids.agent, eventType: "tool.execution.started", stream: "system", + payload: { name: "request_confirmation", executionId: "source-question", transport: "process" } }); + } if (scenario === "unknown_action") await appendHeartbeatRunEvent(db, { companyId: ids.company, runId: ids.run, agentId: ids.agent, eventType: "tool.execution.started", stream: "system", payload: { name: "send_email", executionId: "unconfirmed-external-action", transport: "process" } }); + if (kind === "historical_listener_later_candidate") { + const sourceNumber = BigInt(`0x${ids.run.replaceAll("-", "")}`); + const parked = Array.from({ length: 100 }, (_, index) => { + const hex = (sourceNumber - BigInt(index + 1)).toString(16).padStart(32, "0"); + return { runId: `${hex.slice(0,8)}-${hex.slice(8,12)}-${hex.slice(12,16)}-${hex.slice(16,20)}-${hex.slice(20)}`, + issueId: randomUUID(), sessionId: randomUUID() }; + }); + await db.insert(issues).values(parked.map(row => ({ id: row.issueId, companyId: ids.company, + title: "Preserve unavailable source", status: "in_progress", assigneeAgentId: ids.agent }))); + await db.insert(heartbeatRuns).values(parked.map(row => ({ id: row.runId, companyId: ids.company, agentId: ids.agent, + nativeIssueId: row.issueId, nativeSessionId: row.sessionId, runtimeMode: "native", status: "failed", + errorCode: "native_session_interrupted", error: "local_runtime_api_listener_unavailable", runnerProfileJson: { + nativeExecutionInput: { binding: { runId: row.runId, companyId: ids.company, agentId: ids.agent, issueId: row.issueId }, + session: { normalizedSessionId: row.sessionId } }, + } }))); + await db.insert(nativeRunFinalizations).values(parked.map(row => ({ runId: row.runId, companyId: ids.company, issueId: row.issueId, + phase: "terminal_failure", attempt: 1, failureCode: "native_session_interrupted", failureDetail: { + message: "local_runtime_api_listener_unavailable", originalFailureCode: "native_session_interrupted", recoveryMode: "ambiguous_state", + providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false, + } }))); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scanned: 100, scheduled: 0 }); + expect(await executionControlSchedule(db).replacement()).not.toBeNull(); + invalidateNativeReplacementDiscovery(db); + } + if (kind === "historical_listener_lock_timeout") { + let release!: () => void, announce!: () => void; + const locked = new Promise(resolve => { announce = resolve; }); + const released = new Promise(resolve => { release = resolve; }); + const holder = db.transaction(async tx => { + await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, ids.issue)).for("update"); + announce(); await released; + }); + await locked; + try { + await expect(reconcileSafeNativeReplacements(db)).rejects.toThrow(); + expect(await executionControlSchedule(db).replacement()).not.toBeNull(); + } finally { release(); await holder; } + } let replacement; if (scenario === "deletion_race" || scenario === "controller_claim_race") { let release!: () => void, announce!: () => void; @@ -1783,12 +1907,102 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem expect(blocked).toBe(true); } finally { release(); await deletion; replacement = await pending; } } else replacement = await reconcileSafeNativeReplacements(db); - expect(replacement).toMatchObject({ scheduled: scenario === "saved" ? 1 : 0 }); + expect(replacement).toMatchObject({ scheduled: shouldSchedule ? 1 : 0 }); const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, ids.run)); - expect(successors).toHaveLength(scenario === "saved" ? 1 : 0); - if (scenario === "saved") expect(successors[0]).toMatchObject({ status: "scheduled_retry", + expect(successors).toHaveLength(shouldSchedule ? 1 : 0); + if (shouldSchedule) expect(successors[0]).toMatchObject({ status: "scheduled_retry", contextSnapshot: { retryOfRunId: ids.run, forceFreshSession: true } }); - else expect(await fs.readFile(path.join(prepared.remoteRoot, "saved.txt"), "utf8")) + if (kind === "historical_listener_later_candidate") { + for (let pass = 0; pass < 3 && await executionControlSchedule(db).replacement() !== null; pass++) { + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + } + expect(await executionControlSchedule(db).replacement()).toBeNull(); + } + if (kind === "historical_listener_guard") expect(await getExecutionBlocker(db, ids.company, ids.issue)).toBeNull(); + if (["historical_listener_provider_event", "historical_listener_checkpoint", "historical_listener_resumed", "historical_listener_other_message", + "historical_listener_cancelled", "historical_listener_prior_session", "historical_listener_unknown_event", "historical_listener_budget", "historical_listener_stale_block", "historical_listener_bad_preservation", "historical_listener_dirty", "historical_listener_unconfirmed_stop", "historical_listener_absent", "historical_listener_late_proof"].includes(kind)) { + expect(await executionControlSchedule(db).replacement()).toBeNull(); + } + if (interactionWake) { + expect(await executionControlSchedule(db).replacement()).toBeNull(); + let wakes = 0; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.executionReplacement, event => { + if (event === "settled") { wakes++; invalidateNativeReplacementDiscovery(db); } + }); + try { + await issueThreadInteractionService(db).skipInteraction({ id: ids.issue, companyId: ids.company, + status: kind === "historical_listener_interaction_paused" ? "backlog" : "blocked" }, interactionId, {}, { userId: "local-board" }); + expect(wakes).toBeGreaterThan(0); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: kind === "historical_listener_late_interaction" ? 1 : 0 }); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, ids.run))) + .toHaveLength(kind === "historical_listener_late_interaction" ? 1 : 0); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + } finally { unsubscribe(); } + } + if (kind === "historical_listener_late_proof") { + let wakes = 0; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.executionReplacement, event => { + if (event === "settled") { wakes++; invalidateNativeReplacementDiscovery(db); } + }); + try { + const saved = await saveRemoteWorkspace({ db, storage, workspace: await workspace(reserved.id), runId: ids.run, + lease: await lease(), target: first, assertOwned: async () => undefined }); + expect(wakes).toBeGreaterThan(0); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + const priorWakes = wakes, active = await lease(), pluginId = randomUUID(); + await db.update(environmentLeases).set({ metadata: { ...active.metadata, driver: "sandbox", pluginId, + provider: "daytona", sandboxProviderPlugin: true } }).where(eq(environmentLeases.id, ids.lease)); + const runtime = environmentRuntimeService(db, { pluginWorkerManager: { + isRunning: () => true, getWorker: () => ({ supportedMethods: ["environmentReleaseLease"] }), + call: async (_id: string, method: string) => { + expect(method).toBe("environmentReleaseLease"); + return { providerLeaseId: active.providerLeaseId, state: "destroyed" }; + }, + } as never }); + await runtime.releaseRunLeases(ids.run, "failed"); + expect(wakes).toBeGreaterThan(priorWakes); + expect(await executionControlSchedule(db).replacement()).not.toBeNull(); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 1 }); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + const restored = await createSavedWorkspaceLoader(db, storage)({ companyId: ids.company, workspaceId: reserved.id, snapshotId: saved.snapshotId }); + try { expect(await fs.readFile(path.join(restored.root, "saved.txt"), "utf8")).toBe("work survives a failed provider session\n"); } + finally { await restored.cleanup(); } + } finally { unsubscribe(); } + } + if (kind === "historical_listener_bad_preservation") { + expect(await getExecutionBlocker(db, ids.company, ids.issue)).toMatchObject({ cause: "native_continuation_requires_reconciliation" }); + + } + if (processAcceptance) { + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + const { completeHistoricalListenerSuccessor } = await import("../__tests__/helpers/historical-listener-continuation.js"); + await completeHistoricalListenerSuccessor({ db, runId: successors[0]!.id, root: path.join(root, "successor-runtime"), + prepareWorkspace: async () => { + const leaseId = randomUUID(), runId = successors[0]!.id; + const [nextLease] = await db.insert(environmentLeases).values({ id: leaseId, companyId: ids.company, + environmentId: ids.environment, heartbeatRunId: runId, issueId: ids.issue, provider: "daytona", + providerLeaseId: `allocation-${leaseId}`, status: "active" }).returning(); + const next = await reserveRemoteWorkspace({ db, storage, companyId: ids.company, issueId: ids.issue, + runId, agentId: ids.agent, environmentId: ids.environment, projectId: null, projectWorkspaceId: null, + existingWorkspaceId: reserved.id, source: { kind: "empty" } }); + const nextTarget = { ...await target("historical-successor"), leaseId, sandboxId: nextLease.providerLeaseId! }; + const nextPrepared = await prepareRemoteWorkspace({ db, storage, workspace: next, runId, + lease: nextLease as EnvironmentLease, target: nextTarget }); + return { cwd: nextPrepared.remoteRoot, finalize: async () => { + const { workspaceOperationService } = await import("./workspace-operations.js"); + const recorder = workspaceOperationService(db).createRecorder({ companyId: ids.company, issueId: ids.issue, heartbeatRunId: runId }); + const saved = await recorder.recordOperation({ phase: "workspace_finalize", cwd: nextPrepared.remoteRoot, run: async () => { + await saveRemoteWorkspace({ db, storage, workspace: nextPrepared.workspace, runId, lease: nextPrepared.lease, + target: nextTarget, assertOwned: async () => undefined }); + return { status: "succeeded", exitCode: 0 }; + } }); + expect(saved.status).toBe("succeeded"); + } }; + } }); + } + if (!shouldSchedule) expect(await fs.readFile(path.join(prepared.remoteRoot, "saved.txt"), "utf8")) .toBe("work survives a failed provider session\n"); if (scenario === "unknown_action") { const [decision] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, ids.run)); diff --git a/server/src/services/remote-workspaces.ts b/server/src/services/remote-workspaces.ts index 63aa0bfe1e1..fbdd45e1e64 100644 --- a/server/src/services/remote-workspaces.ts +++ b/server/src/services/remote-workspaces.ts @@ -1,3 +1,4 @@ +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { workspaceAllocationFiles } from "./remote-workspace-allocation.js"; import { readNativeWorkspaceMigration } from "./native-runtime/native-workspace-migration.js"; import { createHash, randomUUID } from "node:crypto"; @@ -763,7 +764,10 @@ export async function saveRemoteWorkspace(input: { eq(executionWorkspaceSnapshots.executionWorkspaceId, workspace.id), eq(executionWorkspaceSnapshots.status, "uploaded"))).returning({ id: executionWorkspaceSnapshots.id }); if (!committed) throw new Error("remote_workspace_snapshot_publication_changed"); await notifyRemoteWorkspaceSnapshots(tx, `workspace:${workspace.id}`); - if (boundary === "final") await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`); + if (boundary === "final") { + await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement); + } const receipt = object(workspace.metadata?.remoteWorkspaceSnapshotRepositories); const priorRepositories = receipt.snapshotId === current.snapshotId && Array.isArray(receipt.repositories) ? remoteWorkspaceSourceSchema.parse({ kind: "empty", repositories: receipt.repositories }).repositories ?? [] : []; diff --git a/server/src/services/run-dispatch-notifications.ts b/server/src/services/run-dispatch-notifications.ts index 3cd4f00cbe1..36fad0e81a3 100644 --- a/server/src/services/run-dispatch-notifications.ts +++ b/server/src/services/run-dispatch-notifications.ts @@ -12,6 +12,7 @@ export async function notifyRunDispatch(transaction: object, scope?: RunDispatch await notifyIssueRecovery(transaction, scope ? { companyId: scope.companyId } : undefined); const hint = scope ? JSON.stringify({ companyId: scope.companyId }) : "*"; await notifyDeliveryWork(transaction, DELIVERY_QUEUES.nativeProcessSessions, hint); + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.executionReplacement, hint); await notifyQueuedRunDispatch(transaction, scope); } diff --git a/server/src/services/workspace-recovery.ts b/server/src/services/workspace-recovery.ts index 88f0efc82cc..b4d9a9d6595 100644 --- a/server/src/services/workspace-recovery.ts +++ b/server/src/services/workspace-recovery.ts @@ -390,6 +390,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ const activity = await db.transaction(async tx => { await notifyNativeResultCommit(tx, input); await notifyNativeWorkspaceFinalization(tx, input); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement); await notifyNativeStatusReassessment(tx, input); await tx.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId, heartbeatRunId: input.runId, phase: "workspace_finalize", status: "succeeded", finishedAt: new Date(), From 5d25117ffad69a93fb92026950b9acd9cec08f69 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 11:49:38 -0600 Subject: [PATCH 10/11] fix(heartbeat): recover deferred dependency completion wakes (#19457) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tasks wait for other tasks through saved dependency edges and wake requests. > - A dependency can finish while its assignee answers a question about the blocked task. > - Paperclip saves the completion wake behind that active turn. > - Recovery did not find that saved wake unless it also contained a comment or an interaction response. > - This pull request restores that wake through the existing release and dispatch checks. > - The task can then continue after the active turn and its cleanup finish. ## Linked Issues or Issue Description **What happened?** A task stayed blocked after its dependency finished. The completion wake was saved while an audit turn was active. The wake stayed deferred after that turn ended. It had no comment or interaction response, so both recovery queries and the input guard excluded it. **Expected behavior** Recover the saved dependency wake when the prior turn and its cleanup finish. Start one dependency continuation under the current execution rules. **Steps to reproduce** 1. Run a question response on a dependency-blocked task. 2. Finish the dependency while the response is active. 3. Save the completion wake behind that response. 4. Finish the response and its environment cleanup, or restart the workers. 5. Observe that the saved wake has no run. **Paperclip version or commit** Reproduced against `70d23612dd`. Rechecked against `2f35eb1080` with the final regression fixture. **Deployment mode** Source checkout on Linux x64, Node 25.9.0, pnpm 9.15.4, and isolated embedded PostgreSQL. Related work: #16673 added recovery for saved messages. #17288 and #16966 address assignment wakes after cleanup. This change covers dependency completion receipts with no message. ## What Changed - Include saved `issue_blockers_resolved` receipts in worker reconstruction and stranded queue selection. - Let these receipts pass the input guard without a comment or interaction response. - Use the existing release transaction and dispatch checks. Add no timer, migration, or direct task status change. - Add process-backed regression cases for restart, cleanup release, cancellation, agent pause, pending cleanup, and unresolved blockers. - Update the dependency wake recovery document. - Share the dependency-wake payload predicate and SQL fragment across reconstruction and stranded receipt admission, so these eligibility checks have one owner. ## Verification - The final regression fails on the base code: the receipt stays `deferred_issue_execution` and has no successor. The same test passes with the fix. - Six focused cases pass. They use real PostgreSQL and a real process adapter. They check saved output, one dependency continuation, clean task locks, repeat recovery, and execution gates. They do not call a model provider. - Three nearby suites pass: 125 tests across queue wiring, dependency scheduling, and dependency recovery. - Final-head server layered typechecking passes with `PAPERCLIP_TYPECHECK_LOW_MEMORY=1`. Whitespace and module boundary checks pass. Two parallel checker attempts reached the sandbox memory limit; the completed low-memory check covers all source and test layers. - The full deferred-receipt suite passes: 45 tests. Combined with the nearby suites, 170 local tests pass. These results cover head `8f9d3aa6ac19a6d2532bd869eb8adda7cf122a5a`. Required CI remains in progress. - Local workspace typechecking and build stop at the Runner Rust step because Cargo is absent. The full local test suite was not run. Required CI must supply broad coverage. - The hosted incident has not been recovered or deployed from this branch. - Review follow-up at `182385e9bc433adac06440997de7e4cfbf889a72`: normal merge of master `0d03610d2f73b7cd799547e43929bb2303319024`; no conflict or history rewrite. All 54 receipt/helper cases pass on macOS arm64, Node 26.4.0, real embedded PostgreSQL and process adapters. The six dependency continuation and safety cases also pass separately. Module boundaries and whitespace checks pass. No assertion, timeout, execution gate or retry policy changed for this extraction. - Current shared, SDK and adapter-utils source builds pass. The first broad local server typecheck read stale borrowed adapter declarations; rebuilding them exposed nominal type duplication between the borrowed workspace and this checkout. It is not counted as a server typecheck pass. All 30 SDK and 169 adapter runtime JavaScript outputs match the previously tested artifacts. Final revision `182385e9bc433adac06440997de7e4cfbf889a72` now passes all 55 applicable CI checks (three skipped, one neutral review), including clean-workspace typecheck, build and broad tests. [CI result](https://github.com/paperclipai/paperclip/actions/runs/42262173960). Greptile and Paperclip review both give 5/5 with no open findings. ## Risks - Recovery now considers a saved dependency receipt with no message. Current ownership, source finalization, cleanup, pause, cancellation, and dependency checks still control execution. - The scan remains bounded and uses existing transaction notifications and restart reconstruction. The integration checks verify that it becomes quiet after settlement. ## Model Used - OpenAI Codex, based on GPT-6. The exact provider model ID and context size are not exposed in this session. Used reasoning, tool calls, TypeScript edits, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/dependency-wake-recovery.md | 7 ++ .../deferred-execution-receipt-work.test.ts | 92 ++++++++++++++++++- .../heartbeat/deferred-receipt-work.ts | 2 + server/src/services/heartbeat/queue.ts | 12 ++- .../src/services/issue-dependency-wakeups.ts | 11 ++- 5 files changed, 119 insertions(+), 5 deletions(-) diff --git a/doc/dependency-wake-recovery.md b/doc/dependency-wake-recovery.md index 0187b99c6f3..41e94ef5ed5 100644 --- a/doc/dependency-wake-recovery.md +++ b/doc/dependency-wake-recovery.md @@ -13,6 +13,13 @@ unattributed operation in the same shared workspace does not remove its wait. Normal readiness, duplicate-wake, interaction, pause, and execution gates still apply. A board transition into blocked keeps the existing recovery behavior. +A dependency completion can arrive while the assignee answers a question about +the blocked task. Its saved wake waits behind that turn. Deferred-receipt recovery +reconstructs these dependency wakes after the turn and its cleanup finish, even +when they contain no comment or interaction response. It uses the existing +release and dispatch checks; task cancellation, agent pause, and pending cleanup +still prevent execution. + Cancelling a run or pausing its agent suppresses immediate stranded-work recovery. A queued user message remains a separate continuation request. The release path still preserves independently authorized deferred input; stopping the old run diff --git a/server/src/__tests__/deferred-execution-receipt-work.test.ts b/server/src/__tests__/deferred-execution-receipt-work.test.ts index be0135fc64e..414ac9d5abf 100644 --- a/server/src/__tests__/deferred-execution-receipt-work.test.ts +++ b/server/src/__tests__/deferred-execution-receipt-work.test.ts @@ -15,7 +15,7 @@ import { issueTreeControlService } from "../services/issue-tree-control.js"; import { randomUUID } from "node:crypto"; import { and, eq, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; -import { activityLog, agents, agentWakeupRequests, authUsers, chatActions, environmentLeases, nativeRunFinalizations, chatEndpoints, toolApplications, toolConnections, companies, companyMemberships, costEvents, createDb, projects, reconcileDatabaseWork, heartbeatRuns, issueComments, issues } from "@paperclipai/db"; +import { activityLog, agents, agentWakeupRequests, authUsers, chatActions, environmentLeases, nativeRunFinalizations, chatEndpoints, toolApplications, toolConnections, companies, companyMemberships, costEvents, createDb, projects, reconcileDatabaseWork, heartbeatRuns, issueComments, issueRelations, issues } from "@paperclipai/db"; import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; @@ -158,6 +158,96 @@ describe("deferred receipt durable dispatch", () => { source: "on_demand", status: "completed", reason: "task_retry", payload: { issueId: f.issue.id } }); await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; await quiet(); }); + async function strandedDependency() { + const f = await fixture(); + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: f.issue.id, + status: "succeeded", errorCode: null, contextSnapshot: { issueId: f.issue.id, + wakeReason: "issue_commented", dependencyBlockedInteraction: true } }).where(eq(heartbeatRuns.id, f.source.id)); + await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id, + runId: f.source.id, phase: "committed" }); + await db.update(issues).set({ status: "blocked", executionRunId: null, checkoutRunId: null }).where(eq(issues.id, f.issue.id)); + const [blocker] = await db.insert(issues).values({ companyId: f.companyId, title: "Completed review", + status: "done", completedAt: new Date() }).returning(); + await db.insert(issueRelations).values({ companyId: f.companyId, issueId: blocker.id, + relatedIssueId: f.issue.id, type: "blocks" }); + const [intent] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, + source: "automation", reason: "issue_blockers_resolved", status: "coalesced", + requestedByActorType: "system", requestedByActorId: "native-status-committer", + payload: { issueId: f.issue.id, resolvedBlockerIssueId: blocker.id }, + }).returning(); + const [receipt] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, + source: "automation", reason: "issue_execution_deferred", status: "deferred_issue_execution", + requestedByActorType: "system", requestedByActorId: `native-status-wake-dispatch:${intent.id}`, + payload: { issueId: f.issue.id, resolvedBlockerIssueId: blocker.id, _paperclipWakeContext: { + issueId: f.issue.id, wakeReason: "issue_blockers_resolved", statusDecisionSource: "native_status_decision", + nativeStatusWakeIntentId: intent.id } }, + }).returning(); + return { ...f, receipt, blocker }; + } + + it.each(["restart", "cleanup"] as const)("recovers a dependency wake after native audit %s into one continuation", async event => { + const f = await strandedDependency(); + const { receipt } = f; + const [lease] = event === "cleanup" ? await db.insert(environmentLeases).values({ companyId: f.companyId, + heartbeatRunId: f.source.id, issueId: f.issue.id, provider: "daytona", providerLeaseId: "dependency-finalizing", + status: "active", leasePolicy: "reuse_by_environment" }).returning() : []; + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; + await heartbeat.registerRunDispatchWork(coordinator, () => !drain).ready; + if (lease) { + await waitForReal(async () => { + await Promise.allSettled([...passes]); + expect(coordinator.nextWakeAt()).toBeNull(); expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 10_000 }); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.id)))[0]) + .toMatchObject({ status: "deferred_issue_execution", runId: null }); + await environmentService(db).releaseLease(lease.id, "retained", { cleanupStatus: "success" }); + } + await waitForReal(async () => { + await Promise.allSettled([...passes]); await heartbeat.drainActiveRunExecutions(); + const [saved] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.id)); + const pending = saved.runId ? (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, saved.runId)))[0] : null; + expect(saved.status, JSON.stringify({ status: pending?.status, error: pending?.error, errors })).toBe("completed"); + expect(saved.runId).not.toBeNull(); + }, { timeout: 15_000 }); + await finish(f.issue.id); + const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId)); + const dependencyRuns = runs.filter(run => run.contextSnapshot?.wakeReason === "issue_blockers_resolved"); + expect(dependencyRuns).toHaveLength(1); + const [resumed] = dependencyRuns; + expect(resumed).toMatchObject({ status: "succeeded", stdoutExcerpt: expect.stringContaining("receipt-proof"), + contextSnapshot: { wakeReason: "issue_blockers_resolved" } }); + expect(resumed.contextSnapshot?.dependencyBlockedInteraction).not.toBe(true); + await heartbeat.resumeQueuedRuns(); + const after = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId)); + expect(after.filter(run => run.contextSnapshot?.wakeReason === "issue_blockers_resolved")).toHaveLength(1); + expect(after.find(run => run.id === f.source.id)?.status).toBe("succeeded"); + expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0]) + .toMatchObject({ status: "done", executionRunId: null, checkoutRunId: null }); + await quiet(); + }); + it.each(["paused", "cleanup", "cancelled"] as const)("keeps the %s gate when recovering a dependency wake", async gate => { + const f = await strandedDependency(); + if (gate === "paused") await db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agent.id)); + if (gate === "cancelled") await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issue.id)); + if (gate === "cleanup") await db.insert(environmentLeases).values({ companyId: f.companyId, + heartbeatRunId: f.source.id, issueId: f.issue.id, provider: "daytona", providerLeaseId: "dependency-cleanup", + status: "active", leasePolicy: "reuse_by_environment" }); + await heartbeat.resumeQueuedRuns(); + const [saved] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.receipt.id)); + expect(saved).toMatchObject({ status: gate === "cancelled" ? "cancelled" : "deferred_issue_execution", runId: null }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1); + }); + it("checks unresolved blockers before a recovered dependency wake can execute", async () => { + const f = await strandedDependency(); + await db.update(issues).set({ status: "blocked", completedAt: null }).where(eq(issues.id, f.blocker.id)); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.receipt.id)); + expect(receipt).toMatchObject({ status: "skipped", runId: expect.any(String) }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, receipt.runId!)); + expect(run).toMatchObject({ status: "cancelled", errorCode: "issue_dependencies_blocked", stdoutExcerpt: null }); + expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0].status).toBe("blocked"); + }); it.each([false, true])("reconsiders every saved receipt during repeated company eligibility changes (expired wait: %s)", async expiredWait => { const f = await fixture(); const ids = Array.from({ length: 178 }, (_, index) => `${(index + 1).toString(16).padStart(8, "0")}-0000-4000-8000-000000000000`); diff --git a/server/src/services/heartbeat/deferred-receipt-work.ts b/server/src/services/heartbeat/deferred-receipt-work.ts index b799f10b0a7..915d72b7b07 100644 --- a/server/src/services/heartbeat/deferred-receipt-work.ts +++ b/server/src/services/heartbeat/deferred-receipt-work.ts @@ -5,6 +5,7 @@ import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; import type { DeferredExecutionScope, DeferredReceiptRecoveryOptions } from "../deferred-execution-notifications.js"; import { registerScopedDeadlineWork } from "../scoped-deadline-work.js"; +import { issueBlockersResolvedWakePayloadSql } from "../issue-dependency-wakeups.js"; type Entry = { id: string; companyId: string; dueAt: number; immediate: boolean }; type NativeIntentOptions = DeferredExecutionScope & { @@ -74,6 +75,7 @@ export function registerDeferredReceiptWork( sql`jsonb_typeof(${agentWakeupRequests.payload}#>'{_paperclipWakeContext,wakeCommentIds}') = 'array' and ${agentWakeupRequests.payload}#>'{_paperclipWakeContext,wakeCommentIds}' <> '[]'::jsonb`, sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`, + issueBlockersResolvedWakePayloadSql, sql`exists (select 1 from heartbeat_runs r where r.company_id = ${agentWakeupRequests.companyId} and r.runtime_mode = 'legacy' and r.status = 'cancelled' and r.result_json->>'queuedCommentInterruptQueueId' = ${agentWakeupRequests.id}::text)`, diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index 4da678e4b38..514d0a005b7 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -144,7 +144,11 @@ import { isThrottleCandidateIssueRewake, } from "../issue-rewake-throttle.js"; import { logActivity } from "../activity-log.js"; -import { ISSUE_BLOCKERS_RESOLVED_WAKE_REASON } from "../issue-dependency-wakeups.js"; +import { + ISSUE_BLOCKERS_RESOLVED_WAKE_REASON, + isIssueBlockersResolvedWakePayload, + issueBlockersResolvedWakePayloadSql, +} from "../issue-dependency-wakeups.js"; import { resumeIfPauseAlreadyReleased } from "../issue-tree-resume.js"; import { ISSUE_PAUSED_ERROR_CODE, @@ -1757,7 +1761,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende or(and( sql`jsonb_typeof(${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}') = 'array'`, sql`${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}' <> '[]'::jsonb`, - ), sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`), + ), sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`, + issueBlockersResolvedWakePayloadSql), sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is null`, sql`NOT EXISTS (SELECT 1 FROM heartbeat_runs live_issue_run WHERE live_issue_run.company_id = ${issues.companyId} @@ -1768,7 +1773,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende .orderBy(asc(agentWakeupRequests.updatedAt)).limit(50); const releasedNativeSources = new Set(); for (const { wake } of strandedQueues) { - if (!queuedCommentIdsFromWakePayload(wake.payload).length && + const dependencyWake = isIssueBlockersResolvedWakePayload(wake.payload); + if (!dependencyWake && !queuedCommentIdsFromWakePayload(wake.payload).length && !await readQueuedInteractionResponse(db, wake.companyId, String(wake.payload?.issueId), wake.payload)) continue; const [latest] = await db.select({ id: heartbeatRuns.id, diff --git a/server/src/services/issue-dependency-wakeups.ts b/server/src/services/issue-dependency-wakeups.ts index ee09f3e55ff..4d891d583a8 100644 --- a/server/src/services/issue-dependency-wakeups.ts +++ b/server/src/services/issue-dependency-wakeups.ts @@ -1,10 +1,19 @@ -import { and, eq, inArray } from "drizzle-orm"; +import { and, eq, inArray, sql } from "drizzle-orm"; import { createHash } from "node:crypto"; import type { Db } from "@paperclipai/db"; import { agentWakeupRequests } from "@paperclipai/db"; export const ISSUE_BLOCKERS_RESOLVED_WAKE_REASON = "issue_blockers_resolved"; +export const issueBlockersResolvedWakePayloadSql = + sql`${agentWakeupRequests.payload}#>>'{_paperclipWakeContext,wakeReason}' = ${ISSUE_BLOCKERS_RESOLVED_WAKE_REASON}`; + +export function isIssueBlockersResolvedWakePayload(payload: Record | null | undefined): boolean { + const context = payload?._paperclipWakeContext; + return context !== null && typeof context === "object" && !Array.isArray(context) + && (context as Record).wakeReason === ISSUE_BLOCKERS_RESOLVED_WAKE_REASON; +} + /** * Whether moving an issue into `blocked` should immediately wake its assignee * because every recorded blocker is already resolved ("blockers restored"). From 1485d3de1b305d9862f724c662b5740ef9ae99db Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 12:00:50 -0600 Subject: [PATCH 11/11] fix(ui): hold announcements until the onboarding first task is finished (#19514) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A new user's first session is the onboarding first task. The agent asks its onboarding questions in that task thread. > - The app also shows an optional in-app announcement. The live feed now serves "Introducing Connectors" (`2026-09-14-connectors`). > - The announcement well only waits while the onboarding wizard dialog is open. It shows 3 seconds after the user lands on the first task. > - In the Nov 5 cloud onboarding test (Claude API path), the tester said: "Don't show the connectors upsell until the second session. It's blocking the user flyout and potentially the onboarding questions." > - This pull request holds announcements until the company's onboarding first task is done or cancelled. > - The benefit is that a new user sees the onboarding questions and the account menu without an upsell on top of them. ## Linked Issues or Issue Description Refs #19511 (places the card in the desktop sidebar, above the account menu. That PR keeps the card off the account menu after onboarding. This PR adds the first-session gate that #19511 leaves open.) **What happened?** On a new company's onboarding first task, the "Introducing Connectors" card shows 3 seconds after the page loads. At 1440x900 the card (352x346 px, bottom-left, z-40) covers the sidebar account menu trigger and the left edge of the task composer. **Expected behavior** Optional announcements do not show in the first session, while the onboarding first task is open. **Steps to reproduce** 1. Create a new company through onboarding. The wizard creates the onboarding first task. 2. Open the first task at 1440x900 and wait 3 seconds. 3. See the announcement card on top of the account menu and the composer. **Paperclip version or commit** `master` at 819420a60a. ## What Changed - `AnnouncementWell` reads the company's onboarding first task (`originKind=onboarding_first_task`, `limit=1`) with React Query under `["issues", companyId, "onboarding-first-task"]`. Issue live-update invalidation of `["issues", companyId]` refreshes it. - The well stays disabled while that task is not `done` or `cancelled`. While the read is loading or after it fails, the optional card stays hidden. A company that never had an onboarding first task is not gated. - "First session" uses the existing `origin_kind` marker. There is no schema change. - New e2e spec `tests/e2e/announcement-first-session.spec.ts` with a mocked feed that copies the live connectors entry. It checks that the card is absent while the onboarding first task is open. Then it marks the task done on the same page and checks that the card appears without a reload (a positive control for the feed mock and the live refresh path). - New unit test `AnnouncementWell.onboarding.test.tsx`. It covers open, finished, no first task, read failure, and the open-to-done transition after `["issues", companyId]` is invalidated. `AnnouncementWell.test.tsx` now gives the well a `QueryClient` with a finished onboarding state. ## Verification Red on `master` (819420a60a), before the fix. Vitest: ``` cd ui && npx vitest run src/components/AnnouncementWell.onboarding.test.tsx × stays hidden while the onboarding first task is open × stays hidden when onboarding state cannot be read Tests 2 failed | 1 passed (3) AssertionError: expected true to be false ``` Red on `master`, Playwright at 1440x900 (an earlier draft of the spec also had a placement check, recorded here as evidence for the flyout part of the report): ``` ✘ announcement-first-session.spec.ts › a brand-new company with its onboarding task open does not show the upsell Error: expect(locator).toHaveCount(expected) failed Locator: getByRole('complementary', { name: 'Paperclip announcements' }) Expected: 0 Received: 1 ✘ … where the upsell shows, it does not cover the account menu or its flyout Error: well {"x":12,"y":542,"width":352,"height":346} covers account trigger {"x":20,"y":856,"width":166,"height":36} ``` Green after the fix, merged with current `master` (f7070d68b5), with the in-page positive control: ``` ✓ announcement-first-session.spec.ts › a brand-new company with its onboarding task open does not show the upsell (10.1s) 1 passed cd ui && npx vitest run src/components/AnnouncementWell.onboarding.test.tsx src/components/AnnouncementWell.test.tsx Tests 6 passed (6) cd ui && npx vitest run src/components/AnnouncementWell.onboarding.test.tsx src/components/AnnouncementWell.test.tsx \ src/components/AnnouncementCard.test.tsx src/hooks/useAnnouncement.test.tsx src/components/SidebarAccountMenu.test.tsx src/components/Layout.test.tsx Test Files 6 passed (6) Tests 95 passed (95) ``` - `pnpm check:token-gates`: All gates clean. - UI `tsc --noEmit`: no errors in app code. The only errors are the git-ignored, generated Storybook catalog files. - CI note: the first CI run had three jobs fail with "The self-hosted runner lost communication with the server" (server tests 12/12, Runner vitest package, e2e shard 5/8). These are infra flakes. I reran them. - Local e2e note: two runs could not start the server because embedded PostgreSQL ports were in use by parallel e2e runs on the same machine. A rerun passed. This is not related to the change. Before (master): the card covers the account menu and the composer on the onboarding first task. ![before: onboarding first task](https://raw.githubusercontent.com/paperclipai/paperclip/c97519868225b0c206cf95fa3fb7150287c88116/n13-before-onboarding-task.png) After: no card on the onboarding first task. The account menu and the composer are clear. ![after: onboarding first task](https://raw.githubusercontent.com/paperclipai/paperclip/c97519868225b0c206cf95fa3fb7150287c88116/n13-after-onboarding-task.png) ## Risks - Low risk. The change only hides optional content. - A user who never finishes or cancels the onboarding first task does not see announcements in that company. This is the intended trade-off for "not in the first session". Closing the task shows them again. - The well makes one extra small issues read (`limit=1`) per company. It is cached for 60 seconds. - `AnnouncementWell.tsx` also changes in #19511. Whichever PR lands second resolves a small conflict in the `enabled` condition and the `