From cb47dbb2dac98bd3527c4eaaabca0b0119e2d9f2 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 16:49:21 -0700 Subject: [PATCH 01/68] fix: resolve git credentials with the issue's identity in background jobs (#17515) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Git cache expiration, isolated source re-home, nested clone migration and workspace reopen fetch from the workspace's remotes to prove history is on the remote or to rebuild workspaces > - They built git credentials from the issue id alone, but the managed GitHub identity is selected from the run's responsible user and agent > - So the credential never resolved, they fell back to anonymous access, and private repositories were unreadable > - This pull request resolves credentials with the issue's responsible user and assigned agent, the same identity agent runs use > - The benefit is that workspaces with private repositories can expire, re-home, migrate and reopen like public ones ## Linked Issues or Issue Description Refs #17464, #17407, #17369. Bug report described in-PR (bug report template). **What happened?** On one instance, 18 long-closed workspaces in one project were kept on every expiration sweep with `managed_checkout_upstream_unavailable`. Their nested checkout is a private GitHub repository. Expiration called `createGitRemoteAuthProvider(db, companyId, { issueId })`. With that context, `resolveManagedGitHubCredential` returned "No managed GitHub identity is available for this run". `withAnonymousGitFallback` then tried anonymous access, which cannot read the private repository. Called with the issue's `responsibleUserId` and `agentId`, the same lookup returned a credential. **Expected behavior** Background workspace jobs authenticate as the issue's identity, as agent runs do, and can verify private repositories. **Steps to reproduce** 1. Configure a project with a private GitHub repository and a managed GitHub identity tied to the issue's responsible user. 2. Close an issue whose workspace includes that repository and let git cache expiration inspect it. 3. It is kept with `managed_checkout_upstream_unavailable`. **Paperclip version or commit:** `master` at fe85fd7ac4. **Deployment mode:** any with managed GitHub identities. **Agent adapter(s) involved:** not adapter-specific (core). ## What Changed - `createIssueGitRemoteAuthProvider(db, companyId, issueId)` in `server/src/services/git-credentials.ts`: - looks up the issue's `responsibleUserId` and `assigneeAgentId` once, lazily, scoped to the company; - builds `createGitRemoteAuthProvider` with `{ issueId, responsibleUserId, agentId }`; - falls back to the issue-only context when the issue is missing. - Callers switched from the issue-only context: - git cache expiration (`expirationGitAuth`); - isolated source re-home; - nested clone migration; - both credential sites in `reopenClosedIsolatedExecutionWorkspaceForIssue`. - Anonymous fallback for public repositories is unchanged. No code comments were added. ## Verification - `pnpm exec vitest run src/__tests__/git-credentials.test.ts src/__tests__/git-expiration-auth-fallback.test.ts` (in `server/`): 32 passed. New cases: - the helper passes the issue's responsible user and agent to the credential lookup; - it looks the issue up once across calls; - it falls back to the issue-only context when the issue is missing. - **Mutation check:** dropping the identity fields from the context fails two cases. - **Live instance:** for an affected workspace, the managed credential resolved only with the responsible user and agent; with the issue id alone, or with standing delegation, it did not. - `pnpm exec vitest run git-credentials git-expiration-auth-fallback git-workspace-expiration git-restore-point isolated-repository-rehome nested-clone-migration` (in `server/`): 296 passed. - `pnpm run typecheck` (in `server/`): pass. - Not run: server build and boot smoke. ## Risks - **Whose identity is used:** background jobs now use the issue's responsible user's managed GitHub identity, which agent runs on that issue already use. They only read: fetch, ls-remote and clone for verification and restore. - **Unchanged failure behavior:** if the identity is unavailable (for example, the user is no longer a member), the existing anonymous fallback applies, as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, tests, read-only diagnosis on a live instance). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- server/src/__tests__/git-credentials.test.ts | 37 +++++++++++++++++++ server/src/services/execution-workspaces.ts | 8 ++-- server/src/services/git-credentials.ts | 14 +++++++ .../src/services/git-workspace-expiration.ts | 4 +- .../services/isolated-repository-rehome.ts | 4 +- server/src/services/nested-clone-migration.ts | 4 +- 6 files changed, 60 insertions(+), 11 deletions(-) diff --git a/server/src/__tests__/git-credentials.test.ts b/server/src/__tests__/git-credentials.test.ts index c286f832b42..e5aee1eb96f 100644 --- a/server/src/__tests__/git-credentials.test.ts +++ b/server/src/__tests__/git-credentials.test.ts @@ -9,6 +9,7 @@ import { GIT_CREDENTIAL_TOKEN_ENV_KEY, buildGitAuthInvocation, createGitRemoteAuthProvider, + createIssueGitRemoteAuthProvider, describeGitAuthFailure, isGitHubHttpsRemoteUrl, scrubGitCredentialText, @@ -379,3 +380,39 @@ describe("DEFAULT_GITHUB_TOKEN_SECRET_NAMES", () => { ]); }); }); + +describe("createIssueGitRemoteAuthProvider", () => { + const githubUrl = "https://github.com/paperclipai/private-repo.git"; + function issueDb(rows: unknown[]) { + const where = vi.fn(async () => rows); + return { db: { select: vi.fn(() => ({ from: () => ({ where }) })) } as unknown as Db, where }; + } + + it("resolves credentials with the issue's responsible user and assigned agent", async () => { + const { db } = issueDb([{ responsibleUserId: "user-1", assigneeAgentId: "agent-1" }]); + const inner = vi.fn(async () => ({ env: {}, configArgs: [], source: "managed_connection" })); + const create = vi.fn(() => inner) as unknown as typeof createGitRemoteAuthProvider; + const provider = createIssueGitRemoteAuthProvider(db, "company-1", "issue-1", create); + await expect(provider(githubUrl)).resolves.toMatchObject({ source: "managed_connection" }); + expect(create).toHaveBeenCalledWith(db, "company-1", { issueId: "issue-1", responsibleUserId: "user-1", agentId: "agent-1" }); + expect(inner).toHaveBeenCalledWith(githubUrl); + }); + + it("looks the issue up once across calls", async () => { + const { db } = issueDb([{ responsibleUserId: "user-1", assigneeAgentId: null }]); + const inner = vi.fn(async () => null); + const create = vi.fn(() => inner) as unknown as typeof createGitRemoteAuthProvider; + const provider = createIssueGitRemoteAuthProvider(db, "company-1", "issue-1", create); + await provider(githubUrl); await provider(githubUrl); + expect(db.select).toHaveBeenCalledTimes(1); + expect(create).toHaveBeenCalledTimes(1); + expect(inner).toHaveBeenCalledTimes(2); + }); + + it("falls back to the issue-only context when the issue is missing", async () => { + const { db } = issueDb([]); + const create = vi.fn(() => vi.fn(async () => null)) as unknown as typeof createGitRemoteAuthProvider; + await createIssueGitRemoteAuthProvider(db, "company-1", "issue-1", create)(githubUrl); + expect(create).toHaveBeenCalledWith(db, "company-1", { issueId: "issue-1", responsibleUserId: null, agentId: null }); + }); +}); diff --git a/server/src/services/execution-workspaces.ts b/server/src/services/execution-workspaces.ts index 63800f66b9e..4d83e2c8a9c 100644 --- a/server/src/services/execution-workspaces.ts +++ b/server/src/services/execution-workspaces.ts @@ -98,7 +98,7 @@ import { type PullRequestMergeDetailsResolver, } from "./github-pull-request-merge.js"; import { visibleIssueCondition } from "./issue-visibility.js"; -import { createGitRemoteAuthProvider } from "./git-credentials.js"; +import { createIssueGitRemoteAuthProvider } from "./git-credentials.js"; import { readProjectWorkspaceRuntimeConfig } from "./project-workspace-runtime-config.js"; import { workspaceGitOperationScheduler } from "./workspace-git-operation-scheduler.js"; import { isRuntimeOwnedGitBranch } from "./execution-workspace-branch-ownership.js"; @@ -4113,7 +4113,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic try { recipe = readGitCacheRestoration(snapshot); } catch { return { ok: false, code: "rebuild_failed", message: new GitCacheRestorationFailure().message }; } const freshStart = recipe !== null && isClosedExecutionWorkspaceStatus(snapshot.status) && gitRestorePointExpired(recipe); - const gitAuth = createGitRemoteAuthProvider(db, snapshot.companyId, { issueId: issue.id }); + const gitAuth = createIssueGitRemoteAuthProvider(db, snapshot.companyId, issue.id); const [ { ensurePersistedExecutionWorkspaceAvailable }, { workspaceOperationService }, @@ -4168,9 +4168,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic }); }; }, - resolveGitAuth: createGitRemoteAuthProvider(db, snapshot.companyId, { - issueId: issue.id, - }), + resolveGitAuth: createIssueGitRemoteAuthProvider(db, snapshot.companyId, issue.id), }); resolvedBaseCwd = managedWorkspace.cwd; } catch { diff --git a/server/src/services/git-credentials.ts b/server/src/services/git-credentials.ts index 21899e5cef1..3fce7fc2b50 100644 --- a/server/src/services/git-credentials.ts +++ b/server/src/services/git-credentials.ts @@ -1,6 +1,7 @@ import { companySecrets, heartbeatRuns, + issues, companyMemberships, connectionGrantDelegations, connectionGrants, @@ -207,6 +208,19 @@ type GitCredentialSecretsDeps = { * provider instance so one run performs at most one secret resolution (and writes at most * one audit event) no matter how many git operations it authenticates. */ +export function createIssueGitRemoteAuthProvider(db: Db, companyId: string, issueId: string, + create: typeof createGitRemoteAuthProvider = createGitRemoteAuthProvider): GitRemoteAuthProvider { + let provider: Promise | null = null; + return async (remoteUrl: string) => { + provider ??= db.select({ responsibleUserId: issues.responsibleUserId, assigneeAgentId: issues.assigneeAgentId }).from(issues) + .where(and(eq(issues.id, issueId), eq(issues.companyId, companyId))) + .then(([issue]) => create(db, companyId, { + issueId, responsibleUserId: issue?.responsibleUserId ?? null, agentId: issue?.assigneeAgentId ?? null, + })); + return (await provider)(remoteUrl); + }; +} + export function createGitRemoteAuthProvider( db: Db, companyId: string, diff --git a/server/src/services/git-workspace-expiration.ts b/server/src/services/git-workspace-expiration.ts index 12cb8e52dda..5fdd85c7f96 100644 --- a/server/src/services/git-workspace-expiration.ts +++ b/server/src/services/git-workspace-expiration.ts @@ -6,7 +6,7 @@ import { acquireGitExpirationLocks } from "./git-workspace-expiration-locks.js"; import { and, eq, inArray, ne, sql } from "drizzle-orm"; import { executionWorkspaces, issues, heartbeatRuns, nativeRunFinalizations, environmentLeases, workspaceRuntimeServices, issueRecoveryActions, projectWorkspaces, withDedicatedDbConnection, type Db } from "@paperclipai/db"; -import { createGitRemoteAuthProvider } from "./git-credentials.js"; +import { createIssueGitRemoteAuthProvider } from "./git-credentials.js"; import type { StorageService } from "../storage/types.js"; import { TASK_DIRECTORY_RETENTION_KEY } from "./isolated-task-directory-snapshot.js"; import { currentRestorePointSnapshot, RestorePointSkip, type RestorePointLimits } from "./git-restore-point.js"; @@ -37,7 +37,7 @@ export function withAnonymousGitFallback(provider: GitRemoteAuthProvider): GitRe } function expirationGitAuth(db: Db, workspace: Workspace) { - return withAnonymousGitFallback(createGitRemoteAuthProvider(db, workspace.companyId, { issueId: workspace.sourceIssueId! })); + return withAnonymousGitFallback(createIssueGitRemoteAuthProvider(db, workspace.companyId, workspace.sourceIssueId!)); } const EXPIRATION_KEY = "gitCacheExpiration"; diff --git a/server/src/services/isolated-repository-rehome.ts b/server/src/services/isolated-repository-rehome.ts index c8c02f85578..6314f5edcc7 100644 --- a/server/src/services/isolated-repository-rehome.ts +++ b/server/src/services/isolated-repository-rehome.ts @@ -23,7 +23,7 @@ import { withExecutionWorkspaceLifecycleLock } from "./execution-workspace-lifec import { loadGitRestorePoint } from "./git-restore-point-storage.js"; import { nestedWorktreeBranchPrefix, readLinkedNestedCheckout, repairMovedNestedRepositoryWorktree, withNestedRepositoryBaseLock } from "./nested-repository-worktrees.js"; import { getStorageService } from "../storage/index.js"; -import { createGitRemoteAuthProvider } from "./git-credentials.js"; +import { createIssueGitRemoteAuthProvider } from "./git-credentials.js"; import { resolvePaperclipInstanceRoot } from "../home-paths.js"; const execFile = promisify(execFileCallback); @@ -598,7 +598,7 @@ async function worktreeOfClone(worktree: string, cloneReal: string, signal?: Abo } function upstreamCheck(db: Db, companyId: string, issueId: string, clone: string, repoUrl: string, signal?: AbortSignal): MissingUpstream { - const auth = withAnonymousGitFallback(createGitRemoteAuthProvider(db, companyId, { issueId })); + const auth = withAnonymousGitFallback(createIssueGitRemoteAuthProvider(db, companyId, issueId)); return (commits, covered) => hasCommitsMissingUpstream(clone, commits, repoUrl, auth, signal, covered); } diff --git a/server/src/services/nested-clone-migration.ts b/server/src/services/nested-clone-migration.ts index eb0e3a87eda..09b4f6663a9 100644 --- a/server/src/services/nested-clone-migration.ts +++ b/server/src/services/nested-clone-migration.ts @@ -13,7 +13,7 @@ import { GIT_CACHE_RESTORATION_KEY, readGitCacheRestoration, type GitCacheRestor import { readManagedCheckoutContents, selectRestoredProjectRepositories } from "./git-workspace-contents.js"; import { applyRestorePoint, captureRestorePoint, currentRestorePointSnapshot, importRestorePoint } from "./git-restore-point.js"; import { managedProjectCheckoutName } from "./managed-project-checkout-name.js"; -import { createGitRemoteAuthProvider, type GitRemoteAuthProvider } from "./git-credentials.js"; +import { createIssueGitRemoteAuthProvider, type GitRemoteAuthProvider } from "./git-credentials.js"; import { isNestedWorktreeBranchName, isVerifiedNestedRepositoryBase, moveNestedRepositoryWorktree, nestedRepositoryBaseCandidates, nestedWorktreeBranchPrefix, readLinkedNestedCheckout, removeStaleNestedRegistrations, trackNestedWorktreeUpstream, withNestedRepositoryBaseLock } from "./nested-repository-worktrees.js"; import { assertSafeBaseConfig, assertTransferredItems, historyKey, linkedCheckoutBases, moveTransferItems, presentCommits, returnTransferItems, @@ -442,7 +442,7 @@ export async function migrateNestedRepositoryClone(db: Db, snapshot: Workspace, const base = source.base; await assertSafeBaseConfig(base, signal); await fetchBase(base, source.repoUrl, options.resolveGitAuth - ?? withAnonymousGitFallback(createGitRemoteAuthProvider(db, row!.companyId, { issueId: row!.sourceIssueId! })), signal); + ?? withAnonymousGitFallback(createIssueGitRemoteAuthProvider(db, row!.companyId, row!.sourceIssueId!)), signal); await assertSafeBaseConfig(base, signal); await assertNoBranchOnlyReflogHistory(inspected, signal); const planned = await planRefs(inspected, base, prefix, signal); From 71f06d84dc41c6e35d368147540588fd8fd8073d Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 17:34:41 -0700 Subject: [PATCH 02/68] fix(db): keep bounded response captures attributable (#17522) Keep more caller groups in one bounded database response capture and verify public caller paths in the production bundle. All local checks and CI passed; Greptile scored the exact head 5/5 with no unresolved comments. Co-Authored-By: Paperclip --- doc/DATABASE.md | 13 +++++++++-- packages/db/src/response-capture.test.ts | 7 +++--- packages/db/src/response-capture.ts | 2 +- server/scripts/bundle-runtime.test.mjs | 29 +++++++++++++++++++++++- 4 files changed, 44 insertions(+), 7 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index bc1292bbe94..e6211773d87 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -332,12 +332,21 @@ The diagnostic measures awaited Postgres.js `unsafe` responses before Drizzle column decoding. It retains the Drizzle builder caller for normal, prepared, transaction, and dedicated-pool execution. It issues no queries. It logs the public server caller locations, static schema table, operation, -query/row counts, and response value sizes. It emits at most 128 aggregate +query/row counts, and response value sizes. It emits at most 512 aggregate groups and a summary at the deadline. It never logs SQL, parameters, result values, connection strings, or query errors. In-flight queries continue normally. +For a bundled server, keep the linked source maps and enable Node's +`--enable-source-maps` option so the diagnostic can resolve public caller +locations. Restore the prior Node options after capture. Sizes still count when +caller locations are unavailable. + Look for `paperclip.db_response_capture.entry.v1` and -`paperclip.db_response_capture.summary.v1` in the process logs. Tagged raw-driver calls, streams, unawaited raw-driver +`paperclip.db_response_capture.summary.v1` in the process logs. A log provider +can store `event` as a structured field instead of message text. Filter by that +field or read the bounded window from the exact deployment. Check the summary +and group totals, including the unattributed overflow group. +Tagged raw-driver calls, streams, unawaited raw-driver dispatch, and protocol overhead are not measured. Relational queries use a driver-level caller fallback. Queries still in flight at the deadline are reported separately. Sizes describe driver values before column decoding, not billed network bytes. Complex parsed diff --git a/packages/db/src/response-capture.test.ts b/packages/db/src/response-capture.test.ts index 65f58d10883..4b87c7924a4 100644 --- a/packages/db/src/response-capture.test.ts +++ b/packages/db/src/response-capture.test.ts @@ -107,11 +107,12 @@ describe("bounded database response capture", () => { }); it("bounds cardinality without dropping response-byte totals", () => { const f = fixture(); - for (let i = 0; i < 200; i += 1) f.capture.execute(() => [i], { ...metadata, callers: [`services/f${i}.ts:1:1`] }); + for (let i = 0; i < 600; i += 1) f.capture.execute(() => [i], { ...metadata, callers: [`services/f${i}.ts:1:1`] }); f.capture.flush(); const summary = f.events.at(-1)!; - expect(summary).toMatchObject({ startedQueries: 200, completedQueries: 200, groups: 128, overflowCalls: 73 }); - expect(summary.responseValueBytes).toBe(Array.from({ length: 200 }, (_, i) => Buffer.byteLength(JSON.stringify([i]))).reduce((a, b) => a + b, 0)); + expect(summary).toMatchObject({ startedQueries: 600, completedQueries: 600, groups: 512, overflowCalls: 89 }); + expect(f.events).toHaveLength(513); + expect(summary.responseValueBytes).toBe(Array.from({ length: 600 }, (_, i) => Buffer.byteLength(JSON.stringify([i]))).reduce((a, b) => a + b, 0)); }); it("does not execute builders or prepared statements until requested", async () => { const f = fixture(), rows = [{ id: 1 }], execute = vi.fn(async () => rows); diff --git a/packages/db/src/response-capture.ts b/packages/db/src/response-capture.ts index 8f738043f4d..f8e6da8e87a 100644 --- a/packages/db/src/response-capture.ts +++ b/packages/db/src/response-capture.ts @@ -99,7 +99,7 @@ export function createDatabaseResponseCapture( completed += 1; try { let key = JSON.stringify(metadata); - if (!entries.has(key) && entries.size >= 127) { + if (!entries.has(key) && entries.size >= 511) { overflowCalls += 1; metadata = { operation: "other", table: "other", callers: [] }; key = "overflow"; diff --git a/server/scripts/bundle-runtime.test.mjs b/server/scripts/bundle-runtime.test.mjs index 14b0d46c681..6e9375f61c4 100644 --- a/server/scripts/bundle-runtime.test.mjs +++ b/server/scripts/bundle-runtime.test.mjs @@ -2,7 +2,7 @@ import { execFile } from "node:child_process"; import { mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import path from "node:path"; -import { pathToFileURL } from "node:url"; +import { fileURLToPath, pathToFileURL } from "node:url"; import { promisify } from "node:util"; import { afterEach, beforeEach, describe, expect, it } from "vitest"; @@ -85,6 +85,33 @@ describe("bundleRuntime", () => { expect(output.entryDir).toBe(path.join(root, "server", "dist")); }); + it("keeps database caller locations public in a source-mapped runtime bundle", async () => { + const captureModule = fileURLToPath(new URL("../../packages/db/src/response-capture.ts", import.meta.url)); + write(path.join(root, "server", "dist", "services", "response-caller.js"), [ + `import { publicDatabaseCallers } from ${JSON.stringify(captureModule)};`, + 'export function callers() { return publicDatabaseCallers(new Error("private-capture-message").stack); }', + ].join("\n")); + write(path.join(root, "server", "dist", "index.js"), [ + 'import { callers } from "./services/response-caller.js";', + 'console.log(JSON.stringify({ callers: callers() }));', + ].join("\n")); + const summary = await bundleRuntime({ entry: path.join(root, "server", "dist", "index.js"), outdir: path.join(root, "server", "dist-runtime") }); + const env = { ...process.env }; + delete env.NODE_OPTIONS; + delete env.PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW; + const options = { cwd: path.join(root, "server"), env, timeout: 30_000 }; + const plain = JSON.parse((await execFileAsync(process.execPath, [summary.outfile], options)).stdout); + const mapped = JSON.parse((await execFileAsync(process.execPath, ["--enable-source-maps", summary.outfile], options)).stdout); + expect(plain.callers).toEqual([]); + expect(mapped.callers).toEqual([expect.stringMatching(/^services\/response-caller\.js:\d+:\d+$/)]); + for (const result of [plain, mapped]) { + const output = JSON.stringify(result); + expect(output).not.toContain(root); + expect(output).not.toContain(captureModule); + expect(output).not.toContain("private-capture-message"); + } + }); + it("emits a server module that loads without starting the server", async () => { const outdir = path.join(root, "server", "dist-runtime"); const summary = await bundleRuntime({ entry: path.join(root, "server", "dist", "index.js"), outdir }); From 64a8b927c1f199c612bcd36dccb6190e6e250eb3 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 17:48:47 -0700 Subject: [PATCH 03/68] fix(native): narrow committed finalization and cleanup reads (#17525) Select fresh metadata for committed finalization and cleanup. Preserve full saved evidence for new finalization, cancelled workspace retention, and legacy Stop. PostgreSQL byte-budget and unchanged-row regressions pass, full CI is green, and Greptile scored the exact head 5/5. Co-Authored-By: Paperclip --- doc/DATABASE.md | 9 +++ server/src/services/heartbeat.ts | 4 ++ .../__tests__/run-control-fixtures.ts | 2 + .../src/services/heartbeat/recovery.test.ts | 2 + server/src/services/heartbeat/recovery.ts | 4 +- .../services/heartbeat/run-control.test.ts | 6 +- server/src/services/heartbeat/run-control.ts | 22 ++++--- .../src/services/heartbeat/run-state.test.ts | 57 +++++++++++++++++++ server/src/services/heartbeat/run-state.ts | 24 ++++++++ .../__tests__/helpers/committed-replay.ts | 34 +++++++++++ .../native-run-finalizer-telemetry.test.ts | 29 ++++++---- .../native-runtime/native-run-finalizer.ts | 55 ++++++++++++------ 12 files changed, 208 insertions(+), 40 deletions(-) create mode 100644 server/src/services/native-runtime/__tests__/helpers/committed-replay.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index e6211773d87..717ec1ca0ed 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -68,6 +68,15 @@ Runner profiles retain the adapter type and complete native execution input and workspace-sync receipts. Full result evidence stays available to the existing retry and workspace checks. Stored context, profiles, and output are unchanged. +Committed native finalization reads run identity and ownership state before it +claims the coordinator. It loads complete saved input only when finalization +still needs to process an accepted result. Review presentation retains the full +result. Muse settlement reads its exact provider binding. Cleanup reads fresh +status and ownership metadata; cancelled native workspace retention and legacy +Stop settlement still load complete run evidence. Workspace settlement reads the +saved sync descriptor, export retry, and first-heartbeat flag. These projections +leave stored prompts, logs, results, and recovery state unchanged. + Conversation ownership checks select process identity and full recovery result evidence without execution prompts or profiles. Committed native reply repair selects its persisted result and identity bindings plus typed presentation flags. diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 543d96a97f8..9a18721aaa7 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -811,6 +811,8 @@ export function heartbeatService( getAgent, resolveSessionBeforeForWakeup, getRun, + getRunCleanupState, + getRunWorkspaceSettlement, getRunStatus, ensureRuntimeState, getTaskSession, @@ -1014,6 +1016,7 @@ export function heartbeatService( environmentRuntime, scheduleNativeSessionResumeDispatch, getRun, + getRunWorkspaceSettlement, executeRun, activeRunExecutionPromises, cancelHeartbeatNativeRun, @@ -1386,6 +1389,7 @@ export function heartbeatService( const runControl = createHeartbeatRunControl(db, { getRun, + getRunCleanupState, options, envOrchestrator, activeRunExecutions, diff --git a/server/src/services/heartbeat/__tests__/run-control-fixtures.ts b/server/src/services/heartbeat/__tests__/run-control-fixtures.ts index 42f6bea5150..2adce9077c1 100644 --- a/server/src/services/heartbeat/__tests__/run-control-fixtures.ts +++ b/server/src/services/heartbeat/__tests__/run-control-fixtures.ts @@ -13,6 +13,7 @@ import type { HeartbeatRunControlDependencies } from "../run-control.js"; type TestCallbacks = HeartbeatRunControlDependencies & ReturnType & { getRun: Mock; getAgent: Mock; + getRunCleanupState: Mock; setRunStatusFromLive: Mock; getSchedulingSuppression: Mock; options: { closeWarmNativeSessionsForRun: Mock> }; @@ -43,6 +44,7 @@ export function callbacks(db: Db): TestCallbacks { ...state, ...lifecycle, getRun: vi.fn(state.getRun), + getRunCleanupState: vi.fn(state.getRunCleanupState), getAgent: vi.fn(state.getAgent), setRunStatusFromLive: vi.fn(lifecycle.setRunStatusFromLive), setRunStatus: vi.fn(lifecycle.setRunStatus), diff --git a/server/src/services/heartbeat/recovery.test.ts b/server/src/services/heartbeat/recovery.test.ts index 8faa8c7e865..34139757468 100644 --- a/server/src/services/heartbeat/recovery.test.ts +++ b/server/src/services/heartbeat/recovery.test.ts @@ -7,6 +7,7 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro import { classifyEnvironmentCapabilities, environmentRuntimeService } from "../environment-runtime.js"; import { environmentService } from "../environments.js"; import { createHeartbeatRecovery, type HeartbeatRecoveryDependencies } from "./recovery.js"; +import { createHeartbeatRunState } from "./run-state.js"; import type { HotRestartIntent } from "../hot-restart.js"; import type { NativeRestartRecoveryClaim, NativeRestartRecoveryDisposition } from "../native-runtime/index.js"; import { NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE } from "../native-runtime/native-runner-ownership.js"; @@ -48,6 +49,7 @@ function callbacks(db: Db) { return { enterShutdown: vi.fn(), getRun: vi.fn(async () => null), + getRunWorkspaceSettlement: vi.fn(createHeartbeatRunState(db).getRunWorkspaceSettlement), getAgent: vi.fn(async () => null), appendRunEvent: vi.fn(async () => undefined), setRunStatusIfRunning: vi.fn(async () => ({ updated: false, run: null })), diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index 73c586bdff3..a5197485881 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -101,6 +101,7 @@ export interface HeartbeatRecoveryDependencies extends Pick["getRun"]>[1]) => Promise< Awaited["getRun"]>> | null >; + getRunWorkspaceSettlement: ReturnType["getRunWorkspaceSettlement"]; environmentRuntime: ReturnType; pluginWorkerManager?: WorkspaceWorkers; environmentsSvc: Pick, "getById" | "getLeaseById" | "releaseLease">; @@ -360,6 +361,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD environmentRuntime, scheduleNativeSessionResumeDispatch, getRun, + getRunWorkspaceSettlement, executeRun, activeRunExecutionPromises, cancelHeartbeatNativeRun, @@ -1858,7 +1860,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD agentId: string; succeeded: boolean; }) { - const settledRun = await getRun(input.runId); + const settledRun = await getRunWorkspaceSettlement(input.runId); const workspaceSyncReference = readNativeWorkspaceSyncReference( parseObject(settledRun?.runnerProfileJson).nativeWorkspaceSync, ); diff --git a/server/src/services/heartbeat/run-control.test.ts b/server/src/services/heartbeat/run-control.test.ts index 7f3991eaa0c..d0d08896538 100644 --- a/server/src/services/heartbeat/run-control.test.ts +++ b/server/src/services/heartbeat/run-control.test.ts @@ -98,7 +98,7 @@ describe("heartbeat run-control module boundary", () => { it("retains leases while native execution ownership is unverified", async () => { const database = guardedDatabase(); const deps = callbacks(database.db); - deps.getRun.mockResolvedValue({ runtimeMode: "native", status: "running", nativePhase: "terminal_failure", errorCode: "native_execution_ownership_unverified" } as Run); + deps.getRunCleanupState.mockResolvedValue({ runtimeMode: "native", status: "running", nativePhase: "terminal_failure", errorCode: "native_execution_ownership_unverified" } as Run); await createHeartbeatRunControl(database.db, deps).releaseEnvironmentLeasesForRun({ runId: "held", companyId: "company", agentId: "agent", status: "failed", providerResourceDisposition: "destroy" }); expect(database.access).not.toHaveBeenCalled(); expect(deps.options.closeWarmNativeSessionsForRun).not.toHaveBeenCalled(); @@ -108,7 +108,7 @@ describe("heartbeat run-control module boundary", () => { it("uses the persisted terminal outcome when choosing warm resource retention", async () => { const database = guardedDatabase(); const deps = callbacks(database.db); - deps.getRun.mockResolvedValue({ status: "failed", runtimeMode: "legacy" } as Run); + deps.getRunCleanupState.mockResolvedValue({ status: "failed", runtimeMode: "legacy" } as Run); await createHeartbeatRunControl(database.db, deps).releaseEnvironmentLeasesForRun({ runId: "failed", companyId: "company", agentId: "agent", status: "succeeded", providerResourceDisposition: "keep_running" }); expect(deps.envOrchestrator.releaseForRun).toHaveBeenCalledWith(expect.objectContaining({ heartbeatRunId: "failed", status: "failed", providerResourceDisposition: "stop_and_retain" })); expect(database.access).not.toHaveBeenCalled(); @@ -117,7 +117,7 @@ describe("heartbeat run-control module boundary", () => { it("defers destruction when a warm session remains busy", async () => { const database = guardedDatabase(); const deps = callbacks(database.db); - deps.getRun.mockResolvedValue({ status: "failed", runtimeMode: "legacy" } as Run); + deps.getRunCleanupState.mockResolvedValue({ status: "failed", runtimeMode: "legacy" } as Run); deps.options.closeWarmNativeSessionsForRun.mockResolvedValue({ closed: 0, busy: 1, failed: 0 }); await createHeartbeatRunControl(database.db, deps).releaseEnvironmentLeasesForRun({ runId: "busy", companyId: "company", agentId: "agent", status: "failed", providerResourceDisposition: "destroy" }); expect(deps.options.closeWarmNativeSessionsForRun).toHaveBeenCalledWith({ runId: "busy", reason: "terminal heartbeat run destroyed its environment lease" }); diff --git a/server/src/services/heartbeat/run-control.ts b/server/src/services/heartbeat/run-control.ts index 7f80f99321a..d7086c52508 100644 --- a/server/src/services/heartbeat/run-control.ts +++ b/server/src/services/heartbeat/run-control.ts @@ -164,7 +164,7 @@ async function withBudgetStopDelivery(db: Db, work: () => Promise): Promis /** Cancellation shares executor ownership and delegates lifecycle and admission effects. */ export interface HeartbeatRunControlDependencies extends Pick, - "getRun" | "getAgent">, + "getRun" | "getAgent" | "getRunCleanupState">, Pick, "mergeRunStopMetadataForAgent" | "setRunStatusFromLive" | "setRunStatus" | "setWakeupStatus" | "appendRunEvent">, Pick { @@ -324,6 +324,7 @@ function runCancellationFenceSql(input: { export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunControlDependencies) { const { getRun, + getRunCleanupState, options, envOrchestrator, activeRunExecutions, @@ -359,7 +360,11 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn"; }; }) { - const leaseOwnerRun = await getRun(input.runId); + let leaseOwnerRun = await getRunCleanupState(input.runId); + if (leaseOwnerRun && isNativeRunnerOwnershipHeld(leaseOwnerRun)) return; + const cancelledRun = leaseOwnerRun?.runtimeMode === "native" && leaseOwnerRun.status === "cancelled" + ? await getRun(input.runId) : null; + if (cancelledRun) leaseOwnerRun = cancelledRun; if (leaseOwnerRun && isNativeRunnerOwnershipHeld(leaseOwnerRun)) return; // Recovery can finish workspace copy-back outside the executor's finally. // Successful copy-back does not earn warm retention for a failed turn. @@ -368,11 +373,11 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont input.providerResourceDisposition, status, ); - if (leaseOwnerRun?.runtimeMode === "native" && status === "cancelled" - && readNativeWorkspaceSyncReference(leaseOwnerRun.runnerProfileJson?.nativeWorkspaceSync)?.state === "prepared" - && !nativeRunHasNoWorkspaceAccess(leaseOwnerRun)) { + if (cancelledRun?.runtimeMode === "native" && status === "cancelled" + && readNativeWorkspaceSyncReference(cancelledRun.runnerProfileJson?.nativeWorkspaceSync)?.state === "prepared" + && !nativeRunHasNoWorkspaceAccess(cancelledRun)) { try { - const preserved = await preserveNativeFailedWorkspace(db, leaseOwnerRun, { allowPreterminal: true }); + const preserved = await preserveNativeFailedWorkspace(db, cancelledRun, { allowPreterminal: true }); if (!preserved || !hasRequiredWorkspaceRecovery(preserved.resultJson)) return; } catch { // Keep the active source eligible for the bounded retention sweep. @@ -478,13 +483,14 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont async function acknowledgeRemoteStop(runId: string, companyId: string) { // The provider receipt arrives after adapter settlement. A remote ACP child // has no host PID, so only this target-aware boundary can acknowledge Stop. - const stopped = await getRun(runId); - if (stopped?.runtimeMode === "native") { + const state = await getRunCleanupState(runId); + if (state?.runtimeMode === "native") { const scopes = await stoppedRemoteCleanupScopes(db, companyId, runId); for (const remoteCleanupScope of scopes ?? []) { completeTerminatedRemoteNativeSessionCleanup({ companyId, runId, remoteCleanupScope }); } } + const stopped = state?.runtimeMode === "legacy" && state.status === "cancelled" ? await getRun(runId) : null; if (stopped?.runtimeMode === "legacy" && stopped.status === "cancelled" && parseObject(stopped.resultJson?.executionCancellation).state === "requested" && await runUsedConversationAdapter(db, stopped) && diff --git a/server/src/services/heartbeat/run-state.test.ts b/server/src/services/heartbeat/run-state.test.ts index 4a729328027..ccc91735df4 100644 --- a/server/src/services/heartbeat/run-state.test.ts +++ b/server/src/services/heartbeat/run-state.test.ts @@ -1,4 +1,5 @@ import { randomBytes, randomUUID } from "node:crypto"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import type { Db } from "@paperclipai/db"; import { agents, @@ -86,6 +87,9 @@ describe("heartbeat run state module", () => { expect(projectedSql(utf8.columns[0].resultJson)).toContain("pg_column_size"); await asciiState.getRun("evidence", { includeExecutionEvidence: true }); expect(projectedSql(ascii.columns[2].resultJson)).toContain("startupPreparationSettledAt"); + await asciiState.getRunWorkspaceSettlement("settlement"); + expect(projectedSql(ascii.columns[3].error)).toBe("NULL"); + expect(ascii.execute).toHaveBeenCalledTimes(1); const raw = readingDatabase("SQL_ASCII"); await extracted.createHeartbeatRunState(raw.db).getRun("raw", { unsafeFullResultJson: true }); expect(raw.execute).not.toHaveBeenCalled(); @@ -154,6 +158,59 @@ describePostgres("heartbeat run state database wiring", () => { return { companyId, agent, issue, state: extracted.createHeartbeatRunState(db) }; } + it("reads cleanup and workspace settlement without transferring saved input or changing the run", async () => { + const { companyId, agent, issue, state } = await fixture(); + const bulk = "cleanup-saved-input-sentinel🙂".repeat(40_000); + const sync = { schema: "paperclip.native-workspace-sync.v1", state: "prepared", resourceDisposition: "destroy" }; + const [run] = await db.insert(heartbeatRuns).values({ + companyId, agentId: agent.id, issueId: issue.id, runtimeMode: "native", status: "failed", + errorCode: "provider_transport_failed", error: "Retain saved files", nativePhase: "terminal_failure", + runnerProfileJson: { nativeWorkspaceSync: sync, savedInput: bulk }, + resultJson: { workspaceExportRetry: { requestId: "retry-source" }, stdout: bulk }, + contextSnapshot: { timerClaimWasFirstHeartbeat: true, savedInput: bulk }, + stdoutExcerpt: bulk, stderrExcerpt: bulk, + }).returning(); + await state.getRunWorkspaceSettlement(run.id); + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + expect(await state.getRunCleanupState(run.id)).toEqual({ + id: run.id, companyId, agentId: agent.id, runtimeMode: run.runtimeMode, + status: run.status, errorCode: run.errorCode, nativePhase: run.nativePhase, + }); + expect(await state.getRunWorkspaceSettlement(run.id)).toEqual({ + id: run.id, status: run.status, error: run.error, + runnerProfileJson: { nativeWorkspaceSync: sync }, + resultJson: { workspaceExportRetry: { requestId: "retry-source" } }, + contextSnapshot: { timerClaimWasFirstHeartbeat: true }, + }); + capture.flush(); + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBe(2); + expect(summary.responseValueBytes).toBeLessThan(2000); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toEqual(run); + expect(JSON.stringify(events)).not.toContain("cleanup-saved-input-sentinel"); + expect(await state.getRunCleanupState(randomUUID())).toBeNull(); + expect(await state.getRunWorkspaceSettlement(randomUUID())).toBeNull(); + }); + + it("preserves JSON types in workspace settlement decisions", async () => { + const { companyId, agent, state } = await fixture(); + for (const value of [null, false, true, "true", 0, [], { future: [1, "retained"] }]) { + const [run] = await db.insert(heartbeatRuns).values({ + companyId, agentId: agent.id, + runnerProfileJson: { nativeWorkspaceSync: value }, + resultJson: { workspaceExportRetry: value }, + contextSnapshot: { timerClaimWasFirstHeartbeat: value }, + }).returning({ id: heartbeatRuns.id }); + const actual = await state.getRunWorkspaceSettlement(run.id); + expect(actual?.runnerProfileJson?.nativeWorkspaceSync).toEqual(value); + expect(actual?.resultJson?.workspaceExportRetry).toEqual(value); + expect(actual?.contextSnapshot?.timerClaimWasFirstHeartbeat).toEqual(value); + } + }); + it("retains the consumed queue identity in issue polling summaries", async () => { const { companyId, agent, issue, state } = await fixture(); const [wake] = await db.insert(agentWakeupRequests).values({ diff --git a/server/src/services/heartbeat/run-state.ts b/server/src/services/heartbeat/run-state.ts index 30f9ef0b001..1911ad1a25c 100644 --- a/server/src/services/heartbeat/run-state.ts +++ b/server/src/services/heartbeat/run-state.ts @@ -1119,6 +1119,28 @@ export function createHeartbeatRunState(db: Db) { .then((rows) => rows[0] ?? null); } + async function getRunCleanupState(runId: string) { + return db.select({ + id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + runtimeMode: heartbeatRuns.runtimeMode, status: heartbeatRuns.status, + errorCode: heartbeatRuns.errorCode, nativePhase: heartbeatRuns.nativePhase, + }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).then(rows => rows[0] ?? null); + } + + async function getRunWorkspaceSettlement(runId: string) { + const safeForLegacyEncoding = await hasUnsafeTextProjectionDatabase(); + return db.select({ + id: heartbeatRuns.id, status: heartbeatRuns.status, + error: safeForLegacyEncoding ? heartbeatRunSqlAsciiSafeColumns.error : heartbeatRuns.error, + runnerProfileJson: sql | null>`case when ${heartbeatRuns.runnerProfileJson} is null then null + else jsonb_build_object('nativeWorkspaceSync', ${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync') end`, + resultJson: sql | null>`case when ${heartbeatRuns.resultJson} is null then null + else jsonb_build_object('workspaceExportRetry', ${heartbeatRuns.resultJson}->'workspaceExportRetry') end`, + contextSnapshot: sql | null>`case when ${heartbeatRuns.contextSnapshot} is null then null + else jsonb_build_object('timerClaimWasFirstHeartbeat', ${heartbeatRuns.contextSnapshot}->'timerClaimWasFirstHeartbeat') end`, + }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).then(rows => rows[0] ?? null); + } + async function getRunStatus(runId: string) { return db .select({ @@ -1688,6 +1710,8 @@ export function createHeartbeatRunState(db: Db) { getAgent, resolveSessionBeforeForWakeup, getRun, + getRunCleanupState, + getRunWorkspaceSettlement, ensureRuntimeState, getTaskSession, getLatestAgentConfigRevision, diff --git a/server/src/services/native-runtime/__tests__/helpers/committed-replay.ts b/server/src/services/native-runtime/__tests__/helpers/committed-replay.ts new file mode 100644 index 00000000000..7caef7cf216 --- /dev/null +++ b/server/src/services/native-runtime/__tests__/helpers/committed-replay.ts @@ -0,0 +1,34 @@ +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { eq } from "drizzle-orm"; +import { expect } from "vitest"; + +export async function verifyCommittedReplayPayloadReads( + db: Db, + binding: { runId: string; companyId: string; agentId: string }, + replay: () => Promise, +) { + const { runId, companyId, agentId } = binding; + const bulk = "saved-native-input-sentinel🙂".repeat(40_000); + const [original] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + await db.update(heartbeatRuns).set({ + contextSnapshot: { ...original.contextSnapshot, savedInput: bulk }, + runnerProfileJson: { savedInput: bulk, nativeExecutionInput: { + schema: "paperclip.native-execution-input.v8", savedInput: bulk, + provider: { kind: "muse", binding: { companyId, agentId, bindingId: runId } }, + } }, stdoutExcerpt: bulk, stderrExcerpt: bulk, + }).where(eq(heartbeatRuns.id, runId)); + const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { await replay(); } + finally { capture.flush(); } + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBeGreaterThan(0); + expect(summary.responseValueBytes).toBeLessThan(100_000); + const [after] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + expect(after).toEqual(before); + expect(JSON.stringify(events)).not.toContain("saved-native-input-sentinel"); +} diff --git a/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts b/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts index 0a8b6ce4ed1..2c2e534081b 100644 --- a/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts +++ b/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts @@ -1,5 +1,6 @@ import { enableRunFailureReporting } from "../../__tests__/helpers/run-failure-reporting.js"; import { randomUUID } from "node:crypto"; +import { verifyCommittedReplayPayloadReads } from "./__tests__/helpers/committed-replay.js"; import { eq } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { @@ -34,6 +35,12 @@ const mockTelemetryClient = vi.hoisted(() => ({ })); vi.mock("../../telemetry.ts", () => ({ getTelemetryClient: () => mockTelemetryClient })); +const reconcileMuseAssignments = vi.hoisted(() => vi.fn(async () => undefined)); +vi.mock("../muse-runner-broker.js", async importOriginal => ({ + ...await importOriginal(), + museRunnerBroker: () => ({ reconcileTerminalAssignments: reconcileMuseAssignments }), +})); + const mockCaptureRunFailure = vi.hoisted(() => vi.fn()); vi.mock("../../sentry.ts", async () => { const actual = await vi.importActual("../../sentry.ts"); @@ -261,18 +268,18 @@ describeEmbeddedPostgres("native run finalizer / status decision committer — a projectRunStatus: true, }); - const callsBefore = mockTelemetryClient.track.mock.calls.length; - // The coordinator is already "committed", so this second call takes the - // projectCommittedRun short-circuit. A succeeded row is eligible so stale - // cleanup errors can be projected separately; an unchanged terminal state - // must still never emit a second agent.task_run event. - await finalizeNativeRun({ - db, - runId: fixture.runId, - workspaceFinalizeStatus: "succeeded", - projectRunStatus: true, + await verifyCommittedReplayPayloadReads(db, { runId: fixture.runId, companyId, agentId }, async () => { + const callsBefore = mockTelemetryClient.track.mock.calls.length; + reconcileMuseAssignments.mockClear(); + await finalizeNativeRun({ + db, + runId: fixture.runId, + workspaceFinalizeStatus: "succeeded", + projectRunStatus: true, + }); + expect(agentTaskRunCalls(callsBefore)).toHaveLength(0); + expect(reconcileMuseAssignments).toHaveBeenCalledExactlyOnceWith(companyId, agentId, fixture.runId); }); - expect(agentTaskRunCalls(callsBefore)).toHaveLength(0); }); it("repairs a stale succeeded-run error once without duplicate telemetry or destroying retained owner evidence", async () => { diff --git a/server/src/services/native-runtime/native-run-finalizer.ts b/server/src/services/native-runtime/native-run-finalizer.ts index 8b2ae73c349..7d60d73e0d7 100644 --- a/server/src/services/native-runtime/native-run-finalizer.ts +++ b/server/src/services/native-runtime/native-run-finalizer.ts @@ -568,7 +568,7 @@ export async function recordNativeFinalizationFailure(input: { }); } -async function resolveCommittedFinalizationRecovery(db: Db, run: typeof heartbeatRuns.$inferSelect, issueId: string) { +async function resolveCommittedFinalizationRecovery(db: Db, run: Pick, issueId: string) { const actions = await db.select().from(issueRecoveryActions).where(and( eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, issueId), eq(issueRecoveryActions.kind, "active_run_watchdog"), inArray(issueRecoveryActions.status, ["active", "escalated"]), @@ -704,7 +704,8 @@ async function projectCommittedRun(input: { async function materializeCommittedReviewResponse(db: Db, runId: string) { try { const [run] = await db - .select() + .select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + nativeIssueId: heartbeatRuns.nativeIssueId, resultJson: heartbeatRuns.resultJson }) .from(heartbeatRuns) .where(eq(heartbeatRuns.id, runId)) .limit(1); @@ -1146,13 +1147,15 @@ export async function finalizeNativeRun(input: { preserveProviderAttempt?: boolean; failpoint?: NativeStatusCommitFailpoint; }) { - const run = await input.db - .select() + const admittedRun = await input.db + .select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + runtimeMode: heartbeatRuns.runtimeMode, status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode, + nativePhase: heartbeatRuns.nativePhase, finishedAt: heartbeatRuns.finishedAt }) .from(heartbeatRuns) .where(eq(heartbeatRuns.id, input.runId)) .limit(1) .then((rows) => rows[0] ?? null); - if (!run || run.runtimeMode !== "native") + if (!admittedRun || admittedRun.runtimeMode !== "native") throw new Error("native_finalization_run_missing"); const claim = await claimCoordinator({ db: input.db, @@ -1161,12 +1164,12 @@ export async function finalizeNativeRun(input: { }); const coordinator = claim.coordinator; if (!claim.leaseOwner && coordinator.phase === "committed") { - if (isNativeRunnerOwnershipHeld(run)) return coordinator; + if (isNativeRunnerOwnershipHeld(admittedRun)) return coordinator; const presentationAlreadyMaterialized = coordinator.decisionId ? await hasMaterializedNativeReviewResponse(input.db, { - companyId: run.companyId, + companyId: admittedRun.companyId, issueId: coordinator.issueId, - runId: run.id, + runId: admittedRun.id, decisionId: coordinator.decisionId, }) : false; @@ -1178,9 +1181,9 @@ export async function finalizeNativeRun(input: { ) { await input.db.transaction((tx) => restoreNativeChatReviewPresentationInTransaction(tx as unknown as Db, { - companyId: run.companyId, + companyId: admittedRun.companyId, issueId: coordinator.issueId, - runId: run.id, + runId: admittedRun.id, decisionId: coordinator.decisionId!, resultId: coordinator.resultId!, assessmentId: coordinator.assessmentId!, @@ -1188,27 +1191,45 @@ export async function finalizeNativeRun(input: { ); } if (input.projectRunStatus) - await projectCommittedRun({ db: input.db, run, coordinator }); + await projectCommittedRun({ db: input.db, run: admittedRun, coordinator }); if (input.projectRunStatus && !presentationAlreadyMaterialized) await materializeCommittedReviewResponse(input.db, input.runId); if (input.projectRunStatus) await repairCommittedNativeChatResponse(input.db, { - companyId: run.companyId, + companyId: admittedRun.companyId, issueId: coordinator.issueId, - runId: run.id, + runId: admittedRun.id, }); if (input.projectRunStatus) { - await settleExternalConversation(input.db, run.companyId, coordinator.issueId).catch((err) => { - logger.warn({ err, runId: run.id }, "Slack conversation settlement deferred to reconciliation"); + await settleExternalConversation(input.db, admittedRun.companyId, coordinator.issueId).catch((err) => { + logger.warn({ err, runId: admittedRun.id }, "Slack conversation settlement deferred to reconciliation"); }); } - await resolveCommittedFinalizationRecovery(input.db, run, coordinator.issueId); + await resolveCommittedFinalizationRecovery(input.db, admittedRun, coordinator.issueId); if (input.projectRunStatus) { - const [projectedRun] = await input.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + const [projectedRun] = await input.db.select({ + id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + runtimeMode: heartbeatRuns.runtimeMode, status: heartbeatRuns.status, + runnerProfileJson: sql | null>`case when ${heartbeatRuns.runnerProfileJson} is null then null + else jsonb_build_object('nativeExecutionInput', jsonb_build_object( + 'schema', ${heartbeatRuns.runnerProfileJson} #> '{nativeExecutionInput,schema}', + 'provider', jsonb_build_object( + 'kind', ${heartbeatRuns.runnerProfileJson} #> '{nativeExecutionInput,provider,kind}', + 'binding', jsonb_build_object( + 'companyId', ${heartbeatRuns.runnerProfileJson} #> '{nativeExecutionInput,provider,binding,companyId}', + 'agentId', ${heartbeatRuns.runnerProfileJson} #> '{nativeExecutionInput,provider,binding,agentId}', + 'bindingId', ${heartbeatRuns.runnerProfileJson} #> '{nativeExecutionInput,provider,binding,bindingId}' + ) + ) + )) end`, + }).from(heartbeatRuns).where(eq(heartbeatRuns.id, admittedRun.id)); if (projectedRun) await reconcileMuseNativeTerminalRun(input.db, projectedRun); } return coordinator; } + const run = await input.db.select().from(heartbeatRuns) + .where(eq(heartbeatRuns.id, input.runId)).limit(1).then(rows => rows[0] ?? null); + if (!run || run.runtimeMode !== "native") throw new Error("native_finalization_run_missing"); const [resultRow, contractRow] = await Promise.all([ input.db .select() From e82d86bfebe34db8757036316b8997162aaa5b10 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 24 Oct 2026 19:01:15 -0700 Subject: [PATCH 04/68] fix(server): gate periodic heartbeat recovery to one chain in flight (#11699) Bind periodic recovery admission to lifecycle tracking. Keep one physical pass active through cleanup, report skipped ticks and long-held ownership, and preserve direct scheduled delivery. Full build, typecheck, 32 focused tests, complete CI, and exact-head Greptile5/5 pass. Co-Authored-By: Senior Engineer Fable Co-Authored-By: Dotta Co-Authored-By: Devin Foley Co-Authored-By: Claude Fable 5 Co-Authored-By: Paperclip --- doc/native-workspace-finalization-recovery.md | 10 ++ server/src/index.ts | 13 +- server/src/lib/single-flight.test.ts | 163 ++++++++++++++++++ server/src/lib/single-flight.ts | 51 ++++++ 4 files changed, 236 insertions(+), 1 deletion(-) create mode 100644 server/src/lib/single-flight.test.ts create mode 100644 server/src/lib/single-flight.ts diff --git a/doc/native-workspace-finalization-recovery.md b/doc/native-workspace-finalization-recovery.md index 8b9c004282c..428c32b3fe4 100644 --- a/doc/native-workspace-finalization-recovery.md +++ b/doc/native-workspace-finalization-recovery.md @@ -1,5 +1,15 @@ # Native workspace finalization ownership and recovery +Periodic heartbeat recovery runs one pass at a time in each server process. +A tick that arrives during a slow pass reports its age and starts no duplicate. +A later tick can start a fresh pass after the current work settles. Shutdown +and idle sleep still track the actual recovery promise through cleanup. Direct +run operations and scheduled work delivery keep their existing paths. A pass +that remains active for three scheduler intervals produces an error log with its +age. A waiting timeout does not admit another pass: the actual work must settle +before ownership is released. A stuck operation requires investigation and +recovery at the operation or provider boundary. + A native provider can fail before it submits a result while its workspace still contains unpublished edits. A prepared workspace reference is not an export receipt. Terminal provider failure pins the exact bound sandbox and records a diff --git a/server/src/index.ts b/server/src/index.ts index 584e9e67697..e83d603e2aa 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -38,6 +38,7 @@ import { chatCompletionDeliveryService } from "./services/chat-completion-delive // instrumentationReady before opening DB connections or constructing the // HTTP server, so trace coverage does not depend on incidental timing. import { instrumentationReady, shutdownInstrumentation } from "./instrumentation.js"; +import { createTrackedSingleFlight } from "./lib/single-flight.js"; import { sentryReady, shutdownSentry, captureException } from "./sentry.js"; import { setSentryRunLinkResolver } from "./sentry-links.js"; import { createSentryRunLinkResolver } from "./services/sentry-run-links.js"; @@ -1209,6 +1210,16 @@ async function startServerWithDatabaseTeardown( }); heartbeatSchedulerInFlight.add(tracked); }; + const runPeriodicRecovery = createTrackedSingleFlight({ + trackWork: trackHeartbeatSchedulerWork, + onSkipped: (inFlightForMs) => { + if (inFlightForMs >= 3 * config.heartbeatSchedulerIntervalMs) { + logger.error({ inFlightForMs }, "periodic heartbeat recovery remains in flight; duplicate pass suppressed"); + } else { + logger.warn({ inFlightForMs }, "skipped periodic heartbeat recovery tick; previous chain still in flight"); + } + }, + }); const waitForHeartbeatSchedulerIdle = async () => { while (heartbeatSchedulerInFlight.size > 0) { await Promise.allSettled([...heartbeatSchedulerInFlight]); @@ -1887,7 +1898,7 @@ async function startServerWithDatabaseTeardown( if (!(await heartbeat.resolveSchedulingSuppression()).suppressed) { // Periodically reap orphaned runs (5-min staleness threshold) and make sure // persisted queued work is still being driven forward. - trackHeartbeatSchedulerWork(heartbeat + runPeriodicRecovery(() => heartbeat .reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000 }) .then(() => heartbeat.promoteDueScheduledRetries()) .then(async (promotion) => { diff --git a/server/src/lib/single-flight.test.ts b/server/src/lib/single-flight.test.ts new file mode 100644 index 00000000000..06dabc92485 --- /dev/null +++ b/server/src/lib/single-flight.test.ts @@ -0,0 +1,163 @@ +import { describe, expect, it, vi } from "vitest"; +import { createSingleFlight, createTrackedSingleFlight } from "./single-flight.js"; +import { beginIdleTrackedWork, idleWorkSnapshot } from "../services/task-admission.js"; + +function deferred() { + let resolve!: () => void; + let reject!: (err: unknown) => void; + const promise = new Promise((res, rej) => { + resolve = res; + reject = rej; + }); + return { promise, resolve, reject }; +} + +describe("createSingleFlight", () => { + it("runs work when idle and reports busy until it settles", async () => { + const gate = createSingleFlight(); + const gap = deferred(); + + const first = gate.run(() => gap.promise); + expect(first.started).toBe(true); + expect(gate.busy).toBe(true); + + gap.resolve(); + await (first as { settled: Promise }).settled; + expect(gate.busy).toBe(false); + }); + + it("skips a tick while the previous invocation is still in flight", async () => { + const clock = { nowMs: 1_000 }; + const gate = createSingleFlight(() => clock.nowMs); + const gap = deferred(); + const work = vi.fn(() => gap.promise); + + const first = gate.run(work); + clock.nowMs = 31_000; + const second = gate.run(work); + + expect(second).toEqual({ started: false, inFlightForMs: 30_000 }); + expect(work).toHaveBeenCalledTimes(1); + + gap.resolve(); + await (first as { settled: Promise }).settled; + }); + + it("admits new work after the previous invocation resolves", async () => { + const gate = createSingleFlight(); + const gap = deferred(); + + const first = gate.run(() => gap.promise); + gap.resolve(); + await (first as { settled: Promise }).settled; + + const second = gate.run(async () => {}); + expect(second.started).toBe(true); + await (second as { settled: Promise }).settled; + }); + + it("releases the gate when work rejects and never rejects the settled promise", async () => { + const gate = createSingleFlight(); + const gap = deferred(); + + const first = gate.run(() => gap.promise); + gap.reject(new Error("chain failed")); + await expect((first as { settled: Promise }).settled).resolves.toBeUndefined(); + expect(gate.busy).toBe(false); + + const second = gate.run(async () => {}); + expect(second.started).toBe(true); + await (second as { settled: Promise }).settled; + }); + + it("propagates a synchronous throw and leaves the gate free", async () => { + const gate = createSingleFlight(); + + expect(() => + gate.run(() => { + throw new Error("sync failure"); + }), + ).toThrow("sync failure"); + expect(gate.busy).toBe(false); + + const second = gate.run(async () => {}); + expect(second.started).toBe(true); + await (second as { settled: Promise }).settled; + }); + + it("keeps the owned pass active after a caller stops waiting", async () => { + const gate = createSingleFlight(), work = deferred(), observer = deferred(); + const first = gate.run(() => work.promise); + if (!first.started) throw new Error("initial pass was not admitted"); + const stopped = Promise.race([first.settled, observer.promise]); + observer.reject(new Error("observer stopped")); + await expect(stopped).rejects.toThrow("observer stopped"); + const duplicate = vi.fn(async () => undefined); + expect(gate.run(duplicate).started).toBe(false); + expect(duplicate).not.toHaveBeenCalled(); + expect(gate.busy).toBe(true); + work.resolve(); + await first.settled; + expect(gate.busy).toBe(false); + }); + + it("keeps cleanup joined after the main operation completes", async () => { + const gate = createSingleFlight(), operation = deferred(), cleanup = deferred(); + const first = gate.run(() => operation.promise.finally(() => cleanup.promise)); + if (!first.started) throw new Error("initial pass was not admitted"); + operation.resolve(); + await Promise.resolve(); + expect(gate.busy).toBe(true); + const duplicate = vi.fn(async () => undefined); + expect(gate.run(duplicate).started).toBe(false); + expect(duplicate).not.toHaveBeenCalled(); + cleanup.resolve(); + await first.settled; + expect(gate.busy).toBe(false); + const next = gate.run(duplicate); + expect(next.started).toBe(true); + if (next.started) await next.settled; + expect(duplicate).toHaveBeenCalledTimes(1); + }); + + it("tracks one periodic pass through scheduler shutdown and idle sleep", async () => { + vi.useFakeTimers(); + const before = idleWorkSnapshot().active; + const operation = deferred(), cleanup = deferred(); + const tracked: Promise[] = []; + const trackWork = vi.fn((work: Promise) => { + const finish = beginIdleTrackedWork(); + tracked.push(work.finally(finish)); + }); + const onSkipped = vi.fn(); + const tick = createTrackedSingleFlight({ trackWork, onSkipped }); + const heartbeat = vi.fn(() => operation.promise.finally(() => cleanup.promise)); + const timer = setInterval(() => tick(heartbeat), 30_000); + try { + await vi.advanceTimersByTimeAsync(60_000); + expect(heartbeat).toHaveBeenCalledOnce(); + expect(trackWork).toHaveBeenCalledOnce(); + expect(onSkipped).toHaveBeenCalledExactlyOnceWith(30_000); + clearInterval(timer); + let shutdownComplete = false; + const shutdown = Promise.all(tracked).then(() => { shutdownComplete = true; }); + await Promise.resolve(); + expect(shutdownComplete).toBe(false); + expect(idleWorkSnapshot().active).toBe(before + 1); + operation.resolve(); + await Promise.resolve(); + expect(shutdownComplete).toBe(false); + expect(idleWorkSnapshot().active).toBe(before + 1); + cleanup.resolve(); + await shutdown; + expect(shutdownComplete).toBe(true); + expect(idleWorkSnapshot().active).toBe(before); + } finally { + clearInterval(timer); + operation.resolve(); + cleanup.resolve(); + await Promise.all(tracked); + vi.useRealTimers(); + } + }); +}); diff --git a/server/src/lib/single-flight.ts b/server/src/lib/single-flight.ts new file mode 100644 index 00000000000..68dc1b035f4 --- /dev/null +++ b/server/src/lib/single-flight.ts @@ -0,0 +1,51 @@ +export interface SingleFlightSkip { + started: false; + inFlightForMs: number; +} + +export interface SingleFlightStart { + started: true; + settled: Promise; +} + +export type SingleFlightResult = SingleFlightSkip | SingleFlightStart; + +export function createSingleFlight(now: () => number = Date.now) { + let inFlight: Promise | null = null; + let startedAtMs = 0; + + return { + get busy(): boolean { + return inFlight !== null; + }, + run(work: () => Promise): SingleFlightResult { + if (inFlight) { + return { started: false, inFlightForMs: now() - startedAtMs }; + } + startedAtMs = now(); + const settled = Promise.resolve(work()) + .then( + () => undefined, + () => undefined, + ) + .finally(() => { + inFlight = null; + }); + inFlight = settled; + return { started: true, settled }; + }, + }; +} + +export function createTrackedSingleFlight(options: { + trackWork: (work: Promise) => void; + onSkipped: (inFlightForMs: number) => void; + now?: () => number; +}) { + const gate = createSingleFlight(options.now); + return (work: () => Promise) => { + const tick = gate.run(work); + if (tick.started) options.trackWork(tick.settled); + else options.onSkipped(tick.inFlightForMs); + }; +} From 39e5c8c1c32ee38d7b40d068560e63b2e93320be Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 19:01:19 -0700 Subject: [PATCH 05/68] perf(server): narrow committed chat authorization reads (#17530) Select current chat authorization fields for committed responses under the existing scoped share lock. Preserve complete evidence for actual retries, source revocation, JSON types, and saved run data. Full build, typecheck, 113 focused tests, complete CI, and exact-head Greptile5/5 pass. Co-Authored-By: Paperclip --- doc/native-workspace-finalization-recovery.md | 6 + .../authorization-run-read.test.ts | 116 ++++++++++++++++++ .../chat-channels/authorization-run-read.ts | 46 +++++++ .../chat-channels/failed-run-retry.ts | 20 +-- 4 files changed, 174 insertions(+), 14 deletions(-) create mode 100644 server/src/services/chat-channels/authorization-run-read.test.ts create mode 100644 server/src/services/chat-channels/authorization-run-read.ts diff --git a/doc/native-workspace-finalization-recovery.md b/doc/native-workspace-finalization-recovery.md index 428c32b3fe4..c888ce1a34f 100644 --- a/doc/native-workspace-finalization-recovery.md +++ b/doc/native-workspace-finalization-recovery.md @@ -475,3 +475,9 @@ For a cancelled native run, the file-recovery form also offers an optional confirmation that no external actions were performed. Leave it unchecked unless the run history verifies that fact. The default records files without certifying action outcomes. Changing the selected source clears the evidence and confirmations. + +Committed chat response authorization reads current run identity, completion +binding, and chat source fields under the existing nonblocking share lock. +It excludes saved execution input, runner profiles, and log excerpts. Actual +retry admission still reads the complete run and its recovery evidence. These +reads preserve stored run data, source revocation checks, and JSON value types. diff --git a/server/src/services/chat-channels/authorization-run-read.test.ts b/server/src/services/chat-channels/authorization-run-read.test.ts new file mode 100644 index 00000000000..3d56b695222 --- /dev/null +++ b/server/src/services/chat-channels/authorization-run-read.test.ts @@ -0,0 +1,116 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { agents, companies, createDb, heartbeatRuns } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { readChatAuthorizationRun } from "./authorization-run-read.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("chat authorization run reads", () => { + let database: Awaited>; + let db: ReturnType; + let companyId: string; + let agentId: string; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("chat-authorization-read-"); + db = createDb(database.connectionString); + const [company] = await db.insert(companies).values({ name: "Chat authorization fixture", issuePrefix: "CAR" }).returning(); + const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Chat agent", adapterType: "process" }).returning(); + companyId = company.id; + agentId = agent.id; + }, 60_000); + + afterAll(async () => { + await db?.$client.end(); + await database?.cleanup(); + }); + + async function seed(contextSnapshot: unknown = {}) { + const [run] = await db.insert(heartbeatRuns).values({ + companyId, agentId, contextSnapshot: contextSnapshot as Record | null, + }).returning(); + return { companyId, agentId, failedRunId: run.id }; + } + + it("reads committed authority below the byte budget and keeps complete retry evidence", async () => { + const bulk = randomBytes(128 * 1024).toString("base64"); + const context = { + source: "chat:slack", issueId: randomUUID(), interactionId: null, taskKey: "CAR-1", + wakeCommentIds: [randomUUID()], wakeCommentId: null, + chatFailedRunRetry: { version: 1, actionId: randomUUID(), failedRunId: randomUUID() }, + retryOfRunId: randomUUID(), + }; + const scope = await seed({ ...context, savedInput: bulk }); + await db.update(heartbeatRuns).set({ runnerProfileJson: { savedInput: bulk }, stdoutExcerpt: bulk, stderrExcerpt: bulk }) + .where(eq(heartbeatRuns.id, scope.failedRunId)); + const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, scope.failedRunId)); + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { + const selected = await db.transaction(tx => readChatAuthorizationRun(tx, scope, true)); + expect(selected.kind).toBe("committed"); + expect(selected.run?.contextSnapshot).toEqual(context); + expect(selected.run).not.toHaveProperty("runnerProfileJson"); + expect(selected.run).not.toHaveProperty("stdoutExcerpt"); + } finally { capture.flush(); } + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBeGreaterThan(0); + expect(summary.responseValueBytes).toBeLessThan(4096); + const retry = await db.transaction(tx => readChatAuthorizationRun(tx, scope, false)); + expect(retry.kind).toBe("retry"); + expect(retry.run).toEqual(before); + const [after] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, scope.failedRunId)); + expect(after).toEqual(before); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + }); + + it.each([{}, { wakeCommentId: null }, { wakeCommentId: false, wakeCommentIds: "invalid", interactionId: [] }])( + "preserves key presence and JSON types in %j", async context => { + const scope = await seed(context); + expect((await readChatAuthorizationRun(db, scope, true)).run?.contextSnapshot).toEqual(context); + }, + ); + + it.each([null, [], "invalid", 42])("handles non-object context %j", async context => { + const scope = await seed(context); + expect((await readChatAuthorizationRun(db, scope, true)).run?.contextSnapshot).toEqual(context === null ? null : {}); + expect((await readChatAuthorizationRun(db, scope, false)).run?.contextSnapshot).toEqual(context); + }); + + it("reads current authority and fails closed for another company or agent", async () => { + const scope = await seed({ source: "chat:slack", taskKey: "CAR-1" }); + expect((await readChatAuthorizationRun(db, scope, true)).run?.contextSnapshot).toEqual({ source: "chat:slack", taskKey: "CAR-1" }); + await db.update(heartbeatRuns).set({ contextSnapshot: { source: "chat:discord", taskKey: "CAR-2" } }) + .where(eq(heartbeatRuns.id, scope.failedRunId)); + expect((await readChatAuthorizationRun(db, scope, true)).run?.contextSnapshot).toEqual({ source: "chat:discord", taskKey: "CAR-2" }); + for (const committed of [true, false]) { + expect((await readChatAuthorizationRun(db, { ...scope, companyId: randomUUID() }, committed)).run).toBeUndefined(); + expect((await readChatAuthorizationRun(db, { ...scope, agentId: randomUUID() }, committed)).run).toBeUndefined(); + expect((await readChatAuthorizationRun(db, { ...scope, failedRunId: randomUUID() }, committed)).run).toBeUndefined(); + } + }); + + it("preserves nonblocking authorization row locks", async () => { + const scope = await seed(); + let acquired!: () => void, release!: () => void; + const ready = new Promise(resolve => { acquired = resolve; }); + const released = new Promise(resolve => { release = resolve; }); + const holder = db.transaction(async tx => { + await tx.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, scope.failedRunId)); + acquired(); + await released; + }); + try { + await ready; + for (const committed of [true, false]) { + await expect(db.transaction(tx => readChatAuthorizationRun(tx, scope, committed))).rejects.toMatchObject({ cause: { code: "55P03" } }); + } + } finally { release(); await holder; } + expect((await readChatAuthorizationRun(db, scope, true)).run?.status).toBe("running"); + }); +}); diff --git a/server/src/services/chat-channels/authorization-run-read.ts b/server/src/services/chat-channels/authorization-run-read.ts new file mode 100644 index 00000000000..3390ba103b7 --- /dev/null +++ b/server/src/services/chat-channels/authorization-run-read.ts @@ -0,0 +1,46 @@ +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { and, eq, sql } from "drizzle-orm"; + +type Database = Db | Parameters[0]>[0]; +type Scope = { companyId: string; agentId: string; failedRunId: string }; + +const contextKeys = [ + "source", "issueId", "interactionId", "taskKey", "wakeCommentIds", + "wakeCommentId", "chatFailedRunRetry", "retryOfRunId", +] as const; + +const committedContext = sql | null>` + case when ${heartbeatRuns.contextSnapshot} is null then null else ( + select coalesce(jsonb_object_agg(field.key, field.value), '{}'::jsonb) + from jsonb_each(case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then ${heartbeatRuns.contextSnapshot} else '{}'::jsonb end) as field(key, value) + where field.key in (${sql.join(contextKeys.map(key => sql`${key}`), sql`, `)}) + ) end +`; + +export async function readChatAuthorizationRun(db: Database, scope: Scope, committed: boolean) { + const where = and( + eq(heartbeatRuns.id, scope.failedRunId), + eq(heartbeatRuns.companyId, scope.companyId), + eq(heartbeatRuns.agentId, scope.agentId), + ); + if (!committed) { + const [run] = await db.select().from(heartbeatRuns).where(where).for("share", { noWait: true }); + return { kind: "retry" as const, run }; + } + const [run] = await db.select({ + id: heartbeatRuns.id, + companyId: heartbeatRuns.companyId, + agentId: heartbeatRuns.agentId, + wakeupRequestId: heartbeatRuns.wakeupRequestId, + status: heartbeatRuns.status, + finishedAt: heartbeatRuns.finishedAt, + runtimeMode: heartbeatRuns.runtimeMode, + errorCode: heartbeatRuns.errorCode, + nativeIssueId: heartbeatRuns.nativeIssueId, + completionContractId: heartbeatRuns.completionContractId, + retryOfRunId: heartbeatRuns.retryOfRunId, + contextSnapshot: committedContext, + }).from(heartbeatRuns).where(where).for("share", { noWait: true }); + return { kind: "committed" as const, run }; +} diff --git a/server/src/services/chat-channels/failed-run-retry.ts b/server/src/services/chat-channels/failed-run-retry.ts index 2b5d2c9b026..6310d448a66 100644 --- a/server/src/services/chat-channels/failed-run-retry.ts +++ b/server/src/services/chat-channels/failed-run-retry.ts @@ -2,6 +2,7 @@ import { continuingTeamsFilePublication, type ContinuingTeamsFileTransfer } from import { readTaskChatRetryIntent, taskChatRetryCanAwaitRecovery, taskChatRetryExecutionStopped } from "./task-retry-authority.js"; import { notifyChatActionWork, notifyChatPublicationWork } from "../chat-work-notifications.js"; import { FAILED_RUN_RETRY_STALE_MS, dueChatAction } from "../chat-action-work.js"; +import { readChatAuthorizationRun } from "./authorization-run-read.js"; import { createHash, @@ -688,17 +689,8 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti ), ) .for("share", { noWait: true }); - const [failedRun] = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, input.failedRunId), - eq(heartbeatRuns.companyId, input.companyId), - eq(heartbeatRuns.agentId, input.agentId), - ), - ) - .for("share", { noWait: true }); + const evidence = await readChatAuthorizationRun(tx, input, Boolean(input.committedResponse)); + const failedRun = evidence.run; if ( !issue || issue.assigneeAgentId !== input.agentId || @@ -719,7 +711,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti "process_exit", "timeout", ].includes(failedRun.errorCode ?? "") || - (await isPreProviderReviewedChatFailure(tx, failedRun)) + (evidence.kind === "retry" && await isPreProviderReviewedChatFailure(tx, evidence.run)) )) || (!input.publication && ["backlog", "done", "cancelled"].includes(issue.status)) @@ -729,8 +721,8 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti companyId: input.companyId, issueId: issue.id, agentId: input.agentId, sourceRunId: failedRun.id, intentId: input.taskRetryIntentId, })) throw failedChatRetryDenied(); - if (failedRun.runtimeMode === "native" && !input.committedResponse && !input.taskRetryIntentId) - await failedNativeRetryCoordinator(tx, failedRun, issue.id); + if (failedRun.runtimeMode === "native" && evidence.kind === "retry" && !input.taskRetryIntentId) + await failedNativeRetryCoordinator(tx, evidence.run, issue.id); const context = failedRun.contextSnapshot ?? {}; const provider = CHAT_PROVIDERS.find( (value) => context.source === `chat:${value}`, From ef0a8db50252c23ca4a1326a27bbb9441c3bc52c Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 19:01:23 -0700 Subject: [PATCH 06/68] perf(native): narrow reconciliation candidate payloads (#17531) Select the complete workspace-sync record and native status wake identifier for reconciliation candidates. Preserve candidate policy, JSON types, full later evidence, and stored data. The committed replay regression includes the real reconciliation scan. Full build, typecheck, 114 focused tests, complete CI, and exact-head Greptile5/5 pass. Co-Authored-By: Paperclip --- doc/native-workspace-finalization-recovery.md | 5 +++++ .../native-runtime/native-finalization-reconciler.ts | 11 +++++------ .../native-run-finalizer-telemetry.test.ts | 5 +++++ 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/doc/native-workspace-finalization-recovery.md b/doc/native-workspace-finalization-recovery.md index c888ce1a34f..3d16ff3aada 100644 --- a/doc/native-workspace-finalization-recovery.md +++ b/doc/native-workspace-finalization-recovery.md @@ -19,6 +19,11 @@ cleanup run. This includes descriptor reads that fail inside runtime selection. preparation failure; another owner or a stale claim prevents cleanup. Settling that claim and recording file retention share a transaction. +The reconciliation candidate scan selects the complete workspace-sync record and +the native status wake identifier from each saved run. It keeps their JSON types +and reads full evidence later where finalization or recovery needs it. Saved +execution input, runner profiles, and logs stay in the stored run. + The recovery card names the original run and requires evidence that its exact allocation stopped and its files were recovered. Recording that evidence does not retry the old turn, certify unknown external actions, change the current task diff --git a/server/src/services/native-runtime/native-finalization-reconciler.ts b/server/src/services/native-runtime/native-finalization-reconciler.ts index ec391457b07..283aac76b2d 100644 --- a/server/src/services/native-runtime/native-finalization-reconciler.ts +++ b/server/src/services/native-runtime/native-finalization-reconciler.ts @@ -742,8 +742,8 @@ async function reconcileNativeFinalizationsPass( resultId: nativeRunFinalizations.resultId, assessmentId: nativeRunFinalizations.assessmentId, decisionId: nativeRunFinalizations.decisionId, - runnerProfileJson: heartbeatRuns.runnerProfileJson, - contextSnapshot: heartbeatRuns.contextSnapshot, + nativeWorkspaceSync: sql`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'`, + nativeStatusWakeIntentId: sql`${heartbeatRuns.contextSnapshot}->'nativeStatusWakeIntentId'`, }) .from(heartbeatRuns) .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)) @@ -955,8 +955,7 @@ async function reconcileNativeFinalizationsPass( if (reviewBarrier && reviewBarrier.status !== "succeeded") continue; const childCompletionRecipient = { companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, runId: row.runId, - sourceIntentId: typeof record(row.contextSnapshot).nativeStatusWakeIntentId === "string" - ? record(row.contextSnapshot).nativeStatusWakeIntentId as string : null, + sourceIntentId: typeof row.nativeStatusWakeIntentId === "string" ? row.nativeStatusWakeIntentId : null, }; const hasPendingChildCompletion = await hasPendingNativeChildCompletion(db, childCompletionRecipient); const reassessmentRow = await recordNativeWorkAssessment({ @@ -1095,7 +1094,7 @@ async function reconcileNativeFinalizationsPass( projectRunStatus: true, preserveProviderAttempt: Boolean( readNativeWorkspaceSyncReference( - record(row.runnerProfileJson).nativeWorkspaceSync, + row.nativeWorkspaceSync, ), ), }); @@ -1153,7 +1152,7 @@ async function reconcileNativeFinalizationsPass( projectRunStatus: true, preserveProviderAttempt: Boolean( readNativeWorkspaceSyncReference( - record(row.runnerProfileJson).nativeWorkspaceSync, + row.nativeWorkspaceSync, ), ), }); diff --git a/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts b/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts index 2c2e534081b..5d1ed177987 100644 --- a/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts +++ b/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts @@ -279,6 +279,11 @@ describeEmbeddedPostgres("native run finalizer / status decision committer — a }); expect(agentTaskRunCalls(callsBefore)).toHaveLength(0); expect(reconcileMuseAssignments).toHaveBeenCalledExactlyOnceWith(companyId, agentId, fixture.runId); + await db.insert(workspaceOperations).values({ companyId, issueId: fixture.issueId, + heartbeatRunId: fixture.runId, phase: "workspace_finalize", status: "succeeded" }); + const reconciled = await reconcileNativeFinalizations(db, [fixture.runId]); + expect(reconciled).toContainEqual(expect.objectContaining({ runId: fixture.runId, phase: "committed" })); + expect(agentTaskRunCalls(callsBefore)).toHaveLength(0); }); }); From ac15d883f7baeda25753aa7631c83a8d30d3d1e5 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 19:26:27 -0700 Subject: [PATCH 07/68] fix: preserve remote-tracking history when expiring nested worktrees (#17535) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Nested project repositories are now worktrees of one shared clone per project repository, and existing standalone clones are converted in place > - Conversion keeps the clone's remote-tracking refs as task-owned branches (`paperclip//remotes/...`) so commits only they hold survive > - Expiration then required every task-owned branch, including those, to be on the remote > - Remote-tracking refs for branches deleted or force-pushed upstream never are, so converted checkouts were kept forever > - This pull request treats those refs the way the standalone-clone path already does: commits missing upstream go into the restore point instead of blocking > - The benefit is that converted workspaces can expire and free disk without losing any history ## Linked Issues or Issue Description Refs #17407, #17382, #17289. Bug report described in-PR (bug report template). **What happened?** On one instance, 17 closed workspaces in a two-repository project were kept on every expiration sweep with `unpushed_managed_checkout_branches`. Each nested checkout is a converted worktree whose task-owned refs include `paperclip//remotes/origin/` for upstream branches since deleted or force-pushed (dependabot, squashed feature branches). Their commits are on no remote, so `provenNestedWorktreeHistory` refused. The task's own branch was fully pushed. **Expected behavior** As in the standalone-clone path (#17289), remote-tracking commits missing upstream are preserved in the restore point, and only genuinely local branches block expiration. **Steps to reproduce** 1. Fetch an upstream branch into a standalone nested clone, then delete that branch upstream. 2. Convert the clone into a worktree (nested clone migration). 3. Close the issue and let expiration inspect it: kept with `unpushed_managed_checkout_branches`. **Paperclip version or commit:** `master` at cb47dbb2da. **Deployment mode:** any. **Agent adapter(s) involved:** not adapter-specific (core). ## What Changed - `provenNestedWorktreeHistory` in `server/src/services/git-workspace-restoration.ts`: - splits task-owned refs into those under `remotes/` and the rest; - still requires the rest (and their reflogs) to be on the remote; - returns commits under `remotes/` that are missing upstream, and their reflogs, as `preserve`, which the restore point bundles. This matches the standalone-clone path. - `restorePointHistoryDigest` in `server/src/services/git-restore-point.ts`: for a linked worktree, it fingerprints the task's own refs and their reflogs instead of the shared clone's `refs/remotes`. Other tasks change those whenever they fetch, and they never covered the task-owned remote refs. It loads `nested-repository-worktrees` lazily to avoid an import cycle. No code comments were added. ## Verification - `pnpm exec vitest run nested-clone-migration` (in `server/`). New cases: - A converted checkout whose only extra history is a remote-tracking commit deleted upstream expires with a restore point. After its refs are removed and the shared clone is garbage-collected, reopening restores the commit. - The digest ignores the shared clone's remote refs and changes when the task's own refs move. - **Mutation checks**, each failing its test: - reverting the prover split; - dropping the preserve list; - reverting the digest change. - The existing case where a converted checkout with a genuinely unpushed branch is kept still passes. - `pnpm exec vitest run git-restore-point nested-repository-worktrees nested-clone-migration git-workspace-expiration git-reproducible-managed-checkouts isolated-task-directory isolated-repository-rehome git-credentials execution-workspace` (in `server/`): 591 passed. - `pnpm run typecheck` (in `server/`): pass. - Not run: server build and boot smoke. ## Risks - **Misclassified branches:** a local branch literally named `remotes/...` in a standalone clone becomes `remotes/...` after conversion and is now preserved in the restore point instead of blocking expiration. Its commits are still saved, not lost. - **Leftover refs:** after expiry, the task's other owned refs stay in the shared clone, as they did before this change, so their commits stay in two places. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, tests, read-only diagnosis on a live instance). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../__tests__/nested-clone-migration.test.ts | 37 ++++++++++++++++++- server/src/services/git-restore-point.ts | 18 +++++++-- .../src/services/git-workspace-restoration.ts | 15 +++++--- .../services/nested-repository-worktrees.ts | 1 + 4 files changed, 62 insertions(+), 9 deletions(-) diff --git a/server/src/__tests__/nested-clone-migration.test.ts b/server/src/__tests__/nested-clone-migration.test.ts index e77222c4b9e..b417a4ce897 100644 --- a/server/src/__tests__/nested-clone-migration.test.ts +++ b/server/src/__tests__/nested-clone-migration.test.ts @@ -14,7 +14,7 @@ import { realizeExecutionWorkspace } from "../services/workspace-runtime.js"; import { executionWorkspaceService } from "../services/execution-workspaces.js"; import { prepareProjectRepositoryWorkspaces } from "../services/heartbeat/workspaces.js"; import { expireReconstructableGitWorkspace } from "../services/git-workspace-expiration.js"; -import { currentRestorePointSnapshot } from "../services/git-restore-point.js"; +import { currentRestorePointSnapshot, restorePointHistoryDigest } from "../services/git-restore-point.js"; import { nestedRepositoryBaseCandidates, nestedWorktreeBranchPrefix, readLinkedNestedCheckout, withNestedRepositoryBaseLock } from "../services/nested-repository-worktrees.js"; import { listNestedCloneMigrationNames, migrateNestedRepositoryClone, nextNestedCloneMigrationBatch, type NestedCloneMigrationOptions } from "../services/nested-clone-migration.js"; import { getStorageService } from "../storage/index.js"; @@ -225,6 +225,41 @@ describeEmbeddedPostgres("nested repository clone migration", () => { expect(git(f.checkout, "cat-file", "-t", feature)).toBe("commit"); }); + it("expires a converted checkout whose only extra history is remote-tracking and restores that history", async () => { + const f = await fixture({ issueStatus: "done" }); + git(f.nested.author, "checkout", "-q", "-b", "squashed-away"); + await fs.writeFile(path.join(f.nested.author, "feature.txt"), "feature\n"); + git(f.nested.author, "add", "."); git(f.nested.author, "commit", "-qm", "Feature"); + const feature = git(f.nested.author, "rev-parse", "HEAD"); + git(f.nested.author, "push", "-q", f.nested.remote, "squashed-away"); + git(f.checkout, "fetch", "-q", "origin"); + git(f.nested.remote, "branch", "-D", "squashed-away"); + git(f.nested.remote, "reflog", "expire", "--expire=now", "--all"); + git(f.nested.remote, "gc", "-q", "--prune=now"); + expect(await f.migrate()).toMatchObject({ outcome: "migrated" }); + + expect(await f.expire()).toMatchObject({ expired: true, restorePoint: true, skipReason: null }); + for (const line of ownedRefs(f.base, f.prefix).split("\n").filter(Boolean)) git(f.base, "update-ref", "-d", line.split(" ")[0]!); + git(f.base, "reflog", "expire", "--expire=now", "--all"); + git(f.base, "gc", "-q", "--prune=now"); + expect(() => git(f.base, "cat-file", "-e", `${feature}^{commit}`)).toThrow(); + expect(await f.reopen()).toMatchObject({ ok: true, reopened: true }); + expect(git(f.checkout, "cat-file", "-t", feature)).toBe("commit"); + }); + + it("fingerprints a converted checkout's own history rather than the shared clone's remote refs", async () => { + const f = await fixture(); + expect(await f.migrate()).toMatchObject({ outcome: "migrated" }); + const head = git(f.checkout, "rev-parse", "HEAD"); + const before = await restorePointHistoryDigest(f.checkout); + git(f.base, "update-ref", "refs/remotes/origin/another-task", head); + expect(await restorePointHistoryDigest(f.checkout)).toBe(before); + git(f.base, "update-ref", `refs/heads/${f.prefix}remotes/origin/extra`, head); + expect(await restorePointHistoryDigest(f.checkout)).not.toBe(before); + for (let index = 0; index < 64; index++) git(f.base, "update-ref", `refs/heads/${f.prefix}remotes/origin/bulk-${index}`, head); + await expect(restorePointHistoryDigest(f.checkout)).rejects.toThrow("managed_checkout_local_tip_budget"); + }); + it("finishes an interrupted conversion after the workspace is archived and forgets one whose workspace is gone", async () => { const f = await fixture(); const { unpushed } = await nestedWork(f.checkout); diff --git a/server/src/services/git-restore-point.ts b/server/src/services/git-restore-point.ts index 698b1395ade..3af9997b693 100644 --- a/server/src/services/git-restore-point.ts +++ b/server/src/services/git-restore-point.ts @@ -595,9 +595,21 @@ export async function restorePointHistoryDigest(input: string, signal?: AbortSig const cwd = await fs.realpath(input); const reflog = await gitText(cwd, ["reflog", "show", "--format=%H", "HEAD"], { signal }).catch(() => ""); const original = await gitText(cwd, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], { signal }).catch(() => ""); - const remotes = await gitText(cwd, ["for-each-ref", "--format=%(objectname) %(refname)", "refs/remotes"], { signal }).catch(() => ""); - const remoteReflogs = (await gitText(cwd, ["reflog", "show", "--all", "--format=%H %gD"], { signal }).catch(() => "")) - .split("\n").filter((line) => / refs\/remotes\//.test(line)); + const linked = (await fs.lstat(path.join(cwd, ".git")).catch(() => null))?.isFile() ?? false; + const nested = linked ? await import("./nested-repository-worktrees.js") : null; + const scope = nested ? `refs/heads/${await nested.nestedWorktreeBranchPrefix(cwd)}` : "refs/remotes"; + const remotes = await gitText(cwd, ["for-each-ref", ...nested ? [`--count=${nested.OWNED_NESTED_REF_LIMIT + 1}`] : [], + "--format=%(objectname) %(refname)", scope], { signal }).catch(() => ""); + const remoteReflogs: string[] = []; + if (nested) { + if (remotes.split("\n").filter(Boolean).length > nested.OWNED_NESTED_REF_LIMIT) throw new Error("managed_checkout_local_tip_budget"); + for (const ref of remotes.split("\n").map((line) => line.split(" ")[1]).filter(Boolean)) { + remoteReflogs.push(...(await gitText(cwd, ["reflog", "show", "--format=%H", ref!], { signal }).catch(() => "")).split("\n").filter((line) => SHA.test(line)).map((sha) => `${sha} ${ref}`)); + } + } else { + remoteReflogs.push(...(await gitText(cwd, ["reflog", "show", "--all", "--format=%H %gD"], { signal }).catch(() => "")) + .split("\n").filter((line) => / refs\/remotes\//.test(line))); + } return createHash("sha256").update(JSON.stringify([reflog.split("\n").filter((line) => SHA.test(line)), SHA.test(original) ? original : null, remotes.split("\n").filter(Boolean), remoteReflogs])).digest("hex"); } diff --git a/server/src/services/git-workspace-restoration.ts b/server/src/services/git-workspace-restoration.ts index f71c1851e7c..5aacb475a80 100644 --- a/server/src/services/git-workspace-restoration.ts +++ b/server/src/services/git-workspace-restoration.ts @@ -18,7 +18,7 @@ import { ensureGitRestorePointCommit, validateGitRestorePointRef, type GitRestor import { getStorageService } from "../storage/index.js"; import type { StorageService } from "../storage/types.js"; import { PROJECT_REPOSITORIES_DIR } from "@paperclipai/adapter-utils/git-workspace-sync"; -import { forgetExpiredNestedRepositoryWorktree, nestedRepositoryBaseCandidates, readLinkedNestedCheckout, recordedNestedRepositoryBase, type LinkedNestedCheckout } from "./nested-repository-worktrees.js"; +import { OWNED_NESTED_REF_LIMIT, forgetExpiredNestedRepositoryWorktree, nestedRepositoryBaseCandidates, readLinkedNestedCheckout, recordedNestedRepositoryBase, type LinkedNestedCheckout } from "./nested-repository-worktrees.js"; const execFile = promisify(execFileCallback); const shaPattern = /^[a-f0-9]{40}$/; @@ -249,7 +249,6 @@ async function assertCommitMessageSaved(cwd: string, gitDir: string, signal?: Ab if (!saved.includes(message.trim())) throw new Error("uncommitted_git_message"); } -const OWNED_NESTED_REF_LIMIT = 64; async function ownedNestedRefs(linked: LinkedNestedCheckout, signal?: AbortSignal) { const refs = (await gitCacheCommand(linked.checkout, ["for-each-ref", `--count=${OWNED_NESTED_REF_LIMIT + 1}`, @@ -380,9 +379,13 @@ async function provenNestedWorktreeHistory(linked: LinkedNestedCheckout, repoUrl const covered = new Set([head, ...shas(await gitCacheCommand(checkout, ["reflog", "show", "--format=%H", "HEAD"], signal).catch(() => "")), ...shas(await gitCacheCommand(checkout, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], signal).catch(() => ""))]); - const owned = await ownedNestedRefs(linked, signal); + const allOwned = await ownedNestedRefs(linked, signal); + const remotePrefix = `refs/heads/${linked.branchPrefix}remotes/`; + const owned = allOwned.filter(([, ref]) => !ref.startsWith(remotePrefix)); + const remoteTracked = [...new Set([...allOwned.filter(([, ref]) => ref.startsWith(remotePrefix)).map(([sha]) => sha), + ...await ownedNestedReflogs(linked, allOwned.filter(([, ref]) => ref.startsWith(remotePrefix)), signal)])]; const recorded = new Set([...owned.map(([sha]) => sha), ...await ownedNestedReflogs(linked, owned, signal)]); - if (recorded.size + covered.size > MANAGED_CHECKOUT_LOCAL_TIP_LIMIT) throw new Error("managed_checkout_local_tip_budget"); + if (recorded.size + covered.size + remoteTracked.length > MANAGED_CHECKOUT_LOCAL_TIP_LIMIT) throw new Error("managed_checkout_local_tip_budget"); const current = (await gitCacheCommand(checkout, ["symbolic-ref", "--quiet", "HEAD"], signal).catch(() => "")).trim(); const otherTips = owned.filter(([, ref]) => ref !== current).map(([sha]) => sha); if (otherTips.length && await hasCommitsMissingUpstream(checkout, otherTips, repoUrl, resolveGitAuth, signal, [head])) { @@ -393,7 +396,9 @@ async function provenNestedWorktreeHistory(linked: LinkedNestedCheckout, repoUrl throw new Error("unpushed_managed_checkout_branches"); } const headOnRemote = !await hasCommitsMissingUpstream(checkout, [head], repoUrl, resolveGitAuth, signal); - return { headOnRemote, preserve: [] }; + const preserve = remoteTracked.length + ? await tipsMissingUpstream(checkout, remoteTracked, repoUrl, resolveGitAuth, signal, [...covered, ...recorded]) : []; + return { headOnRemote, preserve }; } async function tipsMissingUpstream(checkout: string, tips: string[], repoUrl: string, diff --git a/server/src/services/nested-repository-worktrees.ts b/server/src/services/nested-repository-worktrees.ts index 8a2969ec2f3..ad907159cc3 100644 --- a/server/src/services/nested-repository-worktrees.ts +++ b/server/src/services/nested-repository-worktrees.ts @@ -15,6 +15,7 @@ import { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js"; const execFile = promisify(execFileCallback); const NESTED_WORKTREE_GIT_TIMEOUT_MS = 10 * 60 * 1000; const NESTED_WORKTREE_BRANCH = /^paperclip\/[a-f0-9]{12}\/./; +export const OWNED_NESTED_REF_LIMIT = 64; export const DETACHED_REPOSITORIES_DIR = path.join(".paperclip-runtime", "detached-repositories"); export type LinkedNestedCheckout = { checkout: string; base: string; gitDir: string; admin: string; branchPrefix: string }; From e0855fe92ad02250d3a289543e6900a050ccf411 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 19:32:51 -0700 Subject: [PATCH 08/68] perf(wake-queue): narrow native release reads (#17539) Native wake-queue release reads omit saved profiles and excerpts. Legacy release reads keep their complete inputs. Ownership, cleanup, and file retention checks remain unchanged. Verified with 275 wake-queue tests, the full build, and typecheck. All CI checks pass. Greptile scored the exact head 5/5 with Medium impact and no unresolved comments. Co-Authored-By: Paperclip --- doc/native-workspace-finalization-recovery.md | 6 ++ .../wake-queue/adapters/postgres.test.ts | 9 ++- .../modules/wake-queue/adapters/postgres.ts | 27 +++---- .../adapters/release-run-read.test.ts | 74 +++++++++++++++++++ .../wake-queue/adapters/release-run-read.ts | 22 ++++++ 5 files changed, 117 insertions(+), 21 deletions(-) create mode 100644 server/src/modules/wake-queue/adapters/release-run-read.test.ts create mode 100644 server/src/modules/wake-queue/adapters/release-run-read.ts diff --git a/doc/native-workspace-finalization-recovery.md b/doc/native-workspace-finalization-recovery.md index 3d16ff3aada..51d174bb3e7 100644 --- a/doc/native-workspace-finalization-recovery.md +++ b/doc/native-workspace-finalization-recovery.md @@ -24,6 +24,12 @@ the native status wake identifier from each saved run. It keeps their JSON types and reads full evidence later where finalization or recovery needs it. Saved execution input, runner profiles, and logs stay in the stored run. +Wake-queue release reads keep complete context and result receipts. Native runs +omit runner profiles and log excerpts from this read; cleanup and execution +read their own current evidence. Legacy runs retain the full release row so +workspace-restore and pre-dispatch proofs keep their existing inputs. The +selection is one scoped database query for either runtime. + The recovery card names the original run and requires evidence that its exact allocation stopped and its files were recovered. Recording that evidence does not retry the old turn, certify unknown external actions, change the current task diff --git a/server/src/modules/wake-queue/adapters/postgres.test.ts b/server/src/modules/wake-queue/adapters/postgres.test.ts index 16479253a2f..6a8c360e22d 100644 --- a/server/src/modules/wake-queue/adapters/postgres.test.ts +++ b/server/src/modules/wake-queue/adapters/postgres.test.ts @@ -206,7 +206,7 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { return { recorded, rows }; } - it("keeps in-flight work and every recovery receipt while omitting output from release reads", async () => { + it.each(["legacy", "native"])("keeps %s in-flight work and every recovery receipt while omitting output from release reads", async runtimeMode => { const companyId = await seedCompany(); const agentId = await seedAgent({ companyId }); const issueId = await seedIssue({ companyId, assigneeAgentId: agentId }); @@ -218,7 +218,9 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { nativeCancellation: { schema: "paperclip.native-cancellation.v1", runId, companyId, issueId }, executionCancellation: { state: "acknowledged" }, executionRecovery: { kind: "interrupted", providerStopped: true }, configurationIncomplete: { reason: "workspace_unavailable" }, unknownFutureReceipt: { retained: true } }; - await db.update(heartbeatRuns).set({ resultJson: { ...evidence, stdout: payload.repeat(3), stderr: payload } }).where(eq(heartbeatRuns.id, runId)); + await db.update(heartbeatRuns).set({ runtimeMode, + ...(runtimeMode === "native" ? { runnerProfileJson: { savedInput: payload }, stdoutExcerpt: payload, stderrExcerpt: payload } : {}), + resultJson: { ...evidence, stdout: payload.repeat(3), stderr: payload } }).where(eq(heartbeatRuns.id, runId)); await db.update(issues).set({ executionRunId: runId, checkoutRunId: runId }).where(eq(issues.id, issueId)); const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); const [issue] = await db.select().from(issues).where(eq(issues.id, issueId)); @@ -230,7 +232,8 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { ); expect(result.outcome.kind).toBe("released"); expect(rows).toHaveLength(1); - expect(rows[0]).toEqual([{ ...run, resultJson: evidence }]); + expect(rows[0]).toEqual([{ ...run, resultJson: evidence, + ...(runtimeMode === "native" ? { runnerProfileJson: null, stdoutExcerpt: null, stderrExcerpt: null } : {}) }]); expect(Buffer.byteLength(JSON.stringify(rows))).toBeLessThan(8 * 1024); expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]).toEqual(run); expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toEqual(issue); diff --git a/server/src/modules/wake-queue/adapters/postgres.ts b/server/src/modules/wake-queue/adapters/postgres.ts index 6db54a82725..cd2f1aa7a48 100644 --- a/server/src/modules/wake-queue/adapters/postgres.ts +++ b/server/src/modules/wake-queue/adapters/postgres.ts @@ -5,7 +5,8 @@ import { isCompletedOnboardingHandoffWake } from "../../../services/chat-complet import { instanceSettingsService } from "../../../services/instance-settings.js"; import { currentConversationCommentCondition } from "../../../services/agent-conversations.js"; import { getExecutionBlocker, hasPendingRunCleanup } from "../../../services/execution-blocker.js"; -import { and, asc, eq, getTableColumns, inArray, isNull, notInArray, or, sql } from "drizzle-orm"; +import { and, asc, eq, inArray, isNull, notInArray, or, sql } from "drizzle-orm"; +import { readReleaseRun, type ReleaseRun } from "./release-run-read.js"; import type { Db } from "@paperclipai/db"; import { extractIssueReferenceIdentifiers, isUuidLike } from "@paperclipai/shared"; import { @@ -74,14 +75,6 @@ const EXECUTION_PATH_HEARTBEAT_RUN_STATUSES = ["queued", "running", "scheduled_r type HeartbeatRunRow = typeof heartbeatRuns.$inferSelect; type IssueRow = typeof issues.$inferSelect; -const releaseRunFields = { - ...getTableColumns(heartbeatRuns), - resultJson: sql | null>`case - when jsonb_typeof(${heartbeatRuns.resultJson}) = 'object' - then ${heartbeatRuns.resultJson} - 'stdout' - 'stderr' - else ${heartbeatRuns.resultJson} end`, -}; - function normalizeAgentNameKey(value: string | null | undefined): string | null { if (typeof value !== "string") return null; const normalized = value.trim().toLowerCase(); @@ -94,7 +87,7 @@ function toRequestedByActorType(value: string | null): "user" | "agent" | "syste return value === "user" || value === "agent" || value === "system" ? value : null; } -function toRunSnapshot(row: HeartbeatRunRow): RunSnapshot { +function toRunSnapshot(row: ReleaseRun): RunSnapshot { const configurationIncompletePayload = parseObject(parseObject(row.resultJson).configurationIncomplete); return { id: row.id, @@ -206,7 +199,7 @@ function buildHost(_tx: Db, deps: WakeQueuePostgresAdapterDeps): WakeQueueHost { }; } -function buildTransaction(tx: Db, deps: WakeQueuePostgresAdapterDeps, db: Db, run: HeartbeatRunRow): WakeQueueTransaction { +function buildTransaction(tx: Db, deps: WakeQueuePostgresAdapterDeps, db: Db, run: ReleaseRun): WakeQueueTransaction { const treeControlSvc = issueTreeControlService(tx); const issuesSvc = issueService(tx); const interruptQueueId = run.runtimeMode !== "native" && run.status === "cancelled" @@ -750,7 +743,7 @@ function buildTransaction(tx: Db, deps: WakeQueuePostgresAdapterDeps, db: Db, ru }; } -async function recordNativeTerminalRecoveryIfNeeded(tx: Db, run: HeartbeatRunRow, issue: IssueRow, now: Date): Promise { +async function recordNativeTerminalRecoveryIfNeeded(tx: Db, run: ReleaseRun, issue: IssueRow, now: Date): Promise { const applies = run.runtimeMode === "native" && ["failed", "timed_out", "interrupted", "cancelled"].includes(run.status) && @@ -1058,11 +1051,8 @@ export function createPostgresWakeQueueAdapter(db: Db, deps: WakeQueuePostgresAd async withIssueExecutionLock(input, fn): Promise { return db.transaction(async (rawTx) => { const tx = rawTx as unknown as Db; - const run = await tx - .select(releaseRunFields) - .from(heartbeatRuns) - .where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId))) - .then((rows) => rows[0] ?? null); + const evidence = await readReleaseRun(tx, input); + const run = evidence.run; if (!run) { throw new Error(`wake-queue: run ${input.runId} was not found while releasing issue execution`); } @@ -1164,7 +1154,8 @@ export function createPostgresWakeQueueAdapter(db: Db, deps: WakeQueuePostgresAd issueStatus: issueRow?.status ?? "", hasAssigneeUser: Boolean(issueRow?.assigneeUserId), assigneeAgentMatchesRunAgent: issueRow?.assigneeAgentId === run.agentId, - legacyExecutionNeedsReconciliation: await legacyExecutionNeedsReconciliationWithEvidence(tx as unknown as Db, run), + legacyExecutionNeedsReconciliation: evidence.kind === "complete" && + await legacyExecutionNeedsReconciliationWithEvidence(tx as unknown as Db, evidence.run), // An operator stop never promotes old queued work by itself. The // next explicit wake adopts those messages atomically when it // queues a run. diff --git a/server/src/modules/wake-queue/adapters/release-run-read.test.ts b/server/src/modules/wake-queue/adapters/release-run-read.test.ts new file mode 100644 index 00000000000..e4129e6c692 --- /dev/null +++ b/server/src/modules/wake-queue/adapters/release-run-read.test.ts @@ -0,0 +1,74 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { agents, companies, createDb, heartbeatRuns } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js"; +import { readReleaseRun } from "./release-run-read.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("wake release run reads", () => { + let database: Awaited>; + let db: ReturnType; + let companyId: string; + let agentId: string; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("wake-release-read-"); + db = createDb(database.connectionString); + const [company] = await db.insert(companies).values({ name: "Wake release fixture", issuePrefix: "WRR" }).returning(); + const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Release agent", adapterType: "process" }).returning(); + companyId = company.id; + agentId = agent.id; + }, 60_000); + + afterAll(async () => { + await db?.$client.end(); + await database?.cleanup(); + }); + + it("omits native bulk at the driver boundary and preserves the complete legacy fallback", async () => { + const bulk = randomBytes(128 * 1024).toString("base64"); + const context = { issueId: randomUUID(), opaqueFutureContext: { savedInput: "keep this input" }, explicitUserContinuation: true }; + const evidence = { nativeCancellation: { schema: "paperclip.native-cancellation.v1", acknowledged: true }, unknownFutureReceipt: [true, null, 42] }; + const [before] = await db.insert(heartbeatRuns).values({ + companyId, agentId, runtimeMode: "native", status: "running", contextSnapshot: context, + runnerProfileJson: { nativeExecutionInput: { savedInput: bulk } }, stdoutExcerpt: bulk, stderrExcerpt: bulk, + resultJson: { ...evidence, stdout: bulk, stderr: bulk }, + }).returning(); + const scope = { companyId, runId: before.id }; + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { + const native = await db.transaction(tx => readReleaseRun(tx as unknown as typeof db, scope)); + expect(native.kind).toBe("native"); + expect(native.run?.contextSnapshot).toEqual(context); + expect(native.run?.resultJson).toEqual(evidence); + expect(native.run).not.toHaveProperty("runnerProfileJson"); + expect(native.run).not.toHaveProperty("stdoutExcerpt"); + expect(native.run).not.toHaveProperty("stderrExcerpt"); + } finally { capture.flush(); } + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBe(1); + expect(summary.responseValueBytes).toBeLessThan(8192); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, before.id)))[0]).toEqual(before); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + await db.update(heartbeatRuns).set({ runtimeMode: "legacy" }).where(eq(heartbeatRuns.id, before.id)); + const legacy = await readReleaseRun(db, scope); + expect(legacy.kind).toBe("complete"); + expect(legacy.run).toEqual({ ...before, runtimeMode: "legacy", resultJson: evidence }); + expect((await readReleaseRun(db, { ...scope, companyId: randomUUID() })).run).toBeUndefined(); + expect((await readReleaseRun(db, { companyId, runId: randomUUID() })).run).toBeUndefined(); + }); + + it.each([null, ["stdout", { stderr: "evidence" }], "scalar", 42].map(result => ({ result })))( + "preserves non-object native result evidence $result", async ({ result }) => { + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, runtimeMode: "native", + resultJson: result as Record | null }).returning(); + expect((await readReleaseRun(db, { companyId, runId: run.id })).run?.resultJson).toEqual(result); + }, + ); +}); diff --git a/server/src/modules/wake-queue/adapters/release-run-read.ts b/server/src/modules/wake-queue/adapters/release-run-read.ts new file mode 100644 index 00000000000..cc8bcf80388 --- /dev/null +++ b/server/src/modules/wake-queue/adapters/release-run-read.ts @@ -0,0 +1,22 @@ +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { and, eq, getTableColumns, sql } from "drizzle-orm"; + +type Run = typeof heartbeatRuns.$inferSelect; +export type ReleaseRun = Omit; + +export async function readReleaseRun(db: Db, scope: { companyId: string; runId: string }) { + const [selected] = await db.select({ + ...getTableColumns(heartbeatRuns), + runnerProfileJson: sql`case when ${heartbeatRuns.runtimeMode} = 'native' + then null else ${heartbeatRuns.runnerProfileJson} end`, + stdoutExcerpt: sql`case when ${heartbeatRuns.runtimeMode} = 'native' + then null else ${heartbeatRuns.stdoutExcerpt} end`, + stderrExcerpt: sql`case when ${heartbeatRuns.runtimeMode} = 'native' + then null else ${heartbeatRuns.stderrExcerpt} end`, + resultJson: sql`case when jsonb_typeof(${heartbeatRuns.resultJson}) = 'object' + then ${heartbeatRuns.resultJson} - 'stdout' - 'stderr' else ${heartbeatRuns.resultJson} end`, + }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, scope.runId), eq(heartbeatRuns.companyId, scope.companyId))); + if (!selected || selected.runtimeMode !== "native") return { kind: "complete" as const, run: selected }; + const { runnerProfileJson: _profile, stdoutExcerpt: _stdout, stderrExcerpt: _stderr, ...run } = selected; + return { kind: "native" as const, run }; +} From a12329b8a96208918a03a9f11be93a7319b21cb9 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 19:34:31 -0700 Subject: [PATCH 09/68] fix(db): complete bounded response attribution (#17537) Raise bounded database-response attribution to 2048 groups and add numbered 128-entry log pages. Preserve the overflow bucket and exact completed-call and byte totals. Document complete page retrieval and validation. Verified with 30 focused tests, the full build, and typecheck. All CI checks pass. Exact-head Greptile is 5/5 at Medium impact with no unresolved comments. Co-Authored-By: Paperclip --- doc/DATABASE.md | 21 +++++++++++++++++- packages/db/src/response-capture.test.ts | 27 ++++++++++++++++++++---- packages/db/src/response-capture.ts | 7 ++++-- 3 files changed, 48 insertions(+), 7 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 717ec1ca0ed..17fbb47f400 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -341,7 +341,7 @@ The diagnostic measures awaited Postgres.js `unsafe` responses before Drizzle column decoding. It retains the Drizzle builder caller for normal, prepared, transaction, and dedicated-pool execution. It issues no queries. It logs the public server caller locations, static schema table, operation, -query/row counts, and response value sizes. It emits at most 512 aggregate +query/row counts, and response value sizes. It emits at most 2,048 aggregate groups and a summary at the deadline. It never logs SQL, parameters, result values, connection strings, or query errors. In-flight queries continue normally. @@ -355,6 +355,25 @@ Look for `paperclip.db_response_capture.entry.v1` and can store `event` as a structured field instead of message text. Filter by that field or read the bounded window from the exact deployment. Check the summary and group totals, including the unattributed overflow group. + +Fetch a complete capture as follows: + +1. Read the summary events from the exact deployment and window. Keep each + distinct `captureId` separately. +2. Each entry has a unique `groupIndex` within that capture and a `groupPage`. + The summary gives `groupPageSize` (128) and `groupPages`. Fetch every page + from zero through `groupPages - 1`, filtering on the entry event, `captureId`, + and `groupPage`. Request at least `groupPageSize` matching records per page. + A structured query can use + `@event:paperclip.db_response_capture.entry.v1 AND @captureId: AND @groupPage:0`. +3. If the provider's record limit is smaller than a page, fetch each + `groupIndex` separately with the same capture ID. Each index matches one + entry. Do not split by timestamp: entries can share a timestamp. +4. Deduplicate by `(captureId, groupIndex)`. Require every index from zero + through `groups - 1`, with no gaps. Entry count must equal `groups`, summed + `calls` must equal `completedQueries`, and summed `responseValueBytes` must + equal the summary total. A missing page is an incomplete measurement. + Tagged raw-driver calls, streams, unawaited raw-driver dispatch, and protocol overhead are not measured. Relational queries use a driver-level caller fallback. Queries still in flight at the deadline are reported separately. Sizes describe diff --git a/packages/db/src/response-capture.test.ts b/packages/db/src/response-capture.test.ts index 4b87c7924a4..9fc1c80f5b1 100644 --- a/packages/db/src/response-capture.test.ts +++ b/packages/db/src/response-capture.test.ts @@ -107,12 +107,31 @@ describe("bounded database response capture", () => { }); it("bounds cardinality without dropping response-byte totals", () => { const f = fixture(); - for (let i = 0; i < 600; i += 1) f.capture.execute(() => [i], { ...metadata, callers: [`services/f${i}.ts:1:1`] }); + for (let i = 0; i < 2200; i += 1) f.capture.execute(() => [i], { ...metadata, callers: [`services/f${i}.ts:1:1`] }); f.capture.flush(); const summary = f.events.at(-1)!; - expect(summary).toMatchObject({ startedQueries: 600, completedQueries: 600, groups: 512, overflowCalls: 89 }); - expect(f.events).toHaveLength(513); - expect(summary.responseValueBytes).toBe(Array.from({ length: 600 }, (_, i) => Buffer.byteLength(JSON.stringify([i]))).reduce((a, b) => a + b, 0)); + expect(summary).toMatchObject({ startedQueries: 2200, completedQueries: 2200, groups: 2048, overflowCalls: 153, + groupPageSize: 128, groupPages: 16 }); + expect(f.events).toHaveLength(2049); + const entries = f.events.slice(0, -1); + expect(entries.map(entry => entry.groupIndex)).toEqual(Array.from({ length: 2048 }, (_, index) => index)); + for (let page = 0; page < 16; page += 1) { + expect(entries.filter(entry => entry.groupPage === page)).toHaveLength(128); + } + expect(entries.reduce((sum, entry) => sum + Number(entry.calls), 0)).toBe(2200); + expect(summary.responseValueBytes).toBe(Array.from({ length: 2200 }, (_, i) => Buffer.byteLength(JSON.stringify([i]))).reduce((a, b) => a + b, 0)); + }); + it.each([0, 129])("numbers every page including an empty or partial final page (%s)", count => { + const f = fixture(); + for (let index = 0; index < count; index += 1) { + f.capture.execute(() => [index], { ...metadata, callers: [`services/page${index}.ts:1:1`] }); + } + f.capture.flush(); + const entries = f.events.slice(0, -1), summary = f.events.at(-1)!; + expect(summary).toMatchObject({ groups: count, groupPageSize: 128, groupPages: Math.ceil(count / 128) }); + expect(entries.map(entry => entry.groupIndex)).toEqual(Array.from({ length: count }, (_, index) => index)); + expect(entries.filter(entry => entry.groupPage === 0)).toHaveLength(Math.min(count, 128)); + expect(entries.filter(entry => entry.groupPage === 1)).toHaveLength(Math.max(0, count - 128)); }); it("does not execute builders or prepared statements until requested", async () => { const f = fixture(), rows = [{ id: 1 }], execute = vi.fn(async () => rows); diff --git a/packages/db/src/response-capture.ts b/packages/db/src/response-capture.ts index f8e6da8e87a..675d57eeb82 100644 --- a/packages/db/src/response-capture.ts +++ b/packages/db/src/response-capture.ts @@ -7,6 +7,7 @@ type Entry = Metadata & { calls: number; rows: number; bytes: number; maxBytes: type Event = Record; type Method = (this: unknown, ...args: unknown[]) => unknown; export type ResponseCaptureWindow = { start: number; end: number }; +const GROUP_PAGE_SIZE = 128; export function responseCaptureWindow(value: string | undefined, now = Date.now()): ResponseCaptureWindow | undefined { if (!value) return; @@ -76,7 +77,8 @@ export function createDatabaseResponseCapture( clearTimeout(timer); scope.disable(); const ordered = [...entries.values()].sort((a, b) => b.bytes - a.bytes); - for (const entry of ordered) send({ event: "paperclip.db_response_capture.entry.v1", captureId: id, + for (const [groupIndex, entry] of ordered.entries()) send({ event: "paperclip.db_response_capture.entry.v1", captureId: id, + groupIndex, groupPage: Math.floor(groupIndex / GROUP_PAGE_SIZE), operation: entry.operation, table: entry.table, callers: entry.callers, calls: entry.calls, rows: entry.rows, responseValueBytes: entry.bytes, maxResponseValueBytes: entry.maxBytes, estimatedValues: entry.estimatedValues, queryErrors: entry.queryErrors, serializationErrors: entry.serializationErrors }); @@ -84,6 +86,7 @@ export function createDatabaseResponseCapture( windowStart: new Date(window.start).toISOString(), windowEnd: new Date(window.end).toISOString(), closedAt: new Date(now()).toISOString(), startedQueries: started, completedQueries: completed, inFlightAtEnd: inFlight, groups: entries.size, + groupPageSize: GROUP_PAGE_SIZE, groupPages: Math.ceil(ordered.length / GROUP_PAGE_SIZE), overflowCalls, responseValueBytes: ordered.reduce((sum, entry) => sum + entry.bytes, 0), rowValuesLogged: false, sqlOrParametersLogged: false, wireBytesMeasured: false, measurementBoundary: "postgres-js results before Drizzle column decoding", @@ -99,7 +102,7 @@ export function createDatabaseResponseCapture( completed += 1; try { let key = JSON.stringify(metadata); - if (!entries.has(key) && entries.size >= 511) { + if (!entries.has(key) && entries.size >= 2047) { overflowCalls += 1; metadata = { operation: "other", table: "other", callers: [] }; key = "overflow"; From 80c1261417a578393b5420ca8d6ec95d7efaf8e0 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 19:50:19 -0700 Subject: [PATCH 10/68] perf(recovery): decline native legacy checks before bulk reads (#17540) Decline native legacy-continuation checks before loading saved execution input. Preserve exact nonblank task bindings, skip reasons, complete legacy recovery inputs, company scope, and retry accounting. Verified with 105 recovery tests, the full build, and typecheck; the binding review fix passed 73 affected tests and server typecheck. CI is green and exact-head Greptile is 5/5 at Medium impact with no unresolved comments. Co-Authored-By: Paperclip --- doc/native-workspace-finalization-recovery.md | 7 ++ .../recovery/legacy-disposition-read.test.ts | 88 +++++++++++++++++++ .../recovery/legacy-disposition-read.ts | 24 +++++ server/src/services/recovery/service.ts | 11 ++- 4 files changed, 126 insertions(+), 4 deletions(-) create mode 100644 server/src/services/recovery/legacy-disposition-read.test.ts create mode 100644 server/src/services/recovery/legacy-disposition-read.ts diff --git a/doc/native-workspace-finalization-recovery.md b/doc/native-workspace-finalization-recovery.md index 51d174bb3e7..0875b177449 100644 --- a/doc/native-workspace-finalization-recovery.md +++ b/doc/native-workspace-finalization-recovery.md @@ -492,3 +492,10 @@ binding, and chat source fields under the existing nonblocking share lock. It excludes saved execution input, runner profiles, and log excerpts. Actual retry admission still reads the complete run and its recovery evidence. These reads preserve stored run data, source revocation checks, and JSON value types. + +Legacy continuation recovery rejects native runs before it loads their saved +execution input. The persisted disposition check reads only the task binding +for a native run. It keeps the same task-binding and native-finalization skip +reasons, including the original nonblank binding values. Native input and history +stay stored. Real legacy recovery still reads +its complete context, profile, excerpts, and required failure evidence. diff --git a/server/src/services/recovery/legacy-disposition-read.test.ts b/server/src/services/recovery/legacy-disposition-read.test.ts new file mode 100644 index 00000000000..749a7991879 --- /dev/null +++ b/server/src/services/recovery/legacy-disposition-read.test.ts @@ -0,0 +1,88 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { agents, companies, createDb, heartbeatRuns } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { readLegacyDispositionSource } from "./legacy-disposition-read.js"; +import { recoveryService } from "./service.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("legacy disposition source reads", () => { + let database: Awaited>; + let db: ReturnType; + let companyId: string; + let agentId: string; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("legacy-disposition-read-"); + db = createDb(database.connectionString); + const [company] = await db.insert(companies).values({ name: "Legacy disposition fixture", issuePrefix: "LDF" }).returning(); + const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Disposition agent", adapterType: "process" }).returning(); + companyId = company.id; + agentId = agent.id; + }, 60_000); + + afterAll(async () => { + await db?.$client.end(); + await database?.cleanup(); + }); + + it("declines native input below the byte budget and keeps complete legacy evidence", async () => { + const issueId = randomUUID(), bulk = randomBytes(128 * 1024).toString("base64"); + const context = { issueId, savedInput: bulk, executionRetryAccounting: { version: 1, failureRetries: 2, maxTurnContinuations: 1 } }; + const [before] = await db.insert(heartbeatRuns).values({ companyId, agentId, runtimeMode: "native", status: "succeeded", + contextSnapshot: context, runnerProfileJson: { savedInput: bulk }, stdoutExcerpt: bulk, stderrExcerpt: bulk, + errorCode: "adapter_failed", resultJson: { executionRecovery: { providerWorkStarted: true }, stdout: bulk } }).returning(); + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { + expect(await readLegacyDispositionSource(db, { companyId, issueId, runId: before.id })) + .toEqual({ kind: "skip", decision: { kind: "skip", reason: "native_finalization" } }); + } finally { capture.flush(); } + expect(events.at(-1)).toMatchObject({ completedQueries: 1, groups: 1 }); + expect(Number(events.at(-1)?.responseValueBytes)).toBeLessThan(1024); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, before.id)))[0]).toEqual(before); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + await db.update(heartbeatRuns).set({ runtimeMode: "legacy" }).where(eq(heartbeatRuns.id, before.id)); + const legacy = await readLegacyDispositionSource(db, { companyId, issueId, runId: before.id }); + expect(legacy.kind).toBe("full"); + if (legacy.kind !== "full") throw new Error("expected complete legacy input"); + expect(legacy.run).toEqual({ ...before, runtimeMode: "legacy" }); + expect((await readLegacyDispositionSource(db, { companyId: randomUUID(), issueId, runId: before.id }))) + .toMatchObject({ kind: "full", run: undefined }); + }); + + it.each([ + { context: { issueId: " target " }, issueId: "target", reason: "invalid_issue_binding" }, + { context: { issueId: " ", taskId: "target" }, issueId: "target", reason: "native_finalization" }, + { context: { taskId: " target " }, issueId: "target", reason: "invalid_issue_binding" }, + { context: { issueId: 42, taskId: "target" }, issueId: "target", reason: "native_finalization" }, + { context: { issueId: "other", taskId: "target" }, issueId: "target", reason: "invalid_issue_binding" }, + { context: { issueId: null, taskId: false }, issueId: "target", reason: "invalid_issue_binding" }, + { context: null, issueId: "target", reason: "invalid_issue_binding" }, + ])("keeps native binding decisions for $context", async ({ context, issueId, reason }) => { + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, runtimeMode: "native", contextSnapshot: context }).returning(); + expect(await readLegacyDispositionSource(db, { companyId, issueId, runId: run.id })) + .toEqual({ kind: "skip", decision: { kind: "skip", reason } }); + }); + + it("skips native continuation in SQL without loading or rewriting its saved input", async () => { + const bulk = randomBytes(128 * 1024).toString("base64"); + const [before] = await db.insert(heartbeatRuns).values({ companyId, agentId, runtimeMode: "native", status: "succeeded", + contextSnapshot: { issueId: randomUUID(), savedInput: bulk }, runnerProfileJson: { savedInput: bulk } }).returning(); + const enqueueWakeup = vi.fn(async () => { throw new Error("native work must not enter legacy recovery"); }); + const service = recoveryService(db, { enqueueWakeup }); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { expect(await service.reconcileLegacyContinuation(before.id)).toBe("skipped"); } + finally { capture.flush(); } + expect(events.at(-1)).toMatchObject({ completedQueries: 1, responseValueBytes: 0 }); + expect(enqueueWakeup).not.toHaveBeenCalled(); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, before.id)))[0]).toEqual(before); + }); +}); diff --git a/server/src/services/recovery/legacy-disposition-read.ts b/server/src/services/recovery/legacy-disposition-read.ts new file mode 100644 index 00000000000..08f9204f491 --- /dev/null +++ b/server/src/services/recovery/legacy-disposition-read.ts @@ -0,0 +1,24 @@ +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { and, eq, getTableColumns, sql } from "drizzle-orm"; +import { recoveryRunResultColumn } from "./run-result-read.js"; + +export async function readLegacyDispositionSource( + db: Db, + scope: { companyId: string; issueId: string; runId: string }, +) { + const where = and(eq(heartbeatRuns.id, scope.runId), eq(heartbeatRuns.companyId, scope.companyId)); + const [native] = await db.select({ + issueId: sql`${heartbeatRuns.contextSnapshot}->'issueId'`, + taskId: sql`${heartbeatRuns.contextSnapshot}->'taskId'`, + }).from(heartbeatRuns).where(and(where, eq(heartbeatRuns.runtimeMode, "native"))).limit(1); + if (native) { + const bound = [native.issueId, native.taskId] + .map(value => typeof value === "string" && value.trim().length > 0 ? value : null).find(Boolean) ?? null; + return { kind: "skip" as const, decision: { + kind: "skip" as const, reason: bound === scope.issueId ? "native_finalization" : "invalid_issue_binding", + } }; + } + const [run] = await db.select({ ...getTableColumns(heartbeatRuns), resultJson: recoveryRunResultColumn }) + .from(heartbeatRuns).where(where).limit(1); + return { kind: "full" as const, run }; +} diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index 331122e10d6..5701571e864 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -1,5 +1,6 @@ import { recoveryRunContextColumn, recoveryRunVersionColumn } from "./run-context-read.js"; import { recoveryRunResultColumn } from "./run-result-read.js"; +import { readLegacyDispositionSource } from "./legacy-disposition-read.js"; import { isAgentAwaitingSetup } from "../../modules/agent-lifecycle/index.js"; import { onboardingFirstTaskAwaitingUserSql } from "./onboarding-first-task-wait.js"; import { isAiAuthenticationBlocked } from "../ai-auth-failure.js"; @@ -3793,9 +3794,9 @@ export function recoveryService( episode: LegacyDispositionEpisode, dispatchRunId?: string, ) { - const [run] = await db.select(recoveryRunFields).from(heartbeatRuns).where(and( - eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, issue.companyId), - )).limit(1); + const source = await readLegacyDispositionSource(db, { companyId: issue.companyId, issueId: issue.id, runId }); + if (source.kind === "skip") return source.decision; + const run = source.run; if (!run) return { kind: "skip" as const, reason: "missing_run" }; const context = parseObject(run.contextSnapshot); if ((readNonEmptyString(context.issueId) ?? readNonEmptyString(context.taskId)) !== issue.id) { @@ -3854,7 +3855,9 @@ export function recoveryService( } async function reconcileLegacyContinuation(runId: string) { - const [run] = await db.select(recoveryRunFields).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).limit(1); + const [run] = await db.select(recoveryRunFields).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, runId), not(eq(heartbeatRuns.runtimeMode, "native")), eq(heartbeatRuns.status, "succeeded"), + )).limit(1); if (!run || run.runtimeMode === "native" || run.status !== "succeeded") return "skipped" as const; const context = parseObject(run.contextSnapshot); const issueId = readNonEmptyString(context.issueId) ?? readNonEmptyString(context.taskId); From 2f637fe8c49b5ad0276c3d23e9003859d311ffb5 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 20:05:45 -0700 Subject: [PATCH 11/68] perf(tools): scope connection readiness assignment reads (#17541) Connection readiness reads current permitted connection identities without unrelated catalog rows or complete profile-rule payloads. Reuse profile precedence and selector logic, preserve full public/runtime summaries, and keep installed connections, identity checks, and gateway authorization current. Verified with 106 focused tests, the full build, and typecheck. The direct catalog inventory regression fails at 102 rows when the scope filter is removed and passes at two rows with the filter. One unrelated CI timing assertion differed by 1 ms; the exact local test and the single failed shard rerun passed on unchanged source. CI is green and exact-head Greptile is 5/5 at Medium impact with no unresolved comments. Co-Authored-By: Paperclip --- doc/DATABASE.md | 7 + .../tool-effective-assignments.test.ts | 49 +++++- server/src/services/connection-intents.ts | 4 +- .../tool-access/profiles/projection.ts | 10 +- .../tool-access/profiles/resolution.ts | 161 +++++++++++------- 5 files changed, 167 insertions(+), 64 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 17fbb47f400..aacf4f9f23d 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -471,6 +471,13 @@ bindings, selectors, exclusions, and installed connections on every call. The public effective-profile response continues to include complete tool schemas and annotations. +Connection readiness checks use a scoped assignment reader. It applies the same +current profile precedence, selectors, and exclusions to the candidate connection +IDs. The reader selects rule match fields and catalog identities. It does not +load rule conditions, catalog schemas, profile summaries, or the company-agent +list. Installed connections and later identity and gateway checks remain fresh. +Full effective-profile and effective-tool responses keep their complete shape. + ## Execution identity row locks Identity initialization, credential acquisition, and steering reconciliation lock diff --git a/server/src/__tests__/tool-effective-assignments.test.ts b/server/src/__tests__/tool-effective-assignments.test.ts index 2f69d9e62d0..13de5533fbf 100644 --- a/server/src/__tests__/tool-effective-assignments.test.ts +++ b/server/src/__tests__/tool-effective-assignments.test.ts @@ -1,10 +1,11 @@ -import { randomUUID } from "node:crypto"; +import { randomBytes, randomUUID } from "node:crypto"; import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; import { eq, sql } from "drizzle-orm"; import { agents, companies, createDb, toolApplications, toolCatalogEntries, toolConnections, toolProfileBindings, toolProfileEntries, toolProfiles, type Db, } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { toolAccessService } from "../services/tool-access.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; @@ -49,6 +50,11 @@ describeDatabase("effective tool assignments", () => { expect(assignments).toEqual({ ...full, allowedTools: full.allowedTools.map( ({ id, applicationId, connectionId, toolName }) => ({ id, applicationId, connectionId, toolName }), ) }); + const connections = await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id]); + expect(connections).toEqual({ + installedConnections: full.installedConnections, + allowedConnectionIds: [...new Set(full.allowedTools.filter(tool => tool.connectionId === f.connection.id).map(tool => tool.connectionId))], + }); return { service, full, assignments }; } @@ -130,4 +136,45 @@ describeDatabase("effective tool assignments", () => { expect(observed[1].bytes).toBeLessThan(observed[0].bytes / 100); expect(full.allowedTools[0].inputSchema).toEqual(f.read.inputSchema); }); + it("scopes catalog reads, omits rule payloads, and keeps current full definitions", async () => { + const f = await fixture(); + const bulk = randomBytes(128 * 1024).toString("base64"); + const [rule] = await db.insert(toolProfileEntries).values({ companyId: f.company.id, profileId: f.profile.id, + selectorType: "application", effect: "include", applicationId: f.application.id, conditions: { savedRuleMetadata: bulk } }).returning(); + const [unrelated] = await db.insert(toolConnections).values({ companyId: f.company.id, applicationId: f.application.id, + name: "Other connection", uid: randomUUID(), transport: "mcp_remote", status: "active", enabled: true }).returning(); + await db.insert(toolCatalogEntries).values(Array.from({ length: 100 }, (_, index) => ({ companyId: f.company.id, + applicationId: f.application.id, connectionId: unrelated.id, name: `other_${index}`, toolName: `other_${index}`, versionHash: `v${index}` }))); + const service = toolAccessService(db); + const before = await service.getEffectiveToolAssignmentsForAgent(f.company.id, f.agent.id); + expect(before.allowedTools).toHaveLength(102); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { toolCatalogEntries, toolProfileEntries }, capture); + let scoped: Awaited>; + try { scoped = await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id]); } + finally { capture.flush(); } + expect(Number(events.at(-1)?.completedQueries)).toBeGreaterThan(0); + expect(Number(events.at(-1)?.responseValueBytes)).toBeLessThan(16 * 1024); + const catalogQueries = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && event.table === "tool_catalog_entries" && event.operation === "select"); + expect(catalogQueries).toHaveLength(1); + expect(catalogQueries[0]).toMatchObject({ calls: 1, rows: 2 }); + expect(scoped!).toEqual({ allowedConnectionIds: [f.connection.id], installedConnections: before.installedConnections }); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + expect((await db.select().from(toolProfileEntries).where(eq(toolProfileEntries.id, rule.id)))[0]).toEqual(rule); + expect(await service.getEffectiveToolAssignmentsForAgent(f.company.id, f.agent.id)).toEqual(before); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [])).allowedConnectionIds).toEqual([]); + await db.delete(toolProfileBindings).where(eq(toolProfileBindings.profileId, f.profile.id)); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([]); + }); + + it("keeps other companies' catalog rows out of scoped assignments", async () => { + const f = await fixture(); + const other = await fixture(); + await db.update(toolProfiles).set({ defaultAction: "allow" }).where(eq(toolProfiles.id, f.profile.id)); + expect((await toolAccessService(db).getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [other.connection.id])).allowedConnectionIds).toEqual([]); + await expect(toolAccessService(db).getConnectionAssignmentsForAgent(f.company.id, other.agent.id, [f.connection.id])).rejects.toThrow("same company"); + }); + }); diff --git a/server/src/services/connection-intents.ts b/server/src/services/connection-intents.ts index a6d07331c0c..77dabac142b 100644 --- a/server/src/services/connection-intents.ts +++ b/server/src/services/connection-intents.ts @@ -291,9 +291,9 @@ export function connectionIntentService(db: Db) { && connection.status !== "archived" && connection.connectionPurpose !== "ai" ); if (matching.length === 0) return null; - const effective = await access.getEffectiveToolAssignmentsForAgent(input.companyId, input.agentId); + const effective = await access.getConnectionAssignmentsForAgent(input.companyId, input.agentId, matching.map(connection => connection.id)); const installedIds = new Set(effective.installedConnections.map((connection) => connection.id)); - const permittedIds = new Set(effective.allowedTools.map((tool) => tool.connectionId)); + const permittedIds = new Set(effective.allowedConnectionIds); const usable = (connection: ToolConnection | undefined) => connection && installedIds.has(connection.id) && permittedIds.has(connection.id) && connection.status === "active" && connection.enabled diff --git a/server/src/services/tool-access/profiles/projection.ts b/server/src/services/tool-access/profiles/projection.ts index 50b77ae3a79..53b92ef182f 100644 --- a/server/src/services/tool-access/profiles/projection.ts +++ b/server/src/services/tool-access/profiles/projection.ts @@ -80,9 +80,15 @@ export type ProfileApplication = Pick; +export type ProfileEntryMatch = Pick; + +export type ProfileCatalogMatch = Pick; + export function profileEntryMatchesCatalog( - entry: typeof toolProfileEntries.$inferSelect, - catalogEntry: ProfileCatalogEntry, + entry: ProfileEntryMatch, + catalogEntry: ProfileCatalogMatch, ): boolean { if (entry.selectorType === "application") return entry.applicationId === catalogEntry.applicationId; diff --git a/server/src/services/tool-access/profiles/resolution.ts b/server/src/services/tool-access/profiles/resolution.ts index 80aa6870681..3cc4d36df4b 100644 --- a/server/src/services/tool-access/profiles/resolution.ts +++ b/server/src/services/tool-access/profiles/resolution.ts @@ -3,20 +3,67 @@ import type { Db } from "@paperclipai/db"; import { agents, toolCatalogEntries, toolProfileBindings, toolProfileEntries, toolProfiles } from "@paperclipai/db"; import type { ToolCatalogEntry, ToolConnection, ToolProfileEffectiveSummary, ToolProfileWithDetails } from "@paperclipai/shared"; import { effectiveToolProfileBindings, profileIdsInBindingOrder } from "../../tool-profile-binding-precedence.js"; -import { buildProfileDetails, profileCatalogColumns, profileEntryMatchesCatalog, toProfileBinding, toProfileEntry, type ProfileCatalogEntry } from "./projection.js"; +import { buildProfileDetails, profileCatalogColumns, profileEntryMatchesCatalog, toProfileBinding, toProfileEntry, type ProfileCatalogEntry, type ProfileEntryMatch, type ProfileCatalogMatch } from "./projection.js"; type ProfileAssignmentDependencies = { assertOptionalAgent(companyId: string, agentId: string | null | undefined, label: string): Promise; resolveInstalledConnectionsForAgent(companyId: string, agentId: string): Promise; toCatalogEntry(row: typeof toolCatalogEntries.$inferSelect): ToolCatalogEntry; }; +function resolveCatalogAccess( + profiles: Array>, + entries: Array, + catalog: ProfileCatalogMatch[], +) { + const allowedCatalogIds = new Set(); + const allowedToolNames = new Set(); + const entriesByProfile = new Map(); + for (const entry of entries) { + const group = entriesByProfile.get(entry.profileId) ?? []; + group.push(entry); + entriesByProfile.set(entry.profileId, group); + } + for (const profile of profiles) { + const profileEntries = entriesByProfile.get(profile.id) ?? []; + const includes = profileEntries.filter( + (entry) => entry.effect === "include", + ); + const excludes = profileEntries.filter( + (entry) => entry.effect === "exclude", + ); + for (const catalogEntry of catalog) { + if ( + excludes.some((entry) => + profileEntryMatchesCatalog(entry, catalogEntry), + ) + ) + continue; + if ( + profile.defaultAction === "allow" || + includes.some((entry) => + profileEntryMatchesCatalog(entry, catalogEntry), + ) + ) { + allowedCatalogIds.add(catalogEntry.id); + allowedToolNames.add(catalogEntry.toolName); + } + } + for (const entry of includes.filter( + (item) => item.selectorType === "tool_name" && item.toolName, + )) { + const matchingExclude = excludes.some( + (item) => + item.selectorType === "tool_name" && + item.toolName === entry.toolName, + ); + if (!matchingExclude) allowedToolNames.add(entry.toolName!); + } + } + return { allowedCatalogIds, allowedToolNames }; +} + export function createToolProfileResolver(db: Db, { assertOptionalAgent, resolveInstalledConnectionsForAgent, toCatalogEntry }: ProfileAssignmentDependencies) { - async function resolveEffectiveProfiles( - companyId: string, - agentId: string, - loadCatalog: () => PromiseLike, - toAllowedTool: (entry: TEntry) => TAllowedTool, - ): Promise & { allowedTools: TAllowedTool[] }> { + async function readEffectiveProfileState(companyId: string, agentId: string) { await assertOptionalAgent( companyId, agentId, @@ -36,21 +83,8 @@ export function createToolProfileResolver(db: Db, { assertOptionalAgent, resolve binding.targetId === companyId) || (binding.targetType === "agent" && binding.targetId === agentId), ); - if (matchingBindings.length === 0) { - return { - agentId, - profiles: [], - entries: [], - bindings: [], - allowedTools: [], - allowedToolNames: [], - installedConnections: await resolveInstalledConnectionsForAgent( - companyId, - agentId, - ), - }; - } const candidateProfileIds = profileIdsInBindingOrder(matchingBindings); + if (candidateProfileIds.length === 0) return { bindings: [], activeProfiles: [] }; const candidateProfiles = await db .select() .from(toolProfiles) @@ -73,6 +107,16 @@ export function createToolProfileResolver(db: Db, { assertOptionalAgent, resolve .filter((profile): profile is typeof toolProfiles.$inferSelect => Boolean(profile && profile.status === "active"), ); + return { bindings, activeProfiles }; + } + + async function resolveEffectiveProfiles( + companyId: string, + agentId: string, + loadCatalog: () => PromiseLike, + toAllowedTool: (entry: TEntry) => TAllowedTool, + ): Promise & { allowedTools: TAllowedTool[] }> { + const { bindings, activeProfiles } = await readEffectiveProfileState(companyId, agentId); if (activeProfiles.length === 0) { return { agentId, @@ -114,44 +158,7 @@ export function createToolProfileResolver(db: Db, { assertOptionalAgent, resolve list.push(entry); entriesByProfile.set(entry.profileId, list); } - const allowedCatalogIds = new Set(); - const allowedToolNames = new Set(); - for (const profile of activeProfiles) { - const profileEntries = entriesByProfile.get(profile.id) ?? []; - const includes = profileEntries.filter( - (entry) => entry.effect === "include", - ); - const excludes = profileEntries.filter( - (entry) => entry.effect === "exclude", - ); - for (const catalogEntry of catalog) { - if ( - excludes.some((entry) => - profileEntryMatchesCatalog(entry, catalogEntry), - ) - ) - continue; - if ( - profile.defaultAction === "allow" || - includes.some((entry) => - profileEntryMatchesCatalog(entry, catalogEntry), - ) - ) { - allowedCatalogIds.add(catalogEntry.id); - allowedToolNames.add(catalogEntry.toolName); - } - } - for (const entry of includes.filter( - (item) => item.selectorType === "tool_name" && item.toolName, - )) { - const matchingExclude = excludes.some( - (item) => - item.selectorType === "tool_name" && - item.toolName === entry.toolName, - ); - if (!matchingExclude) allowedToolNames.add(entry.toolName!); - } - } + const { allowedCatalogIds, allowedToolNames } = resolveCatalogAccess(activeProfiles, entries, catalog); const agentIds = companyAgents.map((agent) => agent.id); const details: ToolProfileWithDetails[] = activeProfiles.map((profile) => buildProfileDetails({ @@ -182,7 +189,43 @@ export function createToolProfileResolver(db: Db, { assertOptionalAgent, resolve ), }; } + async function getConnectionAssignmentsForAgent(companyId: string, agentId: string, connectionIds: string[]) { + const { activeProfiles } = await readEffectiveProfileState(companyId, agentId); + if (activeProfiles.length === 0 || connectionIds.length === 0) { + return { allowedConnectionIds: [], installedConnections: await resolveInstalledConnectionsForAgent(companyId, agentId) }; + } + const [entries, catalog] = await Promise.all([ + db.select({ + profileId: toolProfileEntries.profileId, + effect: toolProfileEntries.effect, + selectorType: toolProfileEntries.selectorType, + applicationId: toolProfileEntries.applicationId, + connectionId: toolProfileEntries.connectionId, + catalogEntryId: toolProfileEntries.catalogEntryId, + toolName: toolProfileEntries.toolName, + riskLevel: toolProfileEntries.riskLevel, + }).from(toolProfileEntries).where(and( + eq(toolProfileEntries.companyId, companyId), inArray(toolProfileEntries.profileId, activeProfiles.map(profile => profile.id)), + )), + db.select({ + id: toolCatalogEntries.id, + applicationId: toolCatalogEntries.applicationId, + connectionId: toolCatalogEntries.connectionId, + toolName: toolCatalogEntries.toolName, + riskLevel: toolCatalogEntries.riskLevel, + }).from(toolCatalogEntries).where(and( + eq(toolCatalogEntries.companyId, companyId), eq(toolCatalogEntries.status, "active"), inArray(toolCatalogEntries.connectionId, connectionIds), + )), + ]); + const { allowedCatalogIds } = resolveCatalogAccess(activeProfiles, entries, catalog); + return { + allowedConnectionIds: [...new Set(catalog.filter(entry => allowedCatalogIds.has(entry.id)).map(entry => entry.connectionId))], + installedConnections: await resolveInstalledConnectionsForAgent(companyId, agentId), + }; + } + return { + getConnectionAssignmentsForAgent, getEffectiveProfilesForAgent: (companyId: string, agentId: string) => resolveEffectiveProfiles( companyId, From c8a6b4561bb679df1f507b819f9dc0f67e368473 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 20:18:23 -0700 Subject: [PATCH 12/68] perf(heartbeat): narrow native Stop evidence rechecks (#17544) Read current native Stop identity, acknowledgement, and process evidence without saved execution inputs, profiles, excerpts, or unrelated result data. Preserve the complete nested acknowledgement and every existing lease, remote-termination, process, ownership, and admission check. The physical-stop helper body is unchanged. Verified with 101 tests across the affected suites, the full build, and typecheck. The review fix passed 62 focused tests. Large valid saved inputs stay below 2 KB at the raw driver boundary; large malformed string and array results stay below 1 KB and remain stored. CI is green and exact-head Greptile is 5/5 at Medium impact with no unresolved comments. Co-Authored-By: Paperclip --- doc/native-workspace-finalization-recovery.md | 6 ++ .../src/services/acknowledged-native-stop.ts | 3 +- .../native-stop-evidence-read.test.ts | 93 +++++++++++++++++++ .../heartbeat/native-stop-evidence-read.ts | 20 ++++ server/src/services/heartbeat/run-control.ts | 3 +- 5 files changed, 123 insertions(+), 2 deletions(-) create mode 100644 server/src/services/heartbeat/native-stop-evidence-read.test.ts create mode 100644 server/src/services/heartbeat/native-stop-evidence-read.ts diff --git a/doc/native-workspace-finalization-recovery.md b/doc/native-workspace-finalization-recovery.md index 0875b177449..01e7a7031c4 100644 --- a/doc/native-workspace-finalization-recovery.md +++ b/doc/native-workspace-finalization-recovery.md @@ -499,3 +499,9 @@ for a native run. It keeps the same task-binding and native-finalization skip reasons, including the original nonblank binding values. Native input and history stay stored. Real legacy recovery still reads its complete context, profile, excerpts, and required failure evidence. + +The native Stop continuation recheck reads current run identity, the full Stop +acknowledgement receipt, and process identity. It does not load saved execution +input, runner profiles, excerpts, or unrelated result fields. Lease cleanup, +remote termination receipts, local process checks, task ownership, and admission +checks keep their existing behavior. Stored runs retain all of their data. diff --git a/server/src/services/acknowledged-native-stop.ts b/server/src/services/acknowledged-native-stop.ts index bb6b7fa74c3..9d07f7b61a7 100644 --- a/server/src/services/acknowledged-native-stop.ts +++ b/server/src/services/acknowledged-native-stop.ts @@ -37,7 +37,8 @@ export function isAcknowledgedNativeStop(run: Parameters) { if (!isAcknowledgedNativeStop(run)) return false; const leases = await db.select().from(environmentLeases).where(and( eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id), diff --git a/server/src/services/heartbeat/native-stop-evidence-read.test.ts b/server/src/services/heartbeat/native-stop-evidence-read.test.ts new file mode 100644 index 00000000000..d53a5d4c417 --- /dev/null +++ b/server/src/services/heartbeat/native-stop-evidence-read.test.ts @@ -0,0 +1,93 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { heartbeatRuns, issues, type Db } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { eq, sql } from "drizzle-orm"; +import { describe, expect, it, vi } from "vitest"; +import { isAcknowledgedNativeStop } from "../acknowledged-native-stop.js"; +import { callbacks, runControlPostgresSupport, setupRunControlDatabase } from "./__tests__/run-control-fixtures.js"; +import { readNativeStopExecutionEvidence } from "./native-stop-evidence-read.js"; +import { createHeartbeatRunControl } from "./run-control.js"; + +vi.mock("../local-service-supervisor.js", async importOriginal => ({ + ...await importOriginal(), + terminateLocalService: vi.fn(async () => undefined), +})); + +describe.skipIf(!runControlPostgresSupport.supported)("native Stop evidence reads", () => { + let db: Db; + const { fixture } = setupRunControlDatabase("native-stop-evidence-", value => { db = value; }); + + it("reads current receipts below the byte budget without changing saved data", async () => { + const { company, agent } = await fixture(); + const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Stop evidence fixture" }).returning(); + const id = randomUUID(), bulk = randomBytes(128 * 1024).toString("base64"); + const receipt = { schema: "paperclip.native-cancellation.v1", runId: id, companyId: company.id, issueId: issue.id, + scope: "run", reasonCode: "cancellation_run_only", dispatchState: "acknowledged", dispatched: true, + intentAuditId: randomUUID(), acknowledgementAuditId: randomUUID(), futureProof: { version: 2 } }; + const result = { nativeCancellation: receipt, cancelledByActorType: "user", cancelledByUserId: randomUUID(), + stdout: bulk, stderr: bulk, futureExecutionEvidence: { savedInput: bulk } }; + const [before] = await db.insert(heartbeatRuns).values({ id, companyId: company.id, agentId: agent.id, + nativeIssueId: issue.id, runtimeMode: "native", status: "cancelled", processPid: 123, processGroupId: 456, + contextSnapshot: { savedInput: bulk }, runnerProfileJson: { savedInput: bulk }, stdoutExcerpt: bulk, + stderrExcerpt: bulk, resultJson: result }).returning(); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + let evidence: Awaited>; + try { evidence = await readNativeStopExecutionEvidence(db, company.id, id); } + finally { capture.flush(); } + expect(evidence!).toEqual({ id, companyId: company.id, nativeIssueId: issue.id, status: "cancelled", processPid: 123, + processGroupId: 456, resultJson: { nativeCancellation: receipt, cancelledByActorType: "user", cancelledByUserId: result.cancelledByUserId } }); + expect(isAcknowledgedNativeStop(evidence!)).toBe(true); + expect(events.at(-1)).toMatchObject({ completedQueries: 1, groups: 1 }); + expect(Number(events.at(-1)?.responseValueBytes)).toBeLessThan(2048); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0]).toEqual(before); + await db.update(heartbeatRuns).set({ resultJson: { ...result, nativeCancellation: { ...receipt, dispatchState: "pending" } } }) + .where(eq(heartbeatRuns.id, id)); + expect(isAcknowledgedNativeStop((await readNativeStopExecutionEvidence(db, company.id, id))!)).toBe(false); + expect(await readNativeStopExecutionEvidence(db, randomUUID(), id)).toBeNull(); + expect(await readNativeStopExecutionEvidence(db, company.id, randomUUID())).toBeNull(); + }); + + it.each([null, [], false, 42, "not a receipt", { nativeCancellation: null }, { nativeCancellation: { dispatched: "true" } }])( + "keeps malformed receipts fail-closed: %j", async result => { + const { company, agent } = await fixture(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, + runtimeMode: "native", status: "cancelled" }).returning(); + await db.execute(sql`update ${heartbeatRuns} set result_json = ${JSON.stringify(result)}::jsonb where id = ${run.id}`); + expect(isAcknowledgedNativeStop((await readNativeStopExecutionEvidence(db, company.id, run.id))!)).toBe(false); + }, + ); + + it.each(["string", "array"])("does not transfer a large malformed %s result", async kind => { + const { company, agent } = await fixture(); + const bulk = randomBytes(128 * 1024).toString("base64"); + const saved = kind === "array" ? [bulk] : bulk; + const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, + runtimeMode: "native", status: "cancelled" }).returning(); + await db.execute(sql`update ${heartbeatRuns} set result_json = ${JSON.stringify(saved)}::jsonb where id = ${run.id}`); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { + const evidence = await readNativeStopExecutionEvidence(db, company.id, run.id); + expect(evidence?.resultJson).toBeNull(); + expect(isAcknowledgedNativeStop(evidence!)).toBe(false); + } finally { capture.flush(); } + expect(events.at(-1)).toMatchObject({ completedQueries: 1, groups: 1 }); + expect(Number(events.at(-1)?.responseValueBytes)).toBeLessThan(1024); + expect((await db.select({ result: heartbeatRuns.resultJson }).from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].result).toEqual(saved); + }); + + it("uses the scoped receipt reader for the native continuation recheck", async () => { + const { company, agent } = await fixture(); + const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Native continuation fixture" }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, + nativeIssueId: issue.id, runtimeMode: "native", status: "succeeded" }).returning(); + const deps = callbacks(db); + const control = createHeartbeatRunControl(db, deps); + await control.resumeRemoteStopComments(run); + expect(deps.getRun).not.toHaveBeenCalled(); + }); +}); diff --git a/server/src/services/heartbeat/native-stop-evidence-read.ts b/server/src/services/heartbeat/native-stop-evidence-read.ts new file mode 100644 index 00000000000..6ee776bf03f --- /dev/null +++ b/server/src/services/heartbeat/native-stop-evidence-read.ts @@ -0,0 +1,20 @@ +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { and, eq, sql } from "drizzle-orm"; + +export async function readNativeStopExecutionEvidence(db: Db, companyId: string, runId: string) { + const [run] = await db.select({ + id: heartbeatRuns.id, + companyId: heartbeatRuns.companyId, + nativeIssueId: heartbeatRuns.nativeIssueId, + status: heartbeatRuns.status, + processPid: heartbeatRuns.processPid, + processGroupId: heartbeatRuns.processGroupId, + resultJson: sql`case + when jsonb_typeof(${heartbeatRuns.resultJson}) = 'object' then ( + select coalesce(jsonb_object_agg(key, value), '{}'::jsonb) + from jsonb_each(${heartbeatRuns.resultJson}) + where key = any(array['nativeCancellation', 'cancelledByActorType', 'cancelledByUserId']) + ) else null end`, + }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId))).limit(1); + return run ?? null; +} diff --git a/server/src/services/heartbeat/run-control.ts b/server/src/services/heartbeat/run-control.ts index d7086c52508..b4ba5f0ddea 100644 --- a/server/src/services/heartbeat/run-control.ts +++ b/server/src/services/heartbeat/run-control.ts @@ -1,4 +1,5 @@ import { notifyEnvironmentCleanup } from "../environment-cleanup-notifications.js"; +import { readNativeStopExecutionEvidence } from "./native-stop-evidence-read.js"; import { readTaskRetryIntent, reconcileTaskChatRetryReceipts } from "../task-retry.js"; import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED, readNativeLocalProcessStop, recordedLocalExecutionStopped } from "../native-local-process-stop.js"; import { isHeartbeatRunTerminalStatus } from "./run-lifecycle.js"; @@ -522,7 +523,7 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont !(await recordedLocalExecutionStopped(db, run))) return; const issueId = run.nativeIssueId ?? (typeof run.contextSnapshot?.issueId === "string" ? run.contextSnapshot.issueId : null); if (!issueId) return; - const currentRun = run.runtimeMode === "native" ? await getRun(run.id, { includeExecutionEvidence: true }) : null; + const currentRun = run.runtimeMode === "native" ? await readNativeStopExecutionEvidence(db, run.companyId, run.id) : null; const [coordinator] = currentRun ? await db.select({ phase: nativeRunFinalizations.phase, leaseOwner: nativeRunFinalizations.leaseOwner }).from(nativeRunFinalizations).where(and( eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id), From 8a85d9d647b9081cf6b71c0635383edf542eed2e Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 21:04:36 -0700 Subject: [PATCH 13/68] perf(heartbeat): coalesce repeated native stranded releases (#17547) Skip repeated successful native release checks for the same unchanged source within one recovery pass. Preserve every saved request, failure retry, cursor update, source change, legacy path, and current admission gate. Verified with 201 focused tests, 32 full-control queue cases, build and typecheck. Exact-head CI is green, Greptile is 5/5 with Low impact, and no review threads remain unresolved. Co-Authored-By: Paperclip --- doc/native-workspace-finalization-recovery.md | 8 ++ server/src/services/heartbeat/queue.test.ts | 130 ++++++++++++++++-- server/src/services/heartbeat/queue.ts | 8 +- 3 files changed, 137 insertions(+), 9 deletions(-) diff --git a/doc/native-workspace-finalization-recovery.md b/doc/native-workspace-finalization-recovery.md index 01e7a7031c4..c8353239234 100644 --- a/doc/native-workspace-finalization-recovery.md +++ b/doc/native-workspace-finalization-recovery.md @@ -505,3 +505,11 @@ acknowledgement receipt, and process identity. It does not load saved execution input, runner profiles, excerpts, or unrelated result fields. Lease cleanup, remote termination receipts, local process checks, task ownership, and admission checks keep their existing behavior. Stored runs retain all of their data. + +One periodic stranded-message pass coalesces successful native release checks +for the same company, task, source run, and current source row version. The release +path already drains that +task's queue through current admission checks. Each saved request still advances +its recovery cursor. Rejected attempts, changed source runs, and updated source evidence remain eligible, +and the next pass checks current state again. Legacy per-request handling stays +unchanged. Pending work is awaited before a source is marked checked. diff --git a/server/src/services/heartbeat/queue.test.ts b/server/src/services/heartbeat/queue.test.ts index 5a31b5ee131..ef889747669 100644 --- a/server/src/services/heartbeat/queue.test.ts +++ b/server/src/services/heartbeat/queue.test.ts @@ -1,12 +1,14 @@ import { claimNativeSessionResumptions } from "../native-runtime/native-finalization-reconciler.js"; import type { Db } from "@paperclipai/db"; import { agents, agentWakeupRequests, caseHandoffs, cases, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issueComments, issues, nativeRunFinalizations } from "@paperclipai/db"; -import { eq, sql } from "drizzle-orm"; +import { eq, inArray, sql } from "drizzle-orm"; import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { createPostgresWakeQueueAdapter } from "../../modules/wake-queue/adapters/postgres.js"; import { createReleaseIssueExecution } from "../../modules/wake-queue/application/use-cases.js"; import { createHeartbeatQueue, type HeartbeatQueueDependencies } from "./queue.js"; +import { createHeartbeatRunControl } from "./run-control.js"; +import { callbacks as runControlCallbacks } from "./__tests__/run-control-fixtures.js"; import { createHeartbeatQueueTestDependencies as callbacks } from "../../__tests__/helpers/heartbeat-queue.js"; @@ -113,6 +115,116 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => { return { ...seeded, recipient, pending }; } + async function addStrandedMessages(seeded: Awaited>) { + const messages = [seeded.pending]; + for (let index = 0; index < 2; index++) { + const [comment] = await db.insert(issueComments).values({ companyId: seeded.company.id, issueId: seeded.issue.id, + body: `Additional saved feedback ${index}`, authorAgentId: seeded.agent.id }).returning(); + const [wake] = await db.insert(agentWakeupRequests).values({ companyId: seeded.company.id, agentId: seeded.recipient.id, + source: "automation", reason: "issue_execution_deferred", status: "deferred_issue_execution", + payload: { issueId: seeded.issue.id, _paperclipWakeContext: { issueId: seeded.issue.id, + wakeReason: "issue_comment_mentioned", wakeCommentIds: [comment.id] } } }).returning(); + messages.push(wake); + } + await db.update(agentWakeupRequests).set({ updatedAt: new Date(0) }).where(inArray(agentWakeupRequests.id, messages.map(row => row.id))); + return messages; + } + + async function expectSavedMessages(messages: Awaited>) { + const rows = await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, messages.map(row => row.id))); + expect(rows).toHaveLength(messages.length); + for (const before of messages) { + const row = rows.find(row => row.id === before.id)!; + expect(row).toMatchObject({ payload: before.payload, status: "deferred_issue_execution", runId: null, + finishedAt: null, error: null }); + expect(row.updatedAt.getTime()).toBeGreaterThan(0); + } + } + + it.each(["legacy", "native"])("keeps saved %s messages while avoiding repeated native source release", async runtimeMode => { + const seeded = await strandedMention(runtimeMode); + const messages = await addStrandedMessages(seeded); + const deps = callbacks(db), queue = createHeartbeatQueue(db, deps); + await queue.resumeQueuedRuns(); + expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(runtimeMode === "native" ? 1 : 3); + await expectSavedMessages(messages); + await queue.resumeQueuedRuns(); + expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(runtimeMode === "native" ? 2 : 6); + expect(deps.executeRun).not.toHaveBeenCalled(); + }); + + it("retries a rejected native release on the next saved message", async () => { + const seeded = await strandedMention("native"), messages = await addStrandedMessages(seeded); + const deps = callbacks(db); + deps.releaseIssueExecutionAndPromote.mockRejectedValueOnce(new Error("Fixture release remains pending")); + await createHeartbeatQueue(db, deps).resumeQueuedRuns(); + expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(2); + await expectSavedMessages(messages); + expect(deps.executeRun).not.toHaveBeenCalled(); + }); + + it("joins the native release before it suppresses another message's duplicate check", async () => { + const seeded = await strandedMention("native"), messages = await addStrandedMessages(seeded); + const deps = callbacks(db); + let finish!: () => void; + deps.releaseIssueExecutionAndPromote.mockImplementationOnce(() => new Promise(resolve => { finish = resolve; })); + let settled = false; + const pending = createHeartbeatQueue(db, deps).resumeQueuedRuns().then(() => { settled = true; }); + try { + await vi.waitFor(() => expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(1)); + expect(settled).toBe(false); + } finally { finish?.(); } + await pending; + expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(1); + await expectSavedMessages(messages); + }); + + it("rechecks a different native source that completes during the pass", async () => { + const seeded = await strandedMention("native"), messages = await addStrandedMessages(seeded); + await db.update(heartbeatRuns).set({ createdAt: new Date(0) }).where(eq(heartbeatRuns.id, seeded.run.id)); + const deps = callbacks(db); + let nextId: string | undefined; + deps.releaseIssueExecutionAndPromote.mockImplementationOnce(async () => { + const [next] = await db.insert(heartbeatRuns).values({ companyId: seeded.company.id, agentId: seeded.agent.id, + runtimeMode: "native", nativeIssueId: seeded.issue.id, status: "succeeded", finishedAt: new Date(), + contextSnapshot: { issueId: seeded.issue.id } }).returning(); + nextId = next.id; + }); + await createHeartbeatQueue(db, deps).resumeQueuedRuns(); + expect(deps.releaseIssueExecutionAndPromote.mock.calls.map(([run]) => run.id)).toEqual([seeded.run.id, nextId]); + await expectSavedMessages(messages); + }); + + it("rechecks changed native evidence on the same source run", async () => { + const seeded = await strandedMention("native"), messages = await addStrandedMessages(seeded); + const deps = callbacks(db); + deps.releaseIssueExecutionAndPromote.mockImplementationOnce(async () => { + await db.update(heartbeatRuns).set({ resultJson: { observedReceipt: "new evidence" } }).where(eq(heartbeatRuns.id, seeded.run.id)); + }); + await createHeartbeatQueue(db, deps).resumeQueuedRuns(); + expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(2); + expect(deps.releaseIssueExecutionAndPromote.mock.calls.map(([run]) => run.id)).toEqual([seeded.run.id, seeded.run.id]); + await expectSavedMessages(messages); + }); + + it("checks each unrelated native task in the same pass", async () => { + const seeded = await strandedMention("native"), firstMessages = await addStrandedMessages(seeded); + const [issue] = await db.insert(issues).values({ companyId: seeded.company.id, title: "Another saved task", status: "todo", assigneeAgentId: seeded.agent.id }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: seeded.company.id, agentId: seeded.agent.id, + runtimeMode: "native", nativeIssueId: issue.id, status: "succeeded", finishedAt: new Date(), contextSnapshot: { issueId: issue.id } }).returning(); + const [comment] = await db.insert(issueComments).values({ companyId: seeded.company.id, issueId: issue.id, + body: "Other task feedback", authorAgentId: seeded.agent.id }).returning(); + const [pending] = await db.insert(agentWakeupRequests).values({ companyId: seeded.company.id, agentId: seeded.recipient.id, + source: "automation", reason: "issue_execution_deferred", status: "deferred_issue_execution", + payload: { issueId: issue.id, _paperclipWakeContext: { issueId: issue.id, wakeReason: "issue_comment_mentioned", wakeCommentIds: [comment.id] } } }).returning(); + const secondMessages = await addStrandedMessages({ ...seeded, issue, run, pending }); + const deps = callbacks(db); + await createHeartbeatQueue(db, deps).resumeQueuedRuns(); + expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(2); + expect(new Set(deps.releaseIssueExecutionAndPromote.mock.calls.map(([row]) => row.id))).toEqual(new Set([seeded.run.id, run.id])); + await expectSavedMessages([...firstMessages, ...secondMessages]); + }); + it("rechecks a handoff's assignment version under the issue claim lock", async () => { const { company, agent, issue, wake, run } = await fixture("queued"); const [record] = await db.insert(cases).values({ companyId: company.id, caseNumber: 1, identifier: "QM-CASE-1", @@ -137,7 +249,9 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => { }); it.each(["legacy", "native"])("periodic recovery preserves a paused recipient through the real %s release adapter", async runtimeMode => { - const { company, issue, recipient, pending } = await strandedMention(runtimeMode); + const seeded = await strandedMention(runtimeMode); + const { company, issue, recipient } = seeded; + const messages = await addStrandedMessages(seeded); await db.update(agents).set({ status: "paused" }).where(eq(agents.id, recipient.id)); await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, issue.id)); const deps = callbacks(db); @@ -146,16 +260,16 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => { getRoutineEnv: async () => ({ routineId: null, env: null, responsibleUserId: null }), resolveSessionBeforeForWakeup: async () => null, }), recovery: { escalateStrandedAssignedIssue: async () => undefined, escalateStrandedRecoveryIssueInPlace: async () => undefined } }); - deps.releaseIssueExecutionAndPromote.mockImplementation(async (run, options) => { - await release({ companyId: run.companyId, runId: run.id, now: new Date(), ...options }); - }); + const controlDeps = runControlCallbacks(db); + vi.mocked(controlDeps.wakeQueue.releaseIssueExecution).mockImplementation(release); + const control = createHeartbeatRunControl(db, controlDeps); + deps.releaseIssueExecutionAndPromote.mockImplementation(control.releaseIssueExecutionAndPromote); for (let pass = 0; pass < 2; pass++) { await createHeartbeatQueue(db, deps).resumeQueuedRuns(); - expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, pending.id)))[0]) - .toMatchObject({ status: "deferred_issue_execution", runId: null, finishedAt: null, error: null, payload: pending.payload }); + await expectSavedMessages(messages); expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, company.id))).toHaveLength(1); } - expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(2); + expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(runtimeMode === "native" ? 2 : 6); expect(deps.executeRun).not.toHaveBeenCalled(); }); diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index 515bd683fb3..35ff18e6e88 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -1834,6 +1834,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende AND live_issue_run.status IN ('queued', 'running', 'scheduled_retry'))`, cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined)) .orderBy(asc(agentWakeupRequests.updatedAt)).limit(50); + const releasedNativeSources = new Set(); for (const { wake } of strandedQueues) { if (!queuedCommentIdsFromWakePayload(wake.payload).length && !await readQueuedInteractionResponse(db, wake.companyId, String(wake.payload?.issueId), wake.payload)) continue; @@ -1845,6 +1846,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende startedAt: heartbeatRuns.startedAt, runtimeMode: heartbeatRuns.runtimeMode, errorCode: heartbeatRuns.errorCode, + sourceVersion: sql`${heartbeatRuns}.xmin::text`, }).from(heartbeatRuns).where(and( eq(heartbeatRuns.companyId, wake.companyId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${String(wake.payload?.issueId)}`, @@ -1855,6 +1857,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende )); if (!latest || !latest.finishedAt || !isHeartbeatRunTerminalStatus(latest.status)) continue; if (latest.runtimeMode !== "legacy" && latest.runtimeMode !== "native") continue; + const releaseSource = `${latest.companyId}:${String(wake.payload?.issueId)}:${latest.id}:${latest.sourceVersion}`; + if (latest.runtimeMode === "native" && releasedNativeSources.has(releaseSource)) continue; if (latest.runtimeMode === "legacy") { const cancelledAdmission = latest.status === "cancelled" && !latest.startedAt && latest.errorCode === "execution_reconciliation_required"; @@ -1864,7 +1868,9 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende }); } } - await releaseIssueExecutionAndPromote(latest, { suppressImmediateRecovery: true, resumeStrandedQueue: true }).catch(err => { + await releaseIssueExecutionAndPromote(latest, { suppressImmediateRecovery: true, resumeStrandedQueue: true }).then(() => { + if (latest.runtimeMode === "native") releasedNativeSources.add(releaseSource); + }).catch(err => { logger.warn({ err, queueId: wake.id }, "failed to promote stranded comments"); }); } From 2a9d527c68ab1d2e61b8408eed9a194603bddfca Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 24 Oct 2026 21:05:45 -0700 Subject: [PATCH 14/68] fix: preserve native startup during controller restart (#17511) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The native runtime keeps a durable runner alive across controller restarts. > - A restart can begin while a provider session is still opening. > - The old controller fenced read-only run admission before it could publish that prepared session. > - Admission failure then closed the runner that shutdown needed to detach. > - This pull request lets only the existing startup finish admission within the bounded shutdown wait. > - The controller detaches that exact session before it can submit a turn. New work and state changes stay fenced. ## Linked Issues or Issue Description **What happened?** A controller restart during native provider startup could close the prepared runner with `native control-plane run admission failed`. Recovery then found no running process and placed the task in a reconciliation hold. The same admission path follows both fresh startup and replacement after a failed provider resume. **Expected behavior** Shutdown must detach the existing prepared session so the next controller can recover it through the normal identity checks. Shutdown must not admit a new executor, submit a turn, or override an explicit Stop. **Steps to reproduce** 1. Delay a native backend while it opens its provider session. 2. Request controller restart detachment for that run. 3. Let backend startup finish before the 30-second deadline. 4. Observe that read-only admission fails and shutdown reports the run as inactive instead of detached. **Additional context** Related work: #17073 handles remote process-monitor classification; #17459 handles macOS process fingerprints; #17097 covers broader hot upgrade recovery. This change fixes the startup admission fence. It does not relax process identity or provider launch-profile checks. ## What Changed - Separate the read-only admission guard from the guard on control-plane state changes. - Let shutdown await only the startup already registered for the run. Close its admission window when the wait ends. - Reject a new execution after its run is fenced for shutdown. - Detach the prepared session and exit before runtime-question attachment or provider turn submission. - If Stop is pending before or after publication, retain its exact cancellation acknowledgment and cleanup owner across authority reads and runtime-question attachment. - Share one 30-second admission/Stop-settlement deadline across the restart sweep. Report an unsettled run as pending while healthy runs continue hot-restart adoption. - Cover the real runtime and admission path, deadline expiry, new-start rejection, persisted binding checks, and write fences. - Document the bounded startup handoff in the restart ownership contract. ## Verification - Red: the new actual-runtime/admission regression failed on the previous code. Shutdown reported the run as inactive and did not detach it. - Red: review regressions reproduced detach before a post-publication Stop acknowledgment and an actual hot-restart preparation error when startup Stop cleanup timed out. - Green on `e10d0cb238`: 18 startup/restart tests passed, including Stop during authority reads and runtime-question attachment, the real recovery preparation and shutdown coordinator with a stalled Stop plus healthy peer, and the shared sweep deadline. The earlier 36-case Stop/restart selection also passed before this follow-up. - Both complete changed test files previously passed on head `8e9c2610fe`: 857 tests. They used real PostgreSQL 18 with a fresh database per fixture. An earlier local harness attempt reused one database and hit duplicate fixture keys; that harness was corrected before this complete pass. - Real PostgreSQL: 7 admission and revocation tests passed. These cover wrong runner identity, persisted binding mismatch, checkpoint/result rejection, and in-flight write revocation. The local test harness used a fresh Docker PostgreSQL 18 instance because macOS embedded Postgres exhausted the host semaphore limit. Only database provisioning was replaced; assertions and production port code were unchanged. - Transport safety acceptance: the current source transport controlled a real Linux ARM runner and a deterministic Codex test provider in a disposable Debian container. After startup, detach and generic cleanup preserved the process. A new controller authenticated the same PID and provider session. The provider log had exactly one session start and no turn start. Duplicate launch was forbidden, the retained workspace was unchanged, and no stop or suspend command was issued. - The Linux acceptance check tests the shared native transport safety contract. It is not a live Claude response, a hosted Daytona run, or recovery of the original task. Those remain unverified. No original task, sandbox, credentials, or permissions were changed. - `pnpm -r typecheck` and `pnpm build` passed before the review follow-up. Server typecheck passed on current head `e10d0cb238`; current-head CI also passed the full typecheck and build gates. - The full local `pnpm test:run` command was attempted, then stopped after repeated embedded PostgreSQL bootstrap setup failures. That broad local run is incomplete. The current-head four-file Docker PostgreSQL run passed 879 tests, with one existing warm-copy `claim-uncertain` case timing out and recovery failing to import because the temporary provisioning mock omitted a support export. The harness is corrected and affected cases are being rechecked. The timeout is unexplained; it is not being labeled an infrastructure failure. No assertions are skipped by the database provisioning replacement. - All 55 clean-host CI checks passed on current head `e10d0cb238` (2 conditional skips), including all general test shards and native runner verification. Greptile reviewed that exact head at 5/5 with no findings. The previous Paperclip review findings are addressed in [the review response](https://github.com/paperclipai/paperclip/pull/17511#issuecomment-6391520242); its new advisory review remains queued. ## Risks - Admission must stay read-only. All writes retain the original synchronous controller fence, and both admission checks still validate the exact persisted binding. - Startup that exceeds the 30-second shutdown wait still fails closed. It cannot claim a successful detach or start a turn. - No schema migration or provider configuration change is included. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools, code execution, and test execution. The exact runtime model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 12 + .../native-session-executor.test.ts | 322 +++++++++++++++++- .../native-runtime/native-session-executor.ts | 79 ++++- .../paperclip-control-plane-port.test.ts | 32 ++ .../paperclip-control-plane-port.ts | 8 +- 5 files changed, 435 insertions(+), 18 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 8bc8886b075..b4d63b5687f 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1683,6 +1683,18 @@ controller must recover it through the authenticated ownership checks. This preserves active work and queued messages without treating a server restart as user cancellation. +During controller shutdown, already registered provider startups share one +30-second sweep deadline to finish read-only run admission and publish their exact +sessions for detachment. Its company, run, session, runner and completion-contract binding +must still match persisted authority. The draining controller cannot checkpoint, +publish events, submit a turn or settle the run. A new startup after the shutdown +fence, or admission after the window expires, remains rejected. A Stop pending +before or after session publication keeps cancellation and cleanup with the +current owner. Shutdown waits for that owner only until the same sweep deadline; +it does not detach the session or submit a prompt while Stop is pending. An +unsettled owner remains explicitly pending for reconciliation. Its timeout does +not abort detachment of healthy sessions or trigger a global graceful drain. + Recovery restores a structured result proposal from the durable run events when its append survived but the active turn's session checkpoint did not. Advancing the replay cursor must preserve that proposal for the exact run, session, source, diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index ad754e8e84e..bfb1f4f309f 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -33,6 +33,7 @@ import { dirname, join } from "node:path"; import { heartbeatRuns, heartbeatRunEvents, + issueThreadInteractions, issues, nativeRunFinalizations, nativeRunResults, @@ -49,6 +50,7 @@ import { validatePrpStructuredRunResult, validatePrpEvent, parseNativeExecutionInput, + executeNativeSession, type NativeExecutionInputV1, type NativeExecutionInputV3, type NativeExecutionInput, @@ -343,6 +345,7 @@ import { executePaperclipNativeSession, detachNativeSessionsForRestart, NativeControllerDetachedForRestartError, + NativeCancellationPendingRecoveryError, getNativeSessionSteeringState, NativeSessionSteeringError, assertRemoteRunnerBuildMetadata, @@ -5695,6 +5698,10 @@ function leaseDb( agentId: boundExecution.binding.agentId, companyId: boundExecution.binding.companyId, nativeIssueId: boundExecution.binding.issueId, + nativeSessionId: boundExecution.session.normalizedSessionId ?? `session-${boundExecution.binding.runId}`, + runnerInstanceId: "runner", + completionContractId: boundExecution.completionContract.id, + completionContractSha256: boundExecution.completionContract.sha256, resultJson: runResultJson, runnerProfileJson, runtimeMode: "native", @@ -5892,6 +5899,306 @@ describe("native startup cancellation fence", () => { }); describe("native startup restart detachment", () => { + it("uses one admission deadline for the whole restart sweep", async () => { + const inputs = ["first", "second"].map(label => { + const input = structuredClone(execution); + input.binding.runId = `restart-shared-deadline-${label}`; + input.binding.agentId = `restart-shared-deadline-agent-${label}`; + return input; + }); + const releases: Array<() => void> = []; + const ready: Array> = []; + state.execute.mockReset(); + for (const _input of inputs) { + let started!: () => void; + ready.push(new Promise(resolve => { started = resolve; })); + const opening = new Promise(resolve => { releases.push(resolve); }); + state.execute.mockImplementationOnce(async () => { + started(); await opening; + throw new Error("startup fixture released"); + }); + } + const outcomes = inputs.map(input => executePaperclipNativeSession({ + db: leaseDb(input), execution: input, runnerInstanceId: "runner", + }).catch(error => error)); + await Promise.all(ready); + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout", "Date"] }); + const finished = vi.fn(); + const runIds = inputs.map(input => input.binding.runId); + const shutdown = detachNativeSessionsForRestart(runIds).then(finished); + try { + await vi.advanceTimersByTimeAsync(30_000); + expect(finished).toHaveBeenCalledWith({ detachedRunIds: [], inactiveRunIds: [], + unsupportedRunIds: [], pendingRunIds: runIds }); + } finally { + vi.useRealTimers(); releases.forEach(release => release()); + await Promise.all(outcomes); await shutdown; + } + }); + + it.each(["authority", "questions"])("keeps a post-publication Stop owner through %s attachment and restart", async (seam) => { + const restarting = structuredClone(execution); + restarting.binding.runId = `restart-stop-after-publication-${seam}`; + let published!: () => void, releaseRead!: () => void, acknowledgeStop!: () => void; + let cleaning!: () => void, releaseCleanup!: () => void; + const publication = new Promise(resolve => { published = resolve; }); + const read = new Promise(resolve => { releaseRead = resolve; }); + const receipt = new Promise(resolve => { acknowledgeStop = resolve; }); + const cleanupStarted = new Promise(resolve => { cleaning = resolve; }); + const cleanup = new Promise(resolve => { releaseCleanup = resolve; }); + const detach = vi.fn(async () => undefined); + const cancel = vi.fn(() => ({ cleanup: receipt })); + const submit = vi.fn(); + const db = leaseDb(restarting); + const select = db.select.bind(db); + db.select = ((fields?: Record) => { + if (seam === "authority" && fields && Object.keys(fields).length === 2 && "status" in fields && "resultJson" in fields) { + return { from: () => ({ where: () => ({ limit: () => { + published(); + return read.then(() => [{ status: "running", resultJson: {} }]); + } }) }) }; + } + if (seam === "questions" && !fields) { + const query = select(); + return { from: (table: unknown) => table === issueThreadInteractions + ? { where: () => ({ orderBy: () => { published(); return read.then(() => []); } }) } + : query.from(table as never) }; + } + return select(fields as never); + }) as typeof db.select; + state.execute.mockReset().mockImplementationOnce(async options => { + try { + await options.onSession({ detachControllerForRestart: detach, cancel, resolveRuntimeRequest: vi.fn() }); + submit(); + } finally { + cleaning(); + await cleanup; + await options.onSession(null); + } + }); + const outcome = executePaperclipNativeSession({ db, execution: restarting, runnerInstanceId: "runner" }).catch(error => error); + await publication; + const stopping = cancelNativeSession(restarting.binding.runId, "operator Stop after publication"); + const completed = vi.fn(); + const shutdown = detachNativeSessionsForRestart([restarting.binding.runId]).then(completed); + try { + releaseRead(); + await new Promise(resolve => setImmediate(resolve)); + expect(cancel).toHaveBeenCalledOnce(); + expect(detach).not.toHaveBeenCalled(); + expect(completed).not.toHaveBeenCalled(); + // Admission must not unwind into generic close while Stop awaits its ACK. + const cleanupObserved = vi.fn(); + void cleanupStarted.then(cleanupObserved); + await new Promise(resolve => setImmediate(resolve)); + expect(cleanupObserved).not.toHaveBeenCalled(); + acknowledgeStop(); + expect(await stopping).toBe(true); + await cleanupStarted; + expect(completed).not.toHaveBeenCalled(); + releaseCleanup(); + await outcome; + await shutdown; + expect(completed).toHaveBeenCalledWith(expect.objectContaining({ + detachedRunIds: [], inactiveRunIds: [restarting.binding.runId], pendingRunIds: [], + })); + expect(detach).not.toHaveBeenCalled(); + expect(submit).not.toHaveBeenCalled(); + } finally { + releaseRead(); acknowledgeStop(); releaseCleanup(); + await stopping; await outcome; await shutdown; + } + }); + + it.each(["startup", "active"])("keeps healthy hot-restart adoption when another %s Stop cleanup exceeds the sweep deadline", async (mode) => { + const { coordinateHeartbeatSchedulerShutdown } = await import("../../shutdown.js"); + const { createHeartbeatRecovery } = await import("../heartbeat/recovery.js"); + const { writeHotRestartIntent, readHotRestartIntent } = await import("../hot-restart.js"); + const home = await mkdtemp(join(tmpdir(), "native-restart-stop-sweep-")); + const previousHome = process.env.PAPERCLIP_HOME; + process.env.PAPERCLIP_HOME = home; + await writeHotRestartIntent({ previousServerPid: process.pid, previousServerIdentity: "test-controller" }); + const stoppingExecution = structuredClone(execution); + stoppingExecution.binding.runId = `restart-${mode}-stop-timeout`; + stoppingExecution.session.normalizedSessionId = "d371f9bb-6c46-44c6-9e3a-026fa9be0153"; + const healthyExecution = structuredClone(execution); + healthyExecution.binding.runId = `restart-healthy-with-${mode}-stopping-peer`; + healthyExecution.session.normalizedSessionId = "a8fa0cb8-9cf1-46b0-96f0-78518b15847e"; + healthyExecution.binding.agentId = "healthy-restart-agent"; + let stoppedReady!: () => void, healthyReady!: () => void; + let releaseStopped!: () => void, releaseHealthy!: () => void, opening!: () => void, open!: () => void; + const openingStarted = new Promise(resolve => { opening = resolve; }); + const openingGate = new Promise(resolve => { open = resolve; }); + const stopPublished = new Promise(resolve => { stoppedReady = resolve; }); + const healthyPublished = new Promise(resolve => { healthyReady = resolve; }); + const stoppedLifetime = new Promise(resolve => { releaseStopped = resolve; }); + const healthyLifetime = new Promise(resolve => { releaseHealthy = resolve; }); + const stoppedDetach = vi.fn(async () => undefined); + const healthyDetach = vi.fn(async () => undefined); + state.execute.mockReset() + .mockImplementationOnce(async options => { + opening(); await openingGate; + try { + await options.onSession({ detachControllerForRestart: stoppedDetach, cancel: () => ({ cleanup: Promise.resolve() }) }); + } finally { + stoppedReady(); await stoppedLifetime; + await options.onSession(null); + } + throw new Error("stopped owner settled"); + }) + .mockImplementationOnce(async options => { + await options.onSession({ detachControllerForRestart: healthyDetach }); + healthyReady(); await healthyLifetime; + await options.onSession(null); + throw new Error("healthy controller detached"); + }); + const stoppedOutcome = executePaperclipNativeSession({ db: leaseDb(stoppingExecution), execution: stoppingExecution, runnerInstanceId: "runner" }).catch(error => error); + await openingStarted; + const stopping = mode === "startup" ? cancelNativeSession(stoppingExecution.binding.runId, "Stop while opening") : null; + open(); + await stopPublished; + await stopping; + const healthyOutcome = executePaperclipNativeSession({ db: leaseDb(healthyExecution), execution: healthyExecution, runnerInstanceId: "runner" }).catch(error => error); + await healthyPublished; + if (mode === "active") await cancelNativeSession(stoppingExecution.binding.runId, "operator Stop after active admission"); + const snapshotRows = [stoppingExecution, healthyExecution].map(item => ({ + adapterType: "paperclip_runner", adapterConfig: {}, + run: { id: item.binding.runId, companyId: item.binding.companyId, agentId: item.binding.agentId, + runtimeMode: "native", nativeSessionId: item.session.normalizedSessionId, status: "running", contextSnapshot: {} }, + })); + const snapshotDb = { select: () => ({ from: () => ({ innerJoin: () => ({ where: async () => snapshotRows }) }) }) } as unknown as Db; + const { logger } = await import("../../middleware/logger.js"); + const logged = vi.spyOn(logger, "info").mockImplementation(() => undefined); + const recovery = createHeartbeatRecovery(snapshotDb, { + enterShutdown: vi.fn(), appendRunEvent: vi.fn(async () => undefined), + } as unknown as import("../heartbeat/recovery.js").HeartbeatRecoveryDependencies); + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout", "Date"] }); + const shutdown = coordinateHeartbeatSchedulerShutdown({ signal: "SIGTERM", + waitForHeartbeatSchedulerIdle: async () => undefined, + prepareHotRestartShutdown: recovery.prepareHotRestartShutdown, + }); + try { + // Wait for the real snapshot I/O before advancing its bounded sweep. + await vi.waitFor(() => expect(vi.getTimerCount()).toBeGreaterThan(0)); + await vi.advanceTimersByTimeAsync(30_000); + expect(await shutdown).toMatchObject({ hotRestart: { mode: "hot_restart", skipDrain: true }, preparationError: null }); + expect(logged).toHaveBeenCalledWith(expect.objectContaining({ detachedNativeSessions: { + detachedRunIds: [healthyExecution.binding.runId], inactiveRunIds: [], unsupportedRunIds: [], + pendingRunIds: [stoppingExecution.binding.runId], + } }), expect.any(String)); + expect((await readHotRestartIntent())?.shutdownSnapshot?.activeRuns.map(run => run.runId)) + .toEqual([stoppingExecution.binding.runId, healthyExecution.binding.runId]); + expect(stoppedDetach).not.toHaveBeenCalled(); + expect(healthyDetach).toHaveBeenCalledOnce(); + } finally { + vi.useRealTimers(); releaseStopped(); releaseHealthy(); + await stoppedOutcome; await healthyOutcome; await shutdown; + logged.mockRestore(); + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousHome; + await rm(home, { recursive: true, force: true }); + } + }); + + it.each(["detach", "expired", "stop"])("bounds read-only startup admission during drain: %s", async (mode) => { + const expired = mode === "expired"; + const stopped = mode === "stop"; + const restarting = structuredClone(execution); + restarting.binding.runId = `restart-real-admission-${mode}`; + restarting.session.normalizedSessionId = "restart-real-admission-session"; + const identity = { ...restarting.binding, sessionId: restarting.session.normalizedSessionId }; + let release!: () => void, started!: () => void; + const opening = new Promise(resolve => { release = resolve; }); + const pending = new Promise(resolve => { started = resolve; }); + const detach = vi.fn(async () => undefined); + const close = vi.fn(async () => undefined); + const startTurn = vi.fn(); + let acknowledgeStop!: () => void; + const stopReceipt = new Promise(resolve => { acknowledgeStop = resolve; }); + const cancel = vi.fn(() => ({ cleanup: stopReceipt })); + const backend = { + descriptor: async () => ({ kind: "mock" as const, name: "startup-drain", version: "1", + capabilities: { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }, + runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } }), + openSession: vi.fn(async () => { + started(); await opening; + return { identity: () => identity, detachControllerForRestart: detach, close, startTurn, cancel, + snapshot: async () => ({ sessionId: identity.sessionId, identity, backendKind: "mock" as const }), + }; + }), + }; + let runtimeError: unknown; + state.execute.mockReset().mockImplementationOnce(async options => { + try { return await executeNativeSession({ ...options, onSessionAdmission: undefined }); } catch (error) { runtimeError = error; throw error; } + }); + const outcome = executePaperclipNativeSession({ db: leaseDb(restarting), execution: restarting, + runnerInstanceId: "runner", backend: backend as never }).catch(error => error); + await Promise.race([pending, outcome.then(error => { throw runtimeError ?? error; })]); + const stopping = stopped ? cancelNativeSession(restarting.binding.runId, "operator Stop during startup") : null; + if (expired) vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] }); + const shutdown = detachNativeSessionsForRestart([restarting.binding.runId]).catch(error => error); + try { + if (expired) { + await vi.advanceTimersByTimeAsync(30_000); + expect(await shutdown).toMatchObject({ pendingRunIds: [restarting.binding.runId], detachedRunIds: [] }); + vi.useRealTimers(); + } else { + await new Promise(resolve => setImmediate(resolve)); + } + release(); + if (stopped) { + await vi.waitFor(() => expect(cancel).toHaveBeenCalledOnce()); + expect(detach).not.toHaveBeenCalled(); + acknowledgeStop(); + expect(await stopping).toBe(true); + } + expect(await outcome).toBeInstanceOf(NativeControllerDetachedForRestartError); + if (stopped) { + expect(await shutdown).toMatchObject({ detachedRunIds: [], inactiveRunIds: [restarting.binding.runId] }); + expect(detach).not.toHaveBeenCalled(); + } else if (expired) { + expect(detach).not.toHaveBeenCalled(); + expect(close).toHaveBeenCalledWith({ reason: "native control-plane run admission failed" }); + } else { + expect(await shutdown).toMatchObject({ detachedRunIds: [restarting.binding.runId], inactiveRunIds: [] }); + expect(detach).toHaveBeenCalledOnce(); + expect(close).not.toHaveBeenCalledWith({ reason: "native control-plane run admission failed" }); + } + expect(backend.openSession).toHaveBeenCalledOnce(); + expect(startTurn).not.toHaveBeenCalled(); + } finally { vi.useRealTimers(); release(); acknowledgeStop(); await stopping; await outcome; await shutdown; } + }); + it("bounds a pending startup Stop without detaching its cleanup owner", async () => { + const restarting = structuredClone(execution); + restarting.binding.runId = "restart-startup-stop-cleanup-timeout"; + let open!: () => void, started!: () => void, releaseCleanup!: () => void, cleaning!: () => void; + const opening = new Promise(resolve => { open = resolve; }); + const pending = new Promise(resolve => { started = resolve; }); + const cleanup = new Promise(resolve => { releaseCleanup = resolve; }); + const cleanupStarted = new Promise(resolve => { cleaning = resolve; }); + const detach = vi.fn(async () => undefined); + const cancel = vi.fn(() => ({ cleanup: Promise.resolve() })); + state.execute.mockReset().mockImplementationOnce(async options => { + started(); await opening; + try { await options.onSession({ detachControllerForRestart: detach, cancel }); } + finally { cleaning(); await cleanup; } + }); + const outcome = executePaperclipNativeSession({ db: leaseDb(restarting), execution: restarting, runnerInstanceId: "runner" }).catch(error => error); + await pending; + const stopping = cancelNativeSession(restarting.binding.runId, "operator Stop before restart"); + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] }); + const shutdown = detachNativeSessionsForRestart([restarting.binding.runId]).catch(error => error); + try { + open(); + await cleanupStarted; + expect(await stopping).toBe(true); + await vi.advanceTimersByTimeAsync(30_000); + expect(await shutdown).toMatchObject({ pendingRunIds: [restarting.binding.runId], detachedRunIds: [] }); + expect(detach).not.toHaveBeenCalled(); + expect(cancel).toHaveBeenCalledOnce(); + } finally { vi.useRealTimers(); open(); releaseCleanup(); await stopping; await outcome; await shutdown; } + }); + it("waits for a checkpointed governed settlement before relinquishing its controller", async () => { const restarting = structuredClone(execution); restarting.binding.runId = "restart-governed-settlement"; @@ -6033,7 +6340,7 @@ describe("native startup restart detachment", () => { const detached = detachNativeSessionsForRestart([restarting.binding.runId]).catch(error => error); if (exceedDeadline) { await vi.advanceTimersByTimeAsync(30_000); - expect(await detached).toMatchObject({ message: "native_restart_startup_not_ready" }); + expect(await detached).toMatchObject({ pendingRunIds: [restarting.binding.runId], detachedRunIds: [] }); vi.useRealTimers(); } else { release(); @@ -6049,18 +6356,13 @@ describe("native startup restart detachment", () => { } }); - it("remembers shutdown while the session is still opening and detaches its late publication", async () => { + it("rejects an entirely new startup after the run has been fenced for shutdown", async () => { const restarting = structuredClone(execution); - restarting.binding.runId = "restart-during-session-open"; - const detach = vi.fn(async () => undefined); + restarting.binding.runId = "restart-new-startup-after-drain"; await expect(detachNativeSessionsForRestart([restarting.binding.runId])).resolves.toMatchObject({ inactiveRunIds: [restarting.binding.runId] }); - state.execute.mockReset().mockImplementationOnce(async (options) => { - await options.onSession({ detachControllerForRestart: detach }); - expect(detach).toHaveBeenCalledOnce(); - await options.onSession(null); - throw new Error("detachment closed the old event stream"); - }); + state.execute.mockReset(); await expect(executePaperclipNativeSession({ db: leaseDb(restarting), execution: restarting, runnerInstanceId: "runner" })).rejects.toBeInstanceOf(NativeControllerDetachedForRestartError); + expect(state.execute).not.toHaveBeenCalled(); }); }); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index cc03fcbfa56..ccf020f577b 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -227,6 +227,7 @@ import { type ActiveNativeSession = { session: NativeSession; cancelRequested: boolean; + cancellationSettled?: Promise; restartDetach?: Promise; }; @@ -264,6 +265,7 @@ type NativeSessionStartup = { promise: Promise; resolve: (session: ActiveNativeSession | null) => void; stopRequested?: boolean; + restartAdmissionPending?: boolean; cancellationSettled?: Promise; governedWait?: boolean; settled: Promise; @@ -272,13 +274,14 @@ type NativeSessionStartup = { const nativeSessionStartups = new Map(); function detachActiveNativeSessionForRestart(active: ActiveNativeSession) { - active.restartDetach ??= Promise.resolve().then(() => active.session.detachControllerForRestart!()); + active.restartDetach ??= active.session.detachControllerForRestart!(); return active.restartDetach; } async function waitForNativeSessionStartup( runId: string, timeoutError: () => Error = () => new Error("native_restart_startup_not_ready"), + timeoutMs = 30_000, ) { const startup = nativeSessionStartups.get(runId); if (!startup) return null; @@ -287,7 +290,7 @@ async function waitForNativeSessionStartup( return await Promise.race([ startup.promise, new Promise((_resolve, reject) => { - timer = setTimeout(() => reject(timeoutError()), 30_000); + timer = setTimeout(() => reject(timeoutError()), timeoutMs); }), ]); } finally { @@ -301,10 +304,13 @@ export async function detachNativeSessionsForRestart( detachedRunIds: string[]; inactiveRunIds: string[]; unsupportedRunIds: string[]; + pendingRunIds: string[]; }> { const detachedRunIds: string[] = []; const inactiveRunIds: string[] = []; const unsupportedRunIds: string[] = []; + const pendingRunIds: string[] = []; + const restartDeadline = Date.now() + 30_000; const settlementDeadline = Date.now() + 20_000; for (const runId of new Set(runIds)) { // A governed stop already revoked new work and is collecting terminal @@ -320,11 +326,48 @@ export async function detachNativeSessionsForRestart( } finally { clearTimeout(timer); } } nativeRunsDetachingForRestart.add(runId); - const active = activeNativeSessions.get(runId) ?? await waitForNativeSessionStartup(runId); + let active = activeNativeSessions.get(runId); + if (!active) { + // Only a startup registered before this fence may complete read-only + // admission. It must publish its exact session for detachment; writes and + // new turns remain revoked, including after the bounded wait expires. + const startup = nativeSessionStartups.get(runId); + if (startup && Date.now() >= restartDeadline) { + pendingRunIds.push(runId); + continue; + } + if (startup) startup.restartAdmissionPending = true; + try { + active = await waitForNativeSessionStartup(runId, undefined, Math.max(0, restartDeadline - Date.now())) ?? undefined; + } catch (error) { + if (!(error instanceof Error) || error.message !== "native_restart_startup_not_ready") throw error; + pendingRunIds.push(runId); + continue; + } finally { + if (startup) startup.restartAdmissionPending = false; + } + } if (!active) { inactiveRunIds.push(runId); continue; } + const stopping = nativeSessionStartups.get(runId); + if (stopping?.stopRequested || active.cancelRequested) { + // Stop owns cancellation and final cleanup, including after publication. + // A deadline leaves this run pending; it must not abort healthy adoption + // or claim that the transport/cleanup owner has settled. + let timer: ReturnType | undefined; + try { + const settled = stopping && await Promise.race([ + stopping.settled.then(() => true), + new Promise(resolve => { + timer = setTimeout(() => resolve(false), Math.max(0, restartDeadline - Date.now())); + }), + ]); + (settled ? inactiveRunIds : pendingRunIds).push(runId); + } finally { clearTimeout(timer); } + continue; + } if (active.session.detachControllerForRestart === undefined) { unsupportedRunIds.push(runId); continue; @@ -332,7 +375,7 @@ export async function detachNativeSessionsForRestart( await detachActiveNativeSessionForRestart(active); detachedRunIds.push(runId); } - return { detachedRunIds, inactiveRunIds, unsupportedRunIds }; + return { detachedRunIds, inactiveRunIds, unsupportedRunIds, pendingRunIds }; } const MAX_REMOTE_CHECKPOINT_ARCHIVE_BYTES = 64 * 1024 * 1024; const MAX_REMOTE_CHECKPOINT_EXPANDED_BYTES = 64 * 1024 * 1024; @@ -7230,6 +7273,7 @@ export async function cancelNativeSession( // Do not acknowledge a no-op Stop and let that owner submit a turn afterwards. const startup = runStop && !activeNativeSessions.has(runId) ? nativeSessionStartups.get(runId) : undefined; let settleStartupCancellation: (() => void) | undefined; + let settleActiveCancellation: (() => void) | undefined; if (startup) { startup.stopRequested = true; startup.cancellationSettled = new Promise(resolve => { settleStartupCancellation = resolve; }); @@ -7242,6 +7286,7 @@ export async function cancelNativeSession( dispatched = true; if (!active.cancelRequested) { active.cancelRequested = true; + active.cancellationSettled = new Promise(resolve => { settleActiveCancellation = resolve; }); try { if (active.session.cancel) { const cancellationAbort = new AbortController(); @@ -7451,6 +7496,7 @@ export async function cancelNativeSession( return { dispatched, decision, decisionId, auditId }; } finally { settleStartupCancellation?.(); + settleActiveCancellation?.(); } } @@ -7674,6 +7720,9 @@ export async function executePaperclipNativeSession(input: { } } const runId = input.execution.binding.runId; + if (nativeRunsDetachingForRestart.has(runId)) { + throw new NativeControllerDetachedForRestartError(); + } if (nativeSessionStartups.has(runId)) { throw new Error("native_session_supervisor_busy"); } @@ -8362,6 +8411,12 @@ async function executePaperclipNativeSessionWithinScope( throw new NativeControllerDetachedForRestartError(); } }; + const startupForAdmission = nativeSessionStartups.get(input.execution.binding.runId); + const assertRunAdmissionActive = () => { + if (startupForAdmission?.restartAdmissionPending + && nativeSessionStartups.get(input.execution.binding.runId) === startupForAdmission) return; + assertControllerActive(); + }; const controlPlane = new PaperclipControlPlanePort( input.db, { @@ -8378,6 +8433,7 @@ async function executePaperclipNativeSessionWithinScope( { privateKeyPem: input.runnerEnvironment?.PAPERCLIP_AGENT_PRIVATE_KEY, assertControllerActive, + assertRunAdmissionActive, onCommittedEvent: async (event) => { const textKind = !firstTokenRecorded ? observeDeliverableText(event) : null; if (textKind) { @@ -9220,6 +9276,13 @@ async function executePaperclipNativeSessionWithinScope( eq(heartbeatRuns.agentId, input.execution.binding.agentId), )).limit(1); const cancellation = record(currentRun?.resultJson?.nativeCancellation); + if (active.cancelRequested) { + // A Stop can acquire the published handle while the authority + // read above is in flight. Keep its ACK/cleanup owner before + // unwinding admission; neither detach nor generic close wins. + await active.cancellationSettled; + throw new NativeCancellationPendingRecoveryError(); + } if (!currentRun || currentRun.status !== "running" || currentRun.resultJson?.startupCancellation || (cancellation.scope === "run" && @@ -9229,11 +9292,15 @@ async function executePaperclipNativeSessionWithinScope( // callback fails; no provider turn may follow publication. throw new NativeCancellationPendingRecoveryError(); } - if (session.resolveRuntimeRequest) await liveQuestions.attach(); if (nativeRunsDetachingForRestart.has(input.execution.binding.runId)) { if (session.detachControllerForRestart) await detachActiveNativeSessionForRestart(active); + throw new NativeControllerDetachedForRestartError(); + } + if (session.resolveRuntimeRequest) await liveQuestions.attach(); + if (active.cancelRequested) { + await active.cancellationSettled; + throw new NativeCancellationPendingRecoveryError(); } - if (active.cancelRequested) throw new NativeCancellationPendingRecoveryError(); } else { liveQuestions.close(); activeNativeSessions.delete(input.execution.binding.runId); diff --git a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts index 5321a764f8b..f8b9eb1d022 100644 --- a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts +++ b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts @@ -300,6 +300,38 @@ describe("PaperclipControlPlanePort conformance", () => { } }); + it("permits only read-only admission while an existing startup drains", async () => { + const identity = { ...CONTROL_PLANE_CONFORMANCE_OPEN.identity, runId: randomUUID(), sessionId: randomUUID() }; + const runnerId = randomUUID(); + await db.insert(heartbeatRuns).values({ + id: identity.runId, companyId: identity.companyId, agentId: identity.agentId, + status: "running", runtimeMode: "native", nativeIssueId: identity.issueId, + nativeSessionId: identity.sessionId, runnerInstanceId: runnerId, + completionContractId: contractId, completionContractSha256: contractSha, + }); + let awaitingStartup = true; + const port = new PaperclipControlPlanePort(db, { + ...identity, completionContractId: contractId, completionContractSha256: contractSha, + sourceInstanceId: runnerId, controlPlaneSourceInstanceId: `draining-${identity.runId}`, + }, { + assertControllerActive: () => { throw new Error("controller_detached"); }, + assertRunAdmissionActive: () => { if (!awaitingStartup) throw new Error("controller_detached"); }, + }); + const open = { identity, backendKind: "mock" as const, sourceInstanceId: runnerId }; + await expect(port.openRun({ ...open, sourceInstanceId: randomUUID() })).rejects.toThrow("native_open_run_binding_mismatch"); + await db.update(heartbeatRuns).set({ runnerInstanceId: randomUUID() }).where(eq(heartbeatRuns.id, identity.runId)); + await expect(port.openRun(open)).rejects.toThrow("native_open_run_not_authorized"); + await db.update(heartbeatRuns).set({ runnerInstanceId: runnerId }).where(eq(heartbeatRuns.id, identity.runId)); + await expect(port.openRun(open)).resolves.toBeUndefined(); + await expect(port.checkpointSession({ backendKind: "mock", sessionId: identity.sessionId, identity })).rejects.toThrow("controller_detached"); + await expect(port.completeRun({ result: CONTROL_PLANE_CONFORMANCE_RESULT, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL })).rejects.toThrow("controller_detached"); + awaitingStartup = false; + await expect(port.openRun(open)).rejects.toThrow("controller_detached"); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, identity.runId)); + expect(run.runnerProfileJson?.sessionCheckpoint).toBeUndefined(); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, identity.runId))).toEqual([]); + }); + it.each([ ["checkpoint", "detach"], ["result", "detach"], ["event", "detach"], ["checkpoint", "abort"], ["result", "abort"], ["event", "abort"], diff --git a/server/src/services/native-runtime/paperclip-control-plane-port.ts b/server/src/services/native-runtime/paperclip-control-plane-port.ts index acb02e19b63..d6c6413d060 100644 --- a/server/src/services/native-runtime/paperclip-control-plane-port.ts +++ b/server/src/services/native-runtime/paperclip-control-plane-port.ts @@ -83,6 +83,7 @@ export class PaperclipControlPlanePort implements ControlPlanePort { readonly #binding: PaperclipControlPlaneBinding; #sessionId: string | null = null; readonly #assertControllerActive?: () => void; + readonly #assertRunAdmissionActive?: () => void; readonly #onCommittedEvent?: (event: PrpEvent) => Promise; readonly #onDuplicateEvent?: (event: PrpEvent) => Promise; @@ -94,12 +95,15 @@ export class PaperclipControlPlanePort implements ControlPlanePort { privateKeyPem?: string; /** Synchronous revocation fence for a controller handing off on restart. */ assertControllerActive?: () => void; + /** Read-only admission may finish an existing startup solely for restart detachment. */ + assertRunAdmissionActive?: () => void; onCommittedEvent?: (event: PrpEvent) => Promise; onDuplicateEvent?: (event: PrpEvent) => Promise; } = {}, ) { this.#db = db; this.#assertControllerActive = options.assertControllerActive; + this.#assertRunAdmissionActive = options.assertRunAdmissionActive ?? options.assertControllerActive; this.#identityRedactor = createAgentIdentityRedactor(options.privateKeyPem); this.#binding = structuredClone(binding); this.#onCommittedEvent = options.onCommittedEvent; @@ -123,7 +127,7 @@ export class PaperclipControlPlanePort implements ControlPlanePort { } async openRun(input: OpenControlPlaneRunInput): Promise { - this.#assertControllerActive?.(); + this.#assertRunAdmissionActive?.(); const identity = input.identity; if ( identity.companyId !== this.#binding.companyId @@ -146,7 +150,7 @@ export class PaperclipControlPlanePort implements ControlPlanePort { if (!run || !this.#matchesPersistedBinding(run)) { throw new Error("native_open_run_not_authorized"); } - this.#assertControllerActive?.(); + this.#assertRunAdmissionActive?.(); this.#sessionId = identity.sessionId; } From 970127a0220d3145efe2111b0643aaa883cb162f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 24 Oct 2026 21:32:42 -0700 Subject: [PATCH 15/68] test: retain onboarding Connect failure evidence (#17529) ## Thinking Path > - Paperclip manages agents and their work through normal company controls. > - Product E2E journeys check these controls through the browser and public API. > - Cloud onboarding uses one Connect action to validate the environment and hire the first agent. > - A failed Connect action previously left only a general journey error in the report. > - This pull request retains a closed diagnostic receipt for that action. > - The report can show the observed failure boundary without retaining private response text. ## Linked Issues or Issue Description **What existing behavior does this improve?** Cloud onboarding Product E2E failure evidence and report detail. **Current behavior** A validation or hire failure can reach the journey as a general operation failure. The report does not retain which Connect response failed. **Proposed behavior** Observe only the same-origin, same-company validation and hire POST requests made during one Connect action. Retain a closed receipt with the checkpoint, response state, bounded HTTP status, and allowlisted adapter, outcome, check code, level, and server code. The Get started control and saved-agent readback remain required. **Reason and benefit** The next failure can be attributed to the exact observed browser/API boundary. This change does not claim the cause of a historical failure. **Breaking changes** The new receipt is optional. Historical results remain valid. Explicit validation and hire rejections now stop the journey with a precise closed error code. This changes only E2E source; it does not change product behavior or qualification gates. ## What Changed - Add a bounded Connect observer with exact request matching and listener cleanup. - Retain known server codes from `details.code` or `code` without response messages, hints, headers, request bodies, provider text, or credentials. - Validate the receipt as a closed schema and render its allowlisted fields in existing report detail. - Include the new helper in the journey definition hash. - Add route, privacy, malformed-body, deadline, late-response, delayed-failure, success, and historical compatibility tests. Document the evidence boundary. ## Verification - Focused Product E2E tests passed: 3 files, 58 tests on the review-fix head. - E2E typecheck, workspace typecheck, and full build passed on the preceding implementation head. The review fix changes only report labels, checkpoint timing, and the matching tests. - The local full `pnpm test:run` command stopped progressing at the general-server start header. Its owned session was interrupted. This is not recorded as a pass. - All 58 remote checks completed on the final review-fix head: 55 passed, one neutral, and two expected skips. Both Greptile and Paperclip reviews are 5/5 with no unresolved review threads. - No live campaign was run with this patch. The observed historical failure is preserved. ## Risks - Unknown or missing responses remain explicit diagnostic states. They do not establish a product root cause. - Response body reads stop after five seconds or the action deadline and accept at most 64 KiB of JSON. Checks are capped at 32 retained entries. - New server or check codes appear as `unrecognized` until the closed allowlist is updated. - The production launch classifier, native gateway qualification, and Get started requirement are unchanged. ## Model Used OpenAI Codex (GPT-6 family), with code execution and tool use. The host does not expose the exact model ID or context window to this run; these values are unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- tests/runner-e2e/README.md | 5 + tests/runner-e2e/SECURITY.md | 7 + tests/runner-e2e/case-outcome.ts | 24 +++ .../product-journeys/cloud-onboarding.test.ts | 3 + .../product-journeys/cloud-onboarding.ts | 6 +- .../product-journeys/contracts.test.ts | 54 ++++++ tests/runner-e2e/product-journeys/launch.ts | 2 +- .../onboarding-connect.test.ts | 182 ++++++++++++++++++ .../product-journeys/onboarding-connect.ts | 131 +++++++++++++ tests/runner-e2e/product-journeys/types.ts | 1 + tests/runner-e2e/result-validation.ts | 2 + 11 files changed, 414 insertions(+), 3 deletions(-) create mode 100644 tests/runner-e2e/product-journeys/onboarding-connect.test.ts create mode 100644 tests/runner-e2e/product-journeys/onboarding-connect.ts diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 47853d3ba74..129f75c4e31 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -237,6 +237,11 @@ The explicit [deployment-targeted Product journeys](PRODUCT-JOURNEYS.md) reuse browser actions across managed self-hosted and Cloud targets. They include a model-free task-properties story, fresh Cloud onboarding, credential-free previews, private evidence, and prerequisite reuse; `--all` is unchanged. +Cloud onboarding may retain a versioned `onboardingConnect` receipt for the +Connect action: its last checkpoint, validation/hire response state and HTTP +status, known adapters, and allowlisted outcome/check/server codes. The case +report renders these fields explicitly; older results without the receipt remain +valid. This diagnostic does not change the journey grade or qualify a live run. ## Live provider connection journeys diff --git a/tests/runner-e2e/SECURITY.md b/tests/runner-e2e/SECURITY.md index 26a326b9cca..afbcf8dd494 100644 --- a/tests/runner-e2e/SECURITY.md +++ b/tests/runner-e2e/SECURITY.md @@ -14,6 +14,13 @@ their expected four-field result, exact base64 download bytes, checksums, size, and attachment/run IDs. Extra fields and arbitrary agent output are rejected. It contains no credentials, browser state, request headers, or provider text. +The optional `onboardingConnect` diagnostic is also a closed, versioned receipt. +It retains only a known checkpoint, response states, HTTP statuses, known adapter +names, and allowlisted outcome/check/server codes (at most 32 checks per response). +Unknown codes become `unrecognized`; raw bodies, messages, URLs, headers, and +browser errors are excluded. Reports validate the receipt before rendering its +explicit fields, and historical results may omit it. + ## Attended connection authentication See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only diff --git a/tests/runner-e2e/case-outcome.ts b/tests/runner-e2e/case-outcome.ts index 03316c81995..489f16f8384 100644 --- a/tests/runner-e2e/case-outcome.ts +++ b/tests/runner-e2e/case-outcome.ts @@ -1,5 +1,6 @@ import type { RunnerE2EResult } from "./types.js"; import { isBlockedConnectionResult } from "./connection-evidence.js"; +import { validateOnboardingConnectEvidence } from "./product-journeys/onboarding-connect.js"; const html = (value: unknown) => String(value ?? "").replace( /[&<>"']/g, @@ -8,6 +9,28 @@ const html = (value: unknown) => c ]!, ); + +function renderOnboardingConnect(result: RunnerE2EResult): string { + const receipt = result.productJourney?.onboardingConnect; + if (receipt === undefined) return ""; + try { + validateOnboardingConnectEvidence(receipt); + } catch { + return "

Onboarding Connect diagnostics unavailable: invalid receipt.

"; + } + return `
Onboarding Connect diagnostics +

Last checkpoint: ${html(receipt.checkpoint)}

+ ${([["validation", "Native validation"], ["hire", "Agent hire"]] as const).map(([key, label]) => { + const response = receipt[key]; + return `

${label}

${[ + ["Response state", response.state], ["HTTP status", response.httpStatus], + ["Test endpoint adapter", response.requestedAdapter], ["Returned adapter", response.adapter], + ["Validation outcome", response.outcome], ["Server code", response.serverCode], + ].map(([name, value]) => `
${name}
${html(value ?? "Unavailable")}
`).join("")}
+ ${response.checks.length ? `

Validation checks:

    ${response.checks.map(check => `
  • ${html(check.code)} · ${html(check.level)}
    • `).join("")}
` : ""} + ${response.truncatedChecks ? "

Only the first 32 validation checks were retained.

" : ""}`; + }).join("")}
`; +} /** An interrupted first-task journey is not evidence of failed downstream behavior. */ export function isIncompleteFirstTaskResult( result: RunnerE2EResult, @@ -69,6 +92,7 @@ export function renderCaseOutcome( ${result.productJourney.deployment?.effectiveFeatures ? `

Effective features:

${Object.entries(result.productJourney.deployment.effectiveFeatures).map(([key, value]) => `
${html(key)}
${html(value)}
`).join("")}
` : ""}` : ""} ${result.productJourney?.operations ? `

Eval operations: ${html(result.productJourney.operations.campaignId)} · ${result.productJourney.operations.requestCount} requests · model cost $${html(result.productJourney.operations.modelCostUsd ?? "unknown")} (setup $${html(result.productJourney.operations.setupCostUsd ?? "unknown")}, counted once). Feedback: ${html(result.productJourney.operations.feedback.collection)} · ${result.productJourney.operations.feedback.complaints} complaints · ${result.productJourney.operations.feedback.suggestions} suggestions.

` : ""} ${result.productJourney?.operations?.traceIds.length ? `
Trace correlation IDs
    ${result.productJourney.operations.traceIds.map(id => `
  • ${html(id)}
    • `).join("")}
` : ""} + ${renderOnboardingConnect(result)} ${result.providerConnection ? `

Target: ${html(result.providerConnection.target.origin)} · ${html(result.providerConnection.target.commit ?? "revision unavailable")} · ${result.providerConnection.assisted ? "Assisted login" : "Unassisted"} · ${html(result.providerConnection.authFreshness)} browser · ${html(result.providerConnection.entry)} / ${html(result.providerConnection.method)} · stopped at ${html(result.providerConnection.phase)}

` : ""} ${failures.length ? `

Failed checks:

    ${failures.map((c) => `
  • ${html(c.id)} — ${html(c.detail)}
    • `).join("")}
` : checks.length ? `

No behavioral matcher failed.${notReached.length ? " The journey is incomplete; unexercised checks are not passes." : failed ? " The overall failure came from a separate run, cleanup, or evidence check." : ""}

` : ""} ${notReached.length ? `

Not reached:

    ${notReached.map((c) => `
  • ${html(c.id)} — ${html(c.notReached)}
    • `).join("")}
` : ""} diff --git a/tests/runner-e2e/product-journeys/cloud-onboarding.test.ts b/tests/runner-e2e/product-journeys/cloud-onboarding.test.ts index 94589ca5069..0878ade8065 100644 --- a/tests/runner-e2e/product-journeys/cloud-onboarding.test.ts +++ b/tests/runner-e2e/product-journeys/cloud-onboarding.test.ts @@ -86,6 +86,8 @@ function fixture(entry: Entry, options: { intentFailure?: boolean; qualification }, }); const page = { + on() { return this; }, + off() { return this; }, getByLabel(label: string) { return locator(() => controls().filter(control => control.name === label && control.role === "textbox")); }, getByRole(role: string, { name: label }: { name: string | RegExp }) { return locator(() => controls().filter(control => control.role === role && match(control.name, label))); @@ -93,6 +95,7 @@ function fixture(entry: Entry, options: { intentFailure?: boolean; qualification locator(selector: string) { expect(selector).toBe("summary"); return locator(() => controls().filter(control => control.role === "summary")); }, }; const context = { page, company: { id: "company" }, identity: { kind: "cloud" }, startingState: "recipe", api: { + origin: "https://tenant.example.test", async get(route: string) { if (route.endsWith("/ai-connections")) return { connections: saved ? [connection] : [] }; if (route.endsWith("/agents")) return hired ? [{ id: "agent", name, adapterType: "paperclip_runner", adapterConfig: { provider: "codex", model: gateway.model } }] : []; diff --git a/tests/runner-e2e/product-journeys/cloud-onboarding.ts b/tests/runner-e2e/product-journeys/cloud-onboarding.ts index 3173676c3ad..6f9a75634a7 100644 --- a/tests/runner-e2e/product-journeys/cloud-onboarding.ts +++ b/tests/runner-e2e/product-journeys/cloud-onboarding.ts @@ -1,5 +1,6 @@ import { expect } from "@playwright/test"; import { createConnectionProof } from "../connection-evidence.js"; +import { onboardingConnect } from "./onboarding-connect.js"; import { onboardingFirstTask } from "./onboarding-first-task.js"; import { JourneyError, type JourneyContext, type JourneyEvidence, type TargetSession } from "./types.js"; @@ -69,11 +70,12 @@ export async function cloudOnboarding(input: CloudOnboardingInput): Promise { evidence.onboardingConnect = value; } }); const agents = await api.get(`/api/companies/${company.id}/agents`); const agent = agents.find(a => a.name === name); if (!agent || agents.length !== 1 || agent.adapterType !== "paperclip_runner" || agent.adapterConfig?.provider !== "codex" || agent.adapterConfig?.model !== input.gateway.model) throw new JourneyError("failed", "onboarding_agent_configuration_mismatch"); + connect.checkpoint = "agent-verification"; await input.own({ kind: "agent", name, id: agent.id }); evidence.checks.push({ id: "ui-connection-and-native-agent", passed: true }); await input.own({ kind: "issue", name: "Paperclip onboarding" }); diff --git a/tests/runner-e2e/product-journeys/contracts.test.ts b/tests/runner-e2e/product-journeys/contracts.test.ts index 754d6c7bab9..b02057281a3 100644 --- a/tests/runner-e2e/product-journeys/contracts.test.ts +++ b/tests/runner-e2e/product-journeys/contracts.test.ts @@ -13,6 +13,7 @@ import type { RunnerE2EResult } from "../types.js"; import type { JourneyContext, JourneyEvidence, TargetSession } from "./types.js"; import { createConnectionProof } from "../connection-evidence.js"; import { retainOnboardingArtifact } from "./artifact-evidence.js"; +import type { OnboardingConnectEvidence } from "./onboarding-connect.js"; vi.mock("@playwright/test", () => ({ expect: Object.assign( (locator: { textContent(): Promise }) => ({ async toContainText(value: string) { expect(await locator.textContent()).toContain(value); } }), @@ -29,7 +30,60 @@ function result(): RunnerE2EResult { return { schema: "paperclip.runner-e2e.result/v2", suiteId: "product-journeys", suiteDefinitionHash: "test", executionId: "product-journeys.local-qa.local.task-properties", profileId: "local-qa", environmentId: "local", caseId: "task-properties", provider: null, model: null, runtimeMode: "none", status: "passed", cleanup: "passed", attempt: 1, startedAt: "2026-01-01T00:00:00Z", finishedAt: "2026-01-01T00:00:01Z", durationMs: 1000, productJourney: { outcome: "passed", startingState: "recipe", phases: [{phase: "journey", outcome: "passed"}], checks: [{id:"assignment-persisted", passed:true},{id:"no-agent-work",passed:true}], privacy: "private", resourceDisposition: "released", deployment: { kind: "self-hosted", targetId: "local-qa", origin: "http://127.0.0.1:3100", commit: "a".repeat(40), cloudRelease: null, runtimeImage: null, configurationHash: "test" } } }; } +function connectReceipt(): OnboardingConnectEvidence { + return { + version: 1, checkpoint: "hire", + validation: { state: "json", httpStatus: 200, requestedAdapter: "paperclip_runner", adapter: "paperclip_runner", outcome: "fail", serverCode: null, checks: [{ code: "adapter_auth_missing", level: "error" }], truncatedChecks: false }, + hire: { state: "json", httpStatus: 422, requestedAdapter: null, adapter: null, outcome: null, serverCode: "agent_runner_unavailable", checks: [], truncatedChecks: false }, + }; +} describe("deployment journey contracts", () => { + it("accepts historical results without Connect diagnostics and optional closed receipts", () => { + const retained = result(); + retained.caseId = "cloud-onboarding"; + retained.status = "failed"; + expect(() => validateRetainedRunnerResult(retained)).not.toThrow(); + expect(renderCaseOutcome(retained, true, [])).not.toContain("Onboarding Connect diagnostics"); + retained.productJourney!.onboardingConnect = connectReceipt(); + expect(() => validateRetainedRunnerResult(JSON.parse(JSON.stringify(retained)))).not.toThrow(); + retained.productJourney!.onboardingConnect.validation = { state: "pending", httpStatus: 200, requestedAdapter: "codex_local", adapter: null, outcome: null, serverCode: null, checks: [], truncatedChecks: false }; + expect(() => validateRetainedRunnerResult(retained)).not.toThrow(); + }); + it.each([ + ["request headers", () => ({ ...connectReceipt(), requestHeaders: { authorization: "must-not-retain" } })], + ["response bodies", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, body: "must-not-retain" } })], + ["check messages", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, checks: [{ code: "adapter_auth_missing", level: "error", message: "must-not-retain" }] } })], + ["unknown server codes", () => ({ ...connectReceipt(), hire: { ...connectReceipt().hire, serverCode: "raw-provider-error" } })], + ["unknown check codes", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, checks: [{ code: "raw-provider-error", level: "error" }] } })], + ["unknown adapters", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, adapter: "raw-provider-name" } })], + ["unknown requested adapters", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, requestedAdapter: "raw-provider-name" } })], + ["unknown outcomes", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, outcome: "raw-provider-message" } })], + ["unknown response states", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, state: "raw-provider-message" } })], + ["unknown check levels", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, checks: [{ code: "adapter_auth_missing", level: "fatal" }] } })], + ["oversized check arrays", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, checks: Array.from({ length: 33 }, () => ({ code: "adapter_auth_missing", level: "error" })) } })], + ["invalid HTTP statuses", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, httpStatus: 600 } })], + ["non-boolean truncation", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, truncatedChecks: "true" } })], + ["missing response fields", () => ({ ...connectReceipt(), hire: { state: "missing" } })], + ["unsupported receipt versions", () => ({ ...connectReceipt(), version: 2 })], + ["unknown checkpoints", () => ({ ...connectReceipt(), checkpoint: "raw-browser-text" })], + ] as const)("rejects Connect receipts containing %s", (_label, makeReceipt) => { + const retained = result(); + retained.productJourney!.onboardingConnect = makeReceipt() as OnboardingConnectEvidence; + expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingConnect"); + }); + it("renders only validated Connect fields and marks missing or truncated evidence", () => { + const retained = result(); + retained.productJourney!.onboardingConnect = connectReceipt(); + const output = renderCaseOutcome(retained, true, []); + for (const text of ["Onboarding Connect diagnostics", "Last checkpoint", "Native validation", "Agent hire", "HTTP status", "Test endpoint adapter", "Returned adapter", "200", "422", "paperclip_runner", "adapter_auth_missing", "agent_runner_unavailable", "error", "Unavailable"]) expect(output).toContain(text); + retained.productJourney!.onboardingConnect.validation.truncatedChecks = true; + expect(renderCaseOutcome(retained, true, [])).toContain("Only the first 32 validation checks were retained"); + Object.assign(retained.productJourney!.onboardingConnect.validation, { body: "" }); + const invalid = renderCaseOutcome(retained, false, []); + expect(invalid).toContain("Onboarding Connect diagnostics unavailable: invalid receipt"); + expect(invalid).not.toContain("must-not-render"); + expect(invalid).not.toContain("adapter_auth_missing"); + }); it("validates optional historical artifact evidence without trusting extra fields or a different run", () => { const retained = result(); retained.caseId = "cloud-onboarding"; diff --git a/tests/runner-e2e/product-journeys/launch.ts b/tests/runner-e2e/product-journeys/launch.ts index 2d0233bd2e9..410bb7725bf 100644 --- a/tests/runner-e2e/product-journeys/launch.ts +++ b/tests/runner-e2e/product-journeys/launch.ts @@ -99,7 +99,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri try { sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); } catch { /* Packaged harnesses may not include Git metadata. */ } const source = resolveRunnerE2ESource({ sha, ref: null, workflowRunUrl: null }); const definitionHash = createHash("sha256").update(JSON.stringify(productJourneys)); - for (const name of ["catalog.ts", "task-properties.ts", "cloud-onboarding.ts", "onboarding-first-task.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); + for (const name of ["catalog.ts", "task-properties.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); const suiteDefinitionHash = definitionHash.digest("hex"); const expected = config.journeys.map(id => `product-journeys.${config.id}.${id === "cloud-onboarding" ? config.modelProfileId : "browser"}.${config.executionEnvironment}.${id}`); let reportPersisted = false; diff --git a/tests/runner-e2e/product-journeys/onboarding-connect.test.ts b/tests/runner-e2e/product-journeys/onboarding-connect.test.ts new file mode 100644 index 00000000000..be9535cf0c7 --- /dev/null +++ b/tests/runner-e2e/product-journeys/onboarding-connect.test.ts @@ -0,0 +1,182 @@ +import { EventEmitter } from "node:events"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { onboardingConnect, validateOnboardingConnectEvidence, type OnboardingConnectEvidence } from "./onboarding-connect.js"; + +const origin = "https://tenant.example.test", companyId = "company-1"; +const validation = `/api/companies/${companyId}/adapters/paperclip_runner/test-environment`; +const hire = `/api/companies/${companyId}/agent-hires`; +const secret = "PRIVATE_SENTINEL_do_not_retain"; +function fixture(options: { ready?: boolean; durationMs?: number; click?: () => Promise | void; wait?: () => Promise } = {}) { + const events = new EventEmitter(); + let retained!: OnboardingConnectEvidence; + const page = Object.assign(events, { getByRole(_role: string, { name }: { name: string }) { + return { click: vi.fn(async () => { expect(name).toBe("Connect"); await options.click?.(); }), + waitFor: vi.fn(({ timeout }: { timeout: number }) => { + expect(name).toBe("Get started"); + if (options.wait) return options.wait(); + return options.ready ? Promise.resolve() : new Promise((_resolve, reject) => { + setTimeout(() => reject(Object.assign(new Error(secret), { name: "TimeoutError" })), timeout); + }); + }) }; + } }); + const request = (path = validation, method = "POST", requestOrigin = origin) => ({ url: () => requestOrigin + path, method: () => method }); + const respond = (body: unknown, status = 200, path = validation, method = "POST", requestOrigin = origin) => { + const req = request(path, method, requestOrigin); + const response = { request: () => req, status: () => status, body: () => Promise.resolve(Buffer.isBuffer(body) ? body : Buffer.from(JSON.stringify(body))) }; + events.emit("request", req); events.emit("response", response); + return response; + }; + const run = () => onboardingConnect({ page: page as any, origin, companyId, deadlineAt: Date.now() + (options.durationMs ?? 100), retain: value => { retained = value; } }); + return { page, events, request, respond, run, receipt: () => retained }; +} +afterEach(() => { vi.useRealTimers(); }); + +describe("one onboarding Connect diagnostic boundary", () => { + it("does not press Connect after the action deadline already expired", async () => { + const click = vi.fn(); + const f = fixture({ durationMs: 0, click }); + await expect(f.run()).rejects.toMatchObject({ code: "onboarding_connect_timeout" }); + expect(click).not.toHaveBeenCalled(); + expect(f.receipt().checkpoint).toBe("connect"); + expect(f.events.eventNames()).toEqual([]); + }); + + it("rejects a delayed failure body even after Get started becomes visible", async () => { + let resolveBody!: (bytes: Buffer) => void; + const body = new Promise(resolve => { resolveBody = resolve; }); + let reviewSeen!: () => void; + const review = new Promise(resolve => { reviewSeen = resolve; }); + const f = fixture({ wait: async () => { reviewSeen(); }, click: () => { + const req = f.request(); f.events.emit("request", req); + f.events.emit("response", { request: () => req, status: () => 200, body: () => body }); + } }); + const result = expect(f.run()).rejects.toMatchObject({ code: "onboarding_native_validation_failed" }); + await review; + for (let i = 0; i < 10 && f.events.listenerCount("response"); i++) await Promise.resolve(); + expect(f.events.listenerCount("response")).toBe(0); + resolveBody(Buffer.from(JSON.stringify({ status: "fail", checks: [] }))); + await result; + expect(f.receipt().validation.outcome).toBe("fail"); + expect(f.receipt().checkpoint).toBe("validation"); + expect(f.events.eventNames()).toEqual([]); + }); + + it.each(["details", "top-level"])("retains a known %s server rejection without its free text", async location => { + const f = fixture({ click: () => { f.respond({ error: secret, ...(location === "details" ? { details: { code: "agent_runner_config_incompatible", fields: [secret] } } : { code: "agent_runner_unavailable" }) }, 422); } }); + await expect(f.run()).rejects.toMatchObject({ code: "onboarding_native_validation_failed" }); + expect(f.receipt()).toMatchObject({ checkpoint: "validation", validation: { state: "json", httpStatus: 422, serverCode: location === "details" ? "agent_runner_config_incompatible" : "agent_runner_unavailable" }, hire: { state: "missing" } }); + expect(JSON.stringify(f.receipt())).not.toContain(secret); + validateOnboardingConnectEvidence(f.receipt()); + expect(f.events.listenerCount("response")).toBe(0); + }); + + it.each([{ status: "fail", checks: [] }, { status: "pass", checks: [{ code: "adapter_auth_missing", level: "error" }] }])("fails on a 200 validation rejection: %j", async body => { + const f = fixture({ click: () => { f.respond(body); } }); + await expect(f.run()).rejects.toMatchObject({ code: "onboarding_native_validation_failed" }); + expect(f.receipt().validation.httpStatus).toBe(200); + }); + + it("retains the hire boundary rejection and does not wait for Get started", async () => { + const f = fixture({ click: () => { f.respond({ details: { code: "agent_runner_platform_probe_failed" }, error: secret }, 503, hire); } }); + await expect(f.run()).rejects.toMatchObject({ code: "onboarding_agent_hire_failed" }); + expect(f.receipt()).toMatchObject({ checkpoint: "hire", hire: { httpStatus: 503, serverCode: "agent_runner_platform_probe_failed" } }); + }); + + it("projects only enums from malicious nested fields, including unknown codes and bounded checks", async () => { + const f = fixture({ ready: true, click: () => { f.respond({ adapterType: secret, status: secret, code: { credential: secret }, details: { code: secret }, + message: secret, hint: secret, headers: { authorization: secret }, checks: Array.from({ length: 40 }, () => ({ code: secret, level: secret, message: secret, nested: { apiKey: secret } })) }); } }); + const result = await f.run(); + expect(result.validation).toMatchObject({ adapter: "unrecognized", outcome: "unrecognized", serverCode: "unrecognized", truncatedChecks: true }); + expect(result.validation.checks).toHaveLength(32); + expect(result.validation.checks[0]).toEqual({ code: "unrecognized", level: "unrecognized" }); + expect(JSON.stringify(result)).not.toContain(secret); + validateOnboardingConnectEvidence(result); + expect(() => validateOnboardingConnectEvidence({ ...result, validation: { ...result.validation, requestedAdapter: ["codex_local"] } })).toThrow(); + }); + + it("ignores wrong origin, company, method, query, and responses without an in-action request", async () => { + vi.useFakeTimers(); + const f = fixture({ click: () => { + f.respond({}, 500, validation, "POST", "https://foreign.example.test"); + f.respond({}, 500, validation.replace(companyId, "company-2")); + f.respond({}, 500, validation, "GET"); + f.respond({}, 500, validation + "?private=" + secret); + f.events.emit("response", { request: () => f.request(), status: () => 500 }); + } }); + const result = expect(f.run()).rejects.toMatchObject({ code: "onboarding_get_started_timeout" }); + await vi.advanceTimersByTimeAsync(101); await result; + expect(f.receipt()).toMatchObject({ checkpoint: "connect", validation: { state: "missing" }, hire: { state: "missing" } }); + expect(f.events.eventNames()).toEqual([]); + }); + + it.each([Buffer.from("{not-json"), Buffer.alloc(65_537, "x")])("keeps malformed or oversized bodies unknown and still requires Get started", async bytes => { + vi.useFakeTimers(); + const f = fixture({ click: () => { f.respond(bytes); } }); + const result = expect(f.run()).rejects.toMatchObject({ code: "onboarding_get_started_timeout" }); + await vi.advanceTimersByTimeAsync(101); await result; + expect(f.receipt().validation.state).toBe(bytes.byteLength > 65_536 ? "oversized" : "unreadable"); + expect(f.receipt().checkpoint).toBe("validation"); + }); + + it("records missing responses and ignores late responses after timeout", async () => { + vi.useFakeTimers(); + const f = fixture({ click: () => { f.events.emit("request", f.request()); } }); + const result = expect(f.run()).rejects.toMatchObject({ code: "onboarding_get_started_timeout" }); + await vi.advanceTimersByTimeAsync(101); await result; + const sealed = JSON.stringify(f.receipt()); + f.respond({ status: "pass", adapterType: "paperclip_runner", checks: [] }); + expect(JSON.stringify(f.receipt())).toBe(sealed); + expect(f.receipt().validation.state).toBe("missing"); + }); + + it("bounds stalled JSON reads and cannot mutate retained evidence after return or reject unhandled", async () => { + vi.useFakeTimers(); + let rejectBody!: (reason: Error) => void; + const body = new Promise((_resolve, reject) => { rejectBody = reject; }); + const f = fixture({ ready: true, durationMs: 10_000, click: () => { + const req = f.request(); f.events.emit("request", req); + f.events.emit("response", { request: () => req, status: () => 200, body: () => body }); + } }); + const result = f.run(); + await vi.advanceTimersByTimeAsync(5001); + expect((await result).validation.state).toBe("unreadable"); + const sealed = JSON.stringify(f.receipt()); + rejectBody(new Error(secret)); await Promise.resolve(); + expect(JSON.stringify(f.receipt())).toBe(sealed); + expect(f.events.eventNames()).toEqual([]); + }); + + it("preserves the Get started timeout when a previously received body arrives later", async () => { + vi.useFakeTimers(); + let resolveBody!: (bytes: Buffer) => void; + const body = new Promise(resolve => { resolveBody = resolve; }); + const f = fixture(); + const result = expect(f.run()).rejects.toMatchObject({ code: "onboarding_get_started_timeout" }); + await vi.advanceTimersByTimeAsync(10); + const req = f.request(); f.events.emit("request", req); + f.events.emit("response", { request: () => req, status: () => 200, body: () => body }); + await vi.advanceTimersByTimeAsync(90); + resolveBody(Buffer.from(JSON.stringify({ status: "fail" }))); + await vi.advanceTimersByTimeAsync(1); await result; + expect(f.receipt().checkpoint).toBe("validation"); + expect(f.events.eventNames()).toEqual([]); + }); + + it("keeps a browser error at its own boundary without retaining the browser text", async () => { + const f = fixture({ click: () => { throw new Error(secret); } }); + await expect(f.run()).rejects.toMatchObject({ code: "onboarding_connect_failed" }); + expect(f.receipt().checkpoint).toBe("connect"); + expect(JSON.stringify(f.receipt())).not.toContain(secret); + expect(f.events.eventNames()).toEqual([]); + }); + + it("observes both successful responses and requires the visible review step", async () => { + const f = fixture({ ready: true, click: () => { + f.respond({ adapterType: "paperclip_runner", status: "pass", checks: [{ code: "codex_hello_probe_passed", level: "info", message: secret }] }, 200, validation.replace("paperclip_runner", "codex_local")); + f.respond({ agent: { id: secret }, approval: null }, 201, hire); + } }); + expect(await f.run()).toMatchObject({ checkpoint: "review", validation: { requestedAdapter: "codex_local", adapter: "paperclip_runner", outcome: "pass", checks: [{ code: "codex_hello_probe_passed", level: "info" }] }, hire: { state: "json", httpStatus: 201 } }); + expect(JSON.stringify(f.receipt())).not.toContain(secret); + expect(f.events.eventNames()).toEqual([]); + }); +}); diff --git a/tests/runner-e2e/product-journeys/onboarding-connect.ts b/tests/runner-e2e/product-journeys/onboarding-connect.ts new file mode 100644 index 00000000000..4aa6e43becf --- /dev/null +++ b/tests/runner-e2e/product-journeys/onboarding-connect.ts @@ -0,0 +1,131 @@ +import type { Page, Response, Request } from "@playwright/test"; +import { JourneyError } from "./types.js"; + +export const onboardingServerCodes = ["agent_runner_config_incompatible", "agent_runner_unavailable", "agent_runner_platform_probe_failed", "managed_sandbox_unavailable", "environment_company_mismatch"] as const; +export const onboardingCheckCodes = ["adapter_auth_missing", "codex_hello_probe_failed", "codex_hello_probe_passed", "codex_hello_probe_timeout", "codex_hello_probe_auth_required", "environment_lease_acquire_failed", "environment_workspace_realize_failed", "environment_target_failed", "environment_target_unsupported", "environment_not_found", "environment_target_unavailable", "managed_sandbox_unavailable", "environment_env_binding_missing", "environment_env_resolution_failed", "ai_connection_api_key_rejected", "ai_connection_verification_failed", "sandbox_test_identity"] as const; +const checkpoints = ["connect", "validation", "hire", "review", "agent-verification"] as const; +const states = ["missing", "pending", "json", "unreadable", "oversized"] as const; +const requestedAdapters = ["codex_local", "paperclip_runner"] as const; +const adapters = ["codex_local", "paperclip_runner", "unrecognized"] as const; +const outcomes = ["pass", "warn", "fail", "unrecognized"] as const; +const levels = ["info", "warn", "error", "unrecognized"] as const; +type Known = T[number] | "unrecognized"; +export interface OnboardingResponseEvidence { + state: typeof states[number]; httpStatus: number | null; + requestedAdapter: "codex_local" | "paperclip_runner" | null; + adapter: typeof adapters[number] | null; outcome: typeof outcomes[number] | null; + serverCode: Known | null; + checks: Array<{ code: Known; level: typeof levels[number] }>; + truncatedChecks: boolean; +} +export interface OnboardingConnectEvidence { + version: 1; checkpoint: typeof checkpoints[number]; validation: OnboardingResponseEvidence; hire: OnboardingResponseEvidence; +} +const blank = (): OnboardingResponseEvidence => ({ state: "missing", httpStatus: null, requestedAdapter: null, adapter: null, outcome: null, serverCode: null, checks: [], truncatedChecks: false }); +const record = (v: unknown): v is Record => Boolean(v && typeof v === "object" && !Array.isArray(v)); +const member = (values: T, v: unknown): v is T[number] => typeof v === "string" && values.includes(v); +const closed = (v: unknown, keys: string[]) => record(v) && Object.keys(v).length === keys.length && keys.every(key => Object.hasOwn(v, key)); +export function validateOnboardingConnectEvidence(value: unknown): void { + if (!closed(value, ["version", "checkpoint", "validation", "hire"]) || !record(value) || value.version !== 1 || !member(checkpoints, value.checkpoint)) throw new Error("invalid_onboarding_connect_evidence"); + for (const response of [value.validation, value.hire]) { + if (!closed(response, ["state", "httpStatus", "requestedAdapter", "adapter", "outcome", "serverCode", "checks", "truncatedChecks"]) || !record(response) || + !member(states, response.state) || !(response.httpStatus === null || Number.isSafeInteger(response.httpStatus) && Number(response.httpStatus) >= 100 && Number(response.httpStatus) <= 599) || + !(response.requestedAdapter === null || member(requestedAdapters, response.requestedAdapter)) || + !(response.adapter === null || member(adapters, response.adapter)) || !(response.outcome === null || member(outcomes, response.outcome)) || + !(response.serverCode === null || response.serverCode === "unrecognized" || member(onboardingServerCodes, response.serverCode)) || + typeof response.truncatedChecks !== "boolean" || !Array.isArray(response.checks) || response.checks.length > 32 || + response.checks.some(check => !closed(check, ["code", "level"]) || !record(check) || !(check.code === "unrecognized" || member(onboardingCheckCodes, check.code)) || !member(levels, check.level))) throw new Error("invalid_onboarding_connect_evidence"); + } +} + +/** Observe only responses to this Connect action; never retain bodies, text, headers or browser errors. */ +export async function onboardingConnect(input: { page: Page; origin: string; companyId: string; deadlineAt: number; retain(value: OnboardingConnectEvidence): void }): Promise { + const { page } = input, receipt: OnboardingConnectEvidence = { version: 1, checkpoint: "connect", validation: blank(), hire: blank() }; + input.retain(receipt); + let active = true, failure: JourneyError | undefined; + let signalFailure!: (error: JourneyError) => void; + const failed = new Promise(resolve => { signalFailure = resolve; }); + const requests = new Set(), pending = new Set>(); + const deadlineAt = Math.min(input.deadlineAt, Date.now() + 180_000); + const remaining = () => Math.max(1, deadlineAt - Date.now()); + const advance = (checkpoint: OnboardingConnectEvidence["checkpoint"]) => { + if (checkpoints.indexOf(checkpoint) > checkpoints.indexOf(receipt.checkpoint)) receipt.checkpoint = checkpoint; + }; + const reject = (code: string) => { failure ??= new JourneyError("failed", code); signalFailure(failure); }; + const route = (request: Request): { kind: "validation" | "hire"; adapter: "codex_local" | "paperclip_runner" | null } | null => { + try { + const url = new URL(request.url()); + if (request.method() !== "POST" || url.origin !== input.origin || url.search || url.hash) return null; + const prefix = `/api/companies/${encodeURIComponent(input.companyId)}`; + if (url.pathname === `${prefix}/agent-hires`) return { kind: "hire", adapter: null }; + for (const adapter of ["codex_local", "paperclip_runner"] as const) if (url.pathname === `${prefix}/adapters/${adapter}/test-environment`) return { kind: "validation", adapter }; + } catch { /* An unrelated or malformed URL is not this action's evidence. */ } + return null; + }; + const onRequest = (request: Request) => { if (active && route(request)) requests.add(request); }; + const onResponse = (response: Response) => { + const request = response.request(), matched = route(request); + if (!active || !matched || !requests.delete(request)) return; + const observed = receipt[matched.kind]; + // One Connect action can report each boundary once. Later traffic cannot replace its failure. + if (observed.state !== "missing") return; + advance(matched.kind); observed.state = "pending"; + const status = response.status(); + observed.httpStatus = Number.isSafeInteger(status) && status >= 100 && status <= 599 ? status : null; + observed.requestedAdapter = matched.adapter; + if (response.status() >= 400) reject(matched.kind === "validation" ? "onboarding_native_validation_failed" : "onboarding_agent_hire_failed"); + const classify = async () => { + let timer: ReturnType | undefined; + try { + const bytes = await Promise.race([response.body(), new Promise(resolve => { timer = setTimeout(() => resolve(null), Math.min(5000, remaining())); })]); + if (!active) return; + if (!bytes) { observed.state = "unreadable"; return; } + if (bytes.byteLength > 65_536) { observed.state = "oversized"; return; } + let body: unknown; try { body = JSON.parse(bytes.toString("utf8")); } catch { observed.state = "unreadable"; return; } + if (!record(body)) { observed.state = "unreadable"; return; } + observed.state = "json"; + const detailCode = record(body.details) ? body.details.code : undefined; + const code = member(onboardingServerCodes, detailCode) ? detailCode : member(onboardingServerCodes, body.code) ? body.code : detailCode ?? body.code; + if (code !== undefined) observed.serverCode = member(onboardingServerCodes, code) ? code : "unrecognized"; + if (matched.kind === "validation") { + observed.adapter = member(adapters, body.adapterType) ? body.adapterType : "unrecognized"; + observed.outcome = member(outcomes, body.status) ? body.status : "unrecognized"; + if (Array.isArray(body.checks)) { + observed.truncatedChecks = body.checks.length > 32; + observed.checks = body.checks.slice(0, 32).map(check => ({ code: record(check) && member(onboardingCheckCodes, check.code) ? check.code : "unrecognized", level: record(check) && member(levels, check.level) ? check.level : "unrecognized" })); + if (body.status === "fail" || body.checks.some(check => record(check) && check.code === "adapter_auth_missing")) reject("onboarding_native_validation_failed"); + } else if (body.status === "fail") reject("onboarding_native_validation_failed"); + } + } catch { if (active) observed.state = "unreadable"; } + finally { clearTimeout(timer); } + }; + const work = classify(); pending.add(work); void work.then(() => pending.delete(work), () => pending.delete(work)); + }; + page.on("request", onRequest); page.on("response", onResponse); + try { + if (Date.now() >= deadlineAt) throw new JourneyError("failed", "onboarding_connect_timeout"); + const clicked = page.getByRole("button", { name: "Connect", exact: true }).click({ timeout: remaining() }) + .then(() => null, error => new JourneyError("failed", error?.name === "TimeoutError" ? "onboarding_connect_timeout" : "onboarding_connect_failed")); + const clickResult = await Promise.race([failed, clicked]); + if (clickResult) throw clickResult; + if (failure) throw failure; + const result = await Promise.race([failed, page.getByRole("button", { name: "Get started", exact: true }).waitFor({ timeout: remaining() }) + .then(() => null, error => new JourneyError("failed", error?.name === "TimeoutError" ? "onboarding_get_started_timeout" : "onboarding_get_started_failed"))]); + const boundaryFailure = failure; + // Seal the observed boundary before awaiting body reads. A response arriving + // after Get started or its timeout cannot change this action's outcome. + page.off("request", onRequest); page.off("response", onResponse); + if (pending.size) await Promise.all([...pending]); + if (boundaryFailure) throw boundaryFailure; + if (result) throw result; + if (failure) throw failure; + if (Date.now() >= deadlineAt) throw new JourneyError("failed", "onboarding_get_started_timeout"); + advance("review"); + return receipt; + } finally { + // Settle only already observed responses within their bounded read; detach before later traffic. + page.off("request", onRequest); page.off("response", onResponse); + if (pending.size) await Promise.all([...pending]); + active = false; requests.clear(); + } +} diff --git a/tests/runner-e2e/product-journeys/types.ts b/tests/runner-e2e/product-journeys/types.ts index 65941f30d86..da0b054a606 100644 --- a/tests/runner-e2e/product-journeys/types.ts +++ b/tests/runner-e2e/product-journeys/types.ts @@ -28,6 +28,7 @@ export interface JourneyEvidence { attemptId?: string; operations?: JourneyOperations; artifact?: OnboardingArtifactEvidence; + onboardingConnect?: import("./onboarding-connect.js").OnboardingConnectEvidence; } export interface OnboardingArtifactEvidence { schema: "paperclip.product-journey.artifact/v1"; diff --git a/tests/runner-e2e/result-validation.ts b/tests/runner-e2e/result-validation.ts index 883d928c510..56f38ed0bec 100644 --- a/tests/runner-e2e/result-validation.ts +++ b/tests/runner-e2e/result-validation.ts @@ -4,6 +4,7 @@ import { githubJourneyCheckpoints, gradeGitHubReviewJourney } from "./github-rev import { STARTUP_REUSE_LAYERS, STARTUP_REUSE_REASONS } from "../../packages/adapter-utils/src/startup-reuse.js"; import type { RunnerE2ECampaignAccounting, RunnerE2EResult } from "./types.js"; import { DEPENDABLE_CELLS } from "./dependable-qualification.js"; +import { validateOnboardingConnectEvidence } from "./product-journeys/onboarding-connect.js"; import { validateOnboardingArtifact } from "./product-journeys/artifact-evidence.js"; type Rule = (value: unknown, at: string) => void; @@ -344,6 +345,7 @@ const fields = { productJourney: optional(shape({ targetId: optional(string), campaignId: optional(string), attemptId: optional(string), + onboardingConnect: optional((value, at) => { try { validateOnboardingConnectEvidence(value); } catch { invalid(at); } }), artifact: optional((value, at) => { try { validateOnboardingArtifact(value); } catch { invalid(at); } }), operations: optional(closedShape({ campaignId: string, attemptId: string, modelCostUsd: nullable(number), setupCostUsd: nullable(number), requestCount: integer, feedback: closedShape({ collection: oneOf("complete", "partial", "unavailable"), complaints: integer, suggestions: integer }), traceIds: array(string) })), From a0e5166e974a7f6ffabe2a43f816a83449fdef57 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 24 Oct 2026 21:39:21 -0700 Subject: [PATCH 16/68] Fix GitHub journey registration diagnostics and native retry cleanup (#17513) ## Thinking Path > - Paperclip manages work performed by AI agents. > - The Product E2E harness verifies a GitHub review-bot journey. > - GitHub can return a registration page without the expected creation control. > - The journey could then wait for the full login allowance without useful evidence. > - A failed native run can also retain a retry after its terminal status is visible. > - This change records a safe registration boundary and uses the normal Stop route for owned retries. > - Operators can diagnose the setup failure and confirm cleanup without changing the behavioral oracle. ## Linked Issues or Issue Description Refs: #17457. **What happened?** The GitHub review journey waited at App registration with no useful handoff evidence. Its cleanup also skipped failed native runs that still owned a durable retry. **Expected behavior** The journey should report a bounded registration failure or reauthentication boundary. Cleanup should Stop an owned native retry and require its audited acknowledgement. **Steps to reproduce** 1. Run the configured GitHub review journey. 2. Return a GitHub registration page without the expected creation button, or fail a native review run while its coordinator can retry. 3. Inspect the result and cleanup evidence. **Paperclip version or commit** Source built from current master. This extends the driver merged in #17457. ## What Changed - Bound the registration render wait to 30 seconds. Keep the separate attended confirmation allowance. - Retain GitHub origin, path, and control-presence flags. Retain headings, button labels, and alerts only on known registration error pages without authentication fields. - Exclude query parameters, entered values, page bodies, grant payloads, and authentication content from that diagnostic. - Add an optional private authentication hook for ordinary GitHub access reconfirmation. Detect passkey-first sudo at the manifest URL without retaining its authentication content. - Stop owned failed native retries through the existing run-only cancellation route. Verify company, agent, task, and audit acknowledgement. - Use native Stop correlation only for native runs. Queued legacy or unresolved runs use the ordinary cancel request. Stop scheduled retries and wait for them to leave the live state set. - Keep pending accounting debt intact. Add regression coverage and journey documentation. ## Verification - Focused journey and report suite: 58 tests passed. - Runner E2E TypeScript check passed. - `git diff --check` passed. Credential-value scan passed. - Live fixture reproduced the missing creation control at the manifest handoff. The new diagnostic retained its path and control-presence flags. No model work started. Draft cleanup passed. - All 58 current-head checks are terminal and green. Greptile scored the exact head 5/5, with no unresolved review threads. ## Risks GitHub pages that take more than 30 seconds to render the creation control now fail early. Native cleanup fails closed if the server does not provide the expected Stop acknowledgement. The live setup-to-Done behavioral journey is still unqualified. These changes do not repair a missing provider usage receipt or weaken its accounting gate. ## Model Used OpenAI Codex, based on GPT-6. Exact runtime model identifier and context-window size are not exposed in this session. Used for code execution, browser tooling, and regression analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip Co-authored-by: Paperclip --- tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md | 9 ++ tests/runner-e2e/github-review-browser.ts | 48 +++++++++ tests/runner-e2e/github-review-flow.ts | 8 +- .../runner-e2e/github-review-journey.test.ts | 101 +++++++++++++++++- tests/runner-e2e/github-review-launch.ts | 19 ++-- tests/runner-e2e/github-review-ownership.ts | 31 ++++++ 6 files changed, 203 insertions(+), 13 deletions(-) diff --git a/tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md b/tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md index ceeb56a95e0..12cddb6c387 100644 --- a/tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md +++ b/tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md @@ -89,6 +89,15 @@ connections, and completed task evidence. An unconfirmed cleanup fails the attempt; retain the resource receipt for recovery. Read the receipt before any manual cleanup. Never delete by a guessed name or broad account search. +Registration render waits are bounded separately from attended confirmation. +The driver retains only the GitHub origin/path and control-presence flags when +the creation control is missing. On a known registration error page without auth +fields, it also retains headings, button labels, and alerts. It excludes query +parameters and authentication content. Reauthentication is a setup boundary, +not evidence that an App exists. +Cleanup also stops failed native runs that still own a retry, and requires the +normal run-only Stop acknowledgement. It never clears pending accounting debt. + The configured deadline and observed run limit bound the attempt. They are not a hard provider-dollar cap or proof that no request outlives cancellation. Use the existing budgeted Cloud operations adapter before automated paid campaigns; diff --git a/tests/runner-e2e/github-review-browser.ts b/tests/runner-e2e/github-review-browser.ts index dffb516bcdd..b064d0ad384 100644 --- a/tests/runner-e2e/github-review-browser.ts +++ b/tests/runner-e2e/github-review-browser.ts @@ -10,6 +10,54 @@ export async function enterGitHubJourneyCompany(page: Page, origin: string, comp await page.waitForURL(url => url.origin === origin && url.pathname === pathname); } +export interface GitHubRegistrationBoundary { + origin: string; pathname: string; passwordRequired: boolean; oneTimeCodeRequired: boolean; + reauthenticationRequired: boolean; createControlVisible: boolean; + controls?: { headings: string[]; buttons: string[]; alerts: string[] }; +} + +/** Keep only navigation and control-presence evidence. Authentication fields, + * page content, query parameters, and grant payloads must stay out of reports. */ +export async function waitForGitHubRegistrationControl(page: Page, owner: string, timeoutMs: number, + record: (boundary: GitHubRegistrationBoundary) => Promise = async () => {}, reauthenticate?: () => Promise) { + const control = page.getByRole("button", { name: new RegExp(`^Create GitHub App(?: for ${owner})?$`) }); + const observe = async (includeErrorControls = false) => { + const url = new URL(page.url()); + if (url.origin !== "https://github.com") throw new ConnectionFailure("github_registration_origin_mismatch"); + const boundary: GitHubRegistrationBoundary = { origin: url.origin, pathname: url.pathname, + passwordRequired: await page.locator('input[type="password"]:visible').count() > 0, + oneTimeCodeRequired: await page.locator('input[autocomplete="one-time-code"]:visible').count() > 0, + reauthenticationRequired: await page.getByRole("heading", { name: /^Confirm access$/i }).isVisible() + || await page.getByRole("button", { name: /^Use (?:your password|passkey)$/i }).isVisible(), + createControlVisible: await control.isVisible() }; + boundary.reauthenticationRequired ||= boundary.passwordRequired || boundary.oneTimeCodeRequired + || /\/(?:login|sessions)(?:\/|$)/.test(boundary.pathname); + if (includeErrorControls && !boundary.reauthenticationRequired + && ["/settings/apps/new", "/settings/apps/manifest"].includes(url.pathname)) { + boundary.controls = { headings: await page.getByRole("heading").allTextContents(), + buttons: await page.getByRole("button").allTextContents(), alerts: await page.getByRole("alert").allTextContents() }; + } + await record(boundary); + return boundary; + }; + await observe(); + let authenticationAttempted = false; + for (;;) { + try { await control.waitFor({ timeout: Math.min(timeoutMs, 30_000) }); break; } + catch (error) { + if (page.isClosed()) throw error; + const boundary = await observe(true); + if (boundary.reauthenticationRequired) { + if (reauthenticate && !authenticationAttempted) { authenticationAttempted = true; await reauthenticate(); continue; } + throw new ConnectionBlock("awaiting_user", "github_registration_reauthentication_required"); + } + throw new ConnectionFailure("github_registration_create_control_missing"); + } + } + await observe(); + return control; +} + /** Capture measured pages only, excluding authentication and unrelated GitHub data. */ export function isGitHubJourneyEvidencePage(value: string, targetOrigin: string, repository: string, appSlug?: string) { const url = new URL(value); diff --git a/tests/runner-e2e/github-review-flow.ts b/tests/runner-e2e/github-review-flow.ts index c580ed767fa..5547adf21a2 100644 --- a/tests/runner-e2e/github-review-flow.ts +++ b/tests/runner-e2e/github-review-flow.ts @@ -3,7 +3,7 @@ import type { GitHubTaskReview } from "../../packages/shared/src/types/chat-gith import type { GitHubRepositoryPage } from "../../packages/shared/src/types/chat-channels.js"; import { ConnectionBlock, ConnectionFailure } from "./connection-config.js"; import { ConnectionApi } from "./connection-target.js"; -import { awaitJourneyCondition, createGitHubJourneyPullRequest, enterGitHubJourneyCompany, observeGitHubHead, selectGitHubJourneyRepository } from "./github-review-browser.js"; +import { awaitJourneyCondition, createGitHubJourneyPullRequest, enterGitHubJourneyCompany, observeGitHubHead, selectGitHubJourneyRepository, waitForGitHubRegistrationControl, type GitHubRegistrationBoundary } from "./github-review-browser.js"; import { githubCartFixture, hasOnlyOpenedAutomaticReview, matchesGitHubJourneyEnvironment, retainGitHubReview, verifyGitHubReview, verifyGitHubReviewRun, type GitHubReviewJourneyEvidence } from "./github-review-evidence.js"; import type { GitHubReviewJourneyConfig } from "./github-review-config.js"; import { captureOwnedGitHubApp, readOwnedGitHubReviewRuns, type GitHubJourneyResources } from "./github-review-ownership.js"; @@ -15,6 +15,8 @@ export async function runGitHubReviewFlow(input: { checkpoint(id: keyof GitHubReviewJourneyEvidence["checkpoints"], page: Page): Promise; phase(id: string): Promise; persistResources(): Promise; + recordRegistrationBoundary?(boundary: GitHubRegistrationBoundary): Promise; + authenticateRegistration?(page: Page): Promise; }) { const { page, github, api, config, resources, evidence } = input; const capabilities = await api.get('/api/instance/settings/experimental'); @@ -69,8 +71,8 @@ export async function runGitHubReviewFlow(input: { // Manifest flow navigates the same tab. External App grants remain attended. evidence.assisted = true; await awaitJourneyCondition(async () => new URL(page.url()).hostname === "github.com", 30_000, "github_manifest_handoff_missing", input.signal); - const createApp = page.getByRole("button", { name: new RegExp(`^Create GitHub App(?: for ${config.github.appOwner})?$`) }); - await createApp.waitFor({ timeout: config.browser.loginTimeoutMs }); + const createApp = await waitForGitHubRegistrationControl(page, config.github.appOwner, config.browser.loginTimeoutMs, + input.recordRegistrationBoundary, input.authenticateRegistration ? () => input.authenticateRegistration!(page) : undefined); await page.bringToFront(); await input.checkpoint("app_creation_consent", page); console.log(`[github-review-journey] Authorize creation of only ${resources.appName} in the dedicated QA browser. Install it only on ${config.github.repository}.`); diff --git a/tests/runner-e2e/github-review-journey.test.ts b/tests/runner-e2e/github-review-journey.test.ts index 8489b4596b0..09ce3e869dd 100644 --- a/tests/runner-e2e/github-review-journey.test.ts +++ b/tests/runner-e2e/github-review-journey.test.ts @@ -8,9 +8,9 @@ import type { GitHubTaskReview } from "../../packages/shared/src/types/chat-gith import { verifyConnectionTarget } from "./connection-target.js"; import { parseConnectionConfig } from "./connection-config.js"; import type { APIRequestContext } from "@playwright/test"; -import { captureOwnedGitHubApp, readOwnedGitHubReviewRuns, type GitHubJourneyResources } from "./github-review-ownership.js"; +import { captureOwnedGitHubApp, readOwnedGitHubReviewRuns, stopOwnedGitHubReviewRun, type GitHubJourneyResources } from "./github-review-ownership.js"; import type { ChatEndpoint } from "../../packages/shared/src/types/chat-channels.js"; -import { enterGitHubJourneyCompany, isGitHubJourneyEvidencePage } from "./github-review-browser.js"; +import { enterGitHubJourneyCompany, isGitHubJourneyEvidencePage, waitForGitHubRegistrationControl } from "./github-review-browser.js"; import { attendGitHubJourneyAction } from "./github-review-attendance.js"; import { PassThrough } from "node:stream"; import type { Locator, Page } from "@playwright/test"; @@ -19,7 +19,104 @@ import { runGitHubReviewCampaign } from "./github-review-launch.js"; import type { ConnectionApi } from "./connection-target.js"; import { runGitHubReviewFlow } from "./github-review-flow.js"; +describe("GitHub registration handoff", () => { + const page = (pathname: string, password: boolean, available: boolean) => ({ + url: () => `https://github.com${pathname}?return_to=private-grant&token=never-retain`, isClosed: () => false, + locator: (selector: string) => ({ count: async () => selector.includes('password') && password ? 1 : 0 }), + getByRole: (_role?: string, options?: { name?: string | RegExp }) => ({ isVisible: async () => String(options?.name).includes('Create GitHub App') && available, allTextContents: async () => ['Registration error'], + waitFor: vi.fn(async () => { if (!available) throw new Error("control_missing"); }) }), + }); + it("classifies a reauthentication boundary and retains no query or auth content", async () => { + const record = vi.fn(); + await expect(waitForGitHubRegistrationControl(page('/sessions/sudo', true, false) as unknown as Page, 'fixture', 600000, record)) + .rejects.toThrow('github_registration_reauthentication_required'); + expect(record.mock.calls.at(-1)?.[0]).toEqual({ origin: 'https://github.com', pathname: '/sessions/sudo', passwordRequired: true, oneTimeCodeRequired: false, reauthenticationRequired: true, createControlVisible: false }); + expect(JSON.stringify(record.mock.calls)).not.toMatch(/private-grant|never-retain/); + }); + it("fails a missing creation control on an ordinary registration page", async () => { + const record = vi.fn(); + await expect(waitForGitHubRegistrationControl(page('/settings/apps/manifest', false, false) as unknown as Page, 'fixture', 600000, record)) + .rejects.toThrow('github_registration_create_control_missing'); + expect(record.mock.calls.at(-1)?.[0].controls.alerts).toEqual(['Registration error']); + }); + it("admits the ordinary creation control with a bounded render wait", async () => { + const target = page('/settings/apps/new', false, true); + const control = target.getByRole('button', { name: /^Create GitHub App/ }); + const get = target.getByRole; + target.getByRole = (role, options) => String(options?.name).includes('Create GitHub App') ? control : get(role, options); + await expect(waitForGitHubRegistrationControl(target as unknown as Page, 'fixture', 600000)).resolves.toBe(control); + expect(control.waitFor).toHaveBeenCalledWith({ timeout: 30000 }); + }); + it("keeps passkey-first sudo at the manifest path out of authentication content diagnostics", async () => { + const target = page('/settings/apps/manifest', false, false), get = target.getByRole; + target.getByRole = (role, options) => String(options?.name).includes('Confirm access') + ? { ...get(role, options), isVisible: async () => true } : get(role, options); + const record = vi.fn(); + await expect(waitForGitHubRegistrationControl(target as unknown as Page, 'fixture', 600000, record)) + .rejects.toThrow('github_registration_reauthentication_required'); + expect(record.mock.calls.at(-1)?.[0].reauthenticationRequired).toBe(true); + expect(record.mock.calls.at(-1)?.[0].controls).toBeUndefined(); + }); + it("uses a supplied ordinary reauthentication adapter once before retrying creation readiness", async () => { + const target = page('/settings/apps/manifest', true, false), get = target.getByRole; + let authenticated = false; + const control = { ...get(), isVisible: async () => authenticated, + waitFor: vi.fn(async () => { if (!authenticated) throw new Error('sudo_required'); }) }; + target.getByRole = (role, options) => String(options?.name).includes('Create GitHub App') ? control : get(role, options); + const authenticate = vi.fn(async () => { authenticated = true; }); + await expect(waitForGitHubRegistrationControl(target as unknown as Page, 'fixture', 600000, async () => {}, authenticate)).resolves.toBe(control); + expect(authenticate).toHaveBeenCalledOnce(); + }); +}); + describe("reused target ownership", () => { + it("stops a scheduled legacy retry and waits until its cancellation finishes", async () => { + const scope = { companyId: 'company', agentId: 'agent', issueId: 'issue' }; + const scheduled = { ...scope, id: 'run', status: 'scheduled_retry', runtimeMode: 'legacy' }; + const get = vi.fn().mockResolvedValueOnce(scheduled).mockResolvedValueOnce(scheduled) + .mockResolvedValue({ ...scheduled, status: 'cancelled' }); + const post = vi.fn().mockResolvedValue({}); + await stopOwnedGitHubReviewRun({ get, post } as unknown as ConnectionApi, 'run', scope, new AbortController().signal); + expect(post).toHaveBeenCalledExactlyOnceWith('/api/heartbeat-runs/run/cancel', {}); + expect(get).toHaveBeenCalledTimes(3); + }); + it.each(['legacy', undefined])("stops a queued %s run without a native-only cancellation request", async runtimeMode => { + const scope = { companyId: 'company', agentId: 'agent', issueId: 'issue' }; + const queued = { ...scope, id: 'run', status: 'queued', runtimeMode }; + const get = vi.fn().mockResolvedValueOnce(queued).mockResolvedValue({ ...queued, status: 'cancelled' }); + const post = vi.fn(async (_route: string, body: Record) => { + if (body.cancellationRequestId) throw new Error('Correlated Stop requires a native run'); + return {}; + }); + await stopOwnedGitHubReviewRun({ get, post } as unknown as ConnectionApi, 'run', scope, new AbortController().signal); + expect(post).toHaveBeenCalledExactlyOnceWith('/api/heartbeat-runs/run/cancel', {}); + expect(get).toHaveBeenCalledTimes(2); + }); + it.each([true, false])("stops a failed native retry and verifies its audited acknowledgement (dispatched=%s)", async dispatched => { + const scope = { companyId: 'company', agentId: 'agent', issueId: 'issue' }; + const failed = { ...scope, id: 'run', status: 'failed', runtimeMode: 'native', nativePhase: 'retryable_failure' }; + const get = vi.fn().mockResolvedValueOnce(failed).mockResolvedValue({ ...failed, status: 'cancelled', costAccountingPending: true, + resultJson: { nativeCancellation: { schema: 'paperclip.native-cancellation.v1', runId: 'run', companyId: 'company', issueId: 'issue', scope: 'run', reasonCode: 'cancellation_run_only', dispatched, dispatchState: 'acknowledged', intentAuditId: 'intent', acknowledgementAuditId: 'ack' } } }); + const post = vi.fn().mockResolvedValue({}); + await stopOwnedGitHubReviewRun({ get, post } as unknown as ConnectionApi, 'run', scope, new AbortController().signal); + expect(post).toHaveBeenCalledExactlyOnceWith('/api/heartbeat-runs/run/cancel', { cancellationRequestId: expect.any(String) }); + expect(get).toHaveBeenCalledTimes(2); + }); + it("refuses to stop a related run belonging to a different task", async () => { + const post = vi.fn(); + const api = { get: async () => ({ id: 'run', companyId: 'company', agentId: 'agent', issueId: 'other', status: 'running' }), post }; + await expect(stopOwnedGitHubReviewRun(api as unknown as ConnectionApi, 'run', { companyId: 'company', agentId: 'agent', issueId: 'issue' }, new AbortController().signal)) + .rejects.toThrow('cleanup_run_owner_mismatch'); + expect(post).not.toHaveBeenCalled(); + }); + it("does not accept cancelled status as native Stop acknowledgement", async () => { + const scope = { companyId: 'company', agentId: 'agent', issueId: 'issue' }; + const get = vi.fn().mockResolvedValueOnce({ ...scope, id: 'run', status: 'running', runtimeMode: 'native' }) + .mockResolvedValue({ ...scope, id: 'run', status: 'cancelled', runtimeMode: 'native' }); + await expect(stopOwnedGitHubReviewRun({ get, post: async () => ({}) } as unknown as ConnectionApi, 'run', scope, new AbortController().signal, 1)) + .rejects.toThrow('owned_run_cancellation_unconfirmed'); + expect(get.mock.calls.length).toBeGreaterThan(1); + }); it("admits an inherited environment before setup and still leaves actual-run verification mandatory", async () => { const setup = parseGitHubReviewJourneyConfig(config()); const company = { id: setup.target.companyId, issuePrefix: "QA" }; diff --git a/tests/runner-e2e/github-review-launch.ts b/tests/runner-e2e/github-review-launch.ts index a78365f6add..83f4667d90c 100644 --- a/tests/runner-e2e/github-review-launch.ts +++ b/tests/runner-e2e/github-review-launch.ts @@ -12,7 +12,7 @@ import { createPrivateJourneyRepository, ensureGitHubJourneyLogin, closeOwnedGit import { loadGitHubReviewJourneyConfig } from "./github-review-config.js"; import { gradeGitHubReviewJourney, type GitHubReviewJourneyEvidence } from "./github-review-evidence.js"; import { runGitHubReviewFlow } from "./github-review-flow.js"; -import { captureOwnedGitHubApp, readOwnedGitHubReviewRuns, type GitHubJourneyResources } from "./github-review-ownership.js"; +import { captureOwnedGitHubApp, readOwnedGitHubReviewRuns, stopOwnedGitHubReviewRun, type GitHubJourneyResources } from "./github-review-ownership.js"; import { JourneyDependencyError, resolveJourneyCredential } from "./journey-dependencies.js"; import { sanitizeJson, assertSecretFree } from "./redaction.js"; import { writeGitHubReviewReport } from "./github-review-report.js"; @@ -101,7 +101,14 @@ export async function runGitHubReviewCampaign(input: { executions: readonly Matr const identity = await verifyConnectionTarget(config.target.baseURL, common, browser.context.request); evidence.targetCommit = identity.commit; evidence.checkpoints.target = true; await runGitHubReviewFlow({ page, github, githubUsername: config.github.account, api, config, resources, evidence, signal, - phase: checkpoint, persistResources: () => save("owned-resources.json", resources), checkpoint: async (id, tab) => { + phase: checkpoint, persistResources: () => save("owned-resources.json", resources), + ...(typeof authentication?.authenticateGitHubJourneyRegistration === "function" ? { + authenticateRegistration: (tab: Page) => authentication.authenticateGitHubJourneyRegistration({ page: tab, + configFile: config!.authentication!.configFile, expectedAccount: config!.github.account, signal, + deadlineAt: Date.now() + Math.min(config!.browser.loginTimeoutMs, 120_000), + registerSecret: (value: string) => secrets.push(value), checkpoint: (id: string) => checkpoint(id) }), + } : {}), + recordRegistrationBoundary: boundary => save("github-registration-boundary.json", boundary), checkpoint: async (id, tab) => { evidence.checkpoints[id] = true; // Authentication pages and forms are deliberately never captured. const consentUrl = new URL(tab.url()); @@ -180,12 +187,8 @@ export async function runGitHubReviewCampaign(input: { executions: readonly Matr await save("owned-resources.json", resources); await api.post(`/api/chat-endpoints/${resources.endpointId}/setup`, { action: "remove" }); for (const runId of resources.runIds) { - const run = await api.get(`/api/heartbeat-runs/${runId}`); - if (run.companyId !== config!.target.companyId || run.agentId !== config!.agent.id) throw new ConnectionFailure("cleanup_run_owner_mismatch"); - if (["queued", "running"].includes(run.status)) { - await api.post(`/api/heartbeat-runs/${runId}/cancel`); - await awaitJourneyCondition(async () => !["queued", "running"].includes((await api!.get(`/api/heartbeat-runs/${runId}`)).status), 30_000, "owned_run_cancellation_unconfirmed", cleanupSignal, "failed"); - } + if (!resources.issueId) throw new ConnectionFailure("cleanup_run_issue_missing"); + await stopOwnedGitHubReviewRun(api, runId, { companyId: config!.target.companyId, agentId: config!.agent.id, issueId: resources.issueId }, cleanupSignal); } if (resources.issueId && result.status !== "passed") { const issue = await api.get(`/api/issues/${resources.issueId}`); diff --git a/tests/runner-e2e/github-review-ownership.ts b/tests/runner-e2e/github-review-ownership.ts index f1d15ee4855..867ff16e84c 100644 --- a/tests/runner-e2e/github-review-ownership.ts +++ b/tests/runner-e2e/github-review-ownership.ts @@ -2,6 +2,10 @@ import type { ChatEndpoint } from "../../packages/shared/src/types/chat-channels import { ConnectionFailure } from "./connection-config.js"; import type { GitHubReviewJourneyConfig } from "./github-review-config.js"; import type { ConnectionApi } from "./connection-target.js"; +import { randomUUID } from "node:crypto"; +import { awaitJourneyCondition } from "./github-review-browser.js"; + +const liveRunStatuses = new Set(["queued", "running", "scheduled_retry"]); /** The issue run source includes history beyond company list windows. Related * activity can name a run for another task, so hydrate and check direct ownership. */ @@ -13,6 +17,33 @@ export async function readOwnedGitHubReviewRuns(api: Pick, return runs.filter(run => run.issueId === scope.issueId && run.agentId === scope.agentId); } +/** A failed native run can still own a durable retry. Stop it through the + * normal audited route and require acknowledgement, not just terminal status. */ +export async function stopOwnedGitHubReviewRun(api: Pick, runId: string, + scope: { companyId: string; agentId: string; issueId: string }, signal: AbortSignal, timeoutMs = 30_000) { + const read = async () => { + const run = await api.get>(`/api/heartbeat-runs/${runId}`); + if (run.id !== runId || run.companyId !== scope.companyId || run.agentId !== scope.agentId || run.issueId !== scope.issueId) { + throw new ConnectionFailure("cleanup_run_owner_mismatch"); + } + return run; + }; + const run = await read(); + const retry = run.runtimeMode === "native" && run.status === "failed" && run.nativePhase === "retryable_failure"; + if (!liveRunStatuses.has(run.status) && !retry) return; + await api.post(`/api/heartbeat-runs/${runId}/cancel`, run.runtimeMode === "native" ? { cancellationRequestId: randomUUID() } : {}); + await awaitJourneyCondition(async () => { + const stopped = await read(); + if (run.runtimeMode !== "native") return !liveRunStatuses.has(stopped.status); + const intent = stopped.resultJson?.nativeCancellation; + return stopped.status === "cancelled" && intent?.schema === "paperclip.native-cancellation.v1" + && intent.runId === runId && intent.companyId === scope.companyId && intent.issueId === scope.issueId + && intent.scope === "run" && intent.reasonCode === "cancellation_run_only" && typeof intent.dispatched === "boolean" + && intent.dispatchState === "acknowledged" && typeof intent.intentAuditId === "string" && intent.intentAuditId.length > 0 + && typeof intent.acknowledgementAuditId === "string" && intent.acknowledgementAuditId.length > 0; + }, timeoutMs, "owned_run_cancellation_unconfirmed", signal, "failed"); +} + export interface GitHubJourneyResources { endpointId?: string; pullNumber?: number; branch: string; appName: string; appId?: string; appSlug?: string; appSettingsUrl?: string; installationSettingsUrl?: string; From 8415f4a679689eab68af5d41331639cacf7fd6cd Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 24 Oct 2026 21:58:43 -0700 Subject: [PATCH 17/68] fix(github): drain queued pings during active App setup (#17551) ## Thinking Path > - Paperclip connects external work to an agent and its ordinary tasks. > - A dedicated GitHub App must prove signed webhook delivery during setup. > - Cloud queues the original bytes until the tenant can authenticate them. > - An older personal connection can leave a long empty-queue backoff in that tenant. > - Active App setup can then time out before the queued ping is read. > - This change lets active setup request the existing event drain with a short rate limit. > - The same webhook handler still owns signature verification and setup state. ## Linked Issues or Issue Description Refs: #17457, #17513. **What happened?** Cloud accepted a new App's ping, but the tenant read it only after the setup wait timed out and cleanup archived the draft. An existing personal GitHub connection had left an empty-queue backoff. The poller checked that deadline before it read the new App state. **Expected behavior** An active manager setup request must be able to read the queued ping promptly. It must not bypass signature checks or create another App. **Steps to reproduce** 1. Let a personal GitHub connection build its empty-queue backoff. 2. Start a dedicated App and complete its installation. 3. Return to the waiting setup screen while the signed ping is queued. **Paperclip version or commit** This behavior is present on the base commit of this PR. ## What Changed - Let the authenticated wizard request a queue drain only while its Cloud App still lacks signed-delivery verification. Existing manager, credential, and installation checks run first. - Register a narrow callback to the server-owned GitHub poller. Keep provider handling in the GitHub services. Remove the callback on server close. - Share one in-flight attempt between setup and background requests. Limit setup attempts to one per five seconds, including after errors. - Retain the existing passive backoff, sealed queue, acknowledgement rules, and canonical webhook signature handler. - Add regressions for the old backoff, concurrent requests, failed requests, and wizard signature verification. Update the setup documentation and map. ## Verification - Observed the backoff regression fail against the old service: active setup made no additional Cloud lease request. It passes with this fix. - Three focused event-poller tests pass. They use the real admission/backoff code with stubbed SQL and Cloud boundaries. - All 39 startup tests pass. Server TypeScript, map checks, and diff checks pass. - The Postgres integration suites could not start locally because this host exhausted its semaphore limit. They remain required CI checks. This is not a passing local integration result. - Full current-head CI passes, including the Postgres and chat integration suites. Greptile scored this exact head 5/5 with no unresolved findings. - The additional Paperclip review check ended neutral because its execution did not complete. It did not publish a validated assessment. - A fresh deployed setup attempt remains required before live qualification. ## Risks A waiting setup screen now makes a bounded readiness request to Cloud. The existing connector request timeout still applies. Concurrent calls share the same drain, and failures release it for a later retry. The request stops after persisted verification or when the browser stops advancing setup. No timer, durable queue, fleet setting, or sleep blocker is added or removed. The existing background reconciliation path still has its existing sleep restrictions. Live setup-to-review behavior is not established by these local tests. ## Model Used OpenAI Codex, based on GPT-6. The exact runtime model ID and context-window size are not exposed in this session. Used for code execution and regression analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/connections/GITHUB-REVIEW-BOT.md | 7 + maps/feature-map/connection-setup.md | 9 + .../chat-channels.integration.test.ts | 25 ++- .../github-connection-events.test.ts | 20 +- server/src/index.ts | 12 +- .../src/services/chat-github-cloud-ingress.ts | 12 ++ server/src/services/chat-github-wizard.ts | 5 + .../services/github-connection-events.test.ts | 74 ++++++++ .../src/services/github-connection-events.ts | 171 ++++++++++-------- 9 files changed, 243 insertions(+), 92 deletions(-) create mode 100644 server/src/services/github-connection-events.test.ts diff --git a/doc/connections/GITHUB-REVIEW-BOT.md b/doc/connections/GITHUB-REVIEW-BOT.md index 2d9dc55197b..f63732c56a4 100644 --- a/doc/connections/GITHUB-REVIEW-BOT.md +++ b/doc/connections/GITHUB-REVIEW-BOT.md @@ -40,6 +40,13 @@ Installation may return before GitHub's separately delivered signed webhook ping. Paperclip keeps the same draft and credentials, shows a waiting state, and continues automatically after authenticating the ping. Refreshing or resuming that draft must not create another App or bypass signature verification. +While this Cloud setup screen is waiting, its authenticated advance request also +drains the existing sealed event queue. It can bypass an older idle-connection +backoff, at most once every five seconds. Setup and background requests share +one in-flight lease. The normal stack webhook handler still authenticates the +original GitHub bytes before setup can continue. This adds no background timer; +closing the screen stops these readiness requests. Existing background recovery +and its sleep restrictions remain unchanged. New connections default to authorized mentions, advisory reviews, linked-member access, guests off, and enabled bot GitHub tools. Existing instructions, explicit diff --git a/maps/feature-map/connection-setup.md b/maps/feature-map/connection-setup.md index 22306048e8a..d27fc36c657 100644 --- a/maps/feature-map/connection-setup.md +++ b/maps/feature-map/connection-setup.md @@ -205,6 +205,15 @@ configuration changes, saved retry deadlines, rollback, and parking pending repair while an endpoint or connection is inactive. The provider responses are fixtures. Settled maintenance leaves no discovery timer or database scans. +GitHub Cloud setup also covers an older empty-queue backoff, concurrent setup and +background leases, and retry after a Cloud error in the +[event poller regression](../../server/src/services/github-connection-events.test.ts). +Channel integration checks that active setup drains its signed ping through the +normal webhook handler; a forged ping still cannot verify the endpoint. These +fixtures do not establish live Cloud delivery. For live proof, start a disposable +App after an idle personal GitHub connection, complete installation, and confirm +that the same draft advances after authenticating the queued ping. + Manual: complete setup for one named provider, reload endpoint settings, link the intended sender, and send a disposable message. Verify one task and a useful reply in the originating conversation. Inspect an unlinked sender's denial and diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index 4fd186e6271..beb2c0b436a 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -24,7 +24,7 @@ import { toolActionRequests, toolInvocations, toolPolicies } from "@paperclipai/ import { GitHubPublicationLeaseLost, withGitHubPublicationLease } from "../services/chat-github-publication-lease.js"; import { secretService } from "../services/secrets.js"; import { githubChatWizardService } from "../services/chat-github-wizard.js"; -import { dispatchGitHubBotCloudEvent } from "../services/chat-github-cloud-ingress.js"; +import { dispatchGitHubBotCloudEvent, registerGitHubBotCloudSetupPoller } from "../services/chat-github-cloud-ingress.js"; import { githubChatManagementService } from "../services/chat-github-management.js"; import { githubChatReviewService } from "../services/chat-github-reviews.js"; import { resolveGitHubReviewInstructions } from "../services/chat-channels/github/review-instructions.js"; @@ -5135,13 +5135,22 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { (await f.service.get(f.endpoint.id)).setup.webhookVerifiedAt, ).toBeNull(); expect(f.wakeup).not.toHaveBeenCalled(); - event.payload.githubApp.headers["x-hub-signature-256"] = - `sha256=${createHmac("sha256", f.webhookSecret).update(body).digest("hex")}`; - expect(await dispatchGitHubBotCloudEvent(db, event)).toBe(true); - expect( - (await f.service.get(f.endpoint.id)).setup.webhookVerifiedAt, - ).toBeTruthy(); - expect(f.wakeup).not.toHaveBeenCalled(); + const poll = vi.fn(async () => { await dispatchGitHubBotCloudEvent(db, event); }); + const unregister = registerGitHubBotCloudSetupPoller(db, poll); + try { + const waiting = await f.service.githubWizard.advance(f.endpoint.id, "owner-user"); + expect(waiting.state).toBe("verify"); + expect(poll).toHaveBeenCalledOnce(); + expect((await f.service.get(f.endpoint.id)).setup.webhookVerifiedAt).toBeNull(); + event.payload.githubApp.headers["x-hub-signature-256"] = + `sha256=${createHmac("sha256", f.webhookSecret).update(body).digest("hex")}`; + const ready = await f.service.githubWizard.advance(f.endpoint.id, "owner-user"); + expect(ready.state).not.toBe("verify"); + expect((await f.service.get(f.endpoint.id)).setup.webhookVerifiedAt).toBeTruthy(); + expect(f.wakeup).not.toHaveBeenCalled(); + await f.service.githubWizard.advance(f.endpoint.id, "owner-user"); + expect(poll).toHaveBeenCalledTimes(2); + } finally { unregister(); } }); it("legacy name-only registration preserves an explicit empty repository selection", async () => { const fixture = await seedCompany(); diff --git a/server/src/__tests__/github-connection-events.test.ts b/server/src/__tests__/github-connection-events.test.ts index 77d34f15e6a..6bcd2913396 100644 --- a/server/src/__tests__/github-connection-events.test.ts +++ b/server/src/__tests__/github-connection-events.test.ts @@ -7,6 +7,7 @@ import { connectionGrants, createDb, externalObjects, + EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, toolApplications, toolConnections, } from "@paperclipai/db"; @@ -30,7 +31,7 @@ describeEmbeddedPostgres("GitHub connection event delivery", () => { beforeAll(async () => { tempDb = await startEmbeddedPostgresTestDatabase("paperclip-github-events-"); db = createDb(tempDb.connectionString); - }, 20_000); + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); afterEach(async () => { await db.delete(activityLog); @@ -46,7 +47,7 @@ describeEmbeddedPostgres("GitHub connection event delivery", () => { await tempDb?.cleanup(); }); - it("applies a normalized merged PR once while retaining undecryptable siblings for retry", async () => { + it.each([false, true])("applies a normalized merged PR once while retaining undecryptable siblings for retry (active setup=%s)", async activeSetup => { const companyId = randomUUID(); const userId = `github-owner-${randomUUID()}`; const applicationId = randomUUID(); @@ -151,8 +152,21 @@ describeEmbeddedPostgres("GitHub connection event delivery", () => { let currentTime = new Date("2026-09-04T12:00:05.000Z"); const service = githubConnectionEventService(db, { connector, now: () => currentTime }); + if (activeSetup) { + // An existing personal connection can leave a long empty-queue backoff + // before a new dedicated App starts its signed-ping verification. + for (const delay of [0, 10_000, 20_000, 40_000, 80_000]) { + currentTime = new Date(currentTime.getTime() + delay); + vi.mocked(connector.leaseEvents).mockResolvedValueOnce(null); + await service.pollOnce(); + } + currentTime = new Date(currentTime.getTime() + 5_000); + await expect(service.pollOnce()).resolves.toMatchObject({ leased: 0 }); + expect(connector.leaseEvents).toHaveBeenCalledTimes(5); + } + const warning = vi.spyOn(logger, "warn").mockImplementation(() => undefined); - await expect(service.pollOnce()).resolves.toMatchObject({ leased: 2, processed: 1, duplicate: 0, failed: 1 }); + await expect(service.pollOnce({ forSetup: activeSetup })).resolves.toMatchObject({ leased: 2, processed: 1, duplicate: 0, failed: 1 }); expect(warning.mock.calls).toEqual([[{ deliveryId: "undecryptable-delivery", reason: "inner_event_authentication_failed", }, "GitHub connector event retained for retry"]]); diff --git a/server/src/index.ts b/server/src/index.ts index e83d603e2aa..207f342a2ae 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -136,6 +136,7 @@ import { } from "./services/device-login-reaper.js"; import { createProductionSetupTokenReaper } from "./services/setup-token-reaper.js"; import { localAiLoginService } from "./services/local-ai-login.js"; +import { registerGitHubBotCloudSetupPoller } from "./services/chat-github-cloud-ingress.js"; import { resolveWorktreeRunExecutionActivationState } from "./services/instance-settings.js"; import { parseAdapterRegistryEnv, @@ -1419,9 +1420,9 @@ async function startServerWithDatabaseTeardown( ?? process.env.PAPERCLIP_TOOL_RUNTIME_TRUSTED_HOST ?? null, }); - const scheduleGitHubConnectionEventPoll = () => { + const scheduleGitHubConnectionEventPoll = (forSetup = false) => { if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; - trackHeartbeatSchedulerWork(githubConnectionEvents.pollOnce() + const work = githubConnectionEvents.pollOnce({ forSetup }) .then((result) => { if (result.leased > 0 || result.failed > 0) { logger.info(result, "GitHub connection event poll completed"); @@ -1429,8 +1430,13 @@ async function startServerWithDatabaseTeardown( }) .catch((err) => { logger.error({ err }, "GitHub connection event poll failed"); - })); + }); + trackHeartbeatSchedulerWork(work); + return work; }; + server.on("close", registerGitHubBotCloudSetupPoller(db, async () => { + await scheduleGitHubConnectionEventPoll(true); + })); const scheduleGitHubConnectionContinuitySweep = () => { if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; trackHeartbeatSchedulerWork(tools.sweepGitHubConnectionContinuity() diff --git a/server/src/services/chat-github-cloud-ingress.ts b/server/src/services/chat-github-cloud-ingress.ts index b1493d64779..2f8e75eb8f9 100644 --- a/server/src/services/chat-github-cloud-ingress.ts +++ b/server/src/services/chat-github-cloud-ingress.ts @@ -7,6 +7,18 @@ type Event = SealedConnectorEvents["events"][number]; type Endpoint = typeof chatEndpoints.$inferSelect; type Handler = (endpoint: Endpoint, request: Request) => Promise; const handlers = new WeakMap(); +const setupPollers = new WeakMap Promise>(); + +/** The active wizard reuses the server-owned, serialized event drain. */ +export function registerGitHubBotCloudSetupPoller(db: Db, poll: () => Promise) { + setupPollers.set(db, poll); + return () => { if (setupPollers.get(db) === poll) setupPollers.delete(db); }; +} + +export async function pollGitHubBotCloudSetupEvents(db: Db): Promise { + await setupPollers.get(db)?.(); +} + export function registerGitHubBotCloudIngress(db: Db, handler: Handler) { handlers.set(db, handler); return () => { diff --git a/server/src/services/chat-github-wizard.ts b/server/src/services/chat-github-wizard.ts index 5f63f68db8c..701d77cba51 100644 --- a/server/src/services/chat-github-wizard.ts +++ b/server/src/services/chat-github-wizard.ts @@ -33,6 +33,7 @@ import { type PaperclipCloudConnector, } from "./paperclip-cloud-connector.js"; import { paperclipCloudConnectorEnrollmentStatus } from "./paperclip-cloud-connector-enrollment.js"; +import { pollGitHubBotCloudSetupEvents } from "./chat-github-cloud-ingress.js"; /** Loopback aliases are equivalent locally, but Cloud requires the approved exact origin. */ export function githubWizardBrowserOrigin( @@ -920,6 +921,10 @@ export function githubChatWizardService( // Keep the same vaulted App in the wizard instead of throwing JSON from // the browser callback or relaxing the configure action's signature gate. bot = await endpoint(id, userId); + if (!bot.setup.webhookVerifiedAt && bot.setup.github?.cloudRegistrationId) { + await pollGitHubBotCloudSetupEvents(db); + bot = await endpoint(id, userId); + } if (!bot.setup.webhookVerifiedAt) return { endpointId: id, diff --git a/server/src/services/github-connection-events.test.ts b/server/src/services/github-connection-events.test.ts new file mode 100644 index 00000000000..93031ecae62 --- /dev/null +++ b/server/src/services/github-connection-events.test.ts @@ -0,0 +1,74 @@ +import { describe, expect, it, vi } from "vitest"; +import { chatEndpoints, type Db } from "@paperclipai/db"; +import { githubConnectionEventService } from "./github-connection-events.js"; +import type { PaperclipCloudConnector } from "./paperclip-cloud-connector.js"; + +function fixture() { + let time = Date.parse("2026-09-04T12:00:00Z"); + const bindings = [{ + grant: { id: "grant", companyId: "company", kind: "user", subjectUserId: "owner", status: "active", + providerTenant: { github: { installationIds: ["101"] } } }, + connection: { id: "connection", config: { sourceTemplateKey: "github", oauth: { connectorProfile: "github.code" } } }, + }]; + const bots: Array<{ companyId: string }> = []; + // Only SQL is stubbed. Backoff, lease admission and concurrent execution use + // the real service; relational persistence has its own Postgres suite. + const select = vi.fn(() => ({ from: (table: unknown) => table === chatEndpoints + ? { where: async () => bots } + : { innerJoin: () => ({ where: async () => bindings }) } })); + const leaseEvents = vi.fn().mockResolvedValue(null); + const connector = { leaseEvents } as unknown as PaperclipCloudConnector; + const service = githubConnectionEventService({ select } as unknown as Db, { connector, now: () => new Date(time) }); + return { service, leaseEvents, select, bots, advance: (ms: number) => { time += ms; } }; +} + +describe("GitHub event polling during active App setup", () => { + it("reaches Cloud during an old empty-queue backoff without expanding idle polling", async () => { + const f = fixture(); + for (const delay of [0, 10_000, 20_000, 40_000, 80_000]) { + f.advance(delay); + await f.service.pollOnce(); + } + expect(f.leaseEvents).toHaveBeenCalledTimes(5); + const queries = f.select.mock.calls.length; + f.bots.push({ companyId: "company" }); + f.advance(5_000); + for (let index = 0; index < 3; index++) await f.service.pollOnce(); + expect(f.select).toHaveBeenCalledTimes(queries); + expect(f.leaseEvents).toHaveBeenCalledTimes(5); + await f.service.pollOnce({ forSetup: true }); + expect(f.leaseEvents).toHaveBeenCalledTimes(6); + const setupQueries = f.select.mock.calls.length; + await f.service.pollOnce({ forSetup: true }); + expect(f.leaseEvents).toHaveBeenCalledTimes(6); + expect(f.select).toHaveBeenCalledTimes(setupQueries); + f.advance(5_000); + await f.service.pollOnce({ forSetup: true }); + expect(f.leaseEvents).toHaveBeenCalledTimes(7); + }); + + it("shares one lease between setup and scheduled callers while Cloud is slow", async () => { + const f = fixture(); + let complete!: (value: null) => void; + f.leaseEvents.mockImplementationOnce(() => new Promise(resolve => { complete = resolve; })); + const scheduled = f.service.pollOnce(); + const setup = f.service.pollOnce({ forSetup: true }); + await vi.waitFor(() => expect(f.leaseEvents).toHaveBeenCalledTimes(1)); + expect(complete).toBeTypeOf("function"); + complete(null); + await Promise.all([scheduled, setup]); + expect(f.leaseEvents).toHaveBeenCalledTimes(1); + }); + + it("releases the shared attempt after an error and retries only after the setup throttle", async () => { + const f = fixture(); + f.leaseEvents.mockRejectedValueOnce(new Error("Cloud unavailable")); + await expect(f.service.pollOnce({ forSetup: true })).rejects.toThrow("Cloud unavailable"); + await f.service.pollOnce({ forSetup: true }); + await f.service.pollOnce(); + expect(f.leaseEvents).toHaveBeenCalledTimes(1); + f.advance(5_000); + await f.service.pollOnce({ forSetup: true }); + expect(f.leaseEvents).toHaveBeenCalledTimes(2); + }); +}); diff --git a/server/src/services/github-connection-events.ts b/server/src/services/github-connection-events.ts index 04022c647a1..6848875815f 100644 --- a/server/src/services/github-connection-events.ts +++ b/server/src/services/github-connection-events.ts @@ -221,6 +221,8 @@ export function githubConnectionEventService( const now = options.now ?? (() => new Date()); let nextPollAt = 0; let emptyPolls = 0; + let nextSetupPollAt = 0; + let inFlight: Promise | null = null; async function activeBindings() { const rows = await db.select({ grant: connectionGrants, connection: toolConnections }) @@ -458,88 +460,101 @@ export function githubConnectionEventService( return "processed" as const; } - return { - async pollOnce(): Promise { - if (now().getTime() < nextPollAt) { - return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }; - } - const bindings = await activeBindings(); - const bots = await db.select().from(chatEndpoints).where(and(eq(chatEndpoints.provider, "github"), sql`${chatEndpoints.setup}->'github'->>'cloudRegistrationId' is not null`, sql`${chatEndpoints.status} <> 'archived'`)); - if (bindings.length === 0 && bots.length === 0) { - nextPollAt = now().getTime() + 5_000; - return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }; - } - const config = options.connector ? null : paperclipCloudConnectorConfigFromEnv(options.env); - const connector = options.connector ?? (config ? createPaperclipCloudConnector({ config }) : null); - if (!connector) { - nextPollAt = now().getTime() + 5 * 60_000; - return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }; - } - const first = bindings[0] ?? { subject: "github-bot-events", companyId: bots[0]!.companyId }; - const lease = await connector.leaseEvents({ subject: first.subject, companyId: first.companyId }); - if (!lease) { - emptyPolls += 1; - nextPollAt = now().getTime() + Math.min(bots.length ? 15_000 : 5 * 60_000, 5_000 * (2 ** Math.min(emptyPolls, 6))); - return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }; - } - emptyPolls = 0; + async function runPoll(): Promise { + const bindings = await activeBindings(); + const bots = await db.select().from(chatEndpoints).where(and(eq(chatEndpoints.provider, "github"), sql`${chatEndpoints.setup}->'github'->>'cloudRegistrationId' is not null`, sql`${chatEndpoints.status} <> 'archived'`)); + if (bindings.length === 0 && bots.length === 0) { nextPollAt = now().getTime() + 5_000; - const result: GitHubConnectionEventPollResult = { - leased: lease.events.length + (lease.failedEventIds?.length ?? 0), - processed: 0, - duplicate: 0, - ignored: 0, - failed: lease.failedEventIds?.length ?? 0, - }; - const acknowledge: string[] = []; - for (const event of lease.events) { - if (event.payload.githubApp !== undefined) { - try { await dispatchGitHubBotCloudEvent(db, event); acknowledge.push(event.id); result.processed += 1; } - catch { result.failed += 1; } - continue; - } - const matched = bindings.filter((binding) => event.bindingIds.includes(binding.id)); - if (matched.length === 0) { - result.ignored += 1; - acknowledge.push(event.id); - continue; - } - try { - const companies = new Map(); - for (const binding of matched) companies.set(binding.companyId, [...(companies.get(binding.companyId) ?? []), binding]); - for (const [companyId, companyBindings] of companies) { - const status = await processForCompany(companyId, companyBindings, event); - result[status] += 1; - } - acknowledge.push(event.id); - if (event.event === "installation" && (event.action === "deleted" || event.action === "suspend")) { - await Promise.all(matched.map((binding) => connector.setWebhookBinding({ - subject: binding.subject, - companyId: binding.companyId, - id: binding.id, - installationId: binding.installationId, - connectionId: binding.connectionId, - grantId: binding.grantId, - active: false, - }))); - } - } catch { - result.failed += 1; - } + return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }; + } + const config = options.connector ? null : paperclipCloudConnectorConfigFromEnv(options.env); + const connector = options.connector ?? (config ? createPaperclipCloudConnector({ config }) : null); + if (!connector) { + nextPollAt = now().getTime() + 5 * 60_000; + return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }; + } + const first = bindings[0] ?? { subject: "github-bot-events", companyId: bots[0]!.companyId }; + const lease = await connector.leaseEvents({ subject: first.subject, companyId: first.companyId }); + if (!lease) { + emptyPolls += 1; + nextPollAt = now().getTime() + Math.min(bots.length ? 15_000 : 5 * 60_000, 5_000 * (2 ** Math.min(emptyPolls, 6))); + return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }; + } + emptyPolls = 0; + nextPollAt = now().getTime() + 5_000; + const result: GitHubConnectionEventPollResult = { + leased: lease.events.length + (lease.failedEventIds?.length ?? 0), + processed: 0, + duplicate: 0, + ignored: 0, + failed: lease.failedEventIds?.length ?? 0, + }; + const acknowledge: string[] = []; + for (const event of lease.events) { + if (event.payload.githubApp !== undefined) { + try { await dispatchGitHubBotCloudEvent(db, event); acknowledge.push(event.id); result.processed += 1; } + catch { result.failed += 1; } + continue; } - if (acknowledge.length > 0) { - await connector.acknowledgeEvents({ - subject: first.subject, - companyId: first.companyId, - leaseId: lease.leaseId, - deliveryIds: acknowledge, - }); + const matched = bindings.filter((binding) => event.bindingIds.includes(binding.id)); + if (matched.length === 0) { + result.ignored += 1; + acknowledge.push(event.id); + continue; } - for (const deliveryId of lease.failedEventIds ?? []) { - logger.warn({ deliveryId, reason: "inner_event_authentication_failed" }, - "GitHub connector event retained for retry"); + try { + const companies = new Map(); + for (const binding of matched) companies.set(binding.companyId, [...(companies.get(binding.companyId) ?? []), binding]); + for (const [companyId, companyBindings] of companies) { + const status = await processForCompany(companyId, companyBindings, event); + result[status] += 1; + } + acknowledge.push(event.id); + if (event.event === "installation" && (event.action === "deleted" || event.action === "suspend")) { + await Promise.all(matched.map((binding) => connector.setWebhookBinding({ + subject: binding.subject, + companyId: binding.companyId, + id: binding.id, + installationId: binding.installationId, + connectionId: binding.connectionId, + grantId: binding.grantId, + active: false, + }))); + } + } catch { + result.failed += 1; } - return result; + } + if (acknowledge.length > 0) { + await connector.acknowledgeEvents({ + subject: first.subject, + companyId: first.companyId, + leaseId: lease.leaseId, + deliveryIds: acknowledge, + }); + } + for (const deliveryId of lease.failedEventIds ?? []) { + logger.warn({ deliveryId, reason: "inner_event_authentication_failed" }, + "GitHub connector event retained for retry"); + } + return result; + } + + return { + pollOnce(input: { forSetup?: boolean } = {}): Promise { + // A manager actively awaiting a dedicated App's signed ping must not + // inherit a personal connection's old empty-queue backoff. This is a + // bounded readiness probe, not an additional background polling loop. + if (inFlight) return inFlight; + const time = now().getTime(); + if (time < (input.forSetup ? nextSetupPollAt : nextPollAt)) { + return Promise.resolve({ leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }); + } + nextSetupPollAt = time + 5_000; + nextPollAt = Math.max(nextPollAt, nextSetupPollAt); + const work = runPoll().finally(() => { if (inFlight === work) inFlight = null; }); + inFlight = work; + return work; }, }; } From 1ad8f25c7793e4e1d94c1afd29942d05d7778d87 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 23:53:39 -0700 Subject: [PATCH 18/68] perf(heartbeat): narrow queued comment polling run reads (#17560) Read current control facts and typed issue/adapter bindings for queued-comment polling. Keep full mutation and execution reads, native preparation and steering, authorization, and stored evidence. Validation: full build and typecheck, 160 focused tests plus two assembled route regressions; exact-head Greptile 5/5 (medium), full green CI, zero unresolved threads. Old-route counterfactual transfers 2.5 MB versus less than 2 KB. Co-Authored-By: Paperclip --- doc/DEVELOPING.md | 2 + .../issue-queued-comments-routes.test.ts | 32 +++++++++- server/src/routes/issues.ts | 14 +++-- server/src/services/heartbeat.ts | 6 ++ .../src/services/heartbeat/run-state.test.ts | 59 +++++++++++++++++++ server/src/services/heartbeat/run-state.ts | 25 ++++++++ 6 files changed, 130 insertions(+), 8 deletions(-) diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index e399790fbbb..432eb547d45 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1812,6 +1812,8 @@ It owns bounded run projections, database encoding checks, task session reads an writes, explicit resumes, session compaction, and usage/billing helpers. `createHeartbeatRunState(db)` binds these operations without doing database work during construction. The encoding-check cache belongs to each factory instance. +Queued-comment polling reads current issue control fields, the typed issue binding, +and the selected adapter type. Mutation and execution paths retain their full run reads. `heartbeat.ts` keeps run execution and re-exports the existing public helpers. Keep session policy changes separate from run orchestration changes. diff --git a/server/src/__tests__/issue-queued-comments-routes.test.ts b/server/src/__tests__/issue-queued-comments-routes.test.ts index bb42982787b..60985954bea 100644 --- a/server/src/__tests__/issue-queued-comments-routes.test.ts +++ b/server/src/__tests__/issue-queued-comments-routes.test.ts @@ -4,6 +4,7 @@ import { randomUUID } from "node:crypto"; import express from "express"; import request from "supertest"; import { eq, sql } from "drizzle-orm"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import { agentRuntimeState, @@ -79,7 +80,7 @@ describeEmbeddedPostgres("issue queued-comment routes", () => { await tempDb?.cleanup(); }); - function app(companyId: string, userId = "queue-owner", agentActor?: { agentId: string; runId: string }) { + function app(companyId: string, userId = "queue-owner", agentActor?: { agentId: string; runId: string }, database = db) { const testApp = express(); testApp.use(express.json()); testApp.use((req, _res, next) => { @@ -94,7 +95,7 @@ describeEmbeddedPostgres("issue queued-comment routes", () => { }; next(); }); - testApp.use("/api", issueRoutes(db, {} as any, {})); + testApp.use("/api", issueRoutes(database, {} as any, {})); testApp.use(errorHandler); return testApp; } @@ -192,6 +193,33 @@ describeEmbeddedPostgres("issue queued-comment routes", () => { return { companyId, agentId, issueId, runId, wakeId, commentIds }; } + it.each([false, true])("polls queued comments without transferring run snapshots (agent fallback: %s)", async fallback => { + const f = await seedQueue(), payload = "saved-execution-input🙂".repeat(25_000); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, prompt: payload }, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner", adapterConfig: { instructions: payload } } }, + stdoutExcerpt: payload, stderrExcerpt: payload, + }).where(eq(heartbeatRuns.id, f.runId)); + if (fallback) await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, f.issueId)); + const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + const observed = createDb(tempDb!.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns }, capture); + try { + const response = await request(app(f.companyId, "queue-owner", undefined, observed)) + .get(`/api/issues/${f.issueId}/queued-comments`).expect(200); + expect(response.body.queueId).toBe(f.wakeId); + expect(response.body.targetRunId).toBe(f.runId); + expect(response.body.entries.map((entry: { comment: { id: string } }) => entry.comment.id)).toEqual(f.commentIds); + capture.flush(); + const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(2048); + await request(app(randomUUID(), "queue-owner", undefined, observed)) + .get(`/api/issues/${f.issueId}/queued-comments`).expect(404); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(before); + } finally { capture.flush(); await observed.$client.end(); } + }); + it.each(["cancelled", "running"])("rejects a late Done from an interrupted %s task run at the write boundary", async status => { const seeded = await seedQueue(); await db.update(heartbeatRuns).set({ status, resultJson: { executionCancellation: { state: "requested" } } }) diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index 9a29f1c9d91..6aa65dfb774 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -7247,18 +7247,20 @@ export function issueRoutes( id: string; assigneeAgentId: string | null; executionRunId?: string | null; - }) { + }, options?: { queueStateOnly: true }) { let runToInterrupt = issue.executionRunId - ? await heartbeat.getRun(issue.executionRunId) + ? options?.queueStateOnly + ? await heartbeat.getRunIssueControl(issue.executionRunId) + : await heartbeat.getRun(issue.executionRunId) : null; if ( (!runToInterrupt || runToInterrupt.status !== "running") && issue.assigneeAgentId ) { - const activeRun = await heartbeat.getActiveRunForAgent( - issue.assigneeAgentId, - ); + const activeRun = options?.queueStateOnly + ? await heartbeat.getActiveRunIssueControlForAgent(issue.assigneeAgentId) + : await heartbeat.getActiveRunForAgent(issue.assigneeAgentId); const activeIssueId = activeRun && activeRun.contextSnapshot && @@ -16343,7 +16345,7 @@ export function issueRoutes( const queue = await buildQueuedCommentQueue({ executor: db, issue, - activeRun: await resolveActiveIssueRun(issue), + activeRun: await resolveActiveIssueRun(issue, { queueStateOnly: true }), actor: getActorInfo(req), }); res.json( diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 9a18721aaa7..f177921a36d 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -832,6 +832,8 @@ export function heartbeatService( listEvents, getRetryExhaustedReason, getRunIssueSummary, + getRunIssueControl, + getActiveRunIssueControlForAgent, getActiveRunForAgent, getActiveRunIssueSummaryForAgent, } = createHeartbeatRunState(db); @@ -4153,6 +4155,10 @@ export function heartbeatService( getRunIssueSummary, + getRunIssueControl, + + getActiveRunIssueControlForAgent, + getActiveRunForAgent, getActiveRunIssueSummaryForAgent, diff --git a/server/src/services/heartbeat/run-state.test.ts b/server/src/services/heartbeat/run-state.test.ts index ccc91735df4..e42d93d7286 100644 --- a/server/src/services/heartbeat/run-state.test.ts +++ b/server/src/services/heartbeat/run-state.test.ts @@ -20,6 +20,7 @@ import { import * as legacy from "../heartbeat.js"; import * as extracted from "./run-state.js"; import { readRunCancellation } from "../run-cancellation.js"; +import { decideQueuedCommentQueueSteering } from "../issue-queued-comment-queue.js"; // SQL_ASCII needs a separate database cluster. Inspect its selected SQL here; // the PostgreSQL tests below exercise the session queries and transactions. @@ -158,6 +159,64 @@ describePostgres("heartbeat run state database wiring", () => { return { companyId, agent, issue, state: extracted.createHeartbeatRunState(db) }; } + it("reads fresh issue control facts without transferring execution snapshots", async () => { + const { companyId, agent, issue } = await fixture(); + const payload = randomBytes(128 * 1024).toString("base64"); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, status: "running", + runtimeMode: "legacy", runtimeModeResolvedAt: null, + startedAt: new Date("2020-01-01T00:00:00Z"), + contextSnapshot: { issueId: issue.id, prompt: payload }, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner", adapterConfig: { prompt: payload } }, savedInput: payload }, + resultJson: { stdout: payload }, stdoutExcerpt: payload, stderrExcerpt: payload, + }).returning(); + const observed = createDb(database.connectionString); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns }, capture); + try { + const state = extracted.createHeartbeatRunState(observed); + const direct = await state.getRunIssueControl(run.id), active = await state.getActiveRunIssueControlForAgent(agent.id); + expect(active).toEqual(direct); + expect(direct).toEqual({ id: run.id, companyId, agentId: agent.id, status: "running", runtimeMode: "legacy", + runtimeModeResolvedAt: null, startedAt: run.startedAt, createdAt: run.createdAt, + contextSnapshot: { issueId: issue.id }, runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } } }); + capture.flush(); + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBe(2); + expect(summary.responseValueBytes).toBeLessThan(2048); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toEqual(run); + await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, run.id)); + expect((await state.getRunIssueControl(run.id))?.status).toBe("failed"); + expect(await state.getActiveRunIssueControlForAgent(agent.id)).toBeNull(); + expect(await state.getActiveRunIssueControlForAgent(randomUUID())).toBeNull(); + expect(await state.getRunIssueControl(randomUUID())).toBeNull(); + } finally { capture.flush(); await observed.$client.end(); } + }); + + it("preserves native preparation, legacy steering, and typed issue context in issue control reads", async () => { + const { companyId, agent, issue, state } = await fixture(); + for (const mode of ["native", "legacy"] as const) { + for (const value of [null, false, 42, [], {}, { nested: true }, "paperclip_runner", "process"]) { + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, status: "running", runtimeMode: mode, + contextSnapshot: { issueId: value }, runnerProfileJson: { adapterDispatch: { adapterType: value } }, + }).returning(); + const narrow = await state.getRunIssueControl(run.id), full = await state.getRun(run.id); + expect(narrow?.contextSnapshot?.issueId).toEqual(value); + expect(narrow?.runnerProfileJson).toEqual({ adapterDispatch: { adapterType: value } }); + for (const queueState of ["queued", "deferred", null] as const) { + const facts = { state: queueState, queueRunRuntimeMode: mode, assignedAgentAdapterType: "paperclip_runner", queuedCommentCount: 1 }; + expect(decideQueuedCommentQueueSteering({ ...facts, activeRun: narrow })) + .toEqual(decideQueuedCommentQueueSteering({ ...facts, activeRun: full })); + } + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: issue.id }, runtimeModeResolvedAt: new Date("2020-01-01T00:00:00Z") }) + .where(eq(heartbeatRuns.id, run.id)); + const updated = await state.getRunIssueControl(run.id); + expect(updated?.contextSnapshot).toEqual({ issueId: issue.id }); + expect(updated?.runtimeModeResolvedAt).toEqual(new Date("2020-01-01T00:00:00Z")); + } + } + }); + it("reads cleanup and workspace settlement without transferring saved input or changing the run", async () => { const { companyId, agent, issue, state } = await fixture(); const bulk = "cleanup-saved-input-sentinel🙂".repeat(40_000); diff --git a/server/src/services/heartbeat/run-state.ts b/server/src/services/heartbeat/run-state.ts index 1911ad1a25c..3bf92684a1f 100644 --- a/server/src/services/heartbeat/run-state.ts +++ b/server/src/services/heartbeat/run-state.ts @@ -354,6 +354,18 @@ const heartbeatRunLogAccessColumns = { logRef: heartbeatRuns.logRef, } as const; +const heartbeatRunIssueControlColumns = { + id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + status: heartbeatRuns.status, runtimeMode: heartbeatRuns.runtimeMode, + runtimeModeResolvedAt: heartbeatRuns.runtimeModeResolvedAt, + startedAt: heartbeatRuns.startedAt, createdAt: heartbeatRuns.createdAt, + contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then jsonb_build_object('issueId', ${heartbeatRuns.contextSnapshot}->'issueId') else null end`, + runnerProfileJson: sql | null>`case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' + then jsonb_build_object('adapterDispatch', jsonb_build_object('adapterType', ${heartbeatRuns.runnerProfileJson}->'adapterDispatch'->'adapterType')) + else null end`, +} as const; + const heartbeatRunIssueSummaryColumns = { id: heartbeatRuns.id, runtimeMode: heartbeatRuns.runtimeMode, @@ -1955,6 +1967,19 @@ export function createHeartbeatRunState(db: Db) { return run ?? null; }, + getRunIssueControl: async (runId: string) => { + const [run] = await db.select(heartbeatRunIssueControlColumns).from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runId)).limit(1); + return run ?? null; + }, + + getActiveRunIssueControlForAgent: async (agentId: string) => { + const [run] = await db.select(heartbeatRunIssueControlColumns).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.status, "running"))) + .orderBy(desc(heartbeatRuns.startedAt)).limit(1); + return run ?? null; + }, + getActiveRunForAgent: async (agentId: string) => { const [run] = await db .select() From e6ec781a1abcdcddb98e712bcb00b201303e44de Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 24 Oct 2026 23:54:27 -0700 Subject: [PATCH 19/68] fix(e2e): verify review outcomes and preserve report measurements (#17556) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip's Product E2E journeys verify user outcomes in the real app and GitHub. > - The review journey must show the score and failed check for its exact reviewed commit, then the visible task status before and after reload. > - GitHub includes a hidden “Re-run failed checks” menu item before the visible check result. > - The journey selected that hidden item and failed even after native work and publication completed. A subsequent attempt exposed the same first-text-match problem with Done in the task UI. > - This change reads the selected check's status header and binds its URL, score, and commit. > - Browser fixtures retain this failure and reject plausible wrong projections. ## Linked Issues or Issue Description Refs: #17457, #17513, #17551. **What happened?** The bot completed its native run and task, then published its summary, inline precision finding, and failed 2/5 check. The browser assertion selected a hidden “Re-run failed checks” menu item before the visible result, so the journey still failed. After that was corrected, the next live attempt passed the check projection but selected hidden Done text before the visible task status. **Expected behavior** Verify the actual selected GitHub check. A hidden menu item or a word in the model's narrative cannot prove its conclusion. **Steps to reproduce** 1. Render a hidden “Re-run failed checks” item before a visible failed check. 2. Use the old first text match for “failed”. 3. Observe the false visibility failure. **Paperclip version or commit** The base includes the journey changes from #17457 and #17513. ## What Changed - Validate the check URL against the configured repository and select its exact check-run ID. - Require the expected score and full reviewed commit inside that check. - Read failure status only from that check's header. - Give each projection step its own phase so failures identify the correct boundary. - Stage the verified persisted screenshot as the standard report final-state artifact, rejecting a passing report if it is absent. - Preserve measured billing summaries during regeneration when raw run usage is absent; newer raw usage takes precedence. - Extend the catalog-wide history fixture to include the new journey evidence contract. - Read task completion from the visible Status property picker before and after reload, with separate phase labels. - Add eighteen isolated browser fixtures to the existing browser-support suite and document both boundaries. ## Verification - Reproduced the old selector failure in the live GitHub page and a credential-free browser fixture before fixing it. - All 60 browser-support cases pass, including eighteen focused projection cases. They reject the wrong check status, score, commit, page, and repository reference, plus hidden or unrelated Done text. - All 163 focused billing, history, accounting, report, and journey tests pass, as does runner E2E typechecking. - A regression reproduced discarded saved token totals. Retained totals now survive per-cell and campaign aggregation while missing dollar prices stay unknown; fresh raw usage still overrides stale summaries. - Diff and credential-value checks pass. - The earlier failed attempts remain failed. A fresh complete QA journey passed all 18 checkpoints and cleanup with a successful native run, a below-threshold review and inline precision finding, the correctly failed check, visible Done status, and reload persistence. The later reference-validation commit changes only browser fixtures; the live driver bytes are unchanged. - A report regression reproduced the missing standard final-state artifact. The packager now aliases only the hash-verified persisted screenshot for passing results; missing final-state proof is rejected. Presentation regeneration preserves the original live grade and dispatches no provider work. ## Risks This depends on GitHub's check-run header structure. A markup change fails the journey instead of accepting narrative text as a result. The product runtime and behavioral oracle stay unchanged. Usage-price coverage remains a separate evidence dimension. A saved billing summary is reused only when raw usage is absent, preserving the recorded coverage and unknown prices. The separate advisory Paperclip Review remains queued after its prior assigned attempt timed out. Its normal Retry control rejected the obsolete failed chat request as unsafe to retry. Required repository gates `ci / verify` and `ci / e2e` pass, all 55 completed CI checks are green, and final-head Greptile is 5/5 with no unresolved findings. The advisory queue is not represented as a completed assessment. ## Model Used OpenAI Codex, based on GPT-6. The exact runtime model ID and context-window size are not exposed in this session. Used for code, browser fixtures, and live evidence analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip Co-authored-by: Paperclip --- tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md | 13 ++- tests/runner-e2e/billing.test.ts | 19 ++++ tests/runner-e2e/billing.ts | 5 +- tests/runner-e2e/github-review-browser.ts | 29 ++++++ tests/runner-e2e/github-review-check.spec.ts | 94 +++++++++++++++++++ tests/runner-e2e/github-review-flow.ts | 13 ++- tests/runner-e2e/github-review-report.test.ts | 28 ++++++ tests/runner-e2e/github-review-report.ts | 6 ++ tests/runner-e2e/history.test.ts | 7 ++ tests/runner-e2e/playwright-support.config.ts | 2 +- 10 files changed, 208 insertions(+), 8 deletions(-) create mode 100644 tests/runner-e2e/github-review-check.spec.ts diff --git a/tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md b/tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md index 12cddb6c387..bc765f21390 100644 --- a/tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md +++ b/tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md @@ -79,6 +79,14 @@ completed state alone is also insufficient: the actual run must succeed and the associated task must reach Done. A native failure after publication fails the journey and retains the publication evidence already observed. +The rendered check must match its exact GitHub run URL, score, and reviewed +commit. Its failure status comes from that check's header. Hidden re-run menu +items, a failure on another check, and words in the model's narrative do not +prove the conclusion. `github-review-check.spec.ts` covers this boundary in the +credential-free `test:e2e:runner:browser-support` suite. Task completion reads +the visible Status property's picker, before and after reload. Hidden controls, +SVG titles, and narrative text saying Done cannot satisfy that UI checkpoint. + Each attempt records endpoint, App, PR/branch, task, and run ownership. Cleanup removes the owned endpoint, closes the owned PR/deletes its branch, and asks for the owned App's removal. App identity must match the created endpoint, company, @@ -122,7 +130,10 @@ cleanup. Missing usage remains unknown. Authentication screens are excluded. `github-review-report.ts` uses the existing evidence sanitizer and manifest, stages only declared screenshots, and verifies screenshot hashes and attempt path ownership. The standard report inputs are `evidence/result.json`, -`evidence/evidence-manifest.json`, and `evidence/snapshots/`. Raw private operator +`evidence/evidence-manifest.json`, and `evidence/snapshots/`. A passing result +also stages the hash-verified persisted checkpoint as `evidence/final-state.png` +for the standard report's final-state requirement. A missing persisted screenshot +rejects that passing report. Raw private operator receipts are outside the manifest. Existing report tooling imports these files; no second runner or report service is required. A presentation repair must use a new directory, preserve the original result/hash and failure, and dispatch no diff --git a/tests/runner-e2e/billing.test.ts b/tests/runner-e2e/billing.test.ts index 48352815c0e..037ccae91ec 100644 --- a/tests/runner-e2e/billing.test.ts +++ b/tests/runner-e2e/billing.test.ts @@ -31,6 +31,25 @@ function result(overrides: Partial = {}): RunnerE2EResult { } describe("runner E2E billing summaries", () => { + it("preserves captured token totals when a retained result has only a billing summary", () => { + const billing = summarizeExecutionBilling(result({ usage: { + inputTokens: 59_989, outputTokens: 4_974, cachedInputTokens: 488_798, costStatus: "unpriced", + } })); + const retained = result({ billing }); + expect(summarizeExecutionBilling(retained)).toEqual(billing); + expect(aggregateCampaignBilling([retained])).toMatchObject({ llm: { + inputTokens: 59_989, outputTokens: 4_974, cachedInputTokens: 488_798, + runsWithTokenUsage: 1, runsWithReportedCost: 0, + }, testsWithCompleteBilling: 0 }); + }); + + it("uses newly captured usage instead of a stale retained billing summary", () => { + const billing = summarizeExecutionBilling(result({ usage: { inputTokens: 100, outputTokens: 10, costStatus: "unpriced" } })); + const fresh = summarizeExecutionBilling(result({ billing, usage: { inputTokens: 200, outputTokens: 20, costStatus: "unpriced" } })); + expect(fresh.llm).toMatchObject({ inputTokens: 200, outputTokens: 20, runsWithReportedCost: 0 }); + expect(fresh.complete).toBe(false); + }); + it("keeps missing agent time separate from attempt duration and measured zero", () => { const interrupted = result({ status: "failed", durationMs: 108_850 }); expect(summarizeExecutionBilling(interrupted).runtime.agentRunDurationMs).toBeNull(); diff --git a/tests/runner-e2e/billing.ts b/tests/runner-e2e/billing.ts index 35f70e83569..766167c0bfb 100644 --- a/tests/runner-e2e/billing.ts +++ b/tests/runner-e2e/billing.ts @@ -278,8 +278,11 @@ export function summarizeExecutionBilling( result: Pick< RunnerE2EResult, "runIds" | "usage" | "environmentId" | "durationMs" | "runtimeUsage" | "firstTaskQuality" | "completionQuality" | "publicMcp" | "productJourney" - > & Partial>, + > & Partial>, ): RunnerE2EBillingSummary { + // Some retained journeys store their measured summary without raw run usage. + // Regeneration must preserve those measurements, including unknown prices. + if (result.usage === undefined && result.billing) return structuredClone(result.billing); const browserOnly = Boolean(result.productJourney && result.runtimeMode === "none" && !result.runIds?.length); const requestedRunCount = browserOnly ? 0 : Math.max(result.runIds?.length ?? 0, 1); const entries = browserOnly ? [] : usageEntries(result.usage, requestedRunCount); diff --git a/tests/runner-e2e/github-review-browser.ts b/tests/runner-e2e/github-review-browser.ts index b064d0ad384..f4749931dc2 100644 --- a/tests/runner-e2e/github-review-browser.ts +++ b/tests/runner-e2e/github-review-browser.ts @@ -10,6 +10,35 @@ export async function enterGitHubJourneyCompany(page: Page, origin: string, comp await page.waitForURL(url => url.origin === origin && url.pathname === pathname); } +export async function assertGitHubJourneyCheck(page: Page, expected: { repository: string; url: string; headSha: string; score: number }, timeout = 30_000) { + let target: URL; + try { target = new URL(expected.url); } + catch { throw new ConnectionFailure("github_check_reference_mismatch"); } + const prefix = `/${expected.repository}/runs/`; + const id = target.pathname.startsWith(prefix) ? target.pathname.slice(prefix.length) : ""; + if (target.origin !== "https://github.com" || target.username || target.password || target.search || target.hash + || !/^[1-9][0-9]*$/.test(id) || !/^[a-f0-9]{40,64}$/.test(expected.headSha) + || !Number.isInteger(expected.score) || expected.score < 1 || expected.score > 5) { + throw new ConnectionFailure("github_check_reference_mismatch"); + } + await expect(page).toHaveURL(target.href, { timeout }); + const check = page.locator(`#check_run_${id}`); + await expect(check.getByRole("heading", { name: `${expected.score}/5`, exact: true })).toBeVisible({ timeout }); + // GitHub keeps a hidden "Re-run failed checks" menu item before the visible + // result. Only the selected check's status header can prove its conclusion. + await expect(check.locator(":scope > .border-bottom").getByText(/failed/i)).toBeVisible({ timeout }); + await expect(check.getByText(expected.headSha, { exact: true })).toBeVisible({ timeout }); +} + +export async function assertGitHubJourneyTaskDone(page: Page, timeout = 30_000) { + const status = page.locator('[data-property-row="true"]:visible') + .filter({ has: page.locator('[data-property-label="Status"]') }) + .locator('[data-property-value="true"]'); + // Status glyphs contain SVG titles, and responsive layouts can retain hidden + // controls. Read the visible status picker rather than the first Done text. + await expect(status.getByRole("button", { name: "Change status (current: Done)", exact: true })).toBeVisible({ timeout }); +} + export interface GitHubRegistrationBoundary { origin: string; pathname: string; passwordRequired: boolean; oneTimeCodeRequired: boolean; reauthenticationRequired: boolean; createControlVisible: boolean; diff --git a/tests/runner-e2e/github-review-check.spec.ts b/tests/runner-e2e/github-review-check.spec.ts new file mode 100644 index 00000000000..705bc5246a9 --- /dev/null +++ b/tests/runner-e2e/github-review-check.spec.ts @@ -0,0 +1,94 @@ +import { expect, test, type Page } from "@playwright/test"; +import { assertGitHubJourneyCheck, assertGitHubJourneyTaskDone } from "./github-review-browser.js"; + +const expected = { repository: "fixture/repo", url: "https://github.com/fixture/repo/runs/123", headSha: "a".repeat(40), score: 2 }; + +async function openCheck(page: Page, input: { status?: string; score?: number; headSha?: string; unrelatedFailure?: boolean; url?: string } = {}) { + const body = ` +
+
Fixture bot / Paperclip Review ${input.status ?? "failed"}
+

${input.score ?? expected.score}/5

+

Reviewed commit: ${input.headSha ?? expected.headSha}

+

Review rationale: failed tests are discussed in the summary.

+
+ ${input.unrelatedFailure ? '
Another check failed

2/5

' : ""}`; + await page.route("https://github.com/**", route => route.fulfill({ contentType: "text/html", body })); + await page.goto(input.url ?? expected.url); +} + +test("accepts the visible failed check despite an earlier hidden re-run menu item", async ({ page }) => { + await openCheck(page); + await expect(page.getByText(/failed/i).first()).toBeHidden(); + await assertGitHubJourneyCheck(page, expected, 150); +}); + +test("rejects a successful check even when its narrative and another check say failed", async ({ page }) => { + await openCheck(page, { status: "succeeded", unrelatedFailure: true }); + await expect(assertGitHubJourneyCheck(page, expected, 150)).rejects.toThrow(); +}); + +test("rejects the wrong reviewed commit", async ({ page }) => { + await openCheck(page, { headSha: "b".repeat(40) }); + await expect(assertGitHubJourneyCheck(page, expected, 150)).rejects.toThrow(); +}); + +test("rejects a different score", async ({ page }) => { + await openCheck(page, { score: 3 }); + await expect(assertGitHubJourneyCheck(page, expected, 150)).rejects.toThrow(); +}); + +test("rejects a page for another repository", async ({ page }) => { + await openCheck(page, { url: "https://github.com/foreign/repo/runs/123" }); + await expect(assertGitHubJourneyCheck(page, expected, 150)).rejects.toThrow(); +}); + +test("rejects a check reference outside the configured repository", async ({ page }) => { + const url = "https://github.com/foreign/repo/runs/123"; + await openCheck(page, { url }); + await expect(assertGitHubJourneyCheck(page, { ...expected, url }, 150)).rejects.toThrow(); +}); + +for (const [name, override] of [ + ["foreign origin", { url: "https://example.com/fixture/repo/runs/123" }], + ["embedded credentials", { url: "https://fixture-user:fixture-password@github.com/fixture/repo/runs/123" }], + ["query suffix", { url: `${expected.url}?x=1` }], + ["hash suffix", { url: `${expected.url}#fragment` }], + ["non-numeric run ID", { url: "https://github.com/fixture/repo/runs/abc" }], + ["short commit", { headSha: "a".repeat(7) }], + ["score below range", { score: 0 }], + ["score above range", { score: 6 }], +] as const) { + test(`rejects a malformed check reference: ${name}`, async ({ page }) => { + // These must reject before navigation or DOM inspection. + await expect(assertGitHubJourneyCheck(page, { ...expected, ...override }, 150)) + .rejects.toThrow("github_check_reference_mismatch"); + }); +} + +function taskStatus(status: string, hidden = false) { + return `
+ Status
+ +
`; +} + +test("accepts visible Done status despite hidden controls and SVG title text", async ({ page }) => { + await page.setContent(`${taskStatus("Done", true)}${taskStatus("Done")}`); + await expect(page.getByText("Done", { exact: true }).first()).toBeHidden(); + await assertGitHubJourneyTaskDone(page, 150); +}); + +test("rejects Done in the narrative when the task is still in progress", async ({ page }) => { + await page.setContent(`

Done

${taskStatus("In Progress")}`); + await expect(assertGitHubJourneyTaskDone(page, 150)).rejects.toThrow(); +}); + +test("rejects hidden Done status when the visible status is in progress", async ({ page }) => { + await page.setContent(`${taskStatus("Done", true)}${taskStatus("In Progress")}`); + await expect(assertGitHubJourneyTaskDone(page, 150)).rejects.toThrow(); +}); + +test("rejects a Done picker in an unrelated property", async ({ page }) => { + await page.setContent(taskStatus("Done").replace('data-property-label="Status">Status', 'data-property-label="Assignee">Assignee')); + await expect(assertGitHubJourneyTaskDone(page, 150)).rejects.toThrow(); +}); diff --git a/tests/runner-e2e/github-review-flow.ts b/tests/runner-e2e/github-review-flow.ts index 5547adf21a2..d43e919ddd8 100644 --- a/tests/runner-e2e/github-review-flow.ts +++ b/tests/runner-e2e/github-review-flow.ts @@ -3,7 +3,7 @@ import type { GitHubTaskReview } from "../../packages/shared/src/types/chat-gith import type { GitHubRepositoryPage } from "../../packages/shared/src/types/chat-channels.js"; import { ConnectionBlock, ConnectionFailure } from "./connection-config.js"; import { ConnectionApi } from "./connection-target.js"; -import { awaitJourneyCondition, createGitHubJourneyPullRequest, enterGitHubJourneyCompany, observeGitHubHead, selectGitHubJourneyRepository, waitForGitHubRegistrationControl, type GitHubRegistrationBoundary } from "./github-review-browser.js"; +import { assertGitHubJourneyCheck, assertGitHubJourneyTaskDone, awaitJourneyCondition, createGitHubJourneyPullRequest, enterGitHubJourneyCompany, observeGitHubHead, selectGitHubJourneyRepository, waitForGitHubRegistrationControl, type GitHubRegistrationBoundary } from "./github-review-browser.js"; import { githubCartFixture, hasOnlyOpenedAutomaticReview, matchesGitHubJourneyEnvironment, retainGitHubReview, verifyGitHubReview, verifyGitHubReviewRun, type GitHubReviewJourneyEvidence } from "./github-review-evidence.js"; import type { GitHubReviewJourneyConfig } from "./github-review-config.js"; import { captureOwnedGitHubApp, readOwnedGitHubReviewRuns, type GitHubJourneyResources } from "./github-review-ownership.js"; @@ -222,17 +222,20 @@ export async function runGitHubReviewFlow(input: { text: await inline.locator(".review-comment .comment-body").filter({ hasText: /precision|decimal|floating|cents/i }).first().innerText() }; await inline.scrollIntoViewIfNeeded(); await input.checkpoint("github_inline_delivery", github); + await input.phase("github_check_projection"); + if (!review!.checkUrl) throw new ConnectionFailure("github_check_reference_missing"); const checkPopup = page.context().waitForEvent("page"); await page.getByRole("link", { name: "Check", exact: true }).click(); const check = await checkPopup; - await expect(check.getByText(`${evidence.score}/5`, { exact: false }).first()).toBeVisible(); - await expect(check.getByText(/failed/i).first()).toBeVisible(); + await assertGitHubJourneyCheck(check, { repository: config.github.repository, url: review!.checkUrl, headSha, score: evidence.score }); await input.checkpoint("check_projection", check); + await input.phase("task_done"); await page.getByRole("link", { name: "Paperclip task", exact: true }).click(); - await expect(page.getByText("Done", { exact: true }).first()).toBeVisible(); + await assertGitHubJourneyTaskDone(page); await input.checkpoint("task_done", page); + await input.phase("persisted"); await page.reload(); await github.reload(); - await expect(page.getByText("Done", { exact: true }).first()).toBeVisible(); + await assertGitHubJourneyTaskDone(page); await expect(github.getByText(`${evidence.score}/5`, { exact: false }).first()).toBeVisible(); await input.checkpoint("persisted", page); } diff --git a/tests/runner-e2e/github-review-report.test.ts b/tests/runner-e2e/github-review-report.test.ts index 8840fcf171d..9ad23acbcd5 100644 --- a/tests/runner-e2e/github-review-report.test.ts +++ b/tests/runner-e2e/github-review-report.test.ts @@ -5,6 +5,7 @@ import path from "node:path"; import { afterEach, expect, it } from "vitest"; import { runnerMatrix } from "./catalog.js"; import { writeGitHubReviewReport } from "./github-review-report.js"; +import { githubJourneyCheckpoints } from "./github-review-evidence.js"; import { validateRetainedRunnerResult } from "./result-validation.js"; import type { RunnerE2EResult } from "./types.js"; @@ -49,3 +50,30 @@ it("refuses an altered screenshot rather than retaining its claimed digest", asy await expect(writeGitHubReviewReport({ output, campaignId: "altered", result: fixture(), resources: { branch: "owned", appName: "owned", runIds: [] }, catalog: runnerMatrix, secrets: [] })) .rejects.toThrow("journey_screenshot_digest_mismatch"); }); + +it("stages the verified persisted checkpoint as the standard report's final state", async () => { + const output = await mkdtemp(path.join(os.tmpdir(), "github-report-test-")); temporary.push(output); + const input = fixture(); + input.status = "passed"; delete input.error; + input.githubReviewJourney!.outcome = "passed"; + input.githubReviewJourney!.checkpoints = Object.fromEntries(githubJourneyCheckpoints.map(id => [id, true])); + input.screenshots = [{ id: "persisted", label: "Persisted completion", file: "persisted.png", sha256: createHash("sha256").update(pixel).digest("hex") }]; + await writeFile(path.join(output, "persisted.png"), pixel); + const result = await writeGitHubReviewReport({ output, campaignId: "passing-journey", result: input, resources: { branch: "owned", appName: "owned", runIds: ["run"] }, catalog: runnerMatrix, secrets: [] }); + expect(result.status).toBe("passed"); + const manifest = JSON.parse(await readFile(path.join(output, "evidence/evidence-manifest.json"), "utf8")); + expect(manifest.files).toContain("final-state.png"); + expect(await readFile(path.join(output, "evidence/final-state.png"))).toEqual(pixel); + expect(await readFile(path.join(output, "evidence/snapshots/persisted.png"))).toEqual(pixel); +}); + +it("refuses a passing report without its persisted screenshot", async () => { + const output = await mkdtemp(path.join(os.tmpdir(), "github-report-test-")); temporary.push(output); + const input = fixture(); + input.status = "passed"; delete input.error; + input.githubReviewJourney!.outcome = "passed"; + input.githubReviewJourney!.checkpoints = Object.fromEntries(githubJourneyCheckpoints.map(id => [id, true])); + await writeFile(path.join(output, "failure.png"), pixel); + await expect(writeGitHubReviewReport({ output, campaignId: "missing-final-state", result: input, resources: { branch: "owned", appName: "owned", runIds: ["run"] }, catalog: runnerMatrix, secrets: [] })) + .rejects.toThrow("journey_final_state_missing"); +}); diff --git a/tests/runner-e2e/github-review-report.ts b/tests/runner-e2e/github-review-report.ts index 6ad2cc05149..7d0e3332619 100644 --- a/tests/runner-e2e/github-review-report.ts +++ b/tests/runner-e2e/github-review-report.ts @@ -29,6 +29,7 @@ export async function writeGitHubReviewReport(input: { try { await mkdir(path.join(stage, "snapshots"), { mode: 0o700 }); const result = structuredClone(input.result); + let finalStateStaged = false; for (const shot of result.screenshots ?? []) { const source = path.resolve(output, shot.file); if (!source.startsWith(`${output}${path.sep}`) || !/^[a-z0-9_-]+$/.test(shot.id) || path.extname(source) !== ".png") throw new Error("unsafe_journey_screenshot_reference"); @@ -37,8 +38,13 @@ export async function writeGitHubReviewReport(input: { if (shot.sha256 && createHash("sha256").update(bytes).digest("hex") !== shot.sha256) throw new Error("journey_screenshot_digest_mismatch"); shot.file = `snapshots/${shot.id}.png`; await writeFile(path.join(stage, shot.file), bytes, { mode: 0o600 }); + if (result.status === "passed" && shot.id === "persisted" && result.githubReviewJourney?.checkpoints.persisted === true) { + await writeFile(path.join(stage, "final-state.png"), bytes, { mode: 0o600 }); + finalStateStaged = true; + } } validateRetainedRunnerResult(result); + if (result.status === "passed" && !finalStateStaged) throw new Error("journey_final_state_missing"); await save(stage, "result.json", result); await save(stage, "snapshots/journey-resources.json", input.resources); await save(stage, "snapshots/journey-evidence.json", result.githubReviewJourney); diff --git a/tests/runner-e2e/history.test.ts b/tests/runner-e2e/history.test.ts index c717441e321..b6216af3745 100644 --- a/tests/runner-e2e/history.test.ts +++ b/tests/runner-e2e/history.test.ts @@ -32,6 +32,7 @@ import { renderRunnerHistoryIndex } from "./history-index.js"; import { renderPublicCampaignSummary } from "./public-summary-image.js"; import { isPublicRunnerScreenshotRoute } from "./screenshot-policy.js"; import { connectionCheckpoints, type ConnectionEvidence } from "./connection-evidence.js"; +import { githubJourneyCheckpoints, type GitHubReviewJourneyEvidence } from "./github-review-evidence.js"; import type { MatrixExecution, RunnerE2EResult } from "./types.js"; const temporaryDirectories: string[] = []; @@ -65,6 +66,12 @@ function result(execution: MatrixExecution, status: "passed" | "failed") { durationMs: 1_000, cleanup: "passed", ...(execution.task.flow === "provider_connection" ? { providerConnection: { version: 1, outcome: status, phase: "finished", assisted: false, authFreshness: "signed-in", target: { mode: "attach", origin: "http://127.0.0.1:3100", commit: "a".repeat(40), deploymentMode: "local_trusted" }, entry: "agent", method: "api-key", checkpoints: Object.fromEntries(connectionCheckpoints.map(key => [key, true])), waits: [] } satisfies ConnectionEvidence } : {}), + ...(execution.task.flow === "github_review_journey" ? { githubReviewJourney: { + version: 1, outcome: status, phase: "finished", setupId: "fixture", hosting: "self-hosted", + targetCommit: "a".repeat(40), assisted: false, dependencies: [], + checkpoints: Object.fromEntries(githubJourneyCheckpoints.map(key => [key, true])), + cleanup: { endpoint: true, pullRequest: true, branch: true, app: true }, + } satisfies GitHubReviewJourneyEvidence } : {}), } satisfies RunnerE2EResult; } diff --git a/tests/runner-e2e/playwright-support.config.ts b/tests/runner-e2e/playwright-support.config.ts index bdaefb27292..0256f9ef884 100644 --- a/tests/runner-e2e/playwright-support.config.ts +++ b/tests/runner-e2e/playwright-support.config.ts @@ -3,7 +3,7 @@ import { defineConfig } from "@playwright/test"; /** Browser-only harness regressions: no Paperclip instance or provider credentials. */ export default defineConfig({ testDir: ".", - testMatch: ["screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts", "blocker-input.spec.ts", "native-blocker-visible.spec.ts", "copilot-permission-card.spec.ts"], + testMatch: ["screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts", "blocker-input.spec.ts", "native-blocker-visible.spec.ts", "copilot-permission-card.spec.ts", "github-review-check.spec.ts"], workers: 1, retries: 0, timeout: 10_000, From 08dafde1d65a8f3299237af04df5e2543bb4382d Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 23:58:43 -0700 Subject: [PATCH 20/68] perf(accounting): narrow locked usage receipt run reads (#17559) Select only the five run fields used by usage receipt persistence under the existing company and row locks. Keep receipt history, replay checks, budget updates, and stored execution evidence intact. Validation: 37 focused tests, full build and typecheck, exact-head Greptile 5/5 (medium), and full green CI with no unresolved threads. The first CI attempt failed an unchanged Runner shutdown test; its qualified local case and same-head failed-lane rerun passed. Co-Authored-By: Paperclip --- .../src/services/usage-receipts-read.test.ts | 81 +++++++++++++++++++ server/src/services/usage-receipts.ts | 5 +- 2 files changed, 85 insertions(+), 1 deletion(-) create mode 100644 server/src/services/usage-receipts-read.test.ts diff --git a/server/src/services/usage-receipts-read.test.ts b/server/src/services/usage-receipts-read.test.ts new file mode 100644 index 00000000000..83ddbed9364 --- /dev/null +++ b/server/src/services/usage-receipts-read.test.ts @@ -0,0 +1,81 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, companies, createDb, heartbeatRuns, runUsageReceipts, type Db } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { persistUsageReceipt, type UsageReceiptEnvelope } from "./usage-receipts.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("usage receipt PostgreSQL reads", () => { + let database: Awaited>; + let db: Db; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("usage-receipt-read-"); + db = createDb(database.connectionString); + }, 60_000); + afterAll(async () => { await db?.$client.end(); await database?.cleanup(); }); + + async function fixture() { + const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), sourceId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Receipt fixture", issuePrefix: companyId.slice(0, 8) }); + await db.insert(agents).values({ id: agentId, companyId, name: "Receipt agent", adapterType: "process", status: "active" }); + const payload = randomBytes(128 * 1024).toString("base64"); + const evidence = { contextSnapshot: { prompt: payload }, runnerProfileJson: { instructions: payload }, + resultJson: { stdout: payload }, stdoutExcerpt: payload, stderrExcerpt: payload }; + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", ...evidence, + usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0, preserved: "receipt metadata" } }); + const receipt = (sequence: number): UsageReceiptEnvelope => ({ schema: "paperclip/accounting-receipt/v1", + id: randomUUID(), companyId, runId, sourceId, sequence, receivedAt: new Date().toISOString(), adapterType: "process", + receipt: { complete: false, usageBasis: "per_run", costUsd: sequence / 100, billingType: "metered_api" } }); + return { companyId, runId, evidence, receipt }; + } + + it("updates live usage with a small locked read and preserves all execution evidence", async () => { + const f = await fixture(), reads: unknown[][] = []; + const observed = new Proxy(db, { get(target, key) { + if (key !== "transaction") return Reflect.get(target, key); + return (callback: Parameters[0]) => target.transaction(tx => callback(new Proxy(tx, { + get(transaction, property) { + if (property !== "select") return Reflect.get(transaction, property); + return (...fields: Parameters) => { + const builder = transaction.select(...fields), from = builder.from.bind(builder); + builder.from = ((...args: Parameters) => { + const query = from(...args); + if (args[0] === heartbeatRuns) { + const execute = query.execute.bind(query); + query.execute = async (...executeArgs: Parameters) => { + expect(query.toSQL().sql).toContain("for update"); + const rows = await execute(...executeArgs); reads.push(rows); return rows; + }; + } + return query; + }) as typeof builder.from; + return builder; + }; + }, + }))); + } }); + await persistUsageReceipt(observed, f.receipt(1)); + expect(reads).toHaveLength(1); + expect(reads[0]).toHaveLength(1); + expect(Buffer.byteLength(JSON.stringify(reads))).toBeLessThan(2048); + const [stored] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + expect(stored).toMatchObject(f.evidence); + expect(stored.usageJson).toMatchObject({ preserved: "receipt metadata", accountingReceiptSequence: 1, + budgetExposure: { costCents: "1.0000000" } }); + expect(Buffer.byteLength(JSON.stringify(stored))).toBeGreaterThan(512 * 1024); + }); + + it("serializes competing checkpoints and retains replay conflicts and company isolation", async () => { + const f = await fixture(), first = f.receipt(1), second = f.receipt(2); + await Promise.all([persistUsageReceipt(db, first), persistUsageReceipt(db, second)]); + await persistUsageReceipt(db, first); + const [stored] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + expect(stored.usageJson).toMatchObject({ accountingReceiptSequence: 2, budgetExposure: { costCents: "2.0000000" } }); + expect(await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, f.runId))).toHaveLength(2); + await expect(persistUsageReceipt(db, { ...first, receipt: { ...first.receipt, costUsd: 10 } })).rejects.toThrow("Conflicting"); + const foreign = await fixture(); + await expect(persistUsageReceipt(db, { ...f.receipt(3), companyId: foreign.companyId })).rejects.toThrow("not found"); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(stored); + }); +}); diff --git a/server/src/services/usage-receipts.ts b/server/src/services/usage-receipts.ts index 52bae89f8fb..0f99401d9da 100644 --- a/server/src/services/usage-receipts.ts +++ b/server/src/services/usage-receipts.ts @@ -68,7 +68,10 @@ export async function persistUsageReceipt(db: Db, raw: UsageReceiptEnvelope) { const envelope = envelopeSchema.parse(raw); const hash = receiptFingerprint(envelope); return withAccountingTransaction(db, envelope.companyId, async (tx, publications) => { - const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, envelope.runId), eq(heartbeatRuns.companyId, envelope.companyId))).for("update"); + const [run] = await tx.select({ + id: heartbeatRuns.id, agentId: heartbeatRuns.agentId, status: heartbeatRuns.status, + usageJson: heartbeatRuns.usageJson, costAccountedAt: heartbeatRuns.costAccountedAt, + }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, envelope.runId), eq(heartbeatRuns.companyId, envelope.companyId))).for("update"); if (!run) throw notFound("Accounting run not found"); const [existing] = await tx.select().from(runUsageReceipts).where(and(eq(runUsageReceipts.companyId, envelope.companyId), eq(runUsageReceipts.runId, envelope.runId), eq(runUsageReceipts.sourceId, envelope.sourceId), eq(runUsageReceipts.sequence, envelope.sequence))); if (existing && existing.receiptHash !== hash) throw conflict("Conflicting usage checkpoint replay"); From bf142db65aa90eb4c23bf9084e95b33515234637 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 00:02:32 -0700 Subject: [PATCH 21/68] perf(server): narrow external review wait context reads (#17562) Project current chat authorization fields for external review-wait discovery instead of saved execution input. Preserve typed source bindings, nested question receipts, candidate eligibility, live revocation checks, and stored context. Validation: 181 native chat integration tests, full build and typecheck, exact-head Greptile 5/5 (medium), full green CI, and no unresolved review threads. The full-context counterfactual fails both direct-chat size regressions. Co-Authored-By: Paperclip --- doc/native-workspace-finalization-recovery.md | 3 ++ .../external-chat-wait.integration.test.ts | 46 +++++++++++++++++++ .../native-runtime/external-review-wait.ts | 21 ++++++++- 3 files changed, 68 insertions(+), 2 deletions(-) diff --git a/doc/native-workspace-finalization-recovery.md b/doc/native-workspace-finalization-recovery.md index c8353239234..5b1ffb5ce98 100644 --- a/doc/native-workspace-finalization-recovery.md +++ b/doc/native-workspace-finalization-recovery.md @@ -492,6 +492,9 @@ binding, and chat source fields under the existing nonblocking share lock. It excludes saved execution input, runner profiles, and log excerpts. Actual retry admission still reads the complete run and its recovery evidence. These reads preserve stored run data, source revocation checks, and JSON value types. +External review-wait discovery selects only chat authorization context fields. +It keeps the current endpoint, membership, resource, and question-receipt checks; +saved prompts remain stored and do not cross this discovery query. Legacy continuation recovery rejects native runs before it loads their saved execution input. The persisted disposition check reads only the task binding diff --git a/server/src/services/native-runtime/external-chat-wait.integration.test.ts b/server/src/services/native-runtime/external-chat-wait.integration.test.ts index b0ee8e0c5f6..70a6933e80a 100644 --- a/server/src/services/native-runtime/external-chat-wait.integration.test.ts +++ b/server/src/services/native-runtime/external-chat-wait.integration.test.ts @@ -4,6 +4,7 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { and, eq, ne, sql } from "drizzle-orm"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { agentWakeupRequests, @@ -71,6 +72,7 @@ import { voiceCredentialFingerprint } from "../voice/voice-session-store.js"; import { nativeSha256 } from "./canonical.js"; import { questionResponseDeliveryValues } from "../question-response-delivery.js"; import { resolveExternalChatQuestionResponse } from "./external-chat-question-response.js"; +import { listExternalReviewWaits } from "./external-review-wait.js"; import { materializeExternalChatQuestionResponseInput } from "./external-chat-question-response-input.js"; import * as nativeInteractionBridge from "./native-interaction-bridge.js"; import { defaultGitHubReviewPolicy, type AskUserQuestionsInteraction } from "@paperclipai/shared"; @@ -866,6 +868,50 @@ describe("native external-chat response wait", () => { }; } + it.each(["telegram", "github", "github_board_answer", "slack_board_answer"] as const)( + "reads small current external review waits without saved prompts (%s)", async source => { + const fixture = await (async () => { + if (source === "github_board_answer" || source === "slack_board_answer") { + const answered = await seedBoardAnsweredTurn(true, source === "github_board_answer" ? "github" : "slack"); + await attestAnswer(answered); + return answered; + } + return seedWaitTurn(source); + })(); + const finalized = await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect(finalized.phase).toBe("committed"); + await db.update(statusDecisions).set({ toStatus: "in_review" }).where(eq(statusDecisions.id, finalized.decisionId!)); + await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, fixture.issueId)); + const [decision] = await db.select({ reason: statusDecisions.reasonCode, state: statusDecisions.applicationState }) + .from(statusDecisions).where(eq(statusDecisions.id, finalized.decisionId!)); + const governed = source === "github_board_answer" || source === "slack_board_answer"; + expect(decision).toEqual({ reason: governed ? "governed_response_waiting" : "external_chat_response_waiting", state: "applied" }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect(run!.status).toBe("succeeded"); + const contextSnapshot = { ...run!.contextSnapshot, savedPrompt: "external-wait-saved-input🙂".repeat(30_000) }; + await db.update(heartbeatRuns).set({ contextSnapshot }).where(eq(heartbeatRuns.id, fixture.runId)); + await authorizeChatConversationForBoundRun(db, fixture, contextSnapshot, "read"); + const observed = createDb(temporary?.connectionString ?? externalTestDatabaseUrl!); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { issues, heartbeatRuns }, capture); + try { + expect(await listExternalReviewWaits(observed, fixture.companyId, [fixture.issueId])).toEqual(governed ? [] : [ + expect.objectContaining({ companyId: fixture.companyId, issueId: fixture.issueId }), + ]); + capture.flush(); + const reads = events.filter(event => event.table === "issues" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(4096); + expect(await listExternalReviewWaits(observed, randomUUID(), [fixture.issueId])).toEqual([]); + expect(await listExternalReviewWaits(observed, fixture.companyId, [])).toEqual([]); + await db.update(companyMemberships).set({ status: "suspended" }).where(eq(companyMemberships.companyId, fixture.companyId)); + expect(await listExternalReviewWaits(observed, fixture.companyId, [fixture.issueId])).toEqual([]); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)))[0]?.contextSnapshot).toEqual(contextSnapshot); + } finally { capture.flush(); await observed.$client.end(); } + }, + ); + it("resumes an oversized Slack multi-select answered through its Paperclip handoff", async () => { const fixture = await seedBoardAnsweredTurn(true, "slack", true); const [publication] = await db.select().from(chatPublications).where(eq(chatPublications.id, fixture.publicationId)); diff --git a/server/src/services/native-runtime/external-review-wait.ts b/server/src/services/native-runtime/external-review-wait.ts index fe5347a9cea..4a63ad6510d 100644 --- a/server/src/services/native-runtime/external-review-wait.ts +++ b/server/src/services/native-runtime/external-review-wait.ts @@ -1,6 +1,23 @@ -import { and, eq, inArray } from "drizzle-orm"; +import { and, eq, inArray, sql } from "drizzle-orm"; import { chatConversations, chatEndpoints, heartbeatRuns, issues, statusDecisions, type Db } from "@paperclipai/db"; import { authorizeChatConversationForBoundRun } from "./chat-attachment-reuse.js"; +import { EXTERNAL_CHAT_QUESTION_RESPONSE_KEY } from "./external-chat-question-response.js"; + +const authorizationContextKeys = [ + "source", "issueId", "wakeReason", "interactionKind", "interactionStatus", "externalChatContinuation", + "interactionId", "sourceRunId", "sourceCommentId", "wakeCommentId", "wakeCommentIds", + "paperclipHarnessCheckedOut", "paperclipExternalChatExecutionBound", + EXTERNAL_CHAT_QUESTION_RESPONSE_KEY, +] as const; + +const authorizationContext = sql | null>` + case when ${heartbeatRuns.contextSnapshot} is null then null else ( + select coalesce(jsonb_object_agg(field.key, field.value), '{}'::jsonb) + from jsonb_each(case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then ${heartbeatRuns.contextSnapshot} else '{}'::jsonb end) as field(key, value) + where field.key in (${sql.join(authorizationContextKeys.map(key => sql`${key}`), sql`, `)}) + ) end +`; export async function listExternalReviewWaits(db: Db, companyId: string, issueIds: string[]) { if (!issueIds.length) return []; @@ -12,7 +29,7 @@ export async function listExternalReviewWaits(db: Db, companyId: string, issueId createdAt: statusDecisions.createdAt, runId: heartbeatRuns.id, agentId: heartbeatRuns.agentId, - contextSnapshot: heartbeatRuns.contextSnapshot, + contextSnapshot: authorizationContext, }).from(issues) .innerJoin(statusDecisions, and( eq(statusDecisions.id, issues.lastStatusDecisionId), From 75dbd3050bc2a7f7463ff77e5dffe308871a1f25 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 00:09:56 -0700 Subject: [PATCH 22/68] perf(git): filter managed identity connection reads (#17563) Apply the existing exact GitHub source marker predicate in the company-scoped connection query. Keep the JavaScript guard and every current install, grant, consent, identity, and credential check unchanged. Validation: 74 focused tests, full build and typecheck, exact-head Greptile 5/5 (medium), full green CI, and no unresolved threads. Both marker locations, malformed JSON types, changed/disabled connections, and retained unrelated configuration are covered. The unfiltered counterfactual fails both row-count regressions. Co-Authored-By: Paperclip --- .../github-operation-credentials.test.ts | 37 +++++++++++++++++++ server/src/services/git-credentials.ts | 6 ++- 2 files changed, 42 insertions(+), 1 deletion(-) diff --git a/server/src/__tests__/github-operation-credentials.test.ts b/server/src/__tests__/github-operation-credentials.test.ts index 02d1c864bcd..d9d1ef0c9d6 100644 --- a/server/src/__tests__/github-operation-credentials.test.ts +++ b/server/src/__tests__/github-operation-credentials.test.ts @@ -6,6 +6,7 @@ import { createRuntimeToolsToken } from "../runtime-tools-token.js"; import { errorHandler } from "../middleware/index.js"; import { randomUUID } from "node:crypto"; import { eq } from "drizzle-orm"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { agents, @@ -192,6 +193,42 @@ const support = await getEmbeddedPostgresTestSupport(); }); return { id, connectionId, secretId, definitionId }; } + it.each(["config", "transportConfig"] as const)("reads only current GitHub identity connections via %s", async location => { + const input = await seed(), selected = await grant(input, "A"); + const [connection] = await db.select().from(toolConnections).where(eq(toolConnections.id, selected.connectionId)); + await db.update(toolConnections).set({ config: {}, transportConfig: {}, [location]: { sourceTemplateKey: "github" } }) + .where(eq(toolConnections.id, selected.connectionId)); + const payload = "unrelated-connection-config🙂".repeat(10_000); + const irrelevant = [null, true, 42, ["github"], { name: "github" }, "GitHub", " github ", "other"].map(sourceTemplateKey => { + const id = randomUUID(); + return { id, companyId: input.companyId, applicationId: connection.applicationId, name: id, uid: id, + transport: "mcp_remote" as const, config: { sourceTemplateKey, payload }, transportConfig: { sourceTemplateKey } }; + }); + await db.insert(toolConnections).values(irrelevant); + const observed = createDb(database.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { toolConnections }, capture); + try { + expect(await resolveManagedGitHubIdentitySelection(observed, input.companyId, { agentId: input.agentId, responsibleUserId: "A" })) + .toMatchObject({ configured: true, grant: { id: selected.id } }); + capture.flush(); + const reads = events.filter(event => event.table === "tool_connections" && event.operation === "select"); + expect(reads).toHaveLength(1); + expect(reads[0].rows).toBe(1); + expect(reads[0].responseValueBytes).toBeLessThan(8192); + await db.update(toolConnections).set({ enabled: false }).where(eq(toolConnections.id, selected.connectionId)); + expect((await resolveManagedGitHubIdentitySelection(observed, input.companyId, { agentId: input.agentId, responsibleUserId: "A" })).grant) + .toBeUndefined(); + await db.update(toolConnections).set({ [location]: { sourceTemplateKey: "other" } }).where(eq(toolConnections.id, selected.connectionId)); + expect(await resolveManagedGitHubIdentitySelection(observed, input.companyId, { agentId: input.agentId, responsibleUserId: "A" })) + .toEqual({ configured: false }); + expect(await resolveManagedGitHubIdentitySelection(observed, randomUUID(), { agentId: input.agentId, responsibleUserId: "A" })) + .toEqual({ configured: false }); + const saved = await db.select().from(toolConnections).where(eq(toolConnections.companyId, input.companyId)); + for (const row of irrelevant) expect(saved.find(candidate => candidate.id === row.id)?.config).toEqual(row.config); + } finally { capture.flush(); await observed.$client.end(); } + }); + async function switchTo( input: Awaited>, user: string, diff --git a/server/src/services/git-credentials.ts b/server/src/services/git-credentials.ts index 3fce7fc2b50..69bac2c029e 100644 --- a/server/src/services/git-credentials.ts +++ b/server/src/services/git-credentials.ts @@ -10,7 +10,7 @@ import { userSecretDefinitions, type Db, } from "@paperclipai/db"; -import { and, eq, inArray, or } from "drizzle-orm"; +import { and, eq, inArray, or, sql } from "drizzle-orm"; import { isGitHubDotCom } from "./github-fetch.js"; import { secretService } from "./secrets.js"; import { toolAccessService } from "./tool-access.js"; @@ -324,6 +324,10 @@ export async function resolveManagedGitHubIdentitySelection( }> { const connections = await db.select().from(toolConnections).where(and( eq(toolConnections.companyId, companyId), + or( + sql`${toolConnections.config}->'sourceTemplateKey' = '"github"'::jsonb`, + sql`${toolConnections.transportConfig}->'sourceTemplateKey' = '"github"'::jsonb`, + ), )); const githubConnections = connections.filter((connection) => { const config = connection.config && typeof connection.config === "object" ? connection.config as Record : {}; From efa2724f9a464d27bfaa8c82204c13b886287baf Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 00:11:41 -0700 Subject: [PATCH 23/68] fix(db): pace bounded response capture output (#17564) Close database response measurement at its existing deadline, then emit the bounded metadata at 64 records per second. Preserve group indices, counts, privacy, and query/transaction outcomes. Stop the unreferenced timer when the output queue is empty. Validation: 32 capture and real PostgreSQL tests, full build and typecheck, exact-head Greptile 5/5 (medium), full green CI, and no unresolved comments. A 2,200-query fixture reconciles all 2,048 groups and its summary; dedicated-pool coverage awaits deferred output. Co-Authored-By: Paperclip --- doc/DATABASE.md | 5 +- .../db/src/response-capture-postgres.test.ts | 2 + packages/db/src/response-capture.test.ts | 52 ++++++++++++++++++- packages/db/src/response-capture.ts | 25 ++++++++- 4 files changed, 81 insertions(+), 3 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index aacf4f9f23d..a51b7cb6100 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -342,7 +342,10 @@ column decoding. It retains the Drizzle builder caller for normal, prepared, transaction, and dedicated-pool execution. It issues no queries. It logs the public server caller locations, static schema table, operation, query/row counts, and response value sizes. It emits at most 2,048 aggregate -groups and a summary at the deadline. It never logs SQL, parameters, result +groups and a summary. Measurement stops at the deadline. Environment-enabled +captures emit at most 64 records per second and can take another 33 seconds to +finish writing the bounded output. Wait for the summary before collecting it. +It never logs SQL, parameters, result values, connection strings, or query errors. In-flight queries continue normally. For a bundled server, keep the linked source maps and enable Node's diff --git a/packages/db/src/response-capture-postgres.test.ts b/packages/db/src/response-capture-postgres.test.ts index 370d738d056..96de91ff913 100644 --- a/packages/db/src/response-capture-postgres.test.ts +++ b/packages/db/src/response-capture-postgres.test.ts @@ -182,6 +182,8 @@ describe.skipIf(!support.supported)("database response capture PostgreSQL behavi expect(await withDedicatedDbConnection(db, async dedicated => dedicated.execute(sql`select ${"dedicated-value"}::text as value`))) .toEqual([{ value: "dedicated-value" }]); capture.flush(); + await vi.waitFor(() => expect(logs.some(line => JSON.parse(line).event === "paperclip.db_response_capture.summary.v1")).toBe(true), + { timeout: 1_000, interval: 5 }); const events = logs.map(line => JSON.parse(line) as Record); expect(summary(events)).toMatchObject({ startedQueries: 1, completedQueries: 1 }); expect(logs.join("\n")).not.toContain("dedicated-value"); diff --git a/packages/db/src/response-capture.test.ts b/packages/db/src/response-capture.test.ts index 9fc1c80f5b1..73d25d8fc3c 100644 --- a/packages/db/src/response-capture.test.ts +++ b/packages/db/src/response-capture.test.ts @@ -1,6 +1,6 @@ import { integer, pgTable } from "drizzle-orm/pg-core"; import { afterEach, describe, expect, it, vi } from "vitest"; -import { createDatabaseResponseCapture, installDatabaseResponseCapture, publicDatabaseCallers, responseCaptureWindow } from "./response-capture.js"; +import { createDatabaseResponseCapture, databaseResponseCaptureFromEnv, installDatabaseResponseCapture, publicDatabaseCallers, responseCaptureWindow } from "./response-capture.js"; const epoch = Date.parse("2020-01-01T00:00:00Z"); const metadata = { operation: "select", table: "capture_fixture", callers: ["services/fixture.ts:4:2"] }; @@ -16,6 +16,56 @@ function fixture(emitOverride?: (event: Record) => void) { } describe("bounded database response capture", () => { + it("paces a complete environment capture without extending the measurement window or leaving timers", () => { + vi.useFakeTimers(); + vi.setSystemTime(epoch); + const emitted: Array<{ at: number; event: Record }> = []; + const log = vi.spyOn(console, "info").mockImplementation(value => emitted.push({ at: Date.now(), event: JSON.parse(value) })); + try { + const capture = databaseResponseCaptureFromEnv({ PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW: "2020-01-01T00:00:00Z/2020-01-01T00:01:00Z" })!; + captures.push(capture); + const rows = [{ secret: "capture-response-sentinel" }]; + for (let index = 0; index < 2200; index++) { + expect(capture.execute(() => rows, { ...metadata, callers: [`services/capture.ts:${index + 1}:1`] })).toBe(rows); + } + vi.advanceTimersByTime(60_001); + expect(emitted.length).toBeGreaterThan(0); + expect(emitted.length).toBeLessThanOrEqual(64); + expect(capture.active()).toBe(false); + expect(capture.execute(() => rows, metadata)).toBe(rows); + capture.flush(); + vi.advanceTimersByTime(32_000); + expect(emitted).toHaveLength(2049); + const groups = emitted.filter(item => item.event.event === "paperclip.db_response_capture.entry.v1"); + const summary = emitted.at(-1)!.event; + expect(groups).toHaveLength(2048); + expect(summary).toMatchObject({ event: "paperclip.db_response_capture.summary.v1", completedQueries: 2200, + groups: 2048, closedAt: "2020-01-01T00:01:00.000Z" }); + expect(groups.reduce((calls, item) => calls + Number(item.event.calls), 0)).toBe(summary.completedQueries); + expect(groups.reduce((bytes, item) => bytes + Number(item.event.responseValueBytes), 0)).toBe(summary.responseValueBytes); + expect(groups.map(item => item.event.groupIndex)).toEqual(Array.from({ length: 2048 }, (_, index) => index)); + for (const { at } of emitted) expect(emitted.filter(item => item.at >= at && item.at < at + 1000).length).toBeLessThanOrEqual(64); + expect(JSON.stringify(emitted)).not.toContain("capture-response-sentinel"); + expect(vi.getTimerCount()).toBe(0); + vi.advanceTimersByTime(60_000); + expect(emitted).toHaveLength(2049); + } finally { log.mockRestore(); vi.clearAllTimers(); vi.useRealTimers(); } + }); + + it("contains deferred logger failures and finishes its bounded output", () => { + vi.useFakeTimers(); + vi.setSystemTime(epoch + 120_000); + const log = vi.spyOn(console, "info").mockImplementation(() => { throw new Error("fixture logger failure"); }); + try { + const capture = databaseResponseCaptureFromEnv({ PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW: "2020-01-01T00:02:00Z/2020-01-01T00:03:00Z" })!; + captures.push(capture); + expect(capture.execute(() => 7, metadata)).toBe(7); + expect(() => vi.advanceTimersByTime(60_001)).not.toThrow(); + expect(log).toHaveBeenCalledTimes(2); + expect(vi.getTimerCount()).toBe(0); + } finally { log.mockRestore(); vi.clearAllTimers(); vi.useRealTimers(); } + }); + it.each([undefined, "", "sentinel-invalid-input", "2020-01-01T00:00:00Z", "2020-01-01T00:00:00Z/2020-01-01T00:00:00Z", "2020-01-01T00:00:00Z/2020-01-01T00:01:01Z", "2020-01-01T01:00:00Z/2020-01-01T01:01:00Z", "2020-01-01T00:00:00Z/2020-01-01T00:01:00Z/extra", diff --git a/packages/db/src/response-capture.ts b/packages/db/src/response-capture.ts index 675d57eeb82..d53b5fe0c07 100644 --- a/packages/db/src/response-capture.ts +++ b/packages/db/src/response-capture.ts @@ -143,12 +143,35 @@ export function createDatabaseResponseCapture( type Capture = ReturnType; let environmentCapture: { key: string; capture: Capture } | undefined; + +function pacedCaptureLogger() { + const pending: Event[] = []; + let timer: ReturnType | undefined; + const drain = () => { + timer = undefined; + for (const event of pending.splice(0, 64)) { + try { console.info(JSON.stringify(event)); } catch {} + } + if (pending.length) { + timer = setTimeout(drain, 1_000); + timer.unref(); + } + }; + return (event: Event) => { + pending.push(event); + if (!timer) { + timer = setTimeout(drain, 0); + timer.unref(); + } + }; +} + export function databaseResponseCaptureFromEnv(env: NodeJS.ProcessEnv = process.env): Capture | undefined { const key = env.PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW; const window = responseCaptureWindow(key); if (!key || !window) return; if (environmentCapture?.key === key) return environmentCapture.capture; - const capture = createDatabaseResponseCapture(window, event => console.info(JSON.stringify(event))); + const capture = createDatabaseResponseCapture(window, pacedCaptureLogger()); environmentCapture = { key, capture }; return capture; } From f978a521ff34f1a7aac94892981db88973743097 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 00:21:15 -0700 Subject: [PATCH 24/68] perf(heartbeat): narrow workspace operation access reads (#17565) Read only the current run access fields and typed workspace binding for workspace-operation polling. Keep the current privacy decisions, operation checks, and complete run storage. Verified focused run-state, live-event, and real database privacy-route tests, full build and typecheck, all CI checks, and exact-head Greptile 5/5 with no unresolved threads. Co-Authored-By: Paperclip --- doc/DEVELOPING.md | 2 ++ .../__tests__/agent-live-run-routes.test.ts | 20 +++++++++++++ .../heartbeat-run-privacy-routes.test.ts | 24 ++++++++++++++++ server/src/routes/agents.ts | 5 ++-- server/src/services/heartbeat.ts | 2 ++ .../src/services/heartbeat/run-state.test.ts | 28 +++++++++++++++++++ server/src/services/heartbeat/run-state.ts | 8 ++++++ 7 files changed, 86 insertions(+), 3 deletions(-) diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 432eb547d45..20b6ce6563d 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1814,6 +1814,8 @@ writes, explicit resumes, session compaction, and usage/billing helpers. during construction. The encoding-check cache belongs to each factory instance. Queued-comment polling reads current issue control fields, the typed issue binding, and the selected adapter type. Mutation and execution paths retain their full run reads. +Workspace-operation polling reads the current run access scope and typed workspace +binding, then applies the existing run and operation permission checks. `heartbeat.ts` keeps run execution and re-exports the existing public helpers. Keep session policy changes separate from run orchestration changes. diff --git a/server/src/__tests__/agent-live-run-routes.test.ts b/server/src/__tests__/agent-live-run-routes.test.ts index 94f6d525268..484b88a63ca 100644 --- a/server/src/__tests__/agent-live-run-routes.test.ts +++ b/server/src/__tests__/agent-live-run-routes.test.ts @@ -16,6 +16,7 @@ const mockHeartbeatService = vi.hoisted(() => ({ getActiveRunIssueSummaryForAgent: vi.fn(), getRunReadAccess: vi.fn(), getRunLogAccess: vi.fn(), + getRunWorkspaceAccess: vi.fn(), readLog: vi.fn(), wakeup: vi.fn(), getRun: vi.fn(), @@ -378,6 +379,10 @@ describe("agent live run routes", () => { logStore: "local_file", logRef: "logs/aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa.ndjson", }); + mockHeartbeatService.getRunWorkspaceAccess.mockResolvedValue({ + id: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", companyId: "company-1", scopeKind: "company", issueId: null, + executionWorkspaceId: null, + }); mockHeartbeatService.readLog.mockResolvedValue({ runId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", store: "local_file", @@ -687,6 +692,21 @@ describe("agent live run routes", () => { }, ); + it("lists workspace operations using narrow current run access", async () => { + mockHeartbeatService.getRunWorkspaceAccess.mockResolvedValue({ + id: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", companyId: "company-1", scopeKind: "company", issueId: null, + executionWorkspaceId: " workspace-one ", + }); + mockWorkspaceOperationService.listForRun.mockResolvedValue([]); + const response = await requestApp(await createApp(), baseUrl => request(baseUrl) + .get("/api/heartbeat-runs/aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa/workspace-operations")); + expect(response.status).toBe(200); + expect(response.body).toEqual([]); + expect(mockHeartbeatService.getRunWorkspaceAccess).toHaveBeenCalledWith("aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"); + expect(mockHeartbeatService.getRun).not.toHaveBeenCalled(); + expect(mockWorkspaceOperationService.listForRun).toHaveBeenCalledWith("aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", "workspace-one"); + }); + it("caps company live run polling by default", async () => { const rows = Array.from({ length: 75 }, (_, index) => ({ id: `run-${index}`, diff --git a/server/src/__tests__/heartbeat-run-privacy-routes.test.ts b/server/src/__tests__/heartbeat-run-privacy-routes.test.ts index 03914cadeda..40d966ed8e8 100644 --- a/server/src/__tests__/heartbeat-run-privacy-routes.test.ts +++ b/server/src/__tests__/heartbeat-run-privacy-routes.test.ts @@ -1,6 +1,7 @@ import { randomUUID } from "node:crypto"; import { eq } from "drizzle-orm"; import { drizzle } from "drizzle-orm/postgres-js"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import express from "express"; import request from "supertest"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; @@ -257,6 +258,29 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false }, return app; } + it("lists workspace operations with small run reads and checks a revoked issue grant again", async () => { + const f = await seedFixture(), payload = "workspace-operation-saved-input🙂".repeat(25_000); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, prompt: payload }, + runnerProfileJson: { instructions: payload }, stdoutExcerpt: payload, stderrExcerpt: payload, + }).where(eq(heartbeatRuns.id, f.privateRunId)); + const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.privateRunId)); + const observed = createDb(tempDb!.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns }, capture); + try { + const app = createApp(f.companyId, f.grantedAgentId, observed); + const response = await request(app).get(`/api/heartbeat-runs/${f.privateRunId}/workspace-operations`).expect(200); + expect(response.body).toEqual([expect.objectContaining({ id: f.operationId })]); + capture.flush(); + const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(2048); + await db.delete(issueAccessGrants).where(eq(issueAccessGrants.subjectId, f.grantedAgentId)); + await request(app).get(`/api/heartbeat-runs/${f.privateRunId}/workspace-operations`).expect(404); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.privateRunId)))[0]).toEqual(before); + } finally { capture.flush(); await observed.$client.end(); } + }); + it("returns 404 for direct private-run content and metadata-only list stubs to a non-member", async () => { const fixture = await seedFixture(); const app = createApp(fixture.companyId, fixture.otherAgentId); diff --git a/server/src/routes/agents.ts b/server/src/routes/agents.ts index 0aba86abb8f..a6b71c93b9d 100644 --- a/server/src/routes/agents.ts +++ b/server/src/routes/agents.ts @@ -7589,13 +7589,12 @@ export function agentRoutes( router.get("/heartbeat-runs/:runId/workspace-operations", async (req, res) => { const runId = readHeartbeatRunId(req); - const run = await getAccessibleResource(req, res, heartbeat.getRun(runId), "Heartbeat run not found"); + const run = await getAccessibleResource(req, res, heartbeat.getRunWorkspaceAccess(runId), "Heartbeat run not found"); if (!run) return; if (!(await assertRunTelemetryReadAllowed(req, res, run.companyId))) return; if (!(await assertRunReadAllowed(req, res, run))) return; - const context = asRecord(run.contextSnapshot); - const executionWorkspaceId = asNonEmptyString(context?.executionWorkspaceId); + const executionWorkspaceId = asNonEmptyString(run.executionWorkspaceId); const operations = await workspaceOperations.listForRun(runId, executionWorkspaceId); const visibleOperations = []; for (const operation of operations) { diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index f177921a36d..adb10d852c9 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -825,6 +825,7 @@ export function heartbeatService( hasResolvablePriorSessionWorkspaceForWake, getRunReadAccess, getRunLogAccess, + getRunWorkspaceAccess, listRuns, getRuntimeStateWithSessions, listTaskSessions, @@ -3993,6 +3994,7 @@ export function heartbeatService( getRunReadAccess, getRunLogAccess, + getRunWorkspaceAccess, getRuntimeState: getRuntimeStateWithSessions, diff --git a/server/src/services/heartbeat/run-state.test.ts b/server/src/services/heartbeat/run-state.test.ts index e42d93d7286..702f842e464 100644 --- a/server/src/services/heartbeat/run-state.test.ts +++ b/server/src/services/heartbeat/run-state.test.ts @@ -159,6 +159,34 @@ describePostgres("heartbeat run state database wiring", () => { return { companyId, agent, issue, state: extracted.createHeartbeatRunState(db) }; } + it("reads current workspace access without execution snapshots and retains JSON types", async () => { + const { companyId, agent, issue } = await fixture(); + const payload = randomBytes(128 * 1024).toString("base64"), workspaceId = randomUUID(); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, scopeKind: "issue", issueId: issue.id, + contextSnapshot: { executionWorkspaceId: workspaceId, prompt: payload }, runnerProfileJson: { instructions: payload }, + resultJson: { stdout: payload }, stdoutExcerpt: payload, stderrExcerpt: payload, + }).returning(); + const observed = createDb(database.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns }, capture); + try { + const state = extracted.createHeartbeatRunState(observed); + expect(await state.getRunWorkspaceAccess(run.id)).toEqual({ + id: run.id, companyId, scopeKind: "issue", issueId: issue.id, executionWorkspaceId: workspaceId, + }); + capture.flush(); + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBe(1); + expect(summary.responseValueBytes).toBeLessThan(1024); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toEqual(run); + for (const value of [null, true, 42, " workspace-two ", [workspaceId], { id: workspaceId }]) { + await db.update(heartbeatRuns).set({ contextSnapshot: { executionWorkspaceId: value, prompt: payload } }).where(eq(heartbeatRuns.id, run.id)); + expect((await state.getRunWorkspaceAccess(run.id))?.executionWorkspaceId).toEqual(value); + } + expect(await state.getRunWorkspaceAccess(randomUUID())).toBeNull(); + } finally { capture.flush(); await observed.$client.end(); } + }); + it("reads fresh issue control facts without transferring execution snapshots", async () => { const { companyId, agent, issue } = await fixture(); const payload = randomBytes(128 * 1024).toString("base64"); diff --git a/server/src/services/heartbeat/run-state.ts b/server/src/services/heartbeat/run-state.ts index 3bf92684a1f..f870c3957fb 100644 --- a/server/src/services/heartbeat/run-state.ts +++ b/server/src/services/heartbeat/run-state.ts @@ -1182,6 +1182,13 @@ export function createHeartbeatRunState(db: Db) { .then((rows) => rows[0] ?? null); } + async function getRunWorkspaceAccess(runId: string) { + return db.select({ + ...heartbeatRunReadAccessColumns, + executionWorkspaceId: sql`${heartbeatRuns.contextSnapshot}->'executionWorkspaceId'`, + }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).then(rows => rows[0] ?? null); + } + async function getRuntimeState(agentId: string) { return db .select() @@ -1736,6 +1743,7 @@ export function createHeartbeatRunState(db: Db) { getRunStatus, getRunReadAccess, getRunLogAccess, + getRunWorkspaceAccess, listRuns: async ( companyId: string, From 0d73505098ff9980c5a0947b103a999ab55e5b88 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 00:31:41 -0700 Subject: [PATCH 25/68] perf(git): narrow operation identity snapshot reads (#17567) Select only the run context used by GitHub operation credential checks. Preserve the default complete identity capture for Muse and other consumers, current trust checks, grants, and saved run evidence. Validation: 72 focused GitHub, identity, Slack, chat-project, and project-route tests; full build and typecheck; exact-head Greptile 5/5 (medium), all CI checks green, and zero unresolved comments. Real PostgreSQL tests prove current revocation and response-size bounds. No live external GitHub push was performed. Co-Authored-By: Paperclip --- doc/execution-github-identity.md | 6 ++++ .../github-operation-credentials.test.ts | 31 +++++++++++++++++++ server/src/__tests__/run-identity.test.ts | 17 ++++++++++ .../services/github-operation-credentials.ts | 2 +- server/src/services/run-identity.ts | 14 +++++++-- 5 files changed, 67 insertions(+), 3 deletions(-) diff --git a/doc/execution-github-identity.md b/doc/execution-github-identity.md index 662f20ef11d..3d179337f17 100644 --- a/doc/execution-github-identity.md +++ b/doc/execution-github-identity.md @@ -14,6 +14,12 @@ Delegated work and interactions persist their originating context. Retries retai Executions with managed GitHub configured receive token-free `git` and `gh` launchers. Each launcher invocation resolves one eligible credential from the current run's accepted identity at operation start. A `gh` command's child Git processes inherit that command's captured identity; later steering does not change already-started operations. +GitHub credential capture reads the current task, project, and retained trust +policy bindings without transferring saved prompts, runner profiles, results, +or log excerpts. It keeps the existing task/run lock order and accepted-identity +reconciliation. Other identity callers retain their complete execution input, +including the native input required by project creation authorization. + Native runners retain a session-owned broker and launcher path across warm turns. The provider keeps an opaque transport token, not a GitHub credential or the previous run's signed capability. After acquiring exclusive session ownership, the controller binds the broker to the current company, agent, task, and run. Requests cannot choose another run or responsible person. The broker rejects requests while idle and discards credential responses if their run binding changed during acquisition. Each operation still rechecks the live run, accepted identity, grants, and trust policy through the shared credential resolver. Changing run IDs alone no longer replaces the provider process; changes to authentication mode, provider credentials, permissions, or other session configuration retain their existing retirement rules. The session broker listens only on controller loopback and accepts only its authenticated GitHub credential operation. Remote executions reach it through the existing authenticated callback bridge. Its transport and launchers are retired with the provider session. If remote bridge startup fails, anonymous launchers keep ordinary work available; the next run retries setup with a fresh session. Controller restart/cold recovery still uses the existing checkpoint and process-recovery rules; the broker's in-memory authority is not persisted for adoption. diff --git a/server/src/__tests__/github-operation-credentials.test.ts b/server/src/__tests__/github-operation-credentials.test.ts index d9d1ef0c9d6..699249b5b58 100644 --- a/server/src/__tests__/github-operation-credentials.test.ts +++ b/server/src/__tests__/github-operation-credentials.test.ts @@ -32,6 +32,7 @@ import { } from "./helpers/embedded-postgres.js"; import { initializeRunIdentity, + captureRunIdentity, reserveSteeredIdentity, acceptSteeredIdentity, } from "../services/run-identity.js"; @@ -229,6 +230,36 @@ const support = await getEmbeddedPostgresTestSupport(); } finally { capture.flush(); await observed.$client.end(); } }); + it("resolves current GitHub credentials without transferring execution snapshots", async () => { + const input = await seed(), selected = await grant(input, "A"); + const payload = "github-credential-saved-input🙂".repeat(25_000); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: input.issueId, prompt: payload }, + runnerProfileJson: { nativeExecutionInput: { savedInput: payload }, savedInput: payload }, + resultJson: { stdout: payload }, stdoutExcerpt: payload, stderrExcerpt: payload, + }).where(eq(heartbeatRuns.id, input.runId)); + const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)); + const observed = createDb(database.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns }, capture); + try { + expect(await resolveGitHubOperationCredentials(observed, input)).toMatchObject({ + status: "available", login: "A", grantId: selected.id, + }); + capture.flush(); + const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(8192); + expect(JSON.stringify(events)).not.toContain("github-credential-saved-input"); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)))[0]).toEqual(before); + const full = await captureRunIdentity(observed, input); + expect(full.run.runnerProfileJson).toEqual(before.runnerProfileJson); + expect(full.run.resultJson).toEqual(before.resultJson); + expect(full.run.contextSnapshot).toEqual(before.contextSnapshot); + await db.update(connectionGrants).set({ status: "revoked" }).where(eq(connectionGrants.id, selected.id)); + expect(await resolveGitHubOperationCredentials(observed, input)).toMatchObject({ status: "unavailable" }); + } finally { capture.flush(); await observed.$client.end(); } + }); + async function switchTo( input: Awaited>, user: string, diff --git a/server/src/__tests__/run-identity.test.ts b/server/src/__tests__/run-identity.test.ts index d6dbbbfc3ff..b371c59f043 100644 --- a/server/src/__tests__/run-identity.test.ts +++ b/server/src/__tests__/run-identity.test.ts @@ -29,6 +29,23 @@ const support = await getEmbeddedPostgresTestSupport(); } return { companyId, agentId, issueId, runId, messageIds }; } + it("retains typed GitHub trust bindings and keeps other identity reads complete", async () => { + const input = await seed(); + await initializeRunIdentity(db, { ...input, responsibleUserId: "owner", cause: "queued" }); + for (const value of [null, true, 42, " padded ", [], {}, { future: { opaque: true } }]) { + const contextSnapshot = { issueId: input.issueId, projectId: value, executionPolicy: value, savedInput: { retained: true } }; + const runnerProfileJson = { nativeExecutionInput: { opaque: value }, savedInput: "retained" }; + await db.update(heartbeatRuns).set({ contextSnapshot, runnerProfileJson }).where(eq(heartbeatRuns.id, input.runId)); + const narrow = await captureRunIdentity(db, input, { readScope: "github_credentials" }); + expect(narrow.run.contextSnapshot).toEqual({ issueId: input.issueId, taskId: null, projectId: value, executionPolicy: value }); + expect(narrow.run.runnerProfileJson).toBeNull(); + const full = await captureRunIdentity(db, input); + expect(full.run.contextSnapshot).toEqual(contextSnapshot); + expect(full.run.runnerProfileJson).toEqual(runnerProfileJson); + expect(narrow.context).toEqual(full.context); + } + }); + it("retains mixed-author delivery order without rewriting task ownership", async () => { const input = await seed(); await initializeRunIdentity(db, { ...input, responsibleUserId: "owner", cause: "queued" }); diff --git a/server/src/services/github-operation-credentials.ts b/server/src/services/github-operation-credentials.ts index b82e4be4211..c24aeabb5c1 100644 --- a/server/src/services/github-operation-credentials.ts +++ b/server/src/services/github-operation-credentials.ts @@ -112,7 +112,7 @@ export async function resolveGitHubOperationCredentials( runId: string; }, ) { - const { run, context } = await captureRunIdentity(db, input); + const { run, context } = await captureRunIdentity(db, input, { readScope: "github_credentials" }); if (!context) throw forbidden("This run predates managed GitHub credentials"); let summary: GitHubCredentialSummary; let env: Record = {}; diff --git a/server/src/services/run-identity.ts b/server/src/services/run-identity.ts index cc75397a991..b1e27d1ae72 100644 --- a/server/src/services/run-identity.ts +++ b/server/src/services/run-identity.ts @@ -1,4 +1,4 @@ -import { and, asc, desc, eq, inArray, sql } from "drizzle-orm"; +import { and, asc, desc, eq, getTableColumns, inArray, sql } from "drizzle-orm"; import { agentWakeupRequests, heartbeatRuns, @@ -434,12 +434,22 @@ export async function acceptSteeredIdentity( export async function captureRunIdentity( db: Db, input: { companyId: string; runId: string; agentId: string }, + options?: { readScope: "github_credentials" }, ) { // Lock acquisition serializes with steering delivery and its durable acknowledgement. return db.transaction(async (tx) => { await lockIdentityTask(tx, input.companyId, input.runId); const [run] = await tx - .select() + .select(options?.readScope === "github_credentials" ? { + ...getTableColumns(heartbeatRuns), + contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then jsonb_build_object('issueId', ${heartbeatRuns.contextSnapshot}->'issueId', + 'taskId', ${heartbeatRuns.contextSnapshot}->'taskId', 'projectId', ${heartbeatRuns.contextSnapshot}->'projectId', + 'executionPolicy', ${heartbeatRuns.contextSnapshot}->'executionPolicy') else null end`, + runnerProfileJson: sql | null>`null`, + resultJson: sql | null>`null`, + stdoutExcerpt: sql`null`, stderrExcerpt: sql`null`, + } : getTableColumns(heartbeatRuns)) .from(heartbeatRuns) .where( and( From 7494b8dd4c113a9f290e9dc5fbda4f61776f5d67 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 00:35:07 -0700 Subject: [PATCH 26/68] perf(auth): batch current run history read decisions (#17569) Reuse the existing authorization batch owner for current run-history read decisions after company-scoped existence checks. Preserve missing-task denial, per-request freshness, and the decide-only adapter path. Validation: 97 focused tests, full build and repository typecheck. A real PostgreSQL fixture returns identical decisions with 4 queries instead of 327; the original path fails the query-bound regression. Exact-head Greptile 5/5 (medium), all CI checks green, and no unresolved comments. Co-Authored-By: Paperclip --- doc/DEVELOPING.md | 9 ++ .../heartbeat-run-privacy-routes.test.ts | 105 +++++++++++++++++- server/src/services/access.ts | 1 + server/src/services/heartbeat-run-privacy.ts | 43 ++++++- 4 files changed, 153 insertions(+), 5 deletions(-) diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 20b6ce6563d..3cc2067d727 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -907,6 +907,15 @@ can update an entry only after the current account's summary confirms access. The first healthy live subscription reconciles domain data without re-fetching session and health; reconnects and visibility recovery retain full reconciliation. +### Run history authorization reads + +Run history reuses the authorization service's batch read API for distinct task +bindings. Existence checks remain company-scoped and use batches of at most 500 +IDs. Missing or foreign tasks stay unreadable in every privacy mode. The batch +snapshot lasts for one request only; the next request checks current grants, +membership, and run restrictions again. Adapters that expose only `decide` +retain the individual-read path with duplicate bindings shared within the request. + ### Local Linux runner qualification Before serving requests, the server reads `getconf GNU_LIBC_VERSION` once with diff --git a/server/src/__tests__/heartbeat-run-privacy-routes.test.ts b/server/src/__tests__/heartbeat-run-privacy-routes.test.ts index 40d966ed8e8..ed10d2548f2 100644 --- a/server/src/__tests__/heartbeat-run-privacy-routes.test.ts +++ b/server/src/__tests__/heartbeat-run-privacy-routes.test.ts @@ -483,6 +483,105 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false }, process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = "enforce"; } }); + it("batches distinct issue bindings without changing current authorization decisions", async () => { + const fixture = await seedFixture(); + const ids = Array.from({ length: 64 }, () => randomUUID()); + await db.insert(issues).values(ids.map((id, index) => ({ + id, companyId: fixture.companyId, title: `History ${index}`, status: "in_progress", + visibility: index % 2 ? "private" : "open", privacyRootIssueId: index % 2 ? id : null, + assigneeAgentId: fixture.ownerAgentId, responsibleUserId: fixture.ownerUserId, + }))); + const queries: string[] = []; + const observedDb = drizzle(db.$client, { schema, logger: { logQuery(query) { queries.push(query); } } }) as typeof db; + const access = authorizationService(observedDb); + const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.grantedAgentId }; + const bindings = [...ids, fixture.issueId].map(issueId => ({ companyId: fixture.companyId, scopeKind: "issue" as const, issueId })); + const runs = [...bindings, ...bindings, { ...bindings[0], companyId: randomUUID() }, + { ...bindings[0], issueId: randomUUID() }, { companyId: fixture.companyId, scopeKind: "company" as const }, + { companyId: fixture.companyId, scopeKind: "issue" as const }]; + const individual = await getHeartbeatRunReadDecisions(observedDb, { decide: access.decide }, actor, runs); + const individualCount = queries.length; + queries.length = 0; + const batched = await getHeartbeatRunReadDecisions(observedDb, access, actor, runs); + expect(batched).toEqual(individual); + expect(batched.slice(0, 64)).toEqual(ids.map((_, index) => index % 2 === 0)); + expect(batched.at(64)).toBe(true); + expect(batched.slice(-4)).toEqual([false, false, true, false]); + expect(queries.length).toBeLessThan(25); + expect(queries.length).toBeLessThan(individualCount / 4); + await db.update(issueAccessGrants).set({ revokedAt: new Date() }).where(eq(issueAccessGrants.subjectId, fixture.grantedAgentId)); + expect((await getHeartbeatRunReadDecisions(observedDb, access, actor, runs)).at(64)).toBe(false); + }); + + it.each(["enforce", "shadow", "off"])("keeps missing and foreign issue bindings closed in %s mode", async mode => { + const fixture = await seedFixture(); + process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = mode; + try { + const access = authorizationService(db); + const actor = { type: "board" as const, userId: fixture.ownerUserId }; + const binding = { companyId: fixture.companyId, scopeKind: "issue" as const, issueId: fixture.issueId }; + const runs = [binding, { ...binding, issueId: randomUUID() }, { ...binding, companyId: randomUUID() }]; + const expected = await getHeartbeatRunReadDecisions(db, { decide: access.decide }, actor, runs); + expect(await getHeartbeatRunReadDecisions(db, access, actor, runs)).toEqual(expected); + expect(expected).toEqual([true, false, false]); + await db.delete(companyMemberships).where(eq(companyMemberships.principalId, fixture.ownerUserId)); + expect(await getHeartbeatRunReadDecisions(db, access, actor, [binding])).toEqual( + await getHeartbeatRunReadDecisions(db, { decide: access.decide }, actor, [binding]), + ); + } finally { process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = "enforce"; } + }); + + it("checks more than 500 task bindings with bounded company-scoped queries", async () => { + const fixture = await seedFixture(); + const ids = Array.from({ length: 501 }, () => randomUUID()); + await db.insert(issues).values(ids.map((id, index) => ({ + id, companyId: fixture.companyId, title: `Batch ${index}`, visibility: "open", status: "in_progress", + }))); + const queries: { query: string; params: unknown[] }[] = []; + const observedDb = drizzle(db.$client, { schema, logger: { logQuery(query, params) { queries.push({ query, params }); } } }) as typeof db; + const access = authorizationService(observedDb); + const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.ownerAgentId }; + const runs = ids.map(issueId => ({ companyId: fixture.companyId, scopeKind: "issue" as const, issueId })); + expect(await getHeartbeatRunReadDecisions(observedDb, access, actor, runs)).toEqual(ids.map(() => true)); + const existence = queries.filter(({ query }) => query.startsWith('select "id" from "issues"')); + expect(existence.map(({ params }) => params.length).sort((a, b) => a - b)).toEqual([2, 501]); + expect(existence.every(({ query, params }) => query.includes('"company_id"') && params[0] === fixture.companyId)).toBe(true); + }); + + it("captures batch inputs before awaiting reads and rejects a failed batch", async () => { + const fixture = await seedFixture(); + const access = authorizationService(db); + const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.ownerAgentId }; + const binding = { companyId: fixture.companyId, scopeKind: "issue" as const, issueId: fixture.issueId }; + const decideReads = vi.spyOn(access, "decideReads"); + const pending = getHeartbeatRunReadDecisions(db, access, actor, [binding, binding]); + binding.issueId = randomUUID(); + expect(await pending).toEqual([true, true]); + expect(decideReads).toHaveBeenCalledExactlyOnceWith(actor, [expect.objectContaining({ issueId: fixture.issueId })]); + const error = new Error("authorization unavailable"); + decideReads.mockRejectedValueOnce(error); + const current = { ...binding, issueId: fixture.issueId }; + await expect(getHeartbeatRunReadDecisions(db, access, actor, [current])).rejects.toBe(error); + const transient = Object.assign(new Error("temporary disconnect"), { code: "CONNECTION_CLOSED" }); + decideReads.mockRejectedValueOnce(transient); + expect(await getHeartbeatRunReadDecisions(db, access, actor, [current])).toEqual([true]); + }); + + it("does no database or authorization work for company-only or empty batches", async () => { + const fixture = await seedFixture(); + const queries: string[] = []; + const observedDb = drizzle(db.$client, { schema, logger: { logQuery(query) { queries.push(query); } } }) as typeof db; + const access = authorizationService(observedDb); + const decideReads = vi.spyOn(access, "decideReads"); + const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.ownerAgentId }; + expect(await getHeartbeatRunReadDecisions(observedDb, access, actor, [])).toEqual([]); + expect(await getHeartbeatRunReadDecisions(observedDb, access, actor, [ + { companyId: fixture.companyId, scopeKind: "company" }, { companyId: fixture.companyId, scopeKind: "issue" }, + ])).toEqual([true, false]); + expect(queries).toEqual([]); + expect(decideReads).not.toHaveBeenCalled(); + }); + it.each(["owner", "outsider"])("checks concurrent duplicate issue runs once for the %s", async (viewer) => { const fixture = await seedFixture(); const access = authorizationService(db); @@ -493,7 +592,7 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false }, agentId: viewer === "owner" ? fixture.ownerAgentId : fixture.otherAgentId, }; const binding = { companyId: fixture.companyId, scopeKind: "issue" as const, issueId: fixture.issueId }; - const decisions = await getHeartbeatRunReadDecisions(db, access, actor, Array.from({ length: 200 }, () => ({ ...binding }))); + const decisions = await getHeartbeatRunReadDecisions(db, { decide: access.decide }, actor, Array.from({ length: 200 }, () => ({ ...binding }))); expect(decisions).toEqual(Array(200).fill(viewer === "owner")); expect(decide).toHaveBeenCalledTimes(1); decide.mockRestore(); @@ -517,7 +616,7 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false }, const decide = vi.spyOn(access, "decide"); const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.ownerAgentId }; const binding = { companyId: fixture.companyId, scopeKind: "issue" as const, issueId: fixture.issueId }; - const pending = getHeartbeatRunReadDecisions(db, access, actor, [binding, binding]); + const pending = getHeartbeatRunReadDecisions(db, { decide: access.decide }, actor, [binding, binding]); binding.issueId = randomUUID(); expect(await pending).toEqual([true, true]); expect(decide).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ @@ -591,7 +690,7 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false }, const decide = vi.spyOn(access, "decide").mockRejectedValue(error); const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.ownerAgentId }; const binding = { companyId: fixture.companyId, scopeKind: "issue" as const, issueId: fixture.issueId }; - await expect(getHeartbeatRunReadDecisions(db, access, actor, Array(20).fill(binding))).rejects.toBe(error); + await expect(getHeartbeatRunReadDecisions(db, { decide: access.decide }, actor, Array(20).fill(binding))).rejects.toBe(error); expect(decide).toHaveBeenCalledTimes(1); decide.mockRestore(); expect(await getHeartbeatRunReadDecisions(db, access, actor, [binding])).toEqual([true]); diff --git a/server/src/services/access.ts b/server/src/services/access.ts index fced6f2c698..d7ae99b09a7 100644 --- a/server/src/services/access.ts +++ b/server/src/services/access.ts @@ -1162,6 +1162,7 @@ export function accessService(db: Db) { return { isInstanceAdmin, decide, + decideReads: authorization.decideReads, projectDiscoveryCandidateIds: authorization.projectDiscoveryCandidateIds, canUser, hasPermission, diff --git a/server/src/services/heartbeat-run-privacy.ts b/server/src/services/heartbeat-run-privacy.ts index 17b61f4cbbd..d1cb41a19a2 100644 --- a/server/src/services/heartbeat-run-privacy.ts +++ b/server/src/services/heartbeat-run-privacy.ts @@ -1,7 +1,7 @@ import type { Db } from "@paperclipai/db"; import { issues, heartbeatRuns } from "@paperclipai/db"; -import { and, eq } from "drizzle-orm"; -import { canActorReadExecutionWorkspace, type AuthorizationActor, type AuthorizationDecision } from "./authorization.js"; +import { and, eq, inArray } from "drizzle-orm"; +import { canActorReadExecutionWorkspace, type AuthorizationActor, type AuthorizationDecision, type authorizationService } from "./authorization.js"; import { retryIdempotentDatabaseOperation } from "../database-retry.js"; type RunIssueBinding = { @@ -13,6 +13,7 @@ type RunIssueBinding = { export async function canActorReadHeartbeatRun( db: Db, access: { + decideReads?: ReturnType["decideReads"]; decide(input: { actor: AuthorizationActor; action: "issue:read"; @@ -66,6 +67,44 @@ export async function getHeartbeatRunReadDecisions( actor: AuthorizationActor, runs: readonly RunIssueBinding[], ): Promise { + const decideReads = access.decideReads; + if (decideReads) { + const bindings = runs.map(run => ({ companyId: run.companyId, scopeKind: run.scopeKind, issueId: run.issueId })); + const key = (companyId: string, issueId: string) => JSON.stringify([companyId, issueId]); + const issuesByCompany = new Map>(); + for (const run of bindings) { + if (run.scopeKind === "company" || !run.issueId) continue; + let ids = issuesByCompany.get(run.companyId); + if (!ids) { ids = new Set(); issuesByCompany.set(run.companyId, ids); } + ids.add(run.issueId); + } + const resources: Parameters[1] = []; + for (const [companyId, issueIds] of issuesByCompany) { + const ids = [...issueIds]; + for (let offset = 0; offset < ids.length; offset += 500) { + const chunk = ids.slice(offset, offset + 500); + const existing = await retryIdempotentDatabaseOperation(() => db + .select({ id: issues.id }) + .from(issues) + .where(and(eq(issues.companyId, companyId), inArray(issues.id, chunk)))); + for (const { id: issueId } of existing) resources.push({ + type: "issue", companyId, issueId, projectId: null, parentIssueId: null, + assigneeAgentId: null, assigneeUserId: null, + }); + } + } + const allowed = new Map(); + if (resources.length) { + const decisions = await retryIdempotentDatabaseOperation(() => decideReads(actor, resources)); + for (let index = 0; index < resources.length; index++) { + const resource = resources[index]; + if (resource.type === "issue" && resource.issueId) { + allowed.set(key(resource.companyId, resource.issueId), decisions[index]?.allowed === true); + } + } + } + return bindings.map(run => run.scopeKind === "company" || Boolean(run.issueId && allowed.get(key(run.companyId, run.issueId)))); + } const decisions = new Map>(); return Promise.all(runs.map(run => { const binding = { companyId: run.companyId, scopeKind: run.scopeKind, issueId: run.issueId }; From 50797ea03ce0b2a84e4df80864ac6e3652382ed5 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 00:41:14 -0700 Subject: [PATCH 27/68] perf(issues): narrow queued comment wake display reads (#17570) Read only typed queue display inputs from saved wake payloads and only runtime mode from queued runs. Preserve complete rows for locked edits, discard, steering, and interruption. Validation: 93 focused route and policy tests, full build and repository typecheck. PostgreSQL fixtures preserve identical queue responses and saved input with less than 4 KB of read values; restoring the full wake read fails at about 1.8 MB. Exact-head Greptile 5/5 (medium), all CI checks green, and zero unresolved comments. Co-Authored-By: Paperclip --- doc/DEVELOPING.md | 6 ++++ .../issue-queued-comments-routes.test.ts | 32 +++++++++++++++++++ server/src/routes/issues.ts | 26 ++++++++++++--- 3 files changed, 59 insertions(+), 5 deletions(-) diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 3cc2067d727..fdfc680c06e 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1816,6 +1816,12 @@ dispatch with cancellation and execution, and re-exports the existing public helpers and configuration-incomplete error class. Keep preparation policy changes in this module and its tests. +Queued-comment display reads select only wake identity, queue state, typed comment +and interaction bindings, the displayed wait reason, and fresh-session flags. +They read only the queued run's runtime mode. Mutation locks retain complete +wake and run rows, so edits, ordering, interruption, and saved execution inputs +keep their existing write contract. + Run retrieval and session state are in `server/src/services/heartbeat/run-state.ts`. It owns bounded run projections, database encoding checks, task session reads and writes, explicit resumes, session compaction, and usage/billing helpers. diff --git a/server/src/__tests__/issue-queued-comments-routes.test.ts b/server/src/__tests__/issue-queued-comments-routes.test.ts index 60985954bea..7b0af0e8a3d 100644 --- a/server/src/__tests__/issue-queued-comments-routes.test.ts +++ b/server/src/__tests__/issue-queued-comments-routes.test.ts @@ -220,6 +220,38 @@ describeEmbeddedPostgres("issue queued-comment routes", () => { } finally { capture.flush(); await observed.$client.end(); } }); + it.each(["deferred", "queued"])("reads only queue display inputs from large %s wake payloads", async state => { + const f = await seedQueue(); + if (state === "queued") await promoteQueue(f); + const client = app(f.companyId); + const path = `/api/issues/${f.issueId}/queued-comments`; + const expected = await request(client).get(path).expect(200); + const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId)); + const payload = "saved-wake-input🙂".repeat(30_000); + const context = wake.payload?._paperclipWakeContext as Record; + await db.update(agentWakeupRequests).set({ payload: { + ...wake.payload, prompt: payload, + _paperclipWakeContext: { ...context, prompt: payload, nativeExecutionInput: { instructions: payload } }, + } }).where(eq(agentWakeupRequests.id, f.wakeId)); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, prompt: payload }, + runnerProfileJson: { instructions: payload }, resultJson: { output: payload }, + }).where(eq(heartbeatRuns.companyId, f.companyId)); + const [before] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId)); + const observed = createDb(tempDb!.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { agentWakeupRequests, heartbeatRuns }, capture); + try { + const response = await request(app(f.companyId, "queue-owner", undefined, observed)).get(path).expect(200); + expect(response.body).toEqual(expected.body); + capture.flush(); + const reads = events.filter(event => ["agent_wakeup_requests", "heartbeat_runs"].includes(String(event.table)) && event.operation === "select"); + expect(reads.some(event => event.table === "agent_wakeup_requests")).toBe(true); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(4096); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId)))[0]).toEqual(before); + await request(app(randomUUID(), "queue-owner", undefined, observed)).get(path).expect(404); + } finally { capture.flush(); await observed.$client.end(); } + }); + it.each(["cancelled", "running"])("rejects a late Done from an interrupted %s task run at the write boundary", async status => { const seeded = await seedQueue(); await db.update(heartbeatRuns).set({ status, resultJson: { executionCancellation: { state: "requested" } } }) diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index 6aa65dfb774..bb92497ea3b 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -7287,9 +7287,9 @@ export function issueRoutes( type IssueQueueWake = typeof agentWakeupRequests.$inferSelect; type IssueQueueRun = typeof heartbeatRuns.$inferSelect; type IssueQueueState = { - wake: IssueQueueWake; + wake: Pick; state: "deferred" | "queued"; - queueRun: IssueQueueRun | null; + queueRun: Pick | null; }; async function findQueuedCommentWake( @@ -7298,7 +7298,23 @@ export function issueRoutes( ): Promise { if (!issue.assigneeAgentId) return null; const rows = await executor - .select() + .select({ + id: agentWakeupRequests.id, companyId: agentWakeupRequests.companyId, + status: agentWakeupRequests.status, runId: agentWakeupRequests.runId, + payload: sql>`jsonb_build_object( + 'issueId', ${agentWakeupRequests.payload}->'issueId', + 'mutation', ${agentWakeupRequests.payload}->'mutation', + 'interactionId', ${agentWakeupRequests.payload}->'interactionId', + 'interactionStatus', ${agentWakeupRequests.payload}->'interactionStatus', + 'externalChatContinuation', ${agentWakeupRequests.payload}->'externalChatContinuation', + 'executionWait', jsonb_build_object( + 'reason', ${agentWakeupRequests.payload}->'executionWait'->'reason', + 'message', ${agentWakeupRequests.payload}->'executionWait'->'message'), + '_paperclipWakeContext', jsonb_build_object( + 'wakeCommentIds', ${agentWakeupRequests.payload}->'_paperclipWakeContext'->'wakeCommentIds', + 'forceFreshSession', ${agentWakeupRequests.payload}->'_paperclipWakeContext'->'forceFreshSession') + )`, + }) .from(agentWakeupRequests) .where( and( @@ -7323,7 +7339,7 @@ export function issueRoutes( } if (!wake.runId) continue; const queueRun = await executor - .select() + .select({ runtimeMode: heartbeatRuns.runtimeMode }) .from(heartbeatRuns) .where( and( @@ -7344,7 +7360,7 @@ export function issueRoutes( async function queueCommentsForWake( executor: IssueQueueDb, issueId: string, - wake: IssueQueueWake | null, + wake: Pick | null, ) { if (wake && queuedInteractionId(wake.payload)) { const response = await readQueuedInteractionResponse(executor as Db, wake.companyId, issueId, wake.payload); From 3f36947e0cf18f58f57f34cefd53b49af54c790c Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 02:43:17 -0700 Subject: [PATCH 28/68] perf(realtime): batch task and run authorization per event (#17579) Use one fresh authorization batch for each live event task and run binding. Preserve current membership, grant, company, and payload disclosure checks. Build, typecheck, 70 affected local tests, and the complete required CI matrix passed. Greptile scored the exact head 5/5 at medium impact with no unresolved findings. The separate full local suite remains in progress. Co-Authored-By: Paperclip --- doc/DATABASE.md | 6 ++++ .../__tests__/live-events-run-read.test.ts | 31 ++++++++++++++++-- .../src/__tests__/live-events-toast.test.ts | 5 ++- server/src/realtime/live-events-ws.ts | 32 +++++++++---------- 4 files changed, 54 insertions(+), 20 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index a51b7cb6100..dd8126cbcd1 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -123,6 +123,12 @@ launch context does not travel with each policy or parent-report check. JSON types, task-key precedence, and company and agent identity checks are preserved. Each decision still reads current data; this is not an authorization cache. +WebSocket delivery uses one fresh authorization batch for an event's task and +run bindings. When both bindings name the same task, the batch reads its current +authority once. Different task bindings must both pass. The batch is discarded +after that delivery; later events still recheck grants, membership, task +existence, and company boundaries. + Filesystem preparation uses current task/run and selected-configuration checks. Historical root-reader authorization has been retired; no retained-reader batch is loaded at admission or publication. Workspace-record disclosure still uses task diff --git a/server/src/__tests__/live-events-run-read.test.ts b/server/src/__tests__/live-events-run-read.test.ts index 76704f2f63c..b02b0d4cf45 100644 --- a/server/src/__tests__/live-events-run-read.test.ts +++ b/server/src/__tests__/live-events-run-read.test.ts @@ -58,9 +58,19 @@ describe("live-event run read boundaries", () => { }).returning({ id: heartbeatRuns.id }); const [maintenance] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, scopeKind: "company" }).returning({ id: heartbeatRuns.id }); const [foreign] = await db.insert(heartbeatRuns).values({ companyId: foreignCompanyId, agentId: foreignAgent.id, scopeKind: "company" }).returning({ id: heartbeatRuns.id }); + const hiddenIssueId = randomUUID(); + await db.insert(issues).values({ id: hiddenIssueId, companyId, title: "Other private task", visibility: "private", privacyRootIssueId: hiddenIssueId, responsibleUserId: owner }); + const [hiddenRun] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, scopeKind: "issue", issueId: hiddenIssueId }).returning({ id: heartbeatRuns.id }); const reads: string[] = []; + const queries: string[] = []; let failNextRunRead = false; + let failNextIssueRead = false; const observedDb = drizzle(db.$client, { schema, logger: { logQuery(query) { + queries.push(query); + if (query.startsWith("select ") && query.includes('from "issues"') && failNextIssueRead) { + failNextIssueRead = false; + throw new Error("fixture issue lookup unavailable"); + } if (query.startsWith("select ") && query.includes('from "heartbeat_runs"')) { reads.push(query); if (failNextRunRead) { failNextRunRead = false; throw new Error("fixture run lookup unavailable"); } @@ -78,17 +88,32 @@ describe("live-event run read boundaries", () => { socket.on("message", bytes => messages.push(JSON.parse(bytes.toString()))); try { await once(socket, "open"); - async function deliver(runId: string) { + let deliveries = 0; + async function deliver(runId: string, payloadIssueId?: string) { + deliveries += 1; const start = messages.length; const barrier = randomUUID(); - publishLiveEvent({ companyId, type: "heartbeat.run.status", payload: { runId, status: "running" } }); + publishLiveEvent({ companyId, type: "heartbeat.run.status", payload: { + runId, status: "running", ...(payloadIssueId ? { issueId: payloadIssueId } : {}), + } }); publishLiveEvent({ companyId, type: "agent.status", payload: { agentId: barrier, status: "idle" } }); await expect.poll(() => messages.some(message => message.payload.agentId === barrier)).toBe(true); return messages.slice(start).filter(message => message.type === "heartbeat.run.status"); } expect(await deliver(run.id)).toMatchObject([{ payload: { runId: run.id, status: "running" } }]); + const batchStart = queries.length; + expect(await deliver(run.id, issueId)).toMatchObject([{ payload: { runId: run.id, issueId, status: "running" } }]); + expect(queries.length - batchStart).toBeLessThanOrEqual(6); + expect(queries.slice(batchStart).filter(query => query.startsWith("select ") && query.includes('from "issues"')).length).toBeLessThanOrEqual(2); + expect(await deliver(hiddenRun.id, issueId)).toEqual([]); + expect(await deliver(run.id, hiddenIssueId)).toEqual([]); + expect(await deliver(run.id, randomUUID())).toEqual([]); + failNextIssueRead = true; + expect(await deliver(run.id, issueId)).toEqual([]); + expect(await deliver(run.id, issueId)).toHaveLength(1); await db.update(issueAccessGrants).set({ revokedAt: new Date() }).where(eq(issueAccessGrants.id, grant.id)); + expect(await deliver(run.id, issueId)).toEqual([]); expect(await deliver(run.id)).toEqual([]); expect(await deliver(maintenance.id)).toHaveLength(1); expect(await deliver(foreign.id)).toEqual([]); @@ -101,7 +126,7 @@ describe("live-event run read boundaries", () => { await db.delete(issues).where(eq(issues.id, issueId)); expect(await deliver(run.id)).toEqual([]); - expect(reads).toHaveLength(9); + expect(reads).toHaveLength(deliveries); for (const query of reads) { expect(query).toMatch(/^select "company_id", "scope_kind", "issue_id" from "heartbeat_runs"/); expect(query).not.toMatch(/context_snapshot|runner_profile_json|stdout_excerpt|result_json/); diff --git a/server/src/__tests__/live-events-toast.test.ts b/server/src/__tests__/live-events-toast.test.ts index 5c5fbe7bf6c..e82af433d73 100644 --- a/server/src/__tests__/live-events-toast.test.ts +++ b/server/src/__tests__/live-events-toast.test.ts @@ -14,7 +14,10 @@ vi.mock("../realtime/task-toast-eligibility.js", () => ({ taskToastEligible: vi. vi.mock("../services/authorization.js", () => ({ authorizationService: () => ({ decide: async ({ resource }: { resource: { issueId: string } }) => ({ allowed: resource.issueId !== "hidden" }) }), })); -vi.mock("../services/heartbeat-run-privacy.js", () => ({ canActorReadHeartbeatRun: async () => false })); +vi.mock("../services/heartbeat-run-privacy.js", () => ({ + getHeartbeatRunReadDecisions: async (_db: unknown, _access: unknown, _actor: unknown, runs: { issueId?: string }[]) => + runs.map(run => run.issueId !== "hidden"), +})); vi.mock("../services/case-access.js", () => ({ canActorReadCase: async (_db: unknown, _actor: unknown, record: { id: string }) => record.id !== "hidden-case", })); diff --git a/server/src/realtime/live-events-ws.ts b/server/src/realtime/live-events-ws.ts index 6f299bfe672..95159016122 100644 --- a/server/src/realtime/live-events-ws.ts +++ b/server/src/realtime/live-events-ws.ts @@ -5,7 +5,7 @@ import { createRequire } from "node:module"; import type { Duplex } from "node:stream"; import { and, eq, isNull } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; -import { agentApiKeys, companyMemberships, instanceUserRoles, heartbeatRuns, issues, projects } from "@paperclipai/db"; +import { agentApiKeys, companyMemberships, instanceUserRoles, heartbeatRuns, projects } from "@paperclipai/db"; import type { DeploymentMode, LiveEvent } from "@paperclipai/shared"; import type { BetterAuthSessionResult } from "../auth/better-auth.js"; import { logger } from "../middleware/logger.js"; @@ -45,7 +45,7 @@ const { WebSocket, WebSocketServer } = require("ws") as { }; import { authorizationService, type AuthorizationActor } from "../services/authorization.js"; -import { canActorReadHeartbeatRun } from "../services/heartbeat-run-privacy.js"; +import { getHeartbeatRunReadDecisions } from "../services/heartbeat-run-privacy.js"; import { canActorReadCase } from "../services/case-access.js"; interface UpgradeContext { @@ -301,12 +301,22 @@ export function setupLiveEventsWebSocketServer( const payload = event.payload; const issueId = typeof payload.issueId === "string" ? payload.issueId : payload.entityType === "issue" && typeof payload.entityId === "string" ? payload.entityId : null; + const runId = typeof payload.runId === "string" ? payload.runId + : payload.entityType === "heartbeat_run" && typeof payload.entityId === "string" ? payload.entityId : null; + const readBindings: Parameters[3][number][] = []; if (issueId) { - const [issue] = await db.select({ id: issues.id }).from(issues) - .where(and(eq(issues.id, issueId), eq(issues.companyId, event.companyId))).limit(1); - if (!issue || !(await access.decide({ actor: context!.actor, action: "issue:read", - resource: { type: "issue", companyId: event.companyId, issueId } })).allowed) return null; + readBindings.push({ companyId: event.companyId, scopeKind: "issue", issueId }); } + if (runId) { + const [run] = await db.select({ + companyId: heartbeatRuns.companyId, + scopeKind: heartbeatRuns.scopeKind, + issueId: heartbeatRuns.issueId, + }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, event.companyId))); + if (!run) return null; + readBindings.push(run); + } + if ((await getHeartbeatRunReadDecisions(db, access, context!.actor, readBindings)).some(allowed => !allowed)) return null; if (["company_skill", "company_skill_source"].includes(String(payload.entityType)) && typeof payload.entityId === "string") { try { const skills = skillAccessService(db, context!.actor); @@ -320,16 +330,6 @@ export function setupLiveEventsWebSocketServer( if (!project || !(await access.decide({ actor: context!.actor, action: "project:read", resource: { type: "project", companyId: event.companyId, projectId: project.id } })).allowed) return null; } - const runId = typeof payload.runId === "string" ? payload.runId - : payload.entityType === "heartbeat_run" && typeof payload.entityId === "string" ? payload.entityId : null; - if (runId) { - const [run] = await db.select({ - companyId: heartbeatRuns.companyId, - scopeKind: heartbeatRuns.scopeKind, - issueId: heartbeatRuns.issueId, - }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, event.companyId))); - if (!run || !(await canActorReadHeartbeatRun(db, access, context!.actor, run))) return null; - } if (event.type.startsWith("heartbeat.run.")) return runId ? event : null; if (event.type === "agent.session.goal.changed") return issueId ? event : null; // Company-wide invalidations carry no task-derived content. Authorized From c4c4f7eecb6af501451a61b11c68d37f95ca4700 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 03:20:45 -0700 Subject: [PATCH 29/68] perf(environments): reuse provider schemas after content validation (#17581) Add a separate schema-only provider lookup with current plugin identity and SHA-256 content validation in one statement. Retain bounded schema definitions, return independent objects, and preserve current full driver and secret access paths. Build, repository typecheck, 485 affected tests, the supported-host database tests, and the complete required CI matrix passed. Greptile scored this exact head 5/5 at medium impact with no unresolved findings. Co-Authored-By: Paperclip --- doc/DATABASE.md | 13 ++ .../environment-custom-image-routes.test.ts | 1 + .../environment-instance-routes.test.ts | 1 + .../src/__tests__/environment-probe.test.ts | 2 + .../src/__tests__/environment-routes.test.ts | 2 + .../sandbox-provider-schema-reads.test.ts | 173 ++++++++++++++++++ server/src/services/environment-config.ts | 8 +- .../src/services/plugin-environment-driver.ts | 59 +++++- 8 files changed, 252 insertions(+), 7 deletions(-) create mode 100644 server/src/__tests__/sandbox-provider-schema-reads.test.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index dd8126cbcd1..bf4c88a98ca 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -826,6 +826,19 @@ pnpm secrets:migrate-inline-env --apply Hosted AWS provider notes live in [SECRETS-AWS-PROVIDER.md](./SECRETS-AWS-PROVIDER.md). +Sandbox runtime configuration resolves a provider's secret-reference fields +through its schema-only lookup. Each lookup reads the selected plugin ID +and a SHA-256 hash of its current schema in one database statement. A matching +retained schema is omitted from the query result. Selection order, schema +changes, and deletion remain visible on the next lookup, without a time-based +cache. A database error still fails the lookup. + +The retained schemas are private to one `Db` object, with at most 32 entries of +64 KiB each in serialized form. Larger schemas still work and are read in full. +Callers receive independent schema objects. This reuse stores neither resolved +secrets nor authorization decisions. Default driver lookups still read complete +current plugin records and enforce their existing readiness and worker checks. + ### Persistent agent conversations Migration `0274_agent_chat.sql` adds conversation identity/state and session generation/boundary columns to `issues`, plus idempotent client request IDs and processed session-boundary generations to `issue_comments`. The company/agent/user unique index resolves concurrent first writes to one issue. A check constraint preserves the assigned-agent identity and prevents terminal conversation status. Comment request IDs are unique per issue and user. There is no separate chat/message store. Provider sessions continue to use `agent_task_sessions`; `/new` removes only the matching conversation session, and session writers fence stale generations against the issue row. diff --git a/server/src/__tests__/environment-custom-image-routes.test.ts b/server/src/__tests__/environment-custom-image-routes.test.ts index 82b8eda9d37..43b3ae382e4 100644 --- a/server/src/__tests__/environment-custom-image-routes.test.ts +++ b/server/src/__tests__/environment-custom-image-routes.test.ts @@ -92,6 +92,7 @@ vi.mock("../services/plugin-environment-driver.js", () => ({ REUSABLE_LEASE_WORKER_METHODS: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"], listReadyPluginEnvironmentDrivers: vi.fn(async () => []), resolvePluginSandboxProviderDriverByKey: vi.fn(async () => null), + resolvePluginSandboxProviderConfigSchema: vi.fn(async () => null), validatePluginEnvironmentDriverConfig: vi.fn(async ({ config }) => config), validatePluginSandboxProviderConfig: vi.fn(async ({ provider, config }) => ({ normalizedConfig: config, diff --git a/server/src/__tests__/environment-instance-routes.test.ts b/server/src/__tests__/environment-instance-routes.test.ts index 51664e01d98..293e12b7876 100644 --- a/server/src/__tests__/environment-instance-routes.test.ts +++ b/server/src/__tests__/environment-instance-routes.test.ts @@ -79,6 +79,7 @@ vi.mock("../services/plugin-environment-driver.js", () => ({ REUSABLE_LEASE_WORKER_METHODS: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"], listReadyPluginEnvironmentDrivers: vi.fn(async () => []), resolvePluginSandboxProviderDriverByKey: vi.fn(async () => null), + resolvePluginSandboxProviderConfigSchema: vi.fn(async () => null), validatePluginEnvironmentDriverConfig: vi.fn(async ({ config }) => config), validatePluginSandboxProviderConfig: vi.fn(async ({ provider, config }) => ({ normalizedConfig: config, diff --git a/server/src/__tests__/environment-probe.test.ts b/server/src/__tests__/environment-probe.test.ts index e00c2d3eb6f..9474fd8b5a9 100644 --- a/server/src/__tests__/environment-probe.test.ts +++ b/server/src/__tests__/environment-probe.test.ts @@ -25,6 +25,8 @@ vi.mock("../services/plugin-environment-driver.js", () => ({ probePluginEnvironmentDriver: mockProbePluginEnvironmentDriver, probePluginSandboxProviderDriver: mockProbePluginSandboxProviderDriver, resolvePluginSandboxProviderDriverByKey: mockResolvePluginSandboxProviderDriverByKey, + resolvePluginSandboxProviderConfigSchema: async (input: { db: unknown; driverKey: string }) => + (await mockResolvePluginSandboxProviderDriverByKey(input))?.driver?.configSchema ?? null, })); vi.mock("../services/environment-runtime.js", () => ({ diff --git a/server/src/__tests__/environment-routes.test.ts b/server/src/__tests__/environment-routes.test.ts index 07c90052f37..9ad0bf972c4 100644 --- a/server/src/__tests__/environment-routes.test.ts +++ b/server/src/__tests__/environment-routes.test.ts @@ -129,6 +129,8 @@ vi.mock("../services/plugin-environment-driver.js", () => ({ REUSABLE_LEASE_WORKER_METHODS: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"], listReadyPluginEnvironmentDrivers: mockListReadyPluginEnvironmentDrivers, resolvePluginSandboxProviderDriverByKey: mockResolvePluginSandboxProviderDriverByKey, + resolvePluginSandboxProviderConfigSchema: async (input: { db: unknown; driverKey: string }) => + (await mockResolvePluginSandboxProviderDriverByKey(input))?.driver?.configSchema ?? null, startPluginEnvironmentInteractiveSetup: mockStartPluginEnvironmentInteractiveSetup, getPluginEnvironmentInteractiveSetup: mockGetPluginEnvironmentInteractiveSetup, capturePluginEnvironmentTemplate: mockCapturePluginEnvironmentTemplate, diff --git a/server/src/__tests__/sandbox-provider-schema-reads.test.ts b/server/src/__tests__/sandbox-provider-schema-reads.test.ts new file mode 100644 index 00000000000..55f81f8bfbe --- /dev/null +++ b/server/src/__tests__/sandbox-provider-schema-reads.test.ts @@ -0,0 +1,173 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { eq, sql, type SQL } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest"; +import { createDb, plugins, type Db } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import type { PaperclipPluginManifestV1, PluginEnvironmentDriverDeclaration } from "@paperclipai/shared"; +import { resolvePluginSandboxProviderConfigSchema, resolvePluginSandboxProviderDriverByKey } from "../services/plugin-environment-driver.js"; +import { resolveEnvironmentDriverConfigForRuntime, resolveSandboxProviderSecretRefPaths } from "../services/environment-config.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("sandbox provider schema reads", () => { + let temporary: Awaited>; + let db: Db; + const clients: Db[] = []; + const key = "fixture-schema-provider"; + const driver = (driverKey = key, configSchema: unknown = { type: "object" }) => ({ + driverKey, kind: "sandbox_provider", displayName: "Schema fixture", configSchema, + }) as PluginEnvironmentDriverDeclaration; + const manifest = (id: string, declarations: PluginEnvironmentDriverDeclaration[]): PaperclipPluginManifestV1 => ({ + id: `test.schema.${id}`, apiVersion: 1, version: "1.0.0", displayName: "Schema fixture", + description: "Fixture", author: "Test", categories: ["automation"], + capabilities: ["environment.drivers.register"], entrypoints: { worker: "worker.js" }, environmentDrivers: declarations, + }); + async function seed(declarations = [driver()], installOrder = 0) { + const id = randomUUID(); + await db.insert(plugins).values({ + id, pluginKey: `test.schema.${id}`, packageName: "@test/schema", version: "1.0.0", + installOrder, status: "disabled", manifestJson: manifest(id, declarations), + }); + return id; + } + function observer() { + const observed = createDb(temporary.connectionString); + clients.push(observed); + return observed; + } + const lookup = (client: Db, driverKey = key) => resolvePluginSandboxProviderConfigSchema({ db: client, driverKey }); + async function measure(client: Db, operation: () => Promise) { + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event), () => now); + installDatabaseResponseCapture(client, { plugins }, capture); + try { + const value = await operation(); + capture.flush(); + const entries = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1"); + return { value, bytes: entries.reduce((sum, entry) => sum + Number(entry.responseValueBytes), 0), + calls: entries.reduce((sum, entry) => sum + Number(entry.calls), 0) }; + } finally { capture.flush(); } + } + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("sandbox-schema-reads-"); + db = createDb(temporary.connectionString); + }, 120_000); + beforeEach(async () => { await db.delete(plugins); }); + afterEach(async () => { await Promise.all(clients.splice(0).map(client => client.$client.end())); }); + afterAll(async () => { await temporary?.cleanup(); }); + + it("keeps runtime secret paths current without retransferring an unchanged schema", async () => { + const schema = { type: "object", description: randomBytes(16 * 1024).toString("hex"), properties: { + credentials: { type: "object", properties: { token: { type: "string", format: "secret-ref" } } }, + } }; + const id = await seed([driver(key, schema)]), observed = observer(); + const before = await db.select().from(plugins).where(eq(plugins.id, id)); + const cold = await measure(observed, () => resolveSandboxProviderSecretRefPaths(observed, key)); + const warm = await measure(observed, () => resolveSandboxProviderSecretRefPaths(observed, key)); + expect(cold.value).toEqual(new Set(["credentials.token"])); + expect(warm.value).toEqual(cold.value); + expect(cold.calls).toBe(1); + expect(warm.calls).toBe(1); + expect(cold.bytes).toBeGreaterThan(32 * 1024); + expect(warm.bytes).toBeLessThan(256); + const changed = { type: "object", allOf: [{ properties: { apiKey: { type: "string", format: "secret-ref" } } }] }; + await db.update(plugins).set({ manifestJson: manifest(id, [driver(key, changed)]) }).where(eq(plugins.id, id)); + expect(await resolveSandboxProviderSecretRefPaths(observed, key)).toEqual(new Set(["apiKey"])); + const environment = { id: randomUUID(), driver: "sandbox" as const, config: { provider: key, apiKey: randomUUID() } }; + await expect(resolveEnvironmentDriverConfigForRuntime(observed, null, environment)).rejects.toThrow("companyId"); + await db.update(plugins).set({ manifestJson: before[0].manifestJson }).where(eq(plugins.id, id)); + expect(await db.select().from(plugins).where(eq(plugins.id, id))).toEqual(before); + }); + + it("preserves install order, first matching declarations, and default readiness checks", async () => { + const a = await seed([driver(key, { title: "first" }), driver(key, { title: "second" })], 1); + const b = await seed([driver(key, { title: "other plugin" })], 2), observed = observer(); + expect((await lookup(observed))).toEqual({ title: "first" }); + const full = await resolvePluginSandboxProviderDriverByKey({ db: observed, driverKey: key }); + expect(full?.plugin.id).toBe(a); + expect(full?.plugin.status).toBe("disabled"); + expect(await resolvePluginSandboxProviderDriverByKey({ db: observed, driverKey: key, requireRunning: true })).toBeNull(); + await db.update(plugins).set({ installOrder: 0 }).where(eq(plugins.id, b)); + expect((await lookup(observed))).toEqual({ title: "other plugin" }); + await db.delete(plugins).where(eq(plugins.id, b)); + expect((await lookup(observed))).toEqual({ title: "first" }); + await db.update(plugins).set({ manifestJson: manifest(a, [driver("replacement")]) }).where(eq(plugins.id, a)); + expect(await lookup(observed)).toBeNull(); + }); + + it.each([null, true, "invalid", [], undefined])("preserves non-object schema behavior: %s", async value => { + const first = driver(key, value); + if (value === undefined) delete (first as Partial).configSchema; + await seed([first, driver(key, { properties: { secret: { format: "secret-ref" } } })]); + const observed = observer(); + expect(await resolveSandboxProviderSecretRefPaths(observed, key)).toEqual(new Set()); + expect(await resolveSandboxProviderSecretRefPaths(observed, key)).toEqual(new Set()); + }); + + it("returns independent schema objects and keeps separate database clients cold", async () => { + await seed([driver(key, { description: "x".repeat(4096), properties: { token: { format: "secret-ref" } } })]); + const a = observer(), b = observer(); + const first = await lookup(a); + first!.properties = {}; + expect(await resolveSandboxProviderSecretRefPaths(a, key)).toEqual(new Set(["token"])); + const warm = await measure(a, () => lookup(a)), independent = await measure(b, () => lookup(b)); + expect(warm.bytes).toBeLessThan(256); + expect(independent.bytes).toBeGreaterThan(4096); + warm.value!.properties = {}; + expect(await resolveSandboxProviderSecretRefPaths(a, key)).toEqual(new Set(["token"])); + }); + + it("bounds retained schemas and does not retain oversized definitions", async () => { + const declarations = Array.from({ length: 33 }, (_, index) => driver(`provider-${index}`, { description: "x".repeat(2048) })); + declarations.push(driver("oversized", { description: "x".repeat(128 * 1024) })); + await seed(declarations); + const observed = observer(); + for (let index = 0; index < 33; index++) await lookup(observed, `provider-${index}`); + expect((await measure(observed, () => lookup(observed, "provider-32"))).bytes).toBeLessThan(256); + expect((await measure(observed, () => lookup(observed, "provider-0"))).bytes).toBeGreaterThan(2048); + await lookup(observed, "oversized"); + expect((await measure(observed, () => lookup(observed, "oversized"))).bytes).toBeGreaterThan(128 * 1024); + }); + + it("binds unusual keys and never serves retained schemas after a lookup error", async () => { + const unusual = "provider-\"quoted' OR true --"; + await seed([driver(unusual, { title: "exact provider" })]); + const observed = observer(); + let fail = false; + const guarded = { execute: async (query: SQL) => { + if (fail) throw new Error("fixture database unavailable"); + return observed.execute(query); + } } as unknown as Db; + expect((await lookup(guarded, unusual))).toEqual({ title: "exact provider" }); + fail = true; + await expect(lookup(guarded, unusual)).rejects.toThrow("fixture database unavailable"); + fail = false; + expect((await lookup(guarded, unusual))).toEqual({ title: "exact provider" }); + expect(await lookup(guarded, "unknown-provider")).toBeNull(); + }); + + it("keeps each concurrent lookup tied to the database revision it observed", async () => { + const id = await seed([driver(key, { title: "old" })]), observed = observer(); + let pause = false, release!: () => void, reached!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const paused = new Promise(resolve => { reached = resolve; }); + const raced = { execute: async (query: SQL) => { + const rows = await observed.execute(query); + if (pause) { pause = false; reached(); await gate; } + return rows; + } } as unknown as Db; + expect((await lookup(raced))).toEqual({ title: "old" }); + pause = true; + const oldRead = lookup(raced); + await paused; + try { + await db.update(plugins).set({ manifestJson: manifest(id, [driver(key, { title: "new" })]) }).where(eq(plugins.id, id)); + expect((await lookup(raced))).toEqual({ title: "new" }); + } finally { release(); } + expect((await oldRead)).toEqual({ title: "old" }); + expect((await lookup(raced))).toEqual({ title: "new" }); + expect((await db.execute<{ title: string }>(sql`select ${plugins.manifestJson}->'environmentDrivers'->0->'configSchema'->>'title' as title from ${plugins} where ${plugins.id}=${id}`))[0].title).toBe("new"); + }); +}); diff --git a/server/src/services/environment-config.ts b/server/src/services/environment-config.ts index b9c041a35d1..6431c4802ff 100644 --- a/server/src/services/environment-config.ts +++ b/server/src/services/environment-config.ts @@ -17,7 +17,7 @@ import { unprocessable } from "../errors.js"; import { parseObject } from "../adapters/utils.js"; import { secretService } from "./secrets.js"; import { - resolvePluginSandboxProviderDriverByKey, + resolvePluginSandboxProviderConfigSchema, validatePluginEnvironmentDriverConfig, validatePluginSandboxProviderConfig, } from "./plugin-environment-driver.js"; @@ -179,14 +179,10 @@ async function getSandboxProviderConfigSchema( db: Db, provider: string, ): Promise | null> { - const resolved = await resolvePluginSandboxProviderDriverByKey({ + return resolvePluginSandboxProviderConfigSchema({ db, driverKey: provider, }); - const schema = resolved?.driver.configSchema; - return schema && typeof schema === "object" && !Array.isArray(schema) - ? schema as Record - : null; } export async function resolveSandboxProviderSecretRefPaths( diff --git a/server/src/services/plugin-environment-driver.ts b/server/src/services/plugin-environment-driver.ts index d7f56f2b197..f26f32483ff 100644 --- a/server/src/services/plugin-environment-driver.ts +++ b/server/src/services/plugin-environment-driver.ts @@ -1,4 +1,5 @@ -import type { Db } from "@paperclipai/db"; +import { plugins, type Db } from "@paperclipai/db"; +import { sql } from "drizzle-orm"; import type { EnvironmentProbeResult, PluginEnvironmentConfig, @@ -162,6 +163,62 @@ export async function resolvePluginEnvironmentDriverByKey(input: { }); } +type CachedProviderSchema = { pluginId: string; revision: string; schema: Record | null }; +const providerSchemas = new WeakMap>(); +const MAX_CACHED_PROVIDER_SCHEMAS = 32; +const MAX_CACHED_PROVIDER_SCHEMA_BYTES = 64 * 1024; + +export async function resolvePluginSandboxProviderConfigSchema(input: { + db: Db; driverKey: string; +}): Promise | null> { + const { db, driverKey } = input; + let cache = providerSchemas.get(db); + const cached = cache?.get(driverKey); + const unchanged = sql`coalesce(plugin_id = ${cached?.pluginId ?? null}::uuid + and revision = ${cached?.revision ?? null}, false)`; + const [row] = await db.execute<{ + pluginId: string; revision: string; unchanged: boolean; configSchema: unknown; + }>(sql`with selected_provider as materialized ( + select ${plugins.id} as plugin_id, ( + select declaration.value->'configSchema' + from jsonb_array_elements(${plugins.manifestJson}->'environmentDrivers') + with ordinality as declaration(value, position) + where declaration.value->'driverKey' = to_jsonb(${driverKey}::text) + and declaration.value->>'kind' = 'sandbox_provider' + order by declaration.position limit 1 + ) as schema_value + from ${plugins} + where ${plugins.manifestJson} @> ${JSON.stringify({ environmentDrivers: [{ driverKey, kind: "sandbox_provider" }] })}::jsonb + order by ${plugins.installOrder} limit 1 + ), versioned_provider as materialized ( + select plugin_id, schema_value, + encode(sha256(convert_to(coalesce(schema_value::text, 'null'), 'UTF8')), 'hex') as revision + from selected_provider + ) + select plugin_id as "pluginId", revision, ${unchanged} as unchanged, + case when ${unchanged} then null else schema_value end as "configSchema" + from versioned_provider`); + if (!row) { + cache?.delete(driverKey); + return null; + } + if (row.unchanged) { + if (!cached || cached.pluginId !== row.pluginId || cached.revision !== row.revision) { + throw new Error("Sandbox provider schema revision did not match the retained definition"); + } + return structuredClone(cached.schema); + } + const schema = row.configSchema && typeof row.configSchema === "object" && !Array.isArray(row.configSchema) + ? row.configSchema as Record : null; + cache?.delete(driverKey); + if (Buffer.byteLength(JSON.stringify(schema)) <= MAX_CACHED_PROVIDER_SCHEMA_BYTES) { + if (!cache) { cache = new Map(); providerSchemas.set(db, cache); } + if (cache.size >= MAX_CACHED_PROVIDER_SCHEMAS) cache.delete(cache.keys().next().value!); + cache.set(driverKey, { pluginId: row.pluginId, revision: row.revision, schema: structuredClone(schema) }); + } + return schema; +} + export async function resolvePluginSandboxProviderDriverByKey(input: { db: Db; driverKey: string; From 35d5b0686ab5bcaef733b3741b7cec55048993af Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 03:45:25 -0700 Subject: [PATCH 30/68] perf(runner): narrow current tool authority reads (#17587) Select the current run fields used by Native tool authority without transferring unused saved profiles or output. Preserve full context, receipts, authorization predicates, and locked mutation reads. Verified with full build and typecheck, 2,358 affected tests across 20 suites, a counterfactual real PostgreSQL byte regression, complete exact-head CI, and Greptile 5/5 Medium with no unresolved comments. Co-Authored-By: Paperclip --- doc/DATABASE.md | 6 +++ .../paperclip-runner-tool-authority.test.ts | 46 +++++++++++++++++++ .../paperclip-runner-tool-authority.ts | 14 +++++- 3 files changed, 64 insertions(+), 2 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index bf4c88a98ca..06cf24c9368 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -123,6 +123,12 @@ launch context does not travel with each policy or parent-report check. JSON types, task-key precedence, and company and agent identity checks are preserved. Each decision still reads current data; this is not an authorization cache. +Native tool authority lookups select current run identity, status, invocation, +responsible user, context, and receipt data. Saved profiles and output excerpts +stay in storage and are omitted from that read. Task and agent binding, current +status, budget, and task privacy checks retain their existing predicates. +Mutation transactions still lock and read complete run records. + WebSocket delivery uses one fresh authorization batch for an event's task and run bindings. When both bindings name the same task, the batch reads its current authority once. Different task bindings must both pass. The batch is discarded diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts index 04a9b79a2cc..76815a76482 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts @@ -20,6 +20,7 @@ import { issues, } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { initializeRunIdentity, reserveSteeredIdentity, reconcileSteeredIdentity } from "../run-identity.js"; import { documentService } from "../documents.js"; import { issueService } from "../issues.js"; @@ -198,6 +199,51 @@ describe("PaperclipRunnerToolAuthority", () => { ).rejects.toThrow("paperclip_runner_tool_not_advertised"); }); + it("reads current tool authority without transferring saved profiles or output", async () => { + const binding = { companyId, agentId, issueId, runId }; + const authority = new PaperclipRunnerToolAuthority(db, binding); + const contextCall = { tool: "get_task_context", callId: "context-read-width", arguments: {} }; + const apiCall = { tool: "search_api", callId: "api-read-width", arguments: { query: "read task" } }; + const expectedContext = await authority.execute(contextCall); + const expectedApi = await authority.execute(apiCall); + const [original] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const [actor] = await db.select({ status: agents.status }).from(agents).where(eq(agents.id, agentId)); + const payload = "saved-profile-and-output🙂".repeat(32_768); + await db.update(heartbeatRuns).set({ + runnerProfileJson: { ...original.runnerProfileJson, retainedPayload: payload }, + stdoutExcerpt: payload, + stderrExcerpt: payload, + }).where(eq(heartbeatRuns.id, runId)); + const [expanded] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const observed = createDb(temporary!.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns, issues, agents, companies }, capture); + try { + const reader = new PaperclipRunnerToolAuthority(observed, binding); + const context = await reader.execute(contextCall), api = await reader.execute(apiCall); + expect(context).toEqual(expectedContext); + expect(api).toEqual(expectedApi); + expect(JSON.stringify([context, api])).not.toContain("saved-profile-and-output"); + capture.flush(); + const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(16 * 1024); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]).toEqual(expanded); + await db.update(agents).set({ status: "paused" }).where(eq(agents.id, agentId)); + await expect(reader.execute(contextCall)).rejects.toThrow("paperclip_runner_tool_binding_not_authorized"); + await expect(reader.execute(apiCall)).rejects.toThrow("paperclip_runner_tool_binding_not_authorized"); + } finally { + capture.flush(); + await observed.$client.end(); + await db.update(agents).set({ status: actor.status }).where(eq(agents.id, agentId)); + await db.update(heartbeatRuns).set({ + runnerProfileJson: original.runnerProfileJson, + stdoutExcerpt: original.stdoutExcerpt, + stderrExcerpt: original.stderrExcerpt, + }).where(eq(heartbeatRuns.id, runId)); + } + }); + it("preserves platform receipt provenance in both mixed task history readers", async () => { const at = new Date("2026-10-10T12:00:00Z"); const commentIds = Array.from({ length: 4 }, () => randomUUID()); diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index 56bccadf0b6..40b140f01b2 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -1227,7 +1227,17 @@ export class PaperclipRunnerToolAuthority { async #boundContext() { if (this.binding.museRuntime && !this.binding.externalAuthorizingUserId) throw forbidden("Muse authorizer is unavailable"); - const [row] = await this.db.select({ issue: issues, actor: agents, run: heartbeatRuns }) + const [row] = await this.db.select({ issue: issues, actor: agents, run: { + id: heartbeatRuns.id, + companyId: heartbeatRuns.companyId, + agentId: heartbeatRuns.agentId, + runtimeMode: heartbeatRuns.runtimeMode, + status: heartbeatRuns.status, + invocationSource: heartbeatRuns.invocationSource, + responsibleUserId: heartbeatRuns.responsibleUserId, + contextSnapshot: heartbeatRuns.contextSnapshot, + resultJson: heartbeatRuns.resultJson, + } }) .from(heartbeatRuns) .innerJoin(issues, eq(issues.id, this.binding.issueId)) .innerJoin(agents, eq(agents.id, this.binding.agentId)) @@ -2168,7 +2178,7 @@ export class PaperclipRunnerToolAuthority { }); } - async #sourcePlanApproval(context: { issue: typeof issues.$inferSelect; run: typeof heartbeatRuns.$inferSelect }) { + async #sourcePlanApproval(context: { issue: typeof issues.$inferSelect; run: Pick }) { const plan = await handoffPlanContext(this.db, context.issue, this.#privacyActor(context.run)); if (!plan || !this.binding.museRuntime) return plan; const document = await documentService(this.db).getIssueDocumentByKey(plan.sourceIssueId, "plan"); From b7dc594bfaec15c8c87eb97a61931faf2d0aa518 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 03:54:59 -0700 Subject: [PATCH 31/68] perf(recovery): narrow saved-message execution-wait reads (#17588) Read current ownership, startup, cancellation, task-binding, and adapter evidence without transferring unused saved profiles, launch context, or output. Preserve normal continuation admission, recovery holds, and locked mutation reads. Verified with full build and typecheck, 537 affected tests across 12 suites, a counterfactual real PostgreSQL byte regression, complete exact-head CI, and Greptile 5/5 Medium with no unresolved comments. Co-Authored-By: Paperclip --- doc/DATABASE.md | 5 ++ server/src/services/heartbeat.ts | 2 + server/src/services/heartbeat/run-control.ts | 9 +-- .../src/services/heartbeat/run-state.test.ts | 61 +++++++++++++++++++ server/src/services/heartbeat/run-state.ts | 28 +++++++++ .../src/services/native-local-process-stop.ts | 3 +- 6 files changed, 103 insertions(+), 5 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 06cf24c9368..a814c50cfad 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -140,6 +140,11 @@ Historical root-reader authorization has been retired; no retained-reader batch loaded at admission or publication. Workspace-record disclosure still uses task privacy checks, independently of filesystem access. +Saved-message execution-wait scans read current run state, process ownership, +startup and cancellation receipts, task binding, and the recorded adapter kind. +Saved profiles, output excerpts, and unrelated launch context stay in storage. +Continuation admission and its locked mutation reads retain their existing gates. + The initial agent JWT run lookup reads only the cancellation state and conversation flag from saved run JSON. It does not transfer the full context or result. Run status, responsible-user identity, company and agent boundaries, diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index adb10d852c9..354f680eb1d 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -812,6 +812,7 @@ export function heartbeatService( resolveSessionBeforeForWakeup, getRun, getRunCleanupState, + getRunExecutionWait, getRunWorkspaceSettlement, getRunStatus, ensureRuntimeState, @@ -1393,6 +1394,7 @@ export function heartbeatService( const runControl = createHeartbeatRunControl(db, { getRun, getRunCleanupState, + getRunExecutionWait, options, envOrchestrator, activeRunExecutions, diff --git a/server/src/services/heartbeat/run-control.ts b/server/src/services/heartbeat/run-control.ts index b4ba5f0ddea..7387da1f74d 100644 --- a/server/src/services/heartbeat/run-control.ts +++ b/server/src/services/heartbeat/run-control.ts @@ -165,7 +165,7 @@ async function withBudgetStopDelivery(db: Db, work: () => Promise): Promis /** Cancellation shares executor ownership and delegates lifecycle and admission effects. */ export interface HeartbeatRunControlDependencies extends Pick, - "getRun" | "getAgent" | "getRunCleanupState">, + "getRun" | "getAgent" | "getRunCleanupState" | "getRunExecutionWait">, Pick, "mergeRunStopMetadataForAgent" | "setRunStatusFromLive" | "setRunStatus" | "setWakeupStatus" | "appendRunEvent">, Pick { @@ -326,6 +326,7 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont const { getRun, getRunCleanupState, + getRunExecutionWait, options, envOrchestrator, activeRunExecutions, @@ -509,7 +510,7 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont } } - async function resumeRemoteStopComments(run: typeof heartbeatRuns.$inferSelect, requestId?: string) { + async function resumeRemoteStopComments(run: NonNullable>>, requestId?: string) { if (!isHeartbeatRunTerminalStatus(run.status) || adapterExecutionControls.has(run.id)) return; const [preparationCoordinator] = run.runtimeMode === "legacy" && run.status === "cancelled" && !run.runtimeModeResolvedAt ? await db.select().from(nativeRunFinalizations).where(and( @@ -536,7 +537,7 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont await acknowledgedNativeStopExecutionHasStopped(db, currentRun) && !(await getExecutionBlocker(db, run.companyId, issueId)); const legacyContinuation = run.runtimeMode === "legacy" && - hasConversationContinuationPolicy((await getRun(run.id, { includeExecutionEvidence: true }))?.resultJson) && + hasConversationContinuationPolicy((await getRunExecutionWait(run.companyId, run.id))?.resultJson) && !(await getExecutionBlocker(db, run.companyId, issueId)); if (run.runtimeMode !== "native" && run.runtimeMode !== "legacy") return; const pending = await db.select().from(agentWakeupRequests).where(and( @@ -803,7 +804,7 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont const blocker = await getExecutionBlocker(db, wake.companyId, issueId); const sourceId = blocker?.runId; if (!sourceId || !isUuidLike(sourceId)) continue; - const run = await getRun(sourceId, { includeExecutionEvidence: true }); + const run = await getRunExecutionWait(wake.companyId, sourceId); if (!run || run.companyId !== wake.companyId || run.agentId !== wake.agentId) continue; if (canContinueCancelledRun(run)) { await resumeSavedLegacyComments(wake.companyId, wake.id).catch(err => { diff --git a/server/src/services/heartbeat/run-state.test.ts b/server/src/services/heartbeat/run-state.test.ts index 702f842e464..1b9c4276a61 100644 --- a/server/src/services/heartbeat/run-state.test.ts +++ b/server/src/services/heartbeat/run-state.test.ts @@ -97,6 +97,17 @@ describe("heartbeat run state module", () => { expect(raw.columns[0].resultJson).toBe(heartbeatRuns.resultJson); }); + it.each(["UTF8", "SQL_ASCII"])("keeps execution-wait reads company scoped with %s text handling", async encoding => { + const reader = readingDatabase(encoding), state = extracted.createHeartbeatRunState(reader.db); + expect(await state.getRunExecutionWait("company", "run")).toBeNull(); + expect(await state.getRunExecutionWait("other-company", "run")).toBeNull(); + expect(reader.scopes).toEqual([["company", "run"], ["other-company", "run"]]); + expect(reader.execute).toHaveBeenCalledTimes(1); + if (encoding === "UTF8") expect(reader.columns[0].error).toBe(heartbeatRuns.error); + else expect(new PgDialect().sqlToQuery((reader.columns[0].error as SQL.Aliased).sql).sql).toBe("NULL"); + expect(new PgDialect().sqlToQuery((reader.columns[0].resultJson as SQL.Aliased).sql).sql).toContain("startupPreparationSettledAt"); + }); + it("loads current run status without large fields or an encoding probe", async () => { const binding = { id: "run", companyId: "company", agentId: "agent", status: "running" }; const reader = readingDatabase("SQL_ASCII", [binding], []); @@ -159,6 +170,56 @@ describePostgres("heartbeat run state database wiring", () => { return { companyId, agent, issue, state: extracted.createHeartbeatRunState(db) }; } + it("reads current execution-wait evidence without transferring saved input or output", async () => { + const { companyId, agent, issue } = await fixture(); + const payload = randomBytes(128 * 1024).toString("base64"); + const context = { issueId: issue.id, connectionIntentResolution: "existing_connection", source: "connection_intent.resolved", + interactionKind: "connection_intent", interactionStatus: "expired", mutation: "interaction", interactionId: randomUUID() }; + const result = { startupCancellation: { beforeNativeSelection: true, futureProof: { version: 2 } }, + startupPreparationSettledAt: "2020-01-01T00:00:00Z", cancellation: { source: "provider", expected: false, + initiator: { type: "provider" }, reason: "Cancelled", recordedAt: "2020-01-01T00:00:00Z" }, + acpToolInventoryComplete: true, acpPendingToolCount: 0, stdout: payload }; + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, status: "cancelled", runtimeMode: "legacy", + startedAt: new Date("2020-01-01T00:00:00Z"), finishedAt: new Date("2020-01-01T00:01:00Z"), + contextSnapshot: { ...context, prompt: payload }, resultJson: result, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner", adapterConfig: { prompt: payload } }, savedInput: payload }, + stdoutExcerpt: payload, stderrExcerpt: payload, + }).returning(); + const observed = createDb(database.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns }, capture); + try { + const state = extracted.createHeartbeatRunState(observed); + const read = await state.getRunExecutionWait(companyId, run.id); + capture.flush(); + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBe(2); + expect(summary.responseValueBytes).toBeLessThan(4096); + expect(read).toMatchObject({ id: run.id, companyId, agentId: agent.id, status: "cancelled", contextSnapshot: context, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, + resultJson: { startupCancellation: result.startupCancellation, cancellation: result.cancellation, + acpToolInventoryComplete: true, acpPendingToolCount: 0 } }); + expect(JSON.stringify(read)).not.toContain(payload.slice(0, 64)); + expect(read).not.toHaveProperty("stdoutExcerpt"); + expect(read).not.toHaveProperty("stderrExcerpt"); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toEqual(run); + await db.update(heartbeatRuns).set({ status: "running", resultJson: { ...result, acpPendingToolCount: 1 }, + runnerProfileJson: { adapterDispatch: { adapterType: "process" }, savedInput: payload }, + }).where(eq(heartbeatRuns.id, run.id)); + expect(await state.getRunExecutionWait(companyId, run.id)).toMatchObject({ status: "running", + resultJson: { acpPendingToolCount: 1 }, runnerProfileJson: { adapterDispatch: { adapterType: "process" } } }); + expect(await state.getRunExecutionWait(randomUUID(), run.id)).toBeNull(); + expect(await state.getRunExecutionWait(companyId, randomUUID())).toBeNull(); + for (const value of [null, false, 42, [], {}, { nested: true }, "task"]) { + await db.update(heartbeatRuns).set({ contextSnapshot: { ...context, issueId: value }, + runnerProfileJson: { adapterDispatch: { adapterType: value } }, + }).where(eq(heartbeatRuns.id, run.id)); + expect(await state.getRunExecutionWait(companyId, run.id)).toMatchObject({ contextSnapshot: { issueId: value }, + runnerProfileJson: { adapterDispatch: { adapterType: value } } }); + } + } finally { capture.flush(); await observed.$client.end(); } + }); + it("reads current workspace access without execution snapshots and retains JSON types", async () => { const { companyId, agent, issue } = await fixture(); const payload = randomBytes(128 * 1024).toString("base64"), workspaceId = randomUUID(); diff --git a/server/src/services/heartbeat/run-state.ts b/server/src/services/heartbeat/run-state.ts index f870c3957fb..7ac3a3816a2 100644 --- a/server/src/services/heartbeat/run-state.ts +++ b/server/src/services/heartbeat/run-state.ts @@ -1131,6 +1131,33 @@ export function createHeartbeatRunState(db: Db) { .then((rows) => rows[0] ?? null); } + async function getRunExecutionWait(companyId: string, runId: string) { + const safeForLegacyEncoding = await hasUnsafeTextProjectionDatabase(); + return db.select({ + ...heartbeatRunIssueControlColumns, + finishedAt: heartbeatRuns.finishedAt, + errorCode: heartbeatRuns.errorCode, + error: safeForLegacyEncoding ? heartbeatRunSqlAsciiSafeColumns.error : heartbeatRuns.error, + nativeIssueId: heartbeatRuns.nativeIssueId, + nativeSessionId: heartbeatRuns.nativeSessionId, + processPid: heartbeatRuns.processPid, + processGroupId: heartbeatRunProcessGroupIdColumn, + processStartedAt: heartbeatRuns.processStartedAt, + sessionIdAfter: heartbeatRuns.sessionIdAfter, + executionStage: heartbeatRuns.executionStage, + controllerBootId: heartbeatRuns.controllerBootId, + controllerLeaseExpiresAt: heartbeatRuns.controllerLeaseExpiresAt, + resultJson: heartbeatRunExecutionEvidenceColumn, + contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then (select coalesce(jsonb_object_agg(key, value), '{}'::jsonb) + from jsonb_each(${heartbeatRuns.contextSnapshot}) + where key = any(array['issueId', 'connectionIntentResolution', 'source', 'interactionKind', + 'interactionStatus', 'mutation', 'interactionId'])) else null end`, + }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId), + )).then(rows => rows[0] ?? null); + } + async function getRunCleanupState(runId: string) { return db.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, @@ -1730,6 +1757,7 @@ export function createHeartbeatRunState(db: Db) { resolveSessionBeforeForWakeup, getRun, getRunCleanupState, + getRunExecutionWait, getRunWorkspaceSettlement, ensureRuntimeState, getTaskSession, diff --git a/server/src/services/native-local-process-stop.ts b/server/src/services/native-local-process-stop.ts index 7e74b8eca1c..422d2c24f0f 100644 --- a/server/src/services/native-local-process-stop.ts +++ b/server/src/services/native-local-process-stop.ts @@ -22,7 +22,8 @@ function absent(pid: number): boolean { /** A fresh workspace is not process containment. Bind an OS observation to * the original host, process birth, and launch event before accepting it. */ -export async function recordedLocalExecutionStopped(db: Db, run: typeof heartbeatRuns.$inferSelect) { +export async function recordedLocalExecutionStopped(db: Db, run: Pick) { const [identity] = await db.select({ payload: heartbeatRunEvents.payload }).from(heartbeatRunEvents).where(and( eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id), eq(heartbeatRunEvents.eventType, PROCESS_IDENTITY_RECORDED), isNull(heartbeatRunEvents.sourceEventId), From bd77ffe78d625bb68c430640f4f30231e9cf48e8 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 04:12:14 -0700 Subject: [PATCH 32/68] perf(connections): scope profile rule and binding reads (#17589) Read current company/agent profile bindings and rules matching the exact catalog rows being evaluated. Preserve all five selector kinds, null application matches, exclusions, and the existing access evaluator without a cross-call cache. Verified with full build and typecheck, 652 affected tests across ten suites, all 18 strengthened effective-assignment tests, exclusion mutation and byte counterfactuals, complete exact-head CI, and Greptile 5/5 Medium with no unresolved comments. Co-Authored-By: Paperclip --- doc/DATABASE.md | 7 ++ .../tool-effective-assignments.test.ts | 116 ++++++++++++++++++ .../tool-access/profiles/resolution.ts | 72 ++++++----- 3 files changed, 163 insertions(+), 32 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index a814c50cfad..762ea37f2d9 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -135,6 +135,13 @@ authority once. Different task bindings must both pass. The batch is discarded after that delivery; later events still recheck grants, membership, task existence, and company boundaries. +Connection-assignment checks select profile bindings for the current company or +agent and read rules that can match the selected connection's catalog rows. +All five selector kinds, null application matching, exclusions, and profile +precedence retain their existing evaluator. Rule filtering uses the catalog +values already read by that check, so a later catalog change cannot drop an +exclusion from the decision being evaluated. There is no cross-call cache. + Filesystem preparation uses current task/run and selected-configuration checks. Historical root-reader authorization has been retired; no retained-reader batch is loaded at admission or publication. Workspace-record disclosure still uses task diff --git a/server/src/__tests__/tool-effective-assignments.test.ts b/server/src/__tests__/tool-effective-assignments.test.ts index 13de5533fbf..4b99a23eec2 100644 --- a/server/src/__tests__/tool-effective-assignments.test.ts +++ b/server/src/__tests__/tool-effective-assignments.test.ts @@ -169,6 +169,122 @@ describeDatabase("effective tool assignments", () => { expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([]); }); + it("omits unrelated rules and bindings while retaining current connection decisions", async () => { + const f = await fixture(); + const [otherApplication] = await db.insert(toolApplications).values({ companyId: f.company.id, name: "Other app", type: "mcp_http" }).returning(); + const [otherConnection] = await db.insert(toolConnections).values({ companyId: f.company.id, applicationId: otherApplication.id, + name: "Other connection", uid: randomUUID(), transport: "mcp_remote" }).returning(); + const [otherTool] = await db.insert(toolCatalogEntries).values({ companyId: f.company.id, applicationId: otherApplication.id, + connectionId: otherConnection.id, name: "other", toolName: "other", versionHash: "other", riskLevel: "destructive" }).returning(); + await db.insert(toolProfileEntries).values(Array.from({ length: 200 }, (_, index) => [ + { selectorType: "application" as const, applicationId: otherApplication.id }, + { selectorType: "connection" as const, connectionId: otherConnection.id }, + { selectorType: "catalog_entry" as const, catalogEntryId: otherTool.id }, + { selectorType: "tool_name" as const, toolName: `unrelated_${index}` }, + { selectorType: "risk_level" as const, riskLevel: "destructive" as const }, + ].map(entry => ({ companyId: f.company.id, profileId: f.profile.id, effect: index % 2 ? "include" as const : "exclude" as const, ...entry }))).flat()); + await db.insert(toolProfileBindings).values(Array.from({ length: 200 }, () => ({ companyId: f.company.id, + profileId: f.profile.id, targetType: "agent" as const, targetId: randomUUID() }))); + const [include, exclude] = await db.insert(toolProfileEntries).values([ + { companyId: f.company.id, profileId: f.profile.id, selectorType: "connection", effect: "include", connectionId: f.connection.id }, + { companyId: f.company.id, profileId: f.profile.id, selectorType: "catalog_entry", effect: "exclude", catalogEntryId: f.write.id }, + ]).returning(); + const service = toolAccessService(db), before = await compare(f); + const savedRules = await db.select().from(toolProfileEntries); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { toolProfileEntries, toolProfileBindings }, capture); + try { + const scoped = await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id]); + capture.flush(); + const rules = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" && event.table === "tool_profile_entries"); + const bindings = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" && event.table === "tool_profile_bindings"); + expect(rules).toHaveLength(1); + expect(Number(rules[0].responseValueBytes)).toBeLessThan(4096); + expect(rules[0]).toMatchObject({ calls: 1, rows: 2 }); + expect(bindings).toHaveLength(1); + expect(bindings[0]).toMatchObject({ calls: 1, rows: 1 }); + expect(scoped).toEqual({ allowedConnectionIds: [f.connection.id], installedConnections: before.full.installedConnections }); + expect(await db.select().from(toolProfileEntries)).toEqual(savedRules); + await db.update(toolProfileEntries).set({ effect: "exclude" }).where(eq(toolProfileEntries.id, include.id)); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([]); + await db.delete(toolProfileEntries).where(eq(toolProfileEntries.id, include.id)); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([]); + expect((await db.select().from(toolProfileEntries).where(eq(toolProfileEntries.id, exclude.id)))[0]).toEqual(exclude); + } finally { capture.flush(); } + }); + + it("preserves null application matches and exact special-character tool names", async () => { + const f = await fixture(), toolName = "quoted' path\\tool_🙂"; + await db.update(toolCatalogEntries).set({ status: "removed" }).where(eq(toolCatalogEntries.id, f.write.id)); + await db.update(toolCatalogEntries).set({ applicationId: null, toolName }).where(eq(toolCatalogEntries.id, f.read.id)); + await db.update(toolProfiles).set({ defaultAction: "allow" }).where(eq(toolProfiles.id, f.profile.id)); + const [rule] = await db.insert(toolProfileEntries).values({ companyId: f.company.id, profileId: f.profile.id, + selectorType: "application", applicationId: null, effect: "exclude" }).returning(); + const service = toolAccessService(db); + expect((await compare(f)).assignments.allowedTools).toEqual([]); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([]); + await db.update(toolProfileEntries).set({ selectorType: "tool_name", toolName }).where(eq(toolProfileEntries.id, rule.id)); + expect((await compare(f)).assignments.allowedTools).toEqual([]); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([]); + await db.update(toolProfileEntries).set({ toolName: `${toolName}different` }).where(eq(toolProfileEntries.id, rule.id)); + expect((await compare(f)).assignments.allowedTools.map(tool => tool.id)).toEqual([f.read.id]); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([f.connection.id]); + }); + + it.each(["application", "connection", "catalog_entry", "tool_name", "risk_level"] as const)("denies the connection when a %s exclusion blocks its only active tool", async selectorType => { + const f = await fixture(); + await db.update(toolCatalogEntries).set({ status: "removed" }).where(eq(toolCatalogEntries.id, f.write.id)); + await db.update(toolProfiles).set({ defaultAction: "allow" }).where(eq(toolProfiles.id, f.profile.id)); + const [rule] = await db.insert(toolProfileEntries).values({ companyId: f.company.id, profileId: f.profile.id, selectorType, effect: "exclude", + ...(selectorType === "application" ? { applicationId: f.application.id } : {}), + ...(selectorType === "connection" ? { connectionId: f.connection.id } : {}), + ...(selectorType === "catalog_entry" ? { catalogEntryId: f.read.id } : {}), + ...(selectorType === "tool_name" ? { toolName: f.read.toolName } : {}), + ...(selectorType === "risk_level" ? { riskLevel: "read" as const } : {}), + }).returning(); + const service = toolAccessService(db); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([]); + expect((await compare(f)).assignments.allowedTools).toEqual([]); + await db.delete(toolProfileEntries).where(eq(toolProfileEntries.id, rule.id)); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([f.connection.id]); + }); + + it("filters rules against the catalog rows being evaluated when the catalog changes during the read", async () => { + const f = await fixture(); + await db.update(toolProfiles).set({ defaultAction: "allow" }).where(eq(toolProfiles.id, f.profile.id)); + await db.update(toolCatalogEntries).set({ status: "removed" }).where(eq(toolCatalogEntries.id, f.write.id)); + await db.insert(toolProfileEntries).values({ companyId: f.company.id, profileId: f.profile.id, + selectorType: "tool_name", toolName: f.read.toolName, effect: "exclude" }); + let changed = false; + const changingDb = new Proxy(db, { get(target, key) { + if (key !== "select") return Reflect.get(target, key); + return (...fields: Parameters) => { + const builder = target.select(...fields), from = builder.from.bind(builder); + builder.from = ((...args: Parameters) => { + const selection = from(...args); + if (args[0] === toolCatalogEntries) { + const execute = selection.execute.bind(selection); + selection.execute = async () => { + const rows = await execute(); + if (!changed) { + changed = true; + await db.update(toolCatalogEntries).set({ toolName: "changed-after-read" }).where(eq(toolCatalogEntries.id, f.read.id)); + } + return rows; + }; + } + return selection; + }) as typeof builder.from; + return builder; + }; + } }); + const service = toolAccessService(changingDb); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([]); + expect(changed).toBe(true); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([f.connection.id]); + }); + it("keeps other companies' catalog rows out of scoped assignments", async () => { const f = await fixture(); const other = await fixture(); diff --git a/server/src/services/tool-access/profiles/resolution.ts b/server/src/services/tool-access/profiles/resolution.ts index 3cc4d36df4b..c237636eb2b 100644 --- a/server/src/services/tool-access/profiles/resolution.ts +++ b/server/src/services/tool-access/profiles/resolution.ts @@ -1,4 +1,4 @@ -import { and, asc, eq, inArray } from "drizzle-orm"; +import { and, asc, eq, inArray, or, sql, type AnyColumn } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { agents, toolCatalogEntries, toolProfileBindings, toolProfileEntries, toolProfiles } from "@paperclipai/db"; import type { ToolCatalogEntry, ToolConnection, ToolProfileEffectiveSummary, ToolProfileWithDetails } from "@paperclipai/shared"; @@ -10,6 +10,12 @@ type ProfileAssignmentDependencies = { toCatalogEntry(row: typeof toolCatalogEntries.$inferSelect): ToolCatalogEntry; }; +function catalogValuesContain(column: AnyColumn, values: Array) { + const distinct = [...new Set(values)]; + return sql`(${JSON.stringify(distinct.filter(value => value !== null))}::jsonb ? ${column}::text + or (${distinct.includes(null)} and ${column} is null))`; +} + function resolveCatalogAccess( profiles: Array>, entries: Array, @@ -69,20 +75,17 @@ export function createToolProfileResolver(db: Db, { assertOptionalAgent, resolve agentId, "Tool profile effective agent", ); - const allBindings = await db + const matchingBindings = await db .select() .from(toolProfileBindings) - .where(eq(toolProfileBindings.companyId, companyId)) + .where(and(eq(toolProfileBindings.companyId, companyId), or( + and(eq(toolProfileBindings.targetType, "company"), eq(toolProfileBindings.targetId, companyId)), + and(eq(toolProfileBindings.targetType, "agent"), eq(toolProfileBindings.targetId, agentId)), + ))) .orderBy( asc(toolProfileBindings.priority), asc(toolProfileBindings.createdAt), ); - const matchingBindings = allBindings.filter( - (binding) => - (binding.targetType === "company" && - binding.targetId === companyId) || - (binding.targetType === "agent" && binding.targetId === agentId), - ); const candidateProfileIds = profileIdsInBindingOrder(matchingBindings); if (candidateProfileIds.length === 0) return { bindings: [], activeProfiles: [] }; const candidateProfiles = await db @@ -194,29 +197,34 @@ export function createToolProfileResolver(db: Db, { assertOptionalAgent, resolve if (activeProfiles.length === 0 || connectionIds.length === 0) { return { allowedConnectionIds: [], installedConnections: await resolveInstalledConnectionsForAgent(companyId, agentId) }; } - const [entries, catalog] = await Promise.all([ - db.select({ - profileId: toolProfileEntries.profileId, - effect: toolProfileEntries.effect, - selectorType: toolProfileEntries.selectorType, - applicationId: toolProfileEntries.applicationId, - connectionId: toolProfileEntries.connectionId, - catalogEntryId: toolProfileEntries.catalogEntryId, - toolName: toolProfileEntries.toolName, - riskLevel: toolProfileEntries.riskLevel, - }).from(toolProfileEntries).where(and( - eq(toolProfileEntries.companyId, companyId), inArray(toolProfileEntries.profileId, activeProfiles.map(profile => profile.id)), - )), - db.select({ - id: toolCatalogEntries.id, - applicationId: toolCatalogEntries.applicationId, - connectionId: toolCatalogEntries.connectionId, - toolName: toolCatalogEntries.toolName, - riskLevel: toolCatalogEntries.riskLevel, - }).from(toolCatalogEntries).where(and( - eq(toolCatalogEntries.companyId, companyId), eq(toolCatalogEntries.status, "active"), inArray(toolCatalogEntries.connectionId, connectionIds), - )), - ]); + const catalog = await db.select({ + id: toolCatalogEntries.id, + applicationId: toolCatalogEntries.applicationId, + connectionId: toolCatalogEntries.connectionId, + toolName: toolCatalogEntries.toolName, + riskLevel: toolCatalogEntries.riskLevel, + }).from(toolCatalogEntries).where(and( + eq(toolCatalogEntries.companyId, companyId), eq(toolCatalogEntries.status, "active"), inArray(toolCatalogEntries.connectionId, connectionIds), + )); + const entries = catalog.length === 0 ? [] : await db.select({ + profileId: toolProfileEntries.profileId, + effect: toolProfileEntries.effect, + selectorType: toolProfileEntries.selectorType, + applicationId: toolProfileEntries.applicationId, + connectionId: toolProfileEntries.connectionId, + catalogEntryId: toolProfileEntries.catalogEntryId, + toolName: toolProfileEntries.toolName, + riskLevel: toolProfileEntries.riskLevel, + }).from(toolProfileEntries).where(and( + eq(toolProfileEntries.companyId, companyId), inArray(toolProfileEntries.profileId, activeProfiles.map(profile => profile.id)), + or( + and(eq(toolProfileEntries.selectorType, "application"), catalogValuesContain(toolProfileEntries.applicationId, catalog.map(entry => entry.applicationId))), + and(eq(toolProfileEntries.selectorType, "connection"), catalogValuesContain(toolProfileEntries.connectionId, catalog.map(entry => entry.connectionId))), + and(eq(toolProfileEntries.selectorType, "catalog_entry"), catalogValuesContain(toolProfileEntries.catalogEntryId, catalog.map(entry => entry.id))), + and(eq(toolProfileEntries.selectorType, "tool_name"), catalogValuesContain(toolProfileEntries.toolName, catalog.map(entry => entry.toolName))), + and(eq(toolProfileEntries.selectorType, "risk_level"), catalogValuesContain(toolProfileEntries.riskLevel, catalog.map(entry => entry.riskLevel))), + ), + )); const { allowedCatalogIds } = resolveCatalogAccess(activeProfiles, entries, catalog); return { allowedConnectionIds: [...new Set(catalog.filter(entry => allowedCatalogIds.has(entry.id)).map(entry => entry.connectionId))], From 2e47c42fad91df8c040b33d3bda301f4a35264cf Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 04:21:50 -0700 Subject: [PATCH 33/68] perf(runner): omit unused launch context from tool authority reads (#17591) Omit nine unused named launch values from the unlocked Native authority query. Preserve complete wake payloads, other context, result receipts, stored input, and locked execution and mutation readers. Verified with full build and typecheck, 2,359 affected tests across 20 suites, real PostgreSQL byte/retention/cancellation checks, a counterfactual, complete exact-head CI, and Greptile 5/5 Medium with no unresolved comments. An existing cleanup scheduling test passed on CI retry after its suite passed locally with both base and branch source. Co-Authored-By: Paperclip --- doc/DATABASE.md | 5 ++- .../paperclip-runner-tool-authority.test.ts | 36 +++++++++++++++++++ .../paperclip-runner-tool-authority.ts | 6 +++- 3 files changed, 45 insertions(+), 2 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 762ea37f2d9..4f16a6e4d87 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -127,7 +127,10 @@ Native tool authority lookups select current run identity, status, invocation, responsible user, context, and receipt data. Saved profiles and output excerpts stay in storage and are omitted from that read. Task and agent binding, current status, budget, and task privacy checks retain their existing predicates. -Mutation transactions still lock and read complete run records. +Mutation transactions still lock and read complete run records. The unlocked +authority read also omits named launch renderings and preparation data that its +tools do not consume. Full wake payloads, other context, and result receipts +remain available; the complete saved context stays in storage for execution. WebSocket delivery uses one fresh authorization batch for an event's task and run bindings. When both bindings name the same task, the batch reads its current diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts index 76815a76482..2558b813555 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts @@ -244,6 +244,42 @@ describe("PaperclipRunnerToolAuthority", () => { } }); + it("reads tool authority without transferring unused launch renderings", async () => { + const binding = { companyId, agentId, issueId, runId }; + const authority = new PaperclipRunnerToolAuthority(db, binding); + const contextCall = { tool: "get_task_context", callId: "launch-context-read", arguments: {} }; + const apiCall = { tool: "search_api", callId: "launch-api-read", arguments: { query: "read task" } }; + const expectedContext = await authority.execute(contextCall), expectedApi = await authority.execute(apiCall); + const [original] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const payload = "retained-launch-rendering🙂".repeat(8192); + const context = { ...original.contextSnapshot, executionContinuation: { savedInput: payload }, paperclipTaskMarkdown: payload, + paperclipTaskMarkdownCompact: payload, paperclipTaskMarkdownAssignment: payload, paperclipTaskMarkdownAssignmentCompact: payload, + paperclipWorkspace: { savedInput: payload }, paperclipEnvironment: { savedInput: payload }, + paperclipContinuationSummary: payload, paperclipWakeComment: payload }; + await db.update(heartbeatRuns).set({ contextSnapshot: context }).where(eq(heartbeatRuns.id, runId)); + const [expanded] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const observed = createDb(temporary!.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns, issues, agents, companies }, capture); + try { + const reader = new PaperclipRunnerToolAuthority(observed, binding); + const actualContext = await reader.execute(contextCall), actualApi = await reader.execute(apiCall); + capture.flush(); + const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(16 * 1024); + expect(actualContext).toEqual(expectedContext); + expect(actualApi).toEqual(expectedApi); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]).toEqual(expanded); + await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, runId)); + await expect(reader.execute(contextCall)).rejects.toThrow("paperclip_runner_tool_binding_not_authorized"); + await expect(reader.execute(apiCall)).rejects.toThrow("paperclip_runner_tool_binding_not_authorized"); + } finally { + capture.flush(); await observed.$client.end(); + await db.update(heartbeatRuns).set({ contextSnapshot: original.contextSnapshot, status: original.status }).where(eq(heartbeatRuns.id, runId)); + } + }); + it("preserves platform receipt provenance in both mixed task history readers", async () => { const at = new Date("2026-10-10T12:00:00Z"); const commentIds = Array.from({ length: 4 }, () => randomUUID()); diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index 40b140f01b2..e257744e545 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -1235,7 +1235,11 @@ export class PaperclipRunnerToolAuthority { status: heartbeatRuns.status, invocationSource: heartbeatRuns.invocationSource, responsibleUserId: heartbeatRuns.responsibleUserId, - contextSnapshot: heartbeatRuns.contextSnapshot, + contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then ${heartbeatRuns.contextSnapshot} - array['executionContinuation', 'paperclipTaskMarkdown', + 'paperclipTaskMarkdownCompact', 'paperclipTaskMarkdownAssignment', 'paperclipTaskMarkdownAssignmentCompact', + 'paperclipWorkspace', 'paperclipEnvironment', 'paperclipContinuationSummary', 'paperclipWakeComment'] + else ${heartbeatRuns.contextSnapshot} end`, resultJson: heartbeatRuns.resultJson, } }) .from(heartbeatRuns) From 8c6bd5a83bf9d9b5af765be9bbcf60e400889b64 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 05:36:43 -0700 Subject: [PATCH 34/68] perf(runner): skip wake payloads in ordinary authority reads (#17600) Omit saved wake payloads from ordinary unlocked Native authority reads. Load the current full wake for mail restrictions. Preserve stored rows, locked mutation reads, and current authorization predicates. Verified with full build and typecheck, 2,360 affected tests, real PostgreSQL byte and response checks, counterfactual performance and mail authorization failures, complete exact-head CI, and Greptile 5/5 Medium with no unresolved comments. Co-Authored-By: Paperclip --- doc/DATABASE.md | 6 ++- .../paperclip-runner-tool-authority.test.ts | 47 +++++++++++++++++-- .../paperclip-runner-tool-authority.ts | 5 +- 3 files changed, 50 insertions(+), 8 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 4f16a6e4d87..9e432cbfcc0 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -129,8 +129,10 @@ stay in storage and are omitted from that read. Task and agent binding, current status, budget, and task privacy checks retain their existing predicates. Mutation transactions still lock and read complete run records. The unlocked authority read also omits named launch renderings and preparation data that its -tools do not consume. Full wake payloads, other context, and result receipts -remain available; the complete saved context stays in storage for execution. +tools do not consume. Ordinary tool checks also omit the wake payload. Mail +checks load the current full wake to enforce external-chat restrictions. Other +context and result receipts remain available. The complete saved context stays +in storage for execution. WebSocket delivery uses one fresh authorization batch for an event's task and run bindings. When both bindings name the same task, the batch reads its current diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts index 2558b813555..74524c733f3 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts @@ -1,4 +1,4 @@ -import { connectorToolDefinitionsForSkills } from "../connector-runtime.js"; +import * as connectorRuntime from "../connector-runtime.js"; import * as cloudIdentity from "../cloud-runtime-identity.js"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { randomUUID } from "node:crypto"; @@ -101,7 +101,7 @@ describe("PaperclipRunnerToolAuthority", () => { }); it("registers pinned connector definitions without discovery and rejects revoked assignments at use", async () => { - const connectorAssignments = connectorToolDefinitionsForSkills(new Set([ + const connectorAssignments = connectorRuntime.connectorToolDefinitionsForSkills(new Set([ "paperclipai/paperclip/slack", "paperclipai/paperclip/browser-use-cloud", "custom/agentmail", ])); expect(connectorAssignments).toHaveLength(1); @@ -244,7 +244,7 @@ describe("PaperclipRunnerToolAuthority", () => { } }); - it("reads tool authority without transferring unused launch renderings", async () => { + it("reads tool authority without transferring unused launch renderings or wake payloads", async () => { const binding = { companyId, agentId, issueId, runId }; const authority = new PaperclipRunnerToolAuthority(db, binding); const contextCall = { tool: "get_task_context", callId: "launch-context-read", arguments: {} }; @@ -255,7 +255,7 @@ describe("PaperclipRunnerToolAuthority", () => { const context = { ...original.contextSnapshot, executionContinuation: { savedInput: payload }, paperclipTaskMarkdown: payload, paperclipTaskMarkdownCompact: payload, paperclipTaskMarkdownAssignment: payload, paperclipTaskMarkdownAssignmentCompact: payload, paperclipWorkspace: { savedInput: payload }, paperclipEnvironment: { savedInput: payload }, - paperclipContinuationSummary: payload, paperclipWakeComment: payload }; + paperclipContinuationSummary: payload, paperclipWakeComment: payload, paperclipWake: { retainedPayload: payload } }; await db.update(heartbeatRuns).set({ contextSnapshot: context }).where(eq(heartbeatRuns.id, runId)); const [expanded] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); const observed = createDb(temporary!.connectionString), events: Array> = [], now = Date.now(); @@ -280,6 +280,45 @@ describe("PaperclipRunnerToolAuthority", () => { } }); + it("checks the current full wake before mail actions while ordinary tools omit its payload", async () => { + const connectorAssignments = connectorRuntime.connectorToolDefinitionsForSkills(new Set(["paperclipai/paperclip/agentmail"])); + const name = connectorAssignments.flatMap(assignment => assignment.tools).find(tool => tool.name.startsWith("agentmail_"))!.name; + const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId, connectorAssignments }); + const [original] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const dispatch = vi.spyOn(connectorRuntime, "executeConnectorTool").mockResolvedValue({ outcome: "mail-dispatched" }); + const wake = { + reason: "External chat message received", externalChatProvider: "github", checkedOutByHarness: true, + issue: { id: issueId, identifier: "RNT-1", title: "Conversation", description: "Task", descriptionTruncated: false, + status: "in_progress", workMode: "standard" }, + commentWindow: { requestedCount: 1, includedCount: 1, missingCount: 0 }, commentIds: ["wake-comment"], + latestCommentId: "wake-comment", comments: [{ id: "wake-comment", issueId, body: "Reply here", bodyTruncated: false, authorType: "user" }], + fallbackFetchNeeded: false, + }; + const call = { tool: name, callId: "current-mail-wake", arguments: {} }; + try { + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId, paperclipWake: wake } }).where(eq(heartbeatRuns.id, runId)); + await expect(authority.execute(call)).rejects.toThrow("Restricted chat runs cannot use email actions"); + expect(dispatch).not.toHaveBeenCalled(); + await expect(authority.execute({ tool: "get_task_context", callId: "chat-task-context", arguments: {} })) + .resolves.toMatchObject({ activeTask: { id: issueId } }); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId } }).where(eq(heartbeatRuns.id, runId)); + await expect(authority.execute(call)).resolves.toEqual({ outcome: "mail-dispatched" }); + expect(dispatch).toHaveBeenCalledTimes(1); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId, paperclipWake: { ...wake, checkedOutByHarness: false, + externalChatExecutionBound: true, issue: { ...wake.issue, status: "in_review" } } } }).where(eq(heartbeatRuns.id, runId)); + await expect(authority.execute(call)).rejects.toThrow("Restricted chat runs cannot use email actions"); + expect(dispatch).toHaveBeenCalledTimes(1); + for (const context of [{ issueId, source: "chat:github" }, { issueId, paperclipExternalChatQuestionResponse: true }]) { + await db.update(heartbeatRuns).set({ contextSnapshot: context }).where(eq(heartbeatRuns.id, runId)); + await expect(authority.execute(call)).rejects.toThrow("Restricted chat runs cannot use email actions"); + expect(dispatch).toHaveBeenCalledTimes(1); + } + } finally { + dispatch.mockRestore(); + await db.update(heartbeatRuns).set({ contextSnapshot: original.contextSnapshot }).where(eq(heartbeatRuns.id, runId)); + } + }); + it("preserves platform receipt provenance in both mixed task history readers", async () => { const at = new Date("2026-10-10T12:00:00Z"); const commentIds = Array.from({ length: 4 }, () => randomUUID()); diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index e257744e545..31f65f12fcd 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -449,7 +449,7 @@ export class PaperclipRunnerToolAuthority { } if (isConnectorTool(call.tool)) { if (!(this.binding.connectorAssignments ?? []).some((assignment) => assignment.tools.some((tool) => tool.name === call.tool))) throw forbidden("Connector tool is not available to this run"); - const { run } = await this.#boundContext(); + const { run } = await this.#boundContext({ includeChatWake: call.tool.startsWith("agentmail_") }); const snapshot = record(run.contextSnapshot); if (call.tool.startsWith("agentmail_") && (isPaperclipExternalChatContractTurn(snapshot.paperclipWake) || String(snapshot.source ?? "").startsWith("chat:") || snapshot.paperclipExternalChatQuestionResponse)) throw forbidden("Restricted chat runs cannot use email actions"); return executeConnectorTool(this.db, this.binding, call.tool, call.arguments); @@ -1225,7 +1225,7 @@ export class PaperclipRunnerToolAuthority { ...(this.binding.museRuntime ? { authorizingUserId: this.binding.externalAuthorizingUserId } : {}) }; } - async #boundContext() { + async #boundContext(options: { includeChatWake?: boolean } = {}) { if (this.binding.museRuntime && !this.binding.externalAuthorizingUserId) throw forbidden("Muse authorizer is unavailable"); const [row] = await this.db.select({ issue: issues, actor: agents, run: { id: heartbeatRuns.id, @@ -1239,6 +1239,7 @@ export class PaperclipRunnerToolAuthority { then ${heartbeatRuns.contextSnapshot} - array['executionContinuation', 'paperclipTaskMarkdown', 'paperclipTaskMarkdownCompact', 'paperclipTaskMarkdownAssignment', 'paperclipTaskMarkdownAssignmentCompact', 'paperclipWorkspace', 'paperclipEnvironment', 'paperclipContinuationSummary', 'paperclipWakeComment'] + ${options.includeChatWake ? sql`` : sql`- 'paperclipWake'`} else ${heartbeatRuns.contextSnapshot} end`, resultJson: heartbeatRuns.resultJson, } }) From 6aa711f6c67101252961c1715baafb1bbd88e62f Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 06:04:09 -0700 Subject: [PATCH 35/68] fix(ui): avoid broad task refreshes after known run progress (#17602) Use the existing live-status cache patch result to skip redundant task and history reads. Keep unknown-run discovery, lifecycle, activity, visibility, and reconnect recovery. Verified with full build and typecheck, 9,047 UI tests, final context tests and counterfactual broad-refetch and missing-fallback failures, complete exact-head CI, and Greptile 5/5 Medium with no unresolved comments. Co-Authored-By: Paperclip --- doc/DATABASE.md | 5 ++ ui/src/context/LiveUpdatesProvider.test.ts | 63 ++++++++++++++++++++++ ui/src/context/LiveUpdatesProvider.tsx | 19 +++---- 3 files changed, 78 insertions(+), 9 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 9e432cbfcc0..10e7bc7a1e6 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -140,6 +140,11 @@ authority once. Different task bindings must both pass. The batch is discarded after that delivery; later events still recheck grants, membership, task existence, and company boundaries. +Browser progress events update cached live run status directly. A successful +patch does not refetch task details, activity, or run history. Unknown runs +retain the existing refresh fallback. Run transitions, activity events, and +reconnect recovery still refresh their affected data. + Connection-assignment checks select profile bindings for the current company or agent and read rules that can match the selected connection's catalog rows. All five selector kinds, null application matching, exclusions, and profile diff --git a/ui/src/context/LiveUpdatesProvider.test.ts b/ui/src/context/LiveUpdatesProvider.test.ts index f4f09a518b6..3fc05666db8 100644 --- a/ui/src/context/LiveUpdatesProvider.test.ts +++ b/ui/src/context/LiveUpdatesProvider.test.ts @@ -268,6 +268,69 @@ describe("LiveUpdatesProvider issue invalidation", () => { }); }); + it("patches progress without refetching visible task history but refreshes actual activity and terminal runs", async () => { + vi.stubGlobal("document", { visibilityState: "visible" }); + const client = new QueryClient({ defaultOptions: { queries: { retry: false, staleTime: Infinity } } }); + const issue = { id: "issue-1", identifier: "DEMO-1", companyId: "company-1", assigneeAgentId: "agent-1", executionRunId: "run-1" }; + const run = { id: "run-1", agentId: "agent-1", status: "running" }; + const snapshots = [ + [queryKeys.issues.detail("DEMO-1"), issue], + [queryKeys.issues.activity("DEMO-1"), []], + [queryKeys.issues.runs("DEMO-1"), [{ runId: "run-1" }]], + [queryKeys.issues.liveRuns("DEMO-1"), [run]], + [queryKeys.issues.activeRun("DEMO-1"), run], + [queryKeys.issues.comments("DEMO-1"), { pages: [[]], pageParams: [null] }], + ] as const; + const readers = snapshots.map(() => vi.fn(async () => undefined as unknown)); + const observers = snapshots.map(([queryKey, data], index) => { + client.setQueryData(queryKey, data); + readers[index].mockResolvedValue(data); + return new QueryObserver(client, { queryKey, queryFn: readers[index], staleTime: Infinity }); + }); + const unsubscribe = observers.map(observer => observer.subscribe(() => {})); + const gate = { cooldownHits: new Map(), suppressUntil: 0, observedRunOutcomes: new Set() }; + const deliver = (event: Parameters[3]) => + __liveUpdatesTestUtils.handleLiveEvent(client, "company-1", "/DEMO/issues/DEMO-1", event, () => null, gate, + { userId: "viewer", agentId: null }); + try { + for (let id = 1; id <= 3; id++) { + deliver({ id, companyId: "company-1", type: "heartbeat.run.progress", createdAt: "2020-01-01T00:00:00Z", + payload: { runId: "run-1", issueId: "issue-1", agentId: "agent-1", phase: "working", message: `Step ${id}`, + currentToolName: "read_task", updatedAt: "2020-01-01T00:00:00Z" } }); + } + await Promise.resolve(); + expect(readers.map(reader => reader.mock.calls.length)).toEqual([0, 0, 0, 0, 0, 0]); + expect(client.getQueryData(queryKeys.issues.activeRun("DEMO-1"))).toMatchObject({ currentStatusMessage: "Step 3", currentToolName: "read_task" }); + expect(client.getQueryData(queryKeys.issues.liveRuns("DEMO-1"))).toMatchObject([{ currentStatusMessage: "Step 3" }]); + deliver({ id: 4, companyId: "company-1", type: "heartbeat.run.progress", createdAt: "2020-01-01T00:00:01Z", + payload: { runId: "unknown-run", issueId: "issue-1", agentId: "agent-1", message: "New run" } }); + await vi.waitFor(() => { + expect(readers.slice(0, 5).every(reader => reader.mock.calls.length > 0)).toBe(true); + expect(client.isFetching()).toBe(0); + }); + readers.forEach(reader => reader.mockClear()); + deliver({ id: 5, companyId: "company-1", type: "activity.logged", createdAt: "2020-01-01T00:00:02Z", + payload: { entityType: "issue", entityId: "issue-1", action: "issue.comment_added", actorType: "user", actorId: "another-viewer" } }); + await vi.waitFor(() => { + expect(readers[1]).toHaveBeenCalled(); + expect(readers[5]).toHaveBeenCalled(); + expect(client.isFetching()).toBe(0); + }); + readers.forEach(reader => reader.mockClear()); + deliver({ id: 6, companyId: "company-1", type: "heartbeat.run.status", createdAt: "2020-01-01T00:00:03Z", + payload: { runId: "run-1", issueId: "issue-1", agentId: "agent-1", status: "succeeded" } }); + await vi.waitFor(() => { + expect(readers.every(reader => reader.mock.calls.length > 0)).toBe(true); + expect(client.isFetching()).toBe(0); + }); + } finally { + unsubscribe.forEach(stop => stop()); + observers.forEach(observer => observer.destroy()); + client.clear(); + vi.unstubAllGlobals(); + } + }); + it("applies heartbeat progress payloads directly to cached visible issue runs", () => { const cache = new Map([ [JSON.stringify(queryKeys.liveRuns("company-1")), [{ id: "run-1", currentStatusMessage: null }]], diff --git a/ui/src/context/LiveUpdatesProvider.tsx b/ui/src/context/LiveUpdatesProvider.tsx index 48e318d5b0e..5f634a8cdde 100644 --- a/ui/src/context/LiveUpdatesProvider.tsx +++ b/ui/src/context/LiveUpdatesProvider.tsx @@ -1791,14 +1791,14 @@ function handleLiveEvent( event.createdAt, event.type, ); - if (liveStatusPatch) { - applyRunLiveStatusPatchToCaches( - queryClient, - expectedCompanyId, - pathname, - liveStatusPatch, - ); - } + const liveStatusPatched = liveStatusPatch + ? applyRunLiveStatusPatchToCaches( + queryClient, + expectedCompanyId, + pathname, + liveStatusPatch, + ) + : false; if (event.type === "heartbeat.run.log") { return; } @@ -1831,7 +1831,7 @@ function handleLiveEvent( if (event.type === "heartbeat.run.progress") { invalidateHeartbeatProgressQueries(queryClient, expectedCompanyId, payload); - invalidateVisibleIssueRunQueries(queryClient, pathname, payload); + if (!liveStatusPatched) invalidateVisibleIssueRunQueries(queryClient, pathname, payload); return; } @@ -1942,6 +1942,7 @@ export const __liveUpdatesTestUtils = { buildRunStatusToast, closeSocketQuietly, dispatchLiveEventToSubscribers, + handleLiveEvent, LiveEventSubscriptionContext, applyRunLiveStatusPatchToCaches, hydrateVisibleIssueComment, From 3536b3649f98155d92b45cfb5dc3abcc2fea97d6 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 06:30:25 -0700 Subject: [PATCH 36/68] Read only task creation provenance fields (#17604) Narrow saved-run provenance reads to current identity, scope, and typed legacy bindings. Preserve binding precedence, current viewer checks, and complete stored runs. Real PostgreSQL and route tests pass 23 cases; legacy full-context and type-coercion regressions are caught. Exact-head required CI is green; Greptile is 5/5 with Low impact and no unresolved comments. Co-Authored-By: Paperclip --- doc/DATABASE.md | 5 + .../issue-creation-source-read.test.ts | 109 ++++++++++++++++++ server/src/services/issue-creation-source.ts | 24 +++- 3 files changed, 135 insertions(+), 3 deletions(-) create mode 100644 server/src/__tests__/issue-creation-source-read.test.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 10e7bc7a1e6..c4d5c72e41e 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -162,6 +162,11 @@ startup and cancellation receipts, task binding, and the recorded adapter kind. Saved profiles, output excerpts, and unrelated launch context stay in storage. Continuation admission and its locked mutation reads retain their existing gates. +Task creation provenance reads current run identity, scope, and task binding. +Legacy task references retain their typed values and existing precedence. Saved +profiles, output, and unrelated context stay in storage. Viewer access to the +source task and run is still checked before provenance is returned. + The initial agent JWT run lookup reads only the cancellation state and conversation flag from saved run JSON. It does not transfer the full context or result. Run status, responsible-user identity, company and agent boundaries, diff --git a/server/src/__tests__/issue-creation-source-read.test.ts b/server/src/__tests__/issue-creation-source-read.test.ts new file mode 100644 index 00000000000..2b47f3af580 --- /dev/null +++ b/server/src/__tests__/issue-creation-source-read.test.ts @@ -0,0 +1,109 @@ +import { randomUUID } from "node:crypto"; +import { activityLog, agents, companies, createDb, heartbeatRuns, issues } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveIssueCreationSource, runBoundIssueReference } from "../services/issue-creation-source.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("creation source run projections", () => { + const companyId = randomUUID(), otherCompanyId = randomUUID(), agentId = randomUUID(); + const sourceId = randomUUID(), secondId = randomUUID(), foreignId = randomUUID(); + let database: Awaited>; + let db: ReturnType; + let childNumber = 100; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("creation-source-read-"); + db = createDb(database.connectionString); + await db.insert(companies).values([ + { id: companyId, name: "Creation source fixture", issuePrefix: "CSR" }, + { id: otherCompanyId, name: "Foreign source fixture", issuePrefix: "FSR" }, + ]); + await db.insert(agents).values({ id: agentId, companyId, name: "Source agent", adapterType: "process" }); + await db.insert(issues).values([ + { id: sourceId, companyId, title: "Source task", identifier: "CSR-1", issueNumber: 1 }, + { id: secondId, companyId, title: "Second source task", identifier: "CSR-2", issueNumber: 2 }, + { id: foreignId, companyId: otherCompanyId, title: "Foreign task", identifier: "FSR-1", issueNumber: 1 }, + ]); + }, 60_000); + + afterAll(async () => { + await db?.$client.end(); + await database?.cleanup(); + }); + + async function fixture(values: Partial) { + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded", ...values }).returning(); + const number = childNumber++; + const [child] = await db.insert(issues).values({ companyId, title: "Created task", identifier: `CSR-${number}`, + issueNumber: number, originRunId: run.id }).returning(); + return { run, child }; + } + + it.each(["scoped", "legacy task key", "legacy nested identifier"])("omits saved run payloads for %s bindings while preserving provenance and storage", async (binding) => { + const bulk = "retained-source-context🙂".repeat(32_768); + const contextSnapshot = binding === "scoped" + ? { issueId: sourceId, paperclipIssue: { id: sourceId, retainedDescription: bulk }, retainedLaunchInput: bulk } + : binding === "legacy task key" + ? { taskKey: "CSR-1", paperclipIssue: { retainedDescription: bulk }, retainedLaunchInput: bulk } + : { paperclipIssue: { id: "CSR-1", retainedDescription: bulk }, retainedLaunchInput: bulk }; + const { run, child } = await fixture({ contextSnapshot, + runnerProfileJson: { retainedInput: bulk }, stdoutExcerpt: bulk, stderrExcerpt: bulk, resultJson: { retainedReceipt: bulk } }); + expect(run.issueId).toBe(binding === "scoped" ? sourceId : null); + expect(run.nativeIssueId).toBeNull(); + const observed = createDb(database.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns, issues }, capture); + try { + const result = await resolveIssueCreationSource(observed, child); + capture.flush(); + const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select"); + expect(reads.reduce((count, event) => count + Number(event.calls), 0)).toBe(1); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(8192); + expect(result).toMatchObject({ run: { id: run.id, companyId, agentId }, sourceIssue: { id: sourceId } }); + expect(runBoundIssueReference(result!.run)).toBe(binding === "scoped" ? sourceId : "CSR-1"); + for (const key of ["runnerProfileJson", "stdoutExcerpt", "stderrExcerpt", "resultJson"]) expect(result!.run).not.toHaveProperty(key); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toEqual(run); + expect(JSON.stringify(events)).not.toContain("retained-source-context"); + } finally { + capture.flush(); + await observed.$client.end(); + } + }); + + it.each([ + { label: "native identity", nativeIssueId: sourceId, issueId: secondId, contextSnapshot: { issueId: secondId }, expected: sourceId }, + { label: "issue scope", issueId: sourceId, contextSnapshot: { issueId: secondId }, expected: sourceId }, + { label: "legacy issue id", contextSnapshot: { issueId: sourceId, taskId: secondId }, expected: sourceId }, + { label: "typed task id fallback", contextSnapshot: { issueId: 42, taskId: sourceId }, expected: sourceId }, + { label: "nested issue identifier", contextSnapshot: { issueId: " ", paperclipIssue: { id: "CSR-1", ignored: "payload" }, taskKey: "CSR-2" }, expected: sourceId }, + { label: "typed nested id fallback", contextSnapshot: { paperclipIssue: { id: 42 }, taskKey: "CSR-1" }, expected: sourceId }, + { label: "foreign context", contextSnapshot: { taskId: foreignId }, expected: null }, + { label: "null context", contextSnapshot: null, expected: null }, + ])("preserves binding precedence for $label", async ({ label: _label, expected, ...values }) => { + const { child } = await fixture(values); + expect((await resolveIssueCreationSource(db, child))?.sourceIssue.id ?? null).toBe(expected); + }); + + it.each([42, "scalar", ["issueId", sourceId]])("does not invent a task binding from malformed context %j", async (context) => { + const { child } = await fixture({ contextSnapshot: context as unknown as Record }); + expect(await resolveIssueCreationSource(db, child)).toBeNull(); + }); + + it("preserves historical activity lookup, current bindings, and missing or foreign source denial", async () => { + const { run, child } = await fixture({ contextSnapshot: { taskKey: "CSR-1" } }); + await db.update(issues).set({ originRunId: null }).where(eq(issues.id, child.id)); + await db.insert(activityLog).values({ companyId, actorType: "system", actorId: "fixture", action: "issue.created", + entityType: "issue", entityId: child.id, runId: run.id }); + expect((await resolveIssueCreationSource(db, { ...child, originRunId: null }))?.sourceIssue.id).toBe(sourceId); + await db.update(heartbeatRuns).set({ contextSnapshot: { taskKey: "CSR-2" } }).where(eq(heartbeatRuns.id, run.id)); + expect((await resolveIssueCreationSource(db, child))?.sourceIssue.id).toBe(secondId); + expect(await resolveIssueCreationSource(db, { ...child, companyId: otherCompanyId })).toBeNull(); + expect(await resolveIssueCreationSource(db, { ...child, originRunId: randomUUID() })).toBeNull(); + await db.update(heartbeatRuns).set({ contextSnapshot: { taskId: child.id } }).where(eq(heartbeatRuns.id, run.id)); + expect(await resolveIssueCreationSource(db, child)).toBeNull(); + }); +}); diff --git a/server/src/services/issue-creation-source.ts b/server/src/services/issue-creation-source.ts index 42f5ce816a7..c40c898fd39 100644 --- a/server/src/services/issue-creation-source.ts +++ b/server/src/services/issue-creation-source.ts @@ -1,14 +1,17 @@ -import { and, asc, eq, inArray, isNotNull } from "drizzle-orm"; +import { and, asc, eq, inArray, isNotNull, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { activityLog, agents, heartbeatRuns, issues } from "@paperclipai/db"; import { isUuidLike } from "@paperclipai/shared"; type IssueRow = typeof issues.$inferSelect; type HeartbeatRunRow = typeof heartbeatRuns.$inferSelect; +type CreationSourceRun = Pick & { + contextSnapshot: { issueId: unknown; taskId: unknown; taskKey: unknown; paperclipIssue: { id: unknown } } | null; +}; /** The task a run was executing when it created another task. */ export interface IssueCreationSourceRecord { - run: HeartbeatRunRow; + run: CreationSourceRun; sourceIssue: IssueRow; } @@ -86,7 +89,22 @@ export async function resolveIssueCreationSource( } if (!runId || !isUuidLike(runId)) return null; const [run] = await db - .select() + .select({ + id: heartbeatRuns.id, + companyId: heartbeatRuns.companyId, + agentId: heartbeatRuns.agentId, + scopeKind: heartbeatRuns.scopeKind, + nativeIssueId: heartbeatRuns.nativeIssueId, + issueId: heartbeatRuns.issueId, + contextSnapshot: sql`case + when coalesce(${heartbeatRuns.nativeIssueId}, ${heartbeatRuns.issueId}) is not null then null + else jsonb_build_object( + 'issueId', ${heartbeatRuns.contextSnapshot}->'issueId', + 'taskId', ${heartbeatRuns.contextSnapshot}->'taskId', + 'taskKey', ${heartbeatRuns.contextSnapshot}->'taskKey', + 'paperclipIssue', jsonb_build_object('id', ${heartbeatRuns.contextSnapshot}->'paperclipIssue'->'id') + ) end`, + }) .from(heartbeatRuns) .where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, issue.companyId))) .limit(1); From aec86c794b207fd7796d9e9281a0f9ffc3e60f72 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 25 Oct 2026 09:01:24 -0500 Subject: [PATCH 37/68] fix(runner): preserve native work through controller outages (#17594) Keep remote native work under its authenticated connection lease through controller outages. Hold durable ACPX tools through reconnection, align the native Claude HTTP MCP client budget, and wait for the recovered handler before dispatch. Preserve Stop, cancellation, exact identity and artifact checks. Verified with real-process regressions and a real Claude Opus 5.5 Daytona workflow: 125 seconds offline during useful work, original process and lease retained, no failed native tool calls, queued follow-up completed, independently verified ZIP and committed workspace receipts. All current-head CI gates pass and Greptile is 5/5 with all findings resolved. Co-Authored-By: Paperclip --- .dockerignore | 1 + doc/evals.md | 6 + doc/execution-semantics.md | 24 ++++ packages/paperclip-runner/acpx-profiles.json | 8 +- .../runner-core/src/acpx_provider_backend.rs | 3 +- .../src/generated_acpx_profiles.rs | 4 +- .../tests/native_provider_backend.rs | 2 +- .../scripts/copilot-profile-identity.mjs | 4 +- .../src/cli/acpx-runtime-sidecar.test.ts | 53 +++++++++ .../src/cli/acpx-runtime-sidecar.ts | 12 +- .../control-plane/runner-execution-driver.ts | 4 + .../drivers/acpx/copilot-installation.test.ts | 14 ++- .../src/drivers/acpx/environment.test.ts | 7 ++ .../src/drivers/acpx/environment.ts | 1 + .../src/drivers/acpx/generated-profiles.ts | 8 +- .../src/drivers/acpx/pi-installation.test.ts | 41 ++++--- .../src/drivers/acpx/profile-compatibility.ts | 6 +- .../src/drivers/runner-tool-bridge.test.ts | 21 +++- .../src/drivers/runner-tool-bridge.ts | 16 +-- .../src/live/runnerd-codex-transport.ts | 29 ++++- .../live/runnerd-controller-outage.test.ts | 110 ++++++++++++++++++ ...runnerd-semantic-handler-readiness.test.ts | 54 +++++++++ .../pi-acp/profile-v29-identity.json | 51 ++++++++ .../copilot-profile-v59-identity.json | 85 ++++++++++++++ scripts/pi-public-install-probe.mjs | 4 +- .../native-runtime/native-session-executor.ts | 7 +- tests/runner-e2e/EVERYDAY-WORKFLOWS.md | 13 ++- tests/runner-e2e/README.md | 10 ++ tests/runner-e2e/SECURITY.md | 6 +- tests/runner-e2e/catalog.test.ts | 6 +- tests/runner-e2e/catalog.ts | 14 ++- tests/runner-e2e/everyday-flow.ts | 11 +- .../everyday-remote-restart.test.ts | 20 +++- tests/runner-e2e/everyday-remote-restart.ts | 22 +++- tests/runner-e2e/live-fixtures.ts | 4 +- tests/runner-e2e/runner.spec.ts | 6 +- tests/runner-e2e/server.ts | 16 ++- 37 files changed, 628 insertions(+), 75 deletions(-) create mode 100644 packages/paperclip-runner/src/live/runnerd-controller-outage.test.ts create mode 100644 packages/paperclip-runner/src/live/runnerd-semantic-handler-readiness.test.ts create mode 100644 packages/paperclip-runner/test-fixtures/pi-acp/profile-v29-identity.json create mode 100644 packages/paperclip-runner/test/fixtures/copilot-profile-v59-identity.json diff --git a/.dockerignore b/.dockerignore index 83ef4daeb0d..3bf8e79b7b1 100644 --- a/.dockerignore +++ b/.dockerignore @@ -88,6 +88,7 @@ packages/grok-acp/bin !packages/paperclip-runner/test-fixtures/pi-acp/profile-v27-identity.json !packages/paperclip-runner/test-fixtures/pi-acp/profile-v28-identity.json +!packages/paperclip-runner/test-fixtures/pi-acp/profile-v29-identity.json !packages/paperclip-runner/test/fixtures/copilot-profile-v23-identity.json diff --git a/doc/evals.md b/doc/evals.md index 2c0af7e5ec7..78304500307 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -1,5 +1,11 @@ # Paperclip evaluation guide +The explicit-only [Daytona restart continuity journey](../tests/runner-e2e/README.md) +extends the existing native restart oracle to a 125-second controller outage, +using the recorded Claude and Codex models in disposable development fixtures. +It requires same-process adoption, saved files and finalization without replay; +local regression checks and catalog discovery are separate from live acceptance. + The explicit-only [GitHub review bot journey](../tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md) converts the Ponytail walkthrough into attended App setup, scoped installation, automatic PR review, and independent two-service checks. It supports attached diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index b4d63b5687f..52dbcac821e 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1676,6 +1676,30 @@ waiting for provider startup. An abrupt controller exit during session opening can then recover through the same exact process-identity checks as an active turn. Both graceful and hot restarts detach the old controller from native sessions. +Remote Runner processes use their authenticated connection lease as the bound +on disconnected lifetime. The controller's shorter connection-recovery budget +limits its own attempt; it must not terminate the remote owner while another +controller is starting or classifying runs. Lease expiry still shuts down the +provider, and exact identity, authentication, Stop and revocation checks remain +required before any recovery or new execution. +Native ACPX semantic tool calls wait on their durable PRP invocation across a +controller outage. Their provider-side MCP bridge does not impose a separate +request deadline that could make the provider retry an already queued write. +Runner lease expiry, turn cancellation and bridge closure still end those waits. +Ordinary bridge callers retain their request deadline. Repeating the same call +identity returns its one retained result; it does not execute the operation again. +The native Claude sidecar also sets its HTTP MCP client budget to one hour, +matching the Runner connection lease budget. This prevents Claude's default +60-second HTTP deadline from aborting an otherwise pending durable request. +The setting applies to that native Claude session's MCP client; other provider +sessions and ordinary bridge deadlines remain unchanged. Explicit cancellation +and Runner shutdown still take precedence over that client budget. +An adopted transport can receive authenticated queued semantic calls before +the driver publishes its recovered session and tool handler. It holds those +calls until the handler is installed, then rechecks the original authority and +turn epoch before dispatch. Transport failure, close or controller detachment +ends that wait without invoking a late handler. Missing startup wiring is not +reported to the provider as the result of an external operation. If shutdown begins while a provider session is opening, its eventual publication honors the pending detachment before dispatching a turn. Once detached, an old execution finalizer cannot suspend or signal the durable runner: the next diff --git a/packages/paperclip-runner/acpx-profiles.json b/packages/paperclip-runner/acpx-profiles.json index 6b28b8dad51..fdf1b06255c 100644 --- a/packages/paperclip-runner/acpx-profiles.json +++ b/packages/paperclip-runner/acpx-profiles.json @@ -14,12 +14,12 @@ "requiresProviderPolicy": false }, "pi": { - "agentProfileVersion": 28, + "agentProfileVersion": 29, "agentServerPackage": "pi-acp", "agentServerVersion": "0.0.33", "agentRuntimePackage": "@earendil-works/pi-coding-agent", "agentRuntimeVersion": "1.0.0", - "commandDigest": "sha256:4918a4d5472d66b22b1ec642b68b5c60a8092a39c84ec1ab1c9ad0289cb8c68c", + "commandDigest": "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06", "requiresProviderPolicy": true }, "cursor": { @@ -32,12 +32,12 @@ "requiresProviderPolicy": true }, "copilot": { - "agentProfileVersion": 58, + "agentProfileVersion": 59, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:d490dacdab02051d24c84aae49ac6fb12b8226cf8a47b014ec53f3d3a8410a94", + "commandDigest": "sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40", "requiresProviderPolicy": true }, "claude": { diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs index c3f53e20b84..212852476cd 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs @@ -3212,7 +3212,7 @@ mod tests { #[test] fn pi_profile_matches_published_identity_and_rejects_prior_profiles() { let published: Value = serde_json::from_str(include_str!( - "../../../../test-fixtures/pi-acp/profile-v28-identity.json" + "../../../../test-fixtures/pi-acp/profile-v29-identity.json" )) .unwrap(); let mut value = descriptor("codex"); @@ -3229,6 +3229,7 @@ mod tests { current.validate(&context()).unwrap(); for prior in [ + include_str!("../../../../test-fixtures/pi-acp/profile-v28-identity.json"), include_str!("../../../../test-fixtures/pi-acp/profile-v24-identity.json"), include_str!("../../../../test-fixtures/pi-acp/profile-v23-identity.json"), include_str!("../../../../test-fixtures/pi-acp/profile-v15-identity.json"), diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs index 128ec81306e..eb6c82cefcb 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs @@ -29,7 +29,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option { agent_runtime_package: Some("@earendil-works/pi-coding-agent"), agent_runtime_version: Some("1.0.0"), command_digest: - "sha256:4918a4d5472d66b22b1ec642b68b5c60a8092a39c84ec1ab1c9ad0289cb8c68c", + "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06", requires_provider_policy: true, }, "cursor" => AcpxReleaseProfile { @@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option { agent_runtime_package: None, agent_runtime_version: None, command_digest: - "sha256:d490dacdab02051d24c84aae49ac6fb12b8226cf8a47b014ec53f3d3a8410a94", + "sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40", requires_provider_policy: true, }, "claude" => AcpxReleaseProfile { diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs index 872935a3a03..615a2e74839 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs @@ -17,7 +17,7 @@ use sha2::{Digest, Sha256}; const CODEX_ACPX_DIGEST: &str = "sha256:443491f4e814501a5dc47347e4510e84faeec9c9382e2d4e53bfe69386d5aabc"; const PI_ACPX_DIGEST: &str = - "sha256:4918a4d5472d66b22b1ec642b68b5c60a8092a39c84ec1ab1c9ad0289cb8c68c"; + "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06"; fn temporary_directory(label: &str) -> PathBuf { let nonce = SystemTime::now() diff --git a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs index 4d55c9f8367..7f43d83e8af 100644 --- a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs +++ b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs @@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url"; import { resolve, dirname } from "node:path"; const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); -const fixture = resolve(root, "test/fixtures/copilot-profile-v58-identity.json"); +const fixture = resolve(root, "test/fixtures/copilot-profile-v59-identity.json"); const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8")); -const declaration = { ...historical.declaration, agentProfileVersion: 58 }; +const declaration = { ...historical.declaration, agentProfileVersion: 59 }; delete declaration.permissionIdentitySourceSha256; const sources = { displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts", diff --git a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts index 62dc7e15321..28a0c5814b2 100644 --- a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts +++ b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts @@ -12,6 +12,7 @@ import { createHash, randomUUID } from "node:crypto"; import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process"; import { readFileSync } from "node:fs"; import { fileURLToPath } from "node:url"; +import { createAcpxSidecarHostEnvironment, createSanitizedAcpxSpawnInput } from "../drivers/acpx/environment.js"; import { afterEach, describe, expect, it, vi } from "vitest"; @@ -93,6 +94,22 @@ afterEach(async () => { }); describe("qualified ACPX runtime sidecar", () => { + it("gives native Claude HTTP tools the Runner lease budget without changing sibling providers", () => { + const source = readFileSync(new URL("./acpx-runtime-sidecar.ts", import.meta.url), "utf8"); + const start = source.indexOf(" environment:"); + const end = source.indexOf(" expectedIdentity:", start); + expect(start).toBeGreaterThan(0); expect(end).toBeGreaterThan(start); + const create = new Function("process", "params", "createAcpxSidecarHostEnvironment", `return ({ ${source.slice(start, end)} }).environment;`); + const original = { MCP_TOOL_TIMEOUT: "1000", ANTHROPIC_API_KEY: "selected-key", UNRELATED_SECRET: "private" }; + const claude = create({ env: original }, { agent: "claude", normalizedSessionId: "session" }, createAcpxSidecarHostEnvironment); + expect(createSanitizedAcpxSpawnInput(claude, "claude").env).toMatchObject({ MCP_TOOL_TIMEOUT: "3600000", ANTHROPIC_API_KEY: "selected-key" }); + expect(createSanitizedAcpxSpawnInput(claude, "claude").env).not.toHaveProperty("UNRELATED_SECRET"); + expect(original.MCP_TOOL_TIMEOUT).toBe("1000"); + const codex = create({ env: original }, { agent: "codex", normalizedSessionId: "session" }, createAcpxSidecarHostEnvironment); + expect(codex).toBe(original); + expect(createSanitizedAcpxSpawnInput(codex, "codex").env).not.toHaveProperty("MCP_TOOL_TIMEOUT"); + }); + it("projects the actual sidecar terminal diagnostic block without authoritative accounting", () => { const source = readFileSync(new URL("./acpx-runtime-sidecar.ts", import.meta.url), "utf8"); const start = source.indexOf(" try {", source.indexOf(" const usageAfter = await readSidecarHostStatusWithin(activeHost);")); @@ -279,6 +296,42 @@ describe("qualified ACPX runtime sidecar", () => { for (const agent of ["cursor", "codex", "pi"]) expect(create({ agent, tools: [] }, evidence, () => undefined).captureSemanticReceipt).toBeUndefined(); }); + it("keeps a native tool pending through controller downtime without a duplicate side effect", async () => { + const source = readFileSync(new URL("./acpx-runtime-sidecar.ts", import.meta.url), "utf8"); + const start = source.indexOf(" semanticTools: {"); + const end = source.indexOf(" onGoalUpdate:", start); + expect(start).toBeGreaterThan(0); expect(end).toBeGreaterThan(start); + const create = new Function("params", "activeCopilotEvidence", "waitForTool", `return ({ ${source.slice(start, end)} }).semanticTools;`); + let admitted!: () => void, complete!: (value: unknown) => void; + const started = new Promise(resolve => { admitted = resolve; }); + const result = new Promise(resolve => { complete = resolve; }); + let signal: AbortSignal | undefined; + const handler = vi.fn((call: RunnerToolCall) => { signal = call.signal; admitted(); return result; }); + const bridge = await startRunnerToolBridge(create({ agent: "claude", tools: [{ name: "register_deliverable", inputSchema: { type: "object" } }] }, undefined, handler)); + const call = () => fetch(bridge.url, { method: "POST", headers: { Authorization: `Bearer ${bridge.secret}`, "Content-Type": "application/json" }, + body: JSON.stringify({ jsonrpc: "2.0", id: "saved-call", method: "tools/call", params: { name: "register_deliverable", arguments: {} } }) }).then(response => response.json()); + let first: Promise | undefined; + try { + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] }); + first = call(); + await started; + await vi.advanceTimersByTimeAsync(125_000); + expect(signal?.aborted).toBe(false); + // The same pending invocation can be read again after reconnect. It must + // share the original result rather than create another external write. + const duplicate = call(); + complete({ artifactId: "saved-artifact" }); + expect(await first).toMatchObject({ result: { content: [{ text: '{"artifactId":"saved-artifact"}' }] } }); + expect(await duplicate).toEqual(await first); + expect(handler).toHaveBeenCalledOnce(); + } finally { + complete({ artifactId: "saved-artifact" }); + await first?.catch(() => undefined); + await bridge.close(); + vi.useRealTimers(); + } + }); + it.each(["forwarded", "emit-failed"])("binds native v2 receipts to actual sidecar-normalized input: %s", async mode => { const raw = { reportedWorkDisposition: "done", summary: "Complete", evidence: [], verification: [], completionClaim: { contractRevision: "1", objectiveSatisfied: true, criteria: [], remainingWork: [] } }; diff --git a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts index 2b728b2027e..bc1ffb523c3 100644 --- a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts +++ b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts @@ -305,12 +305,22 @@ async function dispatch( providerPolicy: params.providerPolicy, systemInstructions: params.systemInstructions, runtimeContext: params.runtimeContext, - environment: createAcpxSidecarHostEnvironment(process.env, params.agent, params.normalizedSessionId), + environment: params.agent === "claude" ? { + ...createAcpxSidecarHostEnvironment(process.env, params.agent, params.normalizedSessionId), + // Claude's HTTP MCP client otherwise aborts at 60s, before the + // native Runner connection lease. Cancellation and Runner shutdown + // still revoke the pending call; this is not a new authority lease. + MCP_TOOL_TIMEOUT: String(60 * 60 * 1_000), + } : createAcpxSidecarHostEnvironment(process.env, params.agent, params.normalizedSessionId), expectedIdentity: params.expectedIdentity, assignedMcpTransport: "prp", semanticTools: { tools: params.tools, handler: waitForTool, + // These calls are durable PRP requests. A controller outage must not + // turn a queued write into a provider-visible failure and retry. + // Runner lease expiry, turn cancellation and bridge close own them. + timeoutMs: null, ...(params.agent === "copilot" ? { captureSemanticReceipt: () => activeCopilotEvidence?.captureSemanticReceipt() } : {}), }, onGoalUpdate: (goal) => { diff --git a/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts b/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts index 06e94abc778..7a55b20b843 100644 --- a/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts +++ b/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts @@ -47,6 +47,10 @@ export interface RunnerExecutionDriver { }) => Promise>; /** Active-connection recovery budget. Omitted for the existing local mode. */ runnerReconnectGraceMs?: number; + /** Process lifetime while disconnected. Defaults to the active-connection + * recovery budget for compatibility. Null uses the authenticated connection + * lease as the lifetime bound, so controller downtime cannot kill its owner. */ + runnerProcessReconnectGraceMs?: number | null; /** * A verified Runner that outlived its controller. Adoption registers * the durable authority and waits for this exact process to reconnect; it diff --git a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts index 6df0af8afb3..8bdf9b3950d 100644 --- a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts @@ -87,9 +87,19 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi describe("Copilot build-owned installation", () => { it("admits the current profile with environment launch preparation", () => { - expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(58); + expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(59); expect(QUALIFIED_ACPX_PROFILES.copilot.qualificationStatus).toBeUndefined(); - expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:d490dacdab02051d24c84aae49ac6fb12b8226cf8a47b014ec53f3d3a8410a94"); + expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40"); + }); + + it("preserves v58 while binding the separate remote process reconnect policy in v59", () => { + const prior = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v58-identity.json", import.meta.url), "utf8")); + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v59-identity.json", import.meta.url), "utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([ + "agentProfileVersion", "credentialBindingSourceSha256", "nativeInstructionsSourceSha256", "nativeSessionRuntimeSourceSha256", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256", "semanticBridgeSourceSha256", "semanticSidecarSourceSha256", "transportContractSourceSha256", + ]); + expect(isSupportedAcpxProfileVersion("copilot", 58)).toBe(true); + expect(current.commandDigest).not.toBe(prior.commandDigest); }); it("preserves v57 while binding the shared Pi setup and launch repairs in v58", () => { diff --git a/packages/paperclip-runner/src/drivers/acpx/environment.test.ts b/packages/paperclip-runner/src/drivers/acpx/environment.test.ts index 3bc038a101e..3c7cdee2fe9 100644 --- a/packages/paperclip-runner/src/drivers/acpx/environment.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/environment.test.ts @@ -6,6 +6,13 @@ afterEach(() => vi.unstubAllEnvs()); import { ACPX_CREDENTIAL_BINDING_ENV, createAcpxCredentialBinding, createAcpxSidecarHostEnvironment, createSanitizedAcpxSpawnInput } from "./environment.js"; describe("ACPX launch environment", () => { + it("retains an explicit Claude MCP timeout without inheriting an ambient or sibling setting", () => { + vi.stubEnv("MCP_TOOL_TIMEOUT", "999999"); + expect(createSanitizedAcpxSpawnInput(undefined, "claude").env).not.toHaveProperty("MCP_TOOL_TIMEOUT"); + expect(createSanitizedAcpxSpawnInput({ MCP_TOOL_TIMEOUT: "120000" }, "claude").env.MCP_TOOL_TIMEOUT).toBe("120000"); + expect(createSanitizedAcpxSpawnInput({ MCP_TOOL_TIMEOUT: "120000" }, "codex").env).not.toHaveProperty("MCP_TOOL_TIMEOUT"); + }); + it("excludes general IAM from the Pi sidecar and launch, including selected task values", () => { const iam = { AWS_ACCESS_KEY_ID: "general-key", AWS_SECRET_ACCESS_KEY: "general-secret", AWS_SESSION_TOKEN: "general-session" }; const source = { ...iam, ...configuredEnvironmentProjection({ ...iam, CUSTOM_SETTING: "allowed" }), AWS_BEARER_TOKEN_BEDROCK: "provider-key" }; diff --git a/packages/paperclip-runner/src/drivers/acpx/environment.ts b/packages/paperclip-runner/src/drivers/acpx/environment.ts index 1fb79afba46..f11aeacfa08 100644 --- a/packages/paperclip-runner/src/drivers/acpx/environment.ts +++ b/packages/paperclip-runner/src/drivers/acpx/environment.ts @@ -131,6 +131,7 @@ export function createSanitizedAcpxSpawnInput( ...(environment === undefined ? [] : ["PAPERCLIP_AGENT_KEY_ID", "PAPERCLIP_AGENT_PUBLIC_KEY", "PAPERCLIP_AGENT_PRIVATE_KEY"]), ...credentialNames, ...(agent === "claude" ? CLAUDE_ROUTING_ENV_KEYS : []), + ...(agent === "claude" && environment !== undefined ? ["MCP_TOOL_TIMEOUT"] : []), ...(usesPreconfiguredHome(source) ? [...PRECONFIGURED_HOME_KEYS, "PAPERCLIP_RUNNER_HOME_MODE", "ACPX_CLAUDE_INCLUDE_USER_SETTINGS"] : []), ]); let retainedBytes = 0; diff --git a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts index ca6e53d79d7..36f2bdf29e1 100644 --- a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts +++ b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts @@ -22,12 +22,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = { "protocolVersion": 1, "acpxVersion": "0.13.1", "agent": "pi", - "agentProfileVersion": 28, + "agentProfileVersion": 29, "agentServerPackage": "pi-acp", "agentServerVersion": "0.0.33", "agentRuntimePackage": "@earendil-works/pi-coding-agent", "agentRuntimeVersion": "1.0.0", - "commandDigest": "sha256:4918a4d5472d66b22b1ec642b68b5c60a8092a39c84ec1ab1c9ad0289cb8c68c", + "commandDigest": "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06", "permissionPolicy": "interactive" }, "cursor": { @@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = { "protocolVersion": 1, "acpxVersion": "0.13.1", "agent": "copilot", - "agentProfileVersion": 58, + "agentProfileVersion": 59, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:d490dacdab02051d24c84aae49ac6fb12b8226cf8a47b014ec53f3d3a8410a94", + "commandDigest": "sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40", "permissionPolicy": "interactive" }, "claude": { diff --git a/packages/paperclip-runner/src/drivers/acpx/pi-installation.test.ts b/packages/paperclip-runner/src/drivers/acpx/pi-installation.test.ts index 7695e64219c..8d81addb3cf 100644 --- a/packages/paperclip-runner/src/drivers/acpx/pi-installation.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/pi-installation.test.ts @@ -155,11 +155,20 @@ describe("Pi installation factory", () => { expect(current.commandDigest).not.toBe(prior.commandDigest); }); + it("preserves profile 28 while binding the Claude-only explicit MCP client budget source in profile 29", async () => { + const read = async (version: number) => JSON.parse(await readFile(new URL(`../../../test-fixtures/pi-acp/profile-v${version}-identity.json`, import.meta.url), "utf8")); + const prior = await read(28), current = await read(29); + expect(Object.keys(current.declaration).filter(key => JSON.stringify(current.declaration[key]) !== JSON.stringify(prior.declaration[key])).sort()) + .toEqual(["agentProfileVersion", "credentialEnvironmentSourceSha256"]); + expect(current.declaration.closure).toEqual(prior.declaration.closure); + expect(current.commandDigest).not.toBe(prior.commandDigest); + }); + it("binds the profile declaration to the reviewed patch and platform closure pins", async () => { const hash = (bytes: string | Uint8Array) => createHash("sha256").update(bytes).digest("hex"); const canonical = (value: any): string => Array.isArray(value) ? `[${value.map(canonical).join(",")}]` : value && typeof value === "object" ? `{${Object.keys(value).sort().map(key => `${JSON.stringify(key)}:${canonical(value[key])}`).join(",")}}` : JSON.stringify(value); - const identity = JSON.parse(await readFile(new URL("../../../test-fixtures/pi-acp/profile-v28-identity.json", import.meta.url), "utf8")); + const identity = JSON.parse(await readFile(new URL("../../../test-fixtures/pi-acp/profile-v29-identity.json", import.meta.url), "utf8")); const declaration = identity.declaration; expect(declaration.sharedRuntimeContract).toBe("paperclip.acpx-runtime-contract.v1"); expect(identity.commandDigest).toBe(`sha256:${hash(canonical(declaration))}`); @@ -190,21 +199,21 @@ describe("Pi installation factory", () => { it("rejects legacy profiles and tampered executable identities", () => { expect(() => assertPiInstallationProfile(candidate())).not.toThrow(); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 1 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 2 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 3 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 4 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 5 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 6 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 7 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 8 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 9 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 10 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 11 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 12 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 13 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 14 })).toThrow("version 28"); - for (const priorVersion of [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]) expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: priorVersion })).toThrow("version 28"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 1 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 2 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 3 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 4 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 5 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 6 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 7 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 8 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 9 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 10 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 11 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 12 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 13 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 14 })).toThrow("version 29"); + for (const priorVersion of [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28]) expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: priorVersion })).toThrow("version 29"); for (const changed of [{ agentServerVersion: "latest" }, { commandDigest: `sha256:${"0".repeat(64)}` }, { agentRuntimePackage: "ambient-pi" }]) { expect(() => assertPiInstallationProfile({ ...candidate(), ...changed })).toThrow("trusted declaration"); } diff --git a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts index 1119594c0ed..56a5878aa39 100644 --- a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts +++ b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts @@ -2,13 +2,13 @@ import { QUALIFIED_ACPX_PROFILE_DATA } from "./generated-profiles.js"; /** Decodable historical profile revisions, not permission to launch them. * Exact current profile/package/digest admission is checked separately. */ -type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57; +type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58; type CurrentAcpxProfileVersion = typeof QUALIFIED_ACPX_PROFILE_DATA[keyof typeof QUALIFIED_ACPX_PROFILE_DATA]["agentProfileVersion"]; export type AcpxProfileVersion = HistoricalAcpxProfileVersion | CurrentAcpxProfileVersion; const historicalVersions: Readonly> = { - pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5], - grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57], + pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5], + grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58], cursor: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], }; diff --git a/packages/paperclip-runner/src/drivers/runner-tool-bridge.test.ts b/packages/paperclip-runner/src/drivers/runner-tool-bridge.test.ts index 152b8a9b6ea..4b212476c49 100644 --- a/packages/paperclip-runner/src/drivers/runner-tool-bridge.test.ts +++ b/packages/paperclip-runner/src/drivers/runner-tool-bridge.test.ts @@ -246,7 +246,7 @@ describe("runner semantic MCP bridge", () => { }); }); - it("honors MCP cancellation after the handler starts", async () => { + it.each([60_000, null])("honors MCP cancellation after the handler starts (timeout=%s)", async timeoutMs => { let markStarted!: () => void; const started = new Promise((resolve) => { markStarted = resolve; @@ -254,7 +254,7 @@ describe("runner semantic MCP bridge", () => { const aborted = vi.fn(); const bridge = await startRunnerToolBridge({ tools: [tool("documents.read")], - timeoutMs: 60_000, + timeoutMs, handler: ({ signal }) => new Promise((_resolve, reject) => { signal.addEventListener("abort", () => { aborted(); @@ -291,6 +291,23 @@ describe("runner semantic MCP bridge", () => { expect(aborted).toHaveBeenCalledOnce(); }); + it("cancels a lease-owned native tool when its bridge closes", async () => { + let admitted!: () => void; + const started = new Promise(resolve => { admitted = resolve; }); + let signal: AbortSignal | undefined; + const bridge = await startRunnerToolBridge({ tools: [tool("documents.read")], timeoutMs: null, + handler: call => { signal = call.signal; admitted(); return new Promise(() => {}); }, + }); + bridges.push(bridge); + const pending = rpc(bridge, { id: "close-me", method: "tools/call", params: { name: "documents.read", arguments: {} } }); + await started; + await bridge.close(); + expect(signal?.aborted).toBe(true); + expect(await (await pending).json()).toMatchObject({ result: { isError: true, + content: [{ text: "Paperclip tool call cancelled" }], + } }); + }); + it("preserves successful mutation identity when the result is oversized", async () => { const result = { text: `snowman-${"☃".repeat(65 * 1024)}` }; const handler = vi.fn(async () => result); diff --git a/packages/paperclip-runner/src/drivers/runner-tool-bridge.ts b/packages/paperclip-runner/src/drivers/runner-tool-bridge.ts index 3a82d69d209..624569fa195 100644 --- a/packages/paperclip-runner/src/drivers/runner-tool-bridge.ts +++ b/packages/paperclip-runner/src/drivers/runner-tool-bridge.ts @@ -45,7 +45,9 @@ export interface RunnerToolBridgeOptions { handler(call: RunnerToolCall): Promise; /** Copilot-only opt-in; capture the active invocation scope before dispatch. */ captureSemanticReceipt?: () => ((receipt: SemanticToolReceipt) => void) | undefined; - timeoutMs?: number; + /** Null delegates the wait to an owning durable runtime's lease and close. + * Ordinary callers retain the bounded request timeout. */ + timeoutMs?: number | null; privateToolTimeoutMs?: number; maxBodyBytes?: number; /** Internal test seam; production bounds headers and body reads to 30 seconds. */ @@ -143,7 +145,7 @@ export async function startRunnerToolBridge( controllers, handler: options.handler, captureSemanticReceipt: options.captureSemanticReceipt, - timeoutMs: positiveBoundedInteger( + timeoutMs: options.timeoutMs === null ? null : positiveBoundedInteger( options.timeoutMs, DEFAULT_TIMEOUT_MS, 1, @@ -214,7 +216,7 @@ async function handleRequest( controllers: Map; handler: RunnerToolBridgeOptions["handler"]; captureSemanticReceipt: RunnerToolBridgeOptions["captureSemanticReceipt"]; - timeoutMs: number; + timeoutMs: number | null; privateToolTimeoutMs: number; privateToolNames: ReadonlySet; maxBodyBytes: number; @@ -599,24 +601,24 @@ function closeServer(server: Server): Promise { function withCancellationAndTimeout( promise: Promise, controller: AbortController, - timeoutMs: number, + timeoutMs: number | null, ): Promise { return new Promise((resolve, reject) => { let settled = false; const finish = (callback: () => void) => { if (settled) return; settled = true; - clearTimeout(timer); + if (timer !== undefined) clearTimeout(timer); controller.signal.removeEventListener("abort", onAbort); callback(); }; const onAbort = () => finish(() => reject(new Error("Paperclip tool call cancelled"))); - const timer = setTimeout(() => { + const timer = timeoutMs === null ? undefined : setTimeout(() => { finish(() => reject(new Error("Paperclip tool call timed out"))); controller.abort(); }, timeoutMs); - timer.unref(); + timer?.unref(); controller.signal.addEventListener("abort", onAbort, { once: true }); promise.then( (value) => finish(() => resolve(value)), diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index 21452870d1a..b70954e9aa8 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -857,6 +857,11 @@ function runnerCloseGraceMs(options: Pick; + #resolveHandlerReady!: () => void; #core: DurablePrpControlPlane | null = null; #handle: RunnerProcessHandle | null = null; #adoptedRunnerMonitor: NodeJS.Timeout | null = null; @@ -3622,6 +3630,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.#failureSignal = new Promise((_resolve, reject) => { this.#rejectFailureSignal = reject; }); + this.#handlerReady = new Promise(resolve => { + this.#resolveHandlerReady = resolve; + }); this.#closedSignal = new Promise(resolve => { this.#resolveClosedSignal = resolve; }); @@ -3997,6 +4008,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { setServerRequestHandler(handler: CodexServerRequestHandler): void { this.#handler = handler; + this.#handlerInstalled = true; + this.#resolveHandlerReady(); } async #awaitWarmRunAttachmentReady(): Promise { @@ -5074,7 +5087,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { maxOutboxBytes: RUNNERD_MAX_OUTBOX_BYTES, p0ReserveBytes: RUNNERD_P0_RESERVE_BYTES, maxRuntimeMs: 0, - reconnectGraceMs: this.options.runnerReconnectGraceMs, + reconnectGraceMs: runnerProcessReconnectGraceMs(this.options), lifecyclePolicy: this.options.lifecyclePolicy, runnerBinaryPath, runnerVersion: runnerArtifact.version, @@ -5759,7 +5772,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { maxOutboxBytes: RUNNERD_MAX_OUTBOX_BYTES, p0ReserveBytes: RUNNERD_P0_RESERVE_BYTES, maxRuntimeMs: 0, - reconnectGraceMs: this.options.runnerReconnectGraceMs, + reconnectGraceMs: runnerProcessReconnectGraceMs(this.options), lifecyclePolicy: this.options.lifecyclePolicy, runnerBinaryPath, runnerVersion: runnerArtifact.version, @@ -5915,6 +5928,18 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { : await Promise.race([admission.settled, this.#failureSignal]).catch(() => { throw new SemanticToolNotDispatchedError(); }); + // An adopted Runner can authenticate and deliver queued tools before the + // driver has finished reconstructing its session. Handler publication is + // the dispatch barrier; the default RPC diagnostic is not a tool outcome. + // The owning startup/transport lifetime bounds this wait, and the original + // authority/epoch checks below still apply after it becomes ready. + if (!this.#handlerInstalled) { + await Promise.race([ + this.#handlerReady, + this.#failureSignal, + this.#closedSignal.then(() => { throw new SemanticToolNotDispatchedError(); }), + ]).catch(() => { throw new SemanticToolNotDispatchedError(); }); + } try { this.#throwIfFailed(); } catch { diff --git a/packages/paperclip-runner/src/live/runnerd-controller-outage.test.ts b/packages/paperclip-runner/src/live/runnerd-controller-outage.test.ts new file mode 100644 index 00000000000..2988e4845a9 --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-controller-outage.test.ts @@ -0,0 +1,110 @@ +import { createServer } from "node:http"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { expect, it, vi } from "vitest"; +import type { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js"; +import { codexSemanticToolSpecs } from "../drivers/codex/codex-app-server-driver.js"; +import { + createCapabilityRunnerdCodexTransport, + defaultCapabilityRunnerdBinary, +} from "./runnerd-codex-transport.js"; + +it.each([ + { policy: "connection lease", processGraceMs: null, survives: true }, + { policy: "explicit process deadline", processGraceMs: 500, survives: false }, +])("preserves the $policy policy across controller downtime beyond the connection recovery budget", async ({ processGraceMs, survives }) => { + const root = await mkdtemp(join(tmpdir(), "runner-controller-outage-")); + const server = createServer(); + let authority: DurablePrpControlPlane | null = null; + server.on("upgrade", (request, socket, head) => { + if (!authority) { socket.destroy(); return; } + authority.handleUpgrade(request, socket, "/runner", head); + }); + await new Promise(resolveListen => server.listen(0, "127.0.0.1", resolveListen)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("Missing outage test listener"); + const registration = async (next: DurablePrpControlPlane) => { + authority = next; + return { + connectUrl: `ws://127.0.0.1:${address.port}/runner`, + release: () => { if (authority === next) authority = null; }, + }; + }; + const shared = { + runnerBinary: defaultCapabilityRunnerdBinary(), + codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"), + codexArgs: ["--state-file", join(root, "provider-state.json"), "--hold-first-durable-turn"], + stateDirectory: root, + prpIdentity: { + runnerInstanceId: "runner-outage", environmentLeaseId: "lease-outage", runId: "run-outage", + normalizedSessionId: "session-outage", turnId: "turn-outage", itemId: "item-outage", + }, + // Scale the production 120-second reconnect budget down. The live process + // must be bounded by its authenticated connection lease, independently of + // how long a controller spends retrying a connection or booting its server. + runnerReconnectGraceMs: 500, + runnerProcessReconnectGraceMs: processGraceMs, + controlPlaneRegistration: registration, + }; + const first = createCapabilityRunnerdCodexTransport(shared); + let adopted: ReturnType | undefined; + let pid: number | null = null; + try { + const opened = await first.transport.request("thread/start", { + cwd: root, dynamicTools: codexSemanticToolSpecs(), + }); + await first.transport.request("turn/start", { input: [{ type: "text", text: "Keep this task active." }] }); + for await (const notification of first.transport.notifications()) { + if (notification.method === "turn/started") break; + } + pid = first.evidence().runnerPid; + const providerPid = first.evidence().codexPid; + expect(pid).toEqual(expect.any(Number)); + expect(providerPid).toEqual(expect.any(Number)); + await writeFile(join(root, "saved-work.txt"), "completed work survives controller replacement"); + await first.detachControllerForRestart(); + // The deliberate fault is an offline controller, not a slow test assertion. + await new Promise(resolveWait => setTimeout(resolveWait, 1_250)); + const runnerState = JSON.parse(await readFile(join(root, "runner", "runner-state.json"), "utf8")); + if (!survives) { + expect(runnerState.recoverableFailure).toBe("transport_reconnect_grace_exceeded"); + await vi.waitFor(() => expect(() => process.kill(pid!, 0)).toThrow()); + return; + } + expect(runnerState.recoverableFailure).not.toBe("transport_reconnect_grace_exceeded"); + expect(() => process.kill(pid!, 0)).not.toThrow(); + expect(() => process.kill(providerPid!, 0)).not.toThrow(); + const duplicateLaunch = vi.fn(() => { throw new Error("Must adopt the existing runner"); }); + const thread = opened.thread as Record; + adopted = createCapabilityRunnerdCodexTransport({ + ...shared, + runnerReconnectGraceMs: 5_000, + resumeDynamicTools: codexSemanticToolSpecs(), + resumeProviderSession: { driverSessionId: String(thread.id), providerSessionId: String(thread.sessionId) }, + runnerProcessLauncher: duplicateLaunch, + adoptExistingRunner: { + pid: pid!, processGroupId: pid, startedAt: new Date().toISOString(), + isAlive: () => { try { process.kill(pid!, 0); return true; } catch { return false; } }, + signal: signal => { try { process.kill(-pid!, signal); return true; } catch { return false; } }, + }, + }); + adopted.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] })); + const restored = await adopted.transport.request("thread/read", {}); + expect(restored.thread).toMatchObject({ id: thread.id, sessionId: thread.sessionId }); + expect(adopted.evidence()).toMatchObject({ runnerPid: pid, codexPid: providerPid, runnerExited: false }); + expect(duplicateLaunch).not.toHaveBeenCalled(); + expect(await readFile(join(root, "saved-work.txt"), "utf8")).toBe("completed work survives controller replacement"); + // A real Stop still reaches the adopted owner; the lease-only disconnect + // policy cannot turn a detached process into an uncancellable executor. + await adopted.transport.close(); + expect(adopted.evidence().runnerExited).toBe(true); + } finally { + await first.transport.close().catch(() => undefined); + await adopted?.transport.close().catch(() => undefined); + if (pid) { try { process.kill(-pid, "SIGKILL"); } catch { /* already stopped */ } } + server.closeAllConnections(); + if (server.listening) await new Promise(resolveClose => server.close(() => resolveClose())); + await rm(root, { recursive: true, force: true }); + } +}, 30_000); diff --git a/packages/paperclip-runner/src/live/runnerd-semantic-handler-readiness.test.ts b/packages/paperclip-runner/src/live/runnerd-semantic-handler-readiness.test.ts new file mode 100644 index 00000000000..f669c4d4aca --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-semantic-handler-readiness.test.ts @@ -0,0 +1,54 @@ +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { expect, it, vi } from "vitest"; +import type { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js"; +import type { CodexRpcServerRequest } from "../drivers/codex/app-server-transport.js"; +import { createCapabilityRunnerdCodexTransport, defaultCapabilityRunnerdBinary } from "./runnerd-codex-transport.js"; + +it.each(["install", "close", "detach"] as const)("holds an authenticated semantic call until handler readiness or %s", async mode => { + const root = await mkdtemp(join(tmpdir(), "runner-handler-readiness-")); + let core!: DurablePrpControlPlane; + const bundle = createCapabilityRunnerdCodexTransport({ + runnerBinary: defaultCapabilityRunnerdBinary(), + codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"), + codexArgs: ["--state-file", join(root, "provider.json"), "--emit-tool-call"], + stateDirectory: root, + controlPlaneRegistration: async authority => { + core = authority; + await authority.start(); + return { release: () => undefined }; + }, + }); + const handler = vi.fn(async (_request: CodexRpcServerRequest) => ({ success: true, contentItems: [{ type: "inputText", text: "one retained result" }] })); + let pid: number | null = null; + try { + await bundle.transport.request("thread/start", { cwd: root, dynamicTools: [{ name: "get_task_context", description: "Read the task.", inputSchema: { type: "object", properties: {}, additionalProperties: false } }] }); + await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Read the task." }] }); + pid = bundle.evidence().runnerPid; + expect(bundle.evidence().runnerProcessGroupId).toBe(pid); + await vi.waitFor(() => expect(core.store.state.committedEvents.some(event => event.eventType === "semantic_tool.input")).toBe(true)); + // The authenticated provider is already asking for a tool, but the driver + // has not published its recovered session/handler yet. No error result may + // be manufactured in this gap, and no external effect has been dispatched. + await new Promise(resolveWait => setTimeout(resolveWait, 50)); + expect(core.store.state.commands.filter(command => command.type === "semantic_tool.result")).toHaveLength(0); + expect(core.semanticToolResultsSettled()).toBe(false); + if (mode === "close") await bundle.transport.close(); + if (mode === "detach") await bundle.detachControllerForRestart(); + bundle.transport.setServerRequestHandler(handler); + if (mode === "install") { + await vi.waitFor(() => expect(handler).toHaveBeenCalledOnce()); + await vi.waitFor(() => expect(core.store.state.commands.filter(command => command.type === "semantic_tool.result")).toHaveLength(1)); + expect(core.store.state.commands.find(command => command.type === "semantic_tool.result")?.payload).toMatchObject({ isError: false }); + expect(handler.mock.calls[0]?.[0]).toMatchObject({ method: "item/tool/call", params: { tool: "get_task_context" } }); + } else { + await new Promise(resolveWait => setTimeout(resolveWait, 50)); + expect(handler).not.toHaveBeenCalled(); + } + } finally { + await bundle.transport.close().catch(() => undefined); + if (pid) { try { process.kill(-pid, "SIGKILL"); } catch { /* owned child already stopped */ } } + await rm(root, { recursive: true, force: true }); + } +}, 20_000); diff --git a/packages/paperclip-runner/test-fixtures/pi-acp/profile-v29-identity.json b/packages/paperclip-runner/test-fixtures/pi-acp/profile-v29-identity.json new file mode 100644 index 00000000000..664c0ab00c7 --- /dev/null +++ b/packages/paperclip-runner/test-fixtures/pi-acp/profile-v29-identity.json @@ -0,0 +1,51 @@ +{ + "commandDigest": "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "pi", + "agentProfileVersion": 29, + "acpxVersion": "0.13.1", + "agentServerVersion": "0.0.33", + "agentRuntimeVersion": "1.0.0", + "nodeVersion": "24.21.0", + "closure": { + "darwin-arm64": "282022db10150c6632b3444df421342e7d534bdf5d5fb1097a2e79d0625a2bcf", + "darwin-x64": "64e251e19009f755c0b04f73ce2138246faab71a961b0f13d75ebfcc34bef12e", + "linux-x64": "713b1fdff42fb56a1518bdc084f181d70bee8ebadc3e4b1d76321ed9108c8410" + }, + "wrapperSha256": "9d129b3d38772e93e97080aa6c4e574ac5df4e47ce5bc331a484a9fbf8188b36", + "helperSha256": "41e0490b617da0d60e0c8ec58ef311236f945129d99f816cff6897f78da7f91d", + "extensionSha256": "f5818573a355702388072b28926897db546a5cf346b5171f646582ce5f9102a4", + "agentFilesBinding": "registered-runtime-context-v1", + "nativeQuestions": "select-confirm-input-editor-v4-root-field-types", + "pathPolicy": "sdk-normalization-v1", + "mcpToolNames": "provider-safe-exact-source-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "nativeAssistantMessages": "pi1-rpc-deltas-v3-bound-message-and-tool-receipts", + "nativeNotices": "session-extension-v2-bounded-runtime-failure", + "messageProjectionSha256": "17d1e91b948dfa6a56e1905645add52b502041ad843c8d50a13b07956f22aadd", + "noticeProjectionSha256": "014485adc690998e395d334ea3ae67f72be6060a2fc78a1185df7b8d1da28fbf", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "nativeInputDisposition": "queued-only-v1", + "cacheWarming": "disabled-owned-decision-v1", + "thinkingLevel": { + "supported": [ + "off", + "low", + "high", + "max" + ], + "newConfigurationDefault": "low", + "admission": "explicit-native-effective-before-prompt-v1", + "recovery": "mode-bound-v1" + }, + "modelSelection": "caller-selected-native-verified-v1", + "providerConfigurationSourceSha256": "75a9a13f13f224dc092bd8ebf49e8747162172943761e44476c8eb18d80c52d4", + "credentialEnvironmentSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "recoveryIdentitySourceSha256": "d2c9fc161fcc135fb9760110bfa3d984ee2bb38d87fa5ce7cb7418a49dd89e66", + "nativeCancellation": "acknowledged-cancelled-terminal-preserves-usage-v1", + "dependencySecurityPatchSha256": "5273a195b952f233336122842faaf650193c04f36b9359d20d4d63079b0ce814", + "openRouterAttributionSourceSha256": "1211191529d6073b733b311c3d69cada4973eede0f3e540c46e261f78e0b90f9" + } +} diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v59-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v59-identity.json new file mode 100644 index 00000000000..b087d433186 --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v59-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 59, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629", + "semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "3343df6362252d44d12b079ff78c8f6539d1544f3ce5ec36ff36db70ab0fac31", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603", + "semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "972c08e6e320d1384221e180391fe817ea1ba632db687201e457d64e68ef4360", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a", + "providerLaunchSourceSha256": "d4f4a69baeeb2f3b2f350840f81f711e40f08588c28ea28039748dde90302a51", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "7497083b6cf1fc0268187aff948352f3bce6e054ac6e845005846473efdaff15", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "fc7c6568feb0d8d709e9baa51839420c9b90336b4865a9b8405aba06e4d1e9c4", + "profileCompatibilitySourceSha256": "6a26b102b78f282137ff93b0f75eb8788c603fc177802c58715c5ad7b2f3b9e3", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "d248e47a0fe7f2536d1e64cea0b0b410f34104800cd643d995b4473104ecc8e0", + "runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec", + "runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/scripts/pi-public-install-probe.mjs b/scripts/pi-public-install-probe.mjs index 5e6672cc68f..945d8c902de 100644 --- a/scripts/pi-public-install-probe.mjs +++ b/scripts/pi-public-install-probe.mjs @@ -52,8 +52,8 @@ assert.equal(JSON.parse(await readFile(join(server, 'package.json'), 'utf8')).na const { resolvePaperclipRunnerBinary } = await import(pathToFileURL(join(server, 'dist/services/native-runtime/native-codex-runner.js'))); const { createCapabilityRunnerdCodexTransport } = await import(pathToFileURL(join(server, 'dist/vendor/paperclip-runner/live/runnerd-codex-transport.js'))); const { QUALIFIED_ACPX_PROFILES } = await import(pathToFileURL(join(server, 'dist/vendor/paperclip-runner/drivers/acpx/qualified-profiles.js'))); -assert.equal(QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, 28); -assert.equal(QUALIFIED_ACPX_PROFILES.pi.commandDigest, 'sha256:4918a4d5472d66b22b1ec642b68b5c60a8092a39c84ec1ab1c9ad0289cb8c68c'); +assert.equal(QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, 29); +assert.equal(QUALIFIED_ACPX_PROFILES.pi.commandDigest, 'sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06'); assert.equal(Object.hasOwn(QUALIFIED_ACPX_PROFILES.pi, 'qualificationModel'), false); assert.equal(Object.hasOwn(QUALIFIED_ACPX_PROFILES.pi, 'reportedModelId'), false); const daemon = resolvePaperclipRunnerBinary(); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index ccf020f577b..62118309e9d 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -11999,7 +11999,8 @@ async function createRunnerdBackendWithinSessionClaim( claudeEffort: "claudeEffort" in provider ? provider.claudeEffort : undefined, acpxPermissionModePinned: true } : {}), ...(provider.kind === "opencode" ? { opencodePermissionMode: provider.permissionMode } : {}), - runnerReconnectGraceMs: 120_000, lifecyclePolicy: input.execution.session.lifecyclePolicy, + runnerReconnectGraceMs: 120_000, runnerProcessReconnectGraceMs: null, + lifecyclePolicy: input.execution.session.lifecyclePolicy, runtimeContext: "runtimeContext" in input.execution ? input.execution.runtimeContext : null, externallySandboxed: true, baseInstructions: recovery?.baseInstructions, resumeDynamicTools: dynamicTools, resumeCompletionContract: { revision: input.execution.completionContract.contract.revision, @@ -13494,6 +13495,10 @@ async function createRunnerdBackendWithinSessionClaim( ? resolveSourceCodexHome(resolveNativeProviderEnvironment(input.execution.provider, runnerEnvironment)) : undefined, runnerReconnectGraceMs: remoteTarget ? 120_000 : undefined, + // A controller's bounded reconnect attempt is not the remote owner's + // lifetime. Retain it through restarts until its authenticated lease + // expires or explicit Stop/suspension proves that execution ended. + runnerProcessReconnectGraceMs: remoteTarget ? null : undefined, environment: effectiveRunnerEnvironment, observability: input.observability, onDiagnostic: (message) => { diff --git a/tests/runner-e2e/EVERYDAY-WORKFLOWS.md b/tests/runner-e2e/EVERYDAY-WORKFLOWS.md index 567f365adeb..a9e395f53bb 100644 --- a/tests/runner-e2e/EVERYDAY-WORKFLOWS.md +++ b/tests/runner-e2e/EVERYDAY-WORKFLOWS.md @@ -16,7 +16,7 @@ exact execution ID explicitly. | Decide on an installed service action | `service-approve`, `service-decline` | Assign an authenticated local MCP fixture with Ask first; match its tool action and connection ID; no provider call before approval; exactly one after approval and a verified document; none after decline. | | Decline a new connection | `connection-decline` | Start without service connections; match a Notion connection intent; click Not now; verify the saved rejection, no new connection or repeated request, and an explanation followed by Done. | | Request an email address | `agentmail-setup` | Enable Chat connectors, ask for an email address, and require a durable AgentMail card for the requesting agent and user. Reload and verify one password field, the direct API-key link, and no access selectors or modal. Decline and verify no connection or repeated request. | -| Continue work after a controller restart | `recover-controller` | Observe saved source, persist a user message, restart the isolated controller, and independently test the delivered result. | +| Continue work after a controller restart | `recover-controller` | Observe saved source, persist a user message, restart the isolated controller, and independently test the delivered result. Native Daytona cases keep the controller offline for 125 seconds before restarting. | | Stop work and change direction | `stop-redirect` | Click Stop, send one new request, reload, observe exactly one stored user message and the new answer, and reach Done. | | Create and edit a company skill | `create-skill-studio` | Create one skill through the runner, verify its persisted library entry and activity-feed card, open Skill Studio, save an edit, and verify the edit after returning. Local Codex, local ACPX Claude, and warm Daytona cells are explicit. | @@ -47,8 +47,10 @@ restart, a verified first download supplies the persistence checkpoint; the controller is interrupted during a subsequent revision with another queued requirement. -The native Codex Daytona restart cell now requires a completed native tool -receipt while that revision is still running. Before and after restart it pins +The native Codex and ACPX Daytona restart cells require a completed native tool +receipt while that revision is still running. They keep the controller offline +for 125 seconds after its health listener stops, then start its replacement. +Before and after restart each pins the same run, normalized session, Runner identity, PID/start time, and active sandbox lease. The original receipt prefix must remain unchanged. A run that ends before either active observation leaves the fault boundary unexercised; @@ -56,7 +58,10 @@ it cannot pass from the final ZIP alone. The cell no longer exempts an original Codex remote run that fails with `process_lost` and is replaced. Completion requires one original provider turn and control-plane terminal, -the same native receipt prefix, no duplicate completed tool execution IDs, +the same native receipt prefix, no failed native tool receipts, and consistent +results for every stable completed tool identity. Reattached activity can +redeliver an identical start or tool receipt; changed content or a second +submission fails. It also requires a committed run and successful `workspace_finalize` receipt, and no pending workspace operation or recovery. Existing independent ZIP and preserved-download checks still apply. These receipt checks do not prove that arbitrary external diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 129f75c4e31..c6ed181c3e5 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1,5 +1,15 @@ # Paid runner full-stack E2E +The explicit-only `daytona-restart-continuity` suite runs the existing +`recover-controller` journey with the recorded Codex and Claude staging models +on fresh Daytona fixtures. After a completed task tool in an active revision +turn, it keeps the owned development controller offline for 125 seconds, beyond +the controller's two-minute reconnect budget. The existing independent oracle +requires the original process, native session, sandbox lease and receipt prefix, +one completion and successful workspace finalization, plus verified downloaded +work. It permits one attempt per cell and retains failed evidence. This is +development API-key acceptance, distinct from staging subscription recovery. + The explicit-only [GitHub review bot journey](GITHUB-REVIEW-JOURNEY.md) tests attended bot setup and an automatic PR review against an attached deployment. Private account bindings and setup receipts live in the private eval repository. diff --git a/tests/runner-e2e/SECURITY.md b/tests/runner-e2e/SECURITY.md index afbcf8dd494..555a6673b20 100644 --- a/tests/runner-e2e/SECURITY.md +++ b/tests/runner-e2e/SECURITY.md @@ -53,13 +53,17 @@ changes. ## GitHub authorization -The Everyday remote Codex restart oracle reads only the existing public run, +The Everyday remote native restart oracle reads only the existing public run, event, environment-lease, workspace-operation and recovery APIs. It does not execute commands in a provider sandbox or signal a provider process. Its new evidence projects closed receipt metadata and hashes the provider sandbox ID and full native event content. Provider text, tool arguments and command output do not enter these snapshots. Existing image admission, credentials, billing, retention and cleanup rules continue to apply. +The isolated server's private restart request may delay its replacement by at +most five minutes. This fault injection never stops a hosted instance or a +provider sandbox. The explicit Daytona continuity suite has one attempt and +1,000-cent company and agent hard stops. Set `RUNNER_E2E_ALLOWED_ACTOR_IDS` to a non-empty JSON array of numeric GitHub user IDs. Keep the list equal to the owners of `.github/**` in diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 4f36708a091..4b87cac677d 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -219,10 +219,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 13, 10, 2, 48, 16, 10, 2, 1, 1, 4, 116, 2, 2, + 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 13, 10, 2, 48, 16, 10, 2, 1, 1, 4, 116, 2, 2, ]); - expect(validateRunnerCatalog()).toHaveLength(765); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(765); + expect(validateRunnerCatalog()).toHaveLength(767); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(767); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 84c733fe035..a824e9b9dd4 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1155,6 +1155,17 @@ const repositoryModelProfiles = [ ]; export const runnerSuites: readonly RunnerSuiteFixture[] = [ + { + id: "daytona-restart-continuity", label: "Daytona controller outage continuity", manualOnly: true, + description: "Keep the original active native runner, task tools and saved files through more than two minutes without a controller, then commit the result without replay.", + groups: [], profiles: repositoryModelProfiles.map(({ native }) => productionStoryProfile(native)), + environments: runnerEnvironments.filter(environment => environment.id === "daytona"), + tasks: everydayTasks.filter(task => task.id === "recover-controller") + .map(task => ({ ...task, automaticRetryPolicy: "single_attempt" as const })), expectedMatrixSize: 2, + definitionMetadata: { version: 2, scheduling: "explicit-only", controllerDowntimeMs: 125_000, + maximumAttemptsPerCell: 1, budgetMonthlyCents: 1000, instructions: "production", grading: "same-active-process-lease-native-prefix-and-workspace-receipt", + authentication: "api-key", stagingAuthentication: "subscription", stagingQualification: false }, + }, { id: "codie-latency-diagnostic", label: "Native Codex latency diagnostic", @@ -1514,7 +1525,8 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ ...(["hire-reuse", "delegate-feedback"].includes(task.id) ? [] : [`everyday-workflows.runner-opencode.local.${task.id}`]), ]), ], - definitionMetadata: { version: 7, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only", + definitionMetadata: { version: 9, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only", + remoteControllerDowntimeMs: 125_000, remoteCodexRestart: "same-active-process-lease-native-prefix-and-workspace-receipt", procedureCases: { ids: ["hire-reuse", "delegate-feedback"], maximumAttemptsPerCell: 1, companyAndLeadBudgetCents: 1_000 } }, }, diff --git a/tests/runner-e2e/everyday-flow.ts b/tests/runner-e2e/everyday-flow.ts index 3dc84a29769..bebc81eacff 100644 --- a/tests/runner-e2e/everyday-flow.ts +++ b/tests/runner-e2e/everyday-flow.ts @@ -89,7 +89,7 @@ interface Input { workspacePath: string; privateDir: string; deadlineAt: number; - restart(): Promise; + restart(controllerDowntimeMs?: number): Promise; observe(issue: StoryIssue, runs: StoryRun[]): void; capture(id: string, label: string, file: string): Promise; evidence(name: string, value: unknown): Promise; @@ -169,7 +169,8 @@ export async function runEverydayFlow(input: Input) { let project = fixtures.project; const caseId = execution.task.id; const requireRemoteRestart = caseId === "recover-controller" && execution.environment.id === "daytona" - && execution.profile.generation === "native" && execution.profile.provider === "codex"; + && execution.profile.generation === "native" && (execution.profile.provider === "codex" + || execution.profile.provider === "acpx"); const providerChoice = caseId === "provider-decline" || caseId === "provider-second"; const nativeProviderCase = caseId === "provider-native"; let aggregatorFixture: Awaited> | undefined; @@ -949,7 +950,7 @@ export async function runEverydayFlow(input: Input) { ev.remoteRestart = { before }; await input.evidence("remote-restart-before.json", before); } else ev.allowedInterruptedRuns.push(active.id); - await input.restart(); + await input.restart(requireRemoteRestart ? 125_000 : undefined); note("controller-restarted"); await openParent(); if (requireRemoteRestart) { @@ -1643,7 +1644,9 @@ export async function runEverydayFlow(input: Input) { ev.remoteRestart!.final = final; return final; }, accept: final => gradeRemoteRestart({ ...ev.remoteRestart, final }).every(check => check.passed), - reject: final => ["failed", "cancelled", "timed_out"].includes(final.run.status) ? "Original remote run failed after restart" : undefined }); + reject: final => ["failed", "cancelled", "timed_out"].includes(final.run.status) ? "Original remote run failed after restart" + : final.run.status === "succeeded" && final.run.nativePhase === "committed" + && !gradeRemoteRestart({ ...ev.remoteRestart, final })[1]!.passed ? "Original remote turn completed with failed tools or conflicting/repeated execution receipts" : undefined }); await input.evidence("remote-restart-final.json", final); } ev.checks.push( diff --git a/tests/runner-e2e/everyday-remote-restart.test.ts b/tests/runner-e2e/everyday-remote-restart.test.ts index 938c9b30bf6..c492e4f9e72 100644 --- a/tests/runner-e2e/everyday-remote-restart.test.ts +++ b/tests/runner-e2e/everyday-remote-restart.test.ts @@ -111,11 +111,27 @@ describe("remote restart lifecycle evidence", () => { expect(gradeRemoteRestart(missingPrefix)[1]!.passed).toBe(false); }); - it.each(["replacement", "second submission", "duplicate tool receipt", "missing terminal"])("rejects %s at completion", fault => { + it("rejects failed native tool calls even when the original turn eventually succeeds", () => { + const { evidence, event } = fixture(); + const rows = [...fixture().afterRows, event(6, "tool.execution.completed", { executionId: "failed-write", status: "failed", output: "private timeout" }), + event(7, "turn.completed"), event(8, "run.result.accepted", {}, "control_plane", 1), event(9, "run.terminal", {}, "control_plane", 2)]; + evidence.final.events = projectRemoteRestartEvents(evidence.final.run, rows); + expect(gradeRemoteRestart(evidence).every(check => check.passed)).toBe(false); + }); + + it("accepts identical activity receipt redelivery without treating it as another submission", () => { + const { evidence } = fixture(); + evidence.final.events.push({ ...evidence.final.events[1]!, seq: 10, sourceSeq: 10, sourceEventId: "runner:run:10" }, + { ...evidence.final.events[3]!, seq: 11, sourceSeq: 11, sourceEventId: "runner:run:11" }); + expect(gradeRemoteRestart(evidence).every(check => check.passed)).toBe(true); + }); + + it.each(["replacement", "second submission", "changed tool receipt", "changed start receipt", "missing terminal"])("rejects %s at completion", fault => { const { evidence } = fixture(); if (fault === "replacement") evidence.final.run.id = "replacement"; if (fault === "second submission") evidence.final.events.push({ ...evidence.final.events[0]!, seq: 10, sourceSeq: 10, sourceEventId: "runner:run:10" }); - if (fault === "duplicate tool receipt") evidence.final.events.push({ ...evidence.final.events[3]!, seq: 10, sourceSeq: 10, sourceEventId: "runner:run:10" }); + if (fault === "changed tool receipt") evidence.final.events.push({ ...evidence.final.events[3]!, seq: 10, sourceSeq: 10, sourceEventId: "runner:run:10", payloadSha256: "0".repeat(64) }); + if (fault === "changed start receipt") evidence.final.events.push({ ...evidence.final.events[1]!, seq: 10, sourceSeq: 10, sourceEventId: "runner:run:10", payloadSha256: "0".repeat(64) }); if (fault === "missing terminal") evidence.final.events.pop(); expect(gradeRemoteRestart(evidence)[1]!.passed).toBe(false); }); diff --git a/tests/runner-e2e/everyday-remote-restart.ts b/tests/runner-e2e/everyday-remote-restart.ts index 59c1db429e8..9b50934eec6 100644 --- a/tests/runner-e2e/everyday-remote-restart.ts +++ b/tests/runner-e2e/everyday-remote-restart.ts @@ -12,7 +12,8 @@ const digest = (value: unknown) => createHash("sha256").update(canonicalJson(val export interface RemoteRestartEvent { seq: number; sourceSeq: number; sourceEventId: string; eventType: string; sourceKind: "runner" | "control_plane"; turnId: string | null; - toolExecutionId: string | null; toolCompleted: boolean; sha256: string; + toolExecutionId: string | null; toolCompleted: boolean; toolFailed: boolean; + payloadSha256: string; sha256: string; } export interface RemoteRestartBoundary { identity: { runId: string; companyId: string; agentId: string; issueId: string; @@ -78,6 +79,8 @@ export function projectRemoteRestartEvents(run: StoryRun, rows: readonly Row[]): turnId: id(event.turnId) ? event.turnId : null, toolExecutionId: id(payload.executionId) ? payload.executionId : null, toolCompleted: event.eventType === "tool.execution.completed" && payload.status === "completed", + toolFailed: event.eventType === "tool.execution.completed" && payload.status !== "completed", + payloadSha256: digest(payload), sha256: digest(event) }; }); } @@ -132,12 +135,23 @@ export function gradeRemoteRestart(evidence: RemoteRestartEvidence | undefined): const events = final?.events ?? []; const turnId = before?.events.find(event => event.eventType === "turn.started")?.turnId; const once = (kind: string, source: RemoteRestartEvent["sourceKind"]) => events.filter(event => event.eventType === kind && event.sourceKind === source).length === 1; - const completedTools = events.filter(event => event.toolCompleted).map(event => event.toolExecutionId); + // Reattachment can redeliver an already-observed activity notification. Its + // stable identity and exact payload must agree; a new submission, changed + // result or failed tool is not receipt replay and must never pass this cell. + const starts = events.filter(event => event.eventType === "turn.started" && event.sourceKind === "runner"); + const tools = events.filter(event => event.toolCompleted); + const toolResults = new Map(); + let consistentToolResults = true; + for (const event of tools) { + if (!event.toolExecutionId || (toolResults.has(event.toolExecutionId) && toolResults.get(event.toolExecutionId) !== event.payloadSha256)) consistentToolResults = false; + else toolResults.set(event.toolExecutionId, event.payloadSha256); + } const nativeCompletion = finalIdentity && Boolean(before && after && retainsPrefix(after.events, events)) - && once("turn.submitted", "runner") && once("turn.started", "runner") && once("turn.completed", "runner") + && once("turn.submitted", "runner") && starts.length > 0 && new Set(starts.map(event => event.payloadSha256)).size === 1 + && once("turn.completed", "runner") && once("run.result.accepted", "control_plane") && once("run.terminal", "control_plane") && events.filter(event => event.eventType.startsWith("turn.")).every(event => event.turnId === turnId || (event.eventType === "turn.submitted" && event.turnId === null)) - && new Set(completedTools).size === completedTools.length; + && consistentToolResults && !events.some(event => event.toolFailed); const operations = final?.operations ?? []; const saved = finalIdentity && run?.status === "succeeded" && run.nativePhase === "committed" && run.resultJson?.finalizationPhase === "committed" && run.resultJson.workspaceFinalizeStatus === "succeeded" diff --git a/tests/runner-e2e/live-fixtures.ts b/tests/runner-e2e/live-fixtures.ts index cc861c6232c..c5d27531b43 100644 --- a/tests/runner-e2e/live-fixtures.ts +++ b/tests/runner-e2e/live-fixtures.ts @@ -171,7 +171,7 @@ export async function setupLiveFixtures(input: { : ["native-completion", "native-instruction-consolidation", "native-connection-guidance"].includes(execution.suite.id) || (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id)) ? NATIVE_COMPLETION_BUDGET_CENTS : execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS - : ["stock-harness", "task-workspaces", "case-handoffs", "feedback-persistence"].includes(execution.suite.id) ? 1_000 : 0, + : ["stock-harness", "task-workspaces", "case-handoffs", "feedback-persistence", "daytona-restart-continuity"].includes(execution.suite.id) ? 1_000 : 0, }); await input.onCompanyCreated?.(company.id); return company; @@ -365,7 +365,7 @@ export async function setupLiveFixtures(input: { setupProbe.defaultRunner = selected.defaultRunner; } } - if (["stock-harness", "native-connection-guidance", "feedback-persistence"].includes(execution.suite.id) + if (["stock-harness", "native-connection-guidance", "feedback-persistence", "daytona-restart-continuity"].includes(execution.suite.id) || (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id))) { agent.budgetMonthlyCents = 1_000; } diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 420d8c97cf5..8836e27dfe8 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -264,6 +264,7 @@ async function restartIsolatedPaperclipServer(input: { deadlineAt: number; preserveRunner?: RestartRunnerIdentity; crashAfterResult?: FinalizationCrashBinding; + controllerDowntimeMs?: number; }): Promise { const { controlDirectory, @@ -274,7 +275,8 @@ async function restartIsolatedPaperclipServer(input: { const temporaryRequestPath = `${requestPath}.${process.pid}.${input.requestId}.tmp`; await writeFile( temporaryRequestPath, - JSON.stringify({ requestId: input.requestId, ...(input.preserveRunner ? { preserveRunner: input.preserveRunner } : {}), + JSON.stringify({ requestId: input.requestId, ...(input.controllerDowntimeMs !== undefined ? { controllerDowntimeMs: input.controllerDowntimeMs } : {}), + ...(input.preserveRunner ? { preserveRunner: input.preserveRunner } : {}), ...(input.crashAfterResult ? { crashAfterResult: input.crashAfterResult } : {}) }), { encoding: "utf8", mode: 0o600 }, ); @@ -1240,7 +1242,7 @@ for (const execution of executions) { const story = await runEverydayFlow({ page, api, fixtures, execution, nonce, workspacePath, privateDir, deadlineAt: startedAtMs + deadlineMs, - restart: () => restartIsolatedPaperclipServer({ api, requestId: `story-${nonce}`, deadlineAt: startedAtMs + deadlineMs }), + restart: controllerDowntimeMs => restartIsolatedPaperclipServer({ api, requestId: `story-${nonce}`, deadlineAt: startedAtMs + deadlineMs, controllerDowntimeMs }), observe: (storyIssue, storyRuns) => { issue = storyIssue; selectedRuns = storyRuns; }, capture: captureScreenshot, evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets), diff --git a/tests/runner-e2e/server.ts b/tests/runner-e2e/server.ts index 8956ac3f5aa..451ed91e034 100644 --- a/tests/runner-e2e/server.ts +++ b/tests/runner-e2e/server.ts @@ -293,6 +293,7 @@ async function waitForHealthToStop() { interface RestartRequest { requestId: string; + controllerDowntimeMs?: number; preserveRunner?: RestartRunnerIdentity; crashAfterResult?: FinalizationCrashBinding; } @@ -321,9 +322,15 @@ async function readRestartRequest(): Promise { return null; } const preserveRunner = (value as { preserveRunner?: unknown }).preserveRunner; + const controllerDowntimeMs = (value as { controllerDowntimeMs?: unknown }).controllerDowntimeMs; + if (controllerDowntimeMs !== undefined && (!Number.isSafeInteger(controllerDowntimeMs) + || Number(controllerDowntimeMs) < 0 || Number(controllerDowntimeMs) > 300_000)) { + throw new Error("Controller downtime must be an integer between zero and five minutes"); + } const crashAfterResult = (value as { crashAfterResult?: unknown }).crashAfterResult; if (crashAfterResult !== undefined && preserveRunner !== undefined) throw new Error("A finalization crash cannot preserve a running provider"); - return { requestId, ...(preserveRunner === undefined ? {} : { preserveRunner: parseRestartRunnerIdentity(preserveRunner) }), + return { requestId, ...(controllerDowntimeMs === undefined ? {} : { controllerDowntimeMs: Number(controllerDowntimeMs) }), + ...(preserveRunner === undefined ? {} : { preserveRunner: parseRestartRunnerIdentity(preserveRunner) }), ...(crashAfterResult === undefined ? {} : { crashAfterResult: parseFinalizationCrashBinding(crashAfterResult) }) }; } @@ -348,7 +355,7 @@ async function writeRestartAck( await rename(temporaryAckPath, restartAckPath); } -async function restartServer({ requestId, preserveRunner, crashAfterResult }: RestartRequest) { +async function restartServer({ requestId, preserveRunner, crashAfterResult, controllerDowntimeMs = 0 }: RestartRequest) { activeRestartRequestId = requestId; appendLog(`\nRestart request ${requestId}: stopping Paperclip\n`); const previous = child; @@ -365,6 +372,11 @@ async function restartServer({ requestId, preserveRunner, crashAfterResult }: Re // Do not mistake an orphaned old server for a healthy replacement. The port // must stop answering before the next launcher is allowed to start. await waitForHealthToStop(); + const downtimeDeadline = Date.now() + controllerDowntimeMs; + while (Date.now() < downtimeDeadline) { + if (shutdownRequested()) throw new Error("Wrapper shutdown interrupted controller downtime"); + await delay(Math.min(250, downtimeDeadline - Date.now())); + } if (shutdownRequested()) { throw new Error("Wrapper shutdown interrupted the Paperclip restart"); } From a09ef344b3ff6ce1686a1f515820353345462c7b Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 07:38:18 -0700 Subject: [PATCH 38/68] perf(agents): avoid duplicate configuration reads (#17612) Reuse one complete company roster for list responses and narrow other-agent eligibility checks to their five consumed fields. Preserve full selected-agent configuration, terminated-manager evidence, current spend, authorization, and fresh reads. Verified with 170 affected tests, four rejected regression mutations, build and typecheck, exact-head Greptile 5/5, and green required CI. Co-Authored-By: Paperclip --- doc/DATABASE.md | 7 + .../agent-record-read-projections.test.ts | 132 ++++++++++++++++++ server/src/lib/agent-records.ts | 15 +- server/src/services/agents.ts | 14 +- .../services/plugin-agent-runtime-state.ts | 2 +- 5 files changed, 156 insertions(+), 14 deletions(-) create mode 100644 server/src/__tests__/agent-record-read-projections.test.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index c4d5c72e41e..6c47e5d7495 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -231,6 +231,13 @@ The benchmark loads the baseline authorization module with current dependencies, compares complete workspace rows, and prints five warm samples and query plans. It measures SQL execution, not the full repository admission or agent startup. +Agent lists read the complete company roster once and use that same snapshot for +termination filtering and organization-chain normalization. Detail and plugin runtime reads +load only identity, company, name, status, and manager from other company agents +for chain checks. The selected agent's full configuration, current monthly spend, +and existing response redaction remain available. Every call reads current data; +these lookups do not cache authorization or agent configuration. + ## 1. Embedded PostgreSQL — zero config If you don't set `DATABASE_URL`, the server automatically starts an embedded PostgreSQL instance and manages a local data directory. diff --git a/server/src/__tests__/agent-record-read-projections.test.ts b/server/src/__tests__/agent-record-read-projections.test.ts new file mode 100644 index 00000000000..bf0a5183202 --- /dev/null +++ b/server/src/__tests__/agent-record-read-projections.test.ts @@ -0,0 +1,132 @@ +import { randomUUID } from "node:crypto"; +import { agents, companies, costEvents, createDb } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { eq } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { agentRecordQueries } from "../lib/agent-records.js"; +import { agentService } from "../services/agents.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("agent roster read projections", () => { + let database: Awaited>; + let db: ReturnType; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("agent-roster-read-"); + db = createDb(database.connectionString); + }, 60_000); + + beforeEach(() => { + vi.useFakeTimers({ toFake: ["Date"] }); + vi.setSystemTime(new Date("2020-06-15T12:00:00Z")); + }); + + afterEach(() => { vi.useRealTimers(); }); + + afterAll(async () => { + await db?.$client.end(); + await database?.cleanup(); + }); + + async function fixture() { + const companyId = randomUUID(), foreignCompanyId = randomUUID(); + const managerId = randomUUID(), workerId = randomUUID(), unrelatedId = randomUUID(), archivedId = randomUUID(), foreignId = randomUUID(); + await db.insert(companies).values([ + { id: companyId, name: "Roster fixture", issuePrefix: `R${companyId.slice(0, 7).toUpperCase()}` }, + { id: foreignCompanyId, name: "Foreign roster", issuePrefix: `F${foreignCompanyId.slice(0, 7).toUpperCase()}` }, + ]); + const bulk = "retained-other-agent-configuration🙂".repeat(32_768); + await db.insert(agents).values([ + { id: managerId, companyId, name: "Manager", status: "terminated", adapterType: "process", adapterConfig: { instructions: bulk } }, + { id: unrelatedId, companyId, name: "Other worker", status: "idle", adapterType: "process", adapterConfig: { instructions: bulk } }, + { id: archivedId, companyId, name: "Archived worker", status: "terminated", adapterType: "process", runtimeConfig: { retained: bulk } }, + { id: foreignId, companyId: foreignCompanyId, name: "Foreign manager", status: "idle", adapterType: "process", adapterConfig: { instructions: bulk } }, + ]); + const [worker] = await db.insert(agents).values({ id: workerId, companyId, name: "Selected worker", status: "idle", + reportsTo: managerId, adapterType: "process", adapterConfig: { instructions: "selected instructions", model: "fixture-model" }, + runtimeConfig: { heartbeat: { enabled: false } }, metadata: { retained: "selected metadata" }, + permissions: { canCreateAgents: true }, spentMonthlyCents: 999, + }).returning(); + const now = new Date(); + await db.insert(costEvents).values([ + { companyId, agentId: workerId, provider: "fixture", model: "fixture-model", costCents: 19.5, occurredAt: now }, + { companyId, agentId: workerId, provider: "fixture", model: "fixture-model", costCents: 50, + occurredAt: new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth() - 1, 15)) }, + { companyId: foreignCompanyId, agentId: foreignId, provider: "fixture", model: "fixture-model", costCents: 100, occurredAt: now }, + ]); + return { companyId, foreignCompanyId, managerId, workerId, unrelatedId, archivedId, foreignId, worker, bulk }; + } + + async function observe(run: (observed: ReturnType) => Promise) { + const observed = createDb(database.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { agents, costEvents }, capture); + try { + const result = await run(observed); + capture.flush(); + const reads = events.filter(event => event.table === "agents" && event.operation === "select"); + return { result, calls: reads.reduce((n, event) => n + Number(event.calls), 0), + bytes: reads.reduce((n, event) => n + Number(event.responseValueBytes), 0), events }; + } finally { + capture.flush(); + await observed.$client.end(); + } + } + + it.each([false, true])("reads one full roster for list, preserving terminated-manager evidence and configurations (includeTerminated=%s)", async includeTerminated => { + const f = await fixture(); + const captured = await observe(observed => agentService(observed).list(f.companyId, { includeTerminated })); + expect(captured.calls).toBe(1); + expect(captured.result.map(row => row.id).sort()).toEqual((includeTerminated + ? [f.workerId, f.managerId, f.archivedId, f.unrelatedId] : [f.workerId, f.unrelatedId]).sort()); + const selected = captured.result.find(row => row.id === f.workerId)!; + expect(selected).toMatchObject({ adapterConfig: f.worker.adapterConfig, runtimeConfig: f.worker.runtimeConfig, + metadata: f.worker.metadata, permissions: { canCreateAgents: true }, spentMonthlyCents: 19.5, + orgChainHealth: { status: "invalid_org_chain", reason: "terminated_ancestor", firstInvalidAncestor: { id: f.managerId } } }); + expect(captured.result.find(row => row.id === f.unrelatedId)?.adapterConfig.instructions).toBe(f.bulk); + if (includeTerminated) expect(captured.result.find(row => row.id === f.archivedId)?.runtimeConfig.retained).toBe(f.bulk); + expect((await db.select().from(agents).where(eq(agents.id, f.workerId)))[0]).toEqual(f.worker); + expect(JSON.stringify(captured.events)).not.toContain("retained-other-agent-configuration"); + }); + + it("keeps detail reads small while returning the selected agent's complete current configuration", async () => { + const f = await fixture(); + const captured = await observe(observed => agentRecordQueries(observed).getById(f.workerId)); + expect(captured.calls).toBe(2); + expect(captured.bytes).toBeLessThan(8192); + expect(captured.result).toMatchObject({ id: f.workerId, companyId: f.companyId, adapterConfig: f.worker.adapterConfig, + runtimeConfig: f.worker.runtimeConfig, metadata: f.worker.metadata, permissions: { canCreateAgents: true }, spentMonthlyCents: 19.5, + orgChainHealth: { reason: "terminated_ancestor", firstInvalidAncestor: { id: f.managerId, name: "Manager" } } }); + expect((await db.select().from(agents).where(eq(agents.id, f.workerId)))[0]).toEqual(f.worker); + }); + + it("projects only current company chain fields for plugin normalization", async () => { + const f = await fixture(); + const captured = await observe(observed => agentRecordQueries(observed).listCompanyAgentEligibilityRows(f.companyId)); + expect(captured.calls).toBe(1); + expect(captured.bytes).toBeLessThan(4096); + expect(captured.result.map(row => row.id).sort()).toEqual([f.workerId, f.managerId, f.unrelatedId, f.archivedId].sort()); + for (const row of captured.result) expect(Object.keys(row).sort()).toEqual(["companyId", "id", "name", "reportsTo", "status"]); + const normalized = agentRecordQueries(db).normalizeAgentRow(f.worker, captured.result); + expect(normalized).toMatchObject({ adapterConfig: f.worker.adapterConfig, metadata: f.worker.metadata, + orgChainHealth: { reason: "terminated_ancestor", firstInvalidAncestor: { id: f.managerId } } }); + expect(await agentRecordQueries(db).listCompanyAgentEligibilityRows(randomUUID())).toEqual([]); + expect(await agentRecordQueries(db).getById(randomUUID())).toBeNull(); + }); + + it("rechecks manager status, name, selected configuration, missing managers, and cycles on later reads", async () => { + const f = await fixture(), records = agentRecordQueries(db); + expect((await records.getById(f.workerId))?.orgChainHealth.reason).toBe("terminated_ancestor"); + await db.update(agents).set({ status: "paused", name: "Current manager" }).where(eq(agents.id, f.managerId)); + await db.update(agents).set({ adapterConfig: { instructions: "changed instructions" } }).where(eq(agents.id, f.workerId)); + expect(await records.getById(f.workerId)).toMatchObject({ adapterConfig: { instructions: "changed instructions" }, + orgChainHealth: { status: "healthy", pausedAncestors: [{ id: f.managerId, name: "Current manager", status: "paused" }] } }); + await db.update(agents).set({ reportsTo: f.foreignId }).where(eq(agents.id, f.workerId)); + expect((await records.getById(f.workerId))?.orgChainHealth.reason).toBe("missing_manager"); + await db.update(agents).set({ reportsTo: f.managerId }).where(eq(agents.id, f.workerId)); + await db.update(agents).set({ reportsTo: f.workerId }).where(eq(agents.id, f.managerId)); + expect((await records.getById(f.workerId))?.orgChainHealth.reason).toBe("cycle"); + }); +}); diff --git a/server/src/lib/agent-records.ts b/server/src/lib/agent-records.ts index 786f61af550..25c0ed7c9f5 100644 --- a/server/src/lib/agent-records.ts +++ b/server/src/lib/agent-records.ts @@ -98,6 +98,8 @@ export interface CreateAgentOptions { claudeLogin?: ClaudeLoginContext; } +type AgentEligibilityRow = Pick; + interface AgentShortnameRow { id: string; name: string; @@ -340,7 +342,7 @@ export function agentRecordQueries(db: Db) { }; } - function normalizeAgentRows(rows: (typeof agents.$inferSelect)[], allCompanyRows = rows) { + function normalizeAgentRows(rows: (typeof agents.$inferSelect)[], allCompanyRows: AgentEligibilityRow[] = rows) { const eligibilityAgents = allCompanyRows.map(toEligibilityAgent); return rows.map((row) => { const base = normalizeAgentBaseRow(row); @@ -357,7 +359,7 @@ export function agentRecordQueries(db: Db) { }); } - function normalizeAgentRow(row: typeof agents.$inferSelect, allCompanyRows?: (typeof agents.$inferSelect)[]) { + function normalizeAgentRow(row: typeof agents.$inferSelect, allCompanyRows?: AgentEligibilityRow[]) { return normalizeAgentRows([row], allCompanyRows)[0]!; } @@ -365,6 +367,11 @@ export function agentRecordQueries(db: Db) { return db.select().from(agents).where(eq(agents.companyId, companyId)); } + async function listCompanyAgentEligibilityRows(companyId: string): Promise { + return db.select({ id: agents.id, companyId: agents.companyId, name: agents.name, + status: agents.status, reportsTo: agents.reportsTo }).from(agents).where(eq(agents.companyId, companyId)); + } + async function getMonthlySpendByAgentIds(companyId: string, agentIds: string[]) { if (agentIds.length === 0) return new Map(); const { start, end } = currentUtcMonthWindow(); @@ -405,7 +412,7 @@ export function agentRecordQueries(db: Db) { .then((rows) => rows[0] ?? null); if (!row) return null; const [companyRows, hydrated] = await Promise.all([ - listCompanyAgentRows(row.companyId), + listCompanyAgentEligibilityRows(row.companyId), hydrateAgentSpend([row]).then((rows) => rows[0]!), ]); return normalizeAgentRow(hydrated, companyRows); @@ -472,6 +479,6 @@ export function agentRecordQueries(db: Db) { }); } - return { normalizeAgentRows, normalizeAgentRow, listCompanyAgentRows, hydrateAgentSpend, + return { normalizeAgentRows, normalizeAgentRow, listCompanyAgentRows, listCompanyAgentEligibilityRows, hydrateAgentSpend, getById, ensureManager, assertNoCycle, assertCompanyShortnameAvailable, assertBuiltInAgentMetadataMutationAllowed }; } diff --git a/server/src/services/agents.ts b/server/src/services/agents.ts index 4c4cbd7248f..c9dd824f495 100644 --- a/server/src/services/agents.ts +++ b/server/src/services/agents.ts @@ -1,6 +1,6 @@ import { type BudgetServiceHooks } from "./budgets.js"; -import { and, desc, eq, inArray, ne } from "drizzle-orm"; +import { and, desc, eq, inArray } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { agents, agentConfigRevisions, agentApiKeys, heartbeatRuns } from "@paperclipai/db"; import { isUuidLike, normalizeAgentApiKeyScope, normalizeAgentUrlKey, type AgentApiKeyScope } from "@paperclipai/shared"; @@ -31,14 +31,10 @@ export function agentService(db: Db, budgetHooks: BudgetServiceHooks = {}) { createAgentLifecycle(db, budgetHooks).updateConfiguration(...args); return { list: async (companyId: string, options?: { includeTerminated?: boolean }) => { - const conditions = [eq(agents.companyId, companyId)]; - if (!options?.includeTerminated) { - conditions.push(ne(agents.status, "terminated")); - } - const [rows, allCompanyRows] = await Promise.all([ - db.select().from(agents).where(and(...conditions)), - listCompanyAgentRows(companyId), - ]); + const allCompanyRows = await listCompanyAgentRows(companyId); + const rows = options?.includeTerminated + ? allCompanyRows + : allCompanyRows.filter((row) => row.status !== "terminated"); const hydrated = await hydrateAgentSpend(rows); return normalizeAgentRows(hydrated, allCompanyRows); }, diff --git a/server/src/services/plugin-agent-runtime-state.ts b/server/src/services/plugin-agent-runtime-state.ts index a1a4ce22ca7..0770246ef14 100644 --- a/server/src/services/plugin-agent-runtime-state.ts +++ b/server/src/services/plugin-agent-runtime-state.ts @@ -36,7 +36,7 @@ async function readState(db: Db, pluginId: string, companyId: string, agentId: s }; const instructions = await agentInstructionsService().getBundleLocation(agent); const records = agentRecordQueries(tx as unknown as Db); - const companyAgents = await records.listCompanyAgentRows(companyId); + const companyAgents = await records.listCompanyAgentEligibilityRows(companyId); return { revision: String(latest?.id ?? 0), agent: records.normalizeAgentRow(agent, companyAgents) as PluginAgentRuntimeState["agent"], instructionsRootPath: instructions.rootPath, instructionsEntryFile: instructions.entryFile, From bf9a8551c33fad7315fb635781483be260b14fde Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 07:51:28 -0700 Subject: [PATCH 39/68] perf(issues): filter queued comment wakes by task (#17613) Apply the existing typed task identity filter in PostgreSQL before returning queued-comment wake controls. Preserve queue selection, ordering, payloads, and current run ownership checks. All 84 route tests, build, typecheck, exact-head Greptile 5/5, and required CI pass. The old source fails the new deferred and queued byte budgets; corrected reads preserve the complete response and all saved rows. Co-Authored-By: Paperclip --- doc/DATABASE.md | 5 +++ .../issue-queued-comments-routes.test.ts | 39 +++++++++++++++++++ server/src/routes/issues.ts | 1 + 3 files changed, 45 insertions(+) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 6c47e5d7495..780247fc5c8 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -238,6 +238,11 @@ for chain checks. The selected agent's full configuration, current monthly spend and existing response redaction remain available. Every call reads current data; these lookups do not cache authorization or agent configuration. +Queued-comment display filters wake requests by the current company, assigned +agent, status, and exact JSON string task identity in SQL. Other tasks' comment +identifiers stay in the database. The display projection, ordering, current run +ownership checks, and persisted wake payloads retain their existing contract. + ## 1. Embedded PostgreSQL — zero config If you don't set `DATABASE_URL`, the server automatically starts an embedded PostgreSQL instance and manages a local data directory. diff --git a/server/src/__tests__/issue-queued-comments-routes.test.ts b/server/src/__tests__/issue-queued-comments-routes.test.ts index 7b0af0e8a3d..e02d9708a38 100644 --- a/server/src/__tests__/issue-queued-comments-routes.test.ts +++ b/server/src/__tests__/issue-queued-comments-routes.test.ts @@ -252,6 +252,45 @@ describeEmbeddedPostgres("issue queued-comment routes", () => { } finally { capture.flush(); await observed.$client.end(); } }); + it.each(["deferred", "queued"])("filters unrelated task wakes before reading the %s comment queue", async state => { + const f = await seedQueue(); + if (state === "queued") await promoteQueue(f); + const path = `/api/issues/${f.issueId}/queued-comments`; + const expected = await request(app(f.companyId)).get(path).expect(200); + const unrelated = Array.from({ length: 80 }, (_, index) => ({ + companyId: f.companyId, agentId: f.agentId, source: "issue_comment", + status: index % 2 ? "queued" : "deferred_issue_execution", + requestedAt: new Date("2020-01-01T00:00:00Z"), + payload: { issueId: randomUUID(), _paperclipWakeContext: { + wakeCommentIds: Array.from({ length: 60 }, () => randomUUID()), + } }, + })); + const malformed = [null, [f.issueId], { id: f.issueId }, f.issueId.toUpperCase()]; + await db.insert(agentWakeupRequests).values([ + ...unrelated, + ...malformed.map(issueId => ({ companyId: f.companyId, agentId: f.agentId, + source: "issue_comment", status: "deferred_issue_execution", + requestedAt: new Date("2020-01-01T00:00:00Z"), + payload: { issueId, _paperclipWakeContext: { wakeCommentIds: f.commentIds } }, + })), + ]); + const observed = createDb(tempDb!.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { agentWakeupRequests }, capture); + try { + const response = await request(app(f.companyId, "queue-owner", undefined, observed)).get(path).expect(200); + expect(response.body).toEqual(expected.body); + capture.flush(); + const reads = events.filter(event => event.table === "agent_wakeup_requests" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(4096); + expect(reads.reduce((rows, event) => rows + Number(event.rows), 0)).toBe(1); + expect(await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests) + .where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(85); + await request(app(randomUUID(), "queue-owner", undefined, observed)).get(path).expect(404); + } finally { capture.flush(); await observed.$client.end(); } + }); + it.each(["cancelled", "running"])("rejects a late Done from an interrupted %s task run at the write boundary", async status => { const seeded = await seedQueue(); await db.update(heartbeatRuns).set({ status, resultJson: { executionCancellation: { state: "requested" } } }) diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index bb92497ea3b..bbdfdfb031b 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -7320,6 +7320,7 @@ export function issueRoutes( and( eq(agentWakeupRequests.companyId, issue.companyId), eq(agentWakeupRequests.agentId, issue.assigneeAgentId), + sql`${agentWakeupRequests.payload}->'issueId' = to_jsonb(${issue.id}::text)`, inArray(agentWakeupRequests.status, [ "deferred_issue_execution", "queued", From e33526556879e0720e95cd112a5faa3bffac8ce2 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 07:54:47 -0700 Subject: [PATCH 40/68] fix(ui): stop idle task run polling with live updates (#17608) Stop successful empty assigned-task run queries from polling only while the matching company and session connection is open. Preserve active, pending, failed, and disconnected cadence. Share one lifecycle refresh owner between the task and chat views, discover reviewer events without task identity, and refresh after older in-flight reads settle. All 175 focused tests, 9,079 UI tests, build, typecheck, exact-head Greptile 5/5, and required CI pass. Regression mutations catch reviewer discovery, duplicate reads, readiness, failure state, missing refreshes, and scope cleanup. Earlier E2E cleanup deadlock and server cleanup timeout remain documented; failed-only reruns passed on their same commits. Co-Authored-By: Paperclip --- doc/DATABASE.md | 15 + .../LiveUpdatesProvider.recovery.test.tsx | 359 +++++++++++++++++- ui/src/context/LiveUpdatesProvider.tsx | 86 ++++- ui/src/lib/live-runs-cache.test.ts | 18 +- ui/src/lib/live-runs-cache.ts | 10 + ui/src/pages/IssueDetail.tsx | 14 +- 6 files changed, 481 insertions(+), 21 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 780247fc5c8..8056e2468c3 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -145,6 +145,21 @@ patch does not refetch task details, activity, or run history. Unknown runs retain the existing refresh fallback. Run transitions, activity events, and reconnect recovery still refresh their affected data. +Task run queries stop interval polling only after a successful empty response +and an open shared live connection for the current company and session. Initial +reads, active work, errors, connection setup, and disconnections retain their +existing polling cadence. Company or session changes require a new connection. +Views without an assigned agent retain polling. Matching run lifecycle events +refresh the displayed task's run queries independently of route and window +focus. Events that omit task identity conservatively refresh these queries, +including reviewer runs from another agent. One parent view owns both query +observers and reuses refreshes already covered by the focused-route handler. A +matching event that overlaps an older request queues a fresh read after that +request settles. Connection readiness also reconciles the displayed run +queries after any older request. Follow-up reads belong to the current mounted +scope and stop when it changes or unmounts. The first open and reconnect still +reconcile missed events. + Connection-assignment checks select profile bindings for the current company or agent and read rules that can match the selected connection's catalog rows. All five selector kinds, null application matching, exclusions, and profile diff --git a/ui/src/context/LiveUpdatesProvider.recovery.test.tsx b/ui/src/context/LiveUpdatesProvider.recovery.test.tsx index b63381e9b44..591e735fa49 100644 --- a/ui/src/context/LiveUpdatesProvider.recovery.test.tsx +++ b/ui/src/context/LiveUpdatesProvider.recovery.test.tsx @@ -5,18 +5,19 @@ import { QueryClient, QueryClientProvider, useQuery } from "@tanstack/react-quer import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { queryKeys } from "../lib/queryKeys"; -const { pushToast } = vi.hoisted(() => ({ pushToast: vi.fn() })); +const { pushToast, companySelection, routeSelection } = vi.hoisted(() => ({ pushToast: vi.fn(), companySelection: { id: "company-1" }, routeSelection: { pathname: "/tasks" } })); const { sessionRead, healthRead } = vi.hoisted(() => ({ sessionRead: vi.fn(), healthRead: vi.fn() })); vi.mock("./CompanyContext", () => ({ - useCompany: () => ({ selectedCompanyId: "company-1", selectedCompany: { id: "company-1" } }), + useCompany: () => ({ selectedCompanyId: companySelection.id, selectedCompany: { id: companySelection.id } }), })); vi.mock("./ToastContext", () => ({ useToastActions: () => ({ pushToast }) })); -vi.mock("../lib/router", () => ({ useLocation: () => ({ pathname: "/tasks" }) })); +vi.mock("../lib/router", () => ({ useLocation: () => routeSelection })); vi.mock("../api/auth", () => ({ authApi: { getSession: sessionRead }, })); vi.mock("../api/health", () => ({ healthApi: { get: healthRead } })); -import { LiveUpdatesProvider } from "./LiveUpdatesProvider"; +import { LiveUpdatesProvider, useHasCompanyLiveConnection, useTaskRunLiveUpdates } from "./LiveUpdatesProvider"; +import { liveRunRefetchInterval } from "../lib/live-runs-cache"; (globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true; @@ -25,8 +26,8 @@ class Socket { readyState = 1; onopen: (() => void) | null = null; onclose: (() => void) | null = null; - onmessage = null; - onerror = null; + onmessage: ((message: { data: string }) => void) | null = null; + onerror: (() => void) | null = null; constructor() { Socket.instances.push(this); } close() { this.readyState = 3; } } @@ -36,9 +37,13 @@ describe("LiveUpdatesProvider connection recovery", () => { let container: HTMLDivElement; let client: QueryClient; const read = vi.fn(async () => "current task data"); + const pollRead = vi.fn<() => Promise>(); beforeEach(() => { vi.useFakeTimers(); + companySelection.id = "company-1"; + routeSelection.pathname = "/tasks"; + pollRead.mockReset().mockResolvedValue([]); vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible"); read.mockReset(); read.mockResolvedValue("current task data"); @@ -71,6 +76,348 @@ describe("LiveUpdatesProvider connection recovery", () => { await act(async () => vi.advanceTimersByTimeAsync(1)); } + function IdleRuns({ companyId }: { companyId: string }) { + const connected = useHasCompanyLiveConnection(companyId); + useQuery({ + queryKey: [companyId, "idle-run-fixture"], + queryFn: pollRead, + refetchInterval: (query) => liveRunRefetchInterval(connected, query.state, 1000), + }); + return {connected ? "connected" : "fallback"}; + } + + async function renderPolling(companyId = companySelection.id, withProvider = true) { + const child = ; + await act(async () => root.render( + + {withProvider ? {child} : child} + , + )); + await act(async () => vi.advanceTimersByTimeAsync(1)); + } + + async function advancePolls(count: number) { + for (let i = 0; i < count; i += 1) { + await act(async () => vi.advanceTimersByTimeAsync(1001)); + } + } + + it.each([null, []])("keeps setup polling, then stops known-empty polling after connection for %j", async (data) => { + vi.stubGlobal("WebSocket", Socket); + pollRead.mockResolvedValue(data); + await renderPolling(); + expect(container.textContent).toBe("fallback"); + pollRead.mockClear(); + await advancePolls(3); + expect(pollRead.mock.calls.length).toBeGreaterThanOrEqual(3); + await act(async () => Socket.instances[0].onopen?.()); + await act(async () => vi.advanceTimersByTimeAsync(1)); + expect(container.textContent).toBe("connected"); + const readsAfterOpen = pollRead.mock.calls.length; + await advancePolls(10); + expect(pollRead).toHaveBeenCalledTimes(readsAfterOpen); + }); + + it.each([[{ id: "run-1" }], { id: "run-1" }])("keeps active work polling while connected for %j", async (data) => { + vi.stubGlobal("WebSocket", Socket); + pollRead.mockResolvedValue(data); + await renderPolling(); + await act(async () => Socket.instances[0].onopen?.()); + pollRead.mockClear(); + await advancePolls(3); + expect(pollRead.mock.calls.length).toBeGreaterThanOrEqual(3); + }); + + it("keeps polling a failed query with previously empty data until it recovers", async () => { + vi.stubGlobal("WebSocket", Socket); + await renderPolling(); + await act(async () => Socket.instances[0].onopen?.()); + pollRead.mockRejectedValue(new Error("Read unavailable")); + await act(async () => { await client.invalidateQueries({ queryKey: ["company-1", "idle-run-fixture"] }); }); + expect(client.getQueryState(["company-1", "idle-run-fixture"])?.status).toBe("error"); + pollRead.mockClear(); + await advancePolls(3); + expect(pollRead.mock.calls.length).toBeGreaterThanOrEqual(3); + pollRead.mockResolvedValue([]); + await advancePolls(1); + expect(client.getQueryState(["company-1", "idle-run-fixture"])?.status).toBe("success"); + const afterRecovery = pollRead.mock.calls.length; + await advancePolls(5); + expect(pollRead).toHaveBeenCalledTimes(afterRecovery); + }); + + it("retains an initial pending read until it resolves after the connection opens", async () => { + vi.stubGlobal("WebSocket", Socket); + let resolveRead!: (value: unknown) => void; + pollRead.mockImplementation(() => new Promise((resolve) => { resolveRead = resolve; })); + await renderPolling(); + await act(async () => Socket.instances[0].onopen?.()); + expect(container.textContent).toBe("connected"); + expect(client.getQueryState(["company-1", "idle-run-fixture"])?.status).toBe("pending"); + expect(pollRead).toHaveBeenCalledTimes(1); + await act(async () => resolveRead([])); + await act(async () => vi.advanceTimersByTimeAsync(1)); + expect(client.getQueryData(["company-1", "idle-run-fixture"])).toEqual([]); + await advancePolls(5); + expect(pollRead).toHaveBeenCalledTimes(1); + }); + + it("resumes local polling on error before close, and stops it after recovery", async () => { + vi.stubGlobal("WebSocket", Socket); + await renderPolling(); + const first = Socket.instances[0]; + await act(async () => first.onopen?.()); + await act(async () => first.onerror?.()); + expect(container.textContent).toBe("fallback"); + pollRead.mockClear(); + await advancePolls(3); + expect(pollRead.mock.calls.length).toBeGreaterThanOrEqual(3); + await act(async () => first.onclose?.()); + await advancePolls(2); + expect(Socket.instances.length).toBeGreaterThan(1); + await act(async () => Socket.instances.at(-1)?.onopen?.()); + expect(container.textContent).toBe("connected"); + const afterRecovery = pollRead.mock.calls.length; + await advancePolls(10); + expect(pollRead).toHaveBeenCalledTimes(afterRecovery); + }); + + it.each(["company", "session"])("requires a new connection after a %s change and ignores stale callbacks", async (scope) => { + vi.stubGlobal("WebSocket", Socket); + await renderPolling(); + const first = Socket.instances[0]; + await act(async () => first.onopen?.()); + const oldOpen = first.onopen, oldClose = first.onclose, oldError = first.onerror; + if (scope === "company") { + companySelection.id = "company-2"; + await renderPolling(); + } else { + const nextSession = { user: { id: "viewer" }, session: { id: "replacement-session", userId: "viewer" } }; + sessionRead.mockResolvedValue(nextSession); + await act(async () => client.setQueryData(queryKeys.auth.session, nextSession)); + await act(async () => vi.advanceTimersByTimeAsync(1)); + } + expect(container.textContent).toBe("fallback"); + await act(async () => oldOpen?.()); + expect(container.textContent).toBe("fallback"); + pollRead.mockClear(); + await advancePolls(2); + expect(pollRead.mock.calls.length).toBeGreaterThanOrEqual(2); + await act(async () => Socket.instances.at(-1)?.onopen?.()); + expect(container.textContent).toBe("connected"); + await act(async () => { oldClose?.(); oldError?.(); }); + expect(container.textContent).toBe("connected"); + const afterOpen = pollRead.mock.calls.length; + await advancePolls(5); + expect(pollRead).toHaveBeenCalledTimes(afterOpen); + }); + + it("discovers matching new runs after empty polling stops on an unfocused non-task route", async () => { + vi.stubGlobal("WebSocket", Socket); + vi.spyOn(document, "hasFocus").mockReturnValue(false); + const queryRef = "DEMO-1"; + const runRead = vi.fn<() => Promise>().mockResolvedValue([]); + const activeRead = vi.fn<() => Promise>().mockResolvedValue(null); + function TaskRuns() { + const connected = useTaskRunLiveUpdates({ companyId: "company-1", issueId: "issue-1", issueQueryRef: queryRef, assigneeAgentId: "agent-1" }); + const runs = useQuery({ queryKey: queryKeys.issues.liveRuns(queryRef), queryFn: runRead, + refetchInterval: (query) => liveRunRefetchInterval(connected, query.state, 1000) }); + useQuery({ queryKey: queryKeys.issues.activeRun(queryRef), queryFn: activeRead, + refetchInterval: (query) => liveRunRefetchInterval(connected, query.state, 1000) }); + return {JSON.stringify(runs.data)}; + } + await act(async () => root.render()); + await act(async () => vi.advanceTimersByTimeAsync(1)); + const socket = Socket.instances[0]; + await act(async () => socket.onopen?.()); + runRead.mockClear(); activeRead.mockClear(); + await advancePolls(5); + expect(runRead).not.toHaveBeenCalled(); + expect(activeRead).not.toHaveBeenCalled(); + const deliver = async (companyId: string, payload: Record) => { + await act(async () => socket.onmessage?.({ data: JSON.stringify({ id: 1, companyId, + type: "heartbeat.run.queued", createdAt: "2020-01-01T00:00:00Z", payload }) })); + await act(async () => vi.advanceTimersByTimeAsync(1)); + }; + await deliver("company-2", { issueId: "issue-1", agentId: "agent-1", runId: "foreign" }); + await deliver("company-1", { issueId: "other-issue", agentId: "agent-1", runId: "other" }); + expect(runRead).not.toHaveBeenCalled(); + expect(activeRead).not.toHaveBeenCalled(); + runRead.mockResolvedValue([{ id: "new-run", status: "running" }]); + activeRead.mockResolvedValue({ id: "new-run", status: "running" }); + await deliver("company-1", { issueId: "issue-1", agentId: "new-agent", runId: "new-run" }); + expect(runRead).toHaveBeenCalledTimes(1); + expect(activeRead).toHaveBeenCalledTimes(1); + expect(container.textContent).toContain("new-run"); + await deliver("company-1", { agentId: "reviewer-agent", runId: "reviewer-run-without-task-id" }); + expect(runRead).toHaveBeenCalledTimes(2); + expect(activeRead).toHaveBeenCalledTimes(2); + }); + + it.each(["initial", "background", "connection setup"])("refetches after an older empty %s read settles when run events arrive during it", async (phase) => { + vi.stubGlobal("WebSocket", Socket); + vi.spyOn(document, "hasFocus").mockReturnValue(false); + const queryKey = queryKeys.issues.liveRuns("DEMO-1"); + const runRead = vi.fn<() => Promise>().mockResolvedValue([]); + let resolveOlder!: (value: unknown) => void; + const older = new Promise((resolve) => { resolveOlder = resolve; }); + if (phase !== "background") runRead.mockImplementationOnce(() => older); + function TaskRuns() { + const connected = useTaskRunLiveUpdates({ companyId: "company-1", issueId: "issue-1", issueQueryRef: "DEMO-1", assigneeAgentId: "agent-1" }); + const { data } = useQuery({ queryKey, queryFn: runRead, + refetchInterval: (query) => liveRunRefetchInterval(connected, query.state, 1000) }); + return {JSON.stringify(data)}; + } + await act(async () => root.render()); + await act(async () => vi.advanceTimersByTimeAsync(1)); + const socket = Socket.instances[0]; + await act(async () => socket.onopen?.()); + if (phase === "background") { + runRead.mockClear(); + runRead.mockImplementationOnce(() => older); + await act(async () => { void client.invalidateQueries({ queryKey }); }); + } + runRead.mockResolvedValue([{ id: "new-run", status: "running" }]); + expect(runRead).toHaveBeenCalledTimes(1); + await act(async () => { + for (let id = 1; phase !== "connection setup" && id <= 3; id += 1) socket.onmessage?.({ data: JSON.stringify({ id, + companyId: "company-1", type: "heartbeat.run.queued", createdAt: "2020-01-01T00:00:00Z", + payload: { issueId: "issue-1", agentId: "agent-1", runId: "new-run" } }) }); + }); + expect(runRead).toHaveBeenCalledTimes(1); + await act(async () => resolveOlder([])); + await act(async () => vi.advanceTimersByTimeAsync(1)); + expect(runRead).toHaveBeenCalledTimes(2); + expect(client.getQueryData(queryKey)).toEqual([{ id: "new-run", status: "running" }]); + expect(container.textContent).toContain("new-run"); + }); + + it.each(["unmount", "disconnect", "company", "session", "task"])("discards the old follow-up refresh after %s during an older read", async (change) => { + vi.stubGlobal("WebSocket", Socket); + let resolveOlder!: (value: unknown) => void; + const older = new Promise((resolve) => { resolveOlder = resolve; }); + const runRead = vi.fn<() => Promise>().mockResolvedValue([]).mockImplementationOnce(() => older); + function TaskRuns({ task = "1" }: { task?: string }) { + const queryRef = `DEMO-${task}`; + const connected = useTaskRunLiveUpdates({ companyId: "company-1", issueId: `issue-${task}`, issueQueryRef: queryRef, assigneeAgentId: "agent-1" }); + useQuery({ queryKey: queryKeys.issues.liveRuns(queryRef), queryFn: runRead, + refetchInterval: (query) => liveRunRefetchInterval(connected, query.state, 1000) }); + return null; + } + const render = async (task = "1") => { + await act(async () => root.render()); + await act(async () => vi.advanceTimersByTimeAsync(1)); + }; + await render(); + const socket = Socket.instances[0]; + await act(async () => socket.onopen?.()); + await act(async () => socket.onmessage?.({ data: JSON.stringify({ id: 1, companyId: "company-1", + type: "heartbeat.run.queued", createdAt: "2020-01-01T00:00:00Z", + payload: { issueId: "issue-1", agentId: "agent-1", runId: "new-run" } }) })); + if (change === "unmount") await act(async () => root.unmount()); + if (change === "disconnect") await act(async () => socket.onerror?.()); + if (change === "company") { companySelection.id = "company-2"; await render(); } + if (change === "task") await render("2"); + if (change === "session") { + const nextSession = { user: { id: "viewer" }, session: { id: "replacement-session", userId: "viewer" } }; + sessionRead.mockResolvedValue(nextSession); + await act(async () => client.setQueryData(queryKeys.auth.session, nextSession)); + await act(async () => vi.advanceTimersByTimeAsync(1)); + } + const readsAfterChange = runRead.mock.calls.length; + const invalidate = vi.spyOn(client, "invalidateQueries"); + await act(async () => resolveOlder([])); + await act(async () => vi.advanceTimersByTimeAsync(1)); + expect(runRead).toHaveBeenCalledTimes(readsAfterChange); + expect(invalidate).not.toHaveBeenCalled(); + }); + + it.each(["focused", "unfocused"])("refreshes once for a start event with two run observers on a %s task view", async focus => { + vi.stubGlobal("WebSocket", Socket); + vi.spyOn(document, "hasFocus").mockReturnValue(focus === "focused"); + routeSelection.pathname = "/DEMO/issues/DEMO-1"; + const queryRef = "DEMO-1"; + client.setQueryData(queryKeys.issues.detail(queryRef), { id: "issue-1", companyId: "company-1", assigneeAgentId: "agent-1" }); + const runRead = vi.fn<() => Promise>().mockResolvedValue([]); + const activeRead = vi.fn<() => Promise>().mockResolvedValue(null); + function Reads({ connected, interval }: { connected: boolean; interval: number }) { + useQuery({ queryKey: queryKeys.issues.liveRuns(queryRef), queryFn: runRead, + refetchInterval: query => liveRunRefetchInterval(connected, query.state, interval) }); + useQuery({ queryKey: queryKeys.issues.activeRun(queryRef), queryFn: activeRead, + refetchInterval: query => liveRunRefetchInterval(connected, query.state, interval) }); + return null; + } + function TaskRuns() { + const connected = useTaskRunLiveUpdates({ companyId: "company-1", issueId: "issue-1", issueQueryRef: queryRef, assigneeAgentId: "agent-1" }); + return <>; + } + await act(async () => root.render()); + await act(async () => vi.advanceTimersByTimeAsync(1)); + const socket = Socket.instances[0]; + await act(async () => socket.onopen?.()); + await act(async () => vi.advanceTimersByTimeAsync(301)); + runRead.mockClear(); activeRead.mockClear(); + await act(async () => socket.onmessage?.({ data: JSON.stringify({ id: 1, companyId: "company-1", + type: "heartbeat.run.queued", createdAt: "2020-01-01T00:00:00Z", + payload: { agentId: "agent-1", runId: "new-run" } }) })); + await act(async () => vi.advanceTimersByTimeAsync(301)); + expect(runRead).toHaveBeenCalledTimes(1); + expect(activeRead).toHaveBeenCalledTimes(1); + }); + + it("keeps polling unassigned tasks even with a healthy shared connection", async () => { + vi.stubGlobal("WebSocket", Socket); + function UnassignedTask() { + const connected = useTaskRunLiveUpdates({ companyId: "company-1", issueId: "issue-1", issueQueryRef: "issue-1", assigneeAgentId: null }); + useQuery({ queryKey: queryKeys.issues.liveRuns("issue-1"), queryFn: pollRead, + refetchInterval: (query) => liveRunRefetchInterval(connected, query.state, 1000) }); + return {connected ? "connected" : "fallback"}; + } + await act(async () => root.render()); + await act(async () => vi.advanceTimersByTimeAsync(1)); + await act(async () => Socket.instances[0].onopen?.()); + expect(container.textContent).toBe("fallback"); + pollRead.mockClear(); + await advancePolls(2); + expect(pollRead.mock.calls.length).toBeGreaterThanOrEqual(2); + }); + + it("retains polling for another company and outside the provider", async () => { + vi.stubGlobal("WebSocket", Socket); + await renderPolling("company-2"); + await act(async () => Socket.instances[0].onopen?.()); + expect(container.textContent).toBe("fallback"); + pollRead.mockClear(); + await advancePolls(2); + expect(pollRead.mock.calls.length).toBeGreaterThanOrEqual(2); + await renderPolling("company-2", false); + expect(container.textContent).toBe("fallback"); + pollRead.mockClear(); + await advancePolls(2); + expect(pollRead.mock.calls.length).toBeGreaterThanOrEqual(2); + }); + + it("drops readiness while hidden and does not poll or retry after unmount", async () => { + vi.stubGlobal("WebSocket", Socket); + await renderPolling(); + await act(async () => Socket.instances[0].onopen?.()); + const visibility = vi.spyOn(document, "visibilityState", "get"); + await act(async () => { + visibility.mockReturnValue("hidden"); + document.dispatchEvent(new Event("visibilitychange")); + }); + expect(container.textContent).toBe("fallback"); + pollRead.mockClear(); + const count = Socket.instances.length; + await advancePolls(5); + expect(pollRead).not.toHaveBeenCalled(); + expect(Socket.instances).toHaveLength(count); + await act(async () => root.unmount()); + await advancePolls(5); + expect(pollRead).not.toHaveBeenCalled(); + expect(Socket.instances).toHaveLength(count); + }); + it("reconciles updates between the initial query and the first socket connection", async () => { vi.stubGlobal("WebSocket", Socket); await render(); diff --git a/ui/src/context/LiveUpdatesProvider.tsx b/ui/src/context/LiveUpdatesProvider.tsx index 5f634a8cdde..d24879ba378 100644 --- a/ui/src/context/LiveUpdatesProvider.tsx +++ b/ui/src/context/LiveUpdatesProvider.tsx @@ -7,6 +7,7 @@ import { useEffect, useMemo, useRef, + useState, type ReactNode, } from "react"; import { @@ -87,6 +88,13 @@ interface LiveEventSubscription { const LiveEventSubscriptionContext = createContext(null); +const CompanyLiveConnectionContext = createContext(null); + +export function useHasCompanyLiveConnection(companyId: string | null | undefined): boolean { + const connectedCompanyId = useContext(CompanyLiveConnectionContext); + return Boolean(companyId && connectedCompanyId === companyId); +} + function dispatchLiveEventToSubscribers( subscribers: Set, expectedCompanyId: string, @@ -126,6 +134,51 @@ export function useCompanyLiveEvent(handler: CompanyLiveEventHandler): void { }, [subscription]); } +export function useTaskRunLiveUpdates(scope: { + companyId: string | null | undefined; + issueId: string | null | undefined; + issueQueryRef: string; + assigneeAgentId: string | null | undefined; +}): boolean { + const connected = useHasCompanyLiveConnection(scope.companyId); + const queryClient = useQueryClient(); + const location = useLocation(); + const refreshOwner = useRef<{ active: boolean } | null>(null); + const refreshRuns = useCallback((coveredByVisibleRoute = false) => { + const owner = refreshOwner.current; + if (!owner?.active) return; + for (const queryKey of [queryKeys.issues.liveRuns(scope.issueQueryRef), queryKeys.issues.activeRun(scope.issueQueryRef)]) { + const query = queryClient.getQueryCache().find({ queryKey }); + if (query?.state.fetchStatus === "fetching") { + void (query.promise ?? queryClient.invalidateQueries({ queryKey }, { cancelRefetch: false })) + .catch(() => undefined).then(() => { + if (owner.active) return queryClient.invalidateQueries({ queryKey }, { cancelRefetch: false }); + }); + } else if (!coveredByVisibleRoute) { + void queryClient.invalidateQueries({ queryKey }, { cancelRefetch: false }); + } + } + }, [queryClient, scope.issueQueryRef]); + useEffect(() => { + const owner = { active: true }; + refreshOwner.current = owner; + if (connected && scope.issueId && scope.assigneeAgentId) refreshRuns(); + return () => { owner.active = false; }; + }, [connected, refreshRuns, scope.companyId, scope.issueId, scope.assigneeAgentId]); + useCompanyLiveEvent((event) => { + if (event.companyId !== scope.companyId || !scope.issueId || + (event.type !== "heartbeat.run.queued" && event.type !== "heartbeat.run.status")) return; + const issueId = readString(event.payload.issueId); + const matches = issueId + ? issueId === scope.issueId || issueId === scope.issueQueryRef + : true; + if (!matches) return; + const visible = resolveVisibleIssueRouteContext(queryClient, location.pathname); + refreshRuns(Boolean(visible?.issueRefs.has(scope.issueQueryRef) && runEventMatchesVisibleIssue(visible, event.payload))); + }); + return connected && Boolean(scope.issueId && scope.assigneeAgentId); +} + function readString(value: unknown): string | null { return typeof value === "string" && value.length > 0 ? value : null; } @@ -476,6 +529,13 @@ function shouldSuppressRunStatusToastForVisibleIssue( ); } +function runEventMatchesVisibleIssue(context: VisibleIssueRouteContext, payload: Record): boolean { + const runId = readString(payload.runId); + const agentId = readString(payload.agentId); + return (runId !== null && context.runIds.has(runId)) || + Boolean(agentId && context.assigneeAgentId && agentId === context.assigneeAgentId); +} + function invalidateVisibleIssueRunQueries( queryClient: QueryClient, pathname: string, @@ -490,13 +550,7 @@ function invalidateVisibleIssueRunQueries( if (!context) return false; const runId = readString(payload.runId); - const agentId = readString(payload.agentId); - const matchesVisibleIssue = - (runId !== null && context.runIds.has(runId)) || - (!!agentId && - !!context.assigneeAgentId && - agentId === context.assigneeAgentId); - if (!matchesVisibleIssue) return false; + if (!runEventMatchesVisibleIssue(context, payload)) return false; const status = readString(payload.status); if (runId && status && TERMINAL_RUN_STATUSES.has(status)) { @@ -1988,7 +2042,9 @@ export function LiveUpdatesProvider({ children }: { children: ReactNode }) { const currentUserId = session?.user?.id ?? session?.session?.userId ?? (health?.deploymentMode === "local_trusted" ? "local-board" : null); const socketAuthKey = session?.session?.id ?? currentUserId ?? "signed_out"; + const [connectedScopeKey, setConnectedScopeKey] = useState(null); const liveCompanyId = resolveLiveCompanyId(selectedCompanyId, selectedCompany?.id ?? null); + const socketScopeKey = JSON.stringify([liveCompanyId, socketAuthKey]); const canConnectSocket = canUseLiveSession(sessionStatus, session != null, health?.deploymentMode) && liveCompanyId !== null; const currentActorRef = useRef<{ userId: string | null; agentId: string | null }>({ userId: currentUserId, @@ -2104,6 +2160,7 @@ export function LiveUpdatesProvider({ children }: { children: ReactNode }) { closeSocketQuietly(nextSocket, "stale_connection"); return; } + setConnectedScopeKey(socketScopeKey); stopPolling(); if (reconnectAttempt > 0) { gateRef.current.suppressUntil = Date.now() + RECONNECT_SUPPRESS_MS; @@ -2155,6 +2212,8 @@ export function LiveUpdatesProvider({ children }: { children: ReactNode }) { }; nextSocket.onerror = () => { + if (closed || socket !== nextSocket) return; + setConnectedScopeKey((current) => current === socketScopeKey ? null : current); // Wait for onclose to drive the reconnect. Self-closing here is what // produces the "closed before connection established" browser noise. }; @@ -2163,6 +2222,7 @@ export function LiveUpdatesProvider({ children }: { children: ReactNode }) { if (socket !== nextSocket) return; socket = null; if (closed) return; + setConnectedScopeKey((current) => current === socketScopeKey ? null : current); startPolling(); scheduleReconnect(); }; @@ -2175,6 +2235,7 @@ export function LiveUpdatesProvider({ children }: { children: ReactNode }) { return () => { closed = true; + setConnectedScopeKey((current) => current === socketScopeKey ? null : current); window.clearTimeout(connectTimer); clearReconnect(); stopPolling(); @@ -2191,11 +2252,16 @@ export function LiveUpdatesProvider({ children }: { children: ReactNode }) { pushToast, canConnectSocket, socketAuthKey, + socketScopeKey, ]); return ( - - {children} - + + + {children} + + ); } diff --git a/ui/src/lib/live-runs-cache.test.ts b/ui/src/lib/live-runs-cache.test.ts index fc388e1ee24..c908e44c590 100644 --- a/ui/src/lib/live-runs-cache.test.ts +++ b/ui/src/lib/live-runs-cache.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from "vitest"; import type { LiveRunForIssue } from "../api/heartbeats"; -import { patchRunStatusInList, removeRunFromList } from "./live-runs-cache"; +import { liveRunRefetchInterval, patchRunStatusInList, removeRunFromList } from "./live-runs-cache"; function run(id: string, status: string): LiveRunForIssue { return { @@ -62,3 +62,19 @@ describe("patchRunStatusInList", () => { expect(next).toBeUndefined(); }); }); + +describe("liveRunRefetchInterval", () => { + it.each(["pending", "error"] as const)("keeps polling %s queries with retained empty data", (status) => { + expect(liveRunRefetchInterval(true, { status, data: [] }, 1000)).toBe(1000); + expect(liveRunRefetchInterval(true, { status, data: null }, 3000)).toBe(3000); + }); + it.each([null, []])("stops known-empty polling with a connected company for %j", (data) => { + expect(liveRunRefetchInterval(true, { status: "success", data }, 1000)).toBe(false); + expect(liveRunRefetchInterval(false, { status: "success", data }, 1000)).toBe(1000); + }); + + it.each([undefined, {}, [run("active", "running")], "invalid", 0])("preserves polling for unknown or active data %j", (data) => { + expect(liveRunRefetchInterval(true, { status: "success", data }, 3000)).toBe(3000); + expect(liveRunRefetchInterval(false, { status: "success", data }, 3000)).toBe(3000); + }); +}); diff --git a/ui/src/lib/live-runs-cache.ts b/ui/src/lib/live-runs-cache.ts index f0533d20c6d..77b09ba29de 100644 --- a/ui/src/lib/live-runs-cache.ts +++ b/ui/src/lib/live-runs-cache.ts @@ -47,3 +47,13 @@ export function patchRunStatusInList( // or its status already matched) so redundant events don't trigger re-renders. return { next: changed ? next : runs, present }; } + +export function liveRunRefetchInterval( + connected: boolean, + state: { status: "pending" | "error" | "success"; data: unknown }, + fallbackMs: number, +): number | false { + return connected && state.status === "success" && (state.data === null || (Array.isArray(state.data) && state.data.length === 0)) + ? false + : fallbackMs; +} diff --git a/ui/src/pages/IssueDetail.tsx b/ui/src/pages/IssueDetail.tsx index abac74816d5..7394e3180c3 100644 --- a/ui/src/pages/IssueDetail.tsx +++ b/ui/src/pages/IssueDetail.tsx @@ -1,3 +1,5 @@ +import { useTaskRunLiveUpdates } from "../context/LiveUpdatesProvider"; +import { liveRunRefetchInterval } from "../lib/live-runs-cache"; import { useTaskRetry } from "../hooks/useTaskRetry"; import { grokBotApi } from "@/api/grokBot"; import { WorkspaceBaseRefRecoveryProvider } from "../components/WorkspaceBaseRefRecovery"; @@ -1204,6 +1206,7 @@ type IssueDetailChatTabProps = { interruption?: Issue["interruption"]; retryPreparing?: boolean; queryIssueId?: string; + hasLiveConnection: boolean; browsers?: import("@paperclipai/shared").TaskBrowser[]; onOpenBrowser?: (browserId: string) => void; onOpenSkill?: (skillId: string, name: string) => void; @@ -1365,6 +1368,7 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ interruption, retryPreparing, queryIssueId, + hasLiveConnection, browsers, onOpenBrowser, onOpenSkill, @@ -1498,7 +1502,7 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ queryKey: queryKeys.issues.liveRuns(issueQueryRef), queryFn: () => heartbeatsApi.liveRunsForIssue(issueQueryRef), enabled: !!issueId, - refetchInterval: 1000, + refetchInterval: (query) => liveRunRefetchInterval(hasLiveConnection, query.state, 1000), placeholderData: keepPreviousDataForSameQueryTail(issueQueryRef), }); @@ -1515,7 +1519,7 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ queryKey: queryKeys.issues.activeRun(issueQueryRef), queryFn: () => heartbeatsApi.activeRunForIssue(issueQueryRef), enabled: activeRunQueryEnabled, - refetchInterval: liveRunCount > 0 ? false : 1000, + refetchInterval: (query) => liveRunCount > 0 ? false : liveRunRefetchInterval(hasLiveConnection, query.state, 1000), placeholderData: keepPreviousDataForSameQueryTail( issueQueryRef, ), @@ -3275,6 +3279,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS !hasLegacyIssueDetailQuery(location.search) )); const resolvedCompanyId = issue?.companyId ?? selectedCompanyId; + const hasLiveConnection = useTaskRunLiveUpdates({ companyId: resolvedCompanyId, issueId: issue?.id ?? issueId, issueQueryRef: issueId ?? "pending", assigneeAgentId: issue?.assigneeAgentId }); const externalObjectsState = useIssueExternalObjects(conversation && !conversation.issue ? null : issue?.id ?? null); // A closed isolated workspace no longer blocks the composer. The server reopens // the workspace when the next comment or resume arrives, so the composer stays @@ -3414,7 +3419,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS queryKey: queryKeys.issues.liveRuns(issueId!), queryFn: () => heartbeatsApi.liveRunsForIssue(issueId!), enabled: !!issueId, - refetchInterval: 3000, + refetchInterval: (query) => liveRunRefetchInterval(hasLiveConnection, query.state, 3000), select: (runs) => runs.length, placeholderData: keepPreviousDataForSameQueryTail( issueId ?? "pending", @@ -3431,7 +3436,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS queryFn: () => heartbeatsApi.activeRunForIssue(issueId!), enabled: !!issueId && (!!issue?.executionRunId || issue?.status === "in_progress"), - refetchInterval: liveRunCount > 0 ? false : 3000, + refetchInterval: (query) => liveRunCount > 0 ? false : liveRunRefetchInterval(hasLiveConnection, query.state, 3000), select: (run) => !!run, placeholderData: keepPreviousDataForSameQueryTail( issueId ?? "pending", @@ -8150,6 +8155,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS interruption={issue.interruption} retryPreparing={taskRetry.optimistic} queryIssueId={issueId ?? issue.id} + hasLiveConnection={hasLiveConnection} onOpenSkill={handleOpenSkill} browsers={browserQuery.data} onOpenBrowser={(id) => { setOpenBrowserId(id); if (isMobile) setMobilePropsOpen(true); else openTaskSidePanel(); }} From 7ebf98eafd2fea1413fa72876ac705d1d59ae37d Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 08:22:53 -0700 Subject: [PATCH 41/68] fix(recovery): avoid duplicate terminal run status notifications (#17614) Retain pending status deliveries when Native or legacy terminal recovery projects the same state. Preserve completion time on unchanged Native terminal status. Real state, metadata, error, explicit patch, and retained-file changes still notify. The publisher acknowledgement, accepted results, retry deadlines, recovery holds, ownership checks, and workspace cleanup remain intact. Correct the queued Retry browser fixture race by releasing its old-process guard at the exact Retry request, requiring a successful response, and retaining fixture cleanup after failure. Validated by 159 focused tests, 653 wider Native recovery tests, six real Chromium/PostgreSQL browser cases, five queued Retry repeats, build/typecheck, exact-head Apex 5/5, and green required CI. The original detached-button CI timeout is documented in the PR. The additional full local repository Vitest command is still running and is not claimed complete. Co-Authored-By: Paperclip --- doc/DATABASE.md | 11 ++ .../legacy-workspace-restore-recovery.test.ts | 63 ++++++++- .../src/services/legacy-execution-recovery.ts | 14 +- .../native-runtime/native-run-finalizer.ts | 43 +++--- .../native-terminal-status-projection.test.ts | 132 ++++++++++++++++++ tests/e2e/legacy-failure-continuation.spec.ts | 26 +++- 6 files changed, 262 insertions(+), 27 deletions(-) create mode 100644 server/src/services/native-runtime/native-terminal-status-projection.test.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 8056e2468c3..e5478530c02 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -246,6 +246,17 @@ The benchmark loads the baseline authorization module with current dependencies, compares complete workspace rows, and prints five warm samples and query plans. It measures SQL execution, not the full repository admission or agent startup. +Native finalization retries preserve a run's terminal completion time when its +status is unchanged. A retry mints another status delivery only when the current +run's status, finalization metadata, phase, error fields, or missing completion +time changes. An unchanged projection retains its existing pending delivery ID, +including after a publish failure. Status delivery stays at least once; retry +deadlines, recovery, accepted results, and workspace cleanup still run normally. +Legacy recovery also retains the pending delivery when status is unchanged and +no patch was supplied. Explicit patches, new terminal statuses, and newly +retained workspace evidence still create a delivery. File retention, ownership +checks, failed write conditions, lock cleanup, and recovery holds still apply. + Agent lists read the complete company roster once and use that same snapshot for termination filtering and organization-chain normalization. Detail and plugin runtime reads load only identity, company, name, status, and manager from other company agents diff --git a/server/src/__tests__/legacy-workspace-restore-recovery.test.ts b/server/src/__tests__/legacy-workspace-restore-recovery.test.ts index 493304a72fe..4629092e10e 100644 --- a/server/src/__tests__/legacy-workspace-restore-recovery.test.ts +++ b/server/src/__tests__/legacy-workspace-restore-recovery.test.ts @@ -1,6 +1,6 @@ import { randomUUID } from "node:crypto"; import { and, eq, sql } from "drizzle-orm"; -import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { agents, companies, createDb, environmentLeases, environments, executionWorkspaces, heartbeatRuns, issueRecoveryActions, issues, projects } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { recordLegacyWorkspaceRestoreFailure, legacyExecutionNeedsReconciliationWithEvidence, terminalizeLegacyExecution } from "../services/legacy-execution-recovery.js"; @@ -8,6 +8,7 @@ import { hasConversationContinuationPolicy, conversationRecoveryActionPredicate import { preserveWorkspaceRestoreRecoveryMetadataSql } from "../services/legacy-workspace-restore-recovery.js"; import { hasConfirmedSandboxStopAndRetain, prepareSandboxStopAndRetain, settleStopOnlyCleanup } from "../services/sandbox-stop-and-retain.js"; import { environmentService } from "../services/environments.js"; +import { deliverExecutionStatuses } from "../services/execution-status-delivery.js"; import { hasRequiredWorkspaceRecovery } from "../services/workspace-restore-recovery-state.js"; const externalTestDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL?.trim(); @@ -45,6 +46,66 @@ const support = externalTestDatabaseUrl ? { supported: true } : await getEmbedde } }; const readLease = async (id: string) => (await db.select().from(environmentLeases).where(eq(environmentLeases.id, id)))[0]; + it("preserves pending and acknowledged status delivery on unpatched legacy recovery retries", async () => { + await deliverExecutionStatuses(db, { publish: vi.fn() }); + const f = await seed(); + const failed = (await terminalizeLegacyExecution({ db, run: f.run, status: "failed", patch }))!; + const holds = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + expect(holds).toHaveLength(1); + expect(hasRequiredWorkspaceRecovery(failed.resultJson)).toBe(true); + const repeated = (await terminalizeLegacyExecution({ db, run: failed, status: "failed" }))!; + expect(repeated.executionStatusDeliveryId).toBe(failed.executionStatusDeliveryId); + const publish = vi.fn(); + await deliverExecutionStatuses(db, { publish }); + await terminalizeLegacyExecution({ db, run: failed, status: "failed" }); + await deliverExecutionStatuses(db, { publish }); + expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(1); + const after = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + expect(after).toHaveLength(1); + expect(after[0]).toMatchObject({ id: holds[0].id, status: holds[0].status, outcome: holds[0].outcome, + ownerType: holds[0].ownerType, wakePolicy: null, evidence: { ...holds[0].evidence, + automaticRecovery: expect.objectContaining({ policy: "preserve_without_replay_v1", replay: "blocked", + runId: f.runId, actionOutcome: "unknown" }), + } }); + expect(await readLease(f.lease.id)).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure" }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].resultJson).toEqual(failed.resultJson); + }); + + it("delivers new retained workspace evidence even when legacy run status is unchanged", async () => { + await deliverExecutionStatuses(db, { publish: vi.fn() }); + const f = await seed(); + const failed = (await terminalizeLegacyExecution({ db, run: f.run, status: "failed", patch }))!; + const publish = vi.fn(); + await deliverExecutionStatuses(db, { publish }); + const [added] = await db.insert(environmentLeases).values({ ...f.lease, id: randomUUID(), providerLeaseId: randomUUID() }).returning(); + const updated = (await terminalizeLegacyExecution({ db, run: failed, status: "failed" }))!; + expect(updated.executionStatusDeliveryId).toBeTruthy(); + expect(updated.executionStatusDeliveryId).not.toBe(failed.executionStatusDeliveryId); + expect(updated.resultJson?.workspaceRestoreRecovery).toMatchObject({ leaseIds: expect.arrayContaining([f.lease.id, added.id]) }); + await deliverExecutionStatuses(db, { publish }); + expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(2); + await terminalizeLegacyExecution({ db, run: updated, status: "failed" }); + await deliverExecutionStatuses(db, { publish }); + expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(2); + }); + + it("retains write conditions and delivers an explicit patch on unchanged legacy status", async () => { + await deliverExecutionStatuses(db, { publish: vi.fn() }); + const f = await seed(); + const failed = (await terminalizeLegacyExecution({ db, run: f.run, status: "failed", patch }))!; + const publish = vi.fn(); + await deliverExecutionStatuses(db, { publish }); + const holds = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + expect(await terminalizeLegacyExecution({ db, run: failed, status: "failed", writeConditions: [sql`false`] })).toBeNull(); + await deliverExecutionStatuses(db, { publish }); + expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(1); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId))).toEqual(holds); + const patched = (await terminalizeLegacyExecution({ db, run: failed, status: "failed", patch: { error: "Retained workspace still needs repair" } }))!; + expect(patched.error).toBe("Retained workspace still needs repair"); + await deliverExecutionStatuses(db, { publish }); + expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(2); + }); + async function stopAndRetain(f: Awaited>, retry = false) { let stopping = (await prepareSandboxStopAndRetain(db, f.lease))!; expect(stopping.status).toBe("pending_cleanup"); diff --git a/server/src/services/legacy-execution-recovery.ts b/server/src/services/legacy-execution-recovery.ts index f1b16bd0e49..5bbea0e61a5 100644 --- a/server/src/services/legacy-execution-recovery.ts +++ b/server/src/services/legacy-execution-recovery.ts @@ -115,7 +115,8 @@ export async function terminalizeLegacyExecution(input: { .set({ status: input.recordRestoreFailureOnly ? current.status : status, ...patch, - executionStatusDeliveryId: randomUUID(), + executionStatusDeliveryId: input.recordRestoreFailureOnly || current.status !== status || Object.keys(patch ?? {}).length > 0 + ? randomUUID() : current.executionStatusDeliveryId, updatedAt: new Date(), }) .where( @@ -131,10 +132,15 @@ export async function terminalizeLegacyExecution(input: { const retainedLeaseIds = await preserveLegacyWorkspaceRestoreSources(tx as unknown as Db, input.recordRestoreFailureOnly ? { ...updated, status: "failed" } : updated, input.workspaceRestoreSource); if (retainedLeaseIds.length) { - [updated] = await tx.update(heartbeatRuns).set({ resultJson: { + const retainedResult = { ...updated.resultJson, - workspaceRestoreRecovery: { schema: LEGACY_WORKSPACE_RECOVERY_SCHEMA, leaseIds: retainedLeaseIds }, - } }).where(and(eq(heartbeatRuns.id, updated.id), eq(heartbeatRuns.companyId, updated.companyId))).returning(); + workspaceRestoreRecovery: { schema: LEGACY_WORKSPACE_RECOVERY_SCHEMA, leaseIds: [...retainedLeaseIds].sort() }, + }; + [updated] = await tx.update(heartbeatRuns).set({ + resultJson: retainedResult, + executionStatusDeliveryId: sql`case when ${heartbeatRuns.resultJson} is distinct from ${JSON.stringify(retainedResult)}::jsonb + then ${randomUUID()}::uuid else ${heartbeatRuns.executionStatusDeliveryId} end`, + }).where(and(eq(heartbeatRuns.id, updated.id), eq(heartbeatRuns.companyId, updated.companyId))).returning(); } const needsRecovery = input.recordRestoreFailureOnly ? retainedLeaseIds.length > 0 || hasRequiredWorkspaceRecovery(updated.resultJson) diff --git a/server/src/services/native-runtime/native-run-finalizer.ts b/server/src/services/native-runtime/native-run-finalizer.ts index 7d60d73e0d7..805fd623d18 100644 --- a/server/src/services/native-runtime/native-run-finalizer.ts +++ b/server/src/services/native-runtime/native-run-finalizer.ts @@ -1608,33 +1608,42 @@ export async function finalizeNativeRun(input: { statusVersionAfter: Number(committed.issue.statusVersion), workspaceFinalizeStatus: input.workspaceFinalizeStatus, }; + const projectedStatus = terminalState === "succeeded" ? "succeeded" as const + : terminalState === "cancelled" ? "cancelled" as const : "failed" as const; + const projectedResult = sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) + || ${JSON.stringify(finalizationMetadata)}::jsonb + || ${clearExecutionFailure ? recoveredExecutionFailureMetadata() : sql`'{}'::jsonb`}`; + const projectedError = providerFailure ? sql`coalesce(${heartbeatRuns.error}, ${providerFailure.errorMessage})` + : clearExecutionFailure ? sql`null` : heartbeatRuns.error; + const projectedErrorCode = providerFailure ? sql`coalesce(${heartbeatRuns.errorCode}, ${providerFailure.errorCode})` + : clearExecutionFailure ? sql`null` : heartbeatRuns.errorCode; + const projectionChanged = or( + sql`${heartbeatRuns.nativePhase} is distinct from ${finalizationPhase}`, + sql`${heartbeatRuns.resultJson} is distinct from ${projectedResult}`, + sql`${heartbeatRuns.error} is distinct from ${projectedError}`, + sql`${heartbeatRuns.errorCode} is distinct from ${projectedErrorCode}`, + ...(input.projectRunStatus ? [ + sql`${heartbeatRuns.status} is distinct from ${projectedStatus}`, + isNull(heartbeatRuns.finishedAt), + ] : []), + ); const projectRun = (tx: Db) => tx .update(heartbeatRuns) .set({ - executionStatusDeliveryId: randomUUID(), + executionStatusDeliveryId: sql`case when ${projectionChanged} + then ${randomUUID()}::uuid else ${heartbeatRuns.executionStatusDeliveryId} end`, ...(input.projectRunStatus ? { - status: - terminalState === "succeeded" - ? "succeeded" - : terminalState === "cancelled" - ? "cancelled" - : "failed", - finishedAt: now, + status: projectedStatus, + finishedAt: sql`case when ${heartbeatRuns.status} is distinct from ${projectedStatus} + then ${now.toISOString()}::timestamptz else coalesce(${heartbeatRuns.finishedAt}, ${now.toISOString()}::timestamptz) end`, } : {}), nativePhase: finalizationPhase, nativePhaseUpdatedAt: now, ...(clearExecutionFailure ? { error: null, errorCode: null } : {}), - ...(providerFailure ? { - // Recovery can finalize before heartbeat saves the adapter result. - // Retain any independently recorded execution/cleanup failure. - error: sql`coalesce(${heartbeatRuns.error}, ${providerFailure.errorMessage})`, - errorCode: sql`coalesce(${heartbeatRuns.errorCode}, ${providerFailure.errorCode})`, - } : {}), - resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) - || ${JSON.stringify(finalizationMetadata)}::jsonb - || ${clearExecutionFailure ? recoveredExecutionFailureMetadata() : sql`'{}'::jsonb`}`, + ...(providerFailure ? { error: projectedError, errorCode: projectedErrorCode } : {}), + resultJson: projectedResult, updatedAt: now, }) .where(eq(heartbeatRuns.id, run.id)) diff --git a/server/src/services/native-runtime/native-terminal-status-projection.test.ts b/server/src/services/native-runtime/native-terminal-status-projection.test.ts new file mode 100644 index 00000000000..ec23c45f1b6 --- /dev/null +++ b/server/src/services/native-runtime/native-terminal-status-projection.test.ts @@ -0,0 +1,132 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, companies, completionContracts, createDb, heartbeatRuns, issues, + nativeRunFinalizations, nativeRunResults } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { deliverExecutionStatuses } from "../execution-status-delivery.js"; +import { buildIssueMonitorTriggeredPatch, normalizeIssueExecutionPolicy } from "../issue-execution-policy.js"; +import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js"; +import { finalizeNativeRun } from "./native-run-finalizer.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("native terminal status projection", () => { + let temporary: Awaited>; + let db: ReturnType; + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("native-terminal-status-"); + db = createDb(temporary.connectionString); + }); + afterAll(async () => { await temporary?.cleanup(); }); + + async function fixture() { + await deliverExecutionStatuses(db, { publish: vi.fn() }); + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); + const contractId = randomUUID(), resultId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Monitor status", issuePrefix: `S${companyId.slice(0, 7)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "paperclip_runner", status: "active" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Saved check", status: "in_progress", assigneeAgentId: agentId }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issueId, + runtimeMode: "native", status: "running", contextSnapshot: { issueId } }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId }); + await authority.execute({ tool: "set_task_monitor", callId: randomUUID(), arguments: { + idempotencyKey: "saved-monitor", monitor: { nextCheckAt: new Date(Date.now() + 60_000).toISOString(), notes: "Check the saved result." }, + } }); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", + contractJson: { revision: "test", criteria: [] }, canonicalSha256: randomUUID(), + createdByActorType: "system", createdByActorId: "test" }); + const accepted = { result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", + summary: "Wait for the saved check.", completionClaim: { contractRevision: "test", objectiveSatisfied: false, + criteria: [], remainingWork: [{ description: "Check again", blocksCompletion: true }] }, + continuation: { kind: "monitor", summary: "Wait for the check", idempotencyKey: "provider-selected" }, + evidence: [], verification: [], attentionRequests: [], artifacts: [] }, + terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState: "completed", reportedWorkDisposition: "yielded" } }; + const finishedAt = new Date("2020-01-01T00:00:00Z"); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt, completionContractId: contractId }).where(eq(heartbeatRuns.id, runId)); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId, completionContractId: contractId, + serverFingerprint: randomUUID(), canonicalSha256: randomUUID(), schemaStatus: "accepted", resultJson: accepted }); + await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId, phase: "retryable_failure", resultId }); + const [task] = await db.select().from(issues).where(eq(issues.id, issueId)); + await db.update(issues).set({ ...buildIssueMonitorTriggeredPatch({ issue: task!, + policy: normalizeIssueExecutionPolicy(task!.executionPolicy), triggeredAt: new Date() }), executionRunId: null, checkoutRunId: null, + }).where(eq(issues.id, issueId)); + const finalize = () => finalizeNativeRun({ db, runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + const read = async () => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]!; + return { runId, companyId, issueId, finishedAt, resultId, accepted, finalize, read }; + } + + it("preserves the original completion time while retrying an accepted monitor result", async () => { + const f = await fixture(); + await f.finalize(); + expect((await f.read()).finishedAt).toEqual(f.finishedAt); + await f.finalize(); + expect((await f.read()).finishedAt).toEqual(f.finishedAt); + const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(coordinator).toMatchObject({ phase: "retryable_failure", failureCode: "monitor_wait_authority_lost", resultId: f.resultId }); + expect(coordinator.nextAttemptAt!.getTime()).toBeGreaterThan(Date.now()); + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0].resultJson).toEqual(f.accepted); + }); + + it("retains an unacknowledged status delivery across unchanged finalization retries", async () => { + const f = await fixture(); + await f.finalize(); + const before = await f.read(); + expect(before.executionStatusDeliveryId).toBeTruthy(); + await f.finalize(); + expect((await f.read()).executionStatusDeliveryId).toBe(before.executionStatusDeliveryId); + const publish = vi.fn(); + await expect(deliverExecutionStatuses(db, { publish, failpoint: () => { throw new Error("after publish"); } })).rejects.toThrow("after publish"); + await f.finalize(); + expect((await f.read()).executionStatusDeliveryId).toBe(before.executionStatusDeliveryId); + await deliverExecutionStatuses(db, { publish }); + const deliveries = publish.mock.calls.filter(([event]) => event.payload.runId === f.runId); + expect(deliveries).toHaveLength(2); + expect(deliveries[0][0].payload.deliveryId).toBe(deliveries[1][0].payload.deliveryId); + }); + + it("preserves a pending status notification created by another producer", async () => { + const f = await fixture(); + await f.finalize(); + const newer = randomUUID(); + await db.update(heartbeatRuns).set({ executionStatusDeliveryId: newer }).where(eq(heartbeatRuns.id, f.runId)); + await f.finalize(); + expect((await f.read()).executionStatusDeliveryId).toBe(newer); + }); + + it.each(["metadata", "status", "missing_completion_time"])("broadcasts a changed terminal projection after acknowledgement (%s)", async change => { + const f = await fixture(); + await f.finalize(); + const publish = vi.fn(); + await deliverExecutionStatuses(db, { publish }); + if (change === "metadata") { + await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "failed", projectRunStatus: true }); + expect((await f.read()).resultJson?.workspaceFinalizeStatus).toBe("failed"); + } else { + await db.update(heartbeatRuns).set(change === "status" ? { status: "failed" } : { finishedAt: null }) + .where(eq(heartbeatRuns.id, f.runId)); + await f.finalize(); + expect(await f.read()).toMatchObject({ status: "succeeded", finishedAt: expect.any(Date) }); + } + await deliverExecutionStatuses(db, { publish }); + expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(2); + }); + + it("does not broadcast an unchanged terminal projection after acknowledgement or a new DB client", async () => { + const f = await fixture(); + await f.finalize(); + const publish = vi.fn(); + await deliverExecutionStatuses(db, { publish }); + await f.finalize(); + await deliverExecutionStatuses(db, { publish }); + const reopened = createDb(temporary.connectionString); + try { + await finalizeNativeRun({ db: reopened, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + await deliverExecutionStatuses(reopened, { publish }); + } finally { await reopened.$client.end(); } + expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(1); + expect((await f.read()).executionStatusDeliveryId).toBeNull(); + }); +}); diff --git a/tests/e2e/legacy-failure-continuation.spec.ts b/tests/e2e/legacy-failure-continuation.spec.ts index 4eaa73092f9..b04ca8e1e22 100644 --- a/tests/e2e/legacy-failure-continuation.spec.ts +++ b/tests/e2e/legacy-failure-continuation.spec.ts @@ -92,8 +92,18 @@ for (const action of ["task_retry", "retry_after_reload", "inbox_retry", "messag await expect(page.getByRole("button", { name: "Steer", exact: true })).toHaveCount(0); const retry = page.getByRole("button", { name: "Retry", exact: true }); await expect(retry).toBeEnabled(); - await db.update(heartbeatRuns).set({ processPid: 999999999 }).where(eq(heartbeatRuns.id, sourceRunId)); - await retry.click(); + await page.route((url) => url.pathname === `/api/issues/${issue.id}/retry`, async route => { + if (route.request().method() !== "POST") return route.continue(); + await db.update(heartbeatRuns).set({ processPid: 999999999 }).where(eq(heartbeatRuns.id, sourceRunId)); + await route.continue(); + }, { times: 1 }); + const [response] = await Promise.all([ + page.waitForResponse(response => response.request().method() === "POST" && + new URL(response.url()).pathname === `/api/issues/${issue.id}/retry` && + response.request().postDataJSON()?.observedRunId === sourceRunId), + retry.click(), + ]); + expect(response.ok(), `${response.status()} ${await response.text()}`).toBe(true); } else if (action === "automatic_message") { // No Retry, duplicate message, or status change: the saved input runs. } else if (action === "message") { @@ -131,9 +141,15 @@ for (const action of ["task_retry", "retry_after_reload", "inbox_retry", "messag await page.reload(); await expect(page.getByText("Answered the pending follow-up once.", { exact: false })).toBeVisible(); } finally { - await request.patch(`/api/companies/${company.id}`, { data: { status: "archived" } }); - await closeRegisteredClients(url); - await rm(root, { recursive: true, force: true }); + try { + await request.patch(`/api/companies/${company.id}`, { data: { status: "archived" } }); + } finally { + try { + await closeRegisteredClients(url); + } finally { + await rm(root, { recursive: true, force: true }); + } + } } }); } From 34409a92c4c1b4ff18299286d8228563bc098bec Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 08:51:39 -0700 Subject: [PATCH 42/68] fix(native-runtime): stop failover backups from carrying throwaway provider state (#17618) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runner sessions on managed sandboxes keep a failover backup of provider state, so a replacement sandbox can resume the session > - Each backup copies the provider's runtime directory, with a per-directory sha256 manifest, minus `excludeEntries` > - Codex already excludes credentials and re-downloadable plugin caches; OpenCode excluded nothing, and ACPX excluded only per-agent credentials > - OpenCode also leaked a temporary HOME on every launch, because removing it failed on the read-only skill snapshot inside and the error was swallowed > - So every backup carried every earlier launch's HOME (with its npm cache) plus re-installable plugin `node_modules` > - This pull request removes those homes correctly, sweeps homes left by earlier versions, and excludes the throwaway entries from OpenCode and ACPX backups > - The benefit is much smaller backups and live session directories, while session history is still backed up and restored ## Linked Issues or Issue Description Bug report described in-PR (bug report template). **What happened?** On one instance, current native failover backups totalled 24 GB across about 300 sessions. For OpenCode sessions: - `opencode//home-XXXXXX/`: 170 per-launch HOME directories across 67 sessions, 9.3 GB. Each held `.npm` (about 111 MB of npm cache), `.claude` (the skill snapshot) and `.local`. - `opencode//config/opencode/node_modules`: 68 sessions, 4.3 GB of OpenCode plugin installs. One ACPX session also carried `acpx//data/pnpm/store` (229 MB in the current backup, several GB in older copies). Cause of the leaked homes: `startRuntime` in the OpenCode driver creates a fresh `mkdtemp` HOME for each launch and seals the assigned skills inside it (directories `0555`, files `0444`). `close()` and the startup-failure paths removed it with `rm(..., { recursive: true, force: true }).catch(() => undefined)`. That fails with `EACCES` on the sealed directories, and the failure was ignored, so each launch left its HOME behind. **Expected behavior** Per-launch homes are removed when the runtime closes, and backups hold only state needed to resume a session. Caches that the provider re-creates are left out. **Steps to reproduce** 1. Run an OpenCode native session with at least one assigned skill on a managed sandbox, over several turns or runs. 2. Inspect `/` in the OpenCode runtime directory: one `home-XXXXXX` per launch remains. 3. Inspect `failover-backups/current/opencode//`: the same homes and `config/opencode/node_modules` are in the backup. **Paperclip version or commit:** `master` at e3352655687. **Deployment mode:** any with native OpenCode/ACPX sessions on managed sandboxes. **Agent adapter(s) involved:** OpenCode (native), ACPX. ## What Changed - OpenCode driver (`packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts`): - `removeIsolatedHome` unseals the skill snapshot with the existing `releaseMaterializedNativeRuntimeSkills`, then removes the HOME. Close and every startup-failure path use it. It only unseals when `.claude` is a real directory, not a symlink. - Per-launch homes are now created under `/homes/home-XXXXXX`, so one exact path can be excluded. - When a session opens, `home-*` directories left at the session root by earlier versions are removed. History is not in them: OpenCode keeps it in `XDG_DATA_HOME` (`/data/opencode/opencode.db`), and HOME only holds caches, state locks and the skill copy. - Exports `opencodeRuntimeSessionDirectoryName` (the existing session-directory naming) and `OPENCODE_ISOLATED_HOMES_DIRECTORY` for the server. - `resolveNativeHarnessPersistenceProfile` (`server/src/services/native-runtime/native-session-executor.ts`): - OpenCode excludes `/homes` and `/config/opencode/node_modules`. - ACPX (every agent) excludes `acpx//home/.npm` and `acpx//data/pnpm/store`, in addition to the existing per-agent entries. - `excludeEntries` are matched as exact relative paths (wildcards are rejected), so the session directory is computed in the same way as the existing ACPX entries. - Exposes the two new exports through the server's runner vendor shim. No code comments were added. ## Verification - `pnpm exec vitest run src/drivers/opencode src/drivers/runtime-context-materializer` (in `packages/paperclip-runner/`): 108 passed. The 1 failure, `uses provider structure rather than prose length to select a post-tool answer`, also fails on unmodified `master`. - The existing skill-tree test, with and without a startup retry, now asserts that homes are created under `homes/` and that `homes/` is empty after close. - New test: a sealed legacy `home-*` at the session root is removed when the session reopens, and `data/opencode/opencode.db` is kept. - `pnpm exec vitest run src/services/native-runtime/ src/__tests__/native-harness-backup-expiration.test.ts` (in `server/`): 5080 passed. `plugin-runner-probe` could not run locally because of a cargo manifest-path problem in this environment; it is unrelated to this change. - Profile tests cover the new OpenCode and ACPX exclusions, including a session id that needs sanitizing. - New checkpoint/restore round-trip tests for OpenCode and ACPX. They use `syncRemoteRunnerDirectoryOut` for the backup and `stageRemoteRunnerDirectory` for the replacement. Excluded caches are absent from both the backup and the replacement, while history, config and workspace files are restored and the live directory is unchanged. - **Mutation checks**, each failing its test: - removing the skill unsealing (homes leak again; 3 runner tests fail); - removing the legacy sweep (1 test fails); - emptying the OpenCode/ACPX exclusion lists (4 server tests fail). - Checked by hand against the bundled OpenCode 1.18.34 binary, with the driver's environment: - launching with a fresh HOME and no `config/opencode/node_modules` still lists earlier sessions; - OpenCode reinstalls the plugin dependencies on launch. - `pnpm run typecheck` in `server/` and `packages/paperclip-runner/`: pass. - Not run: server build and boot smoke. ## Risks - **Rollout:** existing backups still contain legacy `home-*` directories. After a restore, the sweep removes them when the session opens, and the next backup no longer carries them. - **First launch after restore:** OpenCode reinstalls its plugin dependencies and npm repopulates its cache. This needs network access to the npm registry, the same as a brand-new session. - **Crash leftovers:** a HOME left behind by a hard crash (the runner killed before `close()`) stays under `homes/` until the session directory is removed. It is still excluded from backups. Homes under `homes/` are not swept on open, to avoid touching a HOME that a concurrently running runtime might still use. - **pnpm store:** dropping the pnpm store from ACPX backups means the next `pnpm install` in a replacement sandbox downloads packages again. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, tests, and running the real OpenCode binary to check what state it reads on resume). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- .../opencode/opencode-server-driver.test.ts | 55 +++++++- .../opencode/opencode-server-driver.ts | 64 ++++++--- .../native-session-executor.test.ts | 125 +++++++++++++++++- .../native-runtime/native-session-executor.ts | 24 +++- server/src/vendor/paperclip-runner/index.ts | 4 + 5 files changed, 247 insertions(+), 25 deletions(-) diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts index a63abc777f8..5d568eddf86 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts @@ -1513,7 +1513,7 @@ describe("OpenCodeServerDriver", () => { ); const isolatedHome = runtimeEnvironment.home; expect(startupFailurePending).toBe(false); - expect(dirname(isolatedHome)).toBe(sessionRoot); + expect(dirname(isolatedHome)).toBe(join(sessionRoot, "homes")); expect(basename(isolatedHome)).toMatch(/^home-/); await expect( readFile( @@ -1553,6 +1553,59 @@ describe("OpenCodeServerDriver", () => { readFile(join(instructionRoot, "sibling.md"), "utf8"), ).resolves.toBe("instruction sibling\n"); await session.close({ reason: "test" }); + await expect(readdir(join(sessionRoot, "homes"))).resolves.toEqual([]); + }); + + it("removes isolated homes left by earlier launches when a session reopens", async () => { + await chmod(fixture, 0o755); + const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-legacy-home-")); + const workspace = await mkdtemp( + join(tmpdir(), "paperclip-opencode-legacy-home-workspace-"), + ); + const skillRoot = join(root, "skill-source"); + const instructionRoot = join(root, "instruction-source"); + roots.push(root, workspace); + await Promise.all([ + mkdir(skillRoot, { recursive: true }), + mkdir(instructionRoot, { recursive: true }), + ]); + await writeFile(join(skillRoot, "SKILL.md"), "# Assigned\n"); + await writeFile(join(instructionRoot, "AGENTS.md"), "instructions\n"); + const sessionRoot = join(root, "legacy"); + const legacyHome = join(sessionRoot, "home-AbC123"); + const sealedSkill = join(legacyHome, ".claude", "skills", "assigned"); + await mkdir(sealedSkill, { recursive: true }); + await mkdir(join(legacyHome, ".npm", "_cacache"), { recursive: true }); + await writeFile(join(legacyHome, ".npm", "_cacache", "entry"), "cache"); + await writeFile(join(sealedSkill, "SKILL.md"), "# Assigned\n"); + await chmod(join(sealedSkill, "SKILL.md"), 0o444); + await chmod(sealedSkill, 0o555); + await mkdir(join(sessionRoot, "data", "opencode"), { recursive: true }); + await writeFile(join(sessionRoot, "data", "opencode", "opencode.db"), "history"); + const driver = new OpenCodeServerDriver({ + model: "openrouter/deepseek/deepseek-v4-flash-0731", + runtimeDirectory: root, + command: fixture, + runtimeContext: runtimeContext(skillRoot, instructionRoot), + environment: { + PATH: process.env.PATH, + OPENROUTER_API_KEY: "fixture-key", + }, + }); + const session = await driver.openSession({ + runId: "run-legacy-home", + normalizedSessionId: "legacy", + workingDirectory: workspace, + }); + await expect(lstat(legacyHome)).rejects.toMatchObject({ code: "ENOENT" }); + await session.close({ reason: "test" }); + expect( + (await readdir(sessionRoot)).filter((entry) => entry.startsWith("home-")), + ).toEqual([]); + await expect(readdir(join(sessionRoot, "homes"))).resolves.toEqual([]); + await expect( + readFile(join(sessionRoot, "data", "opencode", "opencode.db"), "utf8"), + ).resolves.toBe("history"); }); it("resends system instructions without repeating the task envelope on recovery", async () => { diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts index a7f80aac85e..4ab25b38b83 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts @@ -6,6 +6,7 @@ import { accessSync, constants, readFileSync, realpathSync, statSync } from "nod import { createRequire } from "node:module"; import { chmod, + lstat, mkdir, mkdtemp, readFile, @@ -85,7 +86,10 @@ import { type OpenCodeMcpBridge, } from "./mcp-bridge.js"; import { nativeMcpLaunchBinding } from "../native-mcp.js"; -import { materializeNativeRuntimeSkills } from "../runtime-context-materializer.js"; +import { + materializeNativeRuntimeSkills, + releaseMaterializedNativeRuntimeSkills, +} from "../runtime-context-materializer.js"; export const OPENCODE_SERVER_DRIVER_KIND = "opencode_server" as const; export const QUALIFIED_OPENCODE_VERSION = "1.18.34" as const; @@ -369,6 +373,7 @@ export class OpenCodeServerDriver implements HarnessDriver { ); await mkdir(root, { recursive: true, mode: 0o700 }); await writeFile(join(root, "workspace"), `${cwd}\n`, { mode: 0o600 }); + await removeLegacyIsolatedHomes(root); const trace = await createProviderTraceFileSink({ path: this.#options.environment?.PAPERCLIP_PROVIDER_TRACE_PATH, provider: "opencode", @@ -2281,7 +2286,9 @@ async function startRuntime(input: { // HOME contains a read-only skill snapshot, whose destination must be fresh. // Keep OpenCode's XDG data stable for provider-session recovery while giving // every launch (including retries and recovery) a new isolated HOME. - const isolatedHome = await mkdtemp(join(input.root, "home-")); + const homesRoot = join(input.root, OPENCODE_ISOLATED_HOMES_DIRECTORY); + await mkdir(homesRoot, { recursive: true, mode: 0o700 }); + const isolatedHome = await mkdtemp(join(homesRoot, "home-")); await chmod(isolatedHome, 0o700); try { await materializeNativeRuntimeSkills( @@ -2289,18 +2296,14 @@ async function startRuntime(input: { join(isolatedHome, ".claude", "skills"), ); } catch (error) { - await rm(isolatedHome, { recursive: true, force: true }).catch( - () => undefined, - ); + await removeIsolatedHome(isolatedHome).catch(() => undefined); throw error; } const bridge = await startOpenCodeMcpBridge({ tools: input.options.dynamicTools, handler: input.dispatch, }).catch(async (error) => { - await rm(isolatedHome, { recursive: true, force: true }).catch( - () => undefined, - ); + await removeIsolatedHome(isolatedHome).catch(() => undefined); throw error; }); const assignedMcp = nativeMcpLaunchBinding( @@ -2335,7 +2338,7 @@ async function startRuntime(input: { const providerProxy = modelProvider === "paperclip" && input.options.environment?.PAPERCLIP_AI_PROVIDER_URL ? await startOpenCodeProviderProxy(input.options.environment.PAPERCLIP_AI_PROVIDER_URL, input.options.environment.PAPERCLIP_AI_PROVIDER_KEY ?? "", providerModelId, input.options.environment).catch(async error => { await bridge.close().catch(() => {}); - await rm(isolatedHome, { recursive: true, force: true }).catch(() => {}); + await removeIsolatedHome(isolatedHome).catch(() => {}); throw error; }) : null; @@ -2568,9 +2571,7 @@ async function startRuntime(input: { await rm(join(configHome, "opencode", "opencode.json"), { force: true, }).catch(() => undefined); - await rm(isolatedHome, { recursive: true, force: true }).catch( - () => undefined, - ); + await removeIsolatedHome(isolatedHome).catch(() => undefined); if (closeInput.finalizeTrace !== false) { await input.trace?.finish({ reason: closeInput.reason ?? null }); } @@ -2585,9 +2586,7 @@ async function startRuntime(input: { await rm(join(configHome, "opencode", "opencode.json"), { force: true, }).catch(() => undefined); - await rm(isolatedHome, { recursive: true, force: true }).catch( - () => undefined, - ); + await removeIsolatedHome(isolatedHome).catch(() => undefined); throw error; } } @@ -2962,8 +2961,9 @@ function sanitizedEnvironmentKeys(): string[] { ]; } -function sessionRoot( - runtimeDirectory: string, +export const OPENCODE_ISOLATED_HOMES_DIRECTORY = "homes"; + +export function opencodeRuntimeSessionDirectoryName( normalizedSessionId: string, ): string { const safe = normalizedSessionId @@ -2971,7 +2971,35 @@ function sessionRoot( .slice(0, 120); if (!safe || safe === "." || safe === "..") throw new Error("Invalid normalized OpenCode session id"); - return join(resolve(runtimeDirectory), safe); + return safe; +} + +function sessionRoot( + runtimeDirectory: string, + normalizedSessionId: string, +): string { + return join( + resolve(runtimeDirectory), + opencodeRuntimeSessionDirectoryName(normalizedSessionId), + ); +} + +async function removeIsolatedHome(home: string): Promise { + const skillsParent = await lstat(join(home, ".claude")).catch(() => null); + if (skillsParent?.isDirectory()) { + await releaseMaterializedNativeRuntimeSkills( + join(home, ".claude", "skills"), + ).catch(() => undefined); + } + await rm(home, { recursive: true, force: true }); +} + +async function removeLegacyIsolatedHomes(root: string): Promise { + const entries = await readdir(root, { withFileTypes: true }).catch(() => []); + for (const entry of entries) { + if (!entry.isDirectory() || !entry.name.startsWith("home-")) continue; + await removeIsolatedHome(join(root, entry.name)).catch(() => undefined); + } } function validateWorkspace(value: string): string { diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index bfb1f4f309f..5c8c3e211f7 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -1880,13 +1880,35 @@ describe("native harness persistence profiles", () => { }); }); - it("excludes only nested Codex launch state from ACPX recovery", () => { + it("excludes per-launch OpenCode homes and re-installable plugins from OpenCode recovery", () => { + const opencode = resolveNativeHarnessPersistenceProfile({ + provider: { kind: "opencode" }, + session: { + driverKind: "opencode_server", + normalizedSessionId: "company:agent/session 1", + protocolVersion: 1, + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, + }, + } as unknown as NativeExecutionInputV1); + expect( + opencode.directories.find((directory) => directory.name === "opencode"), + ).toMatchObject({ + excludeEntries: [ + "company_agent_session_1/homes", + "company_agent_session_1/config/opencode/node_modules", + ], + }); + }); + + it("excludes only nested Codex launch state and package caches from ACPX recovery", () => { const acpx = profile({ kind: "acpx", agent: "codex" }, "acpx_runtime"); const sessionDirectory = acpxRuntimeSessionDirectoryName("session"); expect( acpx.directories.find((directory) => directory.name === "acpx"), ).toMatchObject({ excludeEntries: [ + `acpx/${sessionDirectory}/home/.npm`, + `acpx/${sessionDirectory}/data/pnpm/store`, `acpx/${sessionDirectory}/codex-home/tmp`, `acpx/${sessionDirectory}/codex-home/.tmp`, `acpx/${sessionDirectory}/codex-home/auth.json`, @@ -1907,7 +1929,12 @@ describe("native harness persistence profiles", () => { { kind: "acpx", agent: "claude" }, "acpx_runtime", ).directories.find((directory) => directory.name === "acpx"), - ).toMatchObject({ excludeEntries: [] }); + ).toMatchObject({ + excludeEntries: [ + `acpx/${sessionDirectory}/home/.npm`, + `acpx/${sessionDirectory}/data/pnpm/store`, + ], + }); }); }); @@ -2623,6 +2650,100 @@ describe("remote provider checkpoint snapshots", () => { expect(await readFile(join(home, "logs", "unified.jsonl"), "utf8")).toBe("AUTH_SENTINEL"); } finally { await rm(root, { recursive: true, force: true }); } }); + it.each<{ + name: string; + provider: Record; + driverKind: string; + directory: string; + sessionPath: string; + kept: Record; + omitted: Record; + }>([ + { + name: "OpenCode", + provider: { kind: "opencode" }, + driverKind: "opencode_server", + directory: "opencode", + sessionPath: "company_agent_session_1", + kept: { + "workspace": "/workspace\n", + "data/opencode/opencode.db": "history", + "config/opencode/opencode.json": "{}", + "config/opencode/package.json": "{}", + "home-legacy/.local/state/opencode/locks/lock": "legacy", + }, + omitted: { + "homes/home-AbC123/.npm/_cacache/index": "npm-cache", + "homes/home-AbC123/.claude/skills/assigned/SKILL.md": "skill", + "config/opencode/node_modules/@opencode-ai/plugin/package.json": "plugin", + }, + }, + { + name: "ACPX", + provider: { kind: "acpx", agent: "pi" }, + driverKind: "acpx_runtime", + directory: "acpx", + sessionPath: `acpx/${acpxRuntimeSessionDirectoryName("company:agent/session 1")}`, + kept: { + "workspace": "/workspace\n", + "pi-home/sessions/session.jsonl": "history", + "data/pnpm/config.json": "{}", + "home/.profile": "profile", + }, + omitted: { + "home/.npm/_cacache/index": "npm-cache", + "data/pnpm/store/v10/files/00/blob": "pnpm-store", + }, + }, + ])("omits disposable $name package caches from the checkpoint and restores the session without them", async ({ + provider, driverKind, directory, sessionPath, kept, omitted, + }) => { + const root = await mkdtemp(join(tmpdir(), "provider-cache-checkpoint-")); + try { + const live = join(root, "live", directory); + const backup = join(root, "backup", directory); + const replacement = join(root, "replacement", directory); + const files: Record = { ...kept, ...omitted }; + for (const [path, contents] of Object.entries(files)) { + await mkdir(join(live, sessionPath, path, ".."), { recursive: true }); + await writeFile(join(live, sessionPath, path), contents); + } + const persistence = resolveNativeHarnessPersistenceProfile({ + provider, + session: { normalizedSessionId: "company:agent/session 1", driverKind }, + } as unknown as NativeExecutionInputV1); + const excludeEntries = persistence.directories.find(entry => entry.name === directory)!.excludeEntries; + const execute = async ({ command, args }: { command: string; args: string[] }) => ({ + exitCode: 0, timedOut: false, stdout: execFileSync(command, args, { encoding: "utf8" }), stderr: "", + }); + const copy = async (operations: Array<{ files: Array<{ sourcePath: string; targetPath: string }> }>) => { + for (const operation of operations) for (const file of operation.files) + await cp(file.sourcePath, file.targetPath, { recursive: true }); + }; + await syncRemoteRunnerDirectoryOut({ runner: { execute, syncOut: copy } as never, + sourcePath: live, targetPath: backup, mode: 0o700, excludeEntries }); + for (const [path, contents] of Object.entries(kept)) + expect(await readFile(join(backup, sessionPath, path), "utf8")).toBe(contents); + for (const path of Object.keys(omitted)) + await expect(access(join(backup, sessionPath, path))).rejects.toMatchObject({ code: "ENOENT" }); + for (const [path, contents] of Object.entries(omitted)) + expect(await readFile(join(live, sessionPath, path), "utf8")).toBe(contents); + + await stageRemoteRunnerDirectory({ + target: { kind: "remote", transport: "provider", remoteCwd: root, runner: { syncIn: copy } } as never, + runner: { syncIn: copy } as never, + sourcePath: backup, + targetPath: replacement, + mode: 0o700, + excludeEntries, + }); + for (const [path, contents] of Object.entries(kept)) + expect(await readFile(join(replacement, sessionPath, path), "utf8")).toBe(contents); + for (const path of Object.keys(omitted)) + await expect(access(join(replacement, sessionPath, path))).rejects.toMatchObject({ code: "ENOENT" }); + } finally { await rm(root, { recursive: true, force: true }); } + }); + it("keeps Codex session state while omitting re-downloadable remote plugins from the checkpoint", async () => { const root = await mkdtemp(join(tmpdir(), "codex-plugin-checkpoint-")); try { diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 62118309e9d..7f67a8aa2fc 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -109,6 +109,8 @@ import { completeRetainedNativeSessionCleanup, completeTerminatedLocalNativeSessionCleanup, acpxRuntimeSessionDirectoryName, + OPENCODE_ISOLATED_HOMES_DIRECTORY, + opencodeRuntimeSessionDirectoryName, createNativeSessionBackend, createRunnerdCodexTransport, executeNativeSession, @@ -410,6 +412,14 @@ const CODEX_HOME_CHECKPOINT_EXCLUDED_ENTRIES = [ ...CODEX_HOME_NON_PERSISTENT_ENTRIES, ...CODEX_HOME_REDOWNLOADABLE_ENTRIES, ] as const; +const OPENCODE_RUNTIME_CHECKPOINT_EXCLUDED_ENTRIES = [ + OPENCODE_ISOLATED_HOMES_DIRECTORY, + "config/opencode/node_modules", +] as const; +const ACPX_RUNTIME_CHECKPOINT_EXCLUDED_ENTRIES = [ + "home/.npm", + "data/pnpm/store", +] as const; const RUNNERD_CONTROL_PLANE_STATE_SCHEMA = "paperclip.runner.durable.control-plane-state.v1"; const RUNNERD_STATE_SCHEMA = "paperclip.runner.durable.state.v1"; @@ -5646,7 +5656,10 @@ export function resolveNativeHarnessPersistenceProfile( ? { name: "opencode", location: "filesystem", - excludeEntries: [], + excludeEntries: OPENCODE_RUNTIME_CHECKPOINT_EXCLUDED_ENTRIES.map( + (entry) => + `${opencodeRuntimeSessionDirectoryName(nativeSessionKey(execution))}/${entry}`, + ), } : execution.provider.kind === "acpx" ? { @@ -5658,8 +5671,10 @@ export function resolveNativeHarnessPersistenceProfile( // nor necessary to restore. Credentials and launch-time config // are also re-materialized in the replacement sandbox. Grok // diagnostics can contain auth fields and are not session state. - excludeEntries: - execution.provider.agent === "grok" + excludeEntries: [ + ...ACPX_RUNTIME_CHECKPOINT_EXCLUDED_ENTRIES.map((entry) => + `acpx/${acpxRuntimeSessionDirectoryName(nativeSessionKey(execution))}/${entry}`), + ...(execution.provider.agent === "grok" ? ["auth.json", "auth-refresh.json", "auth-refresh.json.tmp", "config.toml", "logs"].map((entry) => `acpx/${acpxRuntimeSessionDirectoryName(nativeSessionKey(execution))}/grok-home/${entry}`) : execution.provider.agent === "codex" @@ -5667,7 +5682,8 @@ export function resolveNativeHarnessPersistenceProfile( (entry) => `acpx/${acpxRuntimeSessionDirectoryName(nativeSessionKey(execution))}/codex-home/${entry}`, ) - : [], + : []), + ], } : null; return { diff --git a/server/src/vendor/paperclip-runner/index.ts b/server/src/vendor/paperclip-runner/index.ts index 2252b5daf6b..571daf6a929 100644 --- a/server/src/vendor/paperclip-runner/index.ts +++ b/server/src/vendor/paperclip-runner/index.ts @@ -107,6 +107,10 @@ export const PAPERCLIP_EXECUTION_PROMPT_REVISION = runner.PAPERCLIP_EXECUTION_PROMPT_REVISION; export const acpxRuntimeSessionDirectoryName = runner.acpxRuntimeSessionDirectoryName; +export const OPENCODE_ISOLATED_HOMES_DIRECTORY = + runner.OPENCODE_ISOLATED_HOMES_DIRECTORY; +export const opencodeRuntimeSessionDirectoryName = + runner.opencodeRuntimeSessionDirectoryName; export const canonicalNativeRuntimeContextDigest = runner.canonicalNativeRuntimeContextDigest; export const createNativeSessionBackend = runner.createNativeSessionBackend; From b792165ba4ce16d753f7a68f8c8c3685d198d6e7 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 09:20:30 -0700 Subject: [PATCH 43/68] Expire native runner continuity archives and quarantine (#17616) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runner sessions keep durable state under `/runtime/paperclip-runner/durable-sessions/` > - When a session starts a fresh provider session, the retired state is moved into `/continuity-breaks/-/`; when state loses authority, it is moved into `quarantine/..../` > - Nothing ever removes either, and each can hold a full Codex home of several gigabytes > - A sleep-aware sweep already expires failover backup copies in the same tree > - This pull request extends that sweep to remove both kinds of evidence after a retention window, while keeping anything recovery may still read > - The benefit is that native runner disk use stops growing without bound ## Linked Issues or Issue Description Bug report described in-PR (bug report template). **What happened?** On one instance, one day of native runs produced 20 continuity-break archives totaling 4.8 GB, one session alone holding 4 GB. The quarantine folder held 12 entries totaling 2.1 GB, the oldest a month old. Neither is read again by normal operation (continuity-break archives are written by `archiveContinuityState` and never read), and nothing deletes them. They were removed by hand to free space and will regrow. **Expected behavior** Evidence older than a bounded window is removed automatically, as failover backup copies already are, without removing anything a live session or recovery path still needs. **Steps to reproduce** 1. Run a native Codex session that hits a continuity break (fresh provider session), or whose state is quarantined (for example `identity_indeterminate`). 2. Inspect `durable-sessions//continuity-breaks/` and `durable-sessions/quarantine/`. 3. Wait any length of time: the entries stay forever. **Paperclip version or commit:** `master` at e3352655687. **Deployment mode:** any with native runner sessions. **Agent adapter(s) involved:** native runner (Codex, and any provider whose state lands in these folders). ## What Changed - `server/src/services/native-runtime/native-harness-backup-expiration.ts`: the existing tick also retires evidence. - **Continuity-break archives** (`-` under each session): removed when older than the window. Age is the epoch in the name, or the folder mtime when that epoch is in the future. `archiveContinuityState` refuses to archive state with warm-transition evidence, and no code path reads `continuity-breaks`. - **Quarantine entries** (`....`): removed by the same age rule. Native cleanup source archives (`.identity_indeterminate.cleanup.`) are referenced from recovery history, have no epoch, and keep their original inode's mtime, so they are always kept. - Every entry is skipped while its session is executing, initializing, warm, or closing in this process; this is checked again just before removal. Symlinked entries and symlinked `continuity-breaks`/`quarantine` folders are reported as `unsafe_path` and never followed. - Removal renames to `.expired-` in the same folder and then deletes; later sweeps finish interrupted deletions. An empty `continuity-breaks` folder is removed. - Archives, quarantine entries, and leftover deletions count toward `maxEntries` and `maxRemovals`, and stop on `shouldContinue`. Paging resumes inside a session's archives and then in quarantine, so a bounded or dry-run batch always makes progress and never recounts a session's backup. - New result fields: `evidenceCandidates`, `evidenceCandidateBytes`, `evidenceRemoved`, `evidenceReclaimedBytes`, and skip reasons `recent_evidence` and `cleanup_archive`. The service also exposes `noteEvidence(writtenAt)`. - `server/src/services/native-runtime/native-runner-evidence-signal.ts` (new): a process-local signal. `native-session-executor.ts` raises it with the entry's epoch right after `archiveContinuityState` creates an archive and after `quarantineRunnerdStateRoot` moves state into quarantine; `server/src/index.ts` subscribes the sweep and unsubscribes on shutdown. - `server/src/config.ts`: `PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS` (default `7`, minimum `1`). - `server/src/index.ts`: passes the window and logs a batch when it found evidence candidates. - `doc/recoverable-disk-cleanup.md`, `doc/DEVELOPING.md`: document the variable and behavior. **Sleep-aware background work** (`rules/sleep-aware-background-work.mdc`): - *Trigger:* each entry's own expiry deadline (its epoch plus the window), plus a direct call from the executor when it writes new evidence. No timer, interval, or polling loop is added; the pass runs inside the existing sweep's tick and keeps its drain, warm-standby, and shutdown handling. - *Source of truth:* the filesystem entries themselves. After a restart the first sweep reconstructs the next deadline from them. - *Pending-work predicate / next deadline:* after a full pass the sweep keeps the earliest due time of the entries it retained; ticks before that time skip the evidence pass entirely (no readdir, no SQL). A busy session, a failed removal, or a full batch makes it due on the next sweep while that obligation remains. - *Quiet condition:* no evidence, or only evidence not yet due. The pass never queries the database, so retained evidence never causes recurring SQL. Dry run reports each candidate once rather than every hour. No code comments were added. ## Verification - `pnpm exec vitest run src/__tests__/native-harness-backup-expiration` (in `server/`, embedded Postgres): 52 passed. New cases: - old archives and quarantine entries are removed, recent ones kept; the empty `continuity-breaks` folder is removed; - age comes from the name, with mtime used for a future epoch; non-archive folders are ignored; - a busy session keeps its evidence, including when it becomes busy after the first check, and it is retried on the next tick; - dry run reports candidates and bytes, changes nothing (including leftovers), and reports each candidate once until new evidence comes due; - symlinked entries and symlinked evidence folders are skipped without touching the target; - an old quarantine entry is removed even when its session's issue is still open; cleanup source archives are kept; - the pass stays quiet until retained evidence is due, and a `noteEvidence` signal schedules newly written evidence; - inspected archives and leftovers count against `maxEntries`, and leftover removal stops on `shouldContinue` and resumes next tick; - a due pass stopped by drain before any session stays pending; one-entry batches finish without recounting a resumed session; - interrupted `.expired-*` deletions are finished; paging across ticks with `maxRemovals`, in normal and dry run, with no recount of the backup when resuming. - `native-session-executor.test.ts`: quarantining a legacy root raises the evidence signal with the epoch in the entry name. - `native-harness-backup-expiration-config`: default, explicit, minimum, and fallback parsing. - **Mutation checks**, each failing at least one test: removing the age check, the future-epoch fallback, either busy check, the busy retry deadline, the recent-entry deadline, the due gate, the `noteEvidence` update, merging the pass deadline, the entry and both folder symlink checks, the dry-run return, the dry-run leftover guard, leftover removal, the leftover budget check and its retry deadline, counting archives against `maxEntries`, the cleanup-archive skip, the empty-folder removal, the resume cursor, the per-archive budget check, the resume skip of the backup pass, not charging a resumed session, keeping the pass active across drain, and the executor's quarantine signal. - `pnpm exec vitest run config native-harness native-workspace-sync-retention native-session-executor server-startup` (in `server/`): 1272 passed on the previous commit; `config native-harness native-workspace-sync-retention server-startup`: 450 passed on the head commit. - `pnpm run typecheck` (in `server/`): pass. - Not run: server build and boot smoke. The continuity-archive signal call is exercised only by a runner integration test that needs the runnerd binary. ## Risks - **Quarantine as a fail-closed fence:** recovery (`recoverQuiescentRunnerdState`) and failed-run retry (`nativeFailedRunRetryStateIsSafe`) treat a quarantined entry for a session as a reason not to bootstrap fresh. Once such an entry ages out, a later run of that session can start a fresh provider session instead of failing with `quarantine_owner_unverified`. This applies only after the window, matching what backup expiry already accepts for closed work. - **Signal coverage:** the signal is process-local. Evidence written by another process sharing the same state directory is picked up when the next retained deadline is due or after a restart, not immediately. - **Legacy-path quarantine:** entries quarantined from legacy roots carry a legacy digest, so the in-process busy check cannot match them; they are removed by age only. - **Remote sandboxes:** archives moved aside on a remote runner (`.continuity-break-`) are outside this sweep. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DEVELOPING.md | 5 + doc/recoverable-disk-cleanup.md | 33 ++ ...e-harness-backup-expiration-config.test.ts | 11 + .../native-harness-backup-expiration.test.ts | 374 ++++++++++++++++++ server/src/config.ts | 3 + server/src/index.ts | 6 +- .../native-harness-backup-expiration.ts | 282 +++++++++++-- .../native-runner-evidence-signal.ts | 12 + .../native-session-executor.test.ts | 6 +- .../native-runtime/native-session-executor.ts | 9 +- 10 files changed, 712 insertions(+), 29 deletions(-) create mode 100644 server/src/services/native-runtime/native-runner-evidence-signal.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index fdfc680c06e..f8d408cdff9 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -2627,6 +2627,11 @@ Environment overrides: negative, empty, or non-numeric value falls back to the default. `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false` turns expiry off, and `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN=true` only reports candidates. +- `PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS=` sets how long native + session continuity-break archives and quarantined session state are kept (see + `doc/recoverable-disk-cleanup.md`). The default is `7` and the minimum is `1`. A + negative, empty, or non-numeric value falls back to the default. The backup + expiration enabled and dry-run switches apply to it as well. - `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS=` sets how long a settled run's native workspace sync folder (descriptor, manifest and seed archives under `/native-workspace-sync/`) is kept (see diff --git a/doc/recoverable-disk-cleanup.md b/doc/recoverable-disk-cleanup.md index b33a0cc74fd..93f9ccb6fef 100644 --- a/doc/recoverable-disk-cleanup.md +++ b/doc/recoverable-disk-cleanup.md @@ -89,10 +89,43 @@ The rest of the session directory, including its control-plane state, is kept. | `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED` | `true` | Set `false` to stop expiring copies | | `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN` | `false` | Set `true` to report candidates without deleting copies or task sessions | | `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS` | `7` | Days an issue must be closed before its sessions' copies expire (minimum `1`) | +| `PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS` | `7` | Days continuity-break archives and quarantined session state are kept (minimum `1`) | Each batch that finds candidates logs them with candidate and reclaimed allocated bytes, cleared task sessions, and skip reasons. +### Continuity-break archives and quarantined session state + +The same sweep also removes two kinds of evidence that native sessions leave behind +under the same root: + +- `/continuity-breaks/-/`, written when a session starts a + fresh provider session and moves the retired session's state aside. Nothing reads + these archives again. +- `quarantine/..../`, written when a session's + state loses authority. Recovery and failed-run retry still consult these entries. + +An entry's age comes from the epoch in its name, or from its modification time when +that epoch is in the future. An entry is removed once it is older than +`PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS` and its session is not executing, +initializing, warm, or closing in this process. Quarantine entries created by +native cleanup (`.identity_indeterminate.cleanup.`) are referenced +from recovery history and kept. Symlinks are never followed or removed. Once a +quarantined entry is removed, recovery and failed-run retry no longer see it, so a +later run of that session may start a fresh provider session instead of refusing. + +The evidence check runs on the sweep only when something is due: once at startup, +then when the oldest retained entry, or an entry the executor has written since, +reaches the window, and on the next sweep after a busy session, a failed removal, +or a full batch. It only reads the filesystem and never queries the +database. A dry run reports each candidate once until newly written evidence comes due. + +An entry is renamed to `.expired-` in its folder before it is deleted, and a +later sweep finishes an interrupted deletion. An empty `continuity-breaks` folder +is removed. Archives, quarantine entries, and leftovers count toward the batch +limits of 200 entries and 50 removals shared with backup copies, and +`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED` and `_DRY_RUN` apply to both. + ## Native workspace sync folders Each native run that syncs a workspace with a remote sandbox writes a folder at diff --git a/server/src/__tests__/native-harness-backup-expiration-config.test.ts b/server/src/__tests__/native-harness-backup-expiration-config.test.ts index 81967a9bdad..0b79c92e542 100644 --- a/server/src/__tests__/native-harness-backup-expiration-config.test.ts +++ b/server/src/__tests__/native-harness-backup-expiration-config.test.ts @@ -37,4 +37,15 @@ describe("native harness backup expiration config parsing", () => { vi.stubEnv("PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS", "0"); expect(loadConfig()).toMatchObject({ nativeHarnessBackupExpirationDays: 1 }); }); + + it("keeps native runner evidence for seven days by default and at least one day", () => { + vi.stubEnv("PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS", undefined); + expect(loadConfig()).toMatchObject({ nativeRunnerEvidenceRetentionDays: 7 }); + vi.stubEnv("PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS", "14"); + expect(loadConfig()).toMatchObject({ nativeRunnerEvidenceRetentionDays: 14 }); + vi.stubEnv("PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS", "0"); + expect(loadConfig()).toMatchObject({ nativeRunnerEvidenceRetentionDays: 1 }); + vi.stubEnv("PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS", "many"); + expect(loadConfig()).toMatchObject({ nativeRunnerEvidenceRetentionDays: 7 }); + }); }); diff --git a/server/src/__tests__/native-harness-backup-expiration.test.ts b/server/src/__tests__/native-harness-backup-expiration.test.ts index b6330409e72..0123d816a86 100644 --- a/server/src/__tests__/native-harness-backup-expiration.test.ts +++ b/server/src/__tests__/native-harness-backup-expiration.test.ts @@ -301,4 +301,378 @@ describeEmbeddedPostgres("native harness backup expiration", () => { expect(await exists(leftover)).toBe(false); expect(await exists(session.current)).toBe(true); }); + async function seedArchive(root: string, input: { stampedDaysAgo?: number; mtimeDaysAgo?: number; name?: string } = {}) { + const stampedAt = now.getTime() - (input.stampedDaysAgo ?? 10) * DAY_MS; + const archive = path.join(root, "continuity-breaks", input.name ?? `${stampedAt}-${randomUUID()}`); + await fs.mkdir(path.join(archive, "codex-home"), { recursive: true }); + await fs.writeFile(path.join(archive, "codex-home", "history.jsonl"), "x".repeat(64 * 1024)); + const mtime = new Date(now.getTime() - (input.mtimeDaysAgo ?? input.stampedDaysAgo ?? 10) * DAY_MS); + await fs.utimes(archive, mtime, mtime); + return archive; + } + + async function seedQuarantine( + scope: string, + input: { stampedDaysAgo?: number; name?: string } = {}, + ) { + if (!stateBase) stateBase = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-durable-sessions-")); + const stampedAt = now.getTime() - (input.stampedDaysAgo ?? 10) * DAY_MS; + const entry = path.join( + stateBase, + "quarantine", + input.name ?? `${scope}.identity_indeterminate.${stampedAt}.4242.${randomUUID()}`, + ); + await fs.mkdir(path.join(entry, "control-plane"), { recursive: true }); + await fs.writeFile(path.join(entry, "history.jsonl"), "x".repeat(64 * 1024)); + return entry; + } + + function evidenceService(overrides: Parameters[1] = {}) { + return nativeHarnessBackupExpirationService(db, { + stateBase, + now, + evidenceRetentionDays: 7, + isSessionBusy: () => false, + ...overrides, + }); + } + + async function emptyStateBase() { + if (!stateBase) stateBase = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-durable-sessions-")); + return createHash("sha256").update(randomUUID()).digest("hex"); + } + + it("removes continuity archives older than the retention window and keeps recent ones", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const old = await seedArchive(session.root, { stampedDaysAgo: 9 }); + const recent = await seedArchive(session.root, { stampedDaysAgo: 2 }); + + const result = await evidenceService().tick(); + + expect(result).toMatchObject({ evidenceCandidates: 1, evidenceRemoved: 1, failed: 0 }); + expect(result.evidenceReclaimedBytes).toBeGreaterThan(0); + expect(result.skipped.recent_evidence).toBe(1); + expect(await exists(old)).toBe(false); + expect(await exists(recent)).toBe(true); + expect(await exists(session.current)).toBe(true); + }); + + it("removes the continuity-breaks folder once its archives are gone", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + await seedArchive(session.root); + + expect(await evidenceService().tick()).toMatchObject({ evidenceRemoved: 1 }); + expect(await exists(path.join(session.root, "continuity-breaks"))).toBe(false); + }); + + it("ages a continuity archive by its name and falls back to mtime for a future stamp", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const stampedRecently = await seedArchive(session.root, { stampedDaysAgo: 1, mtimeDaysAgo: 30 }); + const futureStamp = await seedArchive(session.root, { + name: `${now.getTime() + DAY_MS}-${randomUUID()}`, + mtimeDaysAgo: 30, + }); + const futureRecent = await seedArchive(session.root, { + name: `${now.getTime() + 2 * DAY_MS}-${randomUUID()}`, + mtimeDaysAgo: 1, + }); + + expect(await evidenceService().tick()).toMatchObject({ evidenceRemoved: 1 }); + expect(await exists(stampedRecently)).toBe(true); + expect(await exists(futureStamp)).toBe(false); + expect(await exists(futureRecent)).toBe(true); + }); + + it("ignores folders that are not continuity archives", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const other = await seedArchive(session.root, { name: "notes", stampedDaysAgo: 30 }); + + expect(await evidenceService().tick()).toMatchObject({ evidenceCandidates: 0 }); + expect(await exists(other)).toBe(true); + }); + + it("keeps the evidence of a session that is executing in this process", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archive = await seedArchive(session.root); + const quarantine = await seedQuarantine(session.scope); + + const result = await evidenceService({ isSessionBusy: (scope) => scope === session.scope }).tick(); + + expect(result).toMatchObject({ evidenceCandidates: 0, evidenceRemoved: 0 }); + expect(result.skipped.session_active).toBe(3); + expect(await exists(archive)).toBe(true); + expect(await exists(quarantine)).toBe(true); + }); + + it("keeps evidence when its session starts while the removal is being prepared", async () => { + const scope = await emptyStateBase(); + const root = path.join(stateBase, scope); + const archive = await seedArchive(root); + let checks = 0; + + const result = await evidenceService({ isSessionBusy: () => ++checks > 1 }).tick(); + + expect(result).toMatchObject({ evidenceCandidates: 1, evidenceRemoved: 0 }); + expect(result.skipped.session_active).toBe(1); + expect(await exists(archive)).toBe(true); + }); + + it("reports evidence without deleting it in dry run", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archive = await seedArchive(session.root); + const quarantine = await seedQuarantine(session.scope); + const leftover = path.join(session.root, "continuity-breaks", `.expired-${randomUUID()}`); + await fs.mkdir(leftover); + + const result = await evidenceService({ dryRun: true }).tick(); + + expect(result).toMatchObject({ evidenceCandidates: 2, evidenceRemoved: 0, evidenceReclaimedBytes: 0 }); + expect(result.evidenceCandidateBytes).toBeGreaterThan(0); + expect(await exists(archive)).toBe(true); + expect(await exists(quarantine)).toBe(true); + expect(await exists(leftover)).toBe(true); + }); + + it("skips symlinked evidence without following it", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const outside = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-evidence-outside-")); + try { + await fs.writeFile(path.join(outside, "keep.txt"), "keep"); + const stampedAt = now.getTime() - 10 * DAY_MS; + await fs.mkdir(path.join(session.root, "continuity-breaks"), { recursive: true }); + const archiveLink = path.join(session.root, "continuity-breaks", `${stampedAt}-${randomUUID()}`); + await fs.symlink(outside, archiveLink); + await fs.mkdir(path.join(stateBase, "quarantine"), { recursive: true }); + const quarantineLink = path.join( + stateBase, + "quarantine", + `${session.scope}.identity_indeterminate.${stampedAt}.4242.${randomUUID()}`, + ); + await fs.symlink(outside, quarantineLink); + + const result = await evidenceService().tick(); + + expect(result).toMatchObject({ evidenceCandidates: 0, evidenceRemoved: 0 }); + expect(result.skipped.unsafe_path).toBe(2); + expect(await exists(archiveLink)).toBe(true); + expect(await exists(quarantineLink)).toBe(true); + expect(await exists(path.join(outside, "keep.txt"))).toBe(true); + } finally { + await fs.rm(outside, { recursive: true, force: true }); + } + }); + + it("skips symlinked evidence folders without following them", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const outside = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-evidence-outside-")); + try { + const stampedAt = now.getTime() - 10 * DAY_MS; + const inside = path.join(outside, `${stampedAt}-${randomUUID()}`); + await fs.mkdir(inside); + await fs.utimes(inside, new Date(stampedAt), new Date(stampedAt)); + await fs.symlink(outside, path.join(session.root, "continuity-breaks")); + await fs.symlink(outside, path.join(stateBase, "quarantine")); + + const result = await evidenceService().tick(); + + expect(result).toMatchObject({ evidenceCandidates: 0, evidenceRemoved: 0 }); + expect(result.skipped.unsafe_path).toBe(2); + expect(await exists(inside)).toBe(true); + } finally { + await fs.rm(outside, { recursive: true, force: true }); + } + }); + + it("removes quarantine entries older than the retention window and keeps recent ones", async () => { + const scope = await emptyStateBase(); + const old = await seedQuarantine(scope, { stampedDaysAgo: 9 }); + const recent = await seedQuarantine(scope, { stampedDaysAgo: 2 }); + + const result = await evidenceService().tick(); + + expect(result).toMatchObject({ evidenceCandidates: 1, evidenceRemoved: 1, failed: 0 }); + expect(result.skipped.recent_evidence).toBe(1); + expect(await exists(old)).toBe(false); + expect(await exists(recent)).toBe(true); + expect(await fs.readdir(path.join(stateBase, "quarantine"))).toEqual([path.basename(recent)]); + }); + + it("keeps cleanup source archives in quarantine", async () => { + const scope = await emptyStateBase(); + const entry = await seedQuarantine(scope, { name: `${scope}.identity_indeterminate.cleanup.${randomUUID()}` }); + const old = new Date(now.getTime() - 30 * DAY_MS); + await fs.utimes(entry, old, old); + + const result = await evidenceService().tick(); + + expect(result.evidenceCandidates).toBe(0); + expect(result.skipped.cleanup_archive).toBe(1); + expect(await exists(entry)).toBe(true); + }); + + it("finishes removing evidence left over from an interrupted removal", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archive = await seedArchive(session.root, { stampedDaysAgo: 1 }); + const leftover = path.join(session.root, "continuity-breaks", `.expired-${randomUUID()}`); + await fs.mkdir(path.join(leftover, "codex-home"), { recursive: true }); + const quarantineLeftover = path.join(stateBase, "quarantine", `.expired-${randomUUID()}`); + await fs.mkdir(quarantineLeftover, { recursive: true }); + + await evidenceService().tick(); + + expect(await exists(leftover)).toBe(false); + expect(await exists(quarantineLeftover)).toBe(false); + expect(await exists(archive)).toBe(true); + }); + + it.each([false, true])("pages evidence removals across ticks (dry run %s)", async (dryRun) => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archives = [await seedArchive(session.root), await seedArchive(session.root), await seedArchive(session.root)]; + const quarantine = await seedQuarantine(session.scope); + const retention = evidenceService({ dryRun, maxRemovals: 2 }); + + const results = []; + for (let i = 0; i < 4; i++) { + const result = await retention.tick(); + results.push(result); + if (!result.truncated) break; + } + + expect(results.map((result) => result.evidenceCandidates)).toEqual([2, 2]); + expect(results.map((result) => result.truncated)).toEqual([true, false]); + for (const archive of archives) expect(await exists(archive)).toBe(dryRun); + expect(await exists(quarantine)).toBe(dryRun); + }); + + it("does not recount a session's backup when its archives resume on the next tick", async () => { + const session = await seedSession(); + await seedArchive(session.root); + await seedArchive(session.root); + const retention = evidenceService({ dryRun: true, maxRemovals: 2 }); + + const first = await retention.tick(); + const second = await retention.tick(); + + expect(first).toMatchObject({ candidates: 1, evidenceCandidates: 1, truncated: true }); + expect(second).toMatchObject({ candidates: 0, evidenceCandidates: 1, truncated: false }); + }); + it("removes old quarantine entries even when their session's issue is still open", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + const entry = await seedQuarantine(session.scope); + await fs.cp(path.join(session.root, "control-plane"), path.join(entry, "control-plane"), { recursive: true }); + + expect(await evidenceService().tick()).toMatchObject({ evidenceRemoved: 1 }); + expect(await exists(entry)).toBe(false); + }); + + it("stays quiet until retained evidence is due and wakes when new evidence is written", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + const recent = await seedArchive(session.root, { stampedDaysAgo: 2 }); + const retention = evidenceService(); + const at = (days: number) => ({ now: new Date(now.getTime() + days * DAY_MS) }); + + expect((await retention.tick()).skipped.recent_evidence).toBe(1); + expect((await retention.tick(at(4))).skipped.recent_evidence).toBeUndefined(); + expect(await retention.tick(at(6))).toMatchObject({ evidenceRemoved: 1 }); + expect(await exists(recent)).toBe(false); + + const written = await seedArchive(session.root, { stampedDaysAgo: 0 }); + expect(await retention.tick(at(8))).toMatchObject({ evidenceCandidates: 0 }); + retention.noteEvidence(now.getTime()); + expect(await retention.tick(at(8))).toMatchObject({ evidenceRemoved: 1 }); + expect(await exists(written)).toBe(false); + }); + + it("retries evidence of a busy session on the next tick", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + const archive = await seedArchive(session.root); + let busy = true; + const retention = evidenceService({ isSessionBusy: () => busy }); + + expect(await retention.tick()).toMatchObject({ evidenceRemoved: 0 }); + busy = false; + expect(await retention.tick()).toMatchObject({ evidenceRemoved: 1 }); + expect(await exists(archive)).toBe(false); + }); + + it("reports dry-run evidence once until new evidence is written", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + await seedArchive(session.root); + const retention = evidenceService({ dryRun: true }); + + expect(await retention.tick()).toMatchObject({ evidenceCandidates: 1 }); + expect(await retention.tick()).toMatchObject({ evidenceCandidates: 0 }); + retention.noteEvidence(now.getTime()); + expect(await retention.tick({ now: new Date(now.getTime() + 7 * DAY_MS) })).toMatchObject({ evidenceCandidates: 1 }); + }); + + it("counts inspected archives and leftovers against the entry limit", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + const archives = [ + await seedArchive(session.root, { stampedDaysAgo: 1 }), + await seedArchive(session.root, { stampedDaysAgo: 2 }), + await seedArchive(session.root, { stampedDaysAgo: 3 }), + ]; + const leftovers = [randomUUID(), randomUUID()].map((id) => path.join(session.root, "continuity-breaks", `.expired-${id}`)); + for (const leftover of leftovers) await fs.mkdir(leftover); + const retention = evidenceService({ maxEntries: 2 }); + + const first = await retention.tick(); + expect(first).toMatchObject({ checked: 2, truncated: true }); + expect(first.skipped.recent_evidence).toBeUndefined(); + expect((await fs.readdir(path.join(session.root, "continuity-breaks"))).filter((name) => name.startsWith(".expired-"))).toHaveLength(1); + + const results = [first]; + for (let i = 0; i < 6 && results.at(-1)!.truncated; i++) results.push(await retention.tick()); + expect(results.at(-1)!.truncated).toBe(false); + expect(results.every((result) => result.checked <= 2)).toBe(true); + expect(results.every((result) => (result.skipped.recent_evidence ?? 0) <= 2)).toBe(true); + expect(results.reduce((sum, result) => sum + (result.skipped.recent_evidence ?? 0), 0)).toBe(3); + for (const leftover of leftovers) expect(await exists(leftover)).toBe(false); + for (const archive of archives) expect(await exists(archive)).toBe(true); + }); + + it("stops removing leftovers once the sweep must stop", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + await fs.mkdir(path.join(session.root, "continuity-breaks")); + const leftover = path.join(session.root, "continuity-breaks", `.expired-${randomUUID()}`); + await fs.mkdir(leftover); + let calls = 0; + + const retention = evidenceService({ shouldContinue: () => ++calls !== 2 }); + + await retention.tick(); + expect(await exists(leftover)).toBe(true); + await retention.tick(); + expect(await exists(leftover)).toBe(false); + }); + + it("keeps a due evidence pass pending when drain stops it before any session", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + const archive = await seedArchive(session.root); + let draining = true; + const retention = evidenceService({ shouldContinue: () => !draining }); + + expect(await retention.tick()).toMatchObject({ checked: 0, truncated: true }); + draining = false; + expect(await retention.tick()).toMatchObject({ evidenceRemoved: 1 }); + expect(await exists(archive)).toBe(false); + }); + + it("finishes one-entry batches without recounting a resumed session", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + const archives = [await seedArchive(session.root), await seedArchive(session.root)]; + const quarantine = await seedQuarantine(session.scope); + const retention = evidenceService({ maxEntries: 1 }); + + const results = []; + for (let i = 0; i < 8; i++) { + results.push(await retention.tick()); + if (!results.at(-1)!.truncated) break; + } + + expect(results.at(-1)!.truncated).toBe(false); + for (const archive of archives) expect(await exists(archive)).toBe(false); + expect(await exists(quarantine)).toBe(false); + }); }); diff --git a/server/src/config.ts b/server/src/config.ts index 74e80c77ee1..dac9dba2142 100644 --- a/server/src/config.ts +++ b/server/src/config.ts @@ -95,6 +95,7 @@ export interface Config { nativeHarnessBackupExpirationEnabled: boolean; nativeHarnessBackupExpirationDryRun: boolean; nativeHarnessBackupExpirationDays: number; + nativeRunnerEvidenceRetentionDays: number; nativeWorkspaceSyncRetentionEnabled: boolean; nativeWorkspaceSyncRetentionDryRun: boolean; nativeWorkspaceSyncRetentionDays: number; @@ -351,6 +352,7 @@ export function loadConfig(): Config { const skillVersionRetentionKeepNewest = nonNegativeIntegerEnv("PAPERCLIP_SKILL_VERSION_RETENTION_KEEP", 0); const skillVersionRetentionGraceMinutes = nonNegativeIntegerEnv("PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES", 60); const nativeHarnessBackupExpirationDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS", 7)); + const nativeRunnerEvidenceRetentionDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS", 7)); const nativeWorkspaceSyncRetentionDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS", 7)); const gitRestorePointRetentionDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS", 90)); const gitRestorePointTerminalDelayDays = nonNegativeIntegerEnv("PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS", 3); @@ -417,6 +419,7 @@ export function loadConfig(): Config { nativeHarnessBackupExpirationEnabled: process.env.PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED !== "false", nativeHarnessBackupExpirationDryRun: process.env.PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN === "true", nativeHarnessBackupExpirationDays, + nativeRunnerEvidenceRetentionDays, nativeWorkspaceSyncRetentionEnabled: process.env.PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED !== "false", nativeWorkspaceSyncRetentionDryRun: process.env.PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN === "true", nativeWorkspaceSyncRetentionDays, diff --git a/server/src/index.ts b/server/src/index.ts index 207f342a2ae..4212387dd04 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -13,6 +13,7 @@ import { NATIVE_HARNESS_BACKUP_EXPIRATION_STARTUP_DELAY_MS, nativeHarnessBackupExpirationService, } from "./services/native-runtime/native-harness-backup-expiration.js"; +import { onNativeRunnerEvidence } from "./services/native-runtime/native-runner-evidence-signal.js"; import { gitRestorePointRetentionService, registerGitRestorePointWork } from "./services/git-restore-point-ledger.js"; import { nativeWorkspaceSyncRetentionService, @@ -1269,8 +1270,10 @@ async function startServerWithDatabaseTeardown( const nativeHarnessBackupExpiration = nativeHarnessBackupExpirationService(db as any, { dryRun: config.nativeHarnessBackupExpirationDryRun, closedForDays: config.nativeHarnessBackupExpirationDays, + evidenceRetentionDays: config.nativeRunnerEvidenceRetentionDays, shouldContinue: () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby(), }); + const unsubscribeNativeRunnerEvidence = onNativeRunnerEvidence(nativeHarnessBackupExpiration.noteEvidence); let nativeHarnessBackupExpirationInFlight = false; let nativeHarnessBackupExpirationContinuation: ReturnType | undefined; const sweepNativeHarnessBackups = () => { @@ -1278,7 +1281,7 @@ async function startServerWithDatabaseTeardown( nativeHarnessBackupExpirationInFlight = true; trackHeartbeatSchedulerWork(nativeHarnessBackupExpiration.tick() .then((result) => { - if (result.candidates > 0 || result.failed > 0) { + if (result.candidates > 0 || result.evidenceCandidates > 0 || result.failed > 0) { logger.info({ ...result, dryRun: config.nativeHarnessBackupExpirationDryRun }, "native harness backup expiration"); } if (result.truncated && !heartbeatSchedulerStopped) { @@ -2095,6 +2098,7 @@ async function startServerWithDatabaseTeardown( clearInterval(nativeHarnessBackupExpirationInterval); clearTimeout(nativeHarnessBackupExpirationStartup); clearTimeout(nativeHarnessBackupExpirationContinuation); + unsubscribeNativeRunnerEvidence(); await skillVersionRetention.dispose(); await skillSourceSync.stop(); unsubscribeChatCompletions(); diff --git a/server/src/services/native-runtime/native-harness-backup-expiration.ts b/server/src/services/native-runtime/native-harness-backup-expiration.ts index 448997670e1..d4c0975f0b5 100644 --- a/server/src/services/native-runtime/native-harness-backup-expiration.ts +++ b/server/src/services/native-runtime/native-harness-backup-expiration.ts @@ -17,10 +17,15 @@ import { nativeSessionScopeDigestIsBusy } from "./native-session-executor.js"; export const NATIVE_HARNESS_BACKUP_EXPIRATION_INTERVAL_MS = 60 * 60 * 1000; export const NATIVE_HARNESS_BACKUP_EXPIRATION_STARTUP_DELAY_MS = 2 * 60 * 1000; export const NATIVE_HARNESS_BACKUP_EXPIRATION_DEFAULT_DAYS = 7; +export const NATIVE_RUNNER_EVIDENCE_RETENTION_DEFAULT_DAYS = 7; const DAY_MS = 24 * 60 * 60 * 1000; const SCOPE_PATTERN = /^[0-9a-f]{64}$/; const UUID_PATTERN = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; +const CONTINUITY_ARCHIVE_PATTERN = /^(\d{1,15})-[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; +const QUARANTINE_PATTERN = + /^([0-9a-f]{64})\.(?:identity_indeterminate|identity_mismatch)\.(\d{1,15})\.\d{1,10}\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; +const QUARANTINE_CLEANUP_PATTERN = /^[0-9a-f]{64}\.identity_indeterminate\.cleanup\./; const EXPIRED_PREFIX = ".expired-"; const RETIRING_PREFIX = ".retiring-"; const LIVE_RUN_STATUSES = ["queued", "scheduled_retry", "running"]; @@ -31,6 +36,7 @@ const MAX_CONTROL_PLANE_STATE_BYTES = 4 * 1024 * 1024; export type NativeHarnessBackupExpirationOptions = { closedForDays?: number; + evidenceRetentionDays?: number; dryRun?: boolean; maxEntries?: number; maxRemovals?: number; @@ -47,11 +53,25 @@ export type NativeHarnessBackupExpirationResult = { expired: number; reclaimedBytes: number; clearedTaskSessions: number; + evidenceCandidates: number; + evidenceCandidateBytes: number; + evidenceRemoved: number; + evidenceReclaimedBytes: number; failed: number; truncated: boolean; skipped: Record; }; +type EvidenceContext = { + dryRun: boolean; + isSessionBusy: (scopeDigest: string) => boolean; + now: number; + retentionMs: number; + retainedBefore: number; + canInspect: () => boolean; + dueAt: (at: number) => void; +}; + type SessionIdentity = { runId: string; normalizedSessionId: string; @@ -66,6 +86,10 @@ function emptyResult(): NativeHarnessBackupExpirationResult { expired: 0, reclaimedBytes: 0, clearedTaskSessions: 0, + evidenceCandidates: 0, + evidenceCandidateBytes: 0, + evidenceRemoved: 0, + evidenceReclaimedBytes: 0, failed: 0, truncated: false, skipped: {}, @@ -223,6 +247,13 @@ export function nativeHarnessBackupExpirationService( defaults: NativeHarnessBackupExpirationOptions = {}, ) { let cursor: string | null = null; + let resume: { scope: string; archive: string } | null = null; + let quarantineCursor: string | null = null; + let phase: "sessions" | "quarantine" = "sessions"; + let passActive = false; + let evidencePass = false; + let evidenceDueAt = 0; + let passDueAt = Number.POSITIVE_INFINITY; async function settleLeftovers(backupRoot: string) { const current = path.join(backupRoot, "current"); @@ -310,46 +341,247 @@ export function nativeHarnessBackupExpirationService( result.reclaimedBytes += bytes; } + async function removeExpiredLeftovers(parent: string, context: EvidenceContext, result: NativeHarnessBackupExpirationResult) { + for (const name of await fs.readdir(parent).catch(() => [] as string[])) { + if (!name.startsWith(EXPIRED_PREFIX) || !UUID_PATTERN.test(name.slice(EXPIRED_PREFIX.length))) continue; + if (!context.canInspect()) { + context.dueAt(context.now); + return; + } + result.checked++; + await removeBackupTree(path.join(parent, name)); + } + } + + async function retireEvidence( + parent: string, + name: string, + scopeDigest: string, + stampedAt: number, + context: EvidenceContext, + result: NativeHarnessBackupExpirationResult, + ) { + const skip = (reason: string) => { + result.skipped[reason] = (result.skipped[reason] ?? 0) + 1; + }; + const target = path.join(parent, name); + const stat = await fs.lstat(target).catch(() => null); + if (path.dirname(target) !== parent || !stat?.isDirectory()) { + return skip("unsafe_path"); + } + const createdAt = stampedAt > 0 && stampedAt <= context.now ? stampedAt : stat.mtimeMs; + if (createdAt >= context.retainedBefore) { + context.dueAt(createdAt + context.retentionMs); + return skip("recent_evidence"); + } + if (context.isSessionBusy(scopeDigest)) { + context.dueAt(context.now); + return skip("session_active"); + } + const bytes = await allocatedBytes(target); + result.evidenceCandidates++; + result.evidenceCandidateBytes += bytes; + if (context.dryRun) return; + if (context.isSessionBusy(scopeDigest)) { + context.dueAt(context.now); + return skip("session_active"); + } + const trash = path.join(parent, `${EXPIRED_PREFIX}${randomUUID()}`); + await fs.rename(target, trash); + await removeBackupTree(trash); + result.evidenceRemoved++; + result.evidenceReclaimedBytes += bytes; + } + + async function retireContinuityBreaks( + root: string, + scopeDigest: string, + resumeAfter: string | null, + context: EvidenceContext, + result: NativeHarnessBackupExpirationResult, + ): Promise { + const parent = path.join(root, "continuity-breaks"); + const stat = await fs.lstat(parent).catch(() => null); + if (!stat) return null; + if (!stat.isDirectory()) { + result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; + return null; + } + if (!context.dryRun) await removeExpiredLeftovers(parent, context, result); + let last = resumeAfter ?? ""; + const names = (await fs.readdir(parent)).filter((name) => name > last).sort(); + for (const name of names) { + const match = CONTINUITY_ARCHIVE_PATTERN.exec(name); + if (!match) continue; + if (!context.canInspect()) return last; + result.checked++; + last = name; + try { + await retireEvidence(parent, name, scopeDigest, Number(match[1]), context, result); + } catch (err) { + result.failed++; + context.dueAt(context.now); + logger.warn({ err, scope: scopeDigest }, "native runner evidence retention skipped a continuity archive"); + } + } + if (!context.dryRun) await fs.rmdir(parent).catch(() => undefined); + return null; + } + + async function retireQuarantine( + base: string, + context: EvidenceContext, + result: NativeHarnessBackupExpirationResult, + ): Promise { + const parent = path.join(base, "quarantine"); + const stat = await fs.lstat(parent).catch(() => null); + if (!stat) return true; + if (!stat.isDirectory()) { + result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; + return true; + } + if (!context.dryRun && quarantineCursor === null) await removeExpiredLeftovers(parent, context, result); + const names = (await fs.readdir(parent)).sort(); + const pending = quarantineCursor ? names.filter((name) => name > quarantineCursor!) : names; + for (const name of pending) { + const match = QUARANTINE_PATTERN.exec(name); + const cleanupArchive = QUARANTINE_CLEANUP_PATTERN.test(name); + if (!match && !cleanupArchive) continue; + if (!context.canInspect()) return false; + result.checked++; + quarantineCursor = name; + if (!match) { + result.skipped.cleanup_archive = (result.skipped.cleanup_archive ?? 0) + 1; + continue; + } + try { + await retireEvidence(parent, name, match[1]!, Number(match[2]), context, result); + } catch (err) { + result.failed++; + context.dueAt(context.now); + logger.warn({ err, scope: match[1] }, "native runner evidence retention skipped a quarantine entry"); + } + } + quarantineCursor = null; + return true; + } + + function noteEvidence(writtenAt: number) { + evidenceDueAt = Math.min(evidenceDueAt, writtenAt + evidenceRetentionMs()); + } + + function evidenceRetentionMs(options: NativeHarnessBackupExpirationOptions = {}) { + return Math.max( + 1, + options.evidenceRetentionDays ?? defaults.evidenceRetentionDays ?? NATIVE_RUNNER_EVIDENCE_RETENTION_DEFAULT_DAYS, + ) * DAY_MS; + } + async function tick(options: NativeHarnessBackupExpirationOptions = {}) { const closedForDays = Math.max( 1, options.closedForDays ?? defaults.closedForDays ?? NATIVE_HARNESS_BACKUP_EXPIRATION_DEFAULT_DAYS, ); + const retentionMs = evidenceRetentionMs(options); const dryRun = options.dryRun ?? defaults.dryRun ?? false; const maxEntries = Math.max(1, options.maxEntries ?? defaults.maxEntries ?? 200); const maxRemovals = Math.max(1, options.maxRemovals ?? defaults.maxRemovals ?? 50); const shouldContinue = options.shouldContinue ?? defaults.shouldContinue ?? (() => true); const isSessionBusy = options.isSessionBusy ?? defaults.isSessionBusy ?? nativeSessionScopeDigestIsBusy; const base = path.resolve(options.stateBase ?? defaults.stateBase ?? nativeHarnessStateBase()); - const closedBefore = new Date((options.now ?? defaults.now ?? new Date()).getTime() - closedForDays * DAY_MS); + const now = (options.now ?? defaults.now ?? new Date()).getTime(); + const closedBefore = new Date(now - closedForDays * DAY_MS); const result = emptyResult(); - - const names = (await fs.readdir(base).catch(() => [] as string[])) - .filter((name) => SCOPE_PATTERN.test(name)) - .sort(); - const pending = cursor ? names.filter((name) => name > cursor!) : names; - for (const name of pending) { - if (!shouldContinue() || result.checked >= maxEntries || result.candidates >= maxRemovals) { - result.truncated = true; - break; - } - result.checked++; - cursor = name; - const root = path.join(base, name); - if (path.dirname(root) !== base || !(await isRealDirectory(root))) { - result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; - continue; - } - try { - await expireSession(root, name, { dryRun, isSessionBusy, closedBefore }, result); - } catch (err) { - result.failed++; - logger.warn({ err, scope: name }, "native harness backup expiration skipped a session"); + const hasBudget = () => shouldContinue() && result.candidates + result.evidenceCandidates < maxRemovals; + if (!passActive) { + passActive = true; + evidencePass = evidenceDueAt <= now; + if (evidencePass) { + evidenceDueAt = Number.POSITIVE_INFINITY; + passDueAt = Number.POSITIVE_INFINITY; } } - if (!result.truncated) cursor = null; + const context: EvidenceContext = { + dryRun, + isSessionBusy, + now, + retentionMs, + retainedBefore: now - retentionMs, + canInspect: () => hasBudget() && result.checked < maxEntries, + dueAt: (at) => { + passDueAt = Math.min(passDueAt, at); + }, + }; + + if (phase === "sessions") { + const names = (await fs.readdir(base).catch(() => [] as string[])) + .filter((name) => SCOPE_PATTERN.test(name)) + .sort(); + const pending = cursor ? names.filter((name) => name > cursor!) : names; + for (const name of pending) { + if (!hasBudget() || result.checked >= maxEntries) { + result.truncated = true; + break; + } + const resumeAfter = resume?.scope === name ? resume.archive : null; + if (resume && name >= resume.scope) resume = null; + if (resumeAfter === null) result.checked++; + const previous = cursor; + cursor = name; + const root = path.join(base, name); + if (path.dirname(root) !== base || !(await isRealDirectory(root))) { + result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; + continue; + } + if (resumeAfter === null) { + try { + await expireSession(root, name, { dryRun, isSessionBusy, closedBefore }, result); + } catch (err) { + result.failed++; + logger.warn({ err, scope: name }, "native harness backup expiration skipped a session"); + } + } + if (!evidencePass) continue; + let stoppedAt: string | null = null; + try { + stoppedAt = await retireContinuityBreaks(root, name, resumeAfter, context, result); + } catch (err) { + result.failed++; + context.dueAt(now); + logger.warn({ err, scope: name }, "native runner evidence retention skipped a session"); + } + if (stoppedAt !== null) { + resume = { scope: name, archive: stoppedAt }; + cursor = previous; + result.truncated = true; + break; + } + } + if (result.truncated) return result; + cursor = null; + if (!evidencePass) { + passActive = false; + return result; + } + phase = "quarantine"; + } + try { + if (!(await retireQuarantine(base, context, result))) { + result.truncated = true; + return result; + } + } catch (err) { + result.failed++; + context.dueAt(now); + quarantineCursor = null; + logger.warn({ err }, "native runner evidence retention skipped quarantine"); + } + phase = "sessions"; + passActive = false; + evidencePass = false; + evidenceDueAt = Math.min(evidenceDueAt, passDueAt); return result; } - return { tick }; + return { tick, noteEvidence }; } diff --git a/server/src/services/native-runtime/native-runner-evidence-signal.ts b/server/src/services/native-runtime/native-runner-evidence-signal.ts new file mode 100644 index 00000000000..39d51bee146 --- /dev/null +++ b/server/src/services/native-runtime/native-runner-evidence-signal.ts @@ -0,0 +1,12 @@ +const listeners = new Set<(writtenAt: number) => void>(); + +export function onNativeRunnerEvidence(listener: (writtenAt: number) => void): () => void { + listeners.add(listener); + return () => { + listeners.delete(listener); + }; +} + +export function noteNativeRunnerEvidence(writtenAt: number): void { + for (const listener of listeners) listener(writtenAt); +} diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 5c8c3e211f7..39862333f8c 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -394,6 +394,7 @@ import { verifyNativeHarnessBackup, shouldRestoreNativeHarnessBackupIntoSandbox, } from "./native-session-executor.js"; +import { onNativeRunnerEvidence } from "./native-runner-evidence-signal.js"; beforeEach(() => { usageRecorder.capture.mockReset().mockImplementation(async receipt => receipt); @@ -12442,6 +12443,8 @@ describe("runnerd provider runtime wiring", () => { await writeRunnerAuthority(legacyRoot, identity, "ready"); state.createBackend.mockClear(); state.createTransport.mockClear(); + const evidence: number[] = []; + const stopEvidence = onNativeRunnerEvidence((writtenAt) => evidence.push(writtenAt)); await expect( createRunnerdBackend({ @@ -12449,11 +12452,12 @@ describe("runnerd provider runtime wiring", () => { execution: currentExecution, runnerInstanceId: "runner-after-legacy-terminal-unsuspended", }), - ).rejects.toThrow("runner_state_identity_mismatch"); + ).rejects.toThrow("runner_state_identity_mismatch").finally(stopEvidence); await expect(access(legacyRoot)).rejects.toThrow(); const quarantineEntries = await readdir(join(stateBase, "quarantine")); expect(quarantineEntries).toHaveLength(1); expect(quarantineEntries[0]).toContain(".identity_indeterminate."); + expect(evidence).toEqual([Number(quarantineEntries[0]!.split(".")[2])]); expect(state.createBackend).not.toHaveBeenCalled(); expect(state.createTransport).not.toHaveBeenCalled(); }); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 7f67a8aa2fc..234f673e664 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -209,6 +209,7 @@ import { type NativeRunTrace, } from "./native-run-trace.js"; import { createNativeHarnessBackupStamp } from "./native-harness-backup-stamp.js"; +import { noteNativeRunnerEvidence } from "./native-runner-evidence-signal.js"; import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js"; import { registerLiveRunnerGoalController } from "../runner-goal-control-broker.js"; import { applyRunnerGoalPrpEvent } from "../runner-goals.js"; @@ -1952,11 +1953,13 @@ function quarantineRunnerdStateRoot( if (!isSafeNativeStateDirectory(quarantineRoot)) { throw new Error("runner_state_directory_unsafe"); } + const quarantinedAt = Date.now(); const destination = resolve( quarantineRoot, - `${stateKey}.${reason}.${Date.now()}.${process.pid}.${randomUUID()}`, + `${stateKey}.${reason}.${quarantinedAt}.${process.pid}.${randomUUID()}`, ); renameSync(resolvedRoot, destination); + noteNativeRunnerEvidence(quarantinedAt); return destination; } @@ -13276,9 +13279,11 @@ async function createRunnerdBackendWithinSessionClaim( if (hasRetainedWarmTransitionEvidence(root)) { throw new Error("native_runner_warm_transition_recovery_unproven"); } - const archiveToken = `${Date.now()}-${randomUUID()}`; + const archivedAt = Date.now(); + const archiveToken = `${archivedAt}-${randomUUID()}`; const archiveRoot = resolve(root, "continuity-breaks", archiveToken); mkdirSync(archiveRoot, { recursive: true, mode: 0o700 }); + noteNativeRunnerEvidence(archivedAt); for (const name of [ "control-plane", "runner", From e302f2098ff1f5868a3b3537e635cc380329cdb0 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 09:40:28 -0700 Subject: [PATCH 44/68] perf(wake-queue): omit unused native launch context from release reads (#17619) Read native wake-release control context without saved launch payloads. Keep stored input, full legacy context, cancellation evidence, and release decisions. PostgreSQL response-budget regression, 785 lifecycle tests, typecheck, build, and all 55 CI checks pass. Apex reviewed the exact head at 5/5 with no findings. Co-Authored-By: Paperclip --- doc/DATABASE.md | 12 +++-- .../wake-queue/adapters/postgres.test.ts | 7 ++- .../adapters/release-run-read.test.ts | 49 +++++++++++++++++++ .../wake-queue/adapters/release-run-read.ts | 7 +++ 4 files changed, 69 insertions(+), 6 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index e5478530c02..4d6622bc784 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -50,10 +50,14 @@ and leaves all existing cases passive. Source identities are not grants. ### Task query performance The issue-lock adapter's release read omits `result_json.stdout` and -`result_json.stderr` before transferring a run to the controller. Every other result key, context field, and -stop, retry, lease, and workspace-recovery check stays intact. Full output remains -stored for execution history and detail reads. Terminal telemetry loads only its -run identity, timestamps, usage, and typed task identifier. +`result_json.stderr` before transferring a run to the controller. Native context +also omits saved launch text, execution-continuation bodies, workspace and +environment payloads, and the launch wake envelope. Task identity, retry and +continuation controls, interaction and comment references, execution policy, unknown context fields, and every other +result key stay intact. Legacy reads retain full context. Stop, retry, lease, +and workspace-recovery checks do not change. Full inputs and output remain stored +for execution history and detail reads. Terminal telemetry loads only its run +identity, timestamps, usage, and typed task identifier. Recovery's latest-run queries return the context fields used by recovery decisions. Prompts, execution continuation bodies, and workspace setup payloads diff --git a/server/src/modules/wake-queue/adapters/postgres.test.ts b/server/src/modules/wake-queue/adapters/postgres.test.ts index 6a8c360e22d..83d5bb6ee44 100644 --- a/server/src/modules/wake-queue/adapters/postgres.test.ts +++ b/server/src/modules/wake-queue/adapters/postgres.test.ts @@ -219,7 +219,8 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { executionCancellation: { state: "acknowledged" }, executionRecovery: { kind: "interrupted", providerStopped: true }, configurationIncomplete: { reason: "workspace_unavailable" }, unknownFutureReceipt: { retained: true } }; await db.update(heartbeatRuns).set({ runtimeMode, - ...(runtimeMode === "native" ? { runnerProfileJson: { savedInput: payload }, stdoutExcerpt: payload, stderrExcerpt: payload } : {}), + ...(runtimeMode === "native" ? { runnerProfileJson: { savedInput: payload }, stdoutExcerpt: payload, stderrExcerpt: payload, + contextSnapshot: { issueId, paperclipWake: { savedTask: payload }, paperclipTaskMarkdown: payload } } : {}), resultJson: { ...evidence, stdout: payload.repeat(3), stderr: payload } }).where(eq(heartbeatRuns.id, runId)); await db.update(issues).set({ executionRunId: runId, checkoutRunId: runId }).where(eq(issues.id, issueId)); const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); @@ -233,7 +234,9 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { expect(result.outcome.kind).toBe("released"); expect(rows).toHaveLength(1); expect(rows[0]).toEqual([{ ...run, resultJson: evidence, - ...(runtimeMode === "native" ? { runnerProfileJson: null, stdoutExcerpt: null, stderrExcerpt: null } : {}) }]); + ...(runtimeMode === "native" ? { runnerProfileJson: null, stdoutExcerpt: null, stderrExcerpt: null, + contextSnapshot: { issueId } } : {}) }]); + expect(result.run.contextSnapshot).toEqual({ issueId }); expect(Buffer.byteLength(JSON.stringify(rows))).toBeLessThan(8 * 1024); expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]).toEqual(run); expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toEqual(issue); diff --git a/server/src/modules/wake-queue/adapters/release-run-read.test.ts b/server/src/modules/wake-queue/adapters/release-run-read.test.ts index e4129e6c692..6a669edf617 100644 --- a/server/src/modules/wake-queue/adapters/release-run-read.test.ts +++ b/server/src/modules/wake-queue/adapters/release-run-read.test.ts @@ -71,4 +71,53 @@ describe.skipIf(!support.supported)("wake release run reads", () => { expect((await readReleaseRun(db, { companyId, runId: run.id })).run?.resultJson).toEqual(result); }, ); + + it("keeps launch text in storage while releasing native runs with their control context", async () => { + const bulk = randomBytes(128 * 1024).toString("base64"); + const context = { + issueId: randomUUID(), taskId: randomUUID(), taskKey: "saved-task", + retryReason: "execution_review_participant_recovery", wakeReason: "issue_commented", + explicitUserContinuation: true, + interactionId: randomUUID(), interactionKind: "review", interactionStatus: "resolved", + wakeCommentIds: [randomUUID()], queuedCommentSourceWakeIds: [randomUUID()], + chatConversationId: randomUUID(), executionPolicy: { requireApproval: true }, + opaqueFutureContext: { retained: [null, false, "input"] }, + }; + const launch = { + executionContinuation: { instruction: bulk, state: "saved" }, + paperclipWake: { savedTask: bulk }, paperclipWakeComment: bulk, paperclipTaskMarkdown: bulk, + paperclipTaskMarkdownCompact: bulk, paperclipTaskMarkdownAssignment: bulk, + paperclipTaskMarkdownAssignmentCompact: bulk, paperclipWorkspace: { files: bulk }, + paperclipEnvironment: { input: bulk }, paperclipContinuationSummary: bulk, + }; + const evidence = { nativeCancellation: { schema: "paperclip.native-cancellation.v1", dispatched: true }, + executionCancellation: { state: "acknowledged" }, workspaceRestoreRecovery: { leaseIds: [randomUUID()] }, + configurationIncomplete: { reason: "secret_binding_missing" }, unknownFutureReceipt: { retained: true } }; + const [before] = await db.insert(heartbeatRuns).values({ companyId, agentId, runtimeMode: "native", + contextSnapshot: { ...context, ...launch }, resultJson: evidence }).returning(); + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + let selected: Awaited>; + try { selected = await db.transaction(tx => readReleaseRun(tx as unknown as typeof db, { companyId, runId: before.id })); } + finally { capture.flush(); } + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBe(1); + expect(summary.responseValueBytes).toBeLessThan(8192); + expect(selected!.run?.contextSnapshot).toEqual(context); + expect(selected!.run?.resultJson).toEqual(evidence); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, before.id)))[0]).toEqual(before); + await db.update(heartbeatRuns).set({ runtimeMode: "legacy" }).where(eq(heartbeatRuns.id, before.id)); + expect((await readReleaseRun(db, { companyId, runId: before.id })).run?.contextSnapshot).toEqual(before.contextSnapshot); + }); + + it.each([null, ["paperclipWake", { savedInput: "keep" }], "scalar", 42].map(context => ({ context })))( + "preserves non-object native context $context", async ({ context }) => { + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, runtimeMode: "native", + contextSnapshot: context as Record | null }).returning(); + expect((await readReleaseRun(db, { companyId, runId: run.id })).run?.contextSnapshot).toEqual(context); + }, + ); }); diff --git a/server/src/modules/wake-queue/adapters/release-run-read.ts b/server/src/modules/wake-queue/adapters/release-run-read.ts index cc8bcf80388..3fa7371bc1a 100644 --- a/server/src/modules/wake-queue/adapters/release-run-read.ts +++ b/server/src/modules/wake-queue/adapters/release-run-read.ts @@ -7,6 +7,13 @@ export type ReleaseRun = Omit`case when ${heartbeatRuns.runtimeMode} = 'native' + and jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then ${heartbeatRuns.contextSnapshot} - array['executionContinuation', 'paperclipWake', 'paperclipWakeComment', + 'paperclipTaskMarkdown', 'paperclipTaskMarkdownCompact', 'paperclipTaskMarkdownAssignment', + 'paperclipTaskMarkdownAssignmentCompact', 'paperclipWorkspace', 'paperclipEnvironment', + 'paperclipContinuationSummary'] + else ${heartbeatRuns.contextSnapshot} end`, runnerProfileJson: sql`case when ${heartbeatRuns.runtimeMode} = 'native' then null else ${heartbeatRuns.runnerProfileJson} end`, stdoutExcerpt: sql`case when ${heartbeatRuns.runtimeMode} = 'native' From b65897f9bf05121290895b48e4794c301790ef67 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 09:53:27 -0700 Subject: [PATCH 45/68] perf(recovery): read workspace contract without native launch prompts (#17621) Project native input to the complete schema, binding, provider, and workspace evidence used by execution-blocker checks. Preserve absent and malformed fields and retain all saved inputs and recovery holds. 149 focused tests and 123 workspace-recovery tests pass. Server typecheck, build, all 55 CI checks, and exact-head Apex 5/5 pass. Co-Authored-By: Paperclip --- doc/DATABASE.md | 9 +++-- .../services/execution-blocker-read.test.ts | 37 ++++++++++++++++++- server/src/services/execution-blocker.ts | 5 +++ 3 files changed, 47 insertions(+), 4 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 4d6622bc784..043493acd8f 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -68,9 +68,12 @@ update, including an edit outside the projection, defers repair to a fresh sweep Execution-blocker reads select process, lifecycle, and task-binding fields rather than full execution rows. Context retains typed task and pre-dispatch evidence; -Runner profiles retain the adapter type and complete native execution input and -workspace-sync receipts. Full result evidence stays available to the existing -retry and workspace checks. Stored context, profiles, and output are unchanged. +Runner profiles retain the adapter type, native input schema, binding, provider, +workspace-access evidence, and complete workspace-sync receipts. Native task and +runtime prompt payloads are not needed for these checks. Missing fields and +malformed input values retain their original shape. Full result evidence stays +available to the existing retry and workspace checks. Stored context, profiles, +and output are unchanged. Committed native finalization reads run identity and ownership state before it claims the coordinator. It loads complete saved input only when finalization diff --git a/server/src/services/execution-blocker-read.test.ts b/server/src/services/execution-blocker-read.test.ts index 309385a25cb..5b16c521af1 100644 --- a/server/src/services/execution-blocker-read.test.ts +++ b/server/src/services/execution-blocker-read.test.ts @@ -100,7 +100,9 @@ describe.skipIf(!support.supported)("execution blocker PostgreSQL read projectio }); const input = { schema: "paperclip.native-execution-input.v6", binding: { runId: f.run.id, companyId: f.companyId, agentId: f.agentId, issueId: f.issueId }, - provider: { kind: "grok_bot" }, workspace: { access: "none", ...(hasCwd ? { cwd: null } : {}) } }; + provider: { kind: "grok_bot" }, workspace: { access: "none", ...(hasCwd ? { cwd: null } : {}) }, + task: { description: randomBytes(128 * 1024).toString("base64") }, + runtimeContext: { instructions: randomBytes(128 * 1024).toString("base64") } }; await db.update(heartbeatRuns).set({ nativeIssueId: f.issueId, runnerProfileJson: { ...f.run.runnerProfileJson, nativeExecutionInput: input } }).where(eq(heartbeatRuns.id, f.run.id)); expect(await compare(f)).toMatchObject({ workspaceRepairRequired: !hasCwd, canRetry: false }); @@ -108,6 +110,39 @@ describe.skipIf(!support.supported)("execution blocker PostgreSQL read projectio expect(stored.runnerProfileJson?.nativeExecutionInput).toEqual(input); }); + it.each(["run", "company", "agent", "issue", "provider", "schema", "access", "workspace", "binding", "missing_binding", + "null_input", "array_input", "scalar_input", "number_input"])( + "does not relax native workspace recovery for invalid input (%s)", async kind => { + const f = await fixture({ runtimeMode: "native", driverKind: "grok_bot_mcp" }, { + workspaceRestoreFailure: "restore_unsafe_archive", + workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [] }, + }); + const input: Record = { + schema: "paperclip.native-execution-input.v6", + binding: { runId: f.run.id, companyId: f.companyId, agentId: f.agentId, issueId: f.issueId }, + provider: { kind: "grok_bot" }, workspace: { access: "none", cwd: null }, + task: { description: randomBytes(128 * 1024).toString("base64") }, + }; + const binding = input.binding as Record; + if (kind === "run") binding.runId = randomUUID(); + if (kind === "company") binding.companyId = randomUUID(); + if (kind === "agent") binding.agentId = randomUUID(); + if (kind === "issue") binding.issueId = randomUUID(); + if (kind === "provider") input.provider = { kind: "codex" }; + if (kind === "schema") input.schema = "unsupported"; + if (kind === "access") input.workspace = { cwd: null }; + if (kind === "workspace") input.workspace = "none"; + if (kind === "binding") input.binding = []; + if (kind === "missing_binding") delete input.binding; + const selectedInput = kind === "null_input" ? null : kind === "array_input" ? [] + : kind === "scalar_input" ? "input" : kind === "number_input" ? 42 : input; + await db.update(heartbeatRuns).set({ nativeIssueId: f.issueId, + runnerProfileJson: { ...f.run.runnerProfileJson, nativeExecutionInput: selectedInput } }).where(eq(heartbeatRuns.id, f.run.id)); + expect(await compare(f)).toMatchObject({ workspaceRepairRequired: true, canRetry: false }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)))[0].runnerProfileJson?.nativeExecutionInput).toEqual(selectedInput); + }, + ); + it.each([null, [], "scalar", 42, { issueId: 42, taskId: false }].map(context => ({ context })))("preserves decisions for malformed context $context", async ({ context }) => { const f = await fixture({ contextSnapshot: context as Record | null }); expect(await compare(f)).toMatchObject({ canRetry: true }); diff --git a/server/src/services/execution-blocker.ts b/server/src/services/execution-blocker.ts index 4777d5947b3..3ee94aedb72 100644 --- a/server/src/services/execution-blocker.ts +++ b/server/src/services/execution-blocker.ts @@ -28,6 +28,11 @@ const blockerProfileColumn = sql | null>`case when ${heartbeatRuns.runnerProfileJson} is null then null else ( select coalesce(jsonb_object_agg(field.key, case field.key when 'adapterDispatch' then jsonb_build_object('adapterType', field.value->'adapterType') + when 'nativeExecutionInput' then case when jsonb_typeof(field.value) = 'object' then ( + select coalesce(jsonb_object_agg(input_field.key, input_field.value), '{}'::jsonb) + from jsonb_each(field.value) as input_field(key, value) + where input_field.key in ('schema', 'binding', 'provider', 'workspace') + ) else field.value end else field.value end), '{}'::jsonb) from jsonb_each(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) as field(key, value) From 19ec787e5bdd8684720068e7e39f02996afe7d8c Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 10:06:27 -0700 Subject: [PATCH 46/68] perf(recovery): release native process identity without saved payload reads (#17622) Keep locked process and terminal-state evidence plus JavaScript-compatible checkpoint presence. Preserve every host, provider, lease, coordinator and stop-receipt check without transferring saved prompts or checkpoint bodies. 53 focused tests, server typecheck, build, all 55 CI checks, and exact-head Apex 5/5 pass. Co-Authored-By: Paperclip --- doc/DATABASE.md | 6 +++ .../native-settled-process-release.test.ts | 37 ++++++++++++++++++- .../src/services/native-local-process-stop.ts | 13 +++++-- 3 files changed, 51 insertions(+), 5 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 043493acd8f..51f565e8bed 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -84,6 +84,12 @@ Stop settlement still load complete run evidence. Workspace settlement reads the saved sync descriptor, export retry, and first-heartbeat flag. These projections leave stored prompts, logs, results, and recovery state unchanged. +Settled native process release locks run identity, terminal state, process +identity, and checkpoint presence without loading the checkpoint or execution +payloads. It retains current coordinator checks, provider evidence, local-host +identity checks, and every lease's termination receipt. The stop receipt and +process-field update remain in the same transaction. + Conversation ownership checks select process identity and full recovery result evidence without execution prompts or profiles. Committed native reply repair selects its persisted result and identity bindings plus typed presentation flags. diff --git a/server/src/__tests__/native-settled-process-release.test.ts b/server/src/__tests__/native-settled-process-release.test.ts index be5c00a4516..d2a60b89ccb 100644 --- a/server/src/__tests__/native-settled-process-release.test.ts +++ b/server/src/__tests__/native-settled-process-release.test.ts @@ -1,5 +1,6 @@ import { EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; -import { randomUUID } from "node:crypto"; +import { randomBytes, randomUUID } from "node:crypto"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { spawn } from "node:child_process"; import os from "node:os"; import { and, eq } from "drizzle-orm"; @@ -207,6 +208,40 @@ suite("settled native run process identity", () => { expect((await readRun(runId)).processPid).toBeNull(); }); + it("releases the stopped identity without reading saved execution payloads", async () => { + const { runId } = await seed(); + const bulk = randomBytes(128 * 1024).toString("base64"); + const before = await readRun(runId); + await db.update(heartbeatRuns).set({ contextSnapshot: { ...before.contextSnapshot, savedTask: bulk }, + runnerProfileJson: { ...before.runnerProfileJson, sessionCheckpoint: { providerSessionId: "thread", savedState: bulk }, + nativeExecutionInput: { task: bulk } }, resultJson: { savedResult: bulk }, stdoutExcerpt: bulk, + }).where(eq(heartbeatRuns.id, runId)); + const saved = await readRun(runId); + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { expect(await releaseSettledNativeRunProcess(db, runId)).toBe("released"); } + finally { capture.flush(); } + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.responseValueBytes).toBeLessThan(8192); + const after = await readRun(runId); + expect(after).toEqual({ ...saved, processPid: null, processGroupId: null, processStartedAt: null, + nextEventSeq: saved.nextEventSeq + 1, updatedAt: after.updatedAt }); + expect(await hasNativeLocalProcessStop(db, companyId, runId)).toBe(true); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + }); + + it.each([null, false, 0, "", {}, []].map(checkpoint => ({ checkpoint })))( + "preserves checkpoint presence semantics when releasing an identity ($checkpoint)", async ({ checkpoint }) => { + const { runId, pid } = await seed({ checkpoint: false }); + await db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: checkpoint } }) + .where(eq(heartbeatRuns.id, runId)); + expect(await releaseSettledNativeRunProcess(db, runId)).toBe(checkpoint === null ? "skipped" : "released"); + expect((await readRun(runId)).processPid).toBe(checkpoint === null ? pid : null); + }, + ); + it("keeps the identity while the process is still alive, then releases it after exit", async () => { const child = startDetached(); children.push(child); diff --git a/server/src/services/native-local-process-stop.ts b/server/src/services/native-local-process-stop.ts index 422d2c24f0f..07ff696ec59 100644 --- a/server/src/services/native-local-process-stop.ts +++ b/server/src/services/native-local-process-stop.ts @@ -1,4 +1,4 @@ -import { and, asc, desc, eq, gt, inArray, isNotNull, isNull, or } from "drizzle-orm"; +import { and, asc, desc, eq, gt, inArray, isNotNull, isNull, or, sql } from "drizzle-orm"; import { environmentLeases, heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js"; import { nativeSessionProviderEvidence } from "./native-runtime/native-session-resume.js"; @@ -41,7 +41,8 @@ export async function recordedLocalExecutionStopped(db: Db, run: Pick) { if (run.runtimeMode !== "native" || (!run.processPid && !run.processGroupId)) return false; const leases = await db.select({ provider: environmentLeases.provider }) .from(environmentLeases) @@ -144,13 +145,17 @@ type SettledProcessRelease = "released" | "pending" | "skipped"; export async function releaseSettledNativeRunProcess(db: Db, runId: string): Promise { return db.transaction(async transaction => { const tx = transaction as unknown as Db; - const [run] = await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update").limit(1); + const [run] = await tx.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + runtimeMode: heartbeatRuns.runtimeMode, status: heartbeatRuns.status, finishedAt: heartbeatRuns.finishedAt, + processPid: heartbeatRuns.processPid, processGroupId: heartbeatRuns.processGroupId, processStartedAt: heartbeatRuns.processStartedAt, + hasSessionCheckpoint: sql`coalesce(${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint', 'null'::jsonb) <> 'null'::jsonb`, + }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update").limit(1); if (!run || run.runtimeMode !== "native" || !SETTLED_RUN_STATUSES.includes(run.status) || !run.finishedAt || (run.processPid === null && run.processGroupId === null && run.processStartedAt === null)) return "skipped"; const [coordinator] = await tx.select({ phase: nativeRunFinalizations.phase }).from(nativeRunFinalizations) .where(eq(nativeRunFinalizations.runId, run.id)).limit(1); if (coordinator && !SETTLED_COORDINATOR_PHASES.includes(coordinator.phase)) return "skipped"; - if (run.runnerProfileJson?.sessionCheckpoint == null && + if (!run.hasSessionCheckpoint && !(await nativeSessionProviderEvidence(tx, [run.id])).has(run.id)) return "skipped"; const leases = await tx.select().from(environmentLeases).where(and( eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id), From 332e4c5df9199b41e5d4b3ff757cdc4644179f8c Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 10:47:02 -0700 Subject: [PATCH 47/68] fix(native): settle completed results that retain a valid monitor (#17641) Preserve a valid saved monitor after its successful source releases execution and checkout. Recheck accepted result identity, receipts, ownership and governance under the task lock. Notify the existing scheduler once and preserve accepted live and recovered replies. Apex 5/5 after fixing its reply-presentation finding. All CI checks, 191 focused tests, 293 broader native tests, workspace typecheck and server build pass. Hosted effects require staging verification. Co-Authored-By: Paperclip --- doc/architecture/native-status-arbitration.md | 24 ++ .../__tests__/heartbeat-run-summary.test.ts | 4 +- server/src/services/heartbeat-run-summary.ts | 2 +- .../native-runtime/native-monitor-delivery.ts | 38 ++- .../native-preserved-monitor-wait.test.ts | 217 ++++++++++++++++++ .../native-runtime/native-run-finalizer.ts | 13 +- .../native-runtime/status-arbiter.test.ts | 2 +- .../services/native-runtime/status-arbiter.ts | 10 +- .../status-decision-committer.ts | 15 +- 9 files changed, 315 insertions(+), 10 deletions(-) create mode 100644 server/src/services/native-runtime/native-preserved-monitor-wait.test.ts diff --git a/doc/architecture/native-status-arbitration.md b/doc/architecture/native-status-arbitration.md index f56014fa5b0..7fa6d5782ed 100644 --- a/doc/architecture/native-status-arbitration.md +++ b/doc/architecture/native-status-arbitration.md @@ -374,3 +374,27 @@ and records a recovery action for the reviewer without an automatic retry loop. Both Codex and ACPX providers wait for server completion feedback before their terminal tool call succeeds. A rejected completion report stays correctable in the same provider turn. The runner does not latch it as an accepted result. + +## Completed runs with saved monitors + +A successful accepted result can report a monitor wait after the run has +already released execution and checkout. Finalization can preserve that wait +when the same agent still owns a valid ordinary task monitor and the completed +run has a server-recorded `set_task_monitor` receipt for the task. The current +monitor can have a later deadline; finalization does not restore an older timer. + +The server rechecks the accepted source, current monitor, empty execution and +checkout slots, and governance under the coordinator and task locks. The +`scheduled_monitor_preserved` decision keeps task status and has no task effects. +It records the completed result without dispatching another provider turn. +The existing monitor scheduler receives a transaction-aware notification when +this finalization blocker clears. Replaying the committed decision sends no +second notification. +Live response selection and committed response repair retain the accepted +monitor explanation under the existing reply and publication authorization rules. + +Missing receipts, changed ownership, cancellation intents, cleared or quota +monitors, and unfinished workspace finalization retain their recovery paths. +Due and exhausted monitors still belong to the scheduler. This does not change +their deadlines or attempts, clear unrelated recovery holds, or relax the +Cloud sleep and wake contract. diff --git a/server/src/__tests__/heartbeat-run-summary.test.ts b/server/src/__tests__/heartbeat-run-summary.test.ts index 33b03986b83..c0ed3d6b113 100644 --- a/server/src/__tests__/heartbeat-run-summary.test.ts +++ b/server/src/__tests__/heartbeat-run-summary.test.ts @@ -25,10 +25,10 @@ describe("model-facing run errors", () => { }); }); -describe("scheduled continuation presentation", () => { +describe.each(["scheduled_monitor_waiting", "scheduled_monitor_dispatched", "scheduled_monitor_preserved"])("scheduled continuation presentation (%s)", finalizationReasonCode => { const resultJson = { finalizationPhase: "committed", - finalizationReasonCode: "scheduled_monitor_waiting", + finalizationReasonCode, nativeResult: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", diff --git a/server/src/services/heartbeat-run-summary.ts b/server/src/services/heartbeat-run-summary.ts index 25e0de11276..97368cb1da6 100644 --- a/server/src/services/heartbeat-run-summary.ts +++ b/server/src/services/heartbeat-run-summary.ts @@ -298,7 +298,7 @@ export function readAcceptedTaskContinuationSummary( const result = record(resultJson); if (result.finalizationPhase !== "committed") return null; const kind = result.finalizationReasonCode === "live_continuation_registered" ? "same_agent" - : ["scheduled_monitor_waiting", "scheduled_monitor_dispatched"].includes(String(result.finalizationReasonCode)) ? "monitor" : null; + : ["scheduled_monitor_waiting", "scheduled_monitor_dispatched", "scheduled_monitor_preserved"].includes(String(result.finalizationReasonCode)) ? "monitor" : null; if (!kind) return null; for (const candidate of semanticResultCandidates(result)) { const continuation = record(candidate.continuation); diff --git a/server/src/services/native-runtime/native-monitor-delivery.ts b/server/src/services/native-runtime/native-monitor-delivery.ts index d38e41d8bc1..b2af0e80605 100644 --- a/server/src/services/native-runtime/native-monitor-delivery.ts +++ b/server/src/services/native-runtime/native-monitor-delivery.ts @@ -1,7 +1,8 @@ -import { agentWakeupRequests, heartbeatRuns, type Db } from "@paperclipai/db"; -import { and, desc, eq, inArray, ne, sql } from "drizzle-orm"; +import { agentWakeupRequests, heartbeatRuns, nativeRunFinalizations, nativeRunResults, type Db, type issues } from "@paperclipai/db"; +import { and, desc, eq, inArray, isNotNull, ne, sql } from "drizzle-orm"; import { alias } from "drizzle-orm/pg-core"; import { boundRunIssueIdSql } from "../run-task-binding.js"; +import { scheduledIssueMonitorWait } from "../issue-monitors.js"; function record(value: unknown): Record { return value !== null && typeof value === "object" && !Array.isArray(value) @@ -9,6 +10,39 @@ function record(value: unknown): Record { } export type NativeMonitorDelivery = { wakeId: string; successorRunId: string }; +export type NativePreservedMonitorWait = { agentId: string; nextCheckAt: string; resultId: string }; + +export async function readNativePreservedMonitorWait( + db: Db, + binding: { companyId: string; issueId: string; runId: string; agentId: string }, + issue: typeof issues.$inferSelect, +): Promise { + if (issue.companyId !== binding.companyId || issue.id !== binding.issueId || issue.workMode !== "standard" || + issue.executionRunId !== null || issue.checkoutRunId !== null) return null; + const nextCheckAt = scheduledIssueMonitorWait(issue, binding.agentId); + if (!nextCheckAt) return null; + const [source] = await db.select({ resultId: nativeRunResults.id, hasReceipt: sql`exists ( + select 1 from jsonb_each(case when jsonb_typeof(${heartbeatRuns.resultJson}->'semanticToolReceipts') = 'object' + then ${heartbeatRuns.resultJson}->'semanticToolReceipts' else '{}'::jsonb end) receipt + where receipt.value->>'operationId' = 'set_task_monitor' + and receipt.value->'result'->>'taskId' = ${binding.issueId} + and receipt.value->'result'->'monitor'->>'status' = 'scheduled' + and jsonb_typeof(receipt.value->'result'->'monitor'->'nextCheckAt') = 'string' + )` }).from(heartbeatRuns).innerJoin(nativeRunFinalizations, and( + eq(nativeRunFinalizations.companyId, binding.companyId), eq(nativeRunFinalizations.issueId, binding.issueId), + eq(nativeRunFinalizations.runId, heartbeatRuns.id), + )).innerJoin(nativeRunResults, and( + eq(nativeRunResults.id, nativeRunFinalizations.resultId), eq(nativeRunResults.companyId, binding.companyId), + eq(nativeRunResults.issueId, binding.issueId), eq(nativeRunResults.runId, heartbeatRuns.id), + eq(nativeRunResults.schemaStatus, "accepted"), + )).where(and(eq(heartbeatRuns.id, binding.runId), eq(heartbeatRuns.companyId, binding.companyId), + eq(heartbeatRuns.agentId, binding.agentId), eq(heartbeatRuns.nativeIssueId, binding.issueId), + eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.status, "succeeded"), isNotNull(heartbeatRuns.finishedAt), + ...["cancellation", "nativeCancellation", "executionCancellation", "startupCancellation"].map(key => + sql`coalesce(${heartbeatRuns.resultJson}->${key}, 'null'::jsonb) = 'null'::jsonb`), + )).limit(1); + return source?.hasReceipt ? { agentId: binding.agentId, nextCheckAt, resultId: source.resultId } : null; +} export async function readNativeMonitorSource( db: Pick, diff --git a/server/src/services/native-runtime/native-preserved-monitor-wait.test.ts b/server/src/services/native-runtime/native-preserved-monitor-wait.test.ts new file mode 100644 index 00000000000..ad269c7467c --- /dev/null +++ b/server/src/services/native-runtime/native-preserved-monitor-wait.test.ts @@ -0,0 +1,217 @@ +import { randomUUID } from "node:crypto"; +import { and, eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, agentWakeupRequests, companies, completionContracts, createDb, heartbeatRuns, + issueComments, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, statusDecisions, subscribeDatabaseWork } from "@paperclipai/db"; +import { PROVIDER_QUOTA_MONITOR_SERVICE_NAME } from "@paperclipai/shared"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js"; +import { finalizeNativeRun, repairCommittedNativeChatResponse } from "./native-run-finalizer.js"; +import { nativeSha256 } from "./canonical.js"; +import * as monitorDelivery from "./native-monitor-delivery.js"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; +import { hasPendingRunCleanup } from "../execution-blocker.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("released native monitor results", () => { + let temporary: Awaited>; + let db: ReturnType; + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("native-preserved-monitor-"); + db = createDb(temporary.connectionString); + }); + afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); }); + + async function fixture() { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); + const contractId = randomUUID(), resultId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Saved monitor", issuePrefix: `P${companyId.slice(0, 7)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "paperclip_runner", status: "active" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Saved follow-up", status: "in_progress", assigneeAgentId: agentId }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issueId, + runtimeMode: "native", status: "running", contextSnapshot: { issueId } }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + const nextCheckAt = new Date(Date.now() + 3_600_000).toISOString(); + const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId }); + await authority.execute({ tool: "set_task_monitor", callId: randomUUID(), arguments: { + idempotencyKey: "saved-monitor", monitor: { nextCheckAt, notes: "Check the saved result." }, + } }); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", + contractJson: { revision: "test", criteria: [] }, canonicalSha256: randomUUID(), + createdByActorType: "system", createdByActorId: "test" }); + const result = { result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", + summary: "Wait for the saved check.", completionClaim: { contractRevision: "test", objectiveSatisfied: false, + criteria: [], remainingWork: [{ description: "Check again", blocksCompletion: true }] }, + continuation: { kind: "monitor", summary: "Wait for the check", idempotencyKey: "provider-selected" }, + evidence: [], verification: [], attentionRequests: [], artifacts: [] }, + terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState: "completed", reportedWorkDisposition: "yielded" } }; + const finishedAt = new Date("2020-01-01T00:00:00Z"); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt, completionContractId: contractId }).where(eq(heartbeatRuns.id, runId)); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId, completionContractId: contractId, + serverFingerprint: randomUUID(), canonicalSha256: randomUUID(), schemaStatus: "accepted", resultJson: result }); + await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId, phase: "retryable_failure", resultId }); + await db.update(issues).set({ executionRunId: null, checkoutRunId: null }).where(eq(issues.id, issueId)); + const readIssue = async () => (await db.select().from(issues).where(eq(issues.id, issueId)))[0]!; + const finalize = (workspaceFinalizeStatus: "succeeded" | "failed" = "succeeded") => finalizeNativeRun({ + db, runId, workspaceFinalizeStatus, projectRunStatus: true, + }); + return { companyId, agentId, issueId, runId, resultId, result, finishedAt, nextCheckAt, readIssue, finalize }; + } + + it.each(["original", "rescheduled", "past_due"])("keeps the saved monitor and settles a released successful result (%s)", async state => { + const f = await fixture(); + if (state !== "original") { + const issue = await f.readIssue(); + const nextCheckAt = state === "past_due" ? "2020-01-01T01:00:00Z" : new Date(Date.now() + 7_200_000).toISOString(); + await db.update(issues).set({ monitorNextCheckAt: new Date(nextCheckAt), executionPolicy: { + ...issue.executionPolicy, monitor: { ...(issue.executionPolicy?.monitor as Record), nextCheckAt }, + } }).where(eq(issues.id, f.issueId)); + } + const issueBefore = await f.readIssue(); + const [hold] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId, + kind: "active_run_watchdog", cause: "legacy_execution_requires_reconciliation", fingerprint: randomUUID(), + status: "active", ownerType: "board", nextAction: "Preserve independent recovery.", evidence: { runId: randomUUID() }, + }).returning(); + await f.finalize(); + const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(coordinator.phase).toBe("committed"); + const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId)); + expect(decisions.at(-1)).toMatchObject({ reasonCode: "scheduled_monitor_preserved", toStatus: issueBefore.status }); + expect(decisions.at(-1)?.decisionJson).toMatchObject({ effects: [] }); + expect(await f.readIssue()).toMatchObject({ status: issueBefore.status, executionPolicy: issueBefore.executionPolicy, + monitorNextCheckAt: issueBefore.monitorNextCheckAt, monitorAttemptCount: issueBefore.monitorAttemptCount, + executionRunId: null, checkoutRunId: null }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].finishedAt).toEqual(f.finishedAt); + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0].resultJson).toEqual(f.result); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, hold!.id))).toEqual([hold]); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toEqual([]); + expect(await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toEqual([{ id: f.runId }]); + await f.finalize(); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId))).toEqual(decisions); + }); + + it.each(["cleared", "owner", "checkout", "other_agent", "no_receipt", "foreign_receipt", "unfinished", "failed_source", "quota", "workspace_failed", + "cancellation", "nativeCancellation", "executionCancellation", "startupCancellation"])( + "does not settle a released monitor without current proof (%s)", async state => { + const f = await fixture(); + const issue = await f.readIssue(); + if (state === "cleared") await db.update(issues).set({ monitorNextCheckAt: null, executionPolicy: {} }).where(eq(issues.id, f.issueId)); + if (state === "quota") await db.update(issues).set({ executionPolicy: { ...issue.executionPolicy, + monitor: { ...(issue.executionPolicy?.monitor as Record), serviceName: PROVIDER_QUOTA_MONITOR_SERVICE_NAME }, + } }).where(eq(issues.id, f.issueId)); + if (state === "other_agent") await db.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, f.issueId)); + if (state === "owner" || state === "checkout") { + const [owner] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }).returning(); + await db.update(issues).set(state === "owner" ? { executionRunId: owner!.id } : { checkoutRunId: owner!.id }).where(eq(issues.id, f.issueId)); + } + if (state === "unfinished") await db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, f.runId)); + if (state === "failed_source") await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, f.runId)); + if (state === "no_receipt") await db.update(heartbeatRuns).set({ resultJson: {} }).where(eq(heartbeatRuns.id, f.runId)); + if (["cancellation", "nativeCancellation", "executionCancellation", "startupCancellation"].includes(state)) { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + await db.update(heartbeatRuns).set({ resultJson: { ...run.resultJson, [state]: { state: "pending" } } }).where(eq(heartbeatRuns.id, f.runId)); + } + if (state === "foreign_receipt") await db.update(heartbeatRuns).set({ resultJson: { semanticToolReceipts: { + invalid: { operationId: "set_task_monitor", result: { taskId: randomUUID(), monitor: { status: "scheduled", nextCheckAt: f.nextCheckAt } } }, + } } }).where(eq(heartbeatRuns.id, f.runId)); + const before = await f.readIssue(); + await f.finalize(state === "workspace_failed" ? "failed" : "succeeded"); + const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(coordinator.phase).toBe("retryable_failure"); + expect(await f.readIssue()).toMatchObject({ executionPolicy: before.executionPolicy, monitorNextCheckAt: before.monitorNextCheckAt, + executionRunId: before.executionRunId, checkoutRunId: before.checkoutRunId, assigneeAgentId: before.assigneeAgentId }); + expect(await db.select().from(statusDecisions).where(and(eq(statusDecisions.runId, f.runId), eq(statusDecisions.reasonCode, "scheduled_monitor_preserved")))).toEqual([]); + }, + ); + + it("supersedes an earlier authority error without replaying the accepted result", async () => { + const f = await fixture(), scheduled = await f.readIssue(); + await db.update(issues).set({ executionPolicy: {}, monitorNextCheckAt: null }).where(eq(issues.id, f.issueId)); + await f.finalize(); + const old = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId)); + expect(old.some(decision => decision.reasonCode === "monitor_wait_authority_lost")).toBe(true); + await db.update(issues).set({ executionPolicy: scheduled.executionPolicy, monitorNextCheckAt: scheduled.monitorNextCheckAt }) + .where(eq(issues.id, f.issueId)); + await f.finalize(); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0].phase).toBe("committed"); + const after = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId)); + for (const before of old) expect(after.find(decision => decision.id === before.id)).toEqual(before); + expect(after.some(decision => decision.reasonCode === "scheduled_monitor_preserved")).toBe(true); + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0].resultJson).toEqual(f.result); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toEqual([]); + }); + + it("notifies the existing monitor scheduler once when finalization stops blocking it", async () => { + const f = await fixture(), events: string[] = []; + expect(await hasPendingRunCleanup(db, f.companyId, f.runId)).toBe(true); + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.issueMonitors, event => { events.push(event); }); + try { + await f.finalize(); + expect(events).toEqual(["intent", "settled"]); + expect(await hasPendingRunCleanup(db, f.companyId, f.runId)).toBe(false); + expect((await f.readIssue()).monitorNextCheckAt?.toISOString()).toBe(f.nextCheckAt); + await f.finalize(); + expect(events).toEqual(["intent", "settled"]); + } finally { unsubscribe(); } + }); + + it.each(["live", "recovered"])("publishes the accepted monitor explanation once (%s)", async mode => { + const f = await fixture(), before = await f.readIssue(); + const completionContractSha256 = `sha256:${nativeSha256({ revision: "test", criteria: [] })}`; + const canonicalSha256 = `sha256:${nativeSha256({ ...f.result, turnId: null })}`; + const serverFingerprint = `sha256:${nativeSha256({ runId: f.runId, completionContractSha256, canonicalSha256 })}`; + await db.update(nativeRunResults).set({ canonicalSha256, serverFingerprint }).where(eq(nativeRunResults.id, f.resultId)); + await db.update(heartbeatRuns).set({ completionContractSha256, + contextSnapshot: { issueId: f.issueId, ...(mode === "recovered" ? { skipIssueComment: true } : {}) }, + }).where(eq(heartbeatRuns.id, f.runId)); + const comments = () => db.select({ id: issueComments.id, body: issueComments.body }).from(issueComments) + .where(and(eq(issueComments.companyId, f.companyId), eq(issueComments.createdByRunId, f.runId))); + const repair = () => repairCommittedNativeChatResponse(db, { companyId: f.companyId, issueId: f.issueId, runId: f.runId }); + await f.finalize(); + if (mode === "recovered") { + expect(await comments()).toEqual([]); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId } }).where(eq(heartbeatRuns.id, f.runId)); + expect(await repair()).toBe(true); + } + const published = await comments(); + expect(published).toEqual([{ id: expect.any(String), body: f.result.result.summary }]); + expect(await repair()).toBe(false); + await f.finalize(); + expect(await comments()).toEqual(published); + expect((await f.readIssue()).monitorNextCheckAt).toEqual(before.monitorNextCheckAt); + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0].resultJson).toEqual(f.result); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toEqual([]); + }); + + it.each(["cleared", "owner", "receipt", "rescheduled"])("rechecks monitor proof under the task lock (%s)", async state => { + const f = await fixture(), scheduled = await f.readIssue(); + const [owner] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }).returning(); + const nextCheckAt = new Date(Date.now() + 7_200_000).toISOString(); + const original = monitorDelivery.readNativePreservedMonitorWait; + const events: string[] = []; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.issueMonitors, event => { events.push(event); }); + const race = vi.spyOn(monitorDelivery, "readNativePreservedMonitorWait").mockImplementationOnce(async (...args) => { + const proof = await original(...args); + expect(proof).not.toBeNull(); + if (state === "cleared") await db.update(issues).set({ executionPolicy: {}, monitorNextCheckAt: null }).where(eq(issues.id, f.issueId)); + if (state === "owner") await db.update(issues).set({ executionRunId: owner!.id }).where(eq(issues.id, f.issueId)); + if (state === "receipt") await db.update(heartbeatRuns).set({ resultJson: {} }).where(eq(heartbeatRuns.id, f.runId)); + if (state === "rescheduled") await db.update(issues).set({ monitorNextCheckAt: new Date(nextCheckAt), executionPolicy: { + ...scheduled.executionPolicy, monitor: { ...(scheduled.executionPolicy?.monitor as Record), nextCheckAt }, + } }).where(eq(issues.id, f.issueId)); + return proof; + }); + try { await f.finalize(); } finally { race.mockRestore(); unsubscribe(); } + const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(coordinator.phase).toBe(state === "rescheduled" ? "committed" : "retryable_failure"); + expect(events).toEqual(state === "rescheduled" ? ["intent", "settled"] : []); + const task = await f.readIssue(); + expect(task.executionRunId).toBe(state === "owner" ? owner!.id : null); + expect(task.monitorNextCheckAt?.toISOString() ?? null).toBe(state === "cleared" ? null : state === "rescheduled" ? nextCheckAt : f.nextCheckAt); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, owner!.id)))[0].status).toBe("running"); + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0].resultJson).toEqual(f.result); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toEqual([]); + }); +}); diff --git a/server/src/services/native-runtime/native-run-finalizer.ts b/server/src/services/native-runtime/native-run-finalizer.ts index 805fd623d18..1f8b830ff03 100644 --- a/server/src/services/native-runtime/native-run-finalizer.ts +++ b/server/src/services/native-runtime/native-run-finalizer.ts @@ -4,7 +4,7 @@ import { reconcileMuseNativeTerminalRun } from "./muse-terminal-reconciliation.j import { assertNativeCaseHandoffResult } from "./native-case-handoff.js"; import { scheduledIssueMonitorWait } from "../issue-monitors.js"; import { hasPublishedGitHubReview } from "../chat-channels/github/review-wait.js"; -import { readNativeMonitorDelivery } from "./native-monitor-delivery.js"; +import { readNativeMonitorDelivery, readNativePreservedMonitorWait } from "./native-monitor-delivery.js"; import { isNativePlanWaitResult, readNativePlanWait } from "./native-plan-wait.js"; import { hasPendingNativeChildCompletion } from "./native-child-completion-delivery.js"; import { settleExternalConversation } from "../slack-conversation-lifecycle.js"; @@ -1331,10 +1331,16 @@ export async function finalizeNativeRun(input: { }) : "not_applicable"; const monitorWaitAt = scheduledIssueMonitorWait(authoritativeIssue, run.agentId); + const preservedMonitorWait = terminalState === "succeeded" && input.workspaceFinalizeStatus === "succeeded" && + assessment.reportedDisposition === "yielded" && assessment.continuation?.kind === "monitor" + ? await readNativePreservedMonitorWait(input.db, { + companyId: run.companyId, issueId: authoritativeIssue.id, runId: run.id, agentId: run.agentId, + }, authoritativeIssue) : null; + if (preservedMonitorWait && preservedMonitorWait.resultId !== resultRow.id) throw new NativeStatusRaceError(); const supersededReviews = terminalState === "succeeded" && input.workspaceFinalizeStatus === "succeeded" && assessment.attentionRequests.length === 0 && (assessment.reportedDisposition === "done" || (assessment.reportedDisposition === "yielded" && (externalChatResponseWaitAuthorization === "authorized" || - (assessment.continuation?.kind === "monitor" && monitorWaitAt !== null)))) + (assessment.continuation?.kind === "monitor" && monitorWaitAt !== null && !preservedMonitorWait)))) ? await findSupersededExternalReviewRequests(input.db, { companyId: run.companyId, issueId: authoritativeIssue.id, runId: run.id, agentId: run.agentId, }) : []; @@ -1419,6 +1425,7 @@ export async function finalizeNativeRun(input: { const proposedDecision = resolveNativeFinalizerStatus({ caseHandoffAuthorized: Boolean(await assertNativeCaseHandoffResult(input.db, run.id, authoritativeIssue.id, result)), monitorWaitDispatched: monitorDelivery !== null, + monitorWaitPreserved: preservedMonitorWait !== null, monitorWaitAuthorized: authoritativeIssue.workMode === "standard" && authoritativeIssue.executionRunId === run.id && monitorWaitAt !== null, planWaitAuthorized: planWait !== null, hasPendingChildCompletion, @@ -1522,6 +1529,8 @@ export async function finalizeNativeRun(input: { ? { agentId: run.agentId, nextCheckAt: monitorWaitAt } : undefined, requireMonitorDelivery: decision.reasonCode === "scheduled_monitor_dispatched" ? monitorDelivery ?? undefined : undefined, + requirePreservedMonitorWait: decision.reasonCode === "scheduled_monitor_preserved" + ? preservedMonitorWait ?? undefined : undefined, requirePlanWaitSource: decision.reasonCode === "native_plan_accepted_waiting_for_continuation" ? planWait?.source : undefined, diff --git a/server/src/services/native-runtime/status-arbiter.test.ts b/server/src/services/native-runtime/status-arbiter.test.ts index 6262e01adaf..9a3609f9a2c 100644 --- a/server/src/services/native-runtime/status-arbiter.test.ts +++ b/server/src/services/native-runtime/status-arbiter.test.ts @@ -560,7 +560,7 @@ describe("native status authority", () => { expect.objectContaining({ statusAction: "blocked", toStatus: "blocked", - policyVersion: "phase6-v12", + policyVersion: "phase6-v13", reasonCode: "current_track_blocker_waiting", unblockDescriptor: { owner: "board", diff --git a/server/src/services/native-runtime/status-arbiter.ts b/server/src/services/native-runtime/status-arbiter.ts index 5c6e4679e12..4fdc5496cc1 100644 --- a/server/src/services/native-runtime/status-arbiter.ts +++ b/server/src/services/native-runtime/status-arbiter.ts @@ -1,7 +1,7 @@ import type { NativeEvidenceAssessment } from "./evidence-classifier.js"; import { NATIVE_MODEL_REJECTION_MESSAGE, NATIVE_PROVIDER_CAPACITY_MAX_RETRIES, NATIVE_PROVIDER_OVERLOADED_CODE, NATIVE_PROVIDER_OVERLOADED_MESSAGE } from "./native-provider-failure.js"; -export const NATIVE_STATUS_ARBITER_POLICY_VERSION = "phase6-v12"; +export const NATIVE_STATUS_ARBITER_POLICY_VERSION = "phase6-v13"; export type NativeAuthoritativeIssueStatus = | "backlog" @@ -118,6 +118,7 @@ export function arbitrateNativeStatus(input: { caseHandoffAuthorized?: boolean; /** The saved timer already admitted another run and this run no longer owns checkout. */ monitorWaitDispatched?: boolean; + monitorWaitPreserved?: boolean; boardResponseWaitAuthorized?: boolean; boardResponseWaitOrigin?: boolean; isConversation?: boolean; @@ -483,6 +484,13 @@ export function arbitrateNativeStatus(input: { effects: [], }; } + if (input.monitorWaitPreserved) { + return { + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + statusAction: "preserve", toStatus: input.priorIssueStatus, + reasonCode: "scheduled_monitor_preserved", unblockDescriptor: null, effects: [], + }; + } if (input.monitorWaitAuthorized) { return { policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, diff --git a/server/src/services/native-runtime/status-decision-committer.ts b/server/src/services/native-runtime/status-decision-committer.ts index eafd64245c9..c7684448cbf 100644 --- a/server/src/services/native-runtime/status-decision-committer.ts +++ b/server/src/services/native-runtime/status-decision-committer.ts @@ -3,7 +3,7 @@ import { hasPublishedGitHubReview } from "../chat-channels/github/review-wait.js import { pendingNativeGovernance } from "./native-governance.js"; import { findSupersededExternalReviewRequests, supersedeExternalReviewRequestsInTransaction } from "./superseded-external-review-requests.js"; import { normalizePrpResultSignals, type PrpStructuredRunResult } from "../../vendor/paperclip-runner/index.js"; -import { readNativeMonitorDelivery, type NativeMonitorDelivery } from "./native-monitor-delivery.js"; +import { readNativeMonitorDelivery, readNativePreservedMonitorWait, type NativeMonitorDelivery, type NativePreservedMonitorWait } from "./native-monitor-delivery.js"; import { readNativePlanWait, type NativePlanWaitSource } from "./native-plan-wait.js"; import { activeIssueInteractionCondition } from "../issue-question-context.js"; import { isConversation } from "../agent-conversations.js"; @@ -1565,6 +1565,7 @@ export async function commitNativeStatusDecision(input: { requirePlanWaitSource?: NativePlanWaitSource; requireMonitorWait?: { agentId: string; nextCheckAt: string }; requireMonitorDelivery?: NativeMonitorDelivery; + requirePreservedMonitorWait?: NativePreservedMonitorWait; requireNoPendingChildCompletion?: NativeChildCompletionRecipient; requireModelRejectionOwner?: { agentId: string; reviewContext: NativeReviewAssignmentContext | null }; requireProviderFailureOwner?: { agentId: string; reviewContext: NativeReviewAssignmentContext | null }; @@ -1675,6 +1676,18 @@ export async function commitNativeStatusDecision(input: { input.decision.statusAction !== "preserve" || input.decision.toStatus !== issue.status || input.decision.effects.length !== 0) throw new NativeStatusRaceError(); } + if (reasonCode === "scheduled_monitor_preserved") { + const expected = input.requirePreservedMonitorWait; + const actual = expected ? await readNativePreservedMonitorWait(tx as unknown as Db, { + companyId: input.companyId, issueId: input.issueId, runId: input.runId, agentId: expected.agentId, + }, issue) : null; + if (!expected || !actual || actual.nextCheckAt !== expected.nextCheckAt || actual.resultId !== expected.resultId || + input.decision.statusAction !== "preserve" || input.decision.toStatus !== issue.status || + input.decision.effects.length !== 0 || await pendingNativeGovernance({ db: tx as unknown as Db, + companyId: input.companyId, issueId: input.issueId, runId: input.runId, executionState: record(issue.executionState), + })) throw new NativeStatusRaceError(); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); + } // Recheck under the parent status lock: a child can finish between the // finalizer's read and this commit. Re-arbitrate instead of discarding its wake. if (input.requireNoPendingChildCompletion && await hasPendingNativeChildCompletion( From 82c998233c10ca23f88e0b4ed7dbceb74feb2684 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 11:06:22 -0700 Subject: [PATCH 48/68] perf(recovery): omit unused launch payloads from reconciliation reads (#17645) Project unused launch context and native execution input out of two preliminary recovery reads. Preserve full results, unknown controls, complete legacy profiles, current filters, native rejection and full locked terminalization. Apex 5/5 with no unresolved comments and all CI checks green. Ten combined local suites pass (370 tests), plus workspace typecheck and server build. Hosted traffic effects require staging verification. Co-Authored-By: Paperclip --- doc/DATABASE.md | 6 + .../src/services/legacy-execution-recovery.ts | 20 +++- .../recovery/reconciliation-run-read.test.ts | 107 ++++++++++++++++++ server/src/services/recovery/service.ts | 5 +- 4 files changed, 135 insertions(+), 3 deletions(-) create mode 100644 server/src/services/recovery/reconciliation-run-read.test.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 51f565e8bed..961d532aef9 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -66,6 +66,12 @@ the PostgreSQL row version. Native blocked-wait repair compares that version against a fresh full source read before applying changes. Any intervening row update, including an edit outside the projection, defers repair to a fresh sweep. +The two preliminary reconciliation reads omit named launch renderings and +preparation payloads. Native profiles omit unused native execution input. +They retain all other context, full results, legacy profiles, task identity, +retry accounting, and unknown controls. Native runs still reach their native +recovery guard. Terminalization keeps its full locked source and stored input. + Execution-blocker reads select process, lifecycle, and task-binding fields rather than full execution rows. Context retains typed task and pre-dispatch evidence; Runner profiles retain the adapter type, native input schema, binding, provider, diff --git a/server/src/services/legacy-execution-recovery.ts b/server/src/services/legacy-execution-recovery.ts index 5bbea0e61a5..3561bff8768 100644 --- a/server/src/services/legacy-execution-recovery.ts +++ b/server/src/services/legacy-execution-recovery.ts @@ -8,7 +8,7 @@ import { normalizeMaxTurnStopReason } from "./heartbeat-stop-metadata.js"; import { claimedAdapterType, hasConversationContinuationPolicy } from "./conversation-continuation.js"; import { canonicalExecutionUuid } from "./execution-identity.js"; import { randomUUID } from "node:crypto"; -import { and, desc, eq, inArray, isNull, or, sql, type SQL } from "drizzle-orm"; +import { and, desc, eq, getTableColumns, inArray, isNull, or, sql, type SQL } from "drizzle-orm"; import { environmentLeases, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, type Db } from "@paperclipai/db"; import { issueRecoveryActionService } from "./issue-recovery-actions.js"; import { parseIssueExecutionState } from "./issue-execution-policy.js"; @@ -20,6 +20,24 @@ import { issueService } from "./issues.js"; type Run = typeof heartbeatRuns.$inferSelect; export const LEGACY_RECOVERY_CAUSE = "legacy_execution_requires_reconciliation"; +const unusedReconciliationContextKeys = [ + "executionContinuation", "paperclipWake", "paperclipWakeComment", + "paperclipTaskMarkdown", "paperclipTaskMarkdownCompact", "paperclipTaskAssignment", + "paperclipTaskAssignmentCompact", "paperclipWorkspace", "paperclipEnvironment", "paperclipContinuationSummary", +] as const; + +export const legacyReconciliationRunFields = { + ...getTableColumns(heartbeatRuns), + contextSnapshot: sql`case + when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then ${heartbeatRuns.contextSnapshot} - ARRAY[${sql.join(unusedReconciliationContextKeys.map(key => sql`${key}`), sql`, `)}]::text[] + else ${heartbeatRuns.contextSnapshot} end`, + runnerProfileJson: sql`case + when ${heartbeatRuns.runtimeMode} = 'native' and jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' + then ${heartbeatRuns.runnerProfileJson} - 'nativeExecutionInput' + else ${heartbeatRuns.runnerProfileJson} end`, +}; + /** Error families describe availability, not whether earlier actions happened. */ export function legacyExecutionNeedsReconciliation( run: Pick & Partial>, diff --git a/server/src/services/recovery/reconciliation-run-read.test.ts b/server/src/services/recovery/reconciliation-run-read.test.ts new file mode 100644 index 00000000000..ff42eb35aac --- /dev/null +++ b/server/src/services/recovery/reconciliation-run-read.test.ts @@ -0,0 +1,107 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { agents, agentWakeupRequests, companies, createDb, heartbeatRuns, issueRecoveryActions, issues } from "@paperclipai/db"; +import { and, eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { executionFailureRetryCount } from "../execution-recovery-attempt.js"; +import { legacyExecutionNeedsReconciliationWithEvidence, legacyReconciliationRunFields, terminalizeLegacyExecution } from "../legacy-execution-recovery.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("reconciliation source PostgreSQL reads", () => { + let temporary: Awaited>; + let db: ReturnType; + let companyId: string, agentId: string; + const launchKeys = ["executionContinuation", "paperclipWake", "paperclipWakeComment", "paperclipTaskMarkdown", + "paperclipTaskMarkdownCompact", "paperclipTaskAssignment", "paperclipTaskAssignmentCompact", "paperclipWorkspace", + "paperclipEnvironment", "paperclipContinuationSummary"]; + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("reconciliation-source-"); + db = createDb(temporary.connectionString); + companyId = randomUUID(); agentId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Recovery reads", issuePrefix: "RCR" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Recovery worker", adapterType: "process", status: "active" }); + }); + afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); }); + + async function fixture(patch: Partial = {}) { + const payload = randomBytes(64 * 1024).toString("base64"); + const contextSnapshot = { ...Object.fromEntries(launchKeys.map(key => [key, { prompt: payload }])), + issueId: randomUUID(), taskId: randomUUID(), executionWorkspaceId: randomUUID(), + executionRetryAccounting: { version: 1, failureRetries: 2, maxTurnContinuations: 3 }, + failureRetriesBeforeAiConnectionWait: 1, failureRetriesBeforeWorkspaceWait: 1, + conversationSessionGeneration: 7, connectionIntentResolution: "existing_connection", source: "connection_intent.resolved", + interactionKind: "connection_intent", interactionStatus: "expired", mutation: "interaction", interactionId: randomUUID(), + futureControl: { nonce: "retained", nested: null }, cancellation: { state: "pending" }, + }; + const [full] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed", runtimeMode: "legacy", + errorCode: "process_lost", contextSnapshot, resultJson: { futureOutput: { retained: true } }, + runnerProfileJson: { adapterDispatch: { adapterType: "process" }, futureControl: { retained: true } }, + ...patch, + }).returning(); + const [selected] = await db.select(legacyReconciliationRunFields).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, full!.id))); + return { full: full!, selected: selected! }; + } + + it.each(["native", "legacy"])("omits unused launch text and preserves current control fields (%s)", async runtimeMode => { + const profile = { nativeExecutionInput: { prompt: randomBytes(128 * 1024).toString("base64") }, futureControl: { retained: true } }; + const f = await fixture({ runtimeMode, ...(runtimeMode === "native" ? { runnerProfileJson: profile } : {}) }); + const expected = { ...f.full.contextSnapshot }; + for (const key of launchKeys) delete expected[key]; + expect(f.selected).toEqual({ ...f.full, contextSnapshot: expected, + runnerProfileJson: runtimeMode === "native" ? { futureControl: profile.futureControl } : f.full.runnerProfileJson }); + expect(Buffer.byteLength(JSON.stringify(f.full))).toBeGreaterThan(800_000); + expect(Buffer.byteLength(JSON.stringify(f.selected))).toBeLessThan(8192); + expect(executionFailureRetryCount(f.selected)).toBe(executionFailureRetryCount(f.full)); + expect(await legacyExecutionNeedsReconciliationWithEvidence(db, f.selected)) + .toBe(await legacyExecutionNeedsReconciliationWithEvidence(db, f.full)); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.full.id)))[0]).toEqual(f.full); + }); + + it.each([null, [], ["unexpected"], "scalar", 42].map(value => ({ value })))("preserves non-object JSON $value", async ({ value }) => { + for (const runtimeMode of ["native", "legacy"]) { + const f = await fixture({ runtimeMode, + contextSnapshot: value as typeof heartbeatRuns.$inferInsert.contextSnapshot, + runnerProfileJson: value as typeof heartbeatRuns.$inferInsert.runnerProfileJson, + }); + expect(f.selected).toEqual(f.full); + } + }); + + it.each([ + { status: "failed", errorCode: "setup_failed", resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } }, + { status: "cancelled", errorCode: "cancelled", resultJson: { executionCancellation: { state: "acknowledged" }, + executionRecovery: { kind: "interrupted", providerStopped: true, sessionPreserved: true, actionOutcomes: "settled" } } }, + { status: "cancelled", errorCode: "cancelled", resultJson: { executionCancellation: { state: "requested" }, + executionRecovery: { kind: "interrupted", providerStopped: false, sessionPreserved: true, actionOutcomes: "unknown" } } }, + { status: "cancelled", errorCode: "ai_connection_busy", resultJson: { executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false } } }, + { status: "cancelled", errorCode: "workspace_busy", resultJson: { executionRecovery: { kind: "workspace_wait", providerWorkStarted: false } } }, + { status: "failed", errorCode: "workspace_restore_failed", resultJson: { workspaceRestoreFailure: "restore_unsafe_archive" } }, + { status: "failed", errorCode: "process_lost", resultJson: { conversationContinuation: "continue_conversation_v1" } }, + { status: "cancelled", errorCode: "issue_waiting_for_response", resultJson: { stopReason: "issue_waiting_for_response", timeoutSource: "stale_queued_run_gate" } }, + ])("keeps reconciliation, cancellation, and retry decisions ($errorCode)", async patch => { + const f = await fixture(patch); + expect(f.selected.resultJson).toEqual(f.full.resultJson); + expect(executionFailureRetryCount(f.selected)).toBe(executionFailureRetryCount(f.full)); + expect(await legacyExecutionNeedsReconciliationWithEvidence(db, f.selected)) + .toBe(await legacyExecutionNeedsReconciliationWithEvidence(db, f.full)); + }); + + it("keeps full stored input and newer ownership when the projected source is terminalized", async () => { + const issueId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId, title: "Recover the stopped source", status: "in_progress", assigneeAgentId: agentId }); + const f = await fixture({ issueId, contextSnapshot: { issueId, paperclipTaskMarkdown: randomBytes(128 * 1024).toString("base64"), + executionRetryAccounting: { version: 1, failureRetries: 2, maxTurnContinuations: 0 } } }); + const [successor] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId, status: "running" }).returning(); + await db.update(issues).set({ executionRunId: successor!.id, checkoutRunId: successor!.id }).where(eq(issues.id, issueId)); + await terminalizeLegacyExecution({ db, run: f.selected, status: "failed", fromStatuses: ["failed"] }); + const [stored] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.full.id)); + expect(stored).toMatchObject({ contextSnapshot: f.full.contextSnapshot, runnerProfileJson: f.full.runnerProfileJson, resultJson: f.full.resultJson }); + expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]) + .toMatchObject({ executionRunId: successor!.id, checkoutRunId: successor!.id }); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))) + .toEqual([expect.objectContaining({ cause: "legacy_execution_requires_reconciliation", ownerType: "board", + evidence: expect.objectContaining({ runId: f.full.id, attempt: 3 }) })]); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId))).toEqual([]); + }); +}); diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index 5701571e864..488ef813306 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -106,6 +106,7 @@ import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { hasUnverifiedNativeWorkspaceFinalizationOwner } from "../native-runtime/native-workspace-finalization-recovery-state.js"; import { legacyExecutionNeedsReconciliationWithEvidence, + legacyReconciliationRunFields, terminalizeLegacyExecution, } from "../legacy-execution-recovery.js"; import { issueTreeControlService } from "../issue-tree-control.js"; @@ -2010,7 +2011,7 @@ export function recoveryService( }) { if (input.retryOfRunId) { const [predecessor] = await db - .select() + .select(legacyReconciliationRunFields) .from(heartbeatRuns) .where( and( @@ -4809,7 +4810,7 @@ export function recoveryService( ) ) { const [source] = await db - .select() + .select(legacyReconciliationRunFields) .from(heartbeatRuns) .where( and( From 1949e726f8515127a04314e3793c427ccfc33c1e Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 11:46:29 -0700 Subject: [PATCH 49/68] perf(recovery): read only legacy terminal control fields under lock (#17646) Select only runtime mode, status, complete result evidence and the delivery token from the locked legacy terminal-state read. Keep filters, row lock, guards, writes, full returned rows, source retention and recovery holds. The synthetic regression reduces a 699,454-byte read to under 4 KB while preserving full stored and returned data. All CI checks, 236 combined tests, workspace typecheck and server build pass. Apex 5/5 after correcting a documentation contradiction, with no unresolved comments. Co-Authored-By: Paperclip --- doc/DATABASE.md | 8 ++++- .../legacy-workspace-restore-recovery.test.ts | 29 ++++++++++++++++++- .../src/services/legacy-execution-recovery.ts | 4 ++- 3 files changed, 38 insertions(+), 3 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 961d532aef9..c7b525877fc 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -70,7 +70,8 @@ The two preliminary reconciliation reads omit named launch renderings and preparation payloads. Native profiles omit unused native execution input. They retain all other context, full results, legacy profiles, task identity, retry accounting, and unknown controls. Native runs still reach their native -recovery guard. Terminalization keeps its full locked source and stored input. +recovery guard. Terminalization locks its current control state and returns +full rows from its writes. Stored input remains intact. Execution-blocker reads select process, lifecycle, and task-binding fields rather than full execution rows. Context retains typed task and pre-dispatch evidence; @@ -96,6 +97,11 @@ payloads. It retains current coordinator checks, provider evidence, local-host identity checks, and every lease's termination receipt. The stop receipt and process-field update remain in the same transaction. +Legacy terminalization locks runtime mode, status, full result evidence, and the +status-delivery token without loading unused input, profile, or log bodies. +It keeps the existing lock, writes, full returned run, workspace-retention +receipts, and source-bound recovery holds. Stored payloads remain unchanged. + Conversation ownership checks select process identity and full recovery result evidence without execution prompts or profiles. Committed native reply repair selects its persisted result and identity bindings plus typed presentation flags. diff --git a/server/src/__tests__/legacy-workspace-restore-recovery.test.ts b/server/src/__tests__/legacy-workspace-restore-recovery.test.ts index 4629092e10e..84da47b84b7 100644 --- a/server/src/__tests__/legacy-workspace-restore-recovery.test.ts +++ b/server/src/__tests__/legacy-workspace-restore-recovery.test.ts @@ -1,4 +1,5 @@ -import { randomUUID } from "node:crypto"; +import { randomBytes, randomUUID } from "node:crypto"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { and, eq, sql } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { agents, companies, createDb, environmentLeases, environments, executionWorkspaces, heartbeatRuns, issueRecoveryActions, issues, projects } from "@paperclipai/db"; @@ -46,6 +47,32 @@ const support = externalTestDatabaseUrl ? { supported: true } : await getEmbedde } }; const readLease = async (id: string) => (await db.select().from(environmentLeases).where(eq(environmentLeases.id, id)))[0]; + it("locks only terminal control state while returning and retaining the complete run", async () => { + const f = await seed(); + const bulk = randomBytes(128 * 1024).toString("base64"); + const [saved] = await db.update(heartbeatRuns).set({ contextSnapshot: { ...f.run.contextSnapshot, savedInput: bulk }, + runnerProfileJson: { savedProfile: bulk }, stdoutExcerpt: bulk, stderrExcerpt: bulk, + }).where(eq(heartbeatRuns.id, f.runId)).returning(); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + let returned: Awaited>; + try { returned = await terminalizeLegacyExecution({ db, run: saved!, status: "failed", patch }); } + finally { capture.flush(); } + const groups = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1"); + const reads = groups.filter(group => group.table === "heartbeat_runs" && group.operation === "select"); + expect(reads.reduce((sum, group) => sum + Number(group.calls), 0)).toBe(1); + expect(reads.reduce((sum, group) => sum + Number(group.responseValueBytes), 0)).toBeLessThan(4096); + expect(returned).toMatchObject({ id: f.runId, status: "failed", contextSnapshot: saved!.contextSnapshot, + runnerProfileJson: saved!.runnerProfileJson, stdoutExcerpt: bulk, stderrExcerpt: bulk }); + expect(hasRequiredWorkspaceRecovery(returned!.resultJson)).toBe(true); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(returned); + expect(await readLease(f.lease.id)).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure" }); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId))) + .toEqual([expect.objectContaining({ evidence: expect.objectContaining({ runId: f.runId }), ownerType: "board" })]); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + }); + it("preserves pending and acknowledged status delivery on unpatched legacy recovery retries", async () => { await deliverExecutionStatuses(db, { publish: vi.fn() }); const f = await seed(); diff --git a/server/src/services/legacy-execution-recovery.ts b/server/src/services/legacy-execution-recovery.ts index 3561bff8768..5d5fe783b3f 100644 --- a/server/src/services/legacy-execution-recovery.ts +++ b/server/src/services/legacy-execution-recovery.ts @@ -118,7 +118,9 @@ export async function terminalizeLegacyExecution(input: { ) .for("update") : []; - const [current] = await tx.select().from(heartbeatRuns).where(and( + const [current] = await tx.select({ runtimeMode: heartbeatRuns.runtimeMode, status: heartbeatRuns.status, + resultJson: heartbeatRuns.resultJson, executionStatusDeliveryId: heartbeatRuns.executionStatusDeliveryId, + }).from(heartbeatRuns).where(and( eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), )).for("update"); if (!current || (current.runtimeMode !== "legacy" && !(input.recordRestoreFailureOnly && current.runtimeMode === "native")) || From 67aa87a93bb1bbc4559105497863f541f9fb4e9b Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 12:58:50 -0700 Subject: [PATCH 50/68] perf(recovery): skip unchanged retained workspace evidence writes (#17650) Guard the retained-evidence update with its existing JSONB comparison. Reuse the full first result when evidence is unchanged, while preserving new evidence delivery, source-bound retention, cleanup, and recovery holds. Verified with 199 recovery and retention tests, full workspace typechecking, the server build, green CI, and exact-head Apex5/5. Co-Authored-By: Paperclip --- doc/DATABASE.md | 5 ++- .../legacy-workspace-restore-recovery.test.ts | 35 +++++++++++++++++++ .../src/services/legacy-execution-recovery.ts | 9 ++--- 3 files changed, 44 insertions(+), 5 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index c7b525877fc..137b7fe1810 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -99,7 +99,7 @@ process-field update remain in the same transaction. Legacy terminalization locks runtime mode, status, full result evidence, and the status-delivery token without loading unused input, profile, or log bodies. -It keeps the existing lock, writes, full returned run, workspace-retention +It keeps the existing lock, full returned run, workspace-retention receipts, and source-bound recovery holds. Stored payloads remain unchanged. Conversation ownership checks select process identity and full recovery result @@ -281,6 +281,9 @@ Legacy recovery also retains the pending delivery when status is unchanged and no patch was supplied. Explicit patches, new terminal statuses, and newly retained workspace evidence still create a delivery. File retention, ownership checks, failed write conditions, lock cleanup, and recovery holds still apply. +The retention-evidence write returns another full run only when that evidence +changes. An unchanged retry keeps the full run returned by the first write and +still processes retained leases and recovery holds. Agent lists read the complete company roster once and use that same snapshot for termination filtering and organization-chain normalization. Detail and plugin runtime reads diff --git a/server/src/__tests__/legacy-workspace-restore-recovery.test.ts b/server/src/__tests__/legacy-workspace-restore-recovery.test.ts index 84da47b84b7..65e0e0cc147 100644 --- a/server/src/__tests__/legacy-workspace-restore-recovery.test.ts +++ b/server/src/__tests__/legacy-workspace-restore-recovery.test.ts @@ -73,6 +73,41 @@ const support = externalTestDatabaseUrl ? { supported: true } : await getEmbedde expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); }); + it.each(["legacy", "native"])("returns one complete run when retained workspace evidence is unchanged for %s", async (runtimeMode) => { + const f = await seed(); + const bulk = randomBytes(128 * 1024).toString("base64"); + const [saved] = await db.update(heartbeatRuns).set({ runtimeMode, status: "failed", finishedAt: new Date(), + contextSnapshot: { ...f.run.contextSnapshot, savedInput: bulk }, runnerProfileJson: { savedProfile: bulk }, + stdoutExcerpt: bulk, stderrExcerpt: bulk, + }).where(eq(heartbeatRuns.id, f.runId)).returning(); + const recordRestoreFailureOnly = runtimeMode === "native"; + const failed = (await terminalizeLegacyExecution({ db, run: saved!, status: "failed", recordRestoreFailureOnly, + workspaceRestoreSource: f.lease, + patch: { resultJson: { workspaceRestoreFailure: "restore_failed", futureResult: bulk } }, + }))!; + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + let returned: Awaited>; + try { returned = await terminalizeLegacyExecution({ db, run: failed, status: "failed", recordRestoreFailureOnly, + workspaceRestoreSource: f.lease, + ...(recordRestoreFailureOnly ? { patch: { resultJson: failed.resultJson } } : {}), + }); } + finally { capture.flush(); } + const writes = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && event.table === "heartbeat_runs" && event.operation === "update"); + expect(writes.reduce((sum, group) => sum + Number(group.rows), 0)).toBe(1); + expect(returned).toMatchObject({ id: f.runId, status: "failed", finishedAt: failed.finishedAt, + contextSnapshot: failed.contextSnapshot, runnerProfileJson: failed.runnerProfileJson, resultJson: failed.resultJson, + stdoutExcerpt: bulk, stderrExcerpt: bulk }); + if (!recordRestoreFailureOnly) expect(returned!.executionStatusDeliveryId).toBe(failed.executionStatusDeliveryId); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(returned); + expect(await readLease(f.lease.id)).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure" }); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId))) + .toEqual([expect.objectContaining({ evidence: expect.objectContaining({ runId: f.runId }), ownerType: "board" })]); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + }); + it("preserves pending and acknowledged status delivery on unpatched legacy recovery retries", async () => { await deliverExecutionStatuses(db, { publish: vi.fn() }); const f = await seed(); diff --git a/server/src/services/legacy-execution-recovery.ts b/server/src/services/legacy-execution-recovery.ts index 5d5fe783b3f..9a3041c88d8 100644 --- a/server/src/services/legacy-execution-recovery.ts +++ b/server/src/services/legacy-execution-recovery.ts @@ -156,11 +156,12 @@ export async function terminalizeLegacyExecution(input: { ...updated.resultJson, workspaceRestoreRecovery: { schema: LEGACY_WORKSPACE_RECOVERY_SCHEMA, leaseIds: [...retainedLeaseIds].sort() }, }; - [updated] = await tx.update(heartbeatRuns).set({ + const [retained] = await tx.update(heartbeatRuns).set({ resultJson: retainedResult, - executionStatusDeliveryId: sql`case when ${heartbeatRuns.resultJson} is distinct from ${JSON.stringify(retainedResult)}::jsonb - then ${randomUUID()}::uuid else ${heartbeatRuns.executionStatusDeliveryId} end`, - }).where(and(eq(heartbeatRuns.id, updated.id), eq(heartbeatRuns.companyId, updated.companyId))).returning(); + executionStatusDeliveryId: randomUUID(), + }).where(and(eq(heartbeatRuns.id, updated.id), eq(heartbeatRuns.companyId, updated.companyId), + sql`${heartbeatRuns.resultJson} is distinct from ${JSON.stringify(retainedResult)}::jsonb`)).returning(); + if (retained) updated = retained; } const needsRecovery = input.recordRestoreFailureOnly ? retainedLeaseIds.length > 0 || hasRequiredWorkspaceRecovery(updated.resultJson) From f0b597172b9ba434e7a490ba4778f63ddc8f1fb7 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 13:37:11 -0700 Subject: [PATCH 51/68] perf(skills): verify bundled releases with current inventory digests (#17652) Use current scoped database inventory fingerprints and the current on-disk expectation before reusing bounded snapshot verification. Preserve full validation, seeding and access checks when content, expectation or row existence changes. Verified177 skill and authorization tests, workspace typechecking, server build, all green CI and exact-head Apex5/5. Prior staging release measured and saved Legacy data fingerprints preserved. Co-Authored-By: Paperclip --- doc/DATABASE.md | 7 ++ .../__tests__/company-skills-service.test.ts | 68 ++++++++++++ .../bundled-skill-release-validation.test.ts | 103 ++++++++++++++++++ .../bundled-skill-release-validation.ts | 45 ++++++++ server/src/services/company-skills.ts | 41 +++---- 5 files changed, 240 insertions(+), 24 deletions(-) create mode 100644 server/src/services/bundled-skill-release-validation.test.ts create mode 100644 server/src/services/bundled-skill-release-validation.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 137b7fe1810..966595b7536 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -185,6 +185,13 @@ queries after any older request. Follow-up reads belong to the current mounted scope and stop when it changes or unmounts. The first open and reconnect still reconcile missed events. +Bundled skill release checks read the current stored version ID and a SHA-256 +fingerprint of its file inventory. A per-database memo keeps at most 64 verified +snapshots. It reuses a verification only when the current database fingerprint +and current on-disk inventory hash both match. A changed or missing snapshot +still follows full validation or seeding. Company access and local source +reconciliation still run for each request; authorization is not cached. + Connection-assignment checks select profile bindings for the current company or agent and read rules that can match the selected connection's catalog rows. All five selector kinds, null application matching, exclusions, and profile diff --git a/server/src/__tests__/company-skills-service.test.ts b/server/src/__tests__/company-skills-service.test.ts index 7217eb4f2be..c1c9a448232 100644 --- a/server/src/__tests__/company-skills-service.test.ts +++ b/server/src/__tests__/company-skills-service.test.ts @@ -16,6 +16,7 @@ import { projectWorkspaces, } from "@paperclipai/db"; import { parseFrontmatterMarkdown } from "@paperclipai/shared"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, @@ -722,6 +723,73 @@ describeEmbeddedPostgres("companySkillService.list", () => { expect(refreshed.filter((skill) => skill.key === key)).toHaveLength(1); }); + it("validates warm bundled releases without returning their stored file bodies", async () => { + const companyId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Release read checks", issuePrefix: "RRC" }); + await svc.list(companyId); + const expected = await svc.list(companyId); + const before = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId)) + .orderBy(companySkillVersions.id); + expect(before.length).toBeGreaterThan(0); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { companySkillVersions }, capture); + try { expect(await companySkillService(db).list(companyId)).toEqual(expected); } + finally { capture.flush(); } + const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && event.table === "company_skill_versions" && event.operation === "select"); + expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThan(0); + expect(reads.reduce((sum, event) => sum + Number(event.responseValueBytes), 0)).toBeLessThan(4096); + expect(await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId)) + .orderBy(companySkillVersions.id)).toEqual(before); + }); + + it.each(["agent", "board"] as const)("checks %s company access before using a warm release validation", async type => { + const companyId = randomUUID(), otherCompanyId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Release access checks", issuePrefix: "RAC" }); + await svc.list(companyId); + await svc.list(companyId); + const actor = type === "agent" + ? { type, companyId: otherCompanyId, agentId: randomUUID() } + : { type, companyIds: [otherCompanyId] }; + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { companySkillVersions }, capture); + try { expect(await companySkillService(db, { actor }).list(companyId)).toEqual([]); } + finally { capture.flush(); } + expect(events.find(event => event.event === "paperclip.db_response_capture.summary.v1")?.completedQueries).toBe(0); + }); + + it("restores a deleted release after warm validation", async () => { + const companyId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Release deletion checks", issuePrefix: "RDC" }); + await svc.list(companyId); + await svc.list(companyId); + const before = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId)); + const original = before[0]!; + await db.delete(companySkillVersions).where(eq(companySkillVersions.id, original.id)); + await companySkillService(db).list(companyId); + const after = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId)); + expect(after).toHaveLength(before.length); + expect(after.find(version => version.releaseId === original.releaseId)).toMatchObject({ + companyId, companySkillId: original.companySkillId, fileInventory: original.fileInventory, + }); + expect(after.some(version => version.id === original.id)).toBe(false); + }); + + it("rejects a changed bundled snapshot after warm validation without overwriting it", async () => { + const companyId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Release integrity checks", issuePrefix: "RIC" }); + await svc.list(companyId); + await svc.list(companyId); + const [version] = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId)); + const changed = [{ path: "SKILL.md", kind: "skill" as const, content: "Changed stored snapshot" }]; + await db.update(companySkillVersions).set({ fileInventory: changed }).where(eq(companySkillVersions.id, version!.id)); + await expect(companySkillService(db).list(companyId)).rejects.toThrow("does not match its seeded snapshot"); + expect((await db.select().from(companySkillVersions).where(eq(companySkillVersions.id, version!.id)))[0]!.fileInventory) + .toEqual(changed); + }); + it("seeds bundled skill releases idempotently and materializes the frozen champion snapshot", async () => { const companyId = randomUUID(); await db.insert(companies).values({ diff --git a/server/src/services/bundled-skill-release-validation.test.ts b/server/src/services/bundled-skill-release-validation.test.ts new file mode 100644 index 00000000000..4c4c1c42857 --- /dev/null +++ b/server/src/services/bundled-skill-release-validation.test.ts @@ -0,0 +1,103 @@ +import { createHash, randomUUID } from "node:crypto"; +import { and, eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { companies, companySkills, companySkillVersions, createDb } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { verifyExistingBundledSkillRelease } from "./bundled-skill-release-validation.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const hashInventory = (files: Array<{ path: string; content: string }>) => createHash("sha256") + .update(JSON.stringify(files.map(file => [file.path, file.content]))).digest("hex"); + +(support.supported ? describe : describe.skip)("bundled release validation with current database fingerprints", () => { + let db: ReturnType; + let temporary: Awaited>; + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("bundled-release-validation-"); + db = createDb(temporary.connectionString); + }, 30_000); + afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); }); + + async function seed(content = "saved release content") { + const companyId = randomUUID(), skillId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Release validation", issuePrefix: companyId.slice(0, 8) }); + await db.insert(companySkills).values({ id: skillId, companyId, key: "fixture/release", slug: "release", + name: "Release", markdown: "# Release", sourceType: "local_path" }); + const files = [{ path: "SKILL.md", kind: "skill" as const, content }]; + const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skillId, + revisionNumber: 1, releaseId: "release-1", fileInventory: files }).returning(); + const expectedInventoryHash = hashInventory(files); + const validateInventory = vi.fn((row: typeof companySkillVersions.$inferSelect) => { + if (hashInventory(row.fileInventory) !== expectedInventoryHash) throw new Error("snapshot mismatch"); + }); + return { version: version!, input: { db, companyId, skillId, releaseId: "release-1", expectedInventoryHash, validateInventory } }; + } + + it("reuses proof only within current company, skill and release bindings", async () => { + const f = await seed(); + expect(await verifyExistingBundledSkillRelease(f.input)).toBe(true); + expect(await verifyExistingBundledSkillRelease(f.input)).toBe(true); + expect(f.input.validateInventory).toHaveBeenCalledTimes(1); + expect(await verifyExistingBundledSkillRelease({ ...f.input, companyId: randomUUID() })).toBe(false); + expect(await verifyExistingBundledSkillRelease({ ...f.input, skillId: randomUUID() })).toBe(false); + expect(await verifyExistingBundledSkillRelease({ ...f.input, releaseId: "other-release" })).toBe(false); + expect(f.input.validateInventory).toHaveBeenCalledTimes(1); + expect(await verifyExistingBundledSkillRelease(f.input)).toBe(true); + }); + + it("revalidates when the expected on-disk inventory changes without poisoning valid proof", async () => { + const f = await seed(); + await verifyExistingBundledSkillRelease(f.input); + const changedHash = hashInventory([{ path: "SKILL.md", content: "new on-disk content" }]); + const validateChanged = vi.fn((row: typeof companySkillVersions.$inferSelect) => { + if (hashInventory(row.fileInventory) !== changedHash) throw new Error("snapshot mismatch"); + }); + await expect(verifyExistingBundledSkillRelease({ ...f.input, expectedInventoryHash: changedHash, + validateInventory: validateChanged })).rejects.toThrow("snapshot mismatch"); + expect(validateChanged).toHaveBeenCalledOnce(); + expect(await verifyExistingBundledSkillRelease(f.input)).toBe(true); + expect(f.input.validateInventory).toHaveBeenCalledOnce(); + }); + + it("rechecks current content after a change overlaps an earlier validation", async () => { + const f = await seed(); + let entered!: () => void, release!: () => void; + const validating = new Promise(resolve => { entered = resolve; }); + const held = new Promise(resolve => { release = resolve; }); + const first = verifyExistingBundledSkillRelease({ ...f.input, validateInventory: async row => { + f.input.validateInventory(row); + entered(); + await held; + } }); + await validating; + try { + await db.update(companySkillVersions).set({ fileInventory: [{ path: "SKILL.md", kind: "skill", content: "changed" }] }) + .where(and(eq(companySkillVersions.companyId, f.input.companyId), eq(companySkillVersions.id, f.version.id))); + } finally { release(); } + expect(await first).toBe(true); + await expect(verifyExistingBundledSkillRelease(f.input)).rejects.toThrow("snapshot mismatch"); + expect(f.input.validateInventory).toHaveBeenCalledTimes(2); + }); + + it("does not accept a deleted row from earlier proof", async () => { + const f = await seed(); + await verifyExistingBundledSkillRelease(f.input); + await db.delete(companySkillVersions).where(eq(companySkillVersions.id, f.version.id)); + expect(await verifyExistingBundledSkillRelease(f.input)).toBe(false); + expect(f.input.validateInventory).toHaveBeenCalledOnce(); + }); + + it("bounds retained proof and safely revalidates an evicted release", async () => { + const f = await seed(); + await verifyExistingBundledSkillRelease(f.input); + for (let revisionNumber = 2; revisionNumber <= 66; revisionNumber++) { + const releaseId = `release-${revisionNumber}`; + await db.insert(companySkillVersions).values({ companyId: f.input.companyId, companySkillId: f.input.skillId, + revisionNumber, releaseId, fileInventory: f.version.fileInventory }); + expect(await verifyExistingBundledSkillRelease({ ...f.input, releaseId })).toBe(true); + } + const before = f.input.validateInventory.mock.calls.length; + expect(await verifyExistingBundledSkillRelease(f.input)).toBe(true); + expect(f.input.validateInventory).toHaveBeenCalledTimes(before + 1); + }); +}); diff --git a/server/src/services/bundled-skill-release-validation.ts b/server/src/services/bundled-skill-release-validation.ts new file mode 100644 index 00000000000..b29f8ee28a6 --- /dev/null +++ b/server/src/services/bundled-skill-release-validation.ts @@ -0,0 +1,45 @@ +import { companySkillVersions, type Db } from "@paperclipai/db"; +import { and, eq, getTableColumns, sql } from "drizzle-orm"; + +type Version = typeof companySkillVersions.$inferSelect; +type Verification = { inventorySha256: string; expectedInventoryHash: string }; +const verifiedInventories = new WeakMap>(); +const inventorySha256 = sql`encode(sha256(convert_to(${companySkillVersions.fileInventory}::text, 'UTF8')), 'hex')`; + +export async function verifyExistingBundledSkillRelease(input: { + db: Db; + companyId: string; + skillId: string; + releaseId: string; + expectedInventoryHash: string; + validateInventory: (version: Version) => void | Promise; +}): Promise { + const scope = and(eq(companySkillVersions.companyId, input.companyId), + eq(companySkillVersions.companySkillId, input.skillId), eq(companySkillVersions.releaseId, input.releaseId)); + const [current] = await input.db.select({ id: companySkillVersions.id, inventorySha256 }) + .from(companySkillVersions).where(scope).limit(1); + if (!current) return false; + let verified = verifiedInventories.get(input.db); + if (!verified) { + verified = new Map(); + verifiedInventories.set(input.db, verified); + } + const previous = verified.get(current.id); + if (previous?.inventorySha256 === current.inventorySha256 + && previous.expectedInventoryHash === input.expectedInventoryHash) { + verified.delete(current.id); + verified.set(current.id, previous); + return true; + } + const [version] = await input.db.select({ ...getTableColumns(companySkillVersions), inventorySha256 }) + .from(companySkillVersions).where(scope).limit(1); + if (!version) return false; + await input.validateInventory(version); + verified.delete(version.id); + verified.set(version.id, { inventorySha256: version.inventorySha256, expectedInventoryHash: input.expectedInventoryHash }); + while (verified.size > 64) { + const oldest = verified.keys().next().value; + if (oldest !== undefined) verified.delete(oldest); + } + return true; +} diff --git a/server/src/services/company-skills.ts b/server/src/services/company-skills.ts index ac114ee692f..f29f4ece222 100644 --- a/server/src/services/company-skills.ts +++ b/server/src/services/company-skills.ts @@ -1,4 +1,5 @@ import { ensureSkillCacheDirectory, touchSkillVersionCache, withSkillVersionCacheLock } from "./skill-version-cache.js"; +import { verifyExistingBundledSkillRelease } from "./bundled-skill-release-validation.js"; import { retainProjectSkillAccess, skillAccessService } from "./skill-access.js"; import type { AuthorizationActor } from "./authorization.js"; import { updateAgentConfigurationInTransaction } from "./agent-configuration-transaction.js"; @@ -3094,30 +3095,22 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const fileInventory = serializeVersionFileInventory( await collectVersionFileInventoryFromDirectory(release.releaseDir), ); - const existing = await db - .select() - .from(companySkillVersions) - .where(and( - eq(companySkillVersions.companyId, companyId), - eq(companySkillVersions.companySkillId, paperclipSkill.id), - eq(companySkillVersions.releaseId, release.id), - )) - .then((rows) => rows[0] ?? null); - if (existing) { - const existingInventory = toCompanySkillVersion(existing).fileInventory; - const existingHash = buildInventoryContentHash(existingInventory.map((entry) => ({ - path: entry.path, - sha256: sha256Buffer(entry.content), - }))); - const registryHash = buildInventoryContentHash(fileInventory.map((entry) => ({ - path: entry.path, - sha256: sha256Buffer(entry.content), - }))); - if (existingHash !== registryHash) { - throw new Error(`Bundled skill release ${release.id} does not match its seeded snapshot.`); - } - continue; - } + const registryHash = buildInventoryContentHash(fileInventory.map((entry) => ({ + path: entry.path, + sha256: sha256Buffer(entry.content), + }))); + if (await verifyExistingBundledSkillRelease({ db, companyId, skillId: paperclipSkill.id, releaseId: release.id, + expectedInventoryHash: registryHash, validateInventory: existing => { + const existingInventory = toCompanySkillVersion(existing).fileInventory; + const existingHash = buildInventoryContentHash(existingInventory.map((entry) => ({ + path: entry.path, + sha256: sha256Buffer(entry.content), + }))); + if (existingHash !== registryHash) { + throw new Error(`Bundled skill release ${release.id} does not match its seeded snapshot.`); + } + }, + })) continue; const releasedAt = new Date(release.releasedAt); if (Number.isNaN(releasedAt.getTime())) { throw new Error(`Invalid bundled skill release date: ${release.releasedAt}`); From 2a45fd95df352b1e9d20ab366181bc715a06fe35 Mon Sep 17 00:00:00 2001 From: Naeem Akhtar <32741588+naeemakhtar110@users.noreply.github.com> Date: Mon, 26 Oct 2026 00:15:03 +0300 Subject: [PATCH 52/68] Reduce newer-run history reads for attention (#15697) Replace broad newer-run responses with bounded company-scoped existence probes. Read keyless history once per unique agent and preserve existing matching and millisecond rules. Keep all stored run data. Verify with 31 real PostgreSQL attention tests, workspace typecheck, server build, response-budget counterfactuals, full green CI, and Apex 5/5 review. Co-Authored-By: Paperclip --- doc/DATABASE.md | 9 + .../src/__tests__/attention-service.test.ts | 174 +++++++++++++++++- server/src/services/attention.ts | 102 ++++++---- 3 files changed, 249 insertions(+), 36 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 966595b7536..ed99988523b 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -49,6 +49,15 @@ and leaves all existing cases passive. Source identities are not grants. ### Task query performance +The attention feed checks for newer runs with company-scoped existence probes. +The probes match the failed run's agent and issue or task context. They preserve +the existing millisecond comparison and empty-context rules. Each batch holds +at most 1,000 failed runs. Retained run history stays in the database; this check +returns only the identifiers of failures that have a newer matching run. +For failures without a task key, it reads the latest keyless timestamp once per +agent. Unique agents are batched separately, so repeated failures do not repeat +the same history scan. Both paths retain the current millisecond comparison. + The issue-lock adapter's release read omits `result_json.stdout` and `result_json.stderr` before transferring a run to the controller. Native context also omits saved launch text, execution-continuation bodies, workspace and diff --git a/server/src/__tests__/attention-service.test.ts b/server/src/__tests__/attention-service.test.ts index 26934470fbd..bf4879957cd 100644 --- a/server/src/__tests__/attention-service.test.ts +++ b/server/src/__tests__/attention-service.test.ts @@ -2,7 +2,8 @@ import { createAgentLifecycle } from "../services/agent-lifecycle.js"; import { randomUUID } from "node:crypto"; import express from "express"; import request from "supertest"; -import { eq } from "drizzle-orm"; +import { eq, sql } from "drizzle-orm"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; import { activityLog, @@ -1034,6 +1035,177 @@ describeEmbeddedPostgres("attention service", () => { expect(feed.items.filter((item) => item.sourceKind === "failed_run")).toEqual([]); }); + it("hides a failed run only after a newer run of the same agent on the same issue or task", async () => { + const { companyId, workerId, reviewerId } = await seedCompany("ATK"); + const taskIssueId = await insertIssue({ companyId, identifier: "ATK-1", title: "Task-keyed", status: "in_progress" }); + const issueKeyedId = await insertIssue({ companyId, identifier: "ATK-2", title: "Issue-keyed", status: "in_progress" }); + const otherIssueId = await insertIssue({ companyId, identifier: "ATK-3", title: "Other work", status: "in_progress" }); + const failedAt = new Date("2026-07-09T12:00:00.000Z"); + const laterAt = new Date("2026-07-09T12:01:00.000Z"); + const taskKeyedFailureId = randomUUID(); + const issueKeyedFailureId = randomUUID(); + const unkeyedHiddenFailureId = randomUUID(); + const unkeyedKeptFailureId = randomUUID(); + const failureIds = [taskKeyedFailureId, issueKeyedFailureId, unkeyedHiddenFailureId, unkeyedKeptFailureId]; + const sameMillisecondRunId = randomUUID(); + const run = (id: string, agentId: string, status: string, contextSnapshot: Record, at: Date) => ({ + id, companyId, agentId, invocationSource: "automation", status, contextSnapshot, + createdAt: at, updatedAt: at, finishedAt: at, ...(status === "failed" ? { error: "adapter failed" } : {}), + }); + + await db.insert(heartbeatRuns).values([ + run(taskKeyedFailureId, workerId, "failed", { taskId: taskIssueId }, failedAt), + run(issueKeyedFailureId, workerId, "failed", { issueId: issueKeyedId }, failedAt), + run(unkeyedHiddenFailureId, workerId, "failed", {}, failedAt), + run(unkeyedKeptFailureId, reviewerId, "failed", {}, failedAt), + run(randomUUID(), workerId, "succeeded", { taskId: taskIssueId }, laterAt), + run(randomUUID(), workerId, "succeeded", { wakeReason: "timer" }, laterAt), + run(randomUUID(), reviewerId, "succeeded", { issueId: issueKeyedId }, laterAt), + run(randomUUID(), reviewerId, "succeeded", { issueId: otherIssueId }, laterAt), + run(randomUUID(), workerId, "succeeded", { issueId: otherIssueId }, laterAt), + run(randomUUID(), workerId, "succeeded", { issueId: otherIssueId, taskId: issueKeyedId }, laterAt), + run(randomUUID(), workerId, "succeeded", { issueId: "", taskId: issueKeyedId }, laterAt), + run(sameMillisecondRunId, workerId, "succeeded", { issueId: issueKeyedId }, failedAt), + run(randomUUID(), workerId, "succeeded", { issueId: issueKeyedId }, new Date("2026-07-09T11:59:00.000Z")), + ]); + await db.execute(sql`update heartbeat_runs set created_at = created_at + interval '400 microseconds' + where id in (${sql.join(failureIds.map((id) => sql`${id}::uuid`), sql`, `)})`); + await db.execute(sql`update heartbeat_runs set created_at = created_at + interval '600 microseconds' + where id = ${sameMillisecondRunId}::uuid`); + await db.insert(heartbeatRunEvents).values( + failureIds.map((runId) => ({ + companyId, + runId, + agentId: runId === unkeyedKeptFailureId ? reviewerId : workerId, + seq: 1, + eventType: "lifecycle", + message: "Bounded retry exhausted after 4 scheduled attempts; no further automatic retry will be queued", + createdAt: new Date("2026-07-09T12:00:01.000Z"), + })), + ); + + const feed = await attentionService(db).list(companyId, { userId: "board-user" }); + + expect(feed.items.filter((item) => item.sourceKind === "failed_run").map((item) => item.subject.id).sort()) + .toEqual([issueKeyedFailureId, unkeyedKeptFailureId].sort()); + }); + + it("keeps newer-run response size bounded despite unrelated retained history", async () => { + const { companyId, workerId } = await seedCompany("ATW"); + const issueId = await insertIssue({ companyId, identifier: "ATW-1", title: "Exhausted task", status: "in_progress" }); + const otherIssueId = await insertIssue({ companyId, identifier: "ATW-2", title: "Other task", status: "done" }); + const failedId = randomUUID(), failedAt = new Date("2026-07-09T12:00:00.000Z"); + await db.insert(heartbeatRuns).values({ + id: failedId, companyId, agentId: workerId, status: "failed", contextSnapshot: { issueId }, + createdAt: failedAt, updatedAt: failedAt, finishedAt: failedAt, + }); + await db.insert(heartbeatRunEvents).values({ + companyId, agentId: workerId, runId: failedId, seq: 1, eventType: "lifecycle", + message: "Bounded retry exhausted after scheduled attempts", + }); + for (let batch = 0; batch < 4; batch += 1) { + await db.insert(heartbeatRuns).values(Array.from({ length: 500 }, (_, index) => ({ + id: randomUUID(), companyId, agentId: workerId, status: "succeeded", + contextSnapshot: { issueId: otherIssueId, prompt: "x".repeat(8_000) }, + createdAt: new Date(failedAt.getTime() + batch * 500 + index + 1), + updatedAt: failedAt, finishedAt: failedAt, + }))); + } + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { + const feed = await attentionService(db).list(companyId, { userId: "board-user" }); + expect(feed.items.filter(item => item.sourceKind === "failed_run").map(item => item.subject.id)).toEqual([failedId]); + } finally { capture.flush(); } + const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && event.table === "heartbeat_runs" && (event.operation === "select" || event.operation === "execute")); + expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThan(0); + expect(reads.reduce((sum, event) => sum + Number(event.rows), 0)).toBeLessThan(64); + expect(reads.reduce((sum, event) => sum + Number(event.responseValueBytes), 0)).toBeLessThan(4096); + expect(reads.filter(event => event.operation === "execute") + .reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThan(0); + const [remaining] = await db.select({ count: sql`count(*)::int` }).from(heartbeatRuns) + .where(eq(heartbeatRuns.companyId, companyId)); + expect(remaining?.count).toBe(2001); + }); + + it("reads keyless history once per agent despite thousands of failures", async () => { + const { companyId, workerId, reviewerId } = await seedCompany("ATB"); + const failedAt = new Date("2026-07-09T12:00:00.000Z"); + const laterAt = new Date("2026-07-09T12:01:00.000Z"); + const hiddenIds = Array.from({ length: 16_390 }, () => randomUUID()); + const keptIds = Array.from({ length: 10 }, () => randomUUID()); + const failure = (id: string, agentId: string) => ({ + id, companyId, agentId, invocationSource: "automation", status: "failed", error: "adapter failed", + contextSnapshot: {}, createdAt: failedAt, updatedAt: failedAt, finishedAt: failedAt, + }); + const failures = hiddenIds.map((id) => failure(id, workerId)); + keptIds.forEach((id, index) => failures.splice(index * 1_700, 0, failure(id, reviewerId))); + for (let start = 0; start < failures.length; start += 1_000) { + const chunk = failures.slice(start, start + 1_000); + await db.insert(heartbeatRuns).values(chunk); + await db.insert(heartbeatRunEvents).values(chunk.map((run) => ({ + companyId, runId: run.id, agentId: run.agentId, seq: 1, eventType: "lifecycle", + message: "Bounded retry exhausted after 4 scheduled attempts; no further automatic retry will be queued", + createdAt: new Date("2026-07-09T12:00:01.000Z"), + }))); + } + await db.insert(heartbeatRuns).values({ + id: randomUUID(), companyId, agentId: workerId, invocationSource: "automation", status: "succeeded", + contextSnapshot: { wakeReason: "timer" }, createdAt: laterAt, updatedAt: laterAt, finishedAt: laterAt, + }); + + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { + const feed = await attentionService(db).list(companyId, { + includeDismissed: true, all: true, allowUnscopedAll: true, + }); + expect(feed.items.filter((item) => item.sourceKind === "failed_run").map((item) => item.subject.id).sort()) + .toEqual([...keptIds].sort()); + } finally { capture.flush(); } + const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && event.table === "heartbeat_runs" && (event.operation === "select" || event.operation === "execute")); + expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThan(0); + expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeLessThan(4); + }, 180_000); + + it("checks more failed runs for a newer run than one statement can bind", async () => { + const { companyId, workerId, reviewerId } = await seedCompany("ATQ"); + const issueId = await insertIssue({ companyId, identifier: "ATQ-1", title: "Keyed retained failures", status: "in_progress" }); + const failedAt = new Date("2026-07-09T12:00:00.000Z"); + const laterAt = new Date("2026-07-09T12:01:00.000Z"); + const hiddenIds = Array.from({ length: 16_390 }, () => randomUUID()); + const keptIds = Array.from({ length: 10 }, () => randomUUID()); + const failure = (id: string, agentId: string) => ({ + id, companyId, agentId, invocationSource: "automation", status: "failed", error: "adapter failed", + contextSnapshot: { issueId }, createdAt: failedAt, updatedAt: failedAt, finishedAt: failedAt, + }); + const failures = hiddenIds.map((id) => failure(id, workerId)); + keptIds.forEach((id, index) => failures.splice(index * 1_700, 0, failure(id, reviewerId))); + for (let start = 0; start < failures.length; start += 1_000) { + const chunk = failures.slice(start, start + 1_000); + await db.insert(heartbeatRuns).values(chunk); + await db.insert(heartbeatRunEvents).values(chunk.map((run) => ({ + companyId, runId: run.id, agentId: run.agentId, seq: 1, eventType: "lifecycle", + message: "Bounded retry exhausted after 4 scheduled attempts; no further automatic retry will be queued", + createdAt: new Date("2026-07-09T12:00:01.000Z"), + }))); + } + await db.insert(heartbeatRuns).values({ + id: randomUUID(), companyId, agentId: workerId, invocationSource: "automation", status: "succeeded", + contextSnapshot: { issueId }, createdAt: laterAt, updatedAt: laterAt, finishedAt: laterAt, + }); + + const feed = await attentionService(db).list(companyId, { + includeDismissed: true, all: true, allowUnscopedAll: true, + }); + expect(feed.items.filter(item => item.sourceKind === "failed_run").map(item => item.subject.id).sort()) + .toEqual([...keptIds].sort()); + }, 180_000); + it("enriches interaction details with project, workspace, plan metadata, and images", async () => { const { companyId, workerId } = await seedCompany("ATE"); const projectId = randomUUID(); diff --git a/server/src/services/attention.ts b/server/src/services/attention.ts index dd46de7d91e..90ed24f135e 100644 --- a/server/src/services/attention.ts +++ b/server/src/services/attention.ts @@ -1099,6 +1099,65 @@ function readRunIssueId(contextSnapshot: Record | null) { return typeof issueId === "string" && issueId.length > 0 ? issueId : null; } +type FailedRunProbe = { id: string; agentId: string; issueId: string | null; createdAt: Date }; + +const FAILED_RUN_PROBE_BATCH_SIZE = 1000; + +async function listFailedRunsWithNewerRun(db: Db, companyId: string, failedRuns: FailedRunProbe[]) { + const withNewerRun = new Set(); + const keyed = failedRuns.filter(run => run.issueId !== null); + for (let start = 0; start < keyed.length; start += FAILED_RUN_PROBE_BATCH_SIZE) { + const batch = keyed.slice(start, start + FAILED_RUN_PROBE_BATCH_SIZE); + for (const id of await listFailedRunBatchWithNewerRun(db, companyId, batch)) withNewerRun.add(id); + } + const unkeyed = failedRuns.filter(run => run.issueId === null); + if (unkeyed.length === 0) return withNewerRun; + const agentIds = [...new Set(unkeyed.map(run => run.agentId))]; + const oldestCreatedAt = unkeyed.reduce((oldest, run) => run.createdAt < oldest ? run.createdAt : oldest, + unkeyed[0]!.createdAt); + const latestByAgent = new Map(); + for (let start = 0; start < agentIds.length; start += FAILED_RUN_PROBE_BATCH_SIZE) { + const rows = await db.select({ + agentId: heartbeatRuns.agentId, + createdAt: sql`max(${heartbeatRuns.createdAt})`.mapWith(heartbeatRuns.createdAt), + }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, companyId), + inArray(heartbeatRuns.agentId, agentIds.slice(start, start + FAILED_RUN_PROBE_BATCH_SIZE)), + gt(heartbeatRuns.createdAt, oldestCreatedAt), + sql`coalesce(${heartbeatRuns.contextSnapshot} ->> 'issueId', ${heartbeatRuns.contextSnapshot} ->> 'taskId', '') = ''`, + )).groupBy(heartbeatRuns.agentId); + for (const row of rows) latestByAgent.set(row.agentId, row.createdAt); + } + for (const run of unkeyed) { + if ((latestByAgent.get(run.agentId)?.getTime() ?? 0) > run.createdAt.getTime()) withNewerRun.add(run.id); + } + return withNewerRun; +} + +async function listFailedRunBatchWithNewerRun(db: Db, companyId: string, failedRuns: FailedRunProbe[]) { + if (failedRuns.length === 0) return []; + const failed = sql.join( + failedRuns.map((run) => sql`(${run.id}::uuid, ${run.agentId}::uuid, ${run.issueId}::text, ${new Date(run.createdAt.getTime() + 1).toISOString()}::timestamptz)`), + sql`, `, + ); + const runIssueId = sql`(${heartbeatRuns.contextSnapshot} ->> 'issueId')`; + const runTaskId = sql`(${heartbeatRuns.contextSnapshot} ->> 'taskId')`; + const newerRunExists = (match: SQL) => sql`exists ( + select 1 from ${heartbeatRuns} + where ${heartbeatRuns.companyId} = ${companyId} + and ${heartbeatRuns.agentId} = failed.agent_id + and ${heartbeatRuns.createdAt} >= failed.newer_from + and ${match} + )`; + const rows = await db.execute(sql` + select failed.id + from (values ${failed}) as failed (id, agent_id, issue_id, newer_from) + where ${newerRunExists(sql`${runIssueId} = failed.issue_id`)} + or ${newerRunExists(sql`${runTaskId} = failed.issue_id and ${runIssueId} is null`)} + `) as unknown as Array<{ id: string }>; + return rows.map((row) => row.id); +} + export function attentionService(db: Db, serviceOptions: AttentionServiceOptions = {}) { const openDecisionLimit = Math.min( Math.max(Math.trunc(serviceOptions.openDecisionLimit ?? OPEN_DECISION_DEFAULT_LIMIT), 1), @@ -1717,12 +1776,7 @@ export function attentionService(db: Db, serviceOptions: AttentionServiceOptions const failedRows = await listAttentionExhaustedRuns(db, companyId); const failedIssueIds = failedRows.map((row) => readRunIssueId(row.contextSnapshot)); - const failedAgentIds = [...new Set(failedRows.map((row) => row.agentId))]; - const oldestFailedRunCreatedAt = failedRows.reduce((oldest, row) => { - if (!oldest || row.createdAt < oldest) return row.createdAt; - return oldest; - }, null); - const [failedIssueMap, failedImageMap, newerRuns] = await Promise.all([ + const [failedIssueMap, failedImageMap, failedRunsWithNewerRun] = await Promise.all([ issueSummaryMap( db, companyId, @@ -1730,38 +1784,16 @@ export function attentionService(db: Db, serviceOptions: AttentionServiceOptions readCondition, ), issueImageMap(db, companyId, failedIssueIds, readCondition), - oldestFailedRunCreatedAt && failedAgentIds.length > 0 - ? db - .select({ - agentId: heartbeatRuns.agentId, - createdAt: heartbeatRuns.createdAt, - // Project just the ids readRunIssueId needs; pulling the whole - // context_snapshot detoasts megabytes per feed build. - runIssueId: sql`${heartbeatRuns.contextSnapshot} ->> 'issueId'`, - runTaskId: sql`${heartbeatRuns.contextSnapshot} ->> 'taskId'`, - }) - .from(heartbeatRuns) - .where(and( - eq(heartbeatRuns.companyId, companyId), - inArray(heartbeatRuns.agentId, failedAgentIds), - gt(heartbeatRuns.createdAt, oldestFailedRunCreatedAt), - )) - : Promise.resolve([]), + listFailedRunsWithNewerRun(db, companyId, failedRows.map((row, index) => ({ + id: row.id, + agentId: row.agentId, + issueId: failedIssueIds[index], + createdAt: row.createdAt, + }))), ]); - const latestRunCreatedAtByKey = new Map(); - for (const newerRun of newerRuns) { - const newerRunIssueId = readRunIssueId({ issueId: newerRun.runIssueId, taskId: newerRun.runTaskId }); - const newerRunKey = `${newerRun.agentId}:${newerRunIssueId ?? ""}`; - const latestCreatedAt = latestRunCreatedAtByKey.get(newerRunKey); - if (!latestCreatedAt || newerRun.createdAt > latestCreatedAt) { - latestRunCreatedAtByKey.set(newerRunKey, newerRun.createdAt); - } - } for (const run of failedRows) { const issueId = readRunIssueId(run.contextSnapshot); - const runKey = `${run.agentId}:${issueId ?? ""}`; - const hasNewerRun = (latestRunCreatedAtByKey.get(runKey)?.getTime() ?? 0) > run.createdAt.getTime(); - if (hasNewerRun) continue; + if (failedRunsWithNewerRun.has(run.id)) continue; const issue = issueId ? failedIssueMap.get(issueId) ?? null : null; if (issueId && !issue) continue; From f9581e533e899ce6c67ad7b74ee1f38321a348e5 Mon Sep 17 00:00:00 2001 From: Naeem Akhtar <32741588+naeemakhtar110@users.noreply.github.com> Date: Mon, 26 Oct 2026 01:35:36 +0300 Subject: [PATCH 53/68] perf(server): find the native event cursor through the source seq index (#16140) Find the contiguous native cursor in PostgreSQL while preserving gaps, late arrivals, source isolation, and event replay. Verify a long-history response budget against current master. Co-Authored-By: Paperclip --- doc/DATABASE.md | 6 ++ .../heartbeat-run-event-sequencing.test.ts | 101 +++++++++++++++++- server/src/services/heartbeat-run-events.ts | 42 +++++--- 3 files changed, 136 insertions(+), 13 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index ed99988523b..e35cf130381 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -49,6 +49,12 @@ and leaves all existing cases passive. Source identities are not grants. ### Task query performance +Native event acknowledgements read a count and maximum through the existing +source-sequence index. A stream with gaps uses indexed lookups for its first +missing successor. The acknowledgement keeps its contiguous-prefix rule without +returning every historical sequence number. All events stay stored. Counting the +index entries still grows with the stream length. + The attention feed checks for newer runs with company-scoped existence probes. The probes match the failed run's agent and issue or task context. They preserve the existing millisecond comparison and empty-context rules. Each batch holds diff --git a/server/src/__tests__/heartbeat-run-event-sequencing.test.ts b/server/src/__tests__/heartbeat-run-event-sequencing.test.ts index 28c2dd00d24..d8e7bd0b9cb 100644 --- a/server/src/__tests__/heartbeat-run-event-sequencing.test.ts +++ b/server/src/__tests__/heartbeat-run-event-sequencing.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it } from "vitest"; -import { eq } from "drizzle-orm"; +import { eq, sql } from "drizzle-orm"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { agents, companies, @@ -15,6 +16,62 @@ import { import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; describe("P6-11..13 / P6-17 canonical event allocator", () => { + it("keeps native cursor responses bounded with long histories and a late gap fill", async () => { + const temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-cursor-budget-"); + const db = createDb(temporary.connectionString); + const companyId = "20000000-0000-4000-8000-000000000001"; + const agentId = "20000000-0000-4000-8000-000000000002"; + const runId = "20000000-0000-4000-8000-000000000003"; + const otherRunId = "20000000-0000-4000-8000-000000000004"; + try { + await db.insert(companies).values({ id: companyId, name: "Cursor budget", issuePrefix: "CBG" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Cursor agent" }); + await db.insert(heartbeatRuns).values([ + { id: runId, companyId, agentId, status: "running", nextEventSeq: 5005 }, + { id: otherRunId, companyId, agentId, status: "running" }, + ]); + for (let batch = 0; batch < 10; batch += 1) { + await db.insert(heartbeatRunEvents).values(Array.from({ length: 500 }, (_, index) => { + const seq = batch * 500 + index + 1; + return { companyId, agentId, runId, seq, eventType: "item.delta", sourceInstanceId: "runner-a", + sourceSeq: seq, sourceEventId: `seed:${seq}` }; + })); + } + await db.insert(heartbeatRunEvents).values([ + ...[-1, 0, null].map((sourceSeq, index) => ({ + companyId, agentId, runId, seq: 5001 + index, eventType: "item.delta", + sourceInstanceId: "runner-a", sourceSeq, sourceEventId: `legacy:${index}`, + })), + { companyId, agentId, runId, seq: 5004, eventType: "item.delta", sourceInstanceId: "runner-b", + sourceSeq: 5001, sourceEventId: "other-source:5001" }, + { companyId, agentId, runId: otherRunId, seq: 1, eventType: "item.delta", sourceInstanceId: "runner-a", + sourceSeq: 5001, sourceEventId: "other-run:5001" }, + ]); + const append = (sourceSeq: number) => appendHeartbeatRunEvent(db, { + companyId, agentId, runId, eventType: "item.delta", payload: { sourceSeq }, + nativeSource: { sourceInstanceId: "runner-a", sourceSeq, sourceEventId: `runner-a:${sourceSeq}`, + protocolSchemaVersion: 1, canonicalPayload: { sourceSeq } }, + }); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRunEvents }, capture); + try { expect((await append(5002)).highestContiguousSourceSeq).toBe(5000); } + finally { capture.flush(); } + const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && event.table === "heartbeat_run_events" && (event.operation === "select" || event.operation === "execute")); + expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThan(0); + expect(reads.reduce((sum, event) => sum + Number(event.rows), 0)).toBeLessThan(16); + expect(reads.reduce((sum, event) => sum + Number(event.responseValueBytes), 0)).toBeLessThan(2048); + expect((await append(5001)).highestContiguousSourceSeq).toBe(5002); + const replay = await append(5002); + expect(replay.disposition).toBe("duplicate"); + expect(replay.highestContiguousSourceSeq).toBe(5002); + const [remaining] = await db.select({ count: sql`count(*)::int` }).from(heartbeatRunEvents) + .where(eq(heartbeatRunEvents.runId, runId)); + expect(remaining?.count).toBe(5006); + } finally { await temporary.cleanup(); } + }, 60_000); + it.each([false, true])("deduplicates concurrent retry exhaustion (historical receipt: %s)", async (historical) => { const temporary = await startEmbeddedPostgresTestDatabase("paperclip-exhaustion-events-"); const db = createDb(temporary.connectionString); @@ -165,4 +222,46 @@ describe("P6-11..13 / P6-17 canonical event allocator", () => { await temporary.cleanup(); } }, 60_000); + + it("reports the highest contiguous source seq per source instance across gaps and late arrivals", async () => { + const temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-cursor-"); + const db = createDb(temporary.connectionString); + const companyId = "20000000-0000-4000-8000-000000000001"; + const agentId = "20000000-0000-4000-8000-000000000002"; + const runId = "20000000-0000-4000-8000-000000000003"; + try { + await db.insert(companies).values({ id: companyId, name: "Cursor fixture", issuePrefix: "CUR" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Cursor agent" }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running" }); + const append = async (sourceInstanceId: string, sourceSeq: number) => (await appendHeartbeatRunEvent(db, { + companyId, + runId, + agentId, + eventType: "item.delta", + payload: { sourceInstanceId, sourceSeq }, + nativeSource: { + sourceInstanceId, + sourceEventId: `${sourceInstanceId}:${sourceSeq}`, + sourceSeq, + protocolSchemaVersion: 1, + canonicalPayload: { sourceInstanceId, sourceSeq }, + }, + })).highestContiguousSourceSeq; + + expect(await append("runner-a", 2)).toBe(0); + expect(await append("runner-a", 1)).toBe(2); + expect(await append("runner-a", 3)).toBe(3); + expect(await append("runner-a", 5)).toBe(3); + expect(await append("runner-a", 6)).toBe(3); + expect(await append("runner-b", 1)).toBe(1); + expect(await append("runner-a", 4)).toBe(6); + for (let sourceSeq = 7; sourceSeq <= 200; sourceSeq += 1) { + expect(await append("runner-a", sourceSeq)).toBe(sourceSeq); + } + expect(await append("runner-b", 3)).toBe(1); + expect(await append("runner-b", 2)).toBe(3); + } finally { + await temporary.cleanup(); + } + }, 60_000); }); diff --git a/server/src/services/heartbeat-run-events.ts b/server/src/services/heartbeat-run-events.ts index ecb7ff5864a..efefbc63b3c 100644 --- a/server/src/services/heartbeat-run-events.ts +++ b/server/src/services/heartbeat-run-events.ts @@ -202,18 +202,36 @@ export async function appendHeartbeatRunEvent( } async function contiguousCursor(db: Db, runId: string, sourceInstanceId: string): Promise { - const rows = await db + const scope = and( + eq(heartbeatRunEvents.runId, runId), + eq(heartbeatRunEvents.sourceInstanceId, sourceInstanceId), + sql`${heartbeatRunEvents.sourceSeq} >= 1`, + ); + const [range] = await db + .select({ + count: sql`count(*)`.mapWith(Number), + max: sql`max(${heartbeatRunEvents.sourceSeq})`.mapWith(Number), + }) + .from(heartbeatRunEvents) + .where(scope); + if (!range || !range.count) return 0; + if (range.max === range.count) return range.count; + const [first] = await db .select({ sourceSeq: heartbeatRunEvents.sourceSeq }) .from(heartbeatRunEvents) - .where(and( - eq(heartbeatRunEvents.runId, runId), - eq(heartbeatRunEvents.sourceInstanceId, sourceInstanceId), - )) - .orderBy(asc(heartbeatRunEvents.sourceSeq)); - let cursor = 0; - for (const row of rows) { - if (row.sourceSeq === cursor + 1) cursor += 1; - else if (row.sourceSeq !== null && row.sourceSeq > cursor + 1) break; - } - return cursor; + .where(and(scope, eq(heartbeatRunEvents.sourceSeq, 1))) + .limit(1); + if (!first) return 0; + const [last] = await db + .select({ sourceSeq: heartbeatRunEvents.sourceSeq }) + .from(heartbeatRunEvents) + .where(and(scope, sql`not exists ( + select 1 from heartbeat_run_events next_event + where next_event.run_id = heartbeat_run_events.run_id + and next_event.source_instance_id = heartbeat_run_events.source_instance_id + and next_event.source_seq = heartbeat_run_events.source_seq + 1 + )`)) + .orderBy(asc(heartbeatRunEvents.sourceSeq)) + .limit(1); + return last?.sourceSeq ?? 0; } From 9f3a5456acda24cab7040daa67cefb321dd73f15 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 15:53:18 -0700 Subject: [PATCH 54/68] perf(server): bound run reads in authentication and conversation recovery (#17663) Read compact run identity during authentication and compact turn state during conversation recovery. Preserve fresh checks, durable steering and reply evidence, locking, and complete stored payloads. Validate 184 targeted PostgreSQL and lifecycle tests, workspace typecheck, server build, full CI, and Apex 5/5. Co-Authored-By: Paperclip --- doc/DATABASE.md | 12 ++ .../agent-auth-identity-transfer.test.ts | 102 +++++++++++++++++ .../conversation-settlement-transfer.test.ts | 105 ++++++++++++++++++ server/src/__tests__/run-identity.test.ts | 22 ++-- server/src/middleware/auth.ts | 3 +- server/src/services/agent-conversations.ts | 31 +++++- server/src/services/recovery/service.ts | 4 +- server/src/services/run-identity.ts | 4 +- 8 files changed, 265 insertions(+), 18 deletions(-) create mode 100644 server/src/__tests__/agent-auth-identity-transfer.test.ts create mode 100644 server/src/__tests__/conversation-settlement-transfer.test.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index e35cf130381..d5f970ac23d 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -49,12 +49,24 @@ and leaves all existing cases passive. Source identities are not grants. ### Task query performance +Agent request authentication uses a compact run-identity read. It omits saved +execution payloads, results, and log excerpts from the query response. The read +still locks the task before the run and reconciles pending steering from durable +acknowledgements on every request. Default identity reads retain the full run. +Stored payloads and separate native tool authorization checks remain unchanged. + Native event acknowledgements read a count and maximum through the existing source-sequence index. A stream with gaps uses indexed lookups for its first missing successor. The acknowledgement keeps its contiguous-prefix rule without returning every historical sequence number. All events stay stored. Counting the index entries still grows with the stream length. +Conversation recovery reads only the saved run fields used to settle a turn. +Settlement locks and reads current conversation state without loading the task +description or comment bodies. It still checks the durable reply or pending +interaction, session generation, owning run, and newer user messages. Saved +execution payloads, task text, and comments remain stored in full. + The attention feed checks for newer runs with company-scoped existence probes. The probes match the failed run's agent and issue or task context. They preserve the existing millisecond comparison and empty-context rules. Each batch holds diff --git a/server/src/__tests__/agent-auth-identity-transfer.test.ts b/server/src/__tests__/agent-auth-identity-transfer.test.ts new file mode 100644 index 00000000000..819b23dc64a --- /dev/null +++ b/server/src/__tests__/agent-auth-identity-transfer.test.ts @@ -0,0 +1,102 @@ +import { randomUUID } from "node:crypto"; +import express from "express"; +import request from "supertest"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issueComments, issues } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { createLocalAgentJwt } from "../agent-auth-jwt.js"; +import { actorMiddleware } from "../middleware/auth.js"; +import { errorHandler } from "../middleware/error-handler.js"; +import { captureRunIdentity, initializeRunIdentity, reserveSteeredIdentity } from "../services/run-identity.js"; + +const support = await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("agent authentication identity transfer", () => { + let database: Awaited>; + let db: ReturnType; + beforeAll(async () => { + vi.stubEnv("PAPERCLIP_AGENT_JWT_SECRET", "identity-transfer-test-secret"); + vi.stubEnv("PAPERCLIP_INSTANCE_ID", "default"); + database = await startEmbeddedPostgresTestDatabase("paperclip-auth-identity-transfer-"); + db = createDb(database.connectionString); + }, 30_000); + afterAll(async () => { + vi.unstubAllEnvs(); + await db?.$client.end({ timeout: 5 }); + await database?.cleanup(); + }, 60_000); + + async function seed() { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Authentication transfer", issuePrefix: companyId.slice(0, 8) }); + await db.insert(agents).values({ id: agentId, companyId, name: "Reader", role: "engineer", adapterType: "codex_local", status: "idle" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Authentication task", responsibleUserId: "owner" }); + const padding = "retained input ".repeat(32_768); + const payloads = { + contextSnapshot: { issueId, launchInput: padding }, + runnerProfileJson: { nativeExecutionInput: { savedInput: padding } }, + resultJson: { savedResult: padding }, stdoutExcerpt: padding, stderrExcerpt: padding, + }; + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", ...payloads }); + const identity = await initializeRunIdentity(db, { companyId, runId, issueId, messageIds: [], responsibleUserId: "owner", cause: "instruction" }); + const app = express(); + app.use(actorMiddleware(db, { deploymentMode: "authenticated", resolveSession: async () => null })); + app.get("/actor", (req, res) => res.json(req.actor)); + app.patch("/actor", (req, res) => res.json(req.actor)); + app.use(errorHandler); + const token = createLocalAgentJwt(agentId, companyId, "codex_local", runId, "stale-user"); + expect(token).not.toBeNull(); + return { companyId, agentId, issueId, runId, app, token: token!, identity, payloads }; + } + + it("bounds actual middleware responses and preserves the complete stored run", async () => { + const fixture = await seed(); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { + const response = await request(fixture.app).get("/actor").set("Authorization", `Bearer ${fixture.token}`); + expect(response.status).toBe(200); + expect(response.body).toMatchObject({ type: "agent", runId: fixture.runId, + identityContextId: fixture.identity.id, onBehalfOfUserId: "owner" }); + } finally { capture.flush(); } + const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && event.table === "heartbeat_runs" && (event.operation === "select" || event.operation === "execute")); + expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThanOrEqual(3); + expect(reads.reduce((sum, event) => sum + Number(event.responseValueBytes), 0)).toBeLessThan(16_384); + expect((await captureRunIdentity(db, fixture)).run).toMatchObject(fixture.payloads); + }); + + it("blocks uncertain steering, accepts its durable acknowledgement, and rechecks cancellation", async () => { + const fixture = await seed(), messageId = randomUUID(); + await db.insert(issueComments).values({ id: messageId, companyId: fixture.companyId, issueId: fixture.issueId, + authorUserId: "next-user", body: "Next instruction" }); + const pending = await reserveSteeredIdentity(db, { ...fixture, messageId }); + expect(pending).not.toBeNull(); + const blocked = await request(fixture.app).get("/actor").set("Authorization", `Bearer ${fixture.token}`); + expect(blocked.status).toBe(409); + await db.insert(heartbeatRunEvents).values({ companyId: fixture.companyId, agentId: fixture.agentId, runId: fixture.runId, + seq: 1, eventType: "item.completed", sourceEventId: randomUUID(), + payload: { prpEvent: { turnId: "auth-turn", itemId: `auth-turn:steer:${messageId}`, + payload: { kind: "steering_acknowledgement" } } } }); + const accepted = await request(fixture.app).get("/actor").set("Authorization", `Bearer ${fixture.token}`); + expect(accepted.status).toBe(200); + expect(accepted.body).toMatchObject({ identityContextId: pending!.id, onBehalfOfUserId: "next-user" }); + await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, fixture.runId)); + const revoked = await request(fixture.app).patch("/actor").set("Authorization", `Bearer ${fixture.token}`); + expect(revoked.status).toBe(403); + expect(revoked.body.code).toBe("agent_run_cancelled"); + const [stored] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect(stored).toMatchObject(fixture.payloads); + }); + + it.each(["company", "agent", "missing", "finished"] as const)("rejects %s run identity for the compact scope", async (fault) => { + const fixture = await seed(); + if (fault === "finished") await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, fixture.runId)); + const input = { companyId: fault === "company" ? randomUUID() : fixture.companyId, + agentId: fault === "agent" ? randomUUID() : fixture.agentId, + runId: fault === "missing" ? randomUUID() : fixture.runId }; + await expect(captureRunIdentity(db, input, { readScope: "request_authentication" })).rejects.toThrow(/active run/); + }); +}); diff --git a/server/src/__tests__/conversation-settlement-transfer.test.ts b/server/src/__tests__/conversation-settlement-transfer.test.ts new file mode 100644 index 00000000000..6a128b00e95 --- /dev/null +++ b/server/src/__tests__/conversation-settlement-transfer.test.ts @@ -0,0 +1,105 @@ +import { randomUUID } from "node:crypto"; +import { and, eq, sql } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, companies, createDb, heartbeatRuns, issueComments, issueThreadInteractions, issues } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { settleConversationTurnForRun } from "../services/agent-conversations.js"; + +const support = await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("conversation settlement transfer", () => { + let database: Awaited>; + let db: ReturnType; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("paperclip-conversation-settlement-"); + db = createDb(database.connectionString); + }, 30_000); + afterAll(async () => { await db?.$client.end({ timeout: 5 }); await database?.cleanup(); }, 60_000); + + async function seed(reply: "response" | "interaction" | "reset" | "none" = "response") { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), wakeId = randomUUID(); + const padding = "saved conversation ".repeat(32_768); + await db.insert(companies).values({ id: companyId, name: "Settlement transfer", issuePrefix: companyId.slice(0, 8) }); + await db.insert(agents).values({ id: agentId, companyId, name: "Conversation agent", role: "engineer" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Conversation", description: padding, + status: "in_progress", conversationState: "active", conversationUserId: "conversation-owner", + conversationAgentId: agentId, assigneeAgentId: agentId, conversationSessionGeneration: 3 }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "succeeded", + contextSnapshot: { issueId, conversationSessionGeneration: 3, wakeCommentId: wakeId, + conversationReset: reply === "reset", savedInput: padding }, + runnerProfileJson: { savedProfile: padding }, resultJson: { savedResult: padding }, + stdoutExcerpt: padding, stderrExcerpt: padding }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + const wakeTime = new Date(Date.now() - 10_000); + await db.insert(issueComments).values({ id: wakeId, companyId, issueId, authorUserId: "conversation-owner", + body: padding, createdAt: wakeTime }); + if (reply === "response") await db.insert(issueComments).values({ companyId, issueId, + createdByRunId: runId, authorAgentId: agentId, origin: "comment", body: padding }); + if (reply === "interaction") await db.insert(issueThreadInteractions).values({ companyId, issueId, + sourceRunId: runId, createdByAgentId: agentId, status: "pending", kind: "request_confirmation", payload: { version: 1 } }); + return { companyId, agentId, issueId, runId, wakeId, wakeTime }; + } + + async function sourceDigests(input: Awaited>) { + return db.execute(sql`select + (select encode(sha256(convert_to(jsonb_build_array(context_snapshot,runner_profile_json,result_json,stdout_excerpt,stderr_excerpt)::text,'UTF8')),'hex') + from heartbeat_runs where id=${input.runId}::uuid) as run_digest, + (select encode(sha256(convert_to(description,'UTF8')),'hex') from issues where id=${input.issueId}::uuid) as description_digest, + (select encode(sha256(convert_to(jsonb_agg(jsonb_build_array(id,body) order by id)::text,'UTF8')),'hex') + from issue_comments where issue_id=${input.issueId}::uuid) as comments_digest`); + } + + it.each(["response", "interaction", "reset"] as const)("bounds %s settlement reads and retains all saved text and run payloads", async reply => { + const input = await seed(reply), before = await sourceDigests(input); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns, issues, issueComments }, capture); + try { expect(await settleConversationTurnForRun(db, input.companyId, input.runId)).toBe(true); } + finally { capture.flush(); } + const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && ["heartbeat_runs", "issues", "issue_comments"].includes(String(event.table)) + && (event.operation === "select" || event.operation === "execute")); + expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThanOrEqual(4); + expect(reads.reduce((sum, event) => sum + Number(event.responseValueBytes), 0)).toBeLessThan(16_384); + const [issue] = await db.select({ status: issues.status, state: issues.conversationState }).from(issues).where(eq(issues.id, input.issueId)); + expect(issue).toEqual({ status: "in_review", state: "waiting" }); + expect(await sourceDigests(input)).toEqual(before); + }); + + it.each(["conversationReplyBoundaryCommentId", "wakeCommentId", "commentId"] as const)("keeps later user messages active through %s", async key => { + const input = await seed(); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: input.issueId, + conversationSessionGeneration: 3, [key]: input.wakeId } }).where(eq(heartbeatRuns.id, input.runId)); + await db.insert(issueComments).values({ companyId: input.companyId, issueId: input.issueId, + authorUserId: "conversation-owner", body: "A new instruction", createdAt: new Date(input.wakeTime.getTime() + 1000) }); + expect(await settleConversationTurnForRun(db, input.companyId, input.runId)).toBe(true); + const [issue] = await db.select({ status: issues.status, state: issues.conversationState }).from(issues).where(eq(issues.id, input.issueId)); + expect(issue).toEqual({ status: "in_progress", state: "active" }); + }); + + it.each(["missing", "company", "issue-company", "generation", "owner", "failed", "no-reply"] as const)("refuses to settle a %s mismatch", async fault => { + const input = await seed(fault === "no-reply" ? "none" : "response"); + if (fault === "generation") await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: input.issueId, + conversationSessionGeneration: 2, wakeCommentId: input.wakeId } }).where(eq(heartbeatRuns.id, input.runId)); + if (fault === "owner") { + const otherRun = randomUUID(); + await db.insert(heartbeatRuns).values({ id: otherRun, companyId: input.companyId, agentId: input.agentId }); + await db.update(issues).set({ executionRunId: otherRun }).where(eq(issues.id, input.issueId)); + } + if (fault === "failed") await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, input.runId)); + if (fault === "issue-company") { + const otherCompany = randomUUID(), otherIssue = randomUUID(), otherAgent = randomUUID(); + await db.insert(companies).values({ id: otherCompany, name: "Other company", issuePrefix: otherCompany.slice(0, 8) }); + await db.insert(agents).values({ id: otherAgent, companyId: otherCompany, name: "Other agent" }); + await db.insert(issues).values({ id: otherIssue, companyId: otherCompany, title: "Other conversation", + conversationUserId: "other-owner", conversationAgentId: otherAgent, assigneeAgentId: otherAgent, + conversationState: "active", conversationSessionGeneration: 3 }); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: otherIssue, conversationSessionGeneration: 3, + conversationReset: true } }).where(eq(heartbeatRuns.id, input.runId)); + } + expect(await settleConversationTurnForRun(db, fault === "company" ? randomUUID() : input.companyId, + fault === "missing" ? randomUUID() : input.runId)).toBe(false); + const [issue] = await db.select({ status: issues.status }).from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId))); + expect(issue.status).toBe("in_progress"); + }); +}); diff --git a/server/src/__tests__/run-identity.test.ts b/server/src/__tests__/run-identity.test.ts index b371c59f043..069139a8123 100644 --- a/server/src/__tests__/run-identity.test.ts +++ b/server/src/__tests__/run-identity.test.ts @@ -29,14 +29,14 @@ const support = await getEmbeddedPostgresTestSupport(); } return { companyId, agentId, issueId, runId, messageIds }; } - it("retains typed GitHub trust bindings and keeps other identity reads complete", async () => { + it.each(["github_credentials", "request_authentication"] as const)("retains typed trust bindings for %s and keeps other identity reads complete", async (readScope) => { const input = await seed(); await initializeRunIdentity(db, { ...input, responsibleUserId: "owner", cause: "queued" }); for (const value of [null, true, 42, " padded ", [], {}, { future: { opaque: true } }]) { const contextSnapshot = { issueId: input.issueId, projectId: value, executionPolicy: value, savedInput: { retained: true } }; const runnerProfileJson = { nativeExecutionInput: { opaque: value }, savedInput: "retained" }; await db.update(heartbeatRuns).set({ contextSnapshot, runnerProfileJson }).where(eq(heartbeatRuns.id, input.runId)); - const narrow = await captureRunIdentity(db, input, { readScope: "github_credentials" }); + const narrow = await captureRunIdentity(db, input, { readScope }); expect(narrow.run.contextSnapshot).toEqual({ issueId: input.issueId, taskId: null, projectId: value, executionPolicy: value }); expect(narrow.run.runnerProfileJson).toBeNull(); const full = await captureRunIdentity(db, input); @@ -123,24 +123,25 @@ const support = await getEmbeddedPostgresTestSupport(); }, ); - it("holds acquisition during uncertain steering, preserves snapshots, and never rewinds on replay", async () => { + it.each([undefined, "github_credentials", "request_authentication"] as const)("holds acquisition during uncertain steering and never rewinds on replay with %s", async (readScope) => { + const options = readScope ? { readScope } : undefined; const input = await seed(); await initializeRunIdentity(db, { ...input, messageIds: [input.messageIds[0]], responsibleUserId: "A", cause: "instruction" }); - const startedUnderA = await captureRunIdentity(db, input); + const startedUnderA = await captureRunIdentity(db, input, options); const b = await reserveSteeredIdentity(db, { ...input, messageId: input.messageIds[1] }); expect(b).not.toBeNull(); - await expect(captureRunIdentity(db, input)).rejects.toThrow(/reconciled/); + await expect(captureRunIdentity(db, input, options)).rejects.toThrow(/reconciled/); await expect(reserveSteeredIdentity(db, { ...input, messageId: input.messageIds[2] })).rejects.toThrow(/reconciled/); await db.transaction(async tx => { await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)).for("update"); await acceptSteeredIdentity(tx, b!); }); - expect((await captureRunIdentity(db, input)).context?.responsibleUserId).toBe("B"); + expect((await captureRunIdentity(db, input, options)).context?.responsibleUserId).toBe("B"); expect(startedUnderA.context?.responsibleUserId).toBe("A"); const a = await reserveSteeredIdentity(db, { ...input, messageId: input.messageIds[2] }); await acceptSteeredIdentity(db, a!); await acceptSteeredIdentity(db, b!); - expect((await captureRunIdentity(db, input)).context?.responsibleUserId).toBe("A"); + expect((await captureRunIdentity(db, input, options)).context?.responsibleUserId).toBe("A"); }); it("keeps rejected steering unchanged and preserves the originating identity on retry", async () => { const input = await seed(); @@ -175,16 +176,17 @@ const support = await getEmbeddedPostgresTestSupport(); expect(captured.run.responsibleUserId).toBeNull(); expect(captured.context?.parentContextId).toBe(origin.id); }); - it("recovers an uncertain acknowledgement from the authenticated native event journal", async () => { + it.each([undefined, "github_credentials", "request_authentication"] as const)("recovers an uncertain native acknowledgement with %s", async (readScope) => { + const options = readScope ? { readScope } : undefined; const input = await seed(); await initializeRunIdentity(db, { ...input, messageIds: [], responsibleUserId: "A", cause: "instruction" }); const pending = await reserveSteeredIdentity(db, { ...input, messageId: input.messageIds[1] }); - await expect(captureRunIdentity(db, input)).rejects.toThrow(/reconciled/); + await expect(captureRunIdentity(db, input, options)).rejects.toThrow(/reconciled/); await db.insert(heartbeatRunEvents).values({ companyId: input.companyId, runId: input.runId, agentId: input.agentId, seq: 1, eventType: "item.completed", sourceEventId: randomUUID(), payload: { prpEvent: { turnId: "turn-1", itemId: `turn-1:steer:${pending!.messageId}`, payload: {kind: "steering_acknowledgement"} } }, }); - expect((await captureRunIdentity(db, input)).context?.responsibleUserId).toBe("B"); + expect((await captureRunIdentity(db, input, options)).context?.responsibleUserId).toBe("B"); await db.update(heartbeatRuns).set({status: "succeeded"}).where(eq(heartbeatRuns.id, input.runId)); expect((await reserveSteeredIdentity(db, { ...input, messageId: input.messageIds[1] }))?.status).toBe("accepted"); }); diff --git a/server/src/middleware/auth.ts b/server/src/middleware/auth.ts index 8504fc3c853..86347bb40d5 100644 --- a/server/src/middleware/auth.ts +++ b/server/src/middleware/auth.ts @@ -429,7 +429,8 @@ export function actorMiddleware(db: Db, opts: ActorMiddlewareOptions): RequestHa return; } if (identityRun?.activeIdentityContextId && identityRun.status === "running") { - const captured = await captureRunIdentity(db, { companyId: claims.company_id, agentId: claims.sub, runId: claims.run_id }); + const captured = await captureRunIdentity(db, { companyId: claims.company_id, agentId: claims.sub, runId: claims.run_id }, + { readScope: "request_authentication" }); identityRun.activeIdentityContextId = captured.context?.id ?? null; identityRun.responsibleUserId = captured.context?.responsibleUserId ?? null; } diff --git a/server/src/services/agent-conversations.ts b/server/src/services/agent-conversations.ts index a2ec8f74b01..b6cfd1a7337 100644 --- a/server/src/services/agent-conversations.ts +++ b/server/src/services/agent-conversations.ts @@ -282,7 +282,7 @@ export async function prepareConversationTurn( /** Finalizers only park a turn with a durable response, interaction, or processed /new. */ export async function settleConversationTurn( db: Db, - run: typeof heartbeatRuns.$inferSelect, + run: Pick, ) { if (run.status !== "succeeded") return false; const context = run.contextSnapshot ?? {}; @@ -291,7 +291,12 @@ export async function settleConversationTurn( let publication: ActivityPublication | null = null; const settled = await db.transaction(async (tx) => { const [issue] = await tx - .select() + .select({ + id: issues.id, companyId: issues.companyId, status: issues.status, + conversationAgentId: issues.conversationAgentId, conversationUserId: issues.conversationUserId, + conversationState: issues.conversationState, conversationSessionGeneration: issues.conversationSessionGeneration, + executionRunId: issues.executionRunId, + }) .from(issues) .where(and(eq(issues.id, issueId), eq(issues.companyId, run.companyId))) .for("update"); @@ -344,7 +349,7 @@ export async function settleConversationTurn( const [wake] = typeof wakeId === "string" ? await tx - .select() + .select({ id: issueComments.id }) .from(issueComments) .where(eq(issueComments.id, wakeId)) : []; @@ -407,6 +412,26 @@ export async function settleConversationTurn( return settled; } +export async function settleConversationTurnForRun( + db: Db, + companyId: string, + runId: string, +) { + const [run] = await db.select({ + id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, status: heartbeatRuns.status, + contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then jsonb_build_object( + 'issueId', ${heartbeatRuns.contextSnapshot}->'issueId', + 'conversationReset', ${heartbeatRuns.contextSnapshot}->'conversationReset', + 'conversationSessionGeneration', ${heartbeatRuns.contextSnapshot}->'conversationSessionGeneration', + 'conversationReplyBoundaryCommentId', ${heartbeatRuns.contextSnapshot}->'conversationReplyBoundaryCommentId', + 'wakeCommentId', ${heartbeatRuns.contextSnapshot}->'wakeCommentId', + 'commentId', ${heartbeatRuns.contextSnapshot}->'commentId' + ) else null end`, + }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId))); + return run ? settleConversationTurn(db, run) : false; +} + /** Fresh provider context uses only messages in this session, up to this turn. */ export async function conversationReplay( db: Db, diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index 488ef813306..3bbf40ab0d2 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -21,7 +21,7 @@ import { import { CONTROLLER_LEASE_LOST_FAILURE_CODE, hasLiveLegacyController } from "../legacy-controller-lease.js"; import { isCompletedOnboardingHandoffWake } from "../chat-completion-delivery.js"; import { instanceSettingsService } from "../instance-settings.js"; -import { isWaitingConversation, settleConversationTurn, deliverConversationComments } from "../agent-conversations.js"; +import { isWaitingConversation, settleConversationTurnForRun, deliverConversationComments } from "../agent-conversations.js"; import { and, asc, @@ -4582,7 +4582,7 @@ export function recoveryService( if (issue.conversationAgentId) { const lastRun = await getLatestIssueRun(issue.companyId, issue.id); if (lastRun?.status === "succeeded") { - if (await settleConversationTurn(db, (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, lastRun.id)))[0]!)) { + if (await settleConversationTurnForRun(db, issue.companyId, lastRun.id)) { const [current] = await db.select().from(issues).where(eq(issues.id, issue.id)); if (current) Object.assign(issue, current); } diff --git a/server/src/services/run-identity.ts b/server/src/services/run-identity.ts index b1e27d1ae72..2f1e9e97d65 100644 --- a/server/src/services/run-identity.ts +++ b/server/src/services/run-identity.ts @@ -434,13 +434,13 @@ export async function acceptSteeredIdentity( export async function captureRunIdentity( db: Db, input: { companyId: string; runId: string; agentId: string }, - options?: { readScope: "github_credentials" }, + options?: { readScope: "github_credentials" | "request_authentication" }, ) { // Lock acquisition serializes with steering delivery and its durable acknowledgement. return db.transaction(async (tx) => { await lockIdentityTask(tx, input.companyId, input.runId); const [run] = await tx - .select(options?.readScope === "github_credentials" ? { + .select(options?.readScope === "github_credentials" || options?.readScope === "request_authentication" ? { ...getTableColumns(heartbeatRuns), contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' then jsonb_build_object('issueId', ${heartbeatRuns.contextSnapshot}->'issueId', From 5a28f7acf8a8c86536c4f5c5369bfeb49a46c41a Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 16:06:36 -0700 Subject: [PATCH 55/68] fix(server): include restored monitor proof in assessment identity (#17665) Include validated preserved monitor proof in the immutable assessment input so restored receipts or completion times can produce a new status decision at the same deadline. Preserve accepted results, earlier decisions, provider attempts, monitor deadlines, and idempotent replay. Validate two baseline regressions, 201 targeted tests on current master, workspace typecheck, server build, full CI, and Apex 5/5. Co-Authored-By: Paperclip --- doc/DATABASE.md | 5 ++++ .../native-preserved-monitor-wait.test.ts | 25 ++++++++++++++++--- .../native-runtime/native-run-finalizer.ts | 1 + .../native-runtime/work-assessments.ts | 4 ++- 4 files changed, 30 insertions(+), 5 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index d5f970ac23d..cb341f8ea79 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -175,6 +175,11 @@ launch context does not travel with each policy or parent-report check. JSON types, task-key precedence, and company and agent identity checks are preserved. Each decision still reads current data; this is not an authorization cache. +Restoring a saved monitor's receipt or completion proof creates a new native +assessment identity even when its scheduled deadline is unchanged. The saved +proof is part of the assessment input. Prior decisions and accepted results stay +intact, and an unchanged replay reuses the same assessment and decision. + Native tool authority lookups select current run identity, status, invocation, responsible user, context, and receipt data. Saved profiles and output excerpts stay in storage and are omitted from that read. Task and agent binding, current diff --git a/server/src/services/native-runtime/native-preserved-monitor-wait.test.ts b/server/src/services/native-runtime/native-preserved-monitor-wait.test.ts index ad269c7467c..1cedcb28be8 100644 --- a/server/src/services/native-runtime/native-preserved-monitor-wait.test.ts +++ b/server/src/services/native-runtime/native-preserved-monitor-wait.test.ts @@ -126,21 +126,38 @@ describe.skipIf(!support.supported)("released native monitor results", () => { }, ); - it("supersedes an earlier authority error without replaying the accepted result", async () => { + it.each(["schedule", "receipt", "finished_time"] as const)("supersedes an earlier authority error after restoring %s without replaying the result", async state => { const f = await fixture(), scheduled = await f.readIssue(); - await db.update(issues).set({ executionPolicy: {}, monitorNextCheckAt: null }).where(eq(issues.id, f.issueId)); + const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + await db.update(nativeRunFinalizations).set({ controllerBootId: randomUUID(), attempt: 1 }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + if (state === "schedule") await db.update(issues).set({ executionPolicy: {}, monitorNextCheckAt: null }).where(eq(issues.id, f.issueId)); + if (state === "receipt") await db.update(heartbeatRuns).set({ resultJson: {} }).where(eq(heartbeatRuns.id, f.runId)); + if (state === "finished_time") await db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, f.runId)); await f.finalize(); const old = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId)); expect(old.some(decision => decision.reasonCode === "monitor_wait_authority_lost")).toBe(true); - await db.update(issues).set({ executionPolicy: scheduled.executionPolicy, monitorNextCheckAt: scheduled.monitorNextCheckAt }) + if (state === "schedule") await db.update(issues).set({ executionPolicy: scheduled.executionPolicy, monitorNextCheckAt: scheduled.monitorNextCheckAt }) .where(eq(issues.id, f.issueId)); + if (state === "receipt") { + const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + await db.update(heartbeatRuns).set({ resultJson: { ...current.resultJson, + semanticToolReceipts: source.resultJson?.semanticToolReceipts } }).where(eq(heartbeatRuns.id, f.runId)); + } + if (state === "finished_time") await db.update(heartbeatRuns).set({ finishedAt: f.finishedAt }).where(eq(heartbeatRuns.id, f.runId)); await f.finalize(); - expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0].phase).toBe("committed"); + const [settled] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(settled).toMatchObject({ phase: "committed", nextAttemptAt: null, attempt: 1 }); const after = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId)); for (const before of old) expect(after.find(decision => decision.id === before.id)).toEqual(before); expect(after.some(decision => decision.reasonCode === "scheduled_monitor_preserved")).toBe(true); expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0].resultJson).toEqual(f.result); expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toEqual([]); + expect(await f.readIssue()).toMatchObject({ monitorNextCheckAt: scheduled.monitorNextCheckAt, + executionRunId: null, checkoutRunId: null }); + await f.finalize(); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId))).toEqual(after); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].finishedAt).toEqual(f.finishedAt); }); it("notifies the existing monitor scheduler once when finalization stops blocking it", async () => { diff --git a/server/src/services/native-runtime/native-run-finalizer.ts b/server/src/services/native-runtime/native-run-finalizer.ts index 1f8b830ff03..5931662d7db 100644 --- a/server/src/services/native-runtime/native-run-finalizer.ts +++ b/server/src/services/native-runtime/native-run-finalizer.ts @@ -1488,6 +1488,7 @@ export async function finalizeNativeRun(input: { workspaceFinalizeStatus: input.workspaceFinalizeStatus, monitorDelivery, monitorWaitAt: assessment.continuation?.kind === "monitor" ? monitorWaitAt : null, + preservedMonitorWait, }); await input.db .update(nativeRunFinalizations) diff --git a/server/src/services/native-runtime/work-assessments.ts b/server/src/services/native-runtime/work-assessments.ts index 829b497040b..58975d83b6d 100644 --- a/server/src/services/native-runtime/work-assessments.ts +++ b/server/src/services/native-runtime/work-assessments.ts @@ -3,7 +3,7 @@ import type { Db } from "@paperclipai/db"; import { workAssessments } from "@paperclipai/db"; import type { NativeEvidenceAssessment } from "./evidence-classifier.js"; import { nativeSha256 } from "./canonical.js"; -import type { NativeMonitorDelivery } from "./native-monitor-delivery.js"; +import type { NativeMonitorDelivery, NativePreservedMonitorWait } from "./native-monitor-delivery.js"; export async function recordNativeWorkAssessment(input: { db: Db; @@ -26,12 +26,14 @@ export async function recordNativeWorkAssessment(input: { monitorDelivery?: NativeMonitorDelivery | null; /** A persisted schedule can regain settlement authority after a policy repair. */ monitorWaitAt?: string | null; + preservedMonitorWait?: NativePreservedMonitorWait | null; supersedesAssessmentId?: string | null; }) { const assessmentJson = { ...input.assessment, ...(input.monitorDelivery ? { monitorDelivery: input.monitorDelivery } : {}), ...(input.monitorWaitAt ? { monitorWaitAt: input.monitorWaitAt } : {}), + ...(input.preservedMonitorWait ? { preservedMonitorWait: input.preservedMonitorWait } : {}), } as unknown as Record; const inputDigest = nativeSha256({ contractId: input.contractId, From a8e63f0e2a9a5af43a2731a52601971307d4da94 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 17:42:54 -0700 Subject: [PATCH 56/68] perf(server): avoid repeated delivery scans in GitHub review repair (#17669) Resolve the exact delivery through its primary key before looking up stale-head cancellations. Backport the predicate from #17664 while preserving the current worker lifecycle and receipt checks. Verified 372 selected tests, workspace typechecking, the server build, green CI, and an Apex 5/5 review. Co-Authored-By: Paperclip --- server/src/services/chat-github-checks.ts | 21 +- .../chat-github-review-repair.test.ts | 183 ++++++++++++++++++ 2 files changed, 196 insertions(+), 8 deletions(-) create mode 100644 server/src/services/chat-github-review-repair.test.ts diff --git a/server/src/services/chat-github-checks.ts b/server/src/services/chat-github-checks.ts index dceb5ae3df5..a47cf3b1021 100644 --- a/server/src/services/chat-github-checks.ts +++ b/server/src/services/chat-github-checks.ts @@ -427,19 +427,24 @@ export function githubReviewCheckService(db: Db, fetchImpl = fetch) { and( inArray(chatGitHubReviews.state, ["queued", "running"]), isNull(chatGitHubReviews.assessment), - sql`exists ( - select 1 from ${chatActions} action - join ${chatDeliveries} delivery - on delivery.id::text = ${chatGitHubReviews.deliveryId} + sql`${chatGitHubReviews.deliveryId} ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'`, + sql`( + select action.id from ${chatActions} action where action.company_id = ${chatGitHubReviews.companyId} and action.endpoint_id = ${chatGitHubReviews.endpointId} - and delivery.endpoint_id = action.endpoint_id and action.kind = 'github_review_check' and action.status = 'cancelled' and action.result->>'code' = 'stale_head' - and action.payload->'event'->>'deliveryId' = - delivery.normalized_event->'githubAutomatic'->'context'->>'deliveryId' - )`, + and action.payload->'event'->>'deliveryId' = ( + select delivery.normalized_event->'githubAutomatic'->'context'->>'deliveryId' + from ${chatDeliveries} delivery + where delivery.id = case + when ${chatGitHubReviews.deliveryId} ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' + then ${chatGitHubReviews.deliveryId}::uuid end + and delivery.endpoint_id = ${chatGitHubReviews.endpointId} + ) + limit 1 + ) is not null`, ), ); const rows = await db diff --git a/server/src/services/chat-github-review-repair.test.ts b/server/src/services/chat-github-review-repair.test.ts new file mode 100644 index 00000000000..01d3649aa2c --- /dev/null +++ b/server/src/services/chat-github-review-repair.test.ts @@ -0,0 +1,183 @@ +import { randomUUID } from "node:crypto"; +import { sql } from "drizzle-orm"; +import { createDb, getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, type Db } from "@paperclipai/db"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { githubReviewCheckService } from "./chat-github-checks.js"; + +type QueryParameters = NonNullable[1]>; +type CapturedQuery = { text: string; parameters: QueryParameters }; +type Plan = { "Node Type"?: string; "Relation Name"?: string; "Index Name"?: string; "Actual Loops"?: number; Plans?: Plan[] }; +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("GitHub stale review repair PostgreSQL behavior", () => { + let temporary: Awaited>; + let db: Db; + let queries: CapturedQuery[]; + const noProviderCall = vi.fn(async () => { throw new Error("Repair must not contact a provider"); }); + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("github-review-repair-"); + const setup = createDb(temporary.connectionString, { maxConnections: 1 }); + try { + await setup.execute(sql`create schema github_review_repair_fixture`); + await setup.execute(sql`set search_path=github_review_repair_fixture`); + await setup.execute(sql`create table chat_github_reviews (id uuid primary key, company_id uuid not null, + endpoint_id uuid not null, delivery_id text not null, state text not null, assessment jsonb, + updated_at timestamptz not null default now())`); + await setup.execute(sql`create table chat_deliveries (id uuid primary key, endpoint_id uuid not null, normalized_event jsonb)`); + await setup.execute(sql`create table chat_actions (id uuid primary key, company_id uuid not null, + endpoint_id uuid not null, provider_action_id text, kind text not null, status text not null, + payload jsonb not null, result jsonb, updated_at timestamptz not null default now())`); + await setup.execute(sql`create unique index chat_actions_provider_action_uq + on chat_actions(company_id, endpoint_id, provider_action_id)`); + } finally { + await setup.$client.end(); + } + }, 90_000); + + beforeEach(async () => { + queries = []; + db = createDb(temporary.connectionString, { maxConnections: 1 }); + db.$client.options.debug = (_connection: number, text: string, parameters: QueryParameters) => { + if (/^update "chat_github_reviews"/i.test(text)) queries.push({ text, parameters: [...parameters] }); + }; + await db.execute(sql`set search_path=github_review_repair_fixture`); + await db.execute(sql`truncate chat_github_reviews, chat_deliveries, chat_actions`); + noProviderCall.mockClear(); + }); + afterEach(async () => { await db?.$client.end(); }); + afterAll(async () => { await temporary?.cleanup(); }); + + async function seed(overrides: Record = {}) { + const companyId = randomUUID(), endpointId = randomUUID(); + const deliveryId = "a" + randomUUID().slice(1), reviewId = randomUUID(), actionId = randomUUID(); + const providerDeliveryId = "provider-" + randomUUID(); + const values = { companyId, endpointId, deliveryId, reviewId, actionId, providerDeliveryId, + actionCompanyId: companyId, actionEndpointId: endpointId, deliveryEndpointId: endpointId, + actionDeliveryId: providerDeliveryId, reviewDeliveryId: deliveryId, state: "queued", assessment: null, + kind: "github_review_check", status: "cancelled", code: "stale_head", ...overrides }; + await db.execute(sql`insert into chat_deliveries(id, endpoint_id, normalized_event) + values (${deliveryId}::uuid, ${values.deliveryEndpointId}::uuid, + ${JSON.stringify({ githubAutomatic: { context: { deliveryId: providerDeliveryId } } })}::jsonb)`); + await db.execute(sql`insert into chat_actions(id, company_id, endpoint_id, kind, status, payload, result) + values (${actionId}::uuid, ${values.actionCompanyId}::uuid, ${values.actionEndpointId}::uuid, + ${values.kind}, ${values.status}, ${JSON.stringify({ event: { deliveryId: values.actionDeliveryId } })}::jsonb, + ${JSON.stringify({ code: values.code })}::jsonb)`); + await db.execute(sql`insert into chat_github_reviews(id, company_id, endpoint_id, delivery_id, state, assessment, updated_at) + values (${reviewId}::uuid, ${companyId}::uuid, ${endpointId}::uuid, ${values.reviewDeliveryId}, + ${values.state}, ${values.assessment === null ? null : JSON.stringify(values.assessment)}::jsonb, '2000-01-01T00:00:00Z')`); + return values; + } + + async function retainedRows() { + return { + reviews: await db.$client.unsafe("select * from chat_github_reviews order by id"), + deliveries: await db.$client.unsafe("select * from chat_deliveries order by id"), + actions: await db.$client.unsafe("select * from chat_actions order by id"), + }; + } + + it.each(["queued", "running"])("supersedes only the exact %s unassessed review", async state => { + const fixture = await seed({ state }); + const before = await retainedRows(); + const service = githubReviewCheckService(db, noProviderCall); + await service.processPending(); + const after = await retainedRows(); + expect(after.reviews).toHaveLength(1); + expect(after.reviews[0]).toMatchObject({ id: fixture.reviewId, state: "superseded", assessment: null }); + expect(after.deliveries).toEqual(before.deliveries); + expect(after.actions).toEqual(before.actions); + await service.processPending(); + expect(await retainedRows()).toEqual(after); + expect(noProviderCall).not.toHaveBeenCalled(); + }); + + it.each([ + ["another action company", { actionCompanyId: randomUUID() }], + ["another action endpoint", { actionEndpointId: randomUUID() }], + ["another delivery endpoint", { deliveryEndpointId: randomUUID() }], + ["another provider delivery", { actionDeliveryId: "another-delivery" }], + ["another cancellation reason", { code: "another_reason" }], + ["another action kind", { kind: "another_kind" }], + ["a processed action", { status: "processed" }], + ["an assessed review", { assessment: { retained: true } }], + ["a completed review", { state: "completed" }], + ])("preserves rows for %s", async (_name, overrides) => { + await seed(overrides as Record); + const before = await retainedRows(); + await githubReviewCheckService(db, noProviderCall).processPending(); + expect(await retainedRows()).toEqual(before); + expect(noProviderCall).not.toHaveBeenCalled(); + }); + + it.each(["malformed", "uppercase", "whitespace", "braces", "unhyphenated"])("preserves %s delivery references without cast errors", async kind => { + const fixture = await seed(); + const reference = kind === "malformed" ? "not-a-uuid" : kind === "uppercase" ? fixture.deliveryId.toUpperCase() + : kind === "whitespace" ? ` ${fixture.deliveryId} ` : kind === "braces" ? `{${fixture.deliveryId}}` + : fixture.deliveryId.replaceAll("-", ""); + await db.execute(sql`update chat_github_reviews set delivery_id=${reference} where id=${fixture.reviewId}::uuid`); + const before = await retainedRows(); + await githubReviewCheckService(db, noProviderCall).processPending(); + expect(await retainedRows()).toEqual(before); + expect(noProviderCall).not.toHaveBeenCalled(); + }); + + it.each(["missing delivery", "missing provider identity"])("preserves a review with %s", async kind => { + const fixture = await seed(); + if (kind === "missing delivery") { + await db.execute(sql`delete from chat_deliveries where id=${fixture.deliveryId}::uuid`); + } else { + await db.execute(sql`update chat_deliveries set normalized_event='{}'::jsonb where id=${fixture.deliveryId}::uuid`); + } + const before = await retainedRows(); + await githubReviewCheckService(db, noProviderCall).processPending(); + expect(await retainedRows()).toEqual(before); + expect(noProviderCall).not.toHaveBeenCalled(); + }); + + it("repairs a late review from the retained cancellation without replaying provider work", async () => { + const fixture = await seed(); + await db.execute(sql`delete from chat_github_reviews where id=${fixture.reviewId}::uuid`); + const service = githubReviewCheckService(db, noProviderCall); + await service.processPending(); + await db.execute(sql`insert into chat_github_reviews(id, company_id, endpoint_id, delivery_id, state) + values (${fixture.reviewId}::uuid, ${fixture.companyId}::uuid, ${fixture.endpointId}::uuid, ${fixture.deliveryId}, 'queued')`); + await service.processPending(); + const rows = await db.$client.unsafe("select state from chat_github_reviews"); + expect(rows).toEqual([{ state: "superseded" }]); + expect(noProviderCall).not.toHaveBeenCalled(); + }); + + it("resolves each candidate delivery once through its primary key with retained history", async () => { + const companyId = randomUUID(), endpointId = randomUUID(); + const candidates = 200; + await db.execute(sql`insert into chat_deliveries(id, endpoint_id, normalized_event) + select gen_random_uuid(), ${endpointId}::uuid, + jsonb_build_object('githubAutomatic', jsonb_build_object('context', jsonb_build_object('deliveryId', 'delivery-' || n))) + from generate_series(1,8000) n`); + await db.execute(sql`insert into chat_github_reviews(id, company_id, endpoint_id, delivery_id, state) + select gen_random_uuid(), ${companyId}::uuid, ${endpointId}::uuid, id::text, 'queued' + from chat_deliveries order by id limit ${candidates}`); + await db.execute(sql`insert into chat_actions(id, company_id, endpoint_id, provider_action_id, kind, status, payload, result) + select gen_random_uuid(), ${companyId}::uuid, ${endpointId}::uuid, 'action-' || n, + case when n<=100 then 'github_review_check' else 'another_kind' end, 'cancelled', + jsonb_build_object('event', jsonb_build_object('deliveryId', 'unmatched-' || n)), + jsonb_build_object('code', 'stale_head') from generate_series(1,1100) n`); + await db.execute(sql`analyze chat_deliveries`); + await db.execute(sql`analyze chat_actions`); + await db.execute(sql`analyze chat_github_reviews`); + await db.execute(sql`set statement_timeout='5s'`); + await githubReviewCheckService(db, noProviderCall).processPending(); + expect(queries).toHaveLength(1); + const query = queries[0]; + const explained = await db.$client.unsafe(`explain (analyze,buffers,format json) ${query.text}`, query.parameters); + const root = (explained[0]["QUERY PLAN"] as Array<{ Plan: Plan }>)[0].Plan; + function flatten(plan: Plan): Plan[] { return [plan, ...(plan.Plans ?? []).flatMap(flatten)]; } + const deliveries = flatten(root).filter(node => node["Relation Name"] === "chat_deliveries"); + expect(deliveries).toHaveLength(1); + expect(deliveries[0]["Index Name"]).toBe("chat_deliveries_pkey"); + expect(deliveries[0]["Actual Loops"]).toBeLessThanOrEqual(candidates); + expect((await db.$client.unsafe("select count(*)::int as count from chat_github_reviews where state='queued'"))[0].count).toBe(candidates); + expect(noProviderCall).not.toHaveBeenCalled(); + }, 30_000); +}); From 1a3f604097d81fb653b271af4c7c3614d2b032a7 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 25 Oct 2026 18:17:43 -0700 Subject: [PATCH 57/68] fix(db): honor the prepared-statement opt-in for ORM queries (#17673) Drizzle calls Postgres.js unsafe queries, which disable preparation unless per-query options enable it. The existing client-level opt-in therefore does not prepare ORM queries. Honor explicit true across root and child clients. Preserve defaults, per-query options, lazy execution, and driver error handling. Document the pooler capability requirement. Test with 92 selected database and connection checks, workspace typechecks, server and database builds, and a read-only hosted ORM protocol probe. Co-Authored-By: Paperclip --- doc/DATABASE.md | 12 +- docs/deploy/database.md | 11 +- packages/db/src/client.ts | 9 +- .../db/src/prepared-queries-postgres.test.ts | 175 ++++++++++++++++++ packages/db/src/prepared-queries.ts | 47 +++++ .../db/src/response-capture-postgres.test.ts | 20 +- 6 files changed, 259 insertions(+), 15 deletions(-) create mode 100644 packages/db/src/prepared-queries-postgres.test.ts create mode 100644 packages/db/src/prepared-queries.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index cb341f8ea79..2032268da2d 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -458,14 +458,14 @@ DATABASE_URL=postgres://postgres.[PROJECT-REF]:[PASSWORD]@aws-0-[REGION].pooler. DATABASE_MIGRATION_URL=postgres://postgres.[PROJECT-REF]:[PASSWORD]@aws-0-[REGION].pooler.supabase.com:5432/postgres ``` -If your hosted database requires transaction-pooling-only connections (pgbouncer transaction mode, Supavisor port 6543, Neon `-pooler` endpoints), set `DATABASE_PREPARED_STATEMENTS=false` so the client does not rely on session-scoped prepared statements, and keep `DATABASE_MIGRATION_URL` on a direct connection. Do not edit database client source files as part of deployment setup. +If your transaction-mode pooler does not support protocol-level named prepared statements, set `DATABASE_PREPARED_STATEMENTS=false`. If your pooler supports them, you can opt in with `DATABASE_PREPARED_STATEMENTS=true`. Check your provider's pooler configuration before you enable this setting. Keep `DATABASE_MIGRATION_URL` on a direct connection. Do not edit database client source files as part of deployment setup. ### Client tuning (optional) All of these are optional; when unset, the driver defaults apply and behavior is unchanged — typical self-hosted setups need none of them: ```sh -DATABASE_PREPARED_STATEMENTS=false # required for transaction-mode poolers; default: enabled +DATABASE_PREPARED_STATEMENTS=false # use false when the pooler does not support named prepared statements DATABASE_POOL_MAX=25 # connection pool size; default: 10 DATABASE_IDLE_TIMEOUT_SECONDS=60 # close idle pooled connections; default: 60 (0 = keep open) DATABASE_CONNECT_TIMEOUT_SECONDS=10 # default: 30 @@ -473,6 +473,14 @@ DATABASE_MAX_LIFETIME_SECONDS=1800 # recycle a pooled connection after this lo DATABASE_APPLICATION_NAME=paperclip # application_name in pg_stat_activity; default: paperclip ``` +An explicit `DATABASE_PREPARED_STATEMENTS=true` enables named prepared statements +for parameterized ORM queries, transactions, reserved connections, and dedicated +clients. Postgres.js manages the statement cache. This does not cache query +results. Bound values and returned data stay fresh. Explicit per-query options +still apply, including `prepare: false` and simple-protocol execution. With the +setting unset, ORM queries keep their existing unprepared behavior and tagged +SQL uses the driver default. With `false`, preparation stays disabled. + ### Bounded response diagnostics `PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW=/` enables one explicit UTC diff --git a/docs/deploy/database.md b/docs/deploy/database.md index 4f72b70c097..6e85cebce7e 100644 --- a/docs/deploy/database.md +++ b/docs/deploy/database.md @@ -56,12 +56,19 @@ For production, use a hosted provider like [Supabase](https://supabase.com/). Use the **direct connection** (port 5432) for migrations and the **pooled connection** (port 6543) for the application. -If using connection pooling (transaction mode), disable prepared statements via the environment — no source edits needed: +If your transaction-mode pooler does not support protocol-level named prepared statements, disable them via the environment: ```sh DATABASE_PREPARED_STATEMENTS=false ``` +For a direct connection or a pooler that supports named prepared statements, +`DATABASE_PREPARED_STATEMENTS=true` also prepares parameterized ORM queries and +queries inside transactions and reserved or dedicated connections. Check the +provider's configuration before you enable it. Preparation caches statement +metadata, not query results. Unset preserves the existing ORM behavior and the +driver default for tagged SQL. Explicit per-query options still apply. + Related optional client tuning: `DATABASE_POOL_MAX`, `DATABASE_IDLE_TIMEOUT_SECONDS`, `DATABASE_CONNECT_TIMEOUT_SECONDS`, `DATABASE_MAX_LIFETIME_SECONDS`, `DATABASE_APPLICATION_NAME`. Driver defaults apply when unset, except that idle pooled connections close after 60 seconds (`DATABASE_IDLE_TIMEOUT_SECONDS=0` keeps them open) and the pool reports `application_name=paperclip`. See [Connection pool settings](#connection-pool-settings). ## Connection Pool Settings @@ -75,7 +82,7 @@ The server opens one postgres.js pool for its own queries (and a second one when | `DATABASE_CONNECT_TIMEOUT_SECONDS` | `30` (driver) | Give up on a connection attempt after this long. | | `DATABASE_MAX_LIFETIME_SECONDS` | 30–60 min, randomized (driver) | Recycle a pooled connection once it is this old. | | `DATABASE_APPLICATION_NAME` | `paperclip` | Value of `application_name` in `pg_stat_activity`, so you can find Paperclip's backends: `SELECT * FROM pg_stat_activity WHERE application_name = 'paperclip';` | -| `DATABASE_PREPARED_STATEMENTS` | `true` (driver) | Set `false` behind a transaction-mode pooler (see above). | +| `DATABASE_PREPARED_STATEMENTS` | Unset | `true` opts parameterized ORM queries into preparation. `false` disables preparation. Use `false` when the pooler does not support named statements. Unset preserves existing behavior. | The server ends its pools during shutdown (SIGINT/SIGTERM) and when startup fails after the pool was opened, so a restarting server does not leave idle backends behind. Size `max_connections` on the PostgreSQL side for at least `DATABASE_POOL_MAX` per server process plus your other clients. Each server process also opens up to 2 connections (`application_name=paperclip-controller-lease`) for agent-run lease renewals, so they never queue behind slow application queries; count those too. diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index f1b8079db45..7f067cbfc1c 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -5,6 +5,7 @@ import { fileURLToPath } from "node:url"; import postgres from "postgres"; import { installDatabaseWorkSignals } from "./work-signals.js"; import { installDatabaseResponseCapture } from "./response-capture.js"; +import { installPreparedQueries } from "./prepared-queries.js"; import * as schema from "./schema/index.js"; const MIGRATIONS_FOLDER = fileURLToPath(new URL("./migrations", import.meta.url)); @@ -129,13 +130,6 @@ export type MigrationState = }; export interface DatabaseClientOptions { - /** - * postgres.js `prepare`. Set false when connecting through a - * transaction-mode pooler (pgbouncer / Neon `-pooler` endpoints / - * Supabase Supavisor transaction ports) so the client does not rely on - * session-scoped prepared statements. Defaults to the driver default - * (enabled), preserving existing behavior on direct connections. - */ prepare?: boolean; /** postgres.js `max` — connection pool size (driver default: 10). */ maxConnections?: number; @@ -301,6 +295,7 @@ export function createDb(url: string, options?: DatabaseClientOptions) { function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: object) { const sql = postgres(url, postgresJsOptions(resolved)); + if (resolved.prepare === true) installPreparedQueries(sql); const key = hostPortKeyOrNull(url); if (key) registerClient(key, sql); // A disconnect can lose the response after a statement has committed. diff --git a/packages/db/src/prepared-queries-postgres.test.ts b/packages/db/src/prepared-queries-postgres.test.ts new file mode 100644 index 00000000000..edef6dc060e --- /dev/null +++ b/packages/db/src/prepared-queries-postgres.test.ts @@ -0,0 +1,175 @@ +import { eq, sql } from "drizzle-orm"; +import { integer, jsonb, pgTable, text } from "drizzle-orm/pg-core"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest"; +import { closeLivenessDb, createDb, databaseClientOptionsFromEnv, livenessDb, withDedicatedDbConnection, type DatabaseClientOptions, type Db } from "./client.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js"; +import { runEventPayload } from "./run-event-payload.js"; + +const records = pgTable("prepared_query_fixture", { id: text("id").primaryKey(), companyId: text("company_id").notNull(), value: integer("value").notNull(), payload: jsonb("payload") }); +const encoded = pgTable("prepared_encoded_fixture", { id: text("id").primaryKey(), payload: runEventPayload("payload") }); +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("explicit ORM prepared statements on PostgreSQL", () => { + let temporary: Awaited>; + let setup: Db; + const clients: Db[] = []; + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("prepared-orm-"); + setup = createDb(temporary.connectionString, { prepare: false, maxConnections: 1 }); + await setup.execute(sql`create table prepared_query_fixture(id text primary key, company_id text not null, value integer not null, payload jsonb)`); + await setup.execute(sql`create table prepared_encoded_fixture(id text primary key, payload jsonb)`); + await setup.execute(sql`create table prepared_schema_change_fixture(id text primary key)`); + await setup.execute(sql`insert into prepared_schema_change_fixture(id) values ('retained')`); + }, 90_000); + beforeEach(async () => { + await setup.execute(sql`truncate prepared_query_fixture, prepared_encoded_fixture`); + await setup.insert(records).values([ + { id: "a", companyId: "company-a", value: 1, payload: { owner: "a" } }, + { id: "b", companyId: "company-b", value: 2, payload: { owner: "b" } }, + ]); + }); + afterEach(async () => { + for (const db of clients.splice(0)) { + await closeLivenessDb(db, 2); + await db.$client.end(); + } + }); + afterAll(async () => { await setup?.$client.end(); await temporary?.cleanup(); }); + function fixture(options: DatabaseClientOptions = {}) { + const db = createDb(temporary.connectionString, { ...options, maxConnections: 1 }); + clients.push(db); return db; + } + async function statements(db: Db, pattern = '%"prepared_query_fixture"%') { + return db.$client.unsafe("select name,statement from pg_prepared_statements where statement like $1", [pattern]); + } + + it.each([undefined, false])("keeps the existing ORM mode when prepare is %s", async prepare => { + const db = fixture({ prepare }); + expect(await db.select().from(records).where(eq(records.companyId, "company-a"))).toEqual([ + { id: "a", companyId: "company-a", value: 1, payload: { owner: "a" } }, + ]); + expect(await statements(db)).toHaveLength(0); + }); + + it("honors the explicit environment opt-in for ORM queries with fresh company bindings", async () => { + const db = fixture(databaseClientOptionsFromEnv({ DATABASE_PREPARED_STATEMENTS: "true" })); + for (let i = 0; i < 12; i++) { + const companyId = i % 2 ? "company-b" : "company-a"; + const rows = await db.select().from(records).where(eq(records.companyId, companyId)); + expect(rows).toEqual([{ id: i % 2 ? "b" : "a", companyId, value: i % 2 ? 2 : 1, payload: { owner: i % 2 ? "b" : "a" } }]); + } + expect(await statements(db)).toHaveLength(1); + }); + + it("preserves explicit per-query opt-out", async () => { + const db = fixture({ prepare: true }); + expect((await db.$client.unsafe("select $1::integer as explicit_opt_out_marker", [7], { prepare: false }))[0].explicit_opt_out_marker).toBe(7); + expect(await statements(db, "%explicit_opt_out_marker%")).toHaveLength(0); + }); + + it("preserves simple queries and the two-argument options overload", async () => { + const db = fixture({ prepare: true }); + const simpleOptions = { prepare: true, simple: true }; + const simple = await db.$client.unsafe("select 1 as first; select 2 as second", [], simpleOptions); + expect(simple).toEqual([[{ first: 1 }], [{ second: 2 }]]); + const unsafe = db.$client.unsafe as unknown as (query: string, options: { simple: boolean }) => Promise; + expect(await unsafe("select 3 as third; select 4 as fourth", { simple: true })) + .toEqual([[{ third: 3 }], [{ fourth: 4 }]]); + }); + + it("does not replay a rejected prepared write", async () => { + const db = fixture({ prepare: true }); + await expect(db.insert(records).values({ id: "a", companyId: "company-b", value: 9 })) + .rejects.toMatchObject({ cause: { code: "23505" } }); + expect(await db.select({ companyId: records.companyId, value: records.value }).from(records).where(eq(records.id, "a"))) + .toEqual([{ companyId: "company-a", value: 1 }]); + }); + + it("keeps write queries lazy and repeated awaits from repeating writes", async () => { + const db = fixture({ prepare: true }); + const pending = db.$client.unsafe("insert into prepared_query_fixture(id,company_id,value) values ($1,$2,$3) returning id", ["lazy", "company-a", 3]); + expect(typeof pending.values).toBe("function"); + expect(await setup.select().from(records).where(eq(records.id, "lazy"))).toEqual([]); + expect(await pending).toEqual([{ id: "lazy" }]); + expect(await pending).toEqual([{ id: "lazy" }]); + expect(await setup.select({ id: records.id }).from(records).where(eq(records.id, "lazy"))).toEqual([{ id: "lazy" }]); + expect(await statements(db, "%insert into prepared_query_fixture%")).toHaveLength(1); + }); + + it("preserves transaction commits, nested savepoint rollback, and scoped reads", async () => { + const db = fixture({ prepare: true }); + await db.transaction(async tx => { + expect(await tx.select({ value: records.value }).from(records).where(eq(records.companyId, "company-a"))).toEqual([{ value: 1 }]); + await expect(tx.transaction(async nested => { + await nested.update(records).set({ value: 99 }).where(eq(records.companyId, "company-a")); + throw new Error("rollback savepoint"); + })).rejects.toThrow("rollback savepoint"); + expect(await tx.select({ value: records.value }).from(records).where(eq(records.companyId, "company-a"))).toEqual([{ value: 1 }]); + await tx.insert(records).values({ id: "committed", companyId: "company-b", value: 3 }); + const names = await tx.execute(sql`select statement from pg_prepared_statements`); + expect(names.some(row => String(row.statement).includes('"prepared_query_fixture"'))).toBe(true); + }, { isolationLevel: "serializable" }); + expect(await db.select({ value: records.value }).from(records).where(eq(records.id, "committed"))).toEqual([{ value: 3 }]); + }); + + it("rolls back failed outer transactions without replay", async () => { + const db = fixture({ prepare: true }); + await expect(db.transaction(async tx => { + await tx.update(records).set({ value: 42 }).where(eq(records.id, "a")); + throw new Error("rollback outer transaction"); + })).rejects.toThrow("rollback outer transaction"); + expect(await db.select({ value: records.value }).from(records).where(eq(records.id, "a"))).toEqual([{ value: 1 }]); + }); + + it("prepares reserved-client queries and keeps their bound values fresh", async () => { + const db = fixture({ prepare: true }); + const reserved = await db.$client.reserve(); + try { + for (const marker of [1, 2, 3]) { + expect((await reserved.unsafe("select $1::integer as reserved_prepared_marker", [marker]))[0].reserved_prepared_marker).toBe(marker); + } + const names = await reserved.unsafe("select statement from pg_prepared_statements where statement like $1", ["%reserved_prepared_marker%"]); + expect(names).toHaveLength(1); + } finally { reserved.release(); } + }); + + it("preserves object and array results for the same prepared shape", async () => { + const db = fixture({ prepare: true }); + expect(await db.$client.unsafe("select $1::integer as result_mode_marker", [41]).values()).toEqual([[41]]); + expect(await db.$client.unsafe("select $1::integer as result_mode_marker", [42])).toEqual([{ result_mode_marker: 42 }]); + }); + + it("keeps custom stored-payload decoding fresh", async () => { + const db = fixture({ prepare: true }); + const first = { text: "encoded first payload🙂".repeat(2048) }; + const second = { text: "encoded second payload🙂".repeat(2048) }; + await setup.insert(encoded).values({ id: "encoded", payload: first }); + expect((await db.select().from(encoded).where(eq(encoded.id, "encoded")))[0].payload).toEqual(first); + await setup.update(encoded).set({ payload: second }).where(eq(encoded.id, "encoded")); + expect((await db.select().from(encoded).where(eq(encoded.id, "encoded")))[0].payload).toEqual(second); + expect(await statements(db, '%"prepared_encoded_fixture"%')).toHaveLength(1); + }); + + it("inherits the opt-in in dedicated and liveness clients", async () => { + const db = fixture({ prepare: true }); + await withDedicatedDbConnection(db, async dedicated => { + expect(await dedicated.select({ value: records.value }).from(records).where(eq(records.companyId, "company-b"))).toEqual([{ value: 2 }]); + expect(await statements(dedicated)).toHaveLength(1); + }); + const live = livenessDb(db); + expect(await live.select({ value: records.value }).from(records).where(eq(records.companyId, "company-a"))).toEqual([{ value: 1 }]); + expect(await statements(live)).toHaveLength(1); + }); + + it("refreshes prepared result metadata after a schema change", async () => { + const db = fixture({ prepare: true }); + const query = "select * from prepared_schema_change_fixture where id=$1"; + expect(await db.$client.unsafe(query, ["retained"])).toEqual([{ id: "retained" }]); + try { + await setup.execute(sql`alter table prepared_schema_change_fixture add column extra text`); + expect(await db.$client.unsafe(query, ["retained"])).toEqual([{ id: "retained", extra: null }]); + } finally { + await setup.execute(sql`alter table prepared_schema_change_fixture drop column if exists extra`); + } + }); +}); diff --git a/packages/db/src/prepared-queries.ts b/packages/db/src/prepared-queries.ts new file mode 100644 index 00000000000..7622722027a --- /dev/null +++ b/packages/db/src/prepared-queries.ts @@ -0,0 +1,47 @@ +type Method = (this: unknown, ...args: unknown[]) => unknown; + +export function installPreparedQueries(client: unknown): void { + const visited = new WeakSet(); + + function visit(value: unknown): void { + if ((typeof value !== "object" || value === null) && typeof value !== "function") return; + if (visited.has(value)) return; + visited.add(value); + const target = value as Record; + const unsafe = target.unsafe; + if (typeof unsafe === "function") { + target.unsafe = function (this: unknown, ...args: unknown[]) { + const index = args.length === 2 && !Array.isArray(args[1]) ? 1 : 2; + const options = args[index]; + if (options !== undefined && (options === null || typeof options !== "object")) { + return Reflect.apply(unsafe, this, args); + } + if (index === 2 && args.length < 2) args[1] = []; + args[index] = { prepare: true, ...options }; + return Reflect.apply(unsafe, this, args); + }; + } + for (const method of ["begin", "savepoint"]) { + const original = target[method]; + if (typeof original !== "function") continue; + target[method] = function (this: unknown, ...args: unknown[]) { + const wrapped = args.map(arg => typeof arg !== "function" ? arg : function (this: unknown, ...inner: unknown[]) { + visit(inner[0]); + return Reflect.apply(arg, this, inner); + }); + return Reflect.apply(original, this, wrapped); + } satisfies Method; + } + const reserve = target.reserve; + if (typeof reserve === "function") { + target.reserve = function (this: unknown, ...args: unknown[]) { + return Promise.resolve(Reflect.apply(reserve, this, args)).then(reserved => { + visit(reserved); + return reserved; + }); + } satisfies Method; + } + } + + visit(client); +} diff --git a/packages/db/src/response-capture-postgres.test.ts b/packages/db/src/response-capture-postgres.test.ts index 96de91ff913..42d5d805a0c 100644 --- a/packages/db/src/response-capture-postgres.test.ts +++ b/packages/db/src/response-capture-postgres.test.ts @@ -31,11 +31,11 @@ describe.skipIf(!support.supported)("database response capture PostgreSQL behavi vi.restoreAllMocks(); vi.unstubAllEnvs(); }); afterAll(async () => { await temporary?.cleanup(); }); - function fixture() { + function fixture(prepare?: boolean) { const events: Array> = []; const now = Date.now(); const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); - const db = createDb(temporary.connectionString); + const db = createDb(temporary.connectionString, { prepare }); clients.push(db); captures.push(capture); expect(installDatabaseResponseCapture(db, { table, encodedTable }, capture)).toBe(db); expect(installDatabaseResponseCapture(db, { table, encodedTable }, capture)).toBe(db); @@ -43,6 +43,18 @@ describe.skipIf(!support.supported)("database response capture PostgreSQL behavi } function summary(events: Array>) { return events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; } + it("retains capture counts when ORM preparation is enabled", async () => { + const f = fixture(true), id = randomUUID(), payload = "prepared-capture-value"; + await f.db.insert(table).values({ id, payload }); + await f.db.transaction(async tx => { + expect(await tx.select().from(table).where(eq(table.id, id))).toEqual([{ id, payload }]); + }); + f.capture.flush(); + expect(summary(f.events)).toMatchObject({ startedQueries: 2, completedQueries: 2 }); + expect(JSON.stringify(f.events)).not.toContain(payload); + expect(JSON.stringify(f.events)).not.toContain(id); + }); + it("keeps rows and builder identity through lazy select and write returning", async () => { const f = fixture(), id = randomUUID(), payload = "sentinel-result-and-parameter-value🙂\n\""; const insert = f.db.insert(table).values({ id, payload }); @@ -173,11 +185,11 @@ describe.skipIf(!support.supported)("database response capture PostgreSQL behavi f.capture.flush(); expect(summary(f.events).completedQueries).toBeGreaterThan(4); }); - it("covers automatically created dedicated pools and retains owner identity", async () => { + it.each([undefined, true])("covers automatically created dedicated pools with prepare %s and retains owner identity", async prepare => { const logs: string[] = [], now = Date.now(); vi.spyOn(console, "info").mockImplementation(value => logs.push(String(value))); vi.stubEnv("PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW", `${new Date(now - 100).toISOString()}/${new Date(now + 59_000).toISOString()}`); - const db = createDb(temporary.connectionString); clients.push(db); + const db = createDb(temporary.connectionString, { prepare }); clients.push(db); const capture = databaseResponseCaptureFromEnv()!; captures.push(capture); expect(await withDedicatedDbConnection(db, async dedicated => dedicated.execute(sql`select ${"dedicated-value"}::text as value`))) .toEqual([{ value: "dedicated-value" }]); From 5cea9c7c0eb176dd8d053f4606230fea165f9d64 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 25 Oct 2026 21:49:52 -0500 Subject: [PATCH 58/68] fix(ui): show eligible wake targets when resuming tasks (#17521) ## Thinking Path > - Paperclip manages agents and their assigned tasks. > - A task pause controls execution without changing task status. > - Resume can remove that pause and request work from eligible assignees. > - The preview counted blocked tasks and former executors as wake targets. > - The release route correctly skipped those targets, so the dialog promised work that did not start. > - Resume also trusted stale pause snapshots after tasks were reopened. > - This PR aligns preview with execution eligibility and rechecks current task state when releasing a pause. ## Linked Issues or Issue Description **What happened?** Pause an assigned blocked task, then open Resume. The dialog says that one task will resume and offers to wake one agent. Applying it only removes the pause. The route skips blocked and backlog tasks. A former executor can also inflate the wake count after reassignment. **Expected behavior** The dialog must distinguish removing a pause from starting work. Its wake count must use the current assignees and statuses accepted by the release route. Blocked work must stay blocked. **Steps to reproduce** 1. Open an assigned blocked task. 2. Choose Pause work, then Resume task. 3. Inspect the wake count and apply Resume work. Related runtime recovery work is tracked separately in #17097 and #17439. This PR does not change remote execution recovery. ## What Changed - Share the resume status list between preview and release. Count only current assignees on todo, in-progress, and in-review tasks for resume wakes. - Explain that Resume removes the pause. Show the existing preview warnings, including tasks that remain idle. - Disable and uncheck the wake option when no agents qualify. Send the same choice to the server. - Reevaluate previously skipped pause members after reopening or releasing another hold. Retain active-pause and unsafe-execution guards. - Add real PostgreSQL and HTTP-route regressions plus browser coverage for both task interfaces. Keep draft, ancestor-pause, and Stop/reconciliation coverage. - Update the task coordination maps. Add an opt-in dedicated test database URL for hosts where embedded PostgreSQL cannot start. ## Verification - Final head `9548eb16736f632cc111a5c13a9a994ec1d95a13`: all 55 completed checks pass, with two optional Storybook checks skipped. [CI 40188094904](https://github.com/paperclipai/paperclip/actions/runs/40188094904) passes, including typecheck, build, all general/serialized suites, and all eight E2E shards. Greptile is 5/5 on this commit; the reopened-task finding is fixed and resolved. - Red: the real PostgreSQL regression receives three agents instead of the one eligible current assignee on the original code. - Green: all 27 tree-control service, route, and unit tests pass. They cover current assignees, blocked/backlog exclusions, reopened tasks, company boundaries, remaining pause holds, and execution blockers. - The reopened-task route regression fails on the previous implementation: skipped members never reach preflight/wake, and an unsafe replay releases the hold instead of returning 409. All cases pass with the fix. - Browser: all five applicable cases pass together (90 seconds): blocked-work resume in both interfaces, draft/ancestor-pause cases in both interfaces, and the process Stop/pause/reconciliation/restart/cancel lifecycle. The optional native-runner fixture is skipped locally because its fixture binary is not configured. - All 157 task-page component tests pass on the prior commit; subsequent changes affect the server route and Stop fixture only. - `pnpm -r typecheck` and `pnpm build` pass before rebase, and CI typecheck/build passed on the previous PR commit. Server typecheck passes again after the route fix. The final-head CI run also passes these full checks. - `node scripts/check-maps.mjs`, `node scripts/check-token-gates.mjs`, and `git diff --check` pass. - Full `pnpm test:run` was attempted but did not complete: this host cannot initialize embedded PostgreSQL, and unrelated workspace-runtime tests timed out. The focused database tests and E2E suite use disposable PostgreSQL 18 instead. All general-test and serialized-test CI shards passed on the previous PR commit; the final-head CI run also passes every test shard. - The first CI run caught the existing Stop test trying to check a disabled wake option for blocked legacy work. It now asserts zero wake targets and retains the no-execution-before-reconciliation checks. - Earlier E2E attempts exposed fixture problems: assignment started the fixture before it was paused, and the classic interface had two matching editor nodes. The fixture now enables dispatch after the hold exists and scopes the final assertion to the task composer. No product assertion was removed. ## Risks Resume preview is a snapshot. The release route still checks current status, assignment, and execution blockers. It does not bypass approval, budget, or recovery gates. Pause, cancel, and restore keep their existing affected-agent semantics. The test database override only accepts a dedicated database name ending in `_test`; it runs migrations before the suite and closes its clients afterward. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, browser control, and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- maps/agent-access-map/task-coordination.md | 6 +- maps/feature-map/task-coordination.md | 7 ++ .../issue-tree-control-routes.test.ts | 36 ++++++++++- .../issue-tree-control-service.test.ts | 64 ++++++++++++++++++- server/src/routes/issue-tree-control.ts | 14 ++-- server/src/services/issue-tree-control.ts | 26 +++++++- tests/e2e/composer-stop.spec.ts | 11 +++- tests/e2e/paused-composer.spec.ts | 50 ++++++++++++++- ui/src/components/TaskTreeControls.tsx | 11 +++- ui/src/pages/IssueDetail.tsx | 3 +- 10 files changed, 209 insertions(+), 19 deletions(-) diff --git a/maps/agent-access-map/task-coordination.md b/maps/agent-access-map/task-coordination.md index a9bd1ddcc56..dda7994a89a 100644 --- a/maps/agent-access-map/task-coordination.md +++ b/maps/agent-access-map/task-coordination.md @@ -27,7 +27,11 @@ API: `present` — representative operation IDs: Agent access: `available` — Native create_task/reassign_task; REST / CLI / MCP. Policy: Task visibility/mutation, task-assignment and ownership policy; review audience -independent; tree controls board-only. +independent; tree controls board-only. Resume preview counts only current agent +assignees on todo, in-progress, and in-review tasks as wake targets. Release keeps +blocked and backlog statuses intact; existing execution gates still apply. +Reopened tasks use current eligibility rather than the pause snapshot's skipped +flag. Other active pause holds and execution reconciliation gates remain effective. Evidence: `source-only` — [implementation](../../server/src/routes/issue-tree-control.ts#L95) at the reviewed diff --git a/maps/feature-map/task-coordination.md b/maps/feature-map/task-coordination.md index 86a60dc5611..67f58c06244 100644 --- a/maps/feature-map/task-coordination.md +++ b/maps/feature-map/task-coordination.md @@ -49,6 +49,13 @@ Automated: [execution policy](../../server/src/__tests__/issue-execution-policy. Manual: Complete implementation under a review-required policy and confirm it waits for the required review outcome. Exercise rejection and later approval. Pause the parent while child work is pending, then verify resumption respects budget and ownership gates. +Resume removes the pause without changing task status. Its wake count includes only +current agent assignees on todo, in-progress, or in-review tasks. The dialog explains +that blocked, backlog, and unassigned tasks remain idle. Verify this in both task +interfaces with [the paused composer browser suite](../../tests/e2e/paused-composer.spec.ts). +Release rechecks tasks reopened after the pause; another active pause still prevents +their execution, and uncertain prior executions must be reconciled before waking. + ## Gotchas - Review completion and task completion are different events under some policies. diff --git a/server/src/__tests__/issue-tree-control-routes.test.ts b/server/src/__tests__/issue-tree-control-routes.test.ts index 7931bb768ec..682650b7c5d 100644 --- a/server/src/__tests__/issue-tree-control-routes.test.ts +++ b/server/src/__tests__/issue-tree-control-routes.test.ts @@ -14,6 +14,7 @@ const mockTreeControlService = vi.hoisted(() => ({ cancelIssueStatusesForHold: vi.fn(), restoreIssueStatusesForHold: vi.fn(), getHold: vi.fn(), + getActivePauseHoldGate: vi.fn(), releaseHold: vi.fn(), cancelUnclaimedWakeupsForTree: vi.fn(), })); @@ -72,6 +73,7 @@ describe("issue tree control routes", () => { mockReplayBlocks.mockResolvedValue([]); mockExecutionBlocker.mockResolvedValue(null); mockTreeControlService.getHold.mockResolvedValue(null); + mockTreeControlService.getActivePauseHoldGate.mockResolvedValue(null); mockIssueService.getById.mockResolvedValue({ id: "11111111-1111-4111-8111-111111111111", companyId: "company-2", @@ -213,6 +215,36 @@ describe("issue tree control routes", () => { }, ); + it.each([ + { skipReason: "terminal_status", stillPaused: false }, + { skipReason: "already_held", stillPaused: false }, + { skipReason: "already_held", stillPaused: true }, + ])("rechecks skipped pause members with $skipReason and remaining pause=$stillPaused", async ({ skipReason, stillPaused }) => { + const rootId = "11111111-1111-4111-8111-111111111111"; + const holdId = "33333333-3333-4333-8333-333333333333"; + const childId = "44444444-4444-4444-8444-444444444444"; + const root = { id: rootId, companyId: "company-2", status: "todo", assigneeAgentId: null }; + const child = { id: childId, companyId: "company-2", status: "todo", assigneeAgentId: "current-agent" }; + mockIssueService.getById.mockImplementation(async (id: string) => id === rootId ? root : child); + const hold = { id: holdId, rootIssueId: rootId, mode: "pause", members: [ + { issueId: childId, skipped: true, skipReason, issueStatus: "done" }, + ] }; + mockTreeControlService.getHold.mockResolvedValue(hold); + mockTreeControlService.releaseHold.mockResolvedValue({ ...hold, status: "released" }); + mockTreeControlService.getActivePauseHoldGate.mockResolvedValue(stillPaused ? { holdId: "remaining-hold" } : null); + mockReplayBlocks.mockResolvedValue([{ id: childId, identifier: "TEST-2" }]); + const app = await createApp({ type: "board", userId: "user-1", companyIds: ["company-2"], source: "session" }); + const response = await request(app) + .post(`/api/issues/${rootId}/tree-holds/${holdId}/release`) + .send({ metadata: { wakeAgents: true } }); + expect(response.status).toBe(200); + expect(mockExecutionBlocker).toHaveBeenCalledWith(expect.anything(), "company-2", childId); + expect(mockHeartbeatService.wakeup).toHaveBeenCalledTimes(stillPaused ? 0 : 1); + if (!stillPaused) expect(mockHeartbeatService.wakeup).toHaveBeenCalledWith("current-agent", expect.objectContaining({ + payload: expect.objectContaining({ issueId: childId }), + })); + }); + it("resumes sandbox-backed work through the shared plugin worker manager", async () => { const rootId = "11111111-1111-4111-8111-111111111111"; const holdId = "33333333-3333-4333-8333-333333333333"; @@ -284,14 +316,14 @@ describe("issue tree control routes", () => { ).toEqual([rootId, "child"]); }); - it("keeps the hold active when resume requests an unsafe execution replay", async () => { + it.each([false, true])("keeps the hold active when resume requests an unsafe execution replay (previously skipped=%s)", async (skipped) => { const rootId = "11111111-1111-4111-8111-111111111111"; const holdId = "33333333-3333-4333-8333-333333333333"; mockTreeControlService.getHold.mockResolvedValue({ id: holdId, rootIssueId: rootId, mode: "pause", - members: [{ issueId: rootId }], + members: [{ issueId: rootId, skipped, skipReason: skipped ? "terminal_status" : null }], }); mockReplayBlocks.mockResolvedValue([{ id: rootId, identifier: "TEST-1" }]); mockExecutionBlocker.mockResolvedValue({ nextAction: "Resume without waking agents until the previous execution stops.", diff --git a/server/src/__tests__/issue-tree-control-service.test.ts b/server/src/__tests__/issue-tree-control-service.test.ts index 43c620053a5..923f399d3bd 100644 --- a/server/src/__tests__/issue-tree-control-service.test.ts +++ b/server/src/__tests__/issue-tree-control-service.test.ts @@ -6,6 +6,8 @@ import { agents, agentWakeupRequests, companies, + applyPendingMigrations, + closeRegisteredClients, createDb, heartbeatRuns, issueComments, @@ -20,8 +22,10 @@ import { } from "./helpers/embedded-postgres.js"; import { issueTreeControlService } from "../services/issue-tree-control.js"; import { issueService } from "../services/issues.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ISSUE_TREE_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; if (!embeddedPostgresSupport.supported) { @@ -35,6 +39,11 @@ describeEmbeddedPostgres("issueTreeControlService", () => { let tempDb: Awaited> | null = null; beforeAll(async () => { + if (externalUrl) { + await applyPendingMigrations(externalUrl); + db = createDb(externalUrl); + return; + } tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issue-tree-control-"); db = createDb(tempDb.connectionString); }, 20_000); @@ -53,6 +62,7 @@ describeEmbeddedPostgres("issueTreeControlService", () => { }); afterAll(async () => { + if (externalUrl) await closeRegisteredClients(externalUrl); await tempDb?.cleanup(); }); @@ -623,6 +633,58 @@ describeEmbeddedPostgres("issueTreeControlService", () => { expect(checkedOutLegacyFullPauseRoot.checkoutRunId).toBe(rootRunId); }); + it("previews only executable current assignees as resume wake targets", async () => { + const companyId = randomUUID(); + const rootId = randomUUID(); + const readyAgentId = randomUUID(); + const parkedAgentId = randomUUID(); + const previousAgentId = randomUUID(); + const runId = randomUUID(); + await db.insert(companies).values({ + id: companyId, name: "Resume preview", issuePrefix: "RESUME", + }); + await db.insert(agents).values([readyAgentId, parkedAgentId, previousAgentId].map((id) => ({ + id, companyId, name: id, role: "engineer", status: "idle", adapterType: "process", + }))); + const children = ["todo", "in_progress", "in_review", "blocked", "backlog", "done"].map((status) => ({ + id: randomUUID(), companyId, parentId: rootId, title: status, status, + assigneeAgentId: status === "blocked" || status === "backlog" ? parkedAgentId : readyAgentId, + })); + await db.insert(issues).values([ + { id: rootId, companyId, title: "Unassigned root", status: "todo" }, + ...children, + ]); + const svc = issueTreeControlService(db); + const pause = await svc.createHold(companyId, rootId, { + mode: "pause", + actor: { actorType: "user", actorId: "board-user", userId: "board-user" }, + }); + expect(pause.hold.members?.find((member) => member.issueId === children[5].id)).toMatchObject({ + skipped: true, skipReason: "terminal_status", + }); + await db.update(issues).set({ status: "todo" }).where(eq(issues.id, children[5].id)); + // A previous executor is observable while stopping, but is not the current assignee to wake. + await db.insert(heartbeatRuns).values({ + id: runId, companyId, agentId: previousAgentId, invocationSource: "assignment", + status: "running", contextSnapshot: { issueId: children[0].id }, + }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, children[0].id)); + + const preview = await svc.preview(companyId, rootId, { mode: "resume" }); + expect(preview.totals.affectedIssues).toBe(7); + expect(preview.affectedAgents).toEqual([{ agentId: readyAgentId, issueCount: 4, activeRunCount: 0 }]); + expect(preview.totals.affectedAgents).toBe(1); + expect(preview.warnings).toContainEqual({ + code: "resume_tasks_remain_idle", + message: "Blocked, backlog, and unassigned tasks will be unpaused but will not start.", + issueIds: [rootId, children[3].id, children[4].id].sort(), + }); + const rows = await db.select().from(issues).where(eq(issues.companyId, companyId)); + expect(rows.find((row) => row.id === children[3].id)?.status).toBe("blocked"); + expect(rows.find((row) => row.id === children[4].id)?.status).toBe("backlog"); + expect((await svc.getHold(companyId, pause.hold.id))?.status).toBe("active"); + }); + it("resumes subtree pauses by releasing matching pause holds", async () => { const companyId = randomUUID(); const rootIssueId = randomUUID(); diff --git a/server/src/routes/issue-tree-control.ts b/server/src/routes/issue-tree-control.ts index c9a00491ba3..05c208756e9 100644 --- a/server/src/routes/issue-tree-control.ts +++ b/server/src/routes/issue-tree-control.ts @@ -6,6 +6,7 @@ import { } from "@paperclipai/db"; import { and, eq, inArray, isNotNull } from "drizzle-orm"; import { getExecutionBlocker } from "../services/execution-blocker.js"; +import { ISSUE_TREE_RESUME_EXECUTABLE_STATUSES } from "../services/issue-tree-control.js"; import { conflict } from "../errors.js"; import { createIssueTreeHoldSchema, @@ -26,7 +27,6 @@ import { assertBoard, getAccessibleResource, getActorInfo } from "./authz.js"; import type { PluginWorkerManager } from "../services/plugin-worker-manager.js"; const TREE_RUN_CANCELLATION_RESPONSE_WAIT_MS = 1_000; -const RESUME_EXECUTABLE_STATUSES = ["todo", "in_progress", "in_review"]; function errorToMessage(error: unknown) { return error instanceof Error ? error.message : String(error); @@ -436,15 +436,13 @@ export function issueTreeControlRoutes( const activeHold = await treeControlSvc.getHold(root.companyId, holdId); const issueIds = activeHold?.mode === "pause" && activeHold.rootIssueId === root.id - ? (activeHold.members ?? []) - .filter((member) => !member.skipped) - .map((member) => member.issueId) + ? (activeHold.members ?? []).map((member) => member.issueId) : []; if (issueIds.length > 0) { const candidates = await db.select({ id: issueRows.id, identifier: issueRows.identifier }) .from(issueRows).where(and( eq(issueRows.companyId, root.companyId), inArray(issueRows.id, issueIds), - inArray(issueRows.status, RESUME_EXECUTABLE_STATUSES), isNotNull(issueRows.assigneeAgentId), + inArray(issueRows.status, ISSUE_TREE_RESUME_EXECUTABLE_STATUSES), isNotNull(issueRows.assigneeAgentId), )); for (const task of candidates) { const blocked = await getExecutionBlocker(db, root.companyId, task.id); @@ -489,16 +487,18 @@ export function issueTreeControlRoutes( const wakeFailures: Array<{ issueId: string; message: string }> = []; if (hold.mode === "pause" && req.body.metadata?.wakeAgents === true) { for (const member of hold.members ?? []) { - if (member.skipped) continue; try { const issue = await issuesSvc.getById(member.issueId); if ( !issue || issue.companyId !== root.companyId || !issue.assigneeAgentId || - !RESUME_EXECUTABLE_STATUSES.includes(issue.status) + !ISSUE_TREE_RESUME_EXECUTABLE_STATUSES.includes(issue.status) ) continue; + // The snapshot may predate reopening a task or releasing another + // pause. Current task state and remaining holds govern execution. + if (await treeControlSvc.getActivePauseHoldGate(root.companyId, issue.id)) continue; await heartbeat.wakeup(issue.assigneeAgentId, { source: "assignment", triggerDetail: "system", diff --git a/server/src/services/issue-tree-control.ts b/server/src/services/issue-tree-control.ts index ecf8a07c6fd..1d8af92fd5b 100644 --- a/server/src/services/issue-tree-control.ts +++ b/server/src/services/issue-tree-control.ts @@ -71,6 +71,7 @@ type RestoreTreeStatusResult = TreeStatusUpdateResult & { }; const TERMINAL_ISSUE_STATUSES = new Set(["done", "cancelled"]); +export const ISSUE_TREE_RESUME_EXECUTABLE_STATUSES = ["todo", "in_progress", "in_review"]; const ACTIVE_RUN_STATUSES = ["queued", "running"] as const; const DEFAULT_RELEASE_POLICY: IssueTreeHoldReleasePolicy = { strategy: "manual" }; const MAX_PAUSE_HOLD_ANCESTOR_DEPTH = 100; @@ -325,13 +326,21 @@ function issueSkipReason(input: { return null; } -function buildAffectedAgents(issuesToPreview: IssueTreePreviewIssue[]): IssueTreePreviewAgent[] { +function canWakeOnResume(issue: IssueTreePreviewIssue): boolean { + return !!issue.assigneeAgentId && ISSUE_TREE_RESUME_EXECUTABLE_STATUSES.includes(issue.status); +} + +function buildAffectedAgents( + issuesToPreview: IssueTreePreviewIssue[], + mode: IssueTreeControlMode, +): IssueTreePreviewAgent[] { const byAgentId = new Map(); for (const issue of issuesToPreview) { if (issue.skipped) continue; + if (mode === "resume" && !canWakeOnResume(issue)) continue; const agentIds = new Set(); if (issue.assigneeAgentId) agentIds.add(issue.assigneeAgentId); - if (issue.activeRun) agentIds.add(issue.activeRun.agentId); + if (mode !== "resume" && issue.activeRun) agentIds.add(issue.activeRun.agentId); for (const agentId of agentIds) { const current = byAgentId.get(agentId) ?? { agentId, issueCount: 0, activeRunCount: 0 }; current.issueCount += 1; @@ -388,6 +397,17 @@ function buildWarnings(input: { }); } + if (input.mode === "resume") { + const idleIssueIds = affectedIssues.filter((issue) => !canWakeOnResume(issue)).map((issue) => issue.id); + if (idleIssueIds.length > 0) { + warnings.push({ + code: "resume_tasks_remain_idle", + message: "Blocked, backlog, and unassigned tasks will be unpaused but will not start.", + issueIds: idleIssueIds.sort(), + }); + } + } + if (input.mode === "restore") { const changedIssueIds = input.issuesToPreview .filter((issue) => issue.skipReason === "changed_after_cancel") @@ -676,7 +696,7 @@ export function issueTreeControlService(db: Db) { const activeRuns = activeRunRows .map(toPreviewRun) .sort((a, b) => a.issueId.localeCompare(b.issueId) || a.id.localeCompare(b.id)); - const affectedAgents = buildAffectedAgents(issuesToPreview); + const affectedAgents = buildAffectedAgents(issuesToPreview, input.mode); return { companyId, diff --git a/tests/e2e/composer-stop.spec.ts b/tests/e2e/composer-stop.spec.ts index a9c747f9bf5..30c31a33a8b 100644 --- a/tests/e2e/composer-stop.spec.ts +++ b/tests/e2e/composer-stop.spec.ts @@ -380,7 +380,16 @@ for (const adapter of ["process", "paperclip_runner"] as const) { ).toEqual([]); } await menu(page, "Resume subtree"); - await page.getByRole("dialog").getByRole("checkbox").check(); + const resumeWake = page.getByRole("dialog").getByRole("checkbox"); + if (adapter === "process") { + // Stopped legacy work is blocked pending reconciliation. Removing the + // pause must not promise or request a wake through that recovery gate. + await expect(resumeWake).toHaveAccessibleName("Wake affected agents (0)"); + await expect(resumeWake).toBeDisabled(); + await expect(resumeWake).not.toBeChecked(); + } else { + await resumeWake.check(); + } await page .getByRole("dialog") .getByRole("button", { name: "Resume subtree", exact: true }) diff --git a/tests/e2e/paused-composer.spec.ts b/tests/e2e/paused-composer.spec.ts index c01d083632a..f3c41525648 100644 --- a/tests/e2e/paused-composer.spec.ts +++ b/tests/e2e/paused-composer.spec.ts @@ -7,6 +7,51 @@ async function json(response: APIResponse) { } for (const classic of [false, true]) { + test(`resume preview: ${classic ? "classic" : "task chat"} does not promise to wake blocked work`, async ({ page, request }) => { + test.setTimeout(120_000); + const company = await json(await request.post("/api/companies", { data: { name: `Resume preview ${Date.now()}` } })); + const settings = await json(await request.get("/api/instance/settings/experimental")); + try { + await json(await request.patch("/api/instance/settings/experimental", { data: { enableClassicTaskInterface: classic } })); + const agent = await json(await request.post(`/api/companies/${company.id}/agents`, { data: { + name: "Resume preview fixture", role: "engineer", adapterType: "process", + adapterConfig: { command: "/usr/bin/true" }, + runtimeConfig: { heartbeat: { enabled: false, wakeOnDemand: false } }, + } })); + const task = await json(await request.post(`/api/companies/${company.id}/issues`, { data: { + title: "Blocked work stays blocked", status: "blocked", assigneeAgentId: agent.id, + } })); + await page.goto(`/${company.issuePrefix}/issues/${task.identifier}`); + await page.getByRole("button", { name: "More task actions", exact: true }).click(); + await page.getByRole("button", { name: "Pause work", exact: true }).click(); + const takeover = page.getByTestId("paused-composer-takeover"); + await expect(takeover).toBeVisible(); + // Enable dispatch only after the hold exists; assignment itself otherwise starts a run. + expect((await json(await request.get(`/api/issues/${task.id}`))).status).toBe("blocked"); + await json(await request.patch(`/api/agents/${agent.id}`, { data: { + runtimeConfig: { heartbeat: { enabled: false, wakeOnDemand: true } }, + } })); + await takeover.getByRole("button", { name: "Resume task" }).click(); + const dialog = page.getByRole("dialog"); + await expect(dialog).toContainText("The pause will be removed from 1 task."); + const wake = dialog.getByRole("checkbox", { name: "Wake affected agents (0)" }); + await expect(wake).toBeDisabled(); + await expect(wake).not.toBeChecked(); + await expect(dialog).toContainText("Blocked, backlog, and unassigned tasks will be unpaused but will not start."); + await dialog.getByRole("button", { name: "Resume work", exact: true }).click(); + await expect(dialog).toHaveCount(0); + await expect(takeover).toHaveCount(0); + await page.reload(); + const editor = (classic ? page.getByTestId("issue-chat-composer") : page).getByRole("textbox", { name: "editable markdown" }); + await expect(editor).toBeVisible(); + expect((await json(await request.get(`/api/issues/${task.id}`))).status).toBe("blocked"); + expect(await json(await request.get(`/api/companies/${company.id}/heartbeat-runs?agentId=${agent.id}`))).toEqual([]); + } finally { + await request.patch(`/api/companies/${company.id}`, { data: { status: "archived" } }); + await request.patch("/api/instance/settings/experimental", { data: { enableClassicTaskInterface: settings.enableClassicTaskInterface } }); + } + }); + test(`paused composer: ${classic ? "classic" : "task chat"} preserves drafts and requires resume`, async ({ page, request }) => { test.setTimeout(120_000); const company = await json(await request.post("/api/companies", { data: { name: `Paused composer ${Date.now()}` } })); @@ -63,8 +108,11 @@ for (const classic of [false, true]) { await takeover.getByRole("link", { name: "Resume subtree" }).click(); await expect(page).toHaveURL(new RegExp(task.identifier)); await takeover.getByRole("button", { name: "Resume subtree" }).click(); - await page.getByRole("dialog").getByRole("checkbox").uncheck(); + const subtreeWake = page.getByRole("dialog").getByRole("checkbox"); + if (await subtreeWake.isEnabled()) await subtreeWake.uncheck(); + else await expect(subtreeWake).not.toBeChecked(); await page.getByRole("dialog").getByRole("button", { name: "Resume subtree", exact: true }).click(); + await expect(page.getByRole("dialog")).toHaveCount(0); await page.goto(`/${company.issuePrefix}/issues/${child.identifier}`); await expect(takeover).toHaveCount(0); await expect(editor).toBeVisible(); diff --git a/ui/src/components/TaskTreeControls.tsx b/ui/src/components/TaskTreeControls.tsx index d2fd9137547..dfd0d1aaf1f 100644 --- a/ui/src/components/TaskTreeControls.tsx +++ b/ui/src/components/TaskTreeControls.tsx @@ -78,6 +78,7 @@ export function TaskTreeControlDialog({ scope, affectedCount, affectedAgentCount, + warnings = [], loading, error, pending, @@ -93,6 +94,7 @@ export function TaskTreeControlDialog({ scope: "leaf" | "subtree"; affectedCount: number; affectedAgentCount: number; + warnings?: string[]; loading: boolean; error?: string | null; pending: boolean; @@ -129,7 +131,9 @@ export function TaskTreeControlDialog({ ? "Loading…" : cancel ? `${tasks} will be cancelled.` - : `${tasks} will ${mode === "restore" ? "be restored" : "resume"}.`} + : mode === "restore" + ? `${tasks} will be restored.` + : `The pause will be removed from ${tasks}.`} {error ? ( @@ -151,13 +155,16 @@ export function TaskTreeControlDialog({ ) : null} + {!loading ? warnings.map((warning) => ( +

{warning}

+ )) : null}