From 4b89eec644d6e1a99d58926e77ae58927c3fa87b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 26 Oct 2026 07:30:27 -0500 Subject: [PATCH 001/133] fix: check selected workspace projects before offering retry (#17552) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task recovery explains access failures before a user retries saved work. > - A task project and its selected workspace are separate settings. > - Moving a task can retain files from a project that the agent cannot access. > - The recovery notice checked only the task project or a past failure record. > - This pull request checks the projects required by the current workspace selection. > - The notice now offers Retry only after those project checks pass. ## Linked Issues or Issue Description Follow-up to #17509. Related: #13230 concerns workspace binding consistency; this fix changes recovery presentation only. **What happened?** A task could show “Project access is ready” after moving to an allowed project while keeping a repository from a denied project. Typed failures could also keep the notice blocked after an authorized workspace replacement. **Expected behavior** Use the current task, workspace, and source project requirements. Honor a valid pending selection for the next admission. Keep stale, archived, or missing selections blocked without offering a project-permission repair. **Steps to reproduce** 1. Assign a low-trust agent to a task whose selected source belongs to a denied project. 2. Persist a project access failure. 3. Move the task to an allowed project without changing its selected files. 4. Reload the task. The notice must still explain the access block. 5. Select an allowed source through the workspace API. Reload the task. The notice must offer an explicit retry without starting work. **Paperclip version or commit** Reproduced after #17509 on master. The regression failed on the old projection and passed with this change. ## What Changed - Batch the current workspace project requirements for tasks with project access failures. - Check retained workspace and source projects, plus valid pending workspace selections. - Keep unknown historical denial projects unnamed. Do not expose unrelated project identifiers. - Keep unavailable workspace selections in the non-retryable failure state. Adding project permissions cannot repair these selections. - Add real database regressions for project moves, pending and applied repairs, stale revisions, archived pending selections, and foreign-company workspaces. - Update the recovery contract and verification recipe. ## Verification - Red-green regression with the real issue update service and execution access check. - Focused tests: 17 tests passed across `agent-project-access.test.ts` and `task-interruption-presentation.test.ts` after rebasing on current master. - Browser: the retained source stays blocked when the task project is allowed. Manage project access opens Permissions / Trust with Allowed projects and Add project. - The normal workspace API accepted an allowed pending selection. Returning to the task showed “Project access is ready” and Retry. Permissions and run count did not change. - Full repository typecheck and build passed before the final presentation adjustment. The final adjustment passed server typecheck, server build, and all 17 focused tests. The stale, archived, and foreign-workspace cases failed before this adjustment. Map and module-boundary checks passed. - Browser: an archived pending workspace shows the existing failure notice without Retry or a project-permission action. - `pnpm test:run` was attempted and stopped after 15 native runtime database tests failed because embedded PostgreSQL could not initialize on this host. The focused database tests passed on isolated PostgreSQL 18 in Docker. All technical CI checks passed on `a6e9f977`. Fresh CI and review are required for the presentation adjustment. One Runner shard passed on its single rerun after a provider shutdown failure. A fresh local Runner build also passed that case and its reuse sibling. The prior Paperclip Review could not publish its assessment. Its saved presentation finding is addressed. Merge waits for a fresh published review and all checks on the current head. ## Risks - Presentation must follow workspace selection precedence. Tests cover retained bindings, valid pending changes, stale changes, and company isolation. - This adds at most three bounded reads for tasks with a project access failure. It adds no reads for this purpose on unaffected tasks. - Dispatch remains the authorization authority. This change does not grant access, move files, or start execution. No migration is required. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and browser tools. The exact serving model identifier and context window were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/SPEC-implementation.md | 10 +- maps/feature-map/recovery.md | 6 ++ .../__tests__/agent-project-access.test.ts | 94 ++++++++++++++++++- .../task-interruption-presentation.ts | 44 ++++++--- server/src/services/task-workspace-access.ts | 50 ++++++++++ 5 files changed, 186 insertions(+), 18 deletions(-) diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index 1048f886509..bfabbd50c6a 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -897,8 +897,14 @@ Permissions / Trust settings. Board operators with `agents:create` can add same-company projects through `POST /api/agents/:id/project-access`; the operation locks the agent, preserves existing permissions and audits added projects. Saving does not retry work. The task offers explicit Retry after the required -project is allowed, and dispatch rechecks authorization. Existing setup failures -with the exact project-boundary denial also receive the explanation on reload. +projects are allowed, and dispatch rechecks authorization. Readiness checks the +task's current organizational project and selected workspace source, including +a valid pending selection. Moving the task to an allowed project does not clear +a retained source's access requirement. Existing setup failures with the exact +project-boundary denial also receive the explanation on reload, without assuming +that the task's current project caused the historical failure. A stale or +unavailable workspace selection stays in the non-retryable failure state; it +does not ask the user to grant project access that cannot repair the selection. Cross-issue writes are contained per heartbeat run. An agent-authored comment may wake the target assignee, including an explicit `resume: true` comment on a diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index b7de12dddc4..a6f49522fef 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -53,6 +53,12 @@ A workspace project denial names the blocked access and links to the agent's project, return to the saved task for an explicit **Retry task** action. Saving permissions alone does not wake the agent. Historical setup denials receive the same notice on reload; unrelated errors and prior assignees do not. +Move a denied task to a project the agent can access while retaining a source +from another project: the notice must stay blocked until the selected source is +also allowed or an authorized workspace selection replaces it. A valid pending +selection is checked for the next admission; a stale or unavailable selection +must show a non-retryable failure without a project-permission repair action. +Project changes alone preserve selected files. Thinking appears immediately after Retry, before the request completes. Technical source/run details remain in deliberate diagnostics. The diff --git a/server/src/__tests__/agent-project-access.test.ts b/server/src/__tests__/agent-project-access.test.ts index 38618a3669c..5b7b26baa34 100644 --- a/server/src/__tests__/agent-project-access.test.ts +++ b/server/src/__tests__/agent-project-access.test.ts @@ -4,12 +4,14 @@ import request from "supertest"; import { and, eq } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it } from "vitest"; import { agents, agentWakeupRequests, activityLog, authUsers, companies, companyMemberships, createDb, - heartbeatRuns, issues, projects, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; + executionWorkspaces, heartbeatRuns, issues, projects, projectWorkspaces, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; import { registerAgentProjectAccessRoutes } from "../routes/agent-project-access.js"; import { errorHandler } from "../middleware/error-handler.js"; -import { assertTaskWorkspaceSourceProjectAccess } from "../services/task-workspace-access.js"; +import { assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "../services/task-workspace-access.js"; import { taskInterruptionPresentation } from "../services/task-interruption-presentation.js"; import { ProjectAccessDeniedError } from "../services/project-access-denial.js"; +import { issueService } from "../services/issues.js"; +import { executionWorkspaceService } from "../services/execution-workspaces.js"; import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; describe("additive project access", () => { @@ -43,6 +45,94 @@ describe("additive project access", () => { } const board = { type: "board" as const, source: "local_implicit" as const }; + it("keeps a historical denial blocked after the task moves to an allowed project but retains its source", async () => { + const f = await seed(); + const [source] = await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: f.selected[1]!.id, + name: "Original repository", cwd: "/tmp/project-access-original" }).returning(); + await db.update(issues).set({ projectWorkspaceId: source!.id, + workspaceSelection: { version: 1, source: "project", selection: { + kind: "configured_source", projectWorkspaceId: source!.id, mode: "managed_isolated", + } } }).where(eq(issues.id, f.issue.id)); + const moved = await issueService(db).update(f.issue.id, { projectId: f.selected[0]!.id }); + expect(moved).toMatchObject({ projectId: f.selected[0]!.id, projectWorkspaceId: source!.id }); + const actor = { type: "agent" as const, source: "agent_key" as const, agentId: f.agentId, companyId: f.companyId }; + await expect(assertTaskWorkspaceExecutionAccess(db, actor, f.companyId, { issueId: f.issue.id })) + .rejects.toBeInstanceOf(ProjectAccessDeniedError); + expect(await taskInterruptionPresentation(db, moved!)).toMatchObject({ + canRetry: false, userAction: "project_access", projectAccess: { ready: false, projectId: null }, + }); + expect((await request(app(board)).post(`/agents/${f.agentId}/project-access`) + .send({ projectIds: [f.selected[1]!.id] })).status).toBe(200); + expect(await taskInterruptionPresentation(db, moved!)).toMatchObject({ canRetry: true, projectAccess: { ready: true } }); + await expect(assertTaskWorkspaceExecutionAccess(db, actor, f.companyId, { issueId: f.issue.id })).resolves.toBeDefined(); + }); + + it.each(["applied", "pending"] as const)("rechecks the selected source after an %s repair instead of the old denied project", async repair => { + const f = await seed(); + const [source] = await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: f.selected[0]!.id, + name: "Allowed repository", cwd: "/tmp/project-access-allowed" }).returning(); + await db.update(heartbeatRuns).set({ errorCode: "project_access_denied", error: null, + resultJson: { projectAccessDenied: { projectId: f.selected[1]!.id } } }).where(eq(heartbeatRuns.id, f.runId)); + const moved = await issueService(db).update(f.issue.id, { projectId: f.selected[0]!.id }); + if (repair === "applied") { + await issueService(db).update(f.issue.id, { projectWorkspaceId: source!.id, workspaceSelectionActor: board }); + } else { + await executionWorkspaceService(db).selectTaskWorkspace({ companyId: f.companyId, issueId: f.issue.id, actor: board, + selection: { kind: "configured_source", projectWorkspaceId: source!.id, mode: "managed_isolated" }, + expectedBindingRevision: 0, requestKey: randomUUID() }); + } + expect(await taskInterruptionPresentation(db, moved!)).toMatchObject({ canRetry: true, projectAccess: { ready: true } }); + // The original denied source was not granted, and presentation creates no execution. + expect((await db.select().from(agents).where(eq(agents.id, f.agentId)))[0]!.permissions).toEqual(f.permissions); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1); + if (repair === "pending") { + await db.update(issues).set({ workspaceBindingRevision: 2 }).where(eq(issues.id, f.issue.id)); + const stale = await taskInterruptionPresentation(db, moved!); + expect(stale).toMatchObject({ state: "failed", canRetry: false, userAction: null }); + expect(stale).not.toHaveProperty("projectAccess"); + } + }); + + it("keeps a pending existing selection blocked if its workspace is archived before retry", async () => { + const f = await seed(); + const task = await issueService(db).update(f.issue.id, { projectId: f.selected[0]!.id }); + const [workspace] = await db.insert(executionWorkspaces).values({ companyId: f.companyId, + projectId: f.selected[0]!.id, name: "Selected files", cwd: "/tmp/project-access-selected", + mode: "isolated_workspace", strategyType: "git_worktree" }).returning(); + const service = executionWorkspaceService(db); + await service.selectTaskWorkspace({ companyId: f.companyId, issueId: f.issue.id, actor: board, + selection: { kind: "existing", workspaceId: workspace!.id }, expectedBindingRevision: 0, requestKey: randomUUID() }); + expect(await taskInterruptionPresentation(db, task!)).toMatchObject({ canRetry: true, projectAccess: { ready: true } }); + expect(await service.archiveWorkspaceUnderLifecycleLock({ id: workspace!.id, patch: {}, closedAt: new Date() })) + .toMatchObject({ outcome: "archived" }); + await expect(service.applyPendingTaskWorkspaceSelection({ companyId: f.companyId, issueId: f.issue.id, + runId: f.runId, actor: { type: "agent", source: "agent_key", agentId: f.agentId, companyId: f.companyId } })) + .rejects.toThrow("Workspace is unavailable or inaccessible"); + const archived = await taskInterruptionPresentation(db, task!); + expect(archived).toMatchObject({ state: "failed", canRetry: false, userAction: null }); + expect(archived).not.toHaveProperty("projectAccess"); + }); + + it.each(["denied_project", "foreign_company"] as const)("retains the %s gate on a bound workspace even when its source is allowed", async boundary => { + const f = await seed(), foreign = boundary === "foreign_company" ? await seed() : null; + const [source] = await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: f.selected[0]!.id, + name: "Allowed source", cwd: "/tmp/project-access-source" }).returning(); + const [workspace] = await db.insert(executionWorkspaces).values({ companyId: foreign?.companyId ?? f.companyId, + projectId: foreign?.selected[0]!.id ?? f.selected[1]!.id, projectWorkspaceId: foreign ? null : source!.id, + name: "Retained files", cwd: "/tmp/project-access-bound", mode: "isolated_workspace", strategyType: "git_worktree" }).returning(); + const [task] = await db.update(issues).set({ projectId: f.selected[0]!.id, projectWorkspaceId: source!.id, + executionWorkspaceId: workspace!.id }).where(eq(issues.id, f.issue.id)).returning(); + const blocked = await taskInterruptionPresentation(db, task!); + if (boundary === "denied_project") { + expect(blocked).toMatchObject({ canRetry: false, + userAction: "project_access", projectAccess: { ready: false, projectId: null } }); + } else { + expect(blocked).toMatchObject({ state: "failed", canRetry: false, userAction: null }); + expect(blocked).not.toHaveProperty("projectAccess"); + } + }); + it("preserves containment, audits concurrent additions, and changes the persisted denial into an explicit retry", async () => { const f = await seed(); const actor = { type: "agent" as const, source: "agent_key" as const, agentId: f.agentId, companyId: f.companyId }; diff --git a/server/src/services/task-interruption-presentation.ts b/server/src/services/task-interruption-presentation.ts index f8dd6664e33..eb489f1ef6d 100644 --- a/server/src/services/task-interruption-presentation.ts +++ b/server/src/services/task-interruption-presentation.ts @@ -6,6 +6,7 @@ import { isAiAuthenticationBlocked } from "./ai-auth-failure.js"; import { readSystemRecovery } from "./system-recovery-state.js"; import { PROJECT_ACCESS_DENIED } from "./project-access-denial.js"; import { authorizationService } from "./authorization.js"; +import { taskWorkspaceProjectRequirements } from "./task-workspace-access.js"; type ProjectionIssue = Pick; type ProjectionRun = Pick; projectAccessAllowed?: boolean; + projectAccessProjectId?: string | null; + workspaceSelectionAvailable?: boolean; project?: Pick; company?: Pick; paused: boolean; @@ -30,7 +33,7 @@ function deniedProjectId(issue: ProjectionIssue, run?: ProjectionRun): string | || (run.errorCode !== PROJECT_ACCESS_DENIED && !historical)) return undefined; const detail = run.resultJson?.projectAccessDenied; return detail && typeof detail === "object" && "projectId" in detail && typeof detail.projectId === "string" - ? detail.projectId : historical ? issue.projectId : null; + ? detail.projectId : null; } /** Presentation never grants authority. Dispatch repeats all gates under its @@ -53,8 +56,11 @@ export function projectTaskInterruption(issue: ProjectionIssue, input: Projectio const deniedProject = deniedProjectId(issue, run); let projectAccess: TaskInterruptionPresentation["projectAccess"]; if (deniedProject !== undefined) { + // Permissions cannot repair a stale or unavailable workspace selection. + if (input.workspaceSelectionAvailable === false) return { ...base, state: "failed" }; const ready = input.projectAccessAllowed === true; - projectAccess = { agentId: run.agentId!, projectId: deniedProject === issue.projectId ? deniedProject : null, ready }; + const projectId = input.projectAccessProjectId === undefined ? deniedProject : input.projectAccessProjectId; + projectAccess = { agentId: run.agentId!, projectId: projectId === issue.projectId ? projectId : null, ready }; if (!ready) return { ...base, responsibility: "user", state: "waiting_for_user", userAction: "project_access", projectAccess }; } @@ -120,13 +126,17 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i // Recheck current authority, including standard trust, root-issue grants and // project policy. Do not reuse the failed run's frozen policy for a new retry. // Only denied tasks need these reads; deduplicate projects and batch per agent. + // A task's organizational project can differ from its retained source. Old + // failures did not record the denied project, and even typed failures can + // become obsolete after a workspace selection changes. + const deniedIssues = selected.filter(issue => issue.assigneeAgentId && !["done", "cancelled"].includes(issue.status) + && deniedProjectId(issue, runMap.get(issue.id)) !== undefined); + const requirements = await taskWorkspaceProjectRequirements(db, companyId, deniedIssues.map(issue => issue.id)); const deniedByAgent = new Map>(); - for (const issue of selected) { - const projectId = deniedProjectId(issue, runMap.get(issue.id)); - if (!projectId || !issue.assigneeAgentId || ["done", "cancelled"].includes(issue.status)) continue; - const ids = deniedByAgent.get(issue.assigneeAgentId) ?? new Set(); - ids.add(projectId); - deniedByAgent.set(issue.assigneeAgentId, ids); + for (const issue of deniedIssues) { + const ids = deniedByAgent.get(issue.assigneeAgentId!) ?? new Set(); + for (const projectId of requirements.get(issue.id)?.projectIds ?? []) ids.add(projectId); + if (ids.size) deniedByAgent.set(issue.assigneeAgentId!, ids); } const currentAccess = new Map(); const authorization = authorizationService(db); @@ -140,12 +150,18 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i const agentMap = new Map(agentRows.map(agent => [agent.id, agent])); const projectMap = new Map(projectRows.map(project => [project.id, project])); const pauses = new Set(pauseRows.map(row => row.issueId)), approvalsSet = new Set(approvalRows.map(row => row.issueId)); - for (const issue of selected) result.set(issue.id, projectTaskInterruption(issue, { - run: runMap.get(issue.id), action: actionMap.get(issue.id), wait: waitMap.get(issue.id), - projectAccessAllowed: currentAccess.get(`${issue.assigneeAgentId}:${deniedProjectId(issue, runMap.get(issue.id))}`), - agent: agentMap.get(issue.assigneeAgentId ?? ""), project: projectMap.get(issue.projectId ?? ""), - company: companyRows[0], paused: pauses.has(issue.id), approval: approvalsSet.has(issue.id), - })); + for (const issue of selected) { + const required = requirements.get(issue.id); + const denied = required?.projectIds.filter(projectId => currentAccess.get(`${issue.assigneeAgentId}:${projectId}`) !== true); + result.set(issue.id, projectTaskInterruption(issue, { + run: runMap.get(issue.id), action: actionMap.get(issue.id), wait: waitMap.get(issue.id), + projectAccessAllowed: denied?.length === 0, + projectAccessProjectId: denied?.includes(issue.projectId ?? "") ? issue.projectId : null, + workspaceSelectionAvailable: required?.available ?? false, + agent: agentMap.get(issue.assigneeAgentId ?? ""), project: projectMap.get(issue.projectId ?? ""), + company: companyRows[0], paused: pauses.has(issue.id), approval: approvalsSet.has(issue.id), + })); + } return result; } diff --git a/server/src/services/task-workspace-access.ts b/server/src/services/task-workspace-access.ts index 31d40f79655..0c4599982e1 100644 --- a/server/src/services/task-workspace-access.ts +++ b/server/src/services/task-workspace-access.ts @@ -10,9 +10,59 @@ import { resolveCoreTrustPreset } from "./trust-preset-resolver.js"; import { isDeepStrictEqual } from "node:util"; import { ProjectAccessDeniedError } from "./project-access-denial.js"; import { taskWorkspaceSelectionSchema, type TaskWorkspaceSelection } from "@paperclipai/shared"; +import { taskWorkspaceSelectableCondition } from "./task-workspace-selection.js"; type DbTransaction = Parameters[0]>[0]; +/** Read the project requirements of the next selected workspace, without + * admitting execution or moving files. Batch only the tasks needing a denial + * presentation; this adds at most three bounded reads, not queries per task. */ +export async function taskWorkspaceProjectRequirements(db: Db, companyId: string, issueIds: string[]) { + const result = new Map(); + if (!issueIds.length) return result; + const tasks = await db.select({ id: issues.id, projectId: issues.projectId, + executionWorkspaceId: issues.executionWorkspaceId, projectWorkspaceId: issues.projectWorkspaceId, + workspaceSelection: issues.workspaceSelection, workspacePendingSelection: issues.workspacePendingSelection, + workspaceBindingRevision: issues.workspaceBindingRevision, + }).from(issues).where(and(eq(issues.companyId, companyId), inArray(issues.id, issueIds))); + const selections = tasks.map(task => { + const pending = task.workspacePendingSelection; + const intent = pending?.intent ?? (!task.executionWorkspaceId ? task.workspaceSelection : null); + const parsed = intent ? taskWorkspaceSelectionSchema.safeParse(intent.selection) : null; + const selection = parsed?.success ? parsed.data : null; + const available = (!pending || pending.expectedBindingRevision === task.workspaceBindingRevision) + && (!parsed || parsed.success); + const workspaceId = selection ? selection.kind === "existing" ? selection.workspaceId : null : task.executionWorkspaceId; + return { task, selection, available, workspaceId }; + }); + const workspaceIds = [...new Set(selections.flatMap(row => row.workspaceId ? [row.workspaceId] : []))]; + const workspaces = workspaceIds.length ? await db.select({ id: executionWorkspaces.id, + projectId: executionWorkspaces.projectId, projectWorkspaceId: executionWorkspaces.projectWorkspaceId, + selectable: sql`${taskWorkspaceSelectableCondition()}`, + }).from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, companyId), inArray(executionWorkspaces.id, workspaceIds))) : []; + const workspaceMap = new Map(workspaces.map(workspace => [workspace.id, workspace])); + const sources = selections.map(row => { + const workspace = row.workspaceId ? workspaceMap.get(row.workspaceId) : undefined; + const sourceId = row.selection + ? row.selection.kind === "configured_source" ? row.selection.projectWorkspaceId : workspace?.projectWorkspaceId + : workspace?.projectWorkspaceId ?? row.task.projectWorkspaceId; + return { ...row, workspace, sourceId }; + }); + const sourceIds = [...new Set(sources.flatMap(row => row.sourceId ? [row.sourceId] : []))]; + const sourceRows = sourceIds.length ? await db.select({ id: projectWorkspaces.id, projectId: projectWorkspaces.projectId }) + .from(projectWorkspaces).where(and(eq(projectWorkspaces.companyId, companyId), inArray(projectWorkspaces.id, sourceIds))) : []; + const sourceMap = new Map(sourceRows.map(source => [source.id, source])); + for (const row of sources) { + const source = row.sourceId ? sourceMap.get(row.sourceId) : undefined; + result.set(row.task.id, { + projectIds: [...new Set([row.task.projectId, row.workspace?.projectId, source?.projectId].filter((id): id is string => Boolean(id)))], + available: row.available && (!row.workspaceId || Boolean(row.workspace)) && (!row.sourceId || Boolean(source)) + && (row.selection?.kind !== "existing" || row.workspace?.selectable === true), + }); + } + return result; +} + export async function assertTaskWorkspaceSourceProjectAccess( db: Db | DbTransaction, actor: AuthorizationActor, companyId: string, projectId: string | null, ) { From 41c42895fa862c586ff841f8c50200d93ffc83a4 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 26 Oct 2026 07:53:57 -0500 Subject: [PATCH 002/133] feat(evals): enforce fixed Runner request allocations (#17452) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner evals execute real provider harnesses against maintained fixtures. > - Turn-level usage arrives after requests have already spent money. > - Parallel cells need distinct allowances before requests start. > - Missing output must preserve the remaining allowance and any known charges. > - This pull request adds a controller that reuses the existing request ledger and gateway. > - It makes a bounded local Runner attempt reviewable before hosted scheduling is enabled. ## Linked Issues or Issue Description Refs #17440. That PR added the explicit Runner gateway route. This PR owns request admission and retained campaign accounting for that route. The existing hosted matrix is not enabled or changed here. Its global allocation, rerun rejection, and private evidence archive remain separate integration work. ## What Changed - Validate a trusted manifest with exact request, CLI, package, daemon, model, and source identities. Assign each selected cell a distinct allowance. Count operating overhead once. Reject allocations above the campaign limit before launch. - Run native Codex or ACPX Claude through the shared request proxy and append-only ledger. Pass only the proxy credential to the child. Refuse local attempt reuse and preserve existing output files. - Close gateway admission on cancellation. Retain the ledger, receipts, route descriptor, child result, and cleanup evidence outside the disposable workspace. - Recompute campaign accounting from retained ledgers. Missing or invalid final evidence holds the full cell allowance while preserving known charges. Reports distinguish provisional and incomplete costs. - Accept namespaces containing only local function/custom tools. Disable web search in the controlled Codex profile. Continue to reject hosted tools, including tools hidden inside a namespace. - Add controller tests to the existing Runner static test path and document this source-checkout entry point. ## Verification - 20 controller tests pass. They use real child processes, local HTTP, and the actual ledger. Cases cover reservations before dispatch, concurrent cell isolation, caps, cancellation, replay rejection, lost artifacts, changed receipts, and Braintrust Claude routing. Upstream responses are synthetic. - 52 shared gateway tests and 55 Runner CLI tests pass. The namespace acceptance and web-search configuration regressions failed before the repair and pass afterward. - Runner TypeScript/Rust checks, Runner build, shared E2E typecheck, and `git diff --check` pass. - The first actual-harness probe found that Codex advertised both a local tool namespace and web search. It reached no upstream transport. That failure is retained; the compatibility repair does not allow paid hosted tools. - An earlier test failure showed that separate dynamic imports created different admission-error classes. The shared runtime bridge now loads one module graph. Another regression showed that a missing final report hid known ledger charges; the collector now preserves those charges. - The repaired actual-harness probe passed its control checks for native Codex and ACPX Claude: each reached exactly one synthetic upstream request after the ledger reservation; both retained the unresolved hold and route evidence; both recorded clean process exit. A scan of 41 retained files found no synthetic upstream credential. Deliberate provider rejection preserved the failed Codex turn and Claude timeout; these are not semantic passes. Prices, operating bounds, and the package archive identity in this probe were synthetic. - Review regressions reproduced lost request/receipt evidence, lost admission metadata, executable replacement after verification, and a SIGTERM-resistant descendant surviving the CLI. The repairs now pass. Executable bytes are staged before launch; a trusted Node resolution hook preserves relative imports. A symlink-path regression also failed before its repair and now passes. The final staged native probe repeated both one-request checks successfully. - Full current-head [CI](https://github.com/paperclipai/paperclip/actions/runs/40115932485) passed on `4781487215116afd10a415d4db53b1ec8e004dd5`: all 58 check runs and the legacy status are successful, neutral, or skipped. Greptile and Paperclip both reviewed this commit at 5/5. All five finding threads are fixed and resolved. No paid provider request ran during this change. ## Risks - This is a trusted source-checkout controller, not a published package command. It needs the repository's installed shared cost dependencies. - The manifest declares reviewed prices and full-lifecycle bounds. Synthetic tests do not qualify real prices, invoices, storage, or Braintrust metering. - The attempt-directory guard is local. Hosted fanout still needs unique global allocations, rerun rejection, and durable private retention. Do not enable the existing matrix on this change alone. - Forced child termination can leave cleanup unverified. That attempt remains incomplete and holds its full allocation. These checks do not claim a host security boundary against an adversarial target. - The current private publisher does not archive the separate controller directory. Native trace relay, private report integration, and live qualification remain open. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. This runtime does not expose the exact model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../docs/runner-protocol-live-evals.md | 98 ++++++ packages/paperclip-runner/package.json | 2 +- .../controlled-evals/controller.test.mjs | 279 ++++++++++++++++++ .../scripts/controlled-evals/manifest.mjs | 93 ++++++ .../scripts/controlled-evals/report.mjs | 84 ++++++ .../scripts/controlled-evals/run.mjs | 193 ++++++++++++ .../controlled-evals/shared-cost-runtime.ts | 3 + .../src/cli/eval-gateway-runtime.test.ts | 1 + .../src/cli/eval-gateway-runtime.ts | 1 + tests/runner-e2e/eval-gateway.test.ts | 23 ++ tests/runner-e2e/eval-gateway.ts | 8 +- 11 files changed, 782 insertions(+), 3 deletions(-) create mode 100644 packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs create mode 100644 packages/paperclip-runner/scripts/controlled-evals/manifest.mjs create mode 100644 packages/paperclip-runner/scripts/controlled-evals/report.mjs create mode 100644 packages/paperclip-runner/scripts/controlled-evals/run.mjs create mode 100644 packages/paperclip-runner/scripts/controlled-evals/shared-cost-runtime.ts diff --git a/packages/paperclip-runner/docs/runner-protocol-live-evals.md b/packages/paperclip-runner/docs/runner-protocol-live-evals.md index 6d3cf586e0b..0232ef8ef37 100644 --- a/packages/paperclip-runner/docs/runner-protocol-live-evals.md +++ b/packages/paperclip-runner/docs/runner-protocol-live-evals.md @@ -159,6 +159,84 @@ change the hosted matrix. The controller must enforce request admission, deadlines, and limits before forwarding traffic. A local fake-API probe only proves routing; it does not qualify live provider or Braintrust behavior. + +## Controlled request budgets + +The source-checkout controller is +`packages/paperclip-runner/scripts/controlled-evals/run.mjs`. Runner/evals owns +this orchestration. It reuses the existing eval request proxy and ledger from +`tests/runner-e2e` through one import graph; it does not copy their reservation +or provider-usage rules. It requires the workspace's installed `tsx` runtime. +It is not a published Runner package entry point. + +The controller accepts the existing `--request` and `--output` contract. Set +`PAPERCLIP_RUNNER_EVAL_CONTROL_MANIFEST` to an explicit reviewed manifest and +supply the selected upstream credential only to this trusted controller. For +Runner Evals, pass this controller path as `eval_program.py run --runner-cli`; +the manifest identifies the actual built `eval-session.js`. The provider child +receives only its temporary proxy credential and private home. + +The manifest schema is `paperclip.runner-eval.controller/v1`. It contains: + +- `campaignId`, `maxUsd`, `deadlineAt`, `pricingVerifiedAt`, and + `pricingExpiresAt`; admission must fit a current four-hour window. +- `sourceRevision` and `definitionsRevision`, each an exact commit ID. + These are the trusted preparer's provenance declarations. +- `runnerCli`, `runnerPackage`, and `runnerd`, each an absolute `path` and + content `sha256`. The controller checks the three files before launch. +- An absolute durable `outputDirectory`, outside the disposable case workspace. +- `route` (`direct` or `braintrust`), plus `projectId` for Braintrust. +- `models`, using the existing exact-model price, request-limit, and reviewed + full-lifecycle `provisionalBound` contract. +- `overhead`, using the existing four-component operating allowance with a + current full-lifecycle bound for every component. This is reserved once. +- `attempts`, each with a unique `executionId` and `attemptId`, + `requestSha256`, `provider`, optional `acpxAgent`, exact `model`, `maxUsd`, + `maxRequests`, `maxConcurrent`, and `timeoutMs`. Native Codex and ACPX + Claude are supported. Each execution has one explicit attempt and no retry. + +Use `requestBindingSha256` from `controlled-evals/manifest.mjs` to bind the +complete compiled request. It excludes only the temporary working directory +and native executable path; the executable digest remains bound and its path +is checked separately. The controller gives the child a retained copy of the +validated request. Changing prompts, fixtures, models, limits, or attempts +requires a new reviewed binding. + +The sum of every cell's allocation and the once-only operating allowance must +fit `maxUsd` before any cell starts. A request reserves against its cell's +ledger before forwarding. Cells cannot borrow unused allocations. Unknown +requests keep their reservations, and failed or missing cell evidence keeps +that cell's full allocation in the campaign report. An existing cell directory +or output is never reset for a retry. Cancellation closes admission, drains +upstream receipts within the proxy limit, and stops the owned CLI process +group. Unverified provider cleanup remains explicit and retains the allocation. + +Each result retains `controllerAccounting` alongside the original provider +outcome and usage. Retain the durable controller directory as private evidence: +its admission, request, gateway descriptor, request receipts, ledger, original +provider artifact, and final result are required for reconciliation. The +private program's existing immutable attempt copy includes the result metadata; +its current default history publisher does not archive this separate directory. +Do not infer receipt retention from that metadata alone. + +```sh +node packages/paperclip-runner/scripts/controlled-evals/report.mjs \ + /private/path/controller-manifest.json > /private/path/controller-report.json +``` + +The report recomputes ledger costs, checks their attempt/manifest bindings, and +preserves known partial charges. It reports operating costs as held and total +costs as provisional when evidence is intact, or incomplete otherwise. It does +not release historical holds or claim invoice +settlement. Existing provider usage is retained separately; do not add the +provider and controller cost observations as if they were separate charges. + +This local controller is not yet wired into the hosted matrix. Its durable +attempt-directory guard applies to one shared filesystem, not independent +GitHub workers. Hosted integration must bind unique cells to the prepared +allocation, reject workflow/job reruns, and preserve the controller evidence +before enabling paid work. Native OTLP routing, report integration, vendor +metering bounds, and actual paid qualification remain separate gates. ## Controller-assigned native observability The same CLI accepts `--observability-config /absolute/path/observability.json` @@ -455,3 +533,23 @@ checksum-verified Grok binary before packaging the portable runtime. Targets without the Grok package keep their existing build behavior. This protocol workflow uses the explicitly selected API key; subscription credentials are not delivered by it. + +The controlled Codex profile disables web search. The shared proxy accepts local +function/custom tool namespaces and still rejects hosted tools, including any +hidden inside a namespace. These routes do not qualify web-search charges. + +The controller stages the exact verified CLI and daemon bytes in its private +attempt directory. A Node module-resolution hook keeps the CLI's relative +imports pointed at the trusted source checkout; it does not reload the original +entry file. Installed dependencies still belong to that trusted checkout. The +staged copies are removed after verified process cleanup and kept if cleanup is +incomplete. This path requires the repository's Node 24.11+ runtime. + +Campaign reports verify the retained request binding and the digest of every +request receipt before reducing a cell's allocation. Missing admission, request, +or receipt evidence preserves known ledger charges and holds the rest of the +full allocation. + +On cancellation, the controller keeps the scheduled process-group SIGKILL until +any remaining owned descendants receive it, even if the CLI exits first. A +forced stop still leaves cleanup unverified and retains the full allocation. diff --git a/packages/paperclip-runner/package.json b/packages/paperclip-runner/package.json index 8080d157740..f42661d79c4 100644 --- a/packages/paperclip-runner/package.json +++ b/packages/paperclip-runner/package.json @@ -85,7 +85,7 @@ "typecheck:browser": "tsc -p tsconfig.browser.json --noEmit", "test": "pnpm run test:typescript && pnpm run test:rust", "test:typescript": "pnpm run test:typescript:prep && vitest run", - "test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node ../../scripts/ci-profile/node-test.mjs test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/provider-pack-runtime-tree.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-runtime-dependency-roots.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs scripts/provider-pack-executable-shims.test.mjs && node --test scripts/build-copilot-distribution.test.mjs scripts/copilot-inner-distribution.test.mjs scripts/copilot-smoke-resources.test.mjs scripts/discover-copilot-acp.test.mjs scripts/materialize-copilot-binary.test.mjs scripts/install-copilot-assets.test.mjs scripts/probe-copilot-acp.test.mjs scripts/qualify-copilot-acp.test.mjs test/acpx-message-boundaries-package-contract.test.mjs", + "test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node ../../scripts/ci-profile/node-test.mjs test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/provider-pack-runtime-tree.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-runtime-dependency-roots.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs scripts/provider-pack-executable-shims.test.mjs scripts/controlled-evals/controller.test.mjs && node --test scripts/build-copilot-distribution.test.mjs scripts/copilot-inner-distribution.test.mjs scripts/copilot-smoke-resources.test.mjs scripts/discover-copilot-acp.test.mjs scripts/materialize-copilot-binary.test.mjs scripts/install-copilot-assets.test.mjs scripts/probe-copilot-acp.test.mjs scripts/qualify-copilot-acp.test.mjs test/acpx-message-boundaries-package-contract.test.mjs", "test:typescript:vitest": "node ./scripts/run-pr-vitest-lane.mjs", "test:rust": "cargo test --release --manifest-path runner/Cargo.toml --locked --workspace", "test:codex": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test codex_provider", diff --git a/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs b/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs new file mode 100644 index 00000000000..a4af19aa000 --- /dev/null +++ b/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs @@ -0,0 +1,279 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; +import { mkdtemp, mkdir, readFile, readdir, rm, symlink, writeFile } from 'node:fs/promises'; +import { writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { canonical, costRuntime, requestBindingSha256, sha256, validateManifest } from './manifest.mjs'; +import { attemptDirectory, runControlledEval } from './run.mjs'; +import { summarizeControlledCampaign } from './report.mjs'; + +const childScript = options => ` +import assert from 'node:assert/strict'; +import {createHash} from 'node:crypto'; +import {spawn} from 'node:child_process'; +import {once} from 'node:events'; +import {readFile,writeFile} from 'node:fs/promises'; +const args=new Map();for(let i=2;i{});process.stdout.write('ready');setInterval(()=>{},1000)"],{stdio:['ignore','pipe','ignore']}); + await once(descendant.stdout,'data'); + await writeFile(request.session.workingDirectory+'/descendant.pid',String(descendant.pid)); +} + +const descriptorRaw=await readFile(args.get('--gateway-config'),'utf8'),descriptor=JSON.parse(descriptorRaw); +for(const key of ['OPENAI_API_KEY','ANTHROPIC_API_KEY','BRAINTRUST_API_KEY','GH_TOKEN','NODE_OPTIONS','PYTHONPATH'])assert.equal(process.env[key],undefined); +const statuses=[]; +for(let i=0;i<${options.requests ?? 3};i++){ + const response=await fetch(descriptor.baseUrl+(descriptor.provider==='openai'?'/v1/responses':'/v1/messages'),{method:'POST',headers:{authorization:'Bearer '+process.env.PAPERCLIP_RUNNER_EVAL_GATEWAY_TOKEN,'content-type':'application/json'},body:JSON.stringify({model:request.model,input:'synthetic fixture'})}); + statuses.push(response.status);await response.text(); +} +if(${Boolean(options.waitForever)})await new Promise(()=>setInterval(()=>{},1000)); +if(!${Boolean(options.missingArtifact)})await writeFile(args.get('--output'),JSON.stringify({schema:'paperclip-runner/eval-session-artifact/v1',attemptId:request.attemptId,requestedModel:request.model, + requestRouting:${options.missingRoute ? 'null' : "{mode:'controller_gateway',provider:descriptor.provider,declaredUpstreamRoute:descriptor.upstreamRoute,campaignId:descriptor.campaignId,executionId:descriptor.executionId,attemptId:descriptor.attemptId,descriptorSha256:createHash('sha256').update(descriptorRaw).digest('hex')}"}, + snapshot:{status:'closed',process:{runnerExited:true,runnerPid:process.pid,providerPid:process.pid}},statuses, + leaked:${options.leakToken ? 'process.env.PAPERCLIP_RUNNER_EVAL_GATEWAY_TOKEN' : 'null'}})); +`; + +async function fixture(t, { count = 1, allocation = '0.01', providerProfile = 'codex', route = 'direct', ...options } = {}) { + const root = await mkdtemp(join(tmpdir(), 'runner-controller-test-')); + t.after(() => rm(root, { recursive: true, force: true })); + const cli = join(root, 'fake-cli.mjs'), binary = join(root, 'runnerd'), archive = join(root, 'runner.tgz'); + await writeFile(cli, childScript(options)); await writeFile(binary, 'synthetic daemon'); await writeFile(archive, 'synthetic package'); + const provenance = async path => ({ path, sha256: sha256(await readFile(path)) }); + const now = Date.now(), verifiedAt = new Date(now - 1000).toISOString(), expiresAt = new Date(now + 86_400_000).toISOString(); + const overhead = { version: 1, reference: 'synthetic-allowance', runner: 'github-hosted', verifiedAt, expiresAt, + components: costRuntime.EVAL_OVERHEAD_COMPONENTS.map(component => ({ component, maxUsd: '0.001', reference: 'synthetic-proof' })) }; + const bound = configuration => ({ version: 1, scope: 'full_lifecycle', evidenceSha256: 'a'.repeat(64), configurationSha256: sha256(canonical(configuration)) }); + for (const component of overhead.components) component.provisionalBound = bound({ reference: overhead.reference, runner: overhead.runner, verifiedAt, expiresAt, component: { ...component } }); + const biller = providerProfile === 'claude' ? 'anthropic' : 'openai'; + const price = { provider: biller, model: 'fixture', version: 'synthetic-pricing', sourceUrl: 'https://provider.invalid/pricing', inputUsdPerMillion: '2', cachedReadUsdPerMillion: '1', outputUsdPerMillion: '10', ...(biller === 'openai' ? { cacheWrite30mUsdPerMillion: '2', cacheWrite12hUsdPerMillion: '2' } : { cacheWrite5mUsdPerMillion: '2', cacheWrite1hUsdPerMillion: '2' }) }; + const model = { provider: biller, price, maximumInputTokens: 1000, maximumOutputTokens: 100, timeoutMs: 1000 }; + model.provisionalBound = bound({ provider: model.provider, mode: route, price, maximumInputTokens: 1000, maximumOutputTokens: 100, reservation: '0.003000000' }); + const manifest = { schema: 'paperclip.runner-eval.controller/v1', campaignId: 'synthetic-controller', maxUsd: '1', + deadlineAt: new Date(now + 60_000).toISOString(), pricingVerifiedAt: verifiedAt, pricingExpiresAt: expiresAt, overhead, + sourceRevision: 'a'.repeat(40), definitionsRevision: 'b'.repeat(40), runnerCli: await provenance(cli), runnerPackage: await provenance(archive), runnerd: await provenance(binary), + outputDirectory: join(root, 'controller'), route, ...(route === 'braintrust' ? { projectId: '11111111-1111-4111-8111-111111111111' } : {}), models: [model], attempts: [] }; + const requests = []; + for (let index = 0; index < count; index++) { + const workspace = join(root, `workspace-${index}`); await mkdir(workspace); + const request = { schema: 'paperclip-runner/eval-session-request/v1', attemptId: `attempt-${index}`, prompt: 'synthetic', provider: providerProfile === 'claude' ? 'acpx' : 'codex', ...(providerProfile === 'claude' ? { acpxAgent: 'claude' } : {}), model: 'fixture', + runnerd: { path: binary, sha256: manifest.runnerd.sha256 }, limits: { turnTimeoutMs: 5000, maxAgentTurns: 1, maxEstimatedCostNanodollars: 1_000_000_000 }, session: { workingDirectory: workspace } }; + const requestPath = join(root, `request-${index}.json`), outputPath = join(root, `output-${index}.json`); + await writeFile(requestPath, JSON.stringify(request)); requests.push({ request, requestPath, outputPath }); + manifest.attempts.push({ executionId: `execution-${index}`, attemptId: request.attemptId, requestSha256: requestBindingSha256(request), provider: request.provider, ...(request.acpxAgent ? { acpxAgent: request.acpxAgent } : {}), model: 'fixture', maxUsd: allocation, maxRequests: 2, maxConcurrent: 1, timeoutMs: 5000 }); + } + const manifestPath = join(root, 'manifest.json'); + const save = () => writeFile(manifestPath, JSON.stringify(manifest)); await save(); + const environment = { PATH: process.env.PATH, OPENAI_API_KEY: 'synthetic-upstream-private', ANTHROPIC_API_KEY: 'other-private', BRAINTRUST_API_KEY: 'other-braintrust', GH_TOKEN: 'other-github', NODE_OPTIONS: '--require must-not-inherit', PYTHONPATH: '/must-not-inherit' }; + return { root, manifest, manifestPath, requests, environment, save, + run: (index = 0, transport = () => success(biller), signal) => runControlledEval({ manifestPath, ...requests[index], environment, transport, signal }) }; +} +let sequence = 0; +function success(provider = 'openai') { + if (provider === 'anthropic') return Response.json({ type: 'message', id: `synthetic-${++sequence}`, model: 'fixture', usage: { input_tokens: 8, cache_read_input_tokens: 2, cache_creation_input_tokens: 0, output_tokens: 5, cache_creation: { ephemeral_5m_input_tokens: 0, ephemeral_1h_input_tokens: 0 } } }, { headers: { 'x-bt-used-endpoint': 'ANTHROPIC_API_KEY', 'x-bt-cached': 'MISS' } }); + return Response.json({ object: 'response', id: `synthetic-${++sequence}`, status: 'completed', model: 'fixture', usage: { + input_tokens: 10, input_tokens_details: { cache_write_tokens: 0, cached_tokens: 2 }, output_tokens: 5, output_tokens_details: { reasoning_tokens: 3 } } }); +} + +test('real child requests reserve before dispatch, stop at the cap and retain only controller credentials', async t => { + const f = await fixture(t); + let calls = 0; + const result = await f.run(0, async (_url, init) => { + calls++; + assert.equal(new Headers(init.headers).get('authorization'), 'Bearer synthetic-upstream-private'); + const ledger = await costRuntime.readEvalCostLedger(join(attemptDirectory(f.manifest, 'attempt-0'), 'ledger.jsonl')); + assert.equal(ledger.state.snapshot().inflight, 1); + return success(); + }); + assert.equal(calls, 2); assert.equal(result.code, 0); assert.equal(result.report.cleanupVerified, true); + const artifact = JSON.parse(await readFile(f.requests[0].outputPath, 'utf8')); + assert.deepEqual(artifact.statuses, [200, 200, 429]); + assert.equal(artifact.controllerAccounting.accounting.knownUsd, '0.000136000'); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.incompleteAttempts, 0); assert.equal(summary.heldUsd, '0.004000000'); assert.equal(summary.committedUsd, '0.004136000'); + for (const name of ['admission.json', 'result.json', 'ledger.jsonl', 'provider-artifact.json', 'stdout.log', 'stderr.log']) { + assert(!(await readFile(join(attemptDirectory(f.manifest, 'attempt-0'), name), 'utf8')).includes(f.environment.OPENAI_API_KEY)); + } + await assert.rejects(f.run(), /output already exists/); + await rm(f.requests[0].outputPath); + await assert.rejects(f.run(), { code: 'EEXIST' }); + assert.equal(calls, 2); +}); + +test('disjoint concurrent cells cannot borrow an allowance and unknown requests retain their reservations', async t => { + const f = await fixture(t, { count: 2, allocation: '0.003' }); f.manifest.maxUsd = '0.01'; await f.save(); + let calls = 0; + const transport = async () => { calls++; return Response.json({ error: 'synthetic refusal' }, { status: 429 }); }; + const results = await Promise.all([f.run(0, transport), f.run(1, transport)]); + assert.equal(calls, 2); assert(results.every(result => result.report.status === 'closed')); + assert(results.every(result => result.report.accounting.heldUsd === '0.003000000')); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.committedUsd, '0.010000000'); assert.equal(summary.knownUsd, '0.000000000'); assert.equal(summary.overBudget, false); + assert(summary.attempts.every(attempt => attempt.boundedUnresolvedRequests === 1)); +}); + +test('rejects oversized or unqualified allocations, changed requests and changed executables before dispatch', async t => { + const f = await fixture(t, { count: 2 }); + for (const mutate of [ + value => { value.maxUsd = '0.01'; }, value => { value.attempts[1].executionId = value.attempts[0].executionId; }, + value => { value.attempts[0].provider = 'acpx'; value.attempts[0].acpxAgent = 'codex'; }, + value => { delete value.overhead.components[0].provisionalBound; }, value => { value.models[0].secret = 'unexpected'; }, + ]) { const value = structuredClone(f.manifest); mutate(value); assert.throws(() => validateManifest(value)); } + f.requests[0].request.prompt = 'drift'; await writeFile(f.requests[0].requestPath, JSON.stringify(f.requests[0].request)); + await assert.rejects(f.run(), /allocated identity/); + await writeFile(f.manifest.runnerCli.path, 'changed CLI'); + await assert.rejects(f.run(1), /provenance changed/); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.incompleteAttempts, 2); assert.equal(summary.heldUsd, '0.024000000'); +}); + +for (const fault of ['missingArtifact', 'missingRoute', 'leakToken']) test(`${fault} keeps the allocation held and preserves the attempt failure`, async t => { + const f = await fixture(t, { [fault]: true, requests: 1 }); + const result = await f.run(); + assert.equal(result.code, 2); assert.equal(result.report.status, 'incomplete'); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.incompleteAttempts, 1); assert.equal(summary.committedUsd, '0.014000000'); assert.equal(summary.costStatus, 'incomplete'); + assert.equal(summary.knownUsd, '0.000068000'); + const artifact = JSON.parse(await readFile(f.requests[0].outputPath, 'utf8')); + assert.equal(artifact.accountingFailure.retryable, false); + if (fault === 'leakToken') { assert.equal(result.report.secretExcluded, false); assert.equal(artifact.leaked, '[redacted]'); } +}); + +test('cancellation closes the proxy, bounds the owned child, and preserves unknown request holds', async t => { + const f = await fixture(t, { requests: 1, waitForever: true }); + const abort = new AbortController(); let calls = 0; + const result = await f.run(0, async (_url, init) => { + calls++; abort.abort(); + return new Promise((_resolve, reject) => init.signal.addEventListener('abort', () => reject(new Error('synthetic interrupted request')), { once: true })); + }, abort.signal); + assert.equal(calls, 1); assert.equal(result.code, 2); assert.equal(result.report.cleanupVerified, false); + assert.equal(result.report.accounting.unknownRequests, 1); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.committedUsd, '0.014000000'); assert.equal(summary.incompleteAttempts, 1); +}); + +test('changed receipts cannot release allocations and expired campaigns remain inspectable', async t => { + const f = await fixture(t, { requests: 1 }); await f.run(); + const root = attemptDirectory(f.manifest, 'attempt-0'); + const resultPath = join(root, 'result.json'), result = JSON.parse(await readFile(resultPath, 'utf8')); + result.accounting.knownUsd = '0.000000000'; await writeFile(resultPath, JSON.stringify(result)); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.incompleteAttempts, 1); assert.equal(summary.committedUsd, '0.014000000'); assert.equal(summary.knownUsd, '0.000068000'); + assert.throws(() => validateManifest(f.manifest, Date.now() + 2 * 86_400_000)); + assert.doesNotThrow(() => validateManifest(f.manifest, Date.now() + 2 * 86_400_000, { admission: false })); +}); + + +test('a lost final report preserves known ledger charges while holding the remaining cell allowance', async t => { + const f = await fixture(t, { requests: 1 }); await f.run(); + await rm(join(attemptDirectory(f.manifest, 'attempt-0'), 'result.json')); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.knownUsd, '0.000068000'); assert.equal(summary.committedUsd, '0.014000000'); + assert.equal(summary.incompleteAttempts, 1); +}); + + +test('Claude cells use the assigned Braintrust provider and keep its credential on the controller', async t => { + const f = await fixture(t, { providerProfile: 'claude', route: 'braintrust', requests: 1 }); + let calls = 0; + const result = await f.run(0, async (url, init) => { + calls++; assert.equal(url, 'https://gateway.braintrust.dev/messages'); + const headers = new Headers(init.headers); + assert.equal(headers.get('authorization'), 'Bearer other-braintrust'); + assert.equal(headers.get('x-bt-endpoint-name'), 'ANTHROPIC_API_KEY'); + assert.equal(headers.get('x-bt-project-id'), f.manifest.projectId); + return success('anthropic'); + }); + assert.equal(calls, 1); assert.equal(result.code, 0); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.incompleteAttempts, 0); assert.equal(summary.knownUsd, '0.000068000'); +}); + + +test('existing outputs and physical workspace aliases are rejected before provider work', async t => { + const f = await fixture(t); + let calls = 0; + const transport = async () => { calls++; return success(); }; + await writeFile(f.requests[0].outputPath, 'existing evidence'); + await assert.rejects(f.run(0, transport), /already exists/); + assert.equal(await readFile(f.requests[0].outputPath, 'utf8'), 'existing evidence'); + await rm(f.requests[0].outputPath); + f.manifest.outputDirectory = join(f.requests[0].request.session.workingDirectory, '..evidence'); await f.save(); + await assert.rejects(f.run(0, transport), /outside the disposable/); + f.manifest.outputDirectory = join(f.root, 'aliased-evidence'); + await symlink(f.requests[0].request.session.workingDirectory, f.manifest.outputDirectory); await f.save(); + await assert.rejects(f.run(0, transport), /resolves inside/); + assert.equal(calls, 0); +}); + +for (const missing of ['request.json', 'admission.json', 'receipt']) test(`lost ${missing} keeps known charges and the full allocation`, async t => { + const f = await fixture(t, { requests: 1 }); await f.run(); + const root = attemptDirectory(f.manifest, 'attempt-0'); + const file = missing === 'receipt' ? join(root, 'receipts', (await readdir(join(root, 'receipts')))[0]) : join(root, missing); + await rm(file); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.knownUsd, '0.000068000'); + assert.equal(summary.committedUsd, '0.014000000'); + assert.equal(summary.incompleteAttempts, 1); +}); + +for (const changed of ['request', 'receipt']) test(`changed ${changed} evidence cannot release the allocation`, async t => { + const f = await fixture(t, { requests: 1 }); await f.run(); + const root = attemptDirectory(f.manifest, 'attempt-0'); + const file = changed === 'receipt' ? join(root, 'receipts', (await readdir(join(root, 'receipts')))[0]) : join(root, 'request.json'); + const value = JSON.parse(await readFile(file, 'utf8')); + if (changed === 'request') value.prompt = 'different request'; else value.costUsd = '0.000000000'; + await writeFile(file, JSON.stringify(value)); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.knownUsd, '0.000068000'); + assert.equal(summary.committedUsd, '0.014000000'); + assert.equal(summary.incompleteAttempts, 1); +}); + +for (const replaced of ['runnerCli', 'runnerd']) test(`launches verified bytes after the original ${replaced} path is replaced`, async t => { + const f = await fixture(t, { requests: 1 }); + await writeFile(f.manifest.runnerCli.path, childScript({ requests: 1 }) + "\nassert.equal(await readFile(request.runnerd.path, 'utf8'), 'synthetic daemon');\n"); + f.manifest.runnerCli.sha256 = sha256(await readFile(f.manifest.runnerCli.path)); await f.save(); + Object.defineProperty(f.environment, 'OPENAI_API_KEY', { get() { + writeFileSync(f.manifest[replaced].path, replaced === 'runnerCli' ? "throw new Error('replacement must not execute');" : 'replaced daemon'); + return 'synthetic-upstream-private'; + }}); + const result = await f.run(); + assert.equal(result.code, 0); assert.equal(result.report.cleanupVerified, true); + assert.equal((await summarizeControlledCampaign(f.manifest)).incompleteAttempts, 0); + await assert.rejects(readdir(join(attemptDirectory(f.manifest, 'attempt-0'), 'executables')), { code: 'ENOENT' }); +}); + +test('staged CLI relative imports survive a symlinked evidence directory', async t => { + const f = await fixture(t, { requests: 1 }); + await writeFile(join(f.root, 'helper.mjs'), "export const marker = 'trusted sibling';\n"); + await writeFile(f.manifest.runnerCli.path, "import { marker } from './helper.mjs';\n" + childScript({ requests: 1 }) + "\nassert.equal(marker, 'trusted sibling');\n"); + f.manifest.runnerCli.sha256 = sha256(await readFile(f.manifest.runnerCli.path)); + const alias = join(f.root, 'evidence-alias'); + await mkdir(join(f.root, 'evidence-real')); await symlink(join(f.root, 'evidence-real'), alias); + f.manifest.outputDirectory = join(alias, 'controller'); await f.save(); + const result = await f.run(); + assert.equal(result.code, 0); + assert.equal((await summarizeControlledCampaign(f.manifest)).incompleteAttempts, 0); +}); + +test('cancellation still kills an owned resistant descendant after the CLI exits', async t => { + const f = await fixture(t, { requests: 1, waitForever: true, resistantDescendant: true }); + f.manifest.attempts[0].timeoutMs = 15000; await f.save(); + const abort = new AbortController(); let ready; + const started = new Promise(resolve => { ready = resolve; }); + const running = f.run(0, async () => { ready(); return success(); }, abort.signal); + await started; + const pid = Number(await readFile(join(f.requests[0].request.session.workingDirectory, 'descendant.pid'), 'utf8')); + t.after(() => { try { process.kill(pid, 'SIGKILL'); } catch {} }); + abort.abort(); + const result = await running; + assert.equal(result.report.child.stopped, 'controller_cancelled'); + const absent = () => { try { process.kill(pid, 0); return false; } catch (error) { return error.code === 'ESRCH'; } }; + for (let retry = 0; retry < 100 && !absent(); retry++) await new Promise(resolve => setTimeout(resolve, 10)); + assert.equal(absent(), true); + assert.equal(result.report.cleanupVerified, false); + assert.equal((await summarizeControlledCampaign(f.manifest)).committedUsd, '0.014000000'); +}); diff --git a/packages/paperclip-runner/scripts/controlled-evals/manifest.mjs b/packages/paperclip-runner/scripts/controlled-evals/manifest.mjs new file mode 100644 index 00000000000..0ca1bb655d0 --- /dev/null +++ b/packages/paperclip-runner/scripts/controlled-evals/manifest.mjs @@ -0,0 +1,93 @@ +import { createHash } from 'node:crypto'; +import { createRequire } from 'node:module'; +import { readFile } from 'node:fs/promises'; +import { isAbsolute } from 'node:path'; + +const require = createRequire(new URL('../../../../server/package.json', import.meta.url)); +const { tsImport } = await import(require.resolve('tsx/esm/api')); +export const costRuntime = await tsImport('./shared-cost-runtime.ts', import.meta.url); +export const sha256 = bytes => createHash('sha256').update(bytes).digest('hex'); +export const canonical = value => JSON.stringify(value, (_key, item) => item && typeof item === 'object' && !Array.isArray(item) + ? Object.fromEntries(Object.keys(item).sort().map(key => [key, item[key]])) : item); +export const usd = value => `${value / 1_000_000_000n}.${String(value % 1_000_000_000n).padStart(9, '0')}`; +export function units(value) { + if (typeof value !== 'string' || !/^(?:0|[1-9][0-9]*)(?:\.[0-9]{1,9})?$/.test(value)) throw new Error('Invalid controller USD amount'); + const [whole, fraction = ''] = value.split('.'); + return BigInt(whole) * 1_000_000_000n + BigInt(fraction.padEnd(9, '0')); +} +export function fields(value, names) { + if (!value || typeof value !== 'object' || Array.isArray(value) || Object.keys(value).some(key => !names.includes(key))) throw new Error('Invalid controller fields'); +} +const hash = value => typeof value === 'string' && /^[a-f0-9]{64}$/.test(value); +const id = value => costRuntime.validEvalLedgerId(value); +export function requestBindingSha256(request) { + const copy = structuredClone(request); + if (!copy.session || !copy.runnerd) throw new Error('Invalid controlled request'); + delete copy.session.workingDirectory; + delete copy.runnerd.path; + return sha256(canonical(copy)); +} +export async function readJson(file, maximum = 1_048_576) { + const bytes = await readFile(file); + if (bytes.length > maximum) throw new Error('Controller document exceeds its byte limit'); + return JSON.parse(bytes.toString('utf8')); +} +export function validateManifest(value, now = Date.now(), { admission = true } = {}) { + fields(value, ['schema', 'campaignId', 'maxUsd', 'deadlineAt', 'pricingVerifiedAt', 'pricingExpiresAt', 'overhead', + 'sourceRevision', 'definitionsRevision', 'runnerCli', 'runnerPackage', 'runnerd', 'outputDirectory', 'route', 'projectId', 'models', 'attempts']); + const deadline = Date.parse(value.deadlineAt); + const verified = Date.parse(value.pricingVerifiedAt), expires = Date.parse(value.pricingExpiresAt); + if (value.schema !== 'paperclip.runner-eval.controller/v1' || !id(value.campaignId) + || !Number.isFinite(now) || !Number.isFinite(deadline) || admission && (deadline <= now || deadline - now > 4 * 3_600_000) + || !Number.isFinite(verified) || !Number.isFinite(expires) || verified >= deadline || expires <= verified || admission && verified > now || expires <= deadline || expires - verified > 31 * 86_400_000 + || !/^[a-f0-9]{40}$/.test(value.sourceRevision ?? '') || !/^[a-f0-9]{40}$/.test(value.definitionsRevision ?? '') + || !isAbsolute(value.outputDirectory ?? '') || !['direct', 'braintrust'].includes(value.route) + || value.route === 'braintrust' && !/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(value.projectId ?? '') + || value.route === 'direct' && value.projectId !== undefined) throw new Error('Invalid controller campaign identity or validity window'); + for (const artifact of [value.runnerCli, value.runnerPackage, value.runnerd]) { + fields(artifact, ['path', 'sha256']); + if (!isAbsolute(artifact.path ?? '') || !hash(artifact.sha256)) throw new Error('Invalid controller executable provenance'); + } + const overhead = costRuntime.validateOperatingOverhead(value.overhead, value.maxUsd); + if (admission && Date.parse(value.overhead.verifiedAt) > now || Date.parse(value.overhead.expiresAt) <= deadline + || value.overhead.components.some(component => !component.provisionalBound)) throw new Error('Controller requires current full-lifecycle operating bounds'); + if (!Array.isArray(value.models) || !value.models.length || value.models.length > 16 + || !Array.isArray(value.attempts) || !value.attempts.length || value.attempts.length > 1000) throw new Error('Invalid controlled campaign selection'); + const prices = new Map(); + const rateFields = ['inputUsdPerMillion', 'cachedReadUsdPerMillion', 'outputUsdPerMillion', 'cacheWrite5mUsdPerMillion', 'cacheWrite1hUsdPerMillion', 'cacheWrite30mUsdPerMillion', 'cacheWrite12hUsdPerMillion']; + for (const model of value.models) { + fields(model, ['provider', 'price', 'maximumInputTokens', 'maximumOutputTokens', 'timeoutMs', 'provisionalBound']); + fields(model.price, ['provider', 'model', 'version', 'sourceUrl', 'longContext', ...rateFields]); + if (model.price.longContext) { fields(model.price.longContext, ['aboveInputTokens', 'rates']); fields(model.price.longContext.rates, rateFields); } + const url = new URL(model.price.sourceUrl); + const key = `${model.provider}:${model.price.model}`; + if (!['openai', 'anthropic'].includes(model.provider) || !model.provisionalBound || url.protocol !== 'https:' + || url.username || url.password || url.search || url.hash || prices.has(key)) throw new Error('Invalid controller model pricing'); + prices.set(key, units(costRuntime.evalRequestReservation({ ...model, mode: value.route, apiKey: 'validation' }))); + } + const attempts = new Set(), executions = new Set(); + let allocated = overhead; + for (const attempt of value.attempts) { + fields(attempt, ['executionId', 'attemptId', 'requestSha256', 'provider', 'acpxAgent', 'model', 'maxUsd', 'maxRequests', 'maxConcurrent', 'timeoutMs']); + const provider = attempt.provider === 'codex' && attempt.acpxAgent === undefined ? 'openai' + : attempt.provider === 'acpx' && attempt.acpxAgent === 'claude' ? 'anthropic' : null; + const reservation = prices.get(`${provider}:${attempt.model}`); + if (!provider || !id(attempt.executionId) || !id(attempt.attemptId) || !hash(attempt.requestSha256) + || attempts.has(attempt.attemptId) || executions.has(attempt.executionId) || reservation === undefined + || !Number.isSafeInteger(attempt.timeoutMs) || attempt.timeoutMs < 1 || attempt.timeoutMs > 4 * 3_600_000 + || !Number.isSafeInteger(attempt.maxRequests) || attempt.maxRequests < 1 || attempt.maxRequests > 1000 + || !Number.isSafeInteger(attempt.maxConcurrent) || attempt.maxConcurrent < 1 || attempt.maxConcurrent > 16 + || units(attempt.maxUsd) < reservation) throw new Error('Invalid controlled attempt allocation'); + attempts.add(attempt.attemptId); executions.add(attempt.executionId); allocated += units(attempt.maxUsd); + } + if (allocated > units(value.maxUsd)) throw new Error('Controlled allocations exceed the campaign budget'); + return { manifest: structuredClone(value), manifestSha256: sha256(canonical(value)), allocatedUsd: usd(allocated), overheadUsd: usd(overhead) }; +} +export function bindRequest(manifest, request) { + const attempt = manifest.attempts.find(item => item.attemptId === request.attemptId); + if (!attempt || requestBindingSha256(request) !== attempt.requestSha256 || request.provider !== attempt.provider + || request.acpxAgent !== (attempt.acpxAgent ?? null) && request.acpxAgent !== attempt.acpxAgent + || request.model !== attempt.model || request.runnerd.sha256.replace(/^sha256:/, '') !== manifest.runnerd.sha256 + || request.runnerd.path !== manifest.runnerd.path) throw new Error('Controlled request does not match its allocated identity'); + return attempt; +} diff --git a/packages/paperclip-runner/scripts/controlled-evals/report.mjs b/packages/paperclip-runner/scripts/controlled-evals/report.mjs new file mode 100644 index 00000000000..23e4f996036 --- /dev/null +++ b/packages/paperclip-runner/scripts/controlled-evals/report.mjs @@ -0,0 +1,84 @@ +import { readFile } from 'node:fs/promises'; +import { join, resolve } from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { canonical, costRuntime, readJson, requestBindingSha256, sha256, units, usd, validateManifest } from './manifest.mjs'; +import { attemptDirectory } from './run.mjs'; + +export async function summarizeControlledCampaign(value) { + const { manifest, manifestSha256, allocatedUsd, overheadUsd } = validateManifest(value, Date.now(), { admission: false }); + let known = 0n, committed = units(overheadUsd); + const attempts = []; + for (const allocation of manifest.attempts) { + const root = attemptDirectory(manifest, allocation.attemptId); + let snapshot, report, verified = false; + try { + const ledgerBytes = await readFile(join(root, 'ledger.jsonl')); + const ledger = await costRuntime.readEvalCostLedger(join(root, 'ledger.jsonl')); + const candidateSnapshot = ledger.state.snapshot(); + const ledgerDeadline = Date.parse(ledger.state.limits.deadlineAt); + const fixedLimits = { campaignId: manifest.campaignId, maxUsd: allocation.maxUsd, deadlineAt: ledger.state.limits.deadlineAt, + maxRequests: allocation.maxRequests, maxConcurrent: allocation.maxConcurrent, maxAttemptsPerExecution: 1 }; + if (canonical(ledger.state.limits) !== canonical(fixedLimits) || !Number.isFinite(ledgerDeadline) + || ledgerDeadline <= Date.parse(manifest.pricingVerifiedAt) || ledgerDeadline > Date.parse(manifest.deadlineAt) + || canonical(candidateSnapshot.admittedAttempts) !== canonical([{ executionId: allocation.executionId, attemptId: allocation.attemptId }]) + || candidateSnapshot.requests.some(request => request.executionId !== allocation.executionId || request.attemptId !== allocation.attemptId || request.phase !== 'agent')) throw new Error('Ledger identity does not match its allocation'); + snapshot = candidateSnapshot; + const admission = await readJson(join(root, 'admission.json')); + const started = Date.parse(admission.startedAt); + const expectedLimits = { campaignId: manifest.campaignId, maxUsd: allocation.maxUsd, + deadlineAt: new Date(Math.min(Date.parse(manifest.deadlineAt), started + allocation.timeoutMs)).toISOString(), + maxRequests: allocation.maxRequests, maxConcurrent: allocation.maxConcurrent, maxAttemptsPerExecution: 1 }; + if (admission.schema !== 'paperclip.runner-eval.admission/v1' || admission.manifestSha256 !== manifestSha256 + || admission.campaignId !== manifest.campaignId || canonical(admission.attempt) !== canonical(allocation) + || canonical(admission.limits) !== canonical(expectedLimits) || canonical(ledger.state.limits) !== canonical(expectedLimits)) throw new Error('Ledger does not match its admission'); + report = await readJson(join(root, 'result.json')); + const request = await readJson(join(root, 'request.json'), 8_388_608); + if (requestBindingSha256(request) !== allocation.requestSha256 || request.runnerd.sha256.replace(/^sha256:/, '') !== manifest.runnerd.sha256 + || canonical(Object.keys(report.receiptHashes ?? {}).sort()) !== canonical(snapshot.requests.map(item => item.requestId).sort())) throw new Error('Retained request evidence does not match'); + for (const item of snapshot.requests) { + if (!/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/.test(item.requestId)) throw new Error('Invalid retained receipt identity'); + const bytes = await readFile(join(root, 'receipts', `${item.requestId}.json`)); + const receipt = JSON.parse(bytes.toString('utf8')); + if (sha256(bytes) !== report.receiptHashes[item.requestId] || receipt.requestId !== item.requestId + || receipt.route !== manifest.route || receipt.costUsd !== item.costUsd) throw new Error('Retained receipt does not match its ledger'); + } + const artifactBytes = await readFile(join(root, 'provider-artifact.json')); + const artifact = JSON.parse(artifactBytes.toString('utf8')); + const descriptorBytes = await readFile(join(root, 'gateway.json')); + const descriptor = JSON.parse(descriptorBytes.toString('utf8')); + if (report.schema !== 'paperclip.runner-eval.control-result/v1' || report.manifestSha256 !== manifestSha256 + || report.campaignId !== manifest.campaignId || report.executionId !== allocation.executionId || report.attemptId !== allocation.attemptId + || report.requestSha256 !== allocation.requestSha256 || report.ledgerSha256 !== sha256(ledgerBytes) + || report.providerArtifactSha256 !== sha256(artifactBytes) || canonical(report.accounting) !== canonical(snapshot) + || report.status !== 'closed' || report.cleanupVerified !== true || report.secretExcluded !== true || report.failure !== null + || report.startedAt !== admission.startedAt || !Number.isFinite(Date.parse(report.finishedAt)) || Date.parse(report.finishedAt) < started + || !report.child || report.child.stopped !== null || artifact.snapshot?.status !== 'closed' || artifact.snapshot?.process?.runnerExited !== true + || descriptor.schema !== 'paperclip.runner-eval.gateway/v1' || descriptor.provider !== (allocation.provider === 'codex' ? 'openai' : 'anthropic') + || descriptor.model !== allocation.model || descriptor.upstreamRoute !== manifest.route + || descriptor.campaignId !== manifest.campaignId || descriptor.executionId !== allocation.executionId || descriptor.attemptId !== allocation.attemptId + || canonical(artifact.requestRouting) !== canonical({ mode: 'controller_gateway', provider: descriptor.provider, declaredUpstreamRoute: manifest.route, + campaignId: manifest.campaignId, executionId: allocation.executionId, attemptId: allocation.attemptId, descriptorSha256: sha256(descriptorBytes) }) + || artifact.attemptId !== allocation.attemptId || artifact.requestedModel !== allocation.model + || snapshot.inflight || snapshot.conflictingRequests || snapshot.overBudget + || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests) throw new Error('Incomplete controller evidence'); + verified = true; + } catch {} + const cost = snapshot ? units(snapshot.knownUsd) : 0n; + const observedCommitment = snapshot ? units(snapshot.committedUsd) : 0n; + const commitment = verified ? observedCommitment : observedCommitment > units(allocation.maxUsd) ? observedCommitment : units(allocation.maxUsd); + known += cost; committed += commitment; + attempts.push({ executionId: allocation.executionId, attemptId: allocation.attemptId, status: verified ? 'retained' : 'incomplete', + allocationUsd: allocation.maxUsd, knownUsd: usd(cost), heldUsd: usd(commitment - cost), committedUsd: usd(commitment), + unknownRequests: snapshot?.unknownRequests ?? null, boundedUnresolvedRequests: snapshot?.boundedUnresolvedRequests ?? null }); + } + return { schema: 'paperclip.runner-eval.control-campaign/v1', campaignId: manifest.campaignId, manifestSha256, + limitUsd: manifest.maxUsd, allocatedUsd, operatingHeldUsd: overheadUsd, knownUsd: usd(known), heldUsd: usd(committed - known), + committedUsd: usd(committed), overBudget: committed > units(manifest.maxUsd), incompleteAttempts: attempts.filter(attempt => attempt.status === 'incomplete').length, + costStatus: attempts.some(attempt => attempt.status === 'incomplete') || committed > units(manifest.maxUsd) ? 'incomplete' : 'provisional', attempts }; +} +if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) { + if (process.argv.length !== 3) throw new Error('Usage: report.mjs '); + const result = await summarizeControlledCampaign(await readJson(process.argv[2])); + process.stdout.write(JSON.stringify(result, null, 2) + '\n'); + if (result.incompleteAttempts || result.overBudget) process.exitCode = 2; +} diff --git a/packages/paperclip-runner/scripts/controlled-evals/run.mjs b/packages/paperclip-runner/scripts/controlled-evals/run.mjs new file mode 100644 index 00000000000..42fda084873 --- /dev/null +++ b/packages/paperclip-runner/scripts/controlled-evals/run.mjs @@ -0,0 +1,193 @@ +import { spawn } from 'node:child_process'; +import { once } from 'node:events'; +import { lstat, mkdir, open, readFile, realpath, rename, rm, writeFile } from 'node:fs/promises'; +import { dirname, isAbsolute, join, relative, resolve, sep } from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { parseArgs } from 'node:util'; +import { bindRequest, canonical, costRuntime, readJson, sha256, validateManifest } from './manifest.mjs'; + +async function atomicJson(file, value) { + const temporary = `${file}.tmp`; + const handle = await open(temporary, 'wx', 0o600); + try { await handle.writeFile(JSON.stringify(value, null, 2) + '\n'); await handle.sync(); } + finally { await handle.close(); } + await rename(temporary, file); +} +const absent = pid => { try { process.kill(pid, 0); return false; } catch (error) { return error.code === 'ESRCH'; } }; +export const attemptDirectory = (manifest, attemptId) => join(manifest.outputDirectory, sha256(attemptId)); + +async function runChild(input) { + const child = spawn(process.execPath, ['--import', input.loader, input.cli, '--request', input.requestPath, '--output', input.artifactPath, '--gateway-config', input.descriptorPath], { + cwd: input.cwd, env: input.environment, detached: true, stdio: ['ignore', 'pipe', 'pipe'], + }); + let stopped = null, force, escalation; + const output = { stdout: [], stderr: [] }, sizes = { stdout: 0, stderr: 0 }; + const stop = reason => { + if (stopped) return; + stopped = reason; + void input.closeGateway().catch(() => {}); + try { process.kill(-child.pid, 'SIGTERM'); } catch {} + escalation = new Promise(resolve => { + force = setTimeout(() => { try { process.kill(-child.pid, 'SIGKILL'); } catch {} finally { resolve(); } }, 5000); + }); + }; + for (const stream of ['stdout', 'stderr']) child[stream].on('data', bytes => { + const available = Math.max(0, 1_048_576 - sizes[stream]); + output[stream].push(bytes.subarray(0, available)); sizes[stream] += bytes.length; + if (sizes[stream] > 1_048_576) stop('output_limit'); + }); + const onAbort = () => stop('controller_cancelled'); + input.signal?.addEventListener('abort', onAbort, { once: true }); + const timeout = setTimeout(() => stop('attempt_deadline'), Math.max(1, input.deadlineAt - Date.now())); + if (input.signal?.aborted) onAbort(); + try { + const [code, signal] = await once(child, 'exit'); + if (stopped) { + let groupPresent = true; + try { process.kill(-child.pid, 0); } catch (error) { if (error.code === 'ESRCH') groupPresent = false; } + if (groupPresent) await escalation; + } + return { code, signal, pid: child.pid, stopped, stdout: Buffer.concat(output.stdout).toString(), stderr: Buffer.concat(output.stderr).toString() }; + } finally { + clearTimeout(timeout); clearTimeout(force); input.signal?.removeEventListener('abort', onAbort); + } +} + +export async function runControlledEval({ manifestPath, requestPath, outputPath, environment = process.env, signal, transport }) { + if (!['linux', 'darwin'].includes(process.platform)) throw new Error('Controlled eval process ownership requires Linux or macOS'); + signal?.throwIfAborted(); + const { manifest, manifestSha256 } = validateManifest(await readJson(manifestPath)); + const request = await readJson(requestPath, 8_388_608), attempt = bindRequest(manifest, request); + try { await lstat(outputPath); throw new Error('Controlled output already exists'); } + catch (error) { if (error.code !== 'ENOENT') throw error; } + if (request.session.workingDirectory) { + const suffix = relative(resolve(request.session.workingDirectory), resolve(manifest.outputDirectory)); + if (!suffix || suffix !== '..' && !suffix.startsWith(`..${sep}`) && !isAbsolute(suffix)) throw new Error('Controller evidence must be outside the disposable workspace'); + } + const originalCliUrl = pathToFileURL(await realpath(manifest.runnerCli.path)).href; + const verified = {}; + for (const name of ['runnerCli', 'runnerPackage', 'runnerd']) { + const artifact = manifest[name], bytes = await readFile(artifact.path); + if (sha256(bytes) !== artifact.sha256) throw new Error('Controlled eval executable provenance changed'); + if (name !== 'runnerPackage') verified[name] = bytes; + } + const provider = attempt.provider === 'codex' ? 'openai' : 'anthropic'; + const keyName = manifest.route === 'braintrust' ? 'BRAINTRUST_API_KEY' : provider === 'openai' ? 'OPENAI_API_KEY' : 'ANTHROPIC_API_KEY'; + const key = environment[keyName]; + if (!key || /[\r\n]/.test(key)) throw new Error('Missing controlled eval upstream credential'); + const model = manifest.models.find(item => item.provider === provider && item.price.model === attempt.model); + const root = attemptDirectory(manifest, attempt.attemptId); + await mkdir(manifest.outputDirectory, { recursive: true, mode: 0o700 }); + if (request.session.workingDirectory) { + const suffix = relative(await realpath(request.session.workingDirectory), await realpath(manifest.outputDirectory)); + if (!suffix || suffix !== '..' && !suffix.startsWith(`..${sep}`) && !isAbsolute(suffix)) throw new Error('Controller evidence resolves inside the disposable workspace'); + } + await mkdir(root, { mode: 0o700 }); + const started = Date.now(), startedAt = new Date(started).toISOString(); + const deadlineAt = Math.min(Date.parse(manifest.deadlineAt), started + attempt.timeoutMs); + const limits = { campaignId: manifest.campaignId, maxUsd: attempt.maxUsd, deadlineAt: new Date(deadlineAt).toISOString(), + maxRequests: attempt.maxRequests, maxConcurrent: attempt.maxConcurrent, maxAttemptsPerExecution: 1 }; + await atomicJson(join(root, 'admission.json'), { schema: 'paperclip.runner-eval.admission/v1', campaignId: manifest.campaignId, + manifestSha256, attempt, limits, startedAt }); + const executables = join(root, 'executables'); + await mkdir(executables, { mode: 0o700 }); + const stagedCli = join(executables, 'entry.mjs'), stagedRunnerd = join(executables, 'runnerd'), loader = join(executables, 'imports.mjs'); + await writeFile(stagedCli, verified.runnerCli, { flag: 'wx', mode: 0o400 }); + await writeFile(stagedRunnerd, verified.runnerd, { flag: 'wx', mode: 0o500 }); + const stagedUrl = JSON.stringify(pathToFileURL(await realpath(stagedCli)).href), originalUrl = JSON.stringify(originalCliUrl); + await writeFile(loader, `import { registerHooks } from 'node:module'; +registerHooks({ resolve(specifier, context, nextResolve) { + return nextResolve(specifier, context.parentURL === ${stagedUrl} ? { ...context, parentURL: ${originalUrl} } : context); +} });\n`, { flag: 'wx', mode: 0o400 }); + const retainedRequestPath = join(root, 'request.json'); + const childRequest = structuredClone(request); childRequest.runnerd.path = stagedRunnerd; + await writeFile(retainedRequestPath, JSON.stringify(childRequest) + '\n', { flag: 'wx', mode: 0o600 }); + await mkdir(join(root, 'receipts'), { mode: 0o700 }); + await mkdir(join(root, 'home'), { mode: 0o700 }); + await mkdir(join(root, 'tmp'), { mode: 0o700 }); + const ledgerPath = join(root, 'ledger.jsonl'); + const ledger = await costRuntime.openEvalCostLedger(ledgerPath, limits); + let gateway, child, artifact, failure = null, cleanupVerified = false, secretExcluded = true; + let providerArtifactSha256; + const secrets = [key]; + const redact = raw => secrets.reduce((text, secret) => text.split(secret).join('[redacted]'), raw); + try { + await ledger.admitAttempt(attempt.executionId, attempt.attemptId); + gateway = await costRuntime.startEvalGateway({ ledger, identity: { executionId: attempt.executionId, attemptId: attempt.attemptId, phase: 'agent' }, + route: { ...model, mode: manifest.route, projectId: manifest.projectId, endpointName: provider === 'openai' ? 'OPENAI_API_KEY' : 'ANTHROPIC_API_KEY', apiKey: key }, + deadlineAt, transport, persist: receipt => atomicJson(join(root, 'receipts', `${receipt.requestId}.json`), receipt), + }); + secrets.push(gateway.token); + const descriptor = { schema: 'paperclip.runner-eval.gateway/v1', provider, model: attempt.model, baseUrl: gateway.baseUrl, + campaignId: manifest.campaignId, executionId: attempt.executionId, attemptId: attempt.attemptId, upstreamRoute: manifest.route }; + const descriptorPath = join(root, 'gateway.json'), descriptorRaw = JSON.stringify(descriptor) + '\n'; + await writeFile(descriptorPath, descriptorRaw, { flag: 'wx', mode: 0o600 }); + const childEnvironment = Object.fromEntries(['PATH', 'LANG', 'LANGUAGE', 'LC_ALL', 'TZ'].flatMap(name => environment[name] === undefined ? [] : [[name, environment[name]]])); + Object.assign(childEnvironment, { HOME: join(root, 'home'), TMPDIR: join(root, 'tmp'), TMP: join(root, 'tmp'), TEMP: join(root, 'tmp'), PAPERCLIP_RUNNER_EVAL_GATEWAY_TOKEN: gateway.token }); + const artifactPath = join(root, 'provider-artifact.json'); + signal?.throwIfAborted(); + if (Date.now() >= deadlineAt) throw new Error('Controlled attempt deadline reached before spawn'); + child = await runChild({ cli: stagedCli, loader, requestPath: retainedRequestPath, artifactPath, descriptorPath, + cwd: request.session.workingDirectory ?? root, environment: childEnvironment, deadlineAt, signal, closeGateway: () => gateway.close() }); + for (const stream of ['stdout', 'stderr']) await writeFile(join(root, `${stream}.log`), redact(child[stream]), { flag: 'wx', mode: 0o600 }); + const raw = await readFile(artifactPath, 'utf8'); + if (Buffer.byteLength(raw) > 16_777_216) throw new Error('Provider artifact exceeds its byte limit'); + secretExcluded = secrets.every(secret => !raw.includes(secret)); + if (!secretExcluded) await writeFile(artifactPath, redact(raw), { mode: 0o600 }); + providerArtifactSha256 = sha256(redact(raw)); + artifact = JSON.parse(redact(raw)); + const expected = { mode: 'controller_gateway', provider, declaredUpstreamRoute: manifest.route, campaignId: manifest.campaignId, + executionId: attempt.executionId, attemptId: attempt.attemptId, descriptorSha256: sha256(descriptorRaw) }; + if (canonical(artifact.requestRouting) !== canonical(expected) || artifact.attemptId !== attempt.attemptId || artifact.requestedModel !== attempt.model) throw new Error('Provider did not retain the assigned route'); + const processInfo = artifact.snapshot?.process; + const pids = [...new Set([child.pid, processInfo?.runnerPid, processInfo?.providerPid, processInfo?.sidecarPid, processInfo?.agentPid].filter(pid => Number.isSafeInteger(pid) && pid > 0))]; + cleanupVerified = !child.stopped && artifact.snapshot?.status === 'closed' && processInfo?.runnerExited === true + && Number.isSafeInteger(processInfo.runnerPid) && processInfo.runnerPid > 0 + && Number.isSafeInteger(processInfo.providerPid) && processInfo.providerPid > 0 && pids.every(absent); + if (!cleanupVerified || !secretExcluded) throw new Error('Controlled provider cleanup or credential exclusion is unverified'); + } catch { + failure = 'controlled_attempt_incomplete'; + } finally { + try { await gateway?.close(); } catch { failure = 'gateway_cleanup_unverified'; } + try { await ledger.close(); } catch { failure = 'ledger_close_failed'; } + } + if (cleanupVerified) { + try { await rm(executables, { recursive: true }); } + catch { cleanupVerified = false; failure ??= 'executable_cleanup_unverified'; } + } + const snapshot = ledger.snapshot(); + if (snapshot.overBudget || snapshot.conflictingRequests || snapshot.inflight || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests) failure ??= 'request_accounting_unbounded'; + const receiptHashes = {}; + try { + for (const request of snapshot.requests) receiptHashes[request.requestId] = sha256(await readFile(join(root, 'receipts', `${request.requestId}.json`))); + } catch { failure ??= 'request_receipt_missing'; } + const ledgerSha256 = sha256(await readFile(ledgerPath)); + const report = { schema: 'paperclip.runner-eval.control-result/v1', campaignId: manifest.campaignId, manifestSha256, + executionId: attempt.executionId, attemptId: attempt.attemptId, requestSha256: attempt.requestSha256, + allocationUsd: attempt.maxUsd, limits, startedAt, finishedAt: new Date().toISOString(), status: failure ? 'incomplete' : 'closed', failure, + child: child ? { code: child.code, signal: child.signal, stopped: child.stopped } : null, + cleanupVerified, secretExcluded, providerArtifactSha256, ledgerSha256, receiptHashes, accounting: snapshot }; + await atomicJson(join(root, 'result.json'), report); + const resultArtifact = artifact ?? { schema: 'paperclip-runner/eval-session-artifact/v1', attemptId: attempt.attemptId, + requestedModel: attempt.model, infrastructureError: 'Controlled provider did not retain an artifact' }; + if (failure) Object.assign(resultArtifact, { accountingError: 'Controlled request accounting or cleanup is incomplete', + accountingFailure: { class: 'provider_budget_coverage_unknown', category: 'provider_budget', retryable: false } }); + resultArtifact.controllerAccounting = report; + await mkdir(dirname(resolve(outputPath)), { recursive: true, mode: 0o700 }); + await writeFile(outputPath, JSON.stringify(resultArtifact, null, 2) + '\n', { flag: 'wx', mode: 0o600 }); + return { code: failure ? 2 : child?.code ?? 2, report }; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) { + const controller = new AbortController(); + const cancel = () => controller.abort(); + process.once('SIGINT', cancel); process.once('SIGTERM', cancel); + try { + const { values } = parseArgs({ options: { request: { type: 'string' }, output: { type: 'string' } } }); + if (!values.request || !values.output || !process.env.PAPERCLIP_RUNNER_EVAL_CONTROL_MANIFEST) throw new Error('A controlled manifest, request and output are required'); + const result = await runControlledEval({ manifestPath: process.env.PAPERCLIP_RUNNER_EVAL_CONTROL_MANIFEST, + requestPath: values.request, outputPath: values.output, signal: controller.signal }); + process.exitCode = result.code; + } catch { process.stderr.write('Controlled Runner eval admission or retention failed; inspect retained controller evidence.\n'); process.exitCode = 2; } + finally { process.removeListener('SIGINT', cancel); process.removeListener('SIGTERM', cancel); } +} diff --git a/packages/paperclip-runner/scripts/controlled-evals/shared-cost-runtime.ts b/packages/paperclip-runner/scripts/controlled-evals/shared-cost-runtime.ts new file mode 100644 index 00000000000..38f45e0839a --- /dev/null +++ b/packages/paperclip-runner/scripts/controlled-evals/shared-cost-runtime.ts @@ -0,0 +1,3 @@ +export { EvalSpendState, openEvalCostLedger, readEvalCostLedger, validEvalLedgerId } from '../../../../tests/runner-e2e/cost-ledger.js'; +export { evalRequestReservation, startEvalGateway } from '../../../../tests/runner-e2e/eval-gateway.js'; +export { EVAL_OVERHEAD_COMPONENTS, validateOperatingOverhead } from '../../../../tests/runner-e2e/operating-overhead.js'; diff --git a/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts b/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts index cd1cb1e03e1..0ef09abd57f 100644 --- a/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts +++ b/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts @@ -73,6 +73,7 @@ it.each([["codex", undefined], ["acpx", "claude"]])("routes %s/%s with only its expect(config).toContain('env_key = "PAPERCLIP_AI_PROVIDER_KEY"'); expect(config).toContain("requires_openai_auth = false"); expect(config).toContain("shell_snapshot = false"); + expect(config).toContain('web_search = "disabled"'); expect(config).not.toContain(token); expect((await stat(join(env.CODEX_HOME!, "config.toml"))).mode & 0o777).toBe(0o600); } diff --git a/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts b/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts index 0d32670c3a6..a4e27f9c180 100644 --- a/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts +++ b/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts @@ -70,6 +70,7 @@ export async function prepareEvalGatewayRuntime( environment.PAPERCLIP_AI_PROVIDER_KEY = token; await writeFile(join(codexHome, "config.toml"), [ 'model_provider = "paperclip_eval"', + 'web_search = "disabled"', `model = ${JSON.stringify(request.model)}`, '[model_providers.paperclip_eval]', 'name = "Paperclip eval controller"', diff --git a/tests/runner-e2e/eval-gateway.test.ts b/tests/runner-e2e/eval-gateway.test.ts index 6e38d356bea..aefca15a9e3 100644 --- a/tests/runner-e2e/eval-gateway.test.ts +++ b/tests/runner-e2e/eval-gateway.test.ts @@ -121,6 +121,29 @@ it.each([ await expect(forwardEvalRequest({ ...f, payload, transport })).rejects.toThrow(); expect(transport).not.toHaveBeenCalled(); expect(f.ledger.snapshot().requests).toHaveLength(0); }); +it("admits local function and custom tools inside a Codex namespace", async () => { + const f = await fixture(); + const tools = [{ type: "namespace", name: "local", tools: [ + { type: "function", name: "read_task", parameters: { type: "object" } }, + { type: "custom", name: "apply_patch" }, + ] }]; + const transport = vi.fn(async (_url, init) => { + expect(JSON.parse(init!.body as string).tools).toEqual(tools); + return response(); + }); + const result = await forwardEvalRequest({ ...f, payload: { ...f.payload, tools }, transport }); + expect(result.outcome).toBe("succeeded"); + expect(transport).toHaveBeenCalledOnce(); +}); +it.each([ + { type: "namespace", name: "local", tools: [{ type: "web_search" }] }, + { type: "namespace", name: "local", tools: [{ type: "namespace", name: "nested", tools: [{ type: "web_search" }] }] }, + { type: "namespace", name: "local" }, +])("rejects unqualified tools inside namespaces before reservation", async tool => { + const f = await fixture(), transport = vi.fn(); + await expect(forwardEvalRequest({ ...f, payload: { ...f.payload, tools: [tool] }, transport })).rejects.toThrow(); + expect(transport).not.toHaveBeenCalled(); expect(f.ledger.snapshot().requests).toHaveLength(0); +}); it("does not dispatch when the private pre-request receipt cannot be persisted", async () => { const f = await fixture(), transport = vi.fn(); await expect(forwardEvalRequest({ ...f, persist: async () => { throw new Error("disk unavailable"); }, transport })).rejects.toThrow("disk unavailable"); diff --git a/tests/runner-e2e/eval-gateway.ts b/tests/runner-e2e/eval-gateway.ts index ecd9b55ede8..d7cc20ab0db 100644 --- a/tests/runner-e2e/eval-gateway.ts +++ b/tests/runner-e2e/eval-gateway.ts @@ -71,8 +71,12 @@ function boundedPayload(value: unknown, route: EvalGatewayRoute) { if (body.background === true || body.mcp_servers !== undefined || body.service_tier !== undefined && !["auto", "default", "standard_only"].includes(String(body.service_tier))) throw new EvalRequestError("unqualified_billing_mode"); if (body.speed !== undefined && body.speed !== "standard" || body.inference_geo !== undefined) throw new EvalRequestError("unqualified_billing_mode"); if (body.tools !== undefined && (!Array.isArray(body.tools) || body.tools.some(tool => { - const type = row(tool).type; - return route.provider === "openai" ? !["function", "custom"].includes(String(type)) : type !== undefined && type !== "custom"; + const definition = row(tool), type = definition.type; + if (route.provider !== "openai") return type !== undefined && type !== "custom"; + if (type === "namespace") return typeof definition.name !== "string" || !definition.name + || !Array.isArray(definition.tools) || !definition.tools.length + || definition.tools.some(nested => !["function", "custom"].includes(String(row(nested).type))); + return !["function", "custom"].includes(String(type)); }))) throw new EvalRequestError("unqualified_hosted_tool"); const inspect = (item: unknown): void => { if (Array.isArray(item)) { item.forEach(inspect); return; } From 5d2515e520f9aefb2ea83accf480090f99fdd943 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 26 Oct 2026 08:11:17 -0500 Subject: [PATCH 003/133] Simplify agent harness configuration choices (#17722) ## Thinking Path > - Paperclip manages AI agents and their work. > - Users choose a harness in agent setup and configuration. > - Supported harnesses already determine their native Runner profile. > - A second ACP agent selector lets users make a conflicting choice. > - The harness list also shows unavailable options marked Coming soon. > - This PR removes these extra choices and keeps each harness's settings. ## Linked Issues or Issue Description Refs #17142 and #17120. Related runner defaults work: #15422. **What existing behavior does this improve?** The Harness dropdown and native agent configuration form. **Current behavior** A Pi or Claude agent shows an ACP agent selector below Harness. Users can select a different agent in that second control. The Harness list also includes disabled Coming soon entries. **Proposed behavior** Use Harness as the only agent choice when it determines the Runner profile. Hide unavailable entries from its list. Keep the generic advanced Runner selector available where no parent harness determines the profile. **Reason and benefit** Users choose the harness once. The form no longer offers a conflicting ACP agent choice or an option they cannot use. **Breaking changes** No API or stored configuration changes. Unavailable harnesses are omitted from the list; existing saved values still display. ## What Changed - Hide the ACP agent selector when the parent Harness control determines the Runner profile. - Remove Coming soon options and their unused disabled-state rendering from the Harness dropdown. - Preserve saved Process and HTTP selections during unrelated edits while omitting them from new choices. - Document the field visibility contract. - Cover new and saved Claude, Pi, Codex, and OpenCode forms. Check that Pi thinking controls remain and an unrelated name edit does not rewrite Runner configuration. ## Verification - Passed after rebasing onto master: 199 tests across AgentConfigForm.render, codex-local/config-fields, and adapter config sections. - Passed: UI TypeScript check, UI production build, token gates, and git diff --check. - Live browser verification passed for the Pi configuration page before and after rebase. Harness remains Pi, ACP agent is absent, and the original model and thinking level remain. The open Harness list contains no Coming soon, Process, or HTTP entries. - Dropdown regression cases open Harness, check the supported options, exclude Coming soon/Process/HTTP options, and preserve saved Process and HTTP configuration through a name-only save. - Passed on final commit b60a02387f: all 57 check runs are terminal and clean (53 success, 2 neutral, 2 skipped). The Snyk status is successful. Greptile and Paperclip review both give 5/5, with no unresolved review threads. ## Risks - Low risk: this change only removes redundant or unavailable UI choices. Runner routing and saved settings keep their existing contracts. - The advanced generic Runner form retains ACP profile selection. It has no parent harness choice to determine that profile. - No new agent task or provider call was needed to verify these form changes. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, command execution, and browser use. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- ui/src/adapters/codex-local/config-fields.tsx | 2 +- ui/src/adapters/types.ts | 1 + .../AgentConfigForm.render.test.tsx | 45 +++++++++++++++++++ ui/src/components/AgentConfigForm.tsx | 5 +-- 4 files changed, 49 insertions(+), 4 deletions(-) diff --git a/ui/src/adapters/codex-local/config-fields.tsx b/ui/src/adapters/codex-local/config-fields.tsx index d691b30c5c2..3ccf6abacf4 100644 --- a/ui/src/adapters/codex-local/config-fields.tsx +++ b/ui/src/adapters/codex-local/config-fields.tsx @@ -309,7 +309,7 @@ export function CodexLocalConfigFields({ updateRunnerSchemaValue("allowUnmeteredProvider", value)} /> } - {runnerManaged && runnerProvider === "acpx" && runnerSchemaValue("acpxAgent", "claude") !== "grok" && ( + {runnerManaged && !hideRunnerHarness && runnerProvider === "acpx" && runnerSchemaValue("acpxAgent", "claude") !== "grok" && ( + + + + + {data.options.map((option) => ( + + {option.login} · {option.connectionName} + + ))} + + + + {data.grantId && ( + + )} + + )} + {data && !data.options.length && ( +

+ Connect your GitHub account and install the Connector for this company to authorize it. +

+ )} + {(authorization.error || save.error) && ( +

+ {(authorization.error || save.error)?.message} +

+ )} + + ); +} diff --git a/ui/src/components/ProjectProperties.tsx b/ui/src/components/ProjectProperties.tsx index a1c631c1546..cdaffaa5d5e 100644 --- a/ui/src/components/ProjectProperties.tsx +++ b/ui/src/components/ProjectProperties.tsx @@ -7,6 +7,7 @@ import { environmentDisplayLabel, filterManagedSandboxSelectableEnvironments } f import { Link } from "@/lib/router"; import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; import type { Project, SharedWorkspaceConcurrency } from "@paperclipai/shared"; +import { ProjectGitAuthorization } from "./ProjectGitAuthorization"; import { ProjectRepositories } from "./ProjectRepositories"; import { cn, formatDate } from "../lib/utils"; import { environmentsApi } from "../api/environments"; @@ -510,6 +511,7 @@ export function ProjectProperties({ project, repositories, onUpdate, onFieldUpda {repositories ?? } + } alignStart From e61ee4b005f8e13f64af15f69cb62305dd869f82 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Mon, 26 Oct 2026 18:02:15 -0700 Subject: [PATCH 037/133] fix(runner): preserve Claude user settings trust in HOME (#17826) --- packages/paperclip-runner/README.md | 5 ++- .../test/acpx-codex-package-contract.test.mjs | 43 ++++++++++++++++++- ...entprotocol__claude-agent-acp@0.73.0.patch | 16 +++++++ 3 files changed, 61 insertions(+), 3 deletions(-) diff --git a/packages/paperclip-runner/README.md b/packages/paperclip-runner/README.md index 243b7928bba..51823ae1d55 100644 --- a/packages/paperclip-runner/README.md +++ b/packages/paperclip-runner/README.md @@ -818,7 +818,10 @@ rules and hooks apply. Native Codex tools can read and modify the execution home (read-only for planning); credentials in that home are not isolated from tools. Native Codex separately denies Runner-owned protocol state; ACPX Claude retains its existing provider trust boundary. Execution is unattended: native Codex uses `never`, and -ACPX rejects requests for approval. The initial ACPX mode requires the Runner +ACPX rejects requests for approval. When Claude starts in its home directory, +its native user settings retain user trust even if the SDK also reports that +same file as project settings. Other project settings keep the native trust filter. +The initial ACPX mode requires the Runner setting `acpxPermissionMode: "approve-all"` (its default); other Runner modes are unqualified here and fail before launch. Local permission rules still apply. Preconfigured ACPX read-only/planning tasks also fail before provider launch diff --git a/packages/paperclip-runner/test/acpx-codex-package-contract.test.mjs b/packages/paperclip-runner/test/acpx-codex-package-contract.test.mjs index 3cd71aa3198..c70486fbdac 100644 --- a/packages/paperclip-runner/test/acpx-codex-package-contract.test.mjs +++ b/packages/paperclip-runner/test/acpx-codex-package-contract.test.mjs @@ -1,8 +1,11 @@ import assert from "node:assert/strict"; import { createHash } from "node:crypto"; -import { readFile } from "node:fs/promises"; +import { execFileSync } from "node:child_process"; +import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises"; import { createRequire } from "node:module"; -import { dirname, resolve } from "node:path"; +import { dirname, join, resolve } from "node:path"; +import { tmpdir } from "node:os"; +import { pathToFileURL } from "node:url"; import test from "node:test"; const runnerPackage = JSON.parse( @@ -217,3 +220,39 @@ test("the Claude patch removes ambient project and local configuration", () => { ); } }); + +test("Claude preserves user permission modes when its working directory is home", async (t) => { + const packagePath = createRequire(import.meta.url).resolve("@agentclientprotocol/claude-agent-acp/package.json"); + const settingsModule = pathToFileURL(resolve(dirname(packagePath), "dist/settings.js")).href; + const root = await mkdtemp(join(tmpdir(), "claude-settings-origin-")); + t.after(() => rm(root, { recursive: true, force: true })); + const home = join(root, "home"); + const project = join(root, "project"); + await mkdir(join(home, ".claude"), { recursive: true }); + await mkdir(join(project, ".claude"), { recursive: true }); + const userFile = join(home, ".claude/settings.json"); + const env = { ...process.env, HOME: home }; + delete env.CLAUDE_CONFIG_DIR; + delete env.PAPERCLIP_ACPX_ISOLATED_CONTEXT; + const inspect = (cwd) => JSON.parse(execFileSync(process.execPath, ["--input-type=module", "-e", ` + import { SettingsManager } from ${JSON.stringify(settingsModule)}; + const manager = new SettingsManager(${JSON.stringify(cwd)}); + try { + await manager.initialize(); + console.log(JSON.stringify(manager.getSettings())); + } finally { manager.dispose(); } + `], { env, encoding: "utf8", timeout: 10_000 })); + for (const defaultMode of ["bypassPermissions", "acceptEdits", "auto", "plan", "default"]) { + const content = JSON.stringify({ permissions: { defaultMode, deny: ["Read(./private)"] } }); + await writeFile(userFile, content); + assert.deepEqual(inspect(home).permissions, { defaultMode, deny: ["Read(./private)"] }); + assert.equal(await readFile(userFile, "utf8"), content, "discovery does not rewrite the profile"); + } + await writeFile(userFile, JSON.stringify({ permissions: { defaultMode: "bypassPermissions" } })); + for (const defaultMode of ["bypassPermissions", "acceptEdits", "auto"]) { + await writeFile(join(project, ".claude/settings.json"), JSON.stringify({ permissions: { defaultMode } })); + assert.equal(inspect(project).permissions?.defaultMode, undefined, "real project escalation stays filtered"); + } + await writeFile(join(home, ".claude/settings.local.json"), JSON.stringify({ permissions: { defaultMode: "plan" } })); + assert.equal(inspect(home).permissions.defaultMode, "plan", "local restrictions still win"); +}); diff --git a/patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch b/patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch index 6774dc6553c..7a16bb7c0f2 100644 --- a/patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch +++ b/patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch @@ -64,3 +64,19 @@ index 0214a509eb5e7b777a69e4e34637941c275d27b1..403be02df66e25eee29e7d5bef146270 // Forward MCP elicitation requests onto ACP elicitation. Only attached // when the client advertised support, so non-supporting clients keep the // SDK's default (auto-decline) behavior. (AskUserQuestion is handled in +diff --git a/dist/settings.js b/dist/settings.js +--- a/dist/settings.js ++++ b/dist/settings.js +@@ -88,6 +88,12 @@ + async loadAllSettings() { + try { + const resolved = await resolveSettings({ cwd: this.cwd }); ++ const userPath = resolved.sources.find((entry) => entry.source === "user")?.path; ++ for (const entry of resolved.sources) { ++ if (userPath && entry.source === "project" && entry.path === userPath) { ++ entry.source = "user"; ++ } ++ } + this.effective = filterEscalatingDefaultMode(resolved); + } + catch (error) { From 4336b9ee0210d51b5f9a19eb195e995a1bd2a50e Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 26 Oct 2026 22:00:01 -0500 Subject: [PATCH 038/133] fix(db): reserve transactions before checking pipeline capacity (#17831) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Interactive requests share a PostgreSQL connection pool with background work. > - Prepared queries can queue behind a slow query on a busy connection. > - Disabling pipelining lets queued work use the next free connection. > - The driver skips its transaction reservation callback when the pipeline limit is zero. > - This pull request runs that callback before the capacity check. > - Operators can test a pool without pipelining while transactions retain their connection ownership. ## Linked Issues or Issue Description **What happened?** With `PGMAX_PIPELINE=0`, postgres.js 3.4.9 skips `onexecute` in its short-circuit capacity check. A transaction then lacks its reserved connection. Real PostgreSQL reproduces `UNSAFE_TRANSACTION` through the app client and an undefined connection error through the raw driver. This prevents safe use of the existing setting to reduce query queue delays. **Expected behavior** A transaction must reserve its connection regardless of pipeline capacity. Commit, rollback, savepoints, and disconnect recovery must preserve their existing contracts. **Steps to reproduce** 1. Use the pre-change driver patch from `e61ee4b005f8e13f64af15f69cb62305dd869f82`. 2. Start disposable PostgreSQL and run the `no-pipeline` scenario in `packages/db/src/__fixtures__/postgres-connection-recovery.mjs` with `TEST_DATABASE_URL`, `DRIVER_FORMAT=esm`, and `RECOVERY_CASE=no-pipeline`. 3. Observe the transaction failure. Apply this patch and repeat with both `esm` and `cjs`. Refs #13643, which introduced the existing connection recovery patch. Searched related open and closed driver PRs. This change extends that patch and preserves its disconnect safeguards. ## What Changed - Move the transaction reservation callback before the pipeline capacity check in both driver distributions. - Add real PostgreSQL regressions for commits, concurrent transaction reads, isolation, rollback, savepoints, reserved connections, and progress on a freed pool connection. - Document the existing `PGMAX_PIPELINE=0` setting and its driver prerequisite. Keep the default pipeline limit unchanged. ## Verification - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - Real PostgreSQL 18.6 in an owned, disposable Docker container: the old patch fails the new transaction scenario in both ESM and CommonJS. The corrected patch passes all 18 scenarios, including all 12 existing disconnect recovery cases. The container was removed afterward. - Local service benchmark: actual `createDb` with prepared statements, two warmed connections, two controlled advisory lock waits, and the complete three-query `getDefaultCompanyGoal` path. Two warmups and five alternating measured samples per setting. Pipeline 100: median 413.44 ms, range 413.33–416.10 ms. Pipeline 0 with the patch: median 12.22 ms, range 11.86–12.57 ms. Complete SQL, bindings, and results match. Five of five reads finish while the unrelated slow query remains blocked. These are synthetic local queue measurements, not hosted latency claims. - App client compatibility on real PostgreSQL: commit, concurrent transaction reads, rollback, nested savepoint rollback, dedicated connections, and liveness connections passed. - `pnpm test:run`: attempted, then stopped after repeated local embedded PostgreSQL startup failures caused by the host semaphore limit. This is an incomplete local full-suite run, not a pass. The focused driver checks above used disposable Docker PostgreSQL instead. The initial sandbox attempt also stopped before test execution because tsx could not bind its IPC socket. Final CI results will be recorded here. - Current head `72039f71f63530300bf241475d42e857297dfb90`: Greptile and Paperclip review are both 5/5. All three earlier findings are resolved. [CI run 40431567516](https://github.com/paperclipai/paperclip/actions/runs/40431567516) ran the complete validation matrix. The database lane passed all 265 tests, including all 18 real PostgreSQL driver regressions. One cleanup timer case timed out in server shard 6/12; its isolated real-PostgreSQL reproduction passed once and failed once, while five runs with the pre-change driver passed. That evidence is retained and does not establish the timer failure's cause. A single CI retry of the failed shard passed. The final aggregate `ci / verify` gate is waiting for a runner. Its first queued request did not start for more than 14 minutes. A guarded queue reset confirmed that it was the only unfinished job and that every test job had passed; attempt 3 restarts only that gate. No test result was replaced or bypassed. The required gate must pass before merge. ## Risks - This changes a shared driver callback order. Both module formats and existing disconnect recovery are covered by real database tests. - This PR does not change pool size, enable the setting by default, retry SQL, or change schemas. Disabling pipelining may reduce batch throughput. Operators must compare the workload before retaining that setting. - The local queue benchmark proves the mechanism. Hosted request improvement still needs acceptance after the patched image is deployed. ## Model Used OpenAI GPT-6 in Codex, with code execution and browser developer tools. The exact serving model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DATABASE.md | 10 +++ .../postgres-connection-recovery.mjs | 76 ++++++++++++++++++- .../src/postgres-connection-recovery.test.ts | 16 ++++ patches/postgres@3.4.9.patch | 24 +++++- 4 files changed, 121 insertions(+), 5 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 2032268da2d..572b3564bf9 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -580,6 +580,16 @@ PostgreSQL backends and checks rejection, pool recovery, and transaction isolati Remove the patch when an upstream release passes these tests. Installs of the unmodified `postgres` package outside this workspace do not include the patch. +The patch also preserves transaction connection reservation when pipelining is +disabled with `PGMAX_PIPELINE=0`. Without this fix, the driver's pipeline capacity +check skips the transaction reservation callback and `BEGIN` fails with +`UNSAFE_TRANSACTION`. Disabling pipelining makes a queued query wait for the next +available pool connection instead of sending it behind unrelated work on a busy +connection. This can help interactive requests that share a prepared-query pool +with slow background work. It does not make the SQL itself faster. The default +pipeline limit remains unchanged. Test transaction behavior before using this +setting with a driver build that does not include the patch. + Trusted-header actor synchronization retries transient connection failures, including `CONNECT_TIMEOUT`, at most twice. This retry applies only to the idempotent actor synchronization operations, not arbitrary transactions. A diff --git a/packages/db/src/__fixtures__/postgres-connection-recovery.mjs b/packages/db/src/__fixtures__/postgres-connection-recovery.mjs index 64ee9db814d..15ac09f8805 100644 --- a/packages/db/src/__fixtures__/postgres-connection-recovery.mjs +++ b/packages/db/src/__fixtures__/postgres-connection-recovery.mjs @@ -8,8 +8,9 @@ const postgres = process.env.DRIVER_FORMAT === "cjs" const mode = process.env.RECOVERY_CASE; const closed = Promise.withResolvers(); const sql = postgres(process.env.TEST_DATABASE_URL, { - max: 1, - max_pipeline: 1, + max: mode === "no-pipeline" || mode === "no-pipeline-fairness" ? 2 : 1, + max_pipeline: mode?.startsWith("no-pipeline") ? 0 : 1, + connection: { application_name: `driver-recovery-${mode}` }, backoff: 0, onnotice() {}, onclose: () => closed.resolve(), @@ -28,7 +29,76 @@ try { await control.unsafe("drop table if exists connection_recovery_probe"); await control.unsafe("create table connection_recovery_probe (id integer)"); - if (mode === "transaction" || mode === "transaction-closed" || mode === "savepoint") { + if (mode === "no-pipeline-fairness") { + const query = "select pg_advisory_xact_lock($1::bigint)"; + const lookup = "select $1::int as value"; + const one = await sql.reserve(); + const two = await sql.reserve(); + for (const connection of [one, two]) { + await connection.unsafe(query, [900001]); + await connection.unsafe(lookup, [42]); + } + one.release(); + two.release(); + let first; + let second; + let fast; + try { + await control.unsafe("select pg_advisory_lock(900001), pg_advisory_lock(900002)"); + const waitForBlocked = async (count) => { + const deadline = performance.now() + 5000; + let observed = 0; + while (performance.now() < deadline) { + const [row] = await control.unsafe("select count(*)::int as count from pg_stat_activity where application_name=$1 and wait_event='advisory'", [`driver-recovery-${mode}`]); + observed = row.count; + if (row.count === count) return; + await setImmediate(); + } + throw new Error(`Expected ${count} advisory lock waits; observed ${observed}`); + }; + let firstFinished = false; + first = sql.unsafe(query, [900001]).then(() => { firstFinished = true; }); + await waitForBlocked(1); + second = sql.unsafe(query, [900002]).then(() => {}); + await waitForBlocked(2); + fast = sql.unsafe(lookup, [42]).then((rows) => rows); + await control.unsafe("select pg_advisory_unlock(900002)"); + assert.equal((await fast)[0].value, 42); + assert.equal(firstFinished, false, "lookup must progress after one pool connection is released while the other stays blocked"); + } finally { + await control.unsafe("select pg_advisory_unlock_all()"); + await Promise.allSettled([first, second, fast].filter(Boolean)); + } + } else if (mode === "no-pipeline" || mode === "no-pipeline-single") { + await sql.begin(async (tx) => { + await tx.unsafe("insert into connection_recovery_probe values (1)"); + const results = await Promise.all([ + tx.unsafe("select id from connection_recovery_probe"), + tx.unsafe("select count(*)::int as count from connection_recovery_probe"), + ]); + assert.deepEqual(results[0].map((row) => row.id), [1]); + assert.equal(results[1][0].count, 1); + assert.equal((await control.unsafe("select count(*)::int as count from connection_recovery_probe"))[0].count, 0); + }); + await assert.rejects(sql.begin(async (tx) => { + await tx.unsafe("insert into connection_recovery_probe values (2)"); + throw new Error("controlled transaction rollback"); + }), /controlled transaction rollback/); + await sql.begin(async (tx) => { + await tx.unsafe("insert into connection_recovery_probe values (3)"); + await assert.rejects(tx.savepoint(async (nested) => { + await nested.unsafe("insert into connection_recovery_probe values (4)"); + throw new Error("controlled savepoint rollback"); + }), /controlled savepoint rollback/); + assert.deepEqual((await tx.unsafe("select id from connection_recovery_probe order by id")).map((row) => row.id), [1, 3]); + }); + const reserved = await sql.reserve(); + try { + assert.deepEqual((await reserved.unsafe("select id from connection_recovery_probe order by id")).map((row) => row.id), [1, 3]); + } finally { + reserved.release(); + } + } else if (mode === "transaction" || mode === "transaction-closed" || mode === "savepoint") { const started = Promise.withResolvers(); const resume = Promise.withResolvers(); const callbackDone = Promise.withResolvers(); diff --git a/packages/db/src/postgres-connection-recovery.test.ts b/packages/db/src/postgres-connection-recovery.test.ts index ddb6bb325b8..4918fe3fc41 100644 --- a/packages/db/src/postgres-connection-recovery.test.ts +++ b/packages/db/src/postgres-connection-recovery.test.ts @@ -21,6 +21,22 @@ describe.skipIf(!support.supported)("postgres connection recovery", () => { afterAll(async () => { await database?.cleanup(); }); for (const format of ["esm", "cjs"]) { + for (const scenario of ["no-pipeline-single", "no-pipeline", "no-pipeline-fairness"]) { + it(`${format}: ${scenario} preserves connection ownership and progress`, async () => { + const { stdout, stderr } = await run(process.execPath, [fixture], { + env: { + ...process.env, + TEST_DATABASE_URL: database.connectionString, + DRIVER_FORMAT: format, + RECOVERY_CASE: scenario, + }, + timeout: 15_000, + }); + expect(stdout.trim()).toBe("recovered"); + expect(stderr).toBe(""); + }, 20_000); + } + for (const scenario of ["transaction", "transaction-closed", "savepoint", "reserve", "transaction-queue", "reserve-queue"]) { it(`${format}: rejects disconnected ${scenario} work and recovers the pool`, async () => { // A driver timer used to crash the whole process. A child also detects diff --git a/patches/postgres@3.4.9.patch b/patches/postgres@3.4.9.patch index ab27a64ba33..c45fb197a7c 100644 --- a/patches/postgres@3.4.9.patch +++ b/patches/postgres@3.4.9.patch @@ -1,7 +1,17 @@ diff --git a/cjs/src/connection.js b/cjs/src/connection.js -index 07f6716702ac888215c86ad6e0071aaf55ae519c..dc24963ce2dd0a219f366f80ba9847548ba23a52 100644 +index 07f6716702ac888215c86ad6e0071aaf55ae519c..85700681ccc7bf98f1fe987978db47e0097227f3 100644 --- a/cjs/src/connection.js +++ b/cjs/src/connection.js +@@ -173,8 +173,8 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose + return write(toBuffer(q)) + && !q.describeFirst + && !q.cursorFn +- && sent.length < max_pipeline + && (!q.options.onexecute || q.options.onexecute(connection)) ++ && sent.length < max_pipeline + } catch (error) { + sent.length === 0 && write(Sync) + errored(error) @@ -438,6 +438,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose remaining = 0 incomings = null @@ -84,9 +94,19 @@ index f09c61c7e998f411560b95c9a8e1ed6a97972c11..04dded5c7e2cdc34ac01b29a87cc9324 ? queries.push(q) : c.execute(q) || move(c, full) diff --git a/src/connection.js b/src/connection.js -index 1b1cccde43b4570d5d071d6ffaab2669b3c2065a..c95c326a2747c680c2c22b04b1c5f2e27a4e8c90 100644 +index 1b1cccde43b4570d5d071d6ffaab2669b3c2065a..81541f9cea75925256d7b02b2079c60802b2f92f 100644 --- a/src/connection.js +++ b/src/connection.js +@@ -173,8 +173,8 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose + return write(toBuffer(q)) + && !q.describeFirst + && !q.cursorFn +- && sent.length < max_pipeline + && (!q.options.onexecute || q.options.onexecute(connection)) ++ && sent.length < max_pipeline + } catch (error) { + sent.length === 0 && write(Sync) + errored(error) @@ -438,6 +438,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose remaining = 0 incomings = null From 832c44f26a1fba8f987a1128314e90cb7f3698f6 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Mon, 26 Oct 2026 20:19:16 -0700 Subject: [PATCH 039/133] fix: resume user follow-ups that reassign a task (#17821) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The execution control plane saves user messages while an earlier run needs recovery. > - A user comment can also assign a task back to its agent. > - That update creates an assignment wake, but continuation accepted only comment wake reasons. > - The saved question could stay blocked after workspace recovery and confirmed process cleanup. > - This pull request admits assignment wakes that carry a valid, undelivered user comment through the existing continuation checks. > - The same rule applies to new messages and saved requests already waiting for recovery. ## Linked Issues or Issue Description **What happened?** A human follow-up that also assigns a task back to its agent produces `issue_assigned`. After a native run stops during handoff, the continuation policy rejects that wake reason. The question remains saved behind the execution recovery hold even after the workspace is recovered and the old sandbox is destroyed. **Expected behavior** Deliver the saved human question once after the existing stop, ownership, authority, and execution gates pass. Assignment alone must not grant permission to continue. **Steps to reproduce** 1. Finish task work by assigning it to a human for review. Let the native run stop with `issue_reassigned` and retain its recovery hold. 2. Recover the workspace and confirm destruction of the old sandbox. 3. Post a new human question while assigning the task back to the agent. 4. Observe that the assignment wake stays deferred instead of delivering the question. Refs: #17288, #14156. These related changes cover legacy reassignment and assignments without comments. This change covers a human comment carried by an assignment wake behind a native or legacy continuation hold. ## What Changed - Share the comment-bearing wake classification between continuation admission and saved-request recovery. - Accept `issue_assigned` only through the existing persisted user-comment validation and execution gates. - Add database regressions for new and saved assignment wakes, concurrent recovery, single delivery, and unchanged prior run results. - Check that bare assignments, invalid authors and comments, changed ownership, active execution, and pending approvals or questions retain their holds. Cover native and legacy cleanup paths. - Update execution semantics and both recovery maps. ## Verification - Red: both new and saved assignment-with-comment cases stayed deferred without a successor on the original code. - Green: the final continuation suite passes all 284 tests. The four companion recovery, retry, and queue suites pass another 208 tests. - `pnpm -r typecheck` and `pnpm build` pass. - `git diff --check`, `node scripts/check-maps.mjs`, and `node scripts/check-module-boundaries.mjs` pass. - Apex review is 5/5 on `56c9c2d90dcda55b3f760946c9aba6aee448c361`, with no unresolved review threads. The only earlier finding concerned the contribution rule against new source comments; those comments were removed. - Full `pnpm test:run` is still running locally. CI is waiting for the `paperclip-public-pr-x64` runner fleet; the superseded workflow was cancelled so the current head could proceed. No passing CI claim is made yet. - Live recovery: corrected one existing saved wake to the supported comment reason. The normal server admitted one successor with the exact saved question and prior-run context, then started real tool calls. This verifies admission on the existing deployment; the new source change is not deployed yet. The resumed turn later failed with a separate provider terminal event whose error was null, followed by a workspace restore failure. The task is blocked again on that failure. Its useful answer has not been delivered, so end-to-end completion remains unverified. ## Risks - Assignment wakes now reach continuation validation when they carry a comment. The persisted comment must still belong to the task and requesting human, follow the stopped run, and remain undelivered. - The existing stop proof, cleanup, current assignment, approval, pause, budget, and competing-execution checks remain effective. Regression cases cover both native and legacy paths. - No schema migration or data rewrite is required for saved requests. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source inspection, tool use, and code execution. The runtime does not expose the exact model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 7 ++ maps/agent-access-map/recovery.md | 5 ++ maps/feature-map/recovery.md | 3 + .../explicit-native-continuation.test.ts | 76 ++++++++++++++++++- .../services/explicit-native-continuation.ts | 6 +- server/src/services/heartbeat/run-control.ts | 3 +- 6 files changed, 95 insertions(+), 5 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 6b1224ef3e2..1228bb99dea 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1480,6 +1480,13 @@ delta when attaching a new run. Other stale-event and authority checks remain. ### Explicit user continuation after execution failure +When a user's comment also assigns the task back to its agent, the assignment +wake retains that comment's continuation intent. Admission validates the saved +comment, its author, task, timing and delivery receipt through the same checks as +an ordinary comment wake. A bare assignment, agent/system input or mention wake +does not authorize continuation. Already deferred assignment-with-comment +receipts use this same path after recovery; they do not require another comment. + An execution recovery hold blocks automatic replay. A new authenticated user comment or exact failed-run Retry can authorize a fresh native or legacy conversation turn after the predecessor's diff --git a/maps/agent-access-map/recovery.md b/maps/agent-access-map/recovery.md index 7c450bd03a2..c9df2c9b13e 100644 --- a/maps/agent-access-map/recovery.md +++ b/maps/agent-access-map/recovery.md @@ -465,6 +465,11 @@ Agent access: `partial` — eligible admitted agent/native bound wake receives s Policy: Exact prior run/lease/workspace stop proof, no competing execution, current owner/responsible user, normal subtree/project pause/budget/review gates. Durable ordered message/wake receipts avoid duplicates; never borrow authority from stale callback/other routine. Unused hold-denied admission needs no-execution/dispatch evidence before automatic reconsideration. +A human-authored comment attached to an assignment wake can authorize the same +fresh continuation as an ordinary comment. The server validates the persisted +comment and its task/author/timing; the assignment label alone grants no +authority. Agents and system wakes cannot inherit this human-input path. + Evidence: `source-only` — [legacy gates](../../server/src/services/recovery/legacy-continuation.ts), [native ownership](../../server/src/services/native-runtime/native-restart-recovery.ts), [queued Stop](../../server/src/modules/wake-queue/adapters/postgres.ts); reviewed at `3bbc2104b65caf9f7763938a01260a5673e12f7a`. No live provider proof. Verification: `not-run` — After confirmed cleanup one useful continuation consumes saved order; repeat sweep/reload no duplicate. Pause/budget/approval/unknown external outcome and reassignment prevent stale admission. Queued Stop leaves another routine cycle intact. diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index da48541232a..03e16003337 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -41,6 +41,9 @@ Agent access: [companion](../agent-access-map/recovery.md). an explicit decision, not automatic replay. - `user-continuation`: `recovery/user-continuation` — saved user messages after a stop are reconsidered once cleanup proves the old execution is stopped; duplicate wakes are avoided. + A user comment that also reassigns the task retains this intent, including + existing deferred assignment receipts. The comment's identity and author, + current assignment, stop proof and normal execution gates remain required. - `reconnect`: `recovery/reconnect` — browser stream recovery refreshes persisted state without claiming the agent itself restarted or that a failed run succeeded. diff --git a/server/src/services/explicit-native-continuation.test.ts b/server/src/services/explicit-native-continuation.test.ts index 06aff68b9b9..361af71a97c 100644 --- a/server/src/services/explicit-native-continuation.test.ts +++ b/server/src/services/explicit-native-continuation.test.ts @@ -57,6 +57,67 @@ const support = await getEmbeddedPostgresTestSupport(); actorType: "user", actorId: "board", reason: "issue_commented" }; } type Fixture = Awaited>; + it.each(["new", "saved"])("delivers a reassignment's saved user question once after the native run stopped (%s)", async origin => { + const f = await seed(); + await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, f.issueId)); + const [source] = await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "issue_reassigned", + processPid: null, processGroupId: null, + }).where(eq(heartbeatRuns.id, f.sourceRunId)).returning(); + await db.update(issueRecoveryActions).set({ cause: "native_continuation_requires_reconciliation" }) + .where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + const lease = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: source.id, + provider: "daytona", providerLeaseId: "reviewed-task-sandbox" }; + await db.insert(environmentLeases).values({ ...lease, status: "expired", leasePolicy: "ephemeral", + releasedAt: new Date(), cleanupStatus: "success", metadata: { + remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }), + }, + }); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); + const heartbeat = heartbeatService(db); + if (origin === "new") { + await heartbeat.wakeup(f.agentId, { source: "assignment", triggerDetail: "system", reason: "issue_assigned", + requestedByActorType: "user", requestedByActorId: f.actorId, + payload: { issueId: f.issueId, commentId: f.commentId, mutation: "update" }, + contextSnapshot: { source: "issue.update", issueId: f.issueId, wakeCommentId: f.commentId }, + }); + } else { + await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, + source: "assignment", triggerDetail: "system", reason: "issue_assigned", status: "deferred_issue_execution", + requestedByActorType: "user", requestedByActorId: f.actorId, payload: { + issueId: f.issueId, commentId: f.commentId, executionWait: { reason: "execution_recovery" }, + _paperclipWakeContext: { source: "issue.update", issueId: f.issueId, wakeReason: "issue_assigned", + wakeCommentId: f.commentId, wakeCommentIds: [f.commentId] }, + }, + }); + } + const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId)); + if (receipt.status === "deferred_issue_execution") { + await db.update(agentWakeupRequests).set({ updatedAt: new Date(0) }).where(eq(agentWakeupRequests.id, receipt.id)); + await Promise.all([heartbeat.resumeExecutionWaitComments(), heartbeat.resumeExecutionWaitComments()]); + } + const [delivered] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.id)); + expect(delivered).toMatchObject({ status: origin === "new" ? "queued" : "coalesced", runId: expect.any(String) }); + const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), + eq(heartbeatRuns.status, "queued"))); + expect(successors).toHaveLength(1); + expect(delivered.runId).toBe(successors[0].id); + expect(successors[0]).toMatchObject({ agentId: f.agentId, contextSnapshot: { + forceFreshSession: true, previousRunId: source.id, wakeCommentIds: [f.commentId], + } }); + await heartbeat.resumeRemoteStopComments(source); + expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), + eq(heartbeatRuns.status, "queued")))).toHaveLength(1); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull(); + const [original] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, source.id)); + expect(original).toMatchObject({ status: "cancelled", errorCode: "issue_reassigned" }); + }); + it("keeps a bare human assignment behind the execution hold", async () => { + const f = await seed(); + expect(await admitExplicitNativeContinuation({ ...f, db, reason: "issue_assigned", commentId: null })).toBeNull(); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull(); + expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), + eq(heartbeatRuns.status, "queued")))).toHaveLength(0); + }); it("automatically admits a saved historical local message once after isolated fallback becomes eligible", async () => { const f = await seed(); const processStartedAt = new Date(); @@ -1451,7 +1512,9 @@ const support = await getEmbeddedPostgresTestSupport(); { runtime: "native", retry: false }, { runtime: "native", retry: true }, { runtime: "legacy", retry: false }, { runtime: "legacy", retry: true }, { runtime: "legacy_startup", retry: false }, { runtime: "legacy_startup", retry: true }, - ])("resumes a user message after confirmed cleanup: %j", async ({ runtime, retry }) => { + { runtime: "native", retry: false, reason: "issue_assigned" }, + { runtime: "legacy", retry: false, reason: "issue_assigned" }, + ])("resumes a user message after confirmed cleanup: %j", async ({ runtime, retry, reason = "issue_commented" }) => { const f = await seed(); if (runtime.startsWith("legacy")) { await db.update(agents).set({ adapterType: "claude_local" }).where(eq(agents.id, f.agentId)); @@ -1474,7 +1537,7 @@ const support = await getEmbeddedPostgresTestSupport(); const heartbeat = heartbeatService(db, retry ? { environmentRuntime: { retryPendingSandboxTeardown: async () => ({ providerLeaseId: identity.providerLeaseId, state: "destroyed" }), } as unknown as HeartbeatEnvironmentRuntime } : {}); - await heartbeat.wakeup(f.agentId, { source: "automation", triggerDetail: "system", reason: "issue_commented", + await heartbeat.wakeup(f.agentId, { source: "automation", triggerDetail: "system", reason, requestedByActorType: "user", requestedByActorId: "board", payload: { issueId: f.issueId, commentId: f.commentId }, contextSnapshot: { issueId: f.issueId, wakeCommentId: f.commentId } }); const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)); @@ -1897,8 +1960,11 @@ const support = await getEmbeddedPostgresTestSupport(); expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull(); } else expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull(); }); - it.each(["live_process", "missing_process", "lease", "coordinator", "successor", "agent_message", "old_comment", "wrong_author", "run_authored", "reassigned", "automatic", "approval", "question", "malformed_comment", "legacy_owner"])("keeps the hold for %s", async kind => { + it.each(["issue_commented", "issue_assigned"].flatMap(reason => + ["live_process", "missing_process", "lease", "coordinator", "successor", "agent_message", "system_message", "old_comment", "wrong_author", "run_authored", "reassigned", "automatic", "mention", "approval", "question", "malformed_comment", "missing_comment", "foreign_comment", "legacy_owner"].map(kind => ({ reason, kind })), + ))("keeps the hold for $kind ($reason)", async ({ reason, kind }) => { const f = await seed(); + f.reason = reason; if (kind === "legacy_owner") await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "failed", runtimeMode: "legacy", processPid: process.pid, contextSnapshot: { issueId: f.issueId }, resultJson: { conversationContinuation: "continue_conversation_v1" } }); @@ -1920,8 +1986,12 @@ const support = await getEmbeddedPostgresTestSupport(); await db.insert(issueApprovals).values({ companyId: f.companyId, issueId: f.issueId, approvalId }); } if (kind === "malformed_comment") f.commentId = "not-a-uuid"; + if (kind === "missing_comment") f.commentId = randomUUID(); + if (kind === "foreign_comment") f.commentId = (await seed()).commentId; if (kind === "agent_message") f.actorType = "agent"; + if (kind === "system_message") f.actorType = "system"; if (kind === "automatic") f.reason = "issue_continuation_needed"; + if (kind === "mention") f.reason = "issue_comment_mentioned"; if (kind === "wrong_author") f.actorId = "someone-else"; if (kind === "run_authored") await db.update(issueComments).set({ createdByRunId: f.sourceRunId }).where(eq(issueComments.id, f.commentId)); if (kind === "old_comment") await db.update(issueComments).set({ createdAt: new Date("2026-09-11T09:00:00Z") }).where(eq(issueComments.id, f.commentId)); diff --git a/server/src/services/explicit-native-continuation.ts b/server/src/services/explicit-native-continuation.ts index 109efade684..0c1ee1ebe21 100644 --- a/server/src/services/explicit-native-continuation.ts +++ b/server/src/services/explicit-native-continuation.ts @@ -31,6 +31,10 @@ import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js type Run = typeof heartbeatRuns.$inferSelect; const terminal = ["failed", "interrupted", "timed_out", "cancelled"]; +export function isUserCommentContinuationReason(reason: string | null | undefined): boolean { + return reason === "issue_commented" || reason === "issue_reopened_via_comment" || reason === "issue_assigned"; +} + function processStopped(pid: number): boolean { try { process.kill(pid, 0); return false; } catch (error) { return (error as NodeJS.ErrnoException).code === "ESRCH"; } @@ -96,7 +100,7 @@ export async function admitExplicitNativeContinuation(input: { const retry = input.reason === "retry_failed_run" && z.string().guid().safeParse(input.failedRunId).success; if (!retry && !input.taskRetryRequestId && !input.queuedCommentInterruptId && (!commentId || !z.string().guid().safeParse(commentId).success || - !["issue_commented", "issue_reopened_via_comment"].includes(input.reason ?? ""))) return null; + !isUserCommentContinuationReason(input.reason))) return null; const [task] = await db.select().from(issues).where(and( eq(issues.companyId, companyId), eq(issues.id, issueId), )); diff --git a/server/src/services/heartbeat/run-control.ts b/server/src/services/heartbeat/run-control.ts index 7387da1f74d..8c87325ee4b 100644 --- a/server/src/services/heartbeat/run-control.ts +++ b/server/src/services/heartbeat/run-control.ts @@ -40,6 +40,7 @@ import { } from "../remote-execution-termination.js"; import { admitExplicitNativeContinuation, + isUserCommentContinuationReason, undeliveredLegacyUserCommentIds, } from "../explicit-native-continuation.js"; import { isCancelledNativeStartup } from "../cancelled-native-startup.js"; @@ -575,7 +576,7 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont } if (legacyContinuation || stoppedNativeContinuation) { if (!commentId || !run.finishedAt || !requestedByActorId || - !["issue_commented", "issue_reopened_via_comment"].includes(reason ?? "")) continue; + !isUserCommentContinuationReason(reason)) continue; const [comment] = await db.select().from(issueComments).where(and( eq(issueComments.companyId, run.companyId), eq(issueComments.issueId, issueId), sql`${issueComments.id}::text = ${commentId}`, eq(issueComments.authorType, "user"), From f0877a9c7c555d8597bfd5fdf91c54f480c20375 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 26 Oct 2026 22:21:58 -0500 Subject: [PATCH 040/133] fix: recover native work and requalify saved computers (#17724) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit https://entire.com/gh/paperclipai/-/trails/32/changes/paperclip/32 ## Thinking Path > - Paperclip manages AI agents and preserves their work after an interruption. > - Native Stop must confirm termination and save the workspace. > - Task and run views must show the current recovery receipt. > - A new external reply can arrive while that cleanup is pending. > - The reply must return to its original chat owner after cleanup. > - A successful fresh retry also becomes the task’s bound workspace for its next ordinary turn. > - Managed computer updates can invalidate a saved snapshot qualification. > - This pull request fixes those instructions, acknowledgements, handoffs and the existing native-agent Test path. > - The operator can continue with new input while the stopped turn remains preserved. ## Linked Issues or Issue Description **What happened?** Stopped native runs could keep the initial workspace-repair instruction after files were recovered. Native Stop could also leave the shared cancellation state at requested after remote termination was confirmed. A new GitHub reply saved during cleanup could then remain queued because cleanup did not ask its original inbound owner to reconsider the receipt. After a managed snapshot update, an existing native agent could also need qualification while the generic environment Test required instance-admin access. **Expected behavior** Show the current saved-work state. Acknowledge Stop only after exact termination proof. Let an authorized operator use the existing saved-agent Run test to qualify its current computer. After cleanup, recheck the saved reply's current authority and admit it once as new input. Preserve the stopped run and uncertain external effects. **Steps to reproduce** 1. Stop an active native turn before it returns a semantic result. 2. Send a new authorized external reply while cleanup is pending. 3. Let workspace recovery and provider cleanup finish. Inspect the run instruction, Stop acknowledgement and queued reply. Related work: #17594 preserves native processes through controller outages. #17401 addresses a separate bootstrap failure. This pull request does not replace those fixes. ## What Changed - Use the exact run's workspace recovery receipt for task and run guidance, including the Activity ledger. - Retire an obsolete exact-session notice through the existing bounded reconciler. Preserve the separate no-replay hold. - Write the shared native Stop acknowledgement only after the audited Stop intent and every exact remote lease receipt match. - Delegate saved external input to the live inbound owner. Recheck its source, actor, endpoint, principal and conversation generation. Promote the same receipt in the transaction that creates its new run. - Use that owner from the cleanup callback and existing stranded-queue catch-up after a lost callback or restart. A denied source cannot abort other items. - Reuse the current private task directory after a successful fresh retry. Re-read the company/task/workspace binding and validate the canonical stable or attempt namespace without recreating missing files; keep a later fresh retry separate. - Extend the existing agent-scoped qualification owner to native agents. The clean saved-agent Test qualifies its exact computer before runtime testing. Reuse a current proof; require a real probe and verified cleanup for a stale proof. Preserve draft tests and metadata-only external/pool previews. - Update execution semantics, capability maps and the recovery verification recipe. Add a production-page Storybook interaction for the normal native Test. ## Verification - **Red:** persisted recovery states returned stale instructions; an exact terminated native lease left shared Stop requested; the real GitHub subscribed-message fixture left zero successors after cleanup. The controlled baseline restored the previous production source, reproduced the failure, then restored the fixed bytes with matching SHA-256 hashes. - **Green:** 382 durable-chat, native-continuation, Stop and recovery tests pass. The real GitHub/PostgreSQL handoff matrix passes 12/12. It covers callback, application restart, callback/sweep race, valid input behind a deleted source, and denial for unknown stop, wrong allocation, paused endpoint, revoked principal, deleted or changed source, changed conversation generation and foreign scope. Repeated callbacks preserve one receipt, its actor, one successor and the unchanged stopped predecessor. - **Workspace continuation red/green:** the real PostgreSQL preparation flow created a fresh attempt and saved a marker, then rejected the next ordinary turn with the same binding mismatch observed in deployed use. The repaired preparation and filesystem suites pass 86/86: stable and attempt-root reuse, saved-marker preservation, a later clean sibling, changed current bindings, foreign company/task, closed workspace, traversal, missing path and symlink refusal. No task row reset or old-turn replay is used. - **Qualification red/green:** a native saved-agent qualification returned 422, and its normal Test skipped qualification. The repaired path passes 224 qualification/immutable-snapshot/environment tests and 176 UI/upgrade tests. It covers revoked authority, foreign or changed computer, current-proof reuse, failed/missing proof, draft settings, settings changed during the probe, and external/pool previews. On exact current source, the real local Harness / Runtime UI also qualified a newly saved disposable Computer with missing proof through actual Daytona, then completed the actual OpenCode/OpenRouter hello probe using `openrouter/deepseek/deepseek-v4.1-flash-1017`. Supported availability changed from Legacy-only to Paperclip Runner after proof persistence. Qualification and runtime leases both completed normal release/cleanup. The five prior task run IDs were unchanged; no task wake was created. This is macOS source-server/UI with real Linux Daytona and provider services, not a hosted UI claim. Test probes are archived by the existing inspection contract; release does not claim physical deletion. - Full workspace `pnpm -r typecheck`, `pnpm build`, token, map and diff checks pass at `f6450f4739fc8141815b8b7c35fde22a655c5f90`. The broad local `pnpm test:run` passed 24,899 tests and reported one existing nested-clone test timeout at 15 seconds. The complete failing file then passed 21/21 with the normal timeout; the exact case also passed on the pinned earlier baseline. The broad runner stopped before its later groups. No timeout or production workaround was added. Prior head `f6450f4` has 53 successful checks, two conditional Storybook skips and Greptile 5/5 with no inline threads. New workspace-reuse head `fcb13399ec70f92f71bd8250081251dca3f48962` passes workspace typecheck, full build, 86 scoped tests, maps and diff checks; Current-head CI has 53 successful checks and two conditional Storybook skips; Greptile is 5/5 with no inline findings. Deployed ordinary-turn reuse acceptance passed as described below. - **Hosted Stop proof:** the real GitHub issue-mention path used native OpenCode, `openrouter/deepseek/deepseek-v4.1-flash-1017`, Daytona and an exact low-trust task boundary. Normal UI Stop interrupted an active Bash call. Native and shared acknowledgements matched the exact lease. Automatic recovery saved exactly `STARTED\n` (8 bytes), with no finish marker. Reload showed Stopped and saved work recovered. Only the original cancelled run remained. Normal five-minute cleanup completed its exact provider-destruction transition. - **Limits:** the provider buffered command output, so the marker was read after recovery. The approved local provider credential could not inspect the hosted sandbox; its 404 is not deletion proof. Cleanup success comes from the exact lease's platform destruction transition. An earlier hosted stale instruction disappeared after rollout with unchanged raw run/action data; that historical cause remains unexplained. Original incident holds and publication receipts were untouched. - **Hosted handoff proof:** after QA deployed the combined candidate containing `46b2e99`, startup admitted the previously saved GitHub author reply without another comment, Retry or manual action. The new native run used the exact original wake receipt and saved comment. It completed once, committed an external-response wait, preserved the stopped predecessor, emptied the queue and left the task In review. The answer appeared in the original GitHub conversation. Workspace finalization and the exact lease's normal destruction transition succeeded. - **Hosted workspace proof:** on the combined candidate containing `fcb13399`, normal Retry preserved the original authenticated GitHub request and completed a fresh native attempt. One later ordinary same-head author reply then completed in the exact same workspace ID and attempt directory, with no Retry/fresh directive. Its input tree digest matched the Retry's finalized digest; final copyback preserved that digest. The persisted assessment row and all five prior run records were unchanged, exactly one new reply run completed, and the task returned to In review with an empty queue and no live run. Both exact leases completed normal verified destruction. This used real native OpenCode, the exact DeepSeek/OpenRouter model and Daytona, without workspace-row repair or a new remote marker. - **Publication limit:** the workspace audit compares the persisted assessment and run history. The ordinary reply exposed a separate destination-summary fallback that changed the visible GitHub summary despite the preserved assessment row. The publication track is repairing that behavior; this proof does not claim the visible summary was preserved. - **Deployment boundary:** current-head CI/review and hosted same-source Retry/ordinary-workspace reuse acceptance passed. The separate visible-summary publication failure remains owned by the publication track. Combined hosted UI confirmation of the native-agent Test path is coordinated separately. Actual missing-proof UI/provider qualification has passed in the isolated local instance above; prior hosted Stop and same-input handoff proof remains separate. No merge or deployment is performed by this pull request. ## Risks Recovery state and termination receipts must match the exact source run and allocation. Missing or foreign proof keeps the hold. The chat owner must retain the original receipt and actor and recheck current authority. Cleanup cannot grant authority or replay the stopped turn. Existing native ownership, Stop, budget, permission and uncertain-effect guards remain in place. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex, with reasoning, repository tools, source editing and test execution. The context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (affected suites and isolated failing file pass; broad timeout retained above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (current fcb has 53 successful checks/two conditional Storybook skips) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 34 +++++- maps/agent-access-map/agents.md | 8 +- maps/agent-access-map/workspaces.md | 6 + maps/feature-map/agents.md | 10 ++ maps/feature-map/recovery.md | 2 +- maps/feature-map/workspaces.md | 9 ++ .../agent-runner-qualification-routes.test.ts | 50 ++++++++ .../chat-channels.integration.test.ts | 115 ++++++++++++++++++ .../src/__tests__/durable-chat-wakeup.test.ts | 16 +++ .../services/agent-runner-qualification.ts | 19 ++- server/src/services/chat-channels.ts | 3 +- .../services/chat-channels/inbound/wakeups.ts | 47 ++++++- server/src/services/durable-chat-wakeup.ts | 19 +++ server/src/services/environment-runtime.ts | 2 +- .../execution-projection-recovery.test.ts | 39 ++++++ .../src/services/execution-projection.test.ts | 33 +++++ server/src/services/execution-projection.ts | 4 +- .../services/execution-recovery-resolution.ts | 58 +++++++-- server/src/services/heartbeat/queue.ts | 24 ++-- .../services/heartbeat/run-control.test.ts | 32 +++++ server/src/services/heartbeat/run-control.ts | 28 +++-- .../heartbeat/workspace-preparation.test.ts | 63 ++++++++++ .../heartbeat/workspace-preparation.ts | 39 ++++-- .../services/isolated-task-directory.test.ts | 31 +++++ .../src/services/isolated-task-directory.ts | 22 +++- .../native-failed-workspace-recovery.test.ts | 40 +++++- .../AgentConfigForm.render.test.tsx | 80 ++++++++++++ ui/src/components/AgentConfigForm.tsx | 23 +++- ui/src/components/IssueRunLedger.test.tsx | 17 +++ ui/src/components/IssueRunLedger.tsx | 5 +- .../agents/codex-runner/upgrade.stories.tsx | 37 +++++- 31 files changed, 849 insertions(+), 66 deletions(-) create mode 100644 server/src/services/execution-projection-recovery.test.ts diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 1228bb99dea..8269a7f4589 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -749,7 +749,7 @@ The handshake failure code is distinct from a session-identity mismatch. A timeo An explicit recovery action is a typed liveness repair path for a source issue. It is the recovery primitive; the action can be rendered directly on the source issue or backed by a separate recovery issue when the repair needs its own work item. -A terminal native failure can retain a result accepted before checkpoint or cleanup failed. That result is historical evidence, not a live controller. New user input may start a fresh turn after the controller and execution environment have stopped and ordinary admission checks pass. Preserve the failed run, its result, and its recovery budget. Do not commit the old result, infer action outcomes, or replay the failed turn. A message saved while cleanup is pending must be reconsidered after verified cleanup and delivered once. Admission must consume its deferred receipt in the same transaction that creates the new run, including in agent chat, where later messages keep their separate turns. Completing that run must not promote the consumed receipt again. Workspace-export repair retains its separate saved-result recovery path. +A terminal native failure can retain a result accepted before checkpoint or cleanup failed. That result is historical evidence, not a live controller. New user input may start a fresh turn after the controller and execution environment have stopped and ordinary admission checks pass. Preserve the failed run, its result, and its recovery budget. Do not commit the old result, infer action outcomes, or replay the failed turn. A message saved while cleanup is pending must be reconsidered after verified cleanup and delivered once. External chat messages retain their original inbound action, wake receipt and actor. Cleanup delegates reconsideration to the live inbound owner, which revalidates the endpoint, conversation generation, principal and exact saved source before normal admission. The existing stranded-queue sweep uses the same owner after a lost callback or restart; it cannot substitute a generic assignment wake. A rejected source must not prevent other saved messages from being reconsidered. Admission must consume its deferred receipt in the same transaction that creates the new run, including in agent chat, where later messages keep their separate turns. Completing that run must not promote the consumed receipt again. Workspace-export repair retains its separate saved-result recovery path. When a legacy turn admitted by an explicit user message fails or times out, a bounded transient retry needs its own durable continuation authorization. Initial admission binds the message's exact body digest and revision. Scheduling requires that original binding and revalidates the human author, source task, assignee, and execution ownership, then records the successor's receipt in the same transaction as its run. Historical receipts without a message binding cannot authorize automatic retries and are not backfilled. The parent receipt remains intact. Dispatch checks the exact successor, retry parent, and unchanged message binding again. Deleted or edited input, changed actors or ownership, superseded work, and duplicate successors cannot renew authority. One-run Retry and Interrupt intents retain their separate admission contracts; they are not renewable user-message authority. Other retry policies cannot borrow this receipt: their scheduling fails closed. Successful explicit turns do not create a prose-only missing-comment follow-up with spent authority. @@ -1444,7 +1444,7 @@ Real gates still apply: company and task ownership, active provider ownership, b A source that already stopped and was released remains fenced until its export settles; restart reconciliation scans these released sources too. Both provider reuse and destruction check the physical allocation for a shutdown fence or file-repair pin, including historical lease rows. -An operator Stop waits for provider termination. Remote sandbox providers may return a stopped/deleted receipt after their control-plane operation completes. Paperclip binds that receipt to the company, run, and exact lease; successful file cleanup, a terminal run row, or an in-sandbox shutdown event is not sufficient. Legacy conversational runs receive their cancellation acknowledgement after all remote leases have confirmed termination. Stop alone never creates a continuation. A user message queued during remote cleanup is reconsidered when the provider confirms termination; it still passes normal admission and adopts pending comment IDs in order. Once stopped, the next explicit wake uses the same queue. A compatible saved ACP session can resume, and an unavailable or incompatible session can start fresh with the full task context. Run credentials and scratch paths remain scoped to the new run. A subtree pause requires Resume; a message does not bypass it. +An operator Stop waits for provider termination. Remote sandbox providers may return a stopped/deleted receipt after their control-plane operation completes. Paperclip binds that receipt to the company, run, and exact lease; successful file cleanup, a terminal run row, or an in-sandbox shutdown event is not sufficient. Legacy conversational runs and native runs with an audited run-only Stop receive their shared cancellation acknowledgement after all remote leases have confirmed termination. A native interrupt acknowledgement alone does not establish containment. Workspace preservation remains a separate obligation and its recovery hold survives this acknowledgement. Stop alone never creates a continuation. A user message queued during remote cleanup is reconsidered when the provider confirms termination; it still passes normal admission and adopts pending comment IDs in order. Once stopped, the next explicit wake uses the same queue. A compatible saved ACP session can resume, and an unavailable or incompatible session can start fresh with the full task context. Run credentials and scratch paths remain scoped to the new run. A subtree pause requires Resume; a message does not bypass it. Cancelled runs and runs with a recorded pending stop lose run-scoped API write authority, while reads remain available for diagnostics. This applies to ordinary tasks as well as conversations. Task and interaction mutations recheck that authority in the write transaction, so a cancelled run cannot overwrite a recovery disposition with a late Done or resolve an interaction after revocation. A task handoff that intentionally stops its own run can commit only with a server receipt tied to that exact request. Direct sandbox CLI adapters such as Grok register cancellation before preparation and keep ownership until host-owned termination is verified, including a sandbox acquired before adapter registration. A failed stop does not acknowledge cancellation or abandon an outstanding remote command. Interrupted workspace restore failures remain recorded for recovery; a stop receipt proves termination, not successful file restoration. @@ -2226,3 +2226,33 @@ use the same activity projection. Board-owned repairs stay actionable until a verified native retry actually starts. Once an explicit board retry is running, its live activity takes precedence over the prior owner and exhausted budget. Resolved and cancelled actions remain resolved. + + +### Task directories after a fresh retry + +A fresh repository-free retry receives its own private attempt directory. Once +that attempt succeeds, later ordinary input reuses the task's currently bound +directory, including an attempt directory, and preserves its saved files. The +binding is re-read before preparation. Only the exact company's task namespace +is eligible; closed or changed bindings, foreign paths, symlinks and missing +retained directories fail validation. A later explicitly fresh retry still +receives a separate clean sibling and does not copy or replay the old turn. + +### Stopped execution workspace recovery instructions + +A stopped run can retain a historical workspace repair instruction after machine +recovery has saved the files. Its execution projection uses the workspace +recovery state bound to that exact run for the next instruction, as the task +blocker does. Recovery in progress, a failed repair, and recovered files must +remain distinct. This read-only instruction does not release the replay hold, +change the stopped run, or authorize another provider turn. A recorded explicit +successor still takes precedence over the old recovery instruction. + + +A terminal native run can leave an older exact-session retry action behind when +workspace recovery takes ownership. The existing execution disposition worker +retires that action only after the exact run and coordinator are terminal, no +controller lease, result, successor or retry remains, and a separate source-bound +workspace hold records that uncertain actions cannot replay. The worker keeps +that hold and all saved work. It does not change the task or start a provider. +The same bounded reconciliation pass handles historical actions after restart. diff --git a/maps/agent-access-map/agents.md b/maps/agent-access-map/agents.md index f091a126d4e..dd89f5abef0 100644 --- a/maps/agent-access-map/agents.md +++ b/maps/agent-access-map/agents.md @@ -123,7 +123,7 @@ API: `present` — representative operation IDs: receives `canManage: false`. Saving never creates a run or wakeup. - `PUT /api/agents/{id}/instructions-bundle/file`. - `PATCH /api/agents/{id}` with an explicit `runner: "paperclip"` upgrade. -- `POST /api/agents/{id}/qualify-runner` to qualify the saved effective sandbox before upgrading. +- `POST /api/agents/{id}/qualify-runner` to qualify the saved effective sandbox before upgrading or testing an existing native runner. - `PATCH /api/agents/{id}` with `runtimeSettings: { harness, model?, aiConnection?, expected: { model, aiConnection } }` for a narrow roster edit. Each request handles one agent; the UI batches at most four writes and collects per-agent outcomes. Agent access: `available` — Native read/update/history/restore instruction tools; REST / @@ -149,7 +149,11 @@ authorization as saving the upgrade. The server resolves the saved computer; `expectedEnvironmentId` rejects stale state and cannot select another computer. Qualification checks company binding, returns only its receipt, and audits within the agent's company. General instance environment probes still require instance -admin. No probe runs merely on page load. Operator browser evidence does not +admin. An existing native agent's clean configuration Test uses the same +saved-computer qualification before its runtime test. A current proof is reused; +a missing or stale proof requires a real successful probe and verified cleanup. +Draft settings and target changes do not qualify another computer. No probe runs +merely on page load. Operator browser evidence does not qualify the agent permission paths for this operation. Evidence: `source-only` — [implementation](../../server/src/routes/agents.ts#L1395) at diff --git a/maps/agent-access-map/workspaces.md b/maps/agent-access-map/workspaces.md index 5b501668c9f..2dc6ba5ad8b 100644 --- a/maps/agent-access-map/workspaces.md +++ b/maps/agent-access-map/workspaces.md @@ -49,6 +49,12 @@ those APIs with their admitted run token. These paths do not require a project. Policy: Company/project/workspace visibility and runtime authority; active run, trust preset and task ownership where required. +Ordinary input may reuse the exact currently bound private task directory after +an explicitly fresh retry. Preparation rechecks company, source task, current +binding and canonical path; missing directories, symlinks and foreign roots are +rejected. This preserves files without granting new access or replaying the +previous turn. An explicitly fresh retry still creates a clean sibling. + Task workspace writes require `issue:mutate`, an active owned task/run for agents, and current source visibility. Shared physical roots also require source-project workspace write authority. Accepting a new pending choice advances its binding diff --git a/maps/feature-map/agents.md b/maps/feature-map/agents.md index c6e9c336570..21dee0c054b 100644 --- a/maps/feature-map/agents.md +++ b/maps/feature-map/agents.md @@ -185,6 +185,16 @@ the original agent, without showing success on the new page or redirecting back. Focused coverage: [save receipt](../../ui/src/pages/AgentRunnerUpgrade.test.tsx) and [configuration form](../../ui/src/components/AgentConfigForm.render.test.tsx). +For an existing native agent on a Daytona snapshot or immutable sandbox image, +use **Test** in its saved Harness or Runtime configuration after a managed +computer update. This action qualifies the saved computer under the existing +agent configuration permission before testing the runtime. A current proof must +remain unchanged. A missing proof requires a real successful probe and verified +cleanup. Failed qualification must stop the runtime test. Unsaved configuration +or a different computer must not qualify a draft target. A company owner must not +need instance-admin permission, and the generic environment probe must remain +restricted. Existing runs retain their recorded allocation and snapshot binding. + Storybook: [Runner upgrade](../../ui/storybook/stories/agents/codex-runner/upgrade.stories.tsx) mounts the production page and shell with mutable HTTP fixtures for both harnesses, pending, failure/retry, incompatible settings, repair, native/unsupported states, diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index 03e16003337..fcf47da1e27 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -150,7 +150,7 @@ Use these checks alongside the entry-point recipes below and the operation recor | Capability ID | Action and persisted result | Denial, race and retry checks | | --- | --- | --- | -| `recovery/diagnosis` | Open the source task and exact run; compare cause, source action, original owner, recovery owner, attempt bounds and next action after reload. | Test an unrelated/private task and a stale run notice. Read-side retirement of a stale action must not create a new attempt, participant, or permission to replay. | +| `recovery/diagnosis` | Open the source task and exact run; compare cause, source action, original owner, recovery owner, attempt bounds and next action after reload. | Test an unrelated/private task and a stale run notice. Read-side retirement of a stale action must not create a new attempt, participant, or permission to replay. After workspace recovery, compare task and run instructions; recovered files must not leave instructions to repair a retained sandbox. The stopped run keeps its replay hold until continuation admission. An obsolete exact-session notice retires only after terminal coordinator proof and a separate exact-run workspace hold; no new execution or file cleanup follows. | | `recovery/bounded-continuation` | Use a deterministic recoverable interruption and inspect the original run/session, attempt receipts, saved result and eventual useful output. | Repeat restart/receipt delivery, race Stop or a new owner, and retain unknown external outcomes. Accepted completion/finalization recovery must not start a new provider turn or repeat its side effect. | | `recovery/source-action` | Inspect active and retained actions, follow the repair owner, and resolve one eligible source-bound action with its exact ID. | Retry the settled resolution and reject a stale/foreign action. Recovery ownership alone cannot mutate the source task; inspect source status, audit and wake independently. | | `recovery/exhaustion` | Exhaust or age a bounded fixture; use the currently offered retry once, then inspect its durable acknowledgement and actual continuation. | Reassign, pause, stop, close, add an unresolved dependency, or impose a budget hold before retry. The old notice must not reopen terminal work or run the wrong owner. | diff --git a/maps/feature-map/workspaces.md b/maps/feature-map/workspaces.md index 0286edafbf9..e82349ce27c 100644 --- a/maps/feature-map/workspaces.md +++ b/maps/feature-map/workspaces.md @@ -57,6 +57,15 @@ Automated: [workspace card](../../ui/src/components/IssueWorkspaceCard.test.tsx) Manual: Run a task that writes a small file, follow the bound workspace, and inspect the file and revision. Reload and verify links still refer to the same workspace, not another checkout or a stale run. +For a repository-free isolated task, complete an explicitly fresh retry, save a +marker, then send ordinary new input. It must retain the bound attempt directory +and marker. A later explicitly fresh retry must use a separate clean sibling. +The [production preparation regression](../../server/src/services/heartbeat/workspace-preparation.test.ts) +uses PostgreSQL and real private directories; it also rejects changed bindings. +The [directory boundary suite](../../server/src/services/isolated-task-directory.test.ts) +checks namespace, missing-directory and symlink refusal. Neither suite proves a +hosted provider turn. + For slow source preparation, configure a short PostgreSQL idle-transaction timeout in a disposable instance. Hold preparation beyond that interval and verify it retains its source lock, rejects competing preparation, and leaves diff --git a/server/src/__tests__/agent-runner-qualification-routes.test.ts b/server/src/__tests__/agent-runner-qualification-routes.test.ts index 3d50ad7d6af..10c1980d248 100644 --- a/server/src/__tests__/agent-runner-qualification-routes.test.ts +++ b/server/src/__tests__/agent-runner-qualification-routes.test.ts @@ -9,6 +9,9 @@ import { grantsForHumanRole } from "../services/company-member-roles.js"; import { agentRoutes } from "../routes/agents.js"; import { environmentRoutes } from "../routes/environments.js"; import { errorHandler } from "../middleware/error-handler.js"; +import { environmentService } from "../services/environments.js"; +import { resolveEnvironmentDriverConfigForRuntime } from "../services/environment-config.js"; +import { fingerprintEnvironmentSandboxProviderConfig, SANDBOX_PLATFORM_PROBE_METADATA_KEY } from "../services/environment-custom-image-runtime.js"; const probe = vi.hoisted(() => vi.fn()); vi.mock("../services/environment-probe.js", async original => ({ @@ -69,6 +72,53 @@ describe.skipIf(!support.supported)("agent-scoped runner qualification", () => { expect(generic.status).toBe(403); expect(probe).toHaveBeenCalledTimes(1); }); + it("lets the owner requalify an existing native runner's saved computer without changing its configuration", async () => { + await db.update(agents).set({ adapterType: "paperclip_runner", adapterConfig: { provider: "opencode", model: "example/model" } }) + .where(eq(agents.id, agentId)); + const before = await db.select().from(agents).where(eq(agents.id, agentId)); + const response = await qualify(); + expect(response.status).toBe(200); + expect(response.body).toMatchObject({ ok: true, details: { platformQualificationRecorded: true } }); + expect(probe).toHaveBeenCalledWith(db, expect.objectContaining({ id: environmentId }), + expect.objectContaining({ companyId, assertCompanyBinding: true, acquireSandboxRuntimeLease: true })); + expect(await db.select().from(agents).where(eq(agents.id, agentId))).toEqual(before); + expect(JSON.stringify(response.body)).not.toMatch(/provider-secret|private-id/); + expect((await request(app).post(`/api/environments/${environmentId}/probe?companyId=${companyId}`).send({})).status).toBe(403); + }); + + it("reuses a current native snapshot proof without invalidating it or allocating another probe", async () => { + await db.update(agents).set({ adapterType: "paperclip_runner", adapterConfig: { provider: "opencode" } }).where(eq(agents.id, agentId)); + const config = { provider: "daytona", snapshot: "managed-snapshot" }; + const fingerprint = fingerprintEnvironmentSandboxProviderConfig(config); + const saved = (await environmentService(db).getById(environmentId))!; + const runtime = await resolveEnvironmentDriverConfigForRuntime(db, companyId, saved, { applyCustomImageTemplate: true, readOnlyCustomImageTemplate: true }); + const runtimeFingerprint = fingerprintEnvironmentSandboxProviderConfig(runtime.config as typeof config); + const metadata = { [SANDBOX_PLATFORM_PROBE_METADATA_KEY]: { [companyId]: { + v: 2, companyId, baseConfigFingerprint: fingerprint, runtimeConfigFingerprint: runtimeFingerprint, + snapshot: config.snapshot, template: null, + snapshotIdentity: { provider: "daytona", requestedRef: config.snapshot, id: randomUUID(), name: config.snapshot, + organizationId: "fixture-organization", createdAt: new Date().toISOString() }, + facts: { platform: "linux", architecture: "x64", libc: "glibc", glibcVersion: "2.36" }, + } } }; + await db.update(environments).set({ metadata }).where(eq(environments.id, environmentId)); + expect((await qualify()).body).toMatchObject({ ok: true, details: { platformQualificationRecorded: true } }); + expect(probe).not.toHaveBeenCalled(); + expect((await db.select().from(environments).where(eq(environments.id, environmentId)))[0]?.metadata).toEqual(metadata); + }); + + it.each(["revoked", "changed-computer", "foreign-computer"])("keeps native saved-computer qualification closed for %s authority", async state => { + await db.update(agents).set({ adapterType: "paperclip_runner", adapterConfig: { provider: "opencode" } }).where(eq(agents.id, agentId)); + if (state === "revoked") await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.principalId, userId)); + if (state === "changed-computer") await db.update(agents).set({ defaultEnvironmentId: null }).where(eq(agents.id, agentId)); + if (state === "foreign-computer") { + const other = randomUUID(); + await db.insert(companies).values({ id: other, name: "Other", issuePrefix: `F${other.slice(0, 7)}` }); + await db.insert(builtInManagedResources).values({ companyId: other, bundleKey: "test", resourceKind: "environment", resourceKey: "computer", resourceId: environmentId, stockVersion: "test", stockHash: "test" }); + } + expect((await qualify()).status).toBe(state === "changed-computer" ? 409 : 403); + expect(probe).not.toHaveBeenCalled(); + }); + it("rejects an owner whose configure grant is absent before any provider work", async () => { await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.principalId, userId)); expect((await qualify()).status).toBe(403); diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index 82f108ec69e..347ec611be1 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -2421,6 +2421,121 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { await db.update(chatEndpoints).set({ botUsername: "gonzo[bot]", status: "active" }).where(eq(chatEndpoints.id, peer.id)); return peer; } + it.each(["callback", "restart", "race", "unknown_stop", "wrong_allocation", "paused_endpoint", "revoked_principal", "deleted_comment", "changed_generation", "foreign_scope", "changed_source", "revoked_before_valid"])( + "reconsiders the same GitHub reply after stopped native workspace cleanup (%s)", async scenario => { + const f = await subscribedGitHubFixture("github:paperclipai/paperclip:419"); + const issueId = f.conversation.issueId!; + const heartbeat = heartbeatService(db); + const sourceRunId = randomUUID(), leaseId = randomUUID(), providerLeaseId = randomUUID(); + const capacityRunId = randomUUID(), stoppedAt = new Date(Date.now() - 10_000); + const initial = (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.assignedAgentId)))[0]; + await db.update(agentWakeupRequests).set({ status: "completed", runId: sourceRunId, finishedAt: stoppedAt }) + .where(eq(agentWakeupRequests.id, initial.id)); + await db.update(companies).set({ defaultResponsibleUserId: "owner-user" }).where(eq(companies.id, f.companyId)); + await db.update(agents).set({ adapterType: "paperclip_runner", status: "running", + runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }).where(eq(agents.id, f.assignedAgentId)); + await db.insert(heartbeatRuns).values({ id: capacityRunId, companyId: f.companyId, agentId: f.assignedAgentId, + status: "running", contextSnapshot: { issueId: randomUUID() } }); + runningProcesses.set(capacityRunId, { child: {} as never, graceSec: 0, processGroupId: null }); + onTestFinished(() => { runningProcesses.delete(capacityRunId); }); + await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId: f.companyId, agentId: f.assignedAgentId, + status: "cancelled", startedAt: new Date(stoppedAt.getTime() - 1000), finishedAt: stoppedAt, + runtimeMode: "native", nativeIssueId: issueId, nativePhase: "observed", + contextSnapshot: { issueId, source: "chat:github", wakeCommentIds: [initial.payload?.commentId] }, + resultJson: { workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { leaseIds: [leaseId] } }, + }); + await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId, + runId: sourceRunId, phase: "terminal_failure" }); + await db.insert(environmentLeases).values({ id: leaseId, companyId: f.companyId, issueId, + heartbeatRunId: sourceRunId, provider: "daytona", providerLeaseId, + status: "pending_cleanup", cleanupStatus: "failed", leasePolicy: "ephemeral" }); + await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: issueId, + kind: "active_run_watchdog", status: "resolved", ownerType: "system", outcome: "blocked", + cause: "native_continuation_requires_reconciliation", fingerprint: randomUUID(), nextAction: "Retained workspace repair", + evidence: { runId: sourceRunId, automaticRecovery: { replay: "blocked" }, + workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { leaseIds: [leaseId] }, + workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: sourceRunId, + status: "recovered", stage: "saving_receipts", attempt: 1, attemptLimit: 6, retryable: true, requestIds: [] }, + systemRecovery: { ...createSystemRecovery(sourceRunId), state: "complete", phase: "reconcile_effects" }, + }, + }); + f.wakeup.mockImplementation(heartbeat.wakeup); + await deliverMessage({ callbacks: f.callbacks, endpointId: f.endpoint.id, provider: "github", thread: f.thread, + trigger: "subscribed_message", message: makeMessage({ id: "41903", text: "A genuinely new author reply after Stop", + userId: "42", userName: "octocat" }) }); + const comment = (await readIssueCommentsByIssueId(issueId)).find(row => row.body.includes("genuinely new author reply"))!; + const [receipt] = (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.assignedAgentId))) + .filter(row => row.id !== initial.id); + expect(receipt).toMatchObject({ status: "deferred_issue_execution", runId: null, + payload: { executionWait: { reason: "remote_cleanup" } } }); + let nextReceipt: typeof receipt | undefined; + let nextCommentId: string | undefined; + if (scenario === "revoked_before_valid") { + const thread = makeThread({ channelId: "paperclipai/paperclip", id: "github:paperclipai/paperclip:419:rc:4332012267" }).thread; + const { id: _id, ...conversation } = f.conversation; + await db.insert(chatConversations).values({ ...conversation, externalThreadId: thread.id }); + await deliverMessage({ callbacks: f.callbacks, endpointId: f.endpoint.id, provider: "github", thread, + trigger: "subscribed_message", message: makeMessage({ id: "41904", text: "A live source behind the deleted source", + userId: "42", userName: "octocat" }) }); + nextCommentId = (await readIssueCommentsByIssueId(issueId)).find(row => row.body.includes("live source behind"))!.id; + nextReceipt = (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.assignedAgentId))) + .find(row => row.id !== initial.id && row.id !== receipt.id); + expect(nextReceipt).toMatchObject({ status: "deferred_issue_execution", runId: null }); + await db.update(issueComments).set({ deletedAt: new Date() }).where(eq(issueComments.id, comment.id)); + } + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId)); + if (scenario !== "unknown_stop") await db.update(environmentLeases).set({ status: "expired", cleanupStatus: "success", + releasedAt: new Date(), metadata: { remoteExecutionTermination: remoteTerminationReceipt(lease, + { providerLeaseId: scenario === "wrong_allocation" ? randomUUID() : providerLeaseId, state: "destroyed" }) } }) + .where(eq(environmentLeases.id, leaseId)); + if (scenario === "paused_endpoint") await db.update(chatEndpoints).set({ status: "paused" }).where(eq(chatEndpoints.id, f.endpoint.id)); + if (scenario === "revoked_principal") await db.update(chatIdentityLinks).set({ status: "revoked", revokedAt: new Date() }) + .where(eq(chatIdentityLinks.principalId, f.principal.id)); + if (scenario === "deleted_comment") await db.update(issueComments).set({ deletedAt: new Date() }).where(eq(issueComments.id, comment.id)); + if (scenario === "changed_generation") await db.update(chatConversations).set({ sessionGeneration: 2 }).where(eq(chatConversations.id, f.conversation.id)); + if (scenario === "changed_source") { + const [action] = await db.select().from(chatActions).where(eq(chatActions.id, receipt.id)); + await db.update(chatActions).set({ payload: { ...action.payload, commentId: initial.payload?.commentId } }) + .where(eq(chatActions.id, action.id)); + } + if (scenario === "restart") { + await f.service.shutdown(); + // Fresh application owner and scheduler must recover from saved data, + // with no in-memory request object left in the retired chat service. + createService(new FakeChatSdkRuntime(), f.providerFetch, { wakeup: heartbeatService(db).wakeup }); + } + const source = (await readHeartbeatRunsByCompanyId(f.companyId)).find(row => row.id === sourceRunId)!; + const { resumeDeferredChatWake } = await import("../services/durable-chat-wakeup.js"); + if (scenario === "foreign_scope") await resumeDeferredChatWake(db, { companyId: randomUUID(), agentId: f.assignedAgentId, + issueId, requestId: receipt.id }); + else if (scenario === "callback" || scenario === "revoked_before_valid") await heartbeat.resumeRemoteStopComments(source); + else if (scenario === "race") await Promise.all([heartbeat.resumeRemoteStopComments(source), heartbeat.resumeQueuedRuns()]); + else await heartbeatService(db).resumeQueuedRuns(); + const [settled] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.id)); + const successors = (await readHeartbeatRunsByCompanyId(f.companyId)).filter(row => row.wakeupRequestId === receipt.id); + const admitted = ["callback", "restart", "race"].includes(scenario); + expect(successors).toHaveLength(admitted ? 1 : 0); + expect(settled).toMatchObject({ id: receipt.id, idempotencyKey: receipt.idempotencyKey, + requestedByActorId: receipt.requestedByActorId, requestedAt: receipt.requestedAt, + status: admitted ? "queued" : "deferred_issue_execution" }); + if (admitted) { + expect(extractWakeCommentIds(successors[0].contextSnapshot)).toEqual([comment.id]); + expect(successors[0].contextSnapshot?.chatConversationId).toBe(f.conversation.id); + expect(successors[0].retryOfRunId).toBeNull(); + await heartbeat.resumeRemoteStopComments(source); + await heartbeat.resumeQueuedRuns(); + expect((await readHeartbeatRunsByCompanyId(f.companyId)).filter(row => row.wakeupRequestId === receipt.id)).toHaveLength(1); + } + if (nextReceipt) { + const admittedBehindRevoked = (await readHeartbeatRunsByCompanyId(f.companyId)).filter(row => row.wakeupRequestId === nextReceipt.id); + expect(admittedBehindRevoked).toHaveLength(1); + expect(extractWakeCommentIds(admittedBehindRevoked[0].contextSnapshot)).toEqual([nextCommentId]); + } + expect((await readHeartbeatRunsByCompanyId(f.companyId)).find(row => row.id === sourceRunId)) + .toMatchObject({ status: "cancelled", resultJson: { workspaceRestoreFailure: "restore_failed" } }); + }, 40_000, + ); + it("loads only the current configured instruction skills, never provider links or another company", async () => { const f = await reviewBotFixture(); const other = await seedCompany(); diff --git a/server/src/__tests__/durable-chat-wakeup.test.ts b/server/src/__tests__/durable-chat-wakeup.test.ts index a979df891c2..9ef4efe4241 100644 --- a/server/src/__tests__/durable-chat-wakeup.test.ts +++ b/server/src/__tests__/durable-chat-wakeup.test.ts @@ -321,6 +321,22 @@ describe("durable inbound chat scheduler receipts", () => { }); }); + it("reconsiders the same held inbound receipt only through its live durable owner", async () => { + const f = await fixture(); + const clearHold = await executionHold(f); + const request = f.request(); + expect(await f.wake(request)).toBeNull(); + await clearHold(); + const reconsidered = createDurableChatWakeupRequest({ ...request, resumeDeferred: true }); + const [first, second] = await Promise.all([f.wake(reconsidered), f.wake(reconsidered)]); + const receipts = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.agentId)); + expect(receipts).toHaveLength(1); + expect(receipts[0]).toMatchObject({ id: request.id, status: "queued", requestedAt: request.requestedAt }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.wakeupRequestId, request.id))).toHaveLength(1); + expect(first?.id ?? second?.id).toBe(receipts[0].runId); + expect(f.authorize.mock.calls.length).toBeGreaterThan(1); + }); + async function retryFixture(deferred = false) { const f = await fixture(deferred); const applicationId = randomUUID(), diff --git a/server/src/services/agent-runner-qualification.ts b/server/src/services/agent-runner-qualification.ts index f6b73618e2e..8533ed19452 100644 --- a/server/src/services/agent-runner-qualification.ts +++ b/server/src/services/agent-runner-qualification.ts @@ -2,7 +2,7 @@ import type { Db } from "@paperclipai/db"; import type { Agent, EnvironmentProbeResult } from "@paperclipai/shared"; import { conflict, unprocessable } from "../errors.js"; import { resolveAgentRunnerEnvironmentForCompany } from "./agent-runner-selection.js"; -import { probeEnvironment } from "./environment-probe.js"; +import { probeEnvironment, readSandboxPlatformQualification } from "./environment-probe.js"; import type { PluginWorkerManager } from "./plugin-worker-manager.js"; export async function qualifyAgentRunner( @@ -11,16 +11,24 @@ export async function qualifyAgentRunner( expectedEnvironmentId: string, pluginWorkerManager?: PluginWorkerManager, ): Promise { - if (!["claude_local", "codex_local"].includes(agent.adapterType)) { - throw unprocessable("This agent does not need a runner upgrade."); + if (!["claude_local", "codex_local", "paperclip_runner"].includes(agent.adapterType)) { + throw unprocessable("This agent does not support computer qualification."); } const { environment } = await resolveAgentRunnerEnvironmentForCompany(db, agent.companyId, agent.defaultEnvironmentId); if (!environment || environment.id !== expectedEnvironmentId) { - throw conflict("The agent's computer changed. Save any environment changes, reload, and retry the upgrade."); + throw conflict(agent.adapterType === "paperclip_runner" + ? "The agent's computer changed. Save the computer selection, reload, and run its test again." + : "The agent's computer changed. Save any environment changes, reload, and retry the upgrade."); } if (environment.driver !== "sandbox") { throw unprocessable("This computer does not need sandbox qualification."); } + // A normal native-agent Test still probes its runtime. Reuse a valid saved + // platform proof so that test does not invalidate a healthy computer first. + if (agent.adapterType === "paperclip_runner" && await readSandboxPlatformQualification(db, environment, agent.companyId)) { + return { ok: true, driver: "sandbox", summary: "The agent's computer is qualified for runner selection.", + details: { platformQualificationRecorded: true } }; + } const result = await probeEnvironment(db, environment, { companyId: agent.companyId, pluginWorkerManager, @@ -33,7 +41,8 @@ export async function qualifyAgentRunner( ok: result.ok, driver: "sandbox", summary: qualified ? "The agent's computer is qualified for runner selection." - : "Could not qualify the agent's computer. Retry the upgrade or keep Legacy runner.", + : agent.adapterType === "paperclip_runner" ? "Could not verify the saved computer. Run test again." + : "Could not qualify the agent's computer. Retry the upgrade or keep Legacy runner.", details: { platformQualificationRecorded: qualified, ...(typeof result.details?.leaseId === "string" ? { leaseId: result.details.leaseId } : {}), diff --git a/server/src/services/chat-channels.ts b/server/src/services/chat-channels.ts index fc41debb711..e0af2b657e3 100644 --- a/server/src/services/chat-channels.ts +++ b/server/src/services/chat-channels.ts @@ -836,7 +836,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { const { publicationFiles } = createOutboundAttachments(db, { get storage() { return options.storage; } }); const { authorizeInboundWakeup, authorizeGitHubTaskRetryBatch, authorizeInboundWorkStart, authorizeFastResponse } = createInboundWakeAuthority(db, { runtimeCallbackEndpoint, lockCurrentPrincipalAuthorization }); const { ingestAttachments, attachmentOmissionDetail } = createInboundAttachments(db, { authorizeInboundWakeup, lifecycleRuntimeFence, sanitizeFilename, issuesSvc, isExternalActionAuthorizationChange, redactError, get storage() { return options.storage; } }); - const { stageInboundWakeup, acceptInboundWakeup, settleRejectedInboundWakeups, processInboundWakeup } = createInboundWakeups(db, { authorizeInboundWakeup, authorizeInboundWorkStart, attachmentOmissionDetail, isExternalActionAuthorizationChange, redactError, fetchImpl, get heartbeat() { return options.heartbeat; } }); + const { stageInboundWakeup, acceptInboundWakeup, settleRejectedInboundWakeups, processInboundWakeup, unregisterDeferredWakeOwner } = createInboundWakeups(db, { authorizeInboundWakeup, authorizeInboundWorkStart, attachmentOmissionDetail, isExternalActionAuthorizationChange, redactError, fetchImpl, get heartbeat() { return options.heartbeat; } }); const { settleTelegramAttachmentOnlyFailure } = createTelegramAttachmentFailure(db, { attachmentOmissionDetail, runtimeContextForRecord, endpointRecord, stageProviderEffect, scheduleProviderEffect, }); @@ -3529,6 +3529,7 @@ return { shutdown: async () => { shuttingDown = true; await Promise.allSettled(pendingFailedRetries()); + unregisterDeferredWakeOwner(); unregisterFailedRetryAuthority(); unregisterSlackTaskAuthority(); voice.unregisterAgentTools(); diff --git a/server/src/services/chat-channels/inbound/wakeups.ts b/server/src/services/chat-channels/inbound/wakeups.ts index a839c9b09b9..9b9f9738310 100644 --- a/server/src/services/chat-channels/inbound/wakeups.ts +++ b/server/src/services/chat-channels/inbound/wakeups.ts @@ -5,13 +5,13 @@ import { enqueueFastResponse } from "../../fast-responses.js"; import { and, eq, sql } from "drizzle-orm"; import { retryChatControlAdmission } from "../../chat-control-admission-retry.js"; import type { Db } from "@paperclipai/db"; -import { createDurableChatWakeupRequest, assertDurableChatWakeupReceipt } from "../../durable-chat-wakeup.js"; +import { createDurableChatWakeupRequest, assertDurableChatWakeupReceipt, registerDeferredChatWakeOwner } from "../../durable-chat-wakeup.js"; import { agentWakeupRequests, chatActions, chatConversations, chatDeliveries, chatEndpoints } from "@paperclipai/db"; import { forbidden } from "../../../errors.js"; import { logger } from "../../../middleware/logger.js"; import { queueIssueAssignmentWakeup } from "../../issue-assignment-wakeup.js"; import { isGitHubAutomaticDelivery, acknowledgeGitHubWakeup } from "../github/wake-authority.js"; -import type { createInboundWakeAuthority } from "./wakeup-authorization.js"; +import { inboundWakeScope, type createInboundWakeAuthority } from "./wakeup-authorization.js"; import type { createInboundAttachments } from "../inbound-attachments.js"; import type { IssueAssignmentWakeupDeps } from "../../issue-assignment-wakeup.js"; type InboundAttachmentResult = Awaited["ingestAttachments"]>>; @@ -425,5 +425,46 @@ export function createInboundWakeups(db: Db, options: WakeupOptions) { return denied; } } - return { stageInboundWakeup, acceptInboundWakeup, settleRejectedInboundWakeups, processInboundWakeup }; + const unregisterDeferredWakeOwner = registerDeferredChatWakeOwner(db, async (scope) => { + const action = await db.select().from(chatActions).where(and( + eq(chatActions.id, scope.requestId), eq(chatActions.companyId, scope.companyId), + eq(chatActions.kind, "inbound_wakeup"), eq(chatActions.status, "processed"), + )).then(rows => rows[0] ?? null); + if (!action || action.payload.agentId !== scope.agentId || action.payload.issueId !== scope.issueId) return; + const [receipt] = await db.select().from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.id, action.id), eq(agentWakeupRequests.companyId, scope.companyId), + eq(agentWakeupRequests.agentId, scope.agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"), + )); + if (!receipt || receipt.runId) return; + const authorize = async (tx: Db) => { + const [source] = await tx.select().from(chatActions).where(and( + eq(chatActions.id, action.id), eq(chatActions.companyId, scope.companyId), + eq(chatActions.kind, "inbound_wakeup"), eq(chatActions.status, "processed"), + )); + if (!source || inboundWakeScope(source) !== inboundWakeScope(action)) + throw forbidden("The saved chat source changed", { code: "chat_action_authorization_changed" }); + const current = await authorizeInboundWorkStart(tx, source); + if (current.delivery.state !== "processed") throw new Error("chat_inbound_wakeup_acceptance_not_committed"); + return current; + }; + // The same authority used at ingress checks the live endpoint, principal, + // immutable action source, conversation generation and original comment. + const context = await db.transaction(tx => authorize(tx as unknown as Db)); + const request = createDurableChatWakeupRequest({ + id: action.id, companyId: scope.companyId, agentId: scope.agentId, issueId: scope.issueId, + commentId: String(action.payload.commentId), + requestedByActorType: action.payload.requestedByActorType as "user" | "system", + requestedByActorId: String(action.payload.requestedByActorId), requestedAt: action.createdAt, + resumeDeferred: true, authorize: async tx => { await authorize(tx); }, + }); + assertDurableChatWakeupReceipt(request, receipt); + await queueIssueAssignmentWakeup({ heartbeat: options.heartbeat, issue: context.issue, + reason: "External chat message received", mutation: "chat_message_received", + contextSource: `chat:${context.endpoint.provider}`, taskKey: context.issue.identifier, + requestedByActorType: request.requestedByActorType, requestedByActorId: request.requestedByActorId, + wakeCommentId: request.commentId, chatConversationId: context.conversation.id, + attachmentOmissionReasons: action.payload.attachmentOmissionReasons as Record | undefined, + durableChatRequest: request, rethrowOnError: true }); + }); + return { stageInboundWakeup, acceptInboundWakeup, settleRejectedInboundWakeups, processInboundWakeup, unregisterDeferredWakeOwner }; } diff --git a/server/src/services/durable-chat-wakeup.ts b/server/src/services/durable-chat-wakeup.ts index b9d9f721a28..9a689254e03 100644 --- a/server/src/services/durable-chat-wakeup.ts +++ b/server/src/services/durable-chat-wakeup.ts @@ -198,6 +198,23 @@ export function unadmittedChatWakeupCondition( )`; } +export interface DeferredChatWakeScope { + companyId: string; + agentId: string; + issueId: string; + requestId: string; +} +const deferredChatWakeOwners = new WeakMap Promise>(); + +/** The live inbound owner reauthorizes the original source. Cleanup owns no chat authority. */ +export function registerDeferredChatWakeOwner(db: Db, owner: (input: DeferredChatWakeScope) => Promise) { + deferredChatWakeOwners.set(db, owner); + return () => { if (deferredChatWakeOwners.get(db) === owner) deferredChatWakeOwners.delete(db); }; +} +export async function resumeDeferredChatWake(db: Db, input: DeferredChatWakeScope) { + await deferredChatWakeOwners.get(db)?.(input); +} + /** Internal authority: this object cannot be supplied through a JSON API. */ export interface DurableChatWakeupRequest { readonly id: string; @@ -210,6 +227,8 @@ export interface DurableChatWakeupRequest { readonly requestedAt: Date; readonly idempotencyKey: string; readonly failedRunRetry?: { readonly failedRunId: string }; + /** Only the current inbound owner may reconsider its already saved, unconsumed receipt. */ + readonly resumeDeferred?: true; readonly authorize: (tx: Db) => Promise; readonly onAdmitted?: (tx: Db, receipt: { runId: string | null }) => Promise; } diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.ts index b3c53737bef..58a27c668a8 100644 --- a/server/src/services/environment-runtime.ts +++ b/server/src/services/environment-runtime.ts @@ -1875,7 +1875,7 @@ function createSandboxEnvironmentDriver( ? qualifiedSandboxSnapshotId(configuredEnvironment, input.companyId, parsed.config, selectedTemplate.value) : null; if (input.adapterType === "paperclip_runner" && !pinnedSnapshotId && hasSandboxSnapshotQualification(configuredEnvironment, input.companyId)) { - throw new Error("sandbox_snapshot_qualification_required: The snapshot qualification is missing or stale. Test this environment again before running Paperclip Runner."); + throw new Error("sandbox_snapshot_qualification_required: The snapshot qualification is missing or stale. Open the agent's Harness / Runtime page and select Run test before running Paperclip Runner."); } if (input.configuredEnvironment) parsed.config = { ...parsed.config, reuseLease: storedParsed.config.reuseLease, archiveOnRelease: storedParsed.config.archiveOnRelease }; if (pinnedSnapshotId) parsed.config = { ...parsed.config, snapshot: pinnedSnapshotId }; diff --git a/server/src/services/execution-projection-recovery.test.ts b/server/src/services/execution-projection-recovery.test.ts new file mode 100644 index 00000000000..a05c868d864 --- /dev/null +++ b/server/src/services/execution-projection-recovery.test.ts @@ -0,0 +1,39 @@ +import { randomUUID } from "node:crypto"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, companies, createDb, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { executionProjectionsForRuns } from "./execution-projection.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("persisted stopped-workspace projection", () => { + let database: Awaited>; + let db: ReturnType; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("stopped-workspace-projection"); + db = createDb(database.connectionString); + }, 60_000); + afterAll(async () => { await db?.$client.end(); await database?.cleanup(); }); + + it("reads recovered files on a cancelled run through the real batched query", async () => { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Stopped workspace", issuePrefix: "SWP" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Reviewer", adapterType: "paperclip_runner" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Review current changes", assigneeAgentId: agentId }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issueId, + runtimeMode: "native", status: "cancelled", finishedAt: new Date(), + contextSnapshot: { issueId }, resultJson: { workspaceRestoreFailure: "restore_failed" } }); + await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId, phase: "observed" }); + await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog", + cause: "native_continuation_requires_reconciliation", fingerprint: `native-workspace:${runId}`, + ownerType: "system", status: "resolved", outcome: "blocked", resolvedAt: new Date(), + nextAction: "The original sandbox is retained for workspace repair.", + evidence: { runId, continuationDelivery: "invalidated", + automaticRecovery: { replay: "blocked", actionOutcome: "unknown", policy: "preserve_without_replay_v1" }, + workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId, status: "recovered", + stage: "saving_receipts", attempt: 1, attemptLimit: 6, retryable: true, requestIds: [] }, + } }); + const projections = await executionProjectionsForRuns(db, companyId, [runId]); + expect(projections.get(runId)).toMatchObject({ phase: "recovery_needed", label: "Stopped", + nextAction: "Saved work recovered. Send a message to continue.", permittedActions: ["inspect_run"] }); + }); +}); diff --git a/server/src/services/execution-projection.test.ts b/server/src/services/execution-projection.test.ts index df55aa800cc..64ed679e73e 100644 --- a/server/src/services/execution-projection.test.ts +++ b/server/src/services/execution-projection.test.ts @@ -37,6 +37,39 @@ const project = ( ) => projectExecution(r, c, pending, undefined, now); describe("execution truth projection", () => { + it.each(["queued", "recovering", "blocked", "recovered"] as const)("shows the exact saved workspace recovery state instead of stale repair instructions: %s", status => { + const sourceRunId = "ec036e06-90a3-4b69-81db-a8eab7dbfbac"; + const source = run({ id: sourceRunId, status: "failed", errorCode: "native_continuation_requires_reconciliation" }); + const action = { + status: "resolved" as const, + cause: "native_continuation_requires_reconciliation", + nextAction: "The original sandbox is retained for workspace repair.", + evidence: { + runId: sourceRunId, + automaticRecovery: { policy: "preserve_without_replay_v1", actionOutcome: "unknown", replay: "blocked" }, + continuationDelivery: "invalidated", + workspaceRecovery: { + schema: "paperclip.workspace-recovery.v1", runId: sourceRunId, status, + stage: "saving_receipts", attempt: 1, requestIds: [], + ...(status === "blocked" ? { message: "The saved source cannot be read safely." } : {}), + }, + }, + }; + const projected = projectExecution(source, coordinator({ phase: "terminal_failure" }), [], action, now); + expect(projected).toMatchObject({ + phase: "recovery_needed", label: "Stopped", cause: action.cause, + nextAction: status === "recovered" ? "Saved work recovered. Send a message to continue." + : status === "blocked" ? "The saved source cannot be read safely." : "Recovering saved work…", + permittedActions: ["inspect_run"], successorRunId: null, + }); + expect(source.status).toBe("failed"); + expect(action.evidence.automaticRecovery.replay).toBe("blocked"); + expect(projectExecution(source, undefined, [], { ...action, + evidence: { ...action.evidence, workspaceRecovery: { ...action.evidence.workspaceRecovery, + runId: "5f227d75-b314-4511-bf23-9c615277e69b" } }, + }, now).nextAction).toBe(action.nextAction); + }); + it("keeps cleanup quarantine visible without a coordinator or recovery row", () => { const stopped = run({ id: "stopped", status: "failed", errorCode: "native_session_cleanup_quarantined" }); expect(projectExecution(stopped, undefined, [], undefined, now)).toMatchObject({ diff --git a/server/src/services/execution-projection.ts b/server/src/services/execution-projection.ts index 1d857e393d9..b778c244f43 100644 --- a/server/src/services/execution-projection.ts +++ b/server/src/services/execution-projection.ts @@ -14,6 +14,7 @@ import type { ExecutionProjection } from "@paperclipai/shared"; import { EXECUTION_CONTROL_DEADLINE_MS } from "./execution-control-deadline.js"; import { executionFailureRetryCount } from "./execution-recovery-attempt.js"; import { retryIdempotentDatabaseOperation } from "../database-retry.js"; +import { workspaceRecoveryDisplay } from "./workspace-recovery-receipt.js"; const text = (v: unknown) => (typeof v === "string" ? v : null); const executionRunColumns = { id: heartbeatRuns.id, @@ -245,7 +246,8 @@ export function projectExecution( Object.assign(projection, { phase, label }); if (recoveryAction?.status === "resolved" && recoveryAction.evidence?.automaticRecovery) { projection.cause = recoveryAction.cause; - projection.nextAction = recoveryAction.nextAction; + projection.nextAction = workspaceRecoveryDisplay(recoveryAction.evidence, run.id)?.message + ?? recoveryAction.nextAction; const continuation = recoveryAction.evidence.explicitUserContinuation as { previousRunId?: unknown; runId?: unknown } | undefined; const explicitSuccessor = text(continuation?.runId); diff --git a/server/src/services/execution-recovery-resolution.ts b/server/src/services/execution-recovery-resolution.ts index e56141565b9..59b2b12cf5a 100644 --- a/server/src/services/execution-recovery-resolution.ts +++ b/server/src/services/execution-recovery-resolution.ts @@ -7,7 +7,7 @@ import { claimedAdapterType, conversationRecoveryActionPredicate, getConversatio import { persistActivity } from "./activity-log.js"; import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js"; import { logger } from "../middleware/logger.js"; -import { and, eq, inArray, isNull, not, or, sql } from "drizzle-orm"; +import { and, eq, exists, inArray, isNull, not, or, sql } from "drizzle-orm"; import { chatActions, environmentLeases, @@ -28,6 +28,7 @@ import { isSupersededConversationRun } from "./agent-conversations.js"; import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js"; import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; import { readSystemRecovery } from "./system-recovery-state.js"; +import { alias } from "drizzle-orm/pg-core"; /** An operator records observed outcomes; this is not permission to blindly retry. */ export async function validateExecutionReconciliation(input: { @@ -344,6 +345,24 @@ export async function deliverReconciledExecutions( } } +const supersedingWorkspaceHold = alias(issueRecoveryActions, "superseding_workspace_hold"); + +function supersedingNativeWorkspaceHold(db: Db, input: { + companyId: string | typeof heartbeatRuns.companyId; + issueId: string | typeof issueRecoveryActions.sourceIssueId; + runId: string | typeof heartbeatRuns.id; +}) { + return db.select({ id: supersedingWorkspaceHold.id }).from(supersedingWorkspaceHold).where(and( + eq(supersedingWorkspaceHold.companyId, input.companyId), + eq(supersedingWorkspaceHold.sourceIssueId, input.issueId), + eq(supersedingWorkspaceHold.status, "resolved"), + eq(supersedingWorkspaceHold.cause, "native_continuation_requires_reconciliation"), + sql`${supersedingWorkspaceHold.evidence}->>'runId' = ${input.runId}::text`, + sql`${supersedingWorkspaceHold.evidence}->'automaticRecovery'->>'replay' = 'blocked'`, + sql`${supersedingWorkspaceHold.evidence}->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'`, + )); +} + export function unresolvedExecutionDispositions(db: Db, limit: number) { return db .select({ action: issueRecoveryActions }) @@ -372,9 +391,14 @@ export function unresolvedExecutionDispositions(db: Db, limit: number) { sql`not (${issueRecoveryActions.evidence} ? 'systemRecovery')`, inArray(issueRecoveryActions.status, ["active", "escalated"]), eq(issueRecoveryActions.kind, "active_run_watchdog"), - inArray(issueRecoveryActions.cause, [ - ...EXECUTION_RECONCILIATION_CAUSES, - ]), + or(inArray(issueRecoveryActions.cause, [...EXECUTION_RECONCILIATION_CAUSES]), and( + eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.nativePhase, "terminal_failure"), + eq(nativeRunFinalizations.phase, "terminal_failure"), isNull(nativeRunFinalizations.nextAttemptAt), + sql`${issueRecoveryActions.wakePolicy}->>'kind' = 'resume_native_run'`, + sql`${issueRecoveryActions.wakePolicy}->>'runId' = ${heartbeatRuns.id}::text`, + exists(supersedingNativeWorkspaceHold(db, { companyId: heartbeatRuns.companyId, + issueId: issueRecoveryActions.sourceIssueId, runId: heartbeatRuns.id })), + )), inArray(heartbeatRuns.status, [ "failed", "timed_out", @@ -502,9 +526,6 @@ export async function settleUnrecoverableExecutions( !run || !action || action.evidence.runId !== runId || - !EXECUTION_RECONCILIATION_CAUSES.includes( - action.cause as (typeof EXECUTION_RECONCILIATION_CAUSES)[number], - ) || !["active", "escalated"].includes(action.status) || (run.nativeIssueId ?? run.contextSnapshot?.issueId) !== task.id || !["failed", "timed_out", "interrupted", "cancelled"].includes( @@ -512,6 +533,29 @@ export async function settleUnrecoverableExecutions( ) ) return; + if (!EXECUTION_RECONCILIATION_CAUSES.includes(action.cause as (typeof EXECUTION_RECONCILIATION_CAUSES)[number])) { + if (run.runtimeMode !== "native" || run.nativePhase !== "terminal_failure" + || coordinator?.phase !== "terminal_failure" || coordinator.leaseOwner || coordinator.resultId + || coordinator.nextAttemptAt || coordinator.failureDetail?.successorRunId + || action.wakePolicy?.kind !== "resume_native_run" || action.wakePolicy.runId !== run.id) return; + const [hold] = await supersedingNativeWorkspaceHold(tx as unknown as Db, { + companyId: run.companyId, issueId: task.id, runId: run.id, + }).limit(1).for("update"); + if (!hold) return; + await notifyExecutionRunWork(tx); + const note = "Exact session recovery stopped. The recorded workspace recovery holds this execution without replaying its actions."; + await tx.update(issueRecoveryActions).set({ status: "cancelled", outcome: "cancelled", + resolvedAt: now, updatedAt: now, wakePolicy: null, monitorPolicy: null, + nextAction: note, resolutionNote: note, + evidence: { ...action.evidence, supersedingRecoveryActionId: hold.id }, + }).where(eq(issueRecoveryActions.id, action.id)); + await persistActivity(tx as unknown as Db, { companyId: run.companyId, actorType: "system", + actorId: "execution-recovery", action: "issue.execution_recovery_settled", entityType: "issue", + entityId: task.id, runId: run.id, + details: { recoveryActionId: action.id, outcome: "cancelled", supersedingRecoveryActionId: hold.id }, + }); + return; + } // Adoption transfers maintenance to the retained-workspace owner. // This compatibility resolver must not close its hold or relabel it // as a user repair merely because old return-owner metadata is absent. diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index 2ac1a3f0960..7d337f06cfb 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -47,6 +47,7 @@ import { adapterExecutionControls } from "../adapter-execution-control.js"; import { executionFailureRetryCount } from "../execution-recovery-attempt.js"; import { slackMentionAllowsRunStart, + resumeDeferredChatWake, assertDurableChatWakeupReceipt, assertDurableChatWakeupRequest, withDurableChatAdmission, @@ -1857,6 +1858,13 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende )); if (!latest || !latest.finishedAt || !isHeartbeatRunTerminalStatus(latest.status)) continue; if (latest.runtimeMode !== "legacy" && latest.runtimeMode !== "native") continue; + if (wake.idempotencyKey?.startsWith("chat-inbound:")) { + await resumeDeferredChatWake(db, { companyId: wake.companyId, agentId: wake.agentId, + issueId: String(wake.payload?.issueId), requestId: wake.id }).catch(() => { + logger.warn({ requestId: wake.id }, "saved chat source could not resume after recovery"); + }); + continue; + } const releaseSource = `${latest.companyId}:${String(wake.payload?.issueId)}:${latest.id}:${latest.sourceVersion}`; if (latest.runtimeMode === "native" && releasedNativeSources.has(releaseSource)) continue; if (latest.runtimeMode === "legacy") { @@ -2373,10 +2381,13 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende receipt.payload?.museBindingGeneration !== museRequest.bindingGeneration || receipt.payload?.museRequestDigest !== museRequest.requestDigest)) { throw conflict("Request ID reused with changed input."); } - if (receipt && durableRequest?.failedRunRetry && opts.taskRetryRequestId && - receipt.status === "deferred_issue_execution" && !receipt.runId && - receipt.payload?.taskRetryRequestId === opts.taskRetryRequestId) { - // A bound ordinary Retry may reconsider its same deferred chat receipt. + if (receipt && durableRequest && receipt.status === "deferred_issue_execution" && !receipt.runId && + ((durableRequest.failedRunRetry && opts.taskRetryRequestId && + receipt.payload?.taskRetryRequestId === opts.taskRetryRequestId) || + (durableRequest.resumeDeferred && !durableRequest.failedRunRetry && + receipt.payload?.issueId === durableRequest.issueId && + queuedCommentIdsFromWakePayload(receipt.payload).includes(durableRequest.commentId)))) { + // The live inbound owner or a bound Retry may reconsider the same saved receipt. // The issue lock, live chat authority and execution holds below still // decide whether it can dispatch; this never creates another wake. executionWaitRequestId = receipt.id; @@ -2822,8 +2833,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende // system wake. Admission validates the saved user comment or board click. (opts.queuedCommentInterruptId ?? opts.queuedCommentRequestId) === executionWaitRequestId ? undefined : eq(agentWakeupRequests.requestedByActorType, - durableRequest?.failedRunRetry && opts.taskRetryRequestId - ? opts.requestedByActorType ?? "" : "user"), + durableRequest ? opts.requestedByActorType ?? "" : "user"), opts.queuedCommentInterruptId === executionWaitRequestId ? sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt'->>'actorId' = ${opts.requestedByActorId ?? ""}` : opts.queuedCommentRequestId === executionWaitRequestId ? undefined @@ -4187,7 +4197,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende }; // A deferred durable chat wake already owns this ID and source. // Promote that receipt in the same transaction as its successor. - const wakeupRequest = durableRequest?.failedRunRetry && executionWaitRequestId === durableRequest.id + const wakeupRequest = durableRequest && executionWaitRequestId === durableRequest.id ? await tx.update(agentWakeupRequests).set({ payload, status: "queued", error: null, finishedAt: null, updatedAt: new Date(), }).where(and(eq(agentWakeupRequests.id, durableRequest.id), diff --git a/server/src/services/heartbeat/run-control.test.ts b/server/src/services/heartbeat/run-control.test.ts index d0d08896538..ab0a9fbb5a8 100644 --- a/server/src/services/heartbeat/run-control.test.ts +++ b/server/src/services/heartbeat/run-control.test.ts @@ -429,6 +429,38 @@ describe.skipIf(!runControlPostgresSupport.supported)("heartbeat run-control dat .toMatchObject({ status: "done", phase: "done", completedAt: expect.any(Date) }); }); + it.each(["confirmed", "missing", "wrong_allocation", "unacknowledged", "still_running", "wrong_intent"] as const)("publishes native Stop acknowledgement only after exact remote containment: %s", async proof => { + const { company, agent } = await fixture(); + const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Save work after Stop", assigneeAgentId: agent.id }).returning(); + const runId = randomUUID(), leaseId = randomUUID(), providerLeaseId = randomUUID(); + const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId: company.id, agentId: agent.id, + nativeIssueId: issue.id, runtimeMode: "native", status: proof === "still_running" ? "running" : "cancelled", finishedAt: proof === "still_running" ? null : new Date(), + resultJson: { executionCancellation: { state: "requested", requestedAt: new Date().toISOString() }, + cancelledByActorType: "user", cancelledByUserId: "operator", nativeCancellation: { + schema: "paperclip.native-cancellation.v1", runId: proof === "wrong_intent" ? randomUUID() : runId, companyId: company.id, issueId: issue.id, + scope: "run", reasonCode: "cancellation_run_only", dispatchState: proof === "unacknowledged" ? "pending" : "acknowledged", dispatched: true, + intentAuditId: randomUUID(), acknowledgementAuditId: randomUUID(), + }, workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { + schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [leaseId], + } }, + }).returning(); + const receipt = { schema: "paperclip.remote-termination.v1", companyId: company.id, runId, leaseId, + provider: "daytona", providerLeaseId: proof === "wrong_allocation" ? randomUUID() : providerLeaseId, + state: "stopped", confirmedAt: new Date().toISOString() }; + await db.insert(environmentLeases).values({ id: leaseId, companyId: company.id, issueId: issue.id, heartbeatRunId: runId, + provider: "daytona", providerLeaseId, status: proof === "missing" ? "active" : "released", leasePolicy: "retain_on_failure", + releasedAt: proof === "missing" ? null : new Date(), cleanupStatus: proof === "missing" ? null : "success", + metadata: proof === "missing" ? {} : { remoteExecutionTermination: receipt } }); + const control = createHeartbeatRunControl(db, callbacks(db)); + await control.acknowledgeRemoteStop(run.id, company.id); + await control.acknowledgeRemoteStop(run.id, company.id); + const [stopped] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + expect(stopped.resultJson?.executionCancellation).toMatchObject({ state: proof === "confirmed" ? "acknowledged" : "requested" }); + if (proof === "confirmed") expect(stopped.resultJson?.executionCancellation).toMatchObject({ proof: "provider_termination_receipt" }); + expect(stopped.resultJson?.workspaceRestoreFailure).toBe("restore_failed"); + expect(stopped.resultJson?.workspaceRestoreRecovery).toMatchObject({ leaseIds: [leaseId] }); + }); + it("keeps pre-Stop input pending until a later user message authorizes one continuation", async () => { const { company, agent } = await fixture(); const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Retain my direction after Stop", diff --git a/server/src/services/heartbeat/run-control.ts b/server/src/services/heartbeat/run-control.ts index 8c87325ee4b..2ce3cf3c363 100644 --- a/server/src/services/heartbeat/run-control.ts +++ b/server/src/services/heartbeat/run-control.ts @@ -1,3 +1,4 @@ +import { resumeDeferredChatWake } from "../durable-chat-wakeup.js"; import { notifyEnvironmentCleanup } from "../environment-cleanup-notifications.js"; import { readNativeStopExecutionEvidence } from "./native-stop-evidence-read.js"; import { readTaskRetryIntent, reconcileTaskChatRetryReceipts } from "../task-retry.js"; @@ -493,21 +494,27 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont completeTerminatedRemoteNativeSessionCleanup({ companyId, runId, remoteCleanupScope }); } } - const stopped = state?.runtimeMode === "legacy" && state.status === "cancelled" ? await getRun(runId) : null; - if (stopped?.runtimeMode === "legacy" && stopped.status === "cancelled" && + const stopped = state?.status === "cancelled" ? await getRun(runId) : null; + // Native interrupt acknowledgement is not physical containment. Publish the + // shared Stop receipt only after every exact remote lease confirms teardown. + // Keep workspace preservation separate: a stopped provider can still owe files. + if (stopped?.status === "cancelled" && parseObject(stopped.resultJson?.executionCancellation).state === "requested" && - await runUsedConversationAdapter(db, stopped) && + (stopped.runtimeMode === "native" ? isAcknowledgedNativeStop(stopped) + : stopped.runtimeMode === "legacy" && await runUsedConversationAdapter(db, stopped)) && await remoteExecutionHasStopped(db, companyId, runId)) { await db.update(heartbeatRuns).set({ resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ executionCancellation: { ...parseObject(stopped.resultJson?.executionCancellation), state: "acknowledged", acknowledgedAt: new Date().toISOString(), proof: "provider_termination_receipt" }, - conversationContinuation: CONVERSATION_CONTINUATION_POLICY, + ...(stopped.runtimeMode === "legacy" ? { conversationContinuation: CONVERSATION_CONTINUATION_POLICY } : {}), })}::jsonb`, updatedAt: new Date(), }).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId), - eq(heartbeatRuns.status, "cancelled"))); + eq(heartbeatRuns.status, "cancelled"), + sql`${heartbeatRuns.resultJson}->'executionCancellation'->>'state' = 'requested'`, + stopped.runtimeMode === "native" ? sql`${heartbeatRuns.resultJson}->'nativeCancellation' = ${JSON.stringify(stopped.resultJson?.nativeCancellation)}::jsonb` : undefined)); } } @@ -544,12 +551,19 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont const pending = await db.select().from(agentWakeupRequests).where(and( eq(agentWakeupRequests.companyId, run.companyId), eq(agentWakeupRequests.agentId, run.agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"), - eq(agentWakeupRequests.requestedByActorType, "user"), + inArray(agentWakeupRequests.requestedByActorType, ["user", "system"]), requestId ? eq(agentWakeupRequests.id, requestId) : undefined, sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`, )).orderBy(asc(agentWakeupRequests.requestedAt)).limit(50); for (const wake of pending) { - if (wake.idempotencyKey?.startsWith("chat-inbound:")) continue; + if (wake.idempotencyKey?.startsWith("chat-inbound:")) { + await resumeDeferredChatWake(db, { companyId: run.companyId, agentId: run.agentId, + issueId, requestId: wake.id }).catch(() => { + logger.warn({ requestId: wake.id }, "saved chat source could not resume after cleanup"); + }); + continue; + } + if (wake.requestedByActorType !== "user") continue; if (readTaskRetryIntent(wake.payload)) { await resumeTaskRetry(wake.companyId, wake.id); continue; } let payload = parseObject(wake.payload); if (payload.queuedCommentInterrupt) { diff --git a/server/src/services/heartbeat/workspace-preparation.test.ts b/server/src/services/heartbeat/workspace-preparation.test.ts index ccbcd9066c5..eb0f0e2abc3 100644 --- a/server/src/services/heartbeat/workspace-preparation.test.ts +++ b/server/src/services/heartbeat/workspace-preparation.test.ts @@ -320,6 +320,69 @@ describe.skipIf(!support.supported)("heartbeat workspace preparation boundary", expect((await readIssue(issue.id)).executionWorkspaceId).toBe(prepared.persistedExecutionWorkspace!.id); }); + it("reuses a successful fresh task attempt for the next ordinary user turn", async () => { + const { input, issue } = await fixture(); + input.policy.useIsolatedTaskDirectory = true; + input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace"; + input.policy.requestedExecutionWorkspaceMode = "isolated_workspace"; + input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id, + sourceRunId: input.run.id, retainedSourceIds: [] }; + const original = await prepareHeartbeatWorkspace(db, input); + const workspace = original.persistedExecutionWorkspace!; + await writeFile(join(workspace.cwd!, "saved-review.txt"), "completed review history\n"); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }) + .where(eq(heartbeatRuns.id, input.run.id)); + await reuse(input, workspace); + input.task.freshWorkspace = null; + const next = await prepareHeartbeatWorkspace(db, input); + expect(next.persistedExecutionWorkspace!.id).toBe(workspace.id); + expect(next.executionWorkspace.cwd).toBe(workspace.cwd); + expect(await readFile(join(next.executionWorkspace.cwd, "saved-review.txt"), "utf8")) + .toBe("completed review history\n"); + expect((await readIssue(issue.id)).executionWorkspaceId).toBe(workspace.id); + await reuse(input, workspace); + input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id, + sourceRunId: input.run.id, retainedSourceIds: [] }; + input.reuse.existingExecutionWorkspace = null; + input.reuse.reusableExistingExecutionWorkspace = null; + input.reuse.requestedShouldReuseExisting = false; + input.reuse.workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({ + issueExecutionWorkspaceId: null, issueExecutionWorkspacePreference: "create_new", existingExecutionWorkspaceStatus: null, + }); + const replacement = await prepareHeartbeatWorkspace(db, input); + expect(replacement.executionWorkspace.cwd).not.toBe(workspace.cwd); + await expect(access(join(replacement.executionWorkspace.cwd, "saved-review.txt"))).rejects.toThrow(); + expect(await readFile(join(workspace.cwd!, "saved-review.txt"), "utf8")).toBe("completed review history\n"); + }); + + it.each(["task_binding", "workspace_row", "foreign_issue", "foreign_company", "closed_workspace"])( + "rejects a retained task attempt after its %s changes", async change => { + const { input, issue, source } = await fixture(); + input.policy.useIsolatedTaskDirectory = true; + input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace"; + input.policy.requestedExecutionWorkspaceMode = "isolated_workspace"; + input.task.freshWorkspace = { mode: "fresh_isolated_workspace", retryIntentId: input.run.id, + sourceRunId: input.run.id, retainedSourceIds: [] }; + const original = await prepareHeartbeatWorkspace(db, input); + const workspace = original.persistedExecutionWorkspace!; + await writeFile(join(workspace.cwd!, "saved-review.txt"), "keep saved work\n"); + await reuse(input, workspace); + input.task.freshWorkspace = null; + if (change === "task_binding") await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.id, issue.id)); + if (change === "workspace_row") await db.update(executionWorkspaces).set({ cwd: source }).where(eq(executionWorkspaces.id, workspace.id)); + if (change === "foreign_issue") await db.update(executionWorkspaces).set({ sourceIssueId: null }).where(eq(executionWorkspaces.id, workspace.id)); + if (change === "foreign_company") { + const [other] = await db.insert(companies).values({ name: "Other workspace company", issuePrefix: "OTH" }).returning(); + await db.update(executionWorkspaces).set({ companyId: other.id }).where(eq(executionWorkspaces.id, workspace.id)); + } + if (change === "closed_workspace") await db.update(executionWorkspaces).set({ status: "archived" }).where(eq(executionWorkspaces.id, workspace.id)); + await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({ + resultJson: { workspaceValidation: { reason: "isolated_task_directory_binding_mismatch", issueId: issue.id } }, + }); + expect(await readFile(join(workspace.cwd!, "saved-review.txt"), "utf8")).toBe("keep saved work\n"); + }, + ); + async function reuse(input: HeartbeatWorkspacePreparationInput, workspace: NonNullable>["persistedExecutionWorkspace"]>) { input.reuse.requestedShouldReuseExisting = true; input.reuse.existingExecutionWorkspace = workspace; diff --git a/server/src/services/heartbeat/workspace-preparation.ts b/server/src/services/heartbeat/workspace-preparation.ts index a23665a96b6..f56bde29125 100644 --- a/server/src/services/heartbeat/workspace-preparation.ts +++ b/server/src/services/heartbeat/workspace-preparation.ts @@ -277,18 +277,33 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace workspaceHints: [], warnings: [], baseCwdFallback: false, materializationFailures: [], additionalWorkspaces: [], referencedProjectFailures: [] }; } if (useIsolatedTaskDirectory && issueRef && !pluginTaskWorkspace) { - const cwd = await materializeIsolatedTaskDirectory({ - companyId: agent.companyId, - issueId: issueRef.id, - ...(input.task.freshWorkspace?.mode === "fresh_isolated_workspace" ? { attemptId: run.id } : {}), - }); - if (reusableExistingExecutionWorkspace && ( - reusableExistingExecutionWorkspace.companyId !== agent.companyId || - reusableExistingExecutionWorkspace.sourceIssueId !== issueRef.id || - reusableExistingExecutionWorkspace.mode !== "isolated_workspace" || - reusableExistingExecutionWorkspace.strategyType !== "project_primary" || - reusableExistingExecutionWorkspace.cwd !== cwd - )) { + // A completed fresh attempt becomes the current task directory. New + // input reuses that binding; only another fresh attempt changes roots. + let cwd: string; + try { + if (reusableExistingExecutionWorkspace) { + const [currentIssue] = await db.select({ workspaceId: issues.executionWorkspaceId }).from(issues) + .where(and(eq(issues.id, issueRef.id), eq(issues.companyId, agent.companyId))); + const stored = await executionWorkspacesSvc.getStoredById(agent.companyId, reusableExistingExecutionWorkspace.id); + if (fresh || currentIssue?.workspaceId !== reusableExistingExecutionWorkspace.id || + reusableExistingExecutionWorkspace.companyId !== agent.companyId || + reusableExistingExecutionWorkspace.sourceIssueId !== issueRef.id || + reusableExistingExecutionWorkspace.mode !== "isolated_workspace" || + reusableExistingExecutionWorkspace.strategyType !== "project_primary" || !stored || + stored.companyId !== agent.companyId || stored.sourceIssueId !== issueRef.id || + stored.mode !== "isolated_workspace" || stored.strategyType !== "project_primary" || + stored.status !== "active" || stored.cwd !== reusableExistingExecutionWorkspace.cwd || !stored.cwd) { + throw new Error("The saved task directory binding changed"); + } + } + cwd = await materializeIsolatedTaskDirectory({ + companyId: agent.companyId, + issueId: issueRef.id, + ...(fresh ? { attemptId: run.id } : {}), + ...(reusableExistingExecutionWorkspace ? { reuseCwd: reusableExistingExecutionWorkspace.cwd! } : {}), + }); + } catch (error) { + if (!reusableExistingExecutionWorkspace) throw error; throw new WorkspaceValidationFailure("The existing execution workspace is not this task's isolated directory.", { workspaceValidation: { reason: "isolated_task_directory_binding_mismatch", issueId: issueRef.id }, }); diff --git a/server/src/services/isolated-task-directory.test.ts b/server/src/services/isolated-task-directory.test.ts index bd3d350ff9f..b2d4125f87b 100644 --- a/server/src/services/isolated-task-directory.test.ts +++ b/server/src/services/isolated-task-directory.test.ts @@ -73,6 +73,37 @@ describe("isolated task directories", () => { .rejects.toThrow("not a private directory"); }); + it("reuses only the exact private task directory, including a prior fresh attempt", async () => { + const identity = await setup(); + for (const attemptId of [undefined, "previous-attempt"]) { + const cwd = await materializeIsolatedTaskDirectory({ ...identity, attemptId }); + await writeFile(path.join(cwd, "saved.txt"), "preserved"); + expect(await materializeIsolatedTaskDirectory({ ...identity, reuseCwd: cwd })).toBe(cwd); + expect(await readFile(path.join(cwd, "saved.txt"), "utf8")).toBe("preserved"); + } + }); + + it.each(["other-company", "other-issue", "outside", "traversal", "missing", "symlink"])( + "does not admit or recreate a reused task directory with %s binding", async change => { + const identity = await setup(); + const cwd = await materializeIsolatedTaskDirectory({ ...identity, attemptId: "prior-attempt" }); + await writeFile(path.join(cwd, "saved.txt"), "preserved"); + let reuseCwd = cwd; + if (change === "other-company") reuseCwd = cwd.replace(identity.companyId, "other-company"); + if (change === "other-issue") reuseCwd = cwd.replace(identity.issueId, "other-issue"); + if (change === "outside") reuseCwd = path.dirname(path.dirname(cwd)); + if (change === "traversal") reuseCwd = `${cwd}/../prior-attempt`; + if (change === "missing") reuseCwd = path.join(path.dirname(cwd), "missing-attempt"); + if (change === "symlink") { + reuseCwd = path.join(path.dirname(cwd), "linked-attempt"); + await symlink(cwd, reuseCwd); + } + await expect(materializeIsolatedTaskDirectory({ ...identity, reuseCwd })).rejects.toThrow(); + if (change === "missing") await expect(readFile(path.join(reuseCwd, "saved.txt"))).rejects.toThrow(); + expect(await readFile(path.join(cwd, "saved.txt"), "utf8")).toBe("preserved"); + }, + ); + it("round-trips sandbox output into the task directory and stages it on the next turn", async () => { const identity = await setup(); const cwd = await materializeIsolatedTaskDirectory(identity); diff --git a/server/src/services/isolated-task-directory.ts b/server/src/services/isolated-task-directory.ts index 9b0298cfe2b..26238976e08 100644 --- a/server/src/services/isolated-task-directory.ts +++ b/server/src/services/isolated-task-directory.ts @@ -26,20 +26,34 @@ export async function materializeIsolatedTaskDirectory(input: { companyId: string; issueId: string; attemptId?: string; + /** Validate retained files in place; never recreate a missing directory. */ + reuseCwd?: string; }): Promise { // Files belong to the task, so reassignment preserves the same workspace. - const identities = [input.companyId, input.issueId, ...(input.attemptId ? [input.attemptId] : [])]; + let identities = [input.companyId, input.issueId, ...(input.attemptId ? [input.attemptId] : [])]; if (identities.some((value) => !/^[a-zA-Z0-9_-]+$/.test(value))) { throw new Error("Invalid isolated task workspace identity"); } const root = resolvePaperclipInstanceRoot(); - await mkdir(root, { recursive: true }); + if (!input.reuseCwd) await mkdir(root, { recursive: true }); let cwd = await realpath(root); + let namespace = input.attemptId ? "isolated-workspace-attempts" : "isolated-workspaces"; + if (input.reuseCwd) { + const segments = path.relative(cwd, input.reuseCwd).split(path.sep); + if (input.attemptId || path.resolve(input.reuseCwd) !== input.reuseCwd || + segments[1] !== input.companyId || segments[2] !== input.issueId || + !((segments[0] === "isolated-workspaces" && segments.length === 3) || + (segments[0] === "isolated-workspace-attempts" && segments.length === 4 && /^[a-zA-Z0-9_-]+$/.test(segments[3])))) { + throw new Error("Isolated task workspace path is not this task's directory"); + } + namespace = segments[0]; + identities = segments.slice(1); + } // A retry is a sibling namespace, never a descendant of the retained root. // An old copyback operation must not be able to overwrite its replacement. - for (const segment of [input.attemptId ? "isolated-workspace-attempts" : "isolated-workspaces", ...identities]) { + for (const segment of [namespace, ...identities]) { cwd = path.join(cwd, segment); - await mkdir(cwd, { mode: 0o700 }).catch((error: NodeJS.ErrnoException) => { + if (!input.reuseCwd) await mkdir(cwd, { mode: 0o700 }).catch((error: NodeJS.ErrnoException) => { if (error.code !== "EEXIST") throw error; }); const stat = await lstat(cwd); diff --git a/server/src/services/native-runtime/native-failed-workspace-recovery.test.ts b/server/src/services/native-runtime/native-failed-workspace-recovery.test.ts index 22570efdb0f..3d5f155bd14 100644 --- a/server/src/services/native-runtime/native-failed-workspace-recovery.test.ts +++ b/server/src/services/native-runtime/native-failed-workspace-recovery.test.ts @@ -4,7 +4,7 @@ import { afterAll, beforeAll, describe, expect, it } from "vitest"; import { activityLog, agents, companies, createDb, environmentLeases, environments, externalAgentHolds, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { getExecutionBlocker } from "../execution-blocker.js"; -import { markExecutionReconciliation, validateExecutionReconciliation } from "../execution-recovery-resolution.js"; +import { markExecutionReconciliation, settleUnrecoverableExecutions, validateExecutionReconciliation } from "../execution-recovery-resolution.js"; import { prepareSandboxStopAndRetain, settleStopOnlyCleanup } from "../sandbox-stop-and-retain.js"; import { preserveCancelledNativeWorkspaces, preserveNativeFailedWorkspace } from "./native-failed-workspace-recovery.js"; import type { NativeRestartRecoveryClaim } from "./native-restart-recovery.js"; @@ -51,6 +51,44 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes const readLease = async (id: string) => (await db.select().from(environmentLeases).where(eq(environmentLeases.id, id)))[0]!; const actions = async (issueId: string) => db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)); + it.each(["settled", "historical", "live_owner", "retry", "nonterminal", "wrong_wake", "wrong_hold_run", "no_hold"])("retires only an exact superseded native resume obligation: %s", async scenario => { + const f = await seed(), actionId = randomUUID(); + await db.insert(issueRecoveryActions).values({ id: actionId, companyId: f.companyId, sourceIssueId: f.issueId, + kind: "active_run_watchdog", status: "active", ownerType: "agent", ownerAgentId: f.agentId, + returnOwnerAgentId: f.agentId, cause: "native_session_interrupted", fingerprint: randomUUID(), + evidence: { runId: f.runId }, nextAction: "Resume the exact persisted session.", + wakePolicy: { kind: "resume_native_run", runId: f.runId }, attemptCount: 1, maxAttempts: 3 }); + await preserveNativeFailedWorkspace(db, f.run); + const hold = (await actions(f.issueId)).find(action => action.id !== actionId)!; + expect(hold.evidence.automaticRecovery).toMatchObject({ replay: "blocked", actionOutcome: "unknown" }); + if (scenario === "live_owner") await db.update(nativeRunFinalizations).set({ leaseOwner: "live-controller" }).where(eq(nativeRunFinalizations.runId, f.runId)); + if (scenario === "retry") await db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(Date.now() + 60_000) }).where(eq(nativeRunFinalizations.runId, f.runId)); + if (scenario === "nonterminal") await db.update(nativeRunFinalizations).set({ phase: "observed" }).where(eq(nativeRunFinalizations.runId, f.runId)); + if (scenario === "wrong_wake") await db.update(issueRecoveryActions).set({ wakePolicy: { kind: "resume_native_run", runId: randomUUID() } }).where(eq(issueRecoveryActions.id, actionId)); + if (scenario === "wrong_hold_run") await db.update(issueRecoveryActions).set({ evidence: { ...hold.evidence, runId: randomUUID() } }).where(eq(issueRecoveryActions.id, hold.id)); + if (scenario === "no_hold") await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.id, hold.id)); + const beforeRun = await readRun(f.runId), beforeLease = await readLease(f.lease.id); + const beforeBlocker = await getExecutionBlocker(db, f.companyId, f.issueId); + const [beforeTask] = await db.select().from(issues).where(eq(issues.id, f.issueId)); + await settleUnrecoverableExecutions(db); + if (scenario === "historical") await settleUnrecoverableExecutions(db); + const retired = (await actions(f.issueId)).find(action => action.id === actionId)!; + const shouldRetire = scenario === "settled" || scenario === "historical"; + expect(retired.status).toBe(shouldRetire ? "cancelled" : "active"); + if (shouldRetire) { + expect(retired).toMatchObject({ outcome: "cancelled", wakePolicy: null, + evidence: { runId: f.runId, supersedingRecoveryActionId: hold.id } }); + expect((await actions(f.issueId)).find(action => action.id === hold.id)).toEqual(hold); + expect(await db.select().from(activityLog).where(and(eq(activityLog.entityId, f.issueId), + eq(activityLog.action, "issue.execution_recovery_settled")))).toHaveLength(1); + } + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toEqual(beforeBlocker); + expect(await readRun(f.runId)).toEqual(beforeRun); + expect(await readLease(f.lease.id)).toEqual(beforeLease); + expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toEqual(beforeTask); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1); + }); + it.each(["acknowledged", "warm", "pending", "foreign", "owned"])("pins historical cancelled work only after an exact Stop acknowledgement: %s", async kind => { const f = await seed(); if (kind === "warm") await db.update(environmentLeases).set({ leasePolicy: "reuse_by_environment" }).where(eq(environmentLeases.id, f.lease.id)); diff --git a/ui/src/components/AgentConfigForm.render.test.tsx b/ui/src/components/AgentConfigForm.render.test.tsx index 97465cbb26b..7b8037b275b 100644 --- a/ui/src/components/AgentConfigForm.render.test.tsx +++ b/ui/src/components/AgentConfigForm.render.test.tsx @@ -1640,6 +1640,86 @@ describe("AgentConfigForm environment selector", () => { expect(result.container.textContent).not.toContain("Cannot read properties of undefined"); }); + it("qualifies an existing native runner's saved computer before its normal Test", async () => { + const computer = makeEnvironment({ id: "saved-computer", driver: "sandbox", status: "active", + config: { provider: "daytona", snapshot: "managed-snapshot" } }); + const order: string[] = []; + mockAgentsApi.qualifyRunner.mockImplementation(async () => { + order.push("qualification"); + return { ok: true, driver: "sandbox", details: { platformQualificationRecorded: true } }; + }); + mockAgentsApi.testEnvironment.mockImplementation(async () => { + order.push("runtime"); + return { adapterType: "paperclip_runner", status: "pass", checks: [], testedAt: new Date(0).toISOString() }; + }); + const result = await renderForm([computer], { adapterType: "paperclip_runner", defaultEnvironmentId: computer.id, + adapterConfig: { provider: "opencode", model: "example/model" } }, { showAdapterTestEnvironmentButton: true }); + roots.push(result.root); + await runTest(result.container); + expect(mockAgentsApi.qualifyRunner).toHaveBeenCalledWith("agent-1", computer.id, "company-1"); + expect(order).toEqual(["qualification", "runtime"]); + expect(result.onSave).not.toHaveBeenCalled(); + }); + + it.each(["failed", "missing-proof"])("does not run the native runtime test after %s saved-computer qualification", async state => { + const computer = makeEnvironment({ id: "saved-computer", driver: "sandbox", status: "active", + config: { provider: "daytona", snapshot: "managed-snapshot" } }); + mockAgentsApi.qualifyRunner.mockResolvedValue({ ok: state !== "failed", driver: "sandbox", summary: "Computer test failed.", + details: { platformQualificationRecorded: false } }); + const result = await renderForm([computer], { adapterType: "paperclip_runner", defaultEnvironmentId: computer.id, + adapterConfig: { provider: "opencode" } }, { showAdapterTestEnvironmentButton: true }); + roots.push(result.root); + await runTest(result.container); + expect(mockAgentsApi.qualifyRunner).toHaveBeenCalledTimes(1); + expect(mockAgentsApi.testEnvironment).not.toHaveBeenCalled(); + expect(result.container.textContent).toContain("Computer test failed."); + }); + + it.each(["openai_dot", "muse", "pool"])("does not turn the %s readiness preview into a computer qualification", async provider => { + const computer = makeEnvironment({ id: "saved-computer", driver: "sandbox", status: "active", + config: { provider: "daytona", snapshot: "managed-snapshot" } }); + mockAgentsApi.qualifyRunner.mockResolvedValue({ ok: true, driver: "sandbox", details: { platformQualificationRecorded: true } }); + const result = await renderForm([computer], { adapterType: "paperclip_runner", defaultEnvironmentId: computer.id, + adapterConfig: { provider: provider === "pool" ? "opencode" : provider }, + ...(provider === "pool" ? { runtimeConfig: { aiConnection: { mode: "router", connectionId: "a8d3a9a3-9866-49d8-93d6-c497ae090d05" } } } : {}) }, + { showAdapterTestEnvironmentButton: true, compactTestFeedback: true }); + roots.push(result.root); + await clickByText(result.container, provider === "pool" ? "Run test" : "Check setup"); + expect(mockAgentsApi.testEnvironment).toHaveBeenCalledTimes(1); + expect(mockAgentsApi.qualifyRunner).not.toHaveBeenCalled(); + }); + + it("does not run a native test after settings change during computer qualification", async () => { + const computer = makeEnvironment({ id: "saved-computer", driver: "sandbox", status: "active", + config: { provider: "daytona", snapshot: "managed-snapshot" } }); + let finish: (() => void) | undefined; + mockAgentsApi.qualifyRunner.mockImplementation(() => new Promise(resolve => { + finish = () => resolve({ ok: true, driver: "sandbox", details: { platformQualificationRecorded: true } }); + })); + const result = await renderForm([computer], { adapterType: "paperclip_runner", defaultEnvironmentId: computer.id, + adapterConfig: { provider: "opencode" } }, { showAdapterTestEnvironmentButton: true }); + roots.push(result.root); + await runTest(result.container); + expect(mockAgentsApi.qualifyRunner).toHaveBeenCalledTimes(1); + await act(async () => setInputValue(result.container.querySelector('[placeholder="Agent name"]')!, "Changed while testing")); + await act(async () => finish!()); + await flushReact(); + expect(mockAgentsApi.testEnvironment).not.toHaveBeenCalled(); + expect(result.container.textContent).toContain("Agent settings changed during the test."); + }); + + it("does not qualify a draft computer through the native runtime Test", async () => { + const computers = ["saved-computer", "draft-computer"].map(id => makeEnvironment({ id, driver: "sandbox", status: "active", + config: { provider: "daytona", snapshot: "managed-snapshot" } })); + const result = await renderForm(computers, { adapterType: "paperclip_runner", defaultEnvironmentId: computers[0]!.id, + adapterConfig: { provider: "opencode" } }, { showAdapterTestEnvironmentButton: true }); + roots.push(result.root); + await selectEnvironment(result.container, computers[1]!.id); + await runTest(result.container); + expect(mockAgentsApi.qualifyRunner).not.toHaveBeenCalled(); + expect(result.onSave).not.toHaveBeenCalled(); + }); + it("omits undefined adapter config entries when testing a create form with the default model", async () => { const result = await renderCreateForm([ makeEnvironment({ id: "local-1", name: "Local", driver: "local" }), diff --git a/ui/src/components/AgentConfigForm.tsx b/ui/src/components/AgentConfigForm.tsx index 55d1ff4aa94..1d7357b6f5d 100644 --- a/ui/src/components/AgentConfigForm.tsx +++ b/ui/src/components/AgentConfigForm.tsx @@ -912,11 +912,11 @@ export function AgentConfigForm(props: AgentConfigFormProps) { && Boolean(runnerAvailability && !runnerAvailability.disabled && !runnerAvailability.overriddenBuiltin) && Boolean(nativeRunnerAvailability && !nativeRunnerAvailability.disabled && !nativeRunnerAvailability.overriddenBuiltin); const upgradeWarnings = harnessType === "claude_local" ? normalizeClaudeRunnerOptions(effectiveConfig).warnings : []; - const upgradeTargetRef = useRef(""); - const upgradeTarget = JSON.stringify([!isCreate ? props.agent.id : null, selectedCompanyId, adapterType, + const runnerSettingsTargetRef = useRef(""); + const runnerSettingsTarget = JSON.stringify([!isCreate ? props.agent.id : null, selectedCompanyId, adapterType, rawCurrentDefaultEnvironmentId, upgradeEnvironmentId, upgradeEnvironment?.config, effectiveConfig, overlay]); - upgradeTargetRef.current = upgradeTarget; - useEffect(() => () => { upgradeTargetRef.current = ""; }, []); + runnerSettingsTargetRef.current = runnerSettingsTarget; + useEffect(() => () => { runnerSettingsTargetRef.current = ""; }, []); const showRunnerUpgrade = !isCreate && ["codex_local", "claude_local"].includes(props.agent.adapterType) && adapterType === props.agent.adapterType @@ -931,7 +931,7 @@ export function AgentConfigForm(props: AgentConfigFormProps) { const nextOverlay = flushedEnv ? { ...overlay, adapterConfig: { ...overlay.adapterConfig, env: flushedEnv } } : overlay; - const pendingTarget = upgradeTargetRef.current; + const pendingTarget = runnerSettingsTargetRef.current; if (runnerAvailability?.defaultRunner !== "paperclip") { if (!canQualifyRunnerUpgrade || !upgradeEnvironmentId || !selectedCompanyId) { throw new Error("Could not determine which environment to verify. Retry the upgrade."); @@ -940,7 +940,7 @@ export function AgentConfigForm(props: AgentConfigFormProps) { adapterType: harnessType, readAdapters: () => adaptersApi.list({ companyId: selectedCompanyId, environmentId: upgradeEnvironmentId }), probeEnvironment: () => agentsApi.qualifyRunner(agentId, upgradeEnvironmentId, selectedCompanyId), - isCurrent: () => upgradeTargetRef.current === pendingTarget, + isCurrent: () => runnerSettingsTargetRef.current === pendingTarget, }); if (!qualified) throw new Error("Agent settings changed during verification. Retry the upgrade with the current settings."); queryClient.setQueryData(queryKeys.adapters.availability(selectedCompanyId, rawCurrentDefaultEnvironmentId || null), qualified.adapters); @@ -1208,6 +1208,17 @@ export function AgentConfigForm(props: AgentConfigFormProps) { const aiConnection = isCreate ? undefined : aiRuntimeConnectionBindingSchema.safeParse( (overlay.runtime.runtimeConfig as Record | undefined)?.aiConnection ?? props.agent.runtimeConfig.aiConnection, ).data; + if (!isCreate && props.agent.adapterType === "paperclip_runner" && adapterType === "paperclip_runner" + && !isExternalRunner && aiConnection?.mode !== "router" && !isDirty && environmentId && canQualifyRunnerEnvironment(environmentList.find(environment => environment.id === environmentId))) { + const target = runnerSettingsTargetRef.current; + // Qualify only the saved computer. The strict agent-scoped endpoint + // accepts no draft provider config, credentials or replacement target. + const qualification = await agentsApi.qualifyRunner(props.agent.id, environmentId, selectedCompanyId); + if (runnerSettingsTargetRef.current !== target) throw new Error("Agent settings changed during the test. Run test again."); + if (qualification.ok !== true || qualification.details?.platformQualificationRecorded !== true) { + throw new Error(qualification.summary || "Could not verify the saved computer. Run test again."); + } + } if (props.compactTestFeedback) { const providerAdapter = adapterType === "paperclip_runner" ? adapterConfig.provider === "codex" ? "codex_local" diff --git a/ui/src/components/IssueRunLedger.test.tsx b/ui/src/components/IssueRunLedger.test.tsx index 117f27ff3b1..f94900ce410 100644 --- a/ui/src/components/IssueRunLedger.test.tsx +++ b/ui/src/components/IssueRunLedger.test.tsx @@ -420,6 +420,23 @@ describe("IssueRunLedger", () => { expect(container.textContent).toContain("all 2 terminal (1 done, 1 cancelled)"); }); + it("shows the current execution instruction and clears stale instructions after a recorded successor", () => { + const execution = { + phase: "recovery_needed" as const, label: "Stopped", cause: "native_continuation_requires_reconciliation", + lastConfirmedActivityAt: null, retryAt: null, attempt: 1, maxAttempts: 3, + recoveryOwner: null, nextAction: "Saved work recovered. Send a message to continue.", + permittedActions: ["inspect_run"] as const, predecessorRunId: null, successorRunId: null, + }; + const stopped = createRun({ status: "failed", nextAction: "Repair the retained sandbox.", + execution: { ...execution, permittedActions: ["inspect_run"] } }); + renderLedger({ runs: [stopped] }); + expect(container.textContent).toContain(execution.nextAction); + expect(container.textContent).not.toContain("Repair the retained sandbox."); + renderLedger({ runs: [{ ...stopped, execution: { ...stopped.execution!, phase: "completed", + label: "Continued in another run", nextAction: null, successorRunId: "successor" } }] }); + expect(container.textContent).not.toContain("Next action:"); + }); + it("uses wrapping-friendly markup for long next action text", () => { renderLedger({ runs: [ diff --git a/ui/src/components/IssueRunLedger.tsx b/ui/src/components/IssueRunLedger.tsx index 6cd7cdee99e..1bee4217d05 100644 --- a/ui/src/components/IssueRunLedger.tsx +++ b/ui/src/components/IssueRunLedger.tsx @@ -842,6 +842,7 @@ export function IssueRunLedgerContent({ ); } const run = item.run; + const nextAction = run.execution ? run.execution.nextAction : run.nextAction; const liveness = livenessCopyForRun(run); const stopReason = stopReasonLabel(run); const duration = formatDuration(run.startedAt, run.finishedAt); @@ -1001,13 +1002,13 @@ export function IssueRunLedgerContent({ /> ) : null} - {run.nextAction ? ( + {nextAction ? (
Next action:{" "} - {run.nextAction} + {nextAction}
) : null} diff --git a/ui/storybook/stories/agents/codex-runner/upgrade.stories.tsx b/ui/storybook/stories/agents/codex-runner/upgrade.stories.tsx index 8f9b103a3f9..d1fe14c18a7 100644 --- a/ui/storybook/stories/agents/codex-runner/upgrade.stories.tsx +++ b/ui/storybook/stories/agents/codex-runner/upgrade.stories.tsx @@ -16,17 +16,21 @@ type Scenario = { upgraded?: boolean; running?: boolean; delay?: number; + nativeComputerTest?: boolean; }; -function installUpgradeFixture({ harness = "codex_local", failOnce = false, incompatible = false, unavailable = false, upgraded = false, running = false, delay = 0 }: Scenario = {}) { +function installUpgradeFixture({ harness = "codex_local", failOnce = false, incompatible = false, unavailable = false, upgraded = false, running = false, delay = 0, nativeComputerTest = false }: Scenario = {}) { resetOnboardingFixtureState(); setOnboardingFixtureState({ environments: "local", authSignal: "present", localLoginStatus: "ready" }); const profile = harness === "claude_local" ? { provider: "acpx", acpxAgent: "claude" } : { provider: "codex" }; + const savedComputer = { id: "environment-native-recovery", name: "Paperclip Computer", driver: "sandbox", status: "active", + config: { provider: "daytona", snapshot: "managed-snapshot" }, metadata: { managedByPaperclip: true } }; + let qualified = false; let agent = { ...storybookAgents[0]!, urlKey: "migration-agent", name: harness === "claude_local" ? "ClaudeWriter" : "CodexCoder", - status: running ? "running" : "idle", lifecycleState: "ready", defaultEnvironmentId: null, - adapterType: upgraded ? "paperclip_runner" : harness, + status: running ? "running" : "idle", lifecycleState: "ready", defaultEnvironmentId: nativeComputerTest ? savedComputer.id : null, + adapterType: upgraded || nativeComputerTest ? "paperclip_runner" : harness, adapterConfig: { model: harness === "claude_local" ? "claude-sonnet-4-6" : "gpt-5.4", modelReasoningEffort: "medium", timeoutSec: 180, - ...(upgraded ? profile : {}), ...(incompatible ? { chrome: true } : {}), env: { MIGRATION_MARKER: { type: "plain", value: "keep-this-setting" } } }, + ...(nativeComputerTest ? { provider: "opencode" } : upgraded ? profile : {}), ...(incompatible ? { chrome: true } : {}), env: { MIGRATION_MARKER: { type: "plain", value: "keep-this-setting" } } }, runtimeConfig: { heartbeat: { enabled: false, wakeOnDemand: true } }, chainOfCommand: [], access: { canAssignTasks: true, taskAssignSource: "explicit_grant", membership: null, grants: [] }, }; @@ -34,6 +38,16 @@ function installUpgradeFixture({ harness = "codex_local", failOnce = false, inco const fixtureFetch: typeof fetch = async (input, init) => { const url = new URL(typeof input === "string" ? input : input instanceof URL ? input.href : input.url, window.location.origin); const method = init?.method ?? "GET"; + if (nativeComputerTest && url.pathname === `/api/companies/${agent.companyId}/environments`) return Response.json([savedComputer]); + if (nativeComputerTest && url.pathname === `/api/agents/${agent.id}/qualify-runner`) { + if (JSON.parse(String(init?.body ?? "{}")).expectedEnvironmentId !== savedComputer.id) return Response.json({ error: "Computer changed." }, { status: 409 }); + qualified = true; + return Response.json({ ok: true, driver: "sandbox", details: { platformQualificationRecorded: true } }); + } + if (nativeComputerTest && url.pathname.endsWith("/adapters/paperclip_runner/test-environment")) { + return Response.json({ adapterType: "paperclip_runner", status: qualified ? "pass" : "fail", testedAt: new Date().toISOString(), + checks: [{ code: "native_runtime_test", level: qualified ? "info" : "error", message: qualified ? "Native runtime ready." : "Saved computer qualification is stale." }] }); + } if (url.pathname === "/api/adapters") { const response = await previous(input, init); return Response.json((await response.json()).map((item: { type: string }) => ["codex_local", "claude_local"].includes(item.type) @@ -140,3 +154,18 @@ export const UnavailableEnvironment: Story = { parameters: { runnerUpgrade: { un export const AgentRunning: Story = { parameters: { runnerUpgrade: { running: true } } }; export const Mobile: Story = { globals: { viewport: { value: "mobile1", isRotated: false } } }; export const Light: Story = { globals: { theme: "light" } }; + +export const NativeComputerRecovery: Story = { + name: "Native saved computer Test", + parameters: { runnerUpgrade: { nativeComputerTest: true }, paperclip: { + notes: "Production native agent page. The normal Run test action qualifies its saved computer before runtime testing. Both responses are simulated; real provider qualification is separate.", + } }, + play: async ({ canvasElement }) => { + const page = within(canvasElement); + await userEvent.click(await page.findByRole("button", { name: /^Run test$/ })); + await expect(await page.findByRole("heading", { name: "Connection successful" })).toBeVisible(); + await userEvent.click(page.getByText("Test details", { exact: true })); + await expect(await page.findByText("Native runtime ready.")).toBeVisible(); + await expect(page.queryByRole("button", { name: "Upgrade runner" })).not.toBeInTheDocument(); + }, +}; From 208a093d19f9f5c9a4a4cf6f6843c47a44d15ec7 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 26 Oct 2026 22:22:47 -0500 Subject: [PATCH 041/133] fix(chat): preserve resumed review waits and retry newer saved input (#17725) https://entire.com/gh/paperclipai/-/trails/36/changes/paperclip/36 ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - GitHub review agents publish findings and wait for an authorized reply or head change. > - Review tools could retain the requester after resume while completion lost that same wait proof. > - A yielded report could then queue its own continuation or repeat a missing-wait error. > - Ordinary Retry could also select an older failed turn while a newer saved request had no admission. > - This change shares the existing authority proofs and canonical admission owners. > - Current access, publication, execution and budget gates remain required. ## Linked Issues or Issue Description Refs #17318. Builds on the canonical saved-input retry owner in #17733. Related #16550 covers deferred legacy comments after ownership waits. The separate recovery repair in #17724 owns the cleanup-to-saved-input handoff; this PR does not add a second admission owner. A resumed review could publish an assessment, then turn its `response_wake` into an immediate `issue_status_changed` wake. A provider that omitted its continuation could instead repeat a completion error. A newer linked-human message saved during a budget hold could also remain unstarted because Retry selected the older failed source. ## What Changed - Resolve passive review waits through the accepted requester ledger used by GitHub tools. Require the latest reviewed head, exact company, task, agent, repository and processed publication under current access. - Fill omitted/null yielded continuation from that server proof. Permit an inherited normal tree resume only through its accepted parent identity. Permit an ordinary follow-up only after this run has a processed useful reply linked to its exact inbound message. Reject acknowledgements, partial assessments, pending or uncertain writes as that reply proof. - Recheck the same run/review/head/source proof during feedback, finalization and final status commit. Preserve explicit malformed-input rejection and Done policy. Add no monitor permission or publication grant. - Keep heartbeat and publication projections narrow. Preserve both published Copilot v62 identities and generate source-bound v63 from the pinned merged base. No v63 live qualification is claimed. - Certify the structured isolated-directory binding failure only at its pre-allocation boundary and only through canonical task Retry. Other workspace validation reasons and unknown effects remain denied. - Route Retry to eligible newer saved input through the existing canonical owner. Preserve the old result and skipped receipt. Require current principal, source, App, runtime fence, budget, native ownership and exact cleanup proof. - Recover absent historical App metadata only through an earlier trusted processed response receipt and an immutable non-null endpoint claim. Explicit malformed or conflicting metadata stays denied. - Revalidate execution through the exact claimed wake, coalesced task Retry and resolved user-continuation receipt after admission. A second normal Retry can recover the exact never-started authorization failure with unchanged saved input. Require empty execution evidence, exact diagnostics and no allocated or active executor. - Explain when a recorded result prevents exact replay. Ordinary board Retry directs a new task message to continue from saved work. Inherited requester identity alone never grants replay. - Keep accepted board-message preparation failures with the canonical task Retry owner. Certify the exact unchanged comment, original user wake, accepted same-user identity and absence of provider execution. Pin that source, recheck it at admission, and bind identity initialization to the coalesced intent and exact claimed successor receipt. Uncertifiable sources retain an actionable fresh-message refusal. - Return safe named authority categories for board Retry refusals. Update execution documentation and capability maps. Add no schema or polling loop. ## Verification - A fresh external review request exposed a separate isolated-directory preparation failure before allocation. Its first normal Retry saved an intent but admitted no successor. This follow-up certifies only that exact structured preparation boundary through the existing canonical Retry owner, with trusted failed settlement and no executor, lease, result or output. A real HTTP/queue regression reproduces the missing successor before the repair. Twenty-three current-source safety cases and the final 98-case Retry/HTTP group pass, including a fresh click after a denied intent, source revocation, accepted results and the older direct API remaining denied. The warning case uses actual configuration preparation and production event append; unknown warning shape or provider provenance stays denied. Full workspace typecheck, build, maps and diff checks pass. Renewed hosted normal Retry now succeeds on the final combined source: it preserves the exact saved request and linked requester, starts one native successor in a fresh attempt workspace, publishes the current-head 2/5 assessment and commits In review with passive response wake. - A later hosted normal Retry published a low-score assessment but lost its accepted source identity because the previous attempt failed before identity initialization. It repeated continuation errors, then submitted Done. This is failed acceptance, not a successful qualification. The initializer now retains the exact source only after execution-time failed-chat Retry authorization; generic Retry and Done policy are unchanged. A real queue/initializer regression fails with a null source on the previous code and passes with the newer saved source. Two real database completion cases verify that source yields the authoritative wait and commits In review. Renewed hosted normal Retry now retains the accepted source and exposes the authoritative current-review wait. It completed after two semantic feedback corrections (wrong contract revision, then wrong criterion IDs); the correct contract was present in emitted input. The third finish was accepted and committed In review, with no missing-wait loop or Done escalation. - Repeated board-message preparation failures now walk their canonical intent, failed dispatch and accepted parent identity, rechecking each contained predecessor. A production queue regression fails the second ordinary HTTP Retry before the repair and passes afterward. Changed source, contradictory execution evidence and cyclic receipts remain denied; traversal stops after twenty links. - New inherited omission cases failed before the repair. The final full PostgreSQL 18 wait/status suite passes **252 tests** under Node 24.22. It covers omitted/null input, malformed-first recovery, same-head resume, exact-source replies, changed heads, revoked access, pending/uncertain/mixed publications, final commit revalidation, no duplicate review and no self-wake. - **15 existing publication tests passed before the pinned merge** on a separate fresh database. They retain default partial-delivery and failure-fallback behavior. Context reads remain below the unchanged 16 KiB heartbeat and publication response limits with large saved output present. - Saved-input tests cover a real signed PR webhook, production accounting-budget refusal, normal policy recovery and canonical Retry. Historical producer-shaped tests prove the missing-App case red, then check trusted prior receipts and malformed, late, revoked or mismatched proof. A broader order-coupled test was reproduced on the prior source; its global fault injector now targets its own authorization query. The 43-case sequence passes without changing production behavior or its safety assertions. - New signed-ingress and real heartbeat execution tests reproduced the admitted-successor failure, then the next normal Retry failure. The final pinned merged source passes 249 full GitHub review/publication and Retry cases on a fresh database, including all 49 saved-input cases. A separate 74-case Retry/HTTP run also passes. Exact dispatch negatives cover source, company, agent, task, successor, receipt, pause, assignment, accepted success, controller and cleanup proof. Bootstrap contradictions remain denied. The canonical Retry suite passes 16 cases through its supported external PostgreSQL test connection. - The normal board HTTP route reproduced the unusable inherited-result refusal. It now returns actionable guidance without starting a successor. The actual UI Retry hook and banner tests pass 12 cases. - A production-shaped accepted board comment followed by a pre-provider qualification failure reproduced HTTP 409 before the repair. The final real HTTP/queue test certifies one canonical successor and preserves the accepted identity. It rejects changed source, actor, task, agent, result, controller, process, profile, allocation and receipt evidence. A post-acceptance source change keeps the same intent pending. Execution revalidation rejects removed or mismatched proof. The probe stops deliberately after production identity initialization; it does not claim provider execution. The final source passes the 75-case Retry group, the focused two-case board/inherited-policy group and 19 canonical Retry cases. Full workspace typecheck, build and map checks pass. - Full workspace `pnpm -r typecheck` and `pnpm build` pass. Map, module-boundary and diff checks pass. Full local `pnpm test:run` remains unverified. The 256-case explicit-native suite cannot initialize its packaged PostgreSQL because libzstd is absent; an existing command-scoped fallback did not resolve it. Current-head remote full-suite checks are required. Merged-source full GitHub+Retry (249), separate Retry/HTTP (74), full wait/status (252), profile/history and completion (75), and visible Retry hook/banner (12) checks pass. The fresh owned PostgreSQL 18 databases applied all 377 migrations. - Actual Rust Runner, OpenCode 1.18.34 and exact DeepSeek used unchanged production-generated synthetic contexts. Resume finished with an explicit server wait. The author-reply trial omitted continuation twice: its first report failed the existing empty-criteria check; the corrected second report still omitted continuation and received the exact server wait. It completed once with no missing-wait loop or Done escalation. Controller acknowledgments were synthetic; these probes do not establish hosted publication or board state. - Hosted combined testing proved automatic admission of the exact saved author reply, a useful GitHub answer and committed passive wait. It required two malformed-continuation rejections before a nullish fallback succeeded. A later normal Pause/Resume completed in one finish with explicit response_wake and no new review/comment. The admitted normal Retry then exposed the execution guard gap repaired here. Those remain checkpoint results. On the final combined source, renewed normal Retry completed actual native provider work and GitHub publication, committed the passive wait, released ownership and cleaned up its environment. A subsequent ordinary unmentioned author reply woke once, made no GitHub tool calls or new assessment, and completed its first finish with an explicit response wake. Both exact leases expired with successful cleanup and the queue was empty. However, final destination inspection found a separate automatic fallback overwrote the existing summary with a generic no-final-response message. External summary preservation therefore remains failed acceptance; the separate publication repair in #17726 owns that producer. Full combined publication preservation has not passed. These hosted runs include the pinned-base native additions; they do not qualify Copilot v63. - Current-head automated review is 5/5 with zero unresolved threads. All current-head CI gates passed (54 successful checks/contexts, four skipped, none pending). The earlier login-cleanup idle-work assertion also reproduced locally while its controlled baseline passed; its cause remains unknown, but the exact assertion passed in the fresh current-head CI shard. No timeout, assertion or production workaround was added. This PR is ready for review within its source, authority, native finalization, state, wake and cleanup scope. The independent automatic fallback publication regression is tracked by #17726; no complete combined workflow pass is claimed. ## Risks - The latest saved review defines the durable head boundary. Ingress must record a new head before admission. A reply receipt proves delivery only for its exact saved source; a shared GitHub comment ID alone cannot authorize this wait. - Historical App recovery cannot infer identity without direct proof or an earlier valid server response receipt. It never replays that response. - This does not infer review-score policy from instructions or override Done. No deployment, merge or repair of historical tasks is claimed by the PR. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, repository inspection, code editing and test execution. The context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused database and native suites; full local suite limitation described above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (verified on immutable latest head) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (verified on the latest head) - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/connections/GITHUB.md | 30 + doc/execution-semantics.md | 46 +- doc/task-recovery.md | 6 + maps/agent-access-map/chat-channels.md | 22 + maps/agent-access-map/recovery.md | 28 +- maps/feature-map/chat-channels.md | 22 + maps/feature-map/recovery.md | 18 + packages/paperclip-runner/acpx-profiles.json | 4 +- .../src/generated_acpx_profiles.rs | 2 +- .../scripts/copilot-profile-identity.mjs | 4 +- .../src/contracts/completion-result.ts | 4 +- .../drivers/acpx/copilot-installation.test.ts | 23 +- .../src/drivers/acpx/generated-profiles.ts | 4 +- .../src/drivers/acpx/profile-compatibility.ts | 4 +- .../drivers/completion-continuation.test.ts | 7 + .../src/drivers/completion-continuation.ts | 4 +- .../opencode/opencode-server-driver.test.ts | 8 +- ...ile-v62-candidate-historical-identity.json | 85 ++ .../copilot-profile-v63-identity.json | 85 ++ .../chat-channels.integration.test.ts | 948 +++++++++++++++++- .../chat-channels/failed-run-retry.ts | 187 +++- .../chat-channels/github/review-wait.ts | 72 +- .../github/task-retry-authorization.ts | 88 +- .../wakeup-authorization-error.test.ts | 16 + .../inbound/wakeup-authorization-error.ts | 19 + .../inbound/wakeup-authorization.ts | 33 +- .../chat-channels/task-retry-authority.ts | 57 +- .../chat-channels/task-retry-input.ts | 23 +- server/src/services/chat-run-publications.ts | 32 +- server/src/services/durable-chat-wakeup.ts | 13 + server/src/services/heartbeat.ts | 2 +- server/src/services/heartbeat/queue.ts | 10 +- .../native-runtime/chat-attachment-reuse.ts | 85 +- .../external-chat-wait.integration.test.ts | 495 ++++++++- .../native-runtime/external-review-wait.ts | 12 +- .../native-completion-feedback.ts | 8 +- .../native-runtime/native-run-finalizer.ts | 17 +- .../paperclip-runner-tool-authority.ts | 2 + .../status-decision-committer.ts | 31 +- server/src/services/run-identity.ts | 36 +- server/src/services/task-retry.ts | 135 ++- ui/src/hooks/useTaskRetry.test.tsx | 20 +- 42 files changed, 2594 insertions(+), 153 deletions(-) create mode 100644 packages/paperclip-runner/test/fixtures/copilot-profile-v62-candidate-historical-identity.json create mode 100644 packages/paperclip-runner/test/fixtures/copilot-profile-v63-identity.json create mode 100644 server/src/services/chat-channels/inbound/wakeup-authorization-error.test.ts create mode 100644 server/src/services/chat-channels/inbound/wakeup-authorization-error.ts diff --git a/doc/connections/GITHUB.md b/doc/connections/GITHUB.md index 90fa895341b..0f93999ab86 100644 --- a/doc/connections/GITHUB.md +++ b/doc/connections/GITHUB.md @@ -329,6 +329,36 @@ another recipient is named. The routing rule does not change automatic issue or event policies. Archived bots and identities from other companies or providers are excluded from the routing lookup. +## Retrying a saved request + +A normal task Retry can continue a newer saved GitHub message after an earlier +attempt failed. Execution revalidates the exact admitted successor, its current +linked principal and repository access, and the durable continuation receipt; +it does not repeat the admission that already consumed the recovery hold. +A failed native predecessor still requires its committed failed result and +exact executor termination proof. Successful results and uncertain delivery do +not grant a replay. A certified failed-chat Retry retains the exact accepted +message identity when its predecessor stopped before identity initialization, +so a published review can expose its authorized passive response wait. +If an inherited task-resume attempt already recorded a result, ordinary Retry +explains that exact replay is unavailable and directs the user to send a new +message to continue from saved work. That new turn still passes normal current +requester, access and executor-containment checks. + +If that successor failed at Retry authorization before runtime or provider +startup, the next normal Retry follows its canonical source receipt. The exact +system diagnostic, settlement events, empty execution evidence and absence of +allocated resources establish that narrow recovery path. A generic setup +failure or a claimed `providerWorkStarted: false` does not establish it. + +The canonical task Retry also recognizes an isolated-directory binding failure +at the workspace preparation boundary before allocation. Its structured reason +and task binding, trusted failed settlement, dispatch-only profile and absence +of execution resources must all agree. Other workspace validation failures do +not inherit this permission. Current request authorization still runs at both +admission and execution; a failed unadmitted Retry intent does not replace the +saved request or prevent a later deliberate Retry. + ## Run projection The resolved token is leased at run start as an audited class-3 secret and is diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 8269a7f4589..17411564865 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -521,13 +521,25 @@ of the invalid call. Unauthorized tools still receive a generic rejection, and corrected calls retain the same permission and approval checks. A native provider may omit `continuation` (or send `null`) when it reports -`yielded` after saving a monitor or requesting repository preparation. The controller reads current authorized task +`yielded` after saving a monitor, requesting repository preparation, or publishing +an authorized current-head GitHub review. The controller reads current authorized task context and fills the canonical continuation only if that same task, run, assignee, and work mode remain eligible. An eligible monitor takes precedence. A pending repository requested by that task in its current workspace offers `same_agent`; the existing status arbiter and wake queue own the next admission. Prepared repositories and requests belonging to another task cannot offer it. -An explicit `done` result does not acquire a continuation. Completion admission +A completed GitHub review with its processed publication may offer `response_wake` +for that exact head and accepted requester under current conversation and +connection authority. A normal tree resume may inherit the published review +through its accepted parent identity. An ordinary GitHub follow-up may inherit it +only after this run has a processed useful reply bound to its exact current +inbound comment; acknowledgements, pending or ambiguous writes and partial +assessment receipts do not suffice. Completion admission, finalization and +status commit recheck the server-issued run/review/head binding; a pending +publication, newer unreviewed head or revoked access cannot fall through to an +immediate continuation. An explicit malformed continuation remains invalid. +An explicit `done` result does not acquire a continuation or change its completion +policy; review scores are not interpreted as task-completion instructions. Completion admission checks that persisted monitor again. This default does not create or renew a schedule. A consumed, cleared, exhausted, or ineligible monitor cannot authorize a passive wait. An explicit `response_wake` still identifies a real response @@ -757,7 +769,7 @@ Explicit retry admission checks the local parent adapter controller and verified A new user message can continue a terminal native run whose process fields were cleared before local stop receipts existed. Admission must verify the exact run, runner, workspace, and provider session in the retained suspended state, with no active provider turn, pending tool call, or undelivered output. Missing or mismatched state keeps the hold. A later recorded process launch also keeps the hold until its stop is verified. Normal assignment, decision, controller, environment cleanup, and active-run gates still apply. The message starts one fresh conversation turn; it does not replay the failed run, reset its recovery budget, or certify unknown action outcomes. -The task thread exposes the guarded Retry action for failed or timed-out conversation runs and native preparation cancelled before provider startup when the never-started proof is verified. The server projects this eligibility on the recovery notice and rechecks it on Retry. Cleanup quarantine retains its inspection path, and non-conversation reconciliation gates remain enforced. Pending decisions, active execution, pause, budget, dependency, and ownership gates remain in force. A refused Retry reports its reason inline and remains available for another attempt. +The task thread exposes the guarded Retry action for failed or timed-out conversation runs and native preparation cancelled before provider startup when the never-started proof is verified. The server projects this eligibility on the recovery notice and rechecks it on Retry. Cleanup quarantine retains its inspection path, and non-conversation reconciliation gates remain enforced. Pending decisions, active execution, pause, budget, dependency, and ownership gates remain in force. A refused Retry reports its reason inline and remains available for another attempt. When a newer linked-human GitHub message was saved but its admission was skipped by a budget or agent-availability gate, ordinary task Retry selects that unstarted message before considering the older failed run. The existing chat task-Retry owner snapshots the complete pending batch, including eligible skipped input, into a new admission generation bound to the canonical processed deliveries, comments, principal, session generation, admitted instruction snapshots, and task Retry intent. It retains the original skipped receipt and failed result unchanged. A terminal failed publication of that old result also remains history for a distinct selected request; pending or uncertain publication still holds work, and the old output is never replayed. Board Retry refusals expose only stable allowlisted authority categories, never source payloads, identifiers, or credential fingerprints. Admission rechecks source integrity and current access, then uses the normal new-user-turn continuation proof: a committed failed result can remain history only after controller, successor, process, workspace, and exact remote-termination guards pass. Unknown ownership stays deferred; Retry cannot certify it. The existing chat retry queue owns delivery and bounded infrastructure retries; a committed receipt settles a lost acknowledgment without another execution. An accepted saved-input receipt discharges only its matching unadmitted-input gate, allowing later genuine conversation or tree-resume wakes without replaying the skipped request. A continuation cancelled by the queued-run gate while waiting for review did not start a provider. Its matching dispatch-gate receipt and unclaimed execution fields keep this deliberate wait out of unknown-action reconciliation. Pending review and decision gates still apply. If an older server already created a recovery hold for this exact state, Retry or a new user message can request one fresh turn after the server verifies there is no provider, coordinator, or unfinished cleanup. Saved user input is reconsidered through the same admission path; the cancelled turn is not replayed. @@ -838,6 +850,7 @@ A healthy `in_review` issue has at least one valid action path: - a human owner via `assigneeUserId` - an active run or queued wake that is expected to process the review state - an active one-shot monitor for an external service or async review loop that the assignee owns +- an authorized GitHub conversation waiting for its next provider event after a published review; a tree resume or same-agent continuation retains this path only through its accepted requester lineage and the latest saved review publication - an open explicit recovery action for an ambiguous review handoff Agent-assigned `in_review` with no typed participant is only healthy when one of the other paths exists. Assignment to the same agent that produced the handoff is not, by itself, a review path. @@ -1674,6 +1687,23 @@ Delivery still requires the provider's verified stop receipt. Periodic queue retries do not gain extra cleanup attempts, and the queue displays the server's waiting reason while cleanup remains unresolved. +A connected task can also fail while preparing an accepted board-authored +message, before the provider starts. Its Retry belongs to the canonical task +intent rather than the external inbound-message ledger. The server certifies +its unchanged, undeleted comment, original user wake and accepted identity in +the same company, task, agent and user scope. A settled bootstrap diagnostic +must agree with the absence of a native result, coordinator, environment lease, +controller, process or provider output. A previous session display hint does +not establish that the failed attempt started a provider. + +That intent pins the saved comment and accepted identity. Queue admission +rechecks the source; dispatch rechecks the coalesced intent and its exact claimed +successor receipt before inheriting the accepted identity. Current permissions, +assignment, pauses, budgets and containment remain normal queue requirements. +This path neither fabricates external inbound provenance nor replays a recorded +terminal result. If this source cannot be certified, Retry directs the user to +send a new task message to continue from saved work. + The legacy task recovery notice shows “Automatic recovery of this task stopped.” in a bordered container with Retry for a failed or timed-out run. A failed Retry shows its error in the same container. New user messages and saved undelivered @@ -1943,6 +1973,16 @@ re-read dependency readiness, since new dependencies need not change the displayed task status. Old blocked rows without a receipt remain held; no historical status backfill is performed. +For legacy GitHub input without a saved App ID or its own response receipt, +Task Retry may prove the endpoint's earlier non-null App claim from a processed +response ledger created before that input. The proof stays within the same +company, endpoint, conversation and agent, with a processed prior delivery and +its inbound source link. Supported setup and reconnect writers preserve an +existing non-null App claim. This historical identity proof grants no publication +replay or access: the saved input and current principal, repository, conversation, +runtime fence and execution containment still pass normal admission. Explicit +conflicting App bindings remain denials. + ### Queued input after a native Stop A run-only Stop ends the current response. It does not discard queued user diff --git a/doc/task-recovery.md b/doc/task-recovery.md index 714d921a97b..cede3a25dc2 100644 --- a/doc/task-recovery.md +++ b/doc/task-recovery.md @@ -28,6 +28,12 @@ comment receipt, then checks current sender, repository, conversation and task access. The retry records the current runtime generation and credential-reference fingerprint. Another reconnect invalidates that admission. Historical deliveries, stale webhook callbacks and the exact-run retry contract retain their fences. +Legacy input that predates saved App IDs can also use an earlier processed +response ledger from the same company, endpoint, conversation and agent. Its +server-owned App binding must predate that input and retain a processed delivery +and inbound source link. The endpoint's non-null App claim cannot be replaced; +this proves historical identity without replaying the earlier response. Missing +proof or an explicit conflicting App binding still rejects Retry. Task Retry keeps the complete saved-input batch, including batches larger than the exact-run endpoint's limit. Source rows, receipts and invalidations are read diff --git a/maps/agent-access-map/chat-channels.md b/maps/agent-access-map/chat-channels.md index d14f5c4237a..aaa0b34679b 100644 --- a/maps/agent-access-map/chat-channels.md +++ b/maps/agent-access-map/chat-channels.md @@ -283,6 +283,20 @@ real human dependencies retain the ordinary attention-request path. A timed wait requires a confirmed `set_task_monitor` schedule. Database coverage lives in `external-chat-wait.integration.test.ts`; this is not live model evidence. +Inherited GitHub review waits (2026-10-26): an explicit native `response_wake` +after tree resume or same-agent continuation may retain the provider-event path +through accepted requester identity lineage and the latest processed review +publication for that task/conversation. This does not grant monitor management +or publication authority. Current task/agent/company binding, repository, +principal and endpoint checks remain effective. PostgreSQL regressions cover +quiet repeated finalization, historical publication reuse, newer unreviewed +heads and revoked access; live GitHub delivery remains separate acceptance. +Omitted/null yielded reports may use the current run's server-projected published +review wait. The same proof is rechecked at feedback, finalization and commit; +older/different-run review, pending publication and revoked connection cannot +supply it. This grants no new publication/monitor authority and leaves Done policy +unchanged; explicit malformed input is not coerced. + ## `outbound-email` Capability ID: `chat-channels/outbound-email`. @@ -300,3 +314,11 @@ Evidence: `source-only` — reviewed at `9ef434fe79c57d976b8b03b14c5242db270b832 Verification: `not-run` — Use the actual permitted writer and run to create one child/publication, confirm recipient delivery and an exact-message reply, and repeat idempotent requests without duplicate sends. Assert run/owner/mode/budget/company/credential/attachment denials and separate read access. Inspect the provider before resolving an unknown write outcome; a 202 or queued publication alone is not proof of delivery. Gaps: Real AgentMail transport, recipient delivery, and each native tool binding still need live proof. Controlled provider receipts and inbound-reply recipes do not establish a newly initiated external email journey. + +Ordinary task Retry can select newer linked-human GitHub input whose original +admission was skipped by a budget or agent-availability gate. The chat owner +revalidates its exact processed delivery, company/task/agent/principal, unchanged +comment, session generation, and admitted instruction snapshot. A separate retry +receipt preserves the original skipped receipt and older failed native result. +Current access and normal native stop/cleanup guards still govern execution; +board wake metadata alone cannot manufacture chat authority or replay old work. diff --git a/maps/agent-access-map/recovery.md b/maps/agent-access-map/recovery.md index c9df2c9b13e..481c17abf8e 100644 --- a/maps/agent-access-map/recovery.md +++ b/maps/agent-access-map/recovery.md @@ -17,8 +17,16 @@ Ordinary board Retry can reauthorize saved GitHub input for the same App after reconnect. It pins current runtime authority and preserves sender, repository, conversation and task checks. Pending input takes precedence over the older failed request and transfers once to the retry owner. This grants no new agent -Retry API or stale webhook authority. Local PostgreSQL and HTTP-route regressions +Retry API or stale webhook authority. Legacy input can use a server-owned, +processed earlier response ledger to prove the same endpoint's App claim before +that input. Company, conversation, agent and prior delivery/source linkage remain +required. Explicit conflicting App IDs cannot use that fallback. Local PostgreSQL and HTTP-route regressions cover these transitions; hosted execution and delivery remain separate acceptance. +Dispatch validates the exact claimed wake and resolved user-continuation receipt +for the current successor. A later board Retry can follow that successor only +when exact durable diagnostics and absent execution/resources prove it never +started. This preserves the original sender/source and grants no agent Retry +permission or accepted-result replay. Source revision: `3bbc2104b65caf9f7763938a01260a5673e12f7a`. @@ -248,6 +256,24 @@ replace the owner's generation or revision. Resolved files-only recovery retains its no-replay restriction. The local PostgreSQL regression checks the producer, worker, and physical source receipt; it is not deployed provider acceptance. +Connected board-message preparation failures use the canonical board Retry +owner only after the exact unchanged comment, original user wake, accepted +same-user identity and pre-provider settlement are certified. Admission and +execution recheck the pinned source and claimed successor receipts. Repeated +pre-provider failures retain the source only through each canonical intent, +failed dispatch and accepted parent identity; cycles and chains beyond twenty +links fail closed. This grants +no external inbound authority or permission to replay a terminal result; an +uncertifiable source directs the user to send a new task message. The real +HTTP/queue regression proves this boundary locally, not hosted provider recovery. + +For external chat, the same canonical Retry owner may certify the structured +isolated-directory binding failure that occurs before environment allocation. +It requires exact task/source authority and trusted settlement with no process, +lease, native result or unknown execution evidence. Other validation reasons and +the older direct chat-retry entry point remain denied. Local HTTP/dispatch proof +does not establish hosted provider completion. + An authorized deliberate Retry also registers missing historical native preservation from an exact prepared descriptor and an unowned terminal finalization. It adopts only the existing bounded maintenance capability. A live diff --git a/maps/feature-map/chat-channels.md b/maps/feature-map/chat-channels.md index 6c98a4e5376..a5a490d0a33 100644 --- a/maps/feature-map/chat-channels.md +++ b/maps/feature-map/chat-channels.md @@ -72,6 +72,17 @@ database suite checks these states and revoked endpoints using stored provider receipts; live model adherence remains a separate check. It also checks saved monitors, disabled repositories, multiple reviewed heads, read-only finish validation, withdrawal rollback, and separate approval gates. +Repeat the review wait after a tree resume and a same-agent continuation with +no new provider message. Accepted requester lineage and the latest saved review +publication keep the task in review without a self-wake or duplicate publication. +A newer unreviewed head, changed requester/repository, or revoked access cannot +inherit the prior head's passive wait. Omitted/null yielded continuation is filled +from authorized server task context after a current-head review, an accepted tree +resume inheriting it, or a same-head follow-up with this run's processed useful +reply bound to its exact current source. Pending or ambiguous replies and +acknowledgements do not authorize this fallback. A malformed first report can recover by omitting the continuation; +explicit malformed input remains rejected. Revocation between context read, +finish and commit preserves the task without self-waking. Done policy is unchanged. Manual: For each supported provider being changed, send a unique message, verify the intended agent/company receives it, and inspect the actual external reply in the original thread. Repeat with an unlinked/unauthorized sender, duplicate inbound delivery, and one supported attachment. Record provider-specific results separately. @@ -217,3 +228,14 @@ Sources: Source: [TaskChatBubble.tsx](../../ui/src/components/task-chat/TaskChat ### Documentation scope: Chat and email channel setup Shared recipes name provider checks; each provider transport/thread/identity/file-size/type combination needs separate live evidence and deeper instructions. + +Saved-input Retry regression: fail a disposable GitHub turn, change the +configured instruction skill, and send a new authorized request while an agent +budget gate blocks admission. Clear that gate and use the task thread's ordinary +**Retry** control. Verify the new saved message and its admitted skill snapshot +reach one run, rather than replaying the previous failed request. Preserve both +original receipts. With a committed failed native result, continuation requires +verified controller and environment cleanup; an unknown executor remains held. +After successful admission, reload and verify that the prior skipped receipt does +not strand a later real provider reply or tree resume. Provider fixtures support +this behavior; live fixed-source runtime and destination proof remain separate. diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index fcf47da1e27..50699dee967 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -82,6 +82,15 @@ stopped-run recovery checks. ### `operator-retry` +The [connected-task regression](../../server/src/__tests__/chat-channels.integration.test.ts) +uses HTTP Retry after an accepted board message fails before provider startup. +It checks the canonical intent, same-user identity and actual queue dispatch, +then repeats Retry after another contained preparation failure with the same +source and a new certified successor; +edited source, contradictory execution evidence and mismatched receipts refuse +or wait. This local boundary probe deliberately stops before a provider call, +so it does not establish hosted recovery of the task. + Use the task's Retry action. It can create an isolated attempt after old execution is contained even when old-file preservation remains unresolved. Approval, credentials, budget and deliberate pause use their existing specific controls. @@ -103,6 +112,15 @@ there must be one successor and no duplicate publication. A second reconnect after admission, changed App, revoked sender or changed repository must prevent dispatch. An unadmitted rejection keeps the follow-up available for another deliberate Retry. Historical deliveries must remain unchanged. +Repeat with legacy input that has no saved App ID and was refused by a budget +before acknowledgement. An earlier processed response from the same endpoint, +conversation and agent can prove its App claim. Missing, late, unlinked or +conflicting proof must reject Retry. Verify actual execution separately. +At dispatch, confirm the admitted successor retains the same source and resolved +continuation receipt. If it fails at Retry authorization before provider startup, +use the next ordinary Retry against that newest failed run. It must retain the +saved follow-up and create one successor; output, an allocated environment or an +active controller must prevent treating that failure as never started. ### `post-stop-message` diff --git a/packages/paperclip-runner/acpx-profiles.json b/packages/paperclip-runner/acpx-profiles.json index 90da6bee2c4..fbe3f2753ad 100644 --- a/packages/paperclip-runner/acpx-profiles.json +++ b/packages/paperclip-runner/acpx-profiles.json @@ -32,12 +32,12 @@ "requiresProviderPolicy": true }, "copilot": { - "agentProfileVersion": 62, + "agentProfileVersion": 63, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:aeeb80bc186569c78911aeaa445c1fdf5e81420cfeadb563d5620497c58dd0f4", + "commandDigest": "sha256:1412842a36a7211d84b6b98a3d1f69e3388bc4814e5825da890a774ae0e3460e", "requiresProviderPolicy": true }, "claude": { diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs index 587a7bd5752..ed82e55fea9 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs @@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option { agent_runtime_package: None, agent_runtime_version: None, command_digest: - "sha256:aeeb80bc186569c78911aeaa445c1fdf5e81420cfeadb563d5620497c58dd0f4", + "sha256:1412842a36a7211d84b6b98a3d1f69e3388bc4814e5825da890a774ae0e3460e", requires_provider_policy: true, }, "claude" => AcpxReleaseProfile { diff --git a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs index bc0b7f399bd..770540a02c5 100644 --- a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs +++ b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs @@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url"; import { resolve, dirname } from "node:path"; const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); -const fixture = resolve(root, "test/fixtures/copilot-profile-v62-identity.json"); +const fixture = resolve(root, "test/fixtures/copilot-profile-v63-identity.json"); const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8")); -const declaration = { ...historical.declaration, agentProfileVersion: 62 }; +const declaration = { ...historical.declaration, agentProfileVersion: 63 }; delete declaration.permissionIdentitySourceSha256; const sources = { displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts", diff --git a/packages/paperclip-runner/src/contracts/completion-result.ts b/packages/paperclip-runner/src/contracts/completion-result.ts index 47862d1fdcb..8542380bbb6 100644 --- a/packages/paperclip-runner/src/contracts/completion-result.ts +++ b/packages/paperclip-runner/src/contracts/completion-result.ts @@ -3,7 +3,7 @@ export const PRP_COMPLETION_TOOL_NAME = "paperclip_finish" as const; export const PRP_BLOCK_TOOL_NAME = "paperclip_block" as const; /** Native Runner guidance; legacy adapters use their own skill/API completion paths. */ export const PRP_COMPLETION_TOOL_DESCRIPTION = - "Report completed work (done), work requiring review (needs_review), or an explicit wait (yielded with response_wake for a response, monitor after set_task_monitor confirms a schedule on this task, or omit continuation to use the saved repository-preparation continuation). Use the current completion contract and supporting evidence. If rejected, correct the report and retry. After acceptance, read the returned outcome; do not claim completion while gated. Explain any required approval with its supplied link and action, then write the final response and end the turn without further tool calls."; + "Report completed work (done), work requiring review (needs_review), or an explicit wait (yielded with response_wake for a response, monitor after set_task_monitor confirms a schedule on this task, or omit continuation to use this run's authorized published-review wait or saved repository-preparation continuation). Use the current completion contract and supporting evidence. If rejected, correct the report and retry. After acceptance, read the returned outcome; do not claim completion while gated. Explain any required approval with its supplied link and action, then write the final response and end the turn without further tool calls."; export const PRP_BLOCK_TOOL_DESCRIPTION = "Report work that cannot continue because of a concrete blocker. Routine repository preparation is not a blocker: use paperclip_finish with yielded and omit continuation to resume through the saved workspace continuation. Identify the blocker, its owner, and the action needed to unblock it; use the current completion contract and supporting evidence. If rejected, correct the report and retry. After acceptance, read the returned outcome, explain the blocker and any required action in the final response, and end the turn without further tool calls."; export const PRP_SEMANTIC_TOOL_NAMES = [ @@ -372,7 +372,7 @@ export const PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA = { ...waitContinuationSchema, type: ["object", "null"], description: - "For yielded, omit or use null to reuse this task's eligible saved monitor or repository-preparation continuation; the controller reads current task state. Otherwise provide a real response_wake or monitor with summary and idempotencyKey. Use null or omit for done, completed, or needs_review.", + "For yielded, omit or use null to reuse this run's authorized published-review wait, eligible saved monitor, or repository-preparation continuation; the controller reads current task state. Otherwise provide a real response_wake or monitor with summary and idempotencyKey. Use null or omit for done, completed, or needs_review.", }, }, // Keep the provider-facing root a concrete object. Codex code-mode renders a diff --git a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts index 2a96fdc9e51..2aa9af9391b 100644 --- a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts @@ -87,9 +87,28 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi describe("Copilot build-owned installation", () => { it("admits the current profile with environment launch preparation", () => { - expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(62); + expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(63); expect(QUALIFIED_ACPX_PROFILES.copilot.qualificationStatus).toBeUndefined(); - expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:aeeb80bc186569c78911aeaa445c1fdf5e81420cfeadb563d5620497c58dd0f4"); + expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:1412842a36a7211d84b6b98a3d1f69e3388bc4814e5825da890a774ae0e3460e"); + }); + + it("preserves both published v62 identities while binding their merged source in v63", async () => { + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v63-identity.json", import.meta.url), "utf8")); + for (const [name, checksum, changed] of [ + ["copilot-profile-v62-identity.json", "faa4d1b35267b54e134a8e0c84586fe1196d8fad5525cb53aa195aae53ca6e83", ["agentProfileVersion", "profileCompatibilitySourceSha256", "semanticCompletionContractSourceSha256", "semanticContinuationSourceSha256"]], + ["copilot-profile-v62-candidate-historical-identity.json", "da3f44a3cfda31cbff558d618e1e9cb1cfb80379b613ff1266e8b866ea666df8", ["agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256"]], + ] as const) { + const bytes = readFileSync(new URL(`../../../test/fixtures/${name}`, import.meta.url)); + expect(createHash("sha256").update(bytes).digest("hex")).toBe(checksum); + const prior = JSON.parse(bytes.toString("utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual(changed); + expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort()); + const sorted = Object.fromEntries(Object.entries(prior.declaration).sort(([a], [b]) => a.localeCompare(b))); + expect(`sha256:${createHash("sha256").update(JSON.stringify(sorted)).digest("hex")}`).toBe(prior.commandDigest); + expect(isSupportedAcpxProfileVersion("copilot", 62)).toBe(true); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: 62, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned"); + } }); it("preserves v61 while binding uncertain process checks and controller retirement in v62", async () => { diff --git a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts index 5aaf18f97c3..4cf0892c5c5 100644 --- a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts +++ b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts @@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = { "protocolVersion": 1, "acpxVersion": "0.13.1", "agent": "copilot", - "agentProfileVersion": 62, + "agentProfileVersion": 63, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:aeeb80bc186569c78911aeaa445c1fdf5e81420cfeadb563d5620497c58dd0f4", + "commandDigest": "sha256:1412842a36a7211d84b6b98a3d1f69e3388bc4814e5825da890a774ae0e3460e", "permissionPolicy": "interactive" }, "claude": { diff --git a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts index e8091c079e2..8f5fd0a22d9 100644 --- a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts +++ b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts @@ -2,13 +2,13 @@ import { QUALIFIED_ACPX_PROFILE_DATA } from "./generated-profiles.js"; /** Decodable historical profile revisions, not permission to launch them. * Exact current profile/package/digest admission is checked separately. */ -type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61; +type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62; type CurrentAcpxProfileVersion = typeof QUALIFIED_ACPX_PROFILE_DATA[keyof typeof QUALIFIED_ACPX_PROFILE_DATA]["agentProfileVersion"]; export type AcpxProfileVersion = HistoricalAcpxProfileVersion | CurrentAcpxProfileVersion; const historicalVersions: Readonly> = { pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5], - grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61], + grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62], cursor: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], }; diff --git a/packages/paperclip-runner/src/drivers/completion-continuation.test.ts b/packages/paperclip-runner/src/drivers/completion-continuation.test.ts index 62051110a60..5a9b60c2118 100644 --- a/packages/paperclip-runner/src/drivers/completion-continuation.test.ts +++ b/packages/paperclip-runner/src/drivers/completion-continuation.test.ts @@ -25,6 +25,13 @@ describe("controller-owned completion continuation", () => { .toEqual({ ...yielded, continuation: preparation }); }); + it.each([undefined, null])("fills a server-owned published response wait (%s)", async absent => { + const responseWait = { kind: "response_wake", summary: "Wait for the author.", idempotencyKey: "published-review:run:receipt" }; + expect(await resolveCompletionContinuation({ ...yielded, continuation: absent }, "run", async () => ({ + ...context, completionContinuation: responseWait, + }))).toEqual({ ...yielded, continuation: responseWait }); + }); + it("rechecks a sidecar-resolved workspace continuation against current server authority", async () => { const preparation = { kind: "same_agent", summary: "Continue with the prepared repository.", idempotencyKey: "repository:task:receipt:run" }; const report = { ...yielded, continuation: preparation }; diff --git a/packages/paperclip-runner/src/drivers/completion-continuation.ts b/packages/paperclip-runner/src/drivers/completion-continuation.ts index 05fd4ad8b75..2f817bb4e28 100644 --- a/packages/paperclip-runner/src/drivers/completion-continuation.ts +++ b/packages/paperclip-runner/src/drivers/completion-continuation.ts @@ -15,10 +15,10 @@ export async function resolveCompletionContinuation( const task = object(context.activeTask); const continuation = object(context.completionContinuation); if (object(context.run).id !== runId || task.executionRunId !== runId || - !["monitor", "same_agent"].includes(String(continuation.kind)) || + !["monitor", "same_agent", "response_wake"].includes(String(continuation.kind)) || typeof continuation.summary !== "string" || !continuation.summary.trim() || typeof continuation.idempotencyKey !== "string" || !continuation.idempotencyKey.trim()) { - throw new Error("Cannot yield without a continuation: no eligible saved monitor or workspace continuation exists on this task. Save a monitor for a deferred check, provide an explicit response_wake for a real response wait, or continue the unfinished work. Do not repeat completed actions."); + throw new Error("Cannot yield without a continuation: no eligible saved monitor or workspace continuation or authorized response wait exists on this task. Save a monitor for a deferred check, provide an explicit response_wake for a real response wait, or continue the unfinished work. Do not repeat completed actions."); } if (report.continuation != null && (continuation.kind !== "same_agent" || supplied.summary !== continuation.summary || supplied.idempotencyKey !== continuation.idempotencyKey)) { diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts index a44136bd364..c8e2ed1c292 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts @@ -2184,12 +2184,13 @@ describe("OpenCodeServerDriver", () => { await session.close({ reason: "test" }); }); - it.each([true, false])("resolves an omitted continuation through the real OpenCode MCP boundary only with a saved monitor: %s", async eligible => { + it.each(["monitor", "response_wake"] as const)("resolves an omitted %s through the real OpenCode MCP boundary only from current server state", async kind => { + for (const eligible of [true, false]) { await chmod(fixture, 0o755); const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-monitor-")); const workspace = await mkdtemp(join(tmpdir(), "paperclip-opencode-monitor-workspace-")); roots.push(root, workspace); - const continuation = { kind: "monitor", summary: "Wait for the saved task monitor.", idempotencyKey: "task-monitor:saved" }; + const continuation = { kind, summary: "Wait for the saved task path.", idempotencyKey: "task-wait:saved" }; const reads: string[] = []; const accepted: unknown[] = []; const driver = new OpenCodeServerDriver({ @@ -2216,7 +2217,8 @@ describe("OpenCodeServerDriver", () => { expect(events.filter(event => event.eventType === "run.result.proposed")).toEqual([]); expect((await session.snapshot()).semanticResult).toBeNull(); } - } finally { await session.close({ reason: "monitor default test complete" }); } + } finally { await session.close({ reason: "saved wait default test complete" }); } + } }); it("keeps assigned discovery operation IDs at the authenticated MCP and controller boundary", async () => { diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v62-candidate-historical-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v62-candidate-historical-identity.json new file mode 100644 index 00000000000..7266c5e13ca --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v62-candidate-historical-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:1251b53cb4716273c74e5839eb112ef8ca639e0967eb90a9f1a5f5ddafdb642c", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 62, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629", + "semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603", + "semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a", + "providerLaunchSourceSha256": "fa8db1c42c378898287c141d8cea9069166981e369c9b9c51422cb6177de5a2c", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "e6c98128d98a4177c42d68a04c1f749554fd5ec844da1bbbdb3f682670189445", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918", + "profileCompatibilitySourceSha256": "be60656e66d7f61a290f8d1c9a3c9ec511fb13b6541825a087b470e971ccc020", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811", + "runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec", + "runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v63-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v63-identity.json new file mode 100644 index 00000000000..10ea2c41afc --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v63-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:1412842a36a7211d84b6b98a3d1f69e3388bc4814e5825da890a774ae0e3460e", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 63, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629", + "semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603", + "semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a", + "providerLaunchSourceSha256": "fa8db1c42c378898287c141d8cea9069166981e369c9b9c51422cb6177de5a2c", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "9be35e006ffffcca48205f9e53548dd453ec6a8cbcf0aa4dd735194e9619dfc7", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918", + "profileCompatibilitySourceSha256": "5b00c3f78ad12321d48e66a90338af2d53d59dd17bb4b72b10ccdb4240b5a150", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811", + "runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec", + "runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index 347ec611be1..f7b82550879 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -1,6 +1,14 @@ +import { adapterExecutionControls, createAdapterExecutionControl } from "../services/adapter-execution-control.js"; +import { chatRunSourceMessageId } from "../services/chat-channels/run-origin.js"; +import * as durableChatWakeup from "../services/durable-chat-wakeup.js"; +import { conflict } from "../errors.js"; +import * as githubTaskPreparation from "../services/chat-channels/github/task-preparation.js"; +import { configureAgentLifecycle } from "../services/agent-lifecycle.js"; +import { budgetPolicies } from "@paperclipai/db"; +import { budgetService } from "../services/budgets.js"; import { idleWorkSnapshot } from "../services/task-admission.js"; import { readFreshWorkspaceReadiness } from "../services/fresh-workspace-continuation.js"; -import { requestTaskRetry } from "../services/task-retry.js"; +import { requestTaskRetry, reconcileTaskChatRetryReceipts } from "../services/task-retry.js"; import { createSystemRecovery } from "../services/system-recovery-state.js"; import { remoteTerminationReceipt } from "../services/remote-execution-termination.js"; import { stageReceiptReactionRemovals } from "../services/chat-receipt-reactions.js"; @@ -39,6 +47,11 @@ import { createAssignedMcpTools } from "../services/native-runtime/assigned-mcp- import { createToolGatewayService } from "../services/tool-gateway.js"; import { resolveGitHubOperationCredentials } from "../services/github-operation-credentials.js"; import { initializeRunIdentity } from "../services/run-identity.js"; +import { prepareHeartbeatExecutionConfiguration } from "../services/heartbeat/execution-configuration.js"; +import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js"; +import { companySkillService } from "../services/company-skills.js"; +import { companyObservability } from "@paperclipai/db"; +import * as runIdentityModule from "../services/run-identity.js"; import { chatGitHubRegistrations, chatGitHubReviews, chatGitHubConfigurations, toolCatalogEntries } from "@paperclipai/db"; import { AsyncLocalStorage } from "node:async_hooks"; import * as cloudRuntimeIdentity from "../services/cloud-runtime-identity.js"; @@ -60722,6 +60735,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { externalActorId?: string, linkedOriginalActor = false, githubUnavailableFile = false, + currentGitHubPr = false, ) { // Keep the policy under which the source was admitted. Inserting an // upgraded GitHub policy after receipt correctly invalidates retry proof. @@ -60733,6 +60747,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { linkedOriginalActor, githubUnavailableFile, false, + currentGitHubPr, ); const [issue] = await readIssuesById(context.conversation.issueId); const [action] = await db @@ -61129,6 +61144,901 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { }, ); + it.each(["contained", "prior_denied_intent", "observability_warning", "observability_reason", "observability_revision", "observability_provenance", "observability_extra", "other_reason", "wrong_issue", "provider_started", "result_extra", "profile", "usage", "output", + "process", "controller", "active_control", "lease", "accepted_result", "event_source", "extra_event", "source_edited", "principal_revoked"] as const)( + "retries an authenticated GitHub request after isolated-directory validation failed before allocation: %s", async mode => { + const f = await failedChatRetryFixture("github", "42", true, false, true); + const companyId = f.fixture.companyId, agentId = f.fixture.assignedAgentId, issueId = f.issue.id; + const diagnostic = "The existing execution workspace is not this task's isolated directory."; + try { + await initializeRunIdentity(db, { companyId, runId: f.runId, issueId, responsibleUserId: "owner-user", + messageIds: [String(f.action.payload.commentId)], cause: "chat_message" }); + await db.update(heartbeatRuns).set({ errorCode: "workspace_validation_failed", error: diagnostic, + runtimeMode: "legacy", sessionIdBefore: "prior-provider-display-hint", + usageJson: { ledgerScope: { issueId, projectId: null, billingCode: null } }, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, stopReason: "adapter_failed", + workspaceValidation: { reason: "isolated_task_directory_binding_mismatch", issueId } } }) + .where(eq(heartbeatRuns.id, f.runId)); + if (mode.startsWith("observability_")) { + await db.insert(companyObservability).values({ companyId, revision: randomUUID(), enabled: true, + destination: { provider: "otlp", endpointBaseUrl: "https://collector.example/otel" }, + capture: { userPrompts: false, assistantResponses: false, toolDetails: false, toolContent: false } }); + const [agent] = await db.update(agents).set({ adapterConfig: { provider: "opencode", model: "openrouter/synthetic", homeMode: "preconfigured" } }) + .where(eq(agents.id, agentId)).returning(); + const [run] = await readHeartbeatRunsById(f.runId); + const prepared = await prepareHeartbeatExecutionConfiguration(db, { run, agent, + task: { issueId, context: { ...run.contextSnapshot }, responsibleUserId: "owner-user", + acceptedPlanContinuationWake: false, acceptedPlanWakeRoutingDecision: null, taskProjectContext: null, + routineEnvContext: { routineId: null, env: null, responsibleUserId: null } }, + config: { sourceWorkspaceConfig: agent.adapterConfig, aiBinding: undefined }, + policy: { trustPreset: { kind: "standard", preset: "standard", boundary: null, sourcePresets: {} }, + useIsolatedTaskDirectory: false, explicitTaskDirectory: false, requestedExecutionWorkspaceMode: "shared_workspace" }, + environment: { selectedEnvironmentId: null, selectedEnvironmentForConfig: null, + resolvedInstanceSettings: await instanceSettingsService(db).get(), persistedNativeExecutionInput: null }, + services: { secretsSvc: secretService(db), companySkills: companySkillService(db), + appendRunEvent: async (source, event) => { await appendHeartbeatRunEvent(db, { companyId: source.companyId, + agentId: source.agentId, runId: source.id, ...event }); }, finalizeAiConnectionBusyDeferral: vi.fn() }, + onFailureSecretsResolved: () => {}, onManagedAiRuntimeAcquired: () => {}, onIdentityRedactorPrepared: () => {}, + }); + expect(prepared.prepared).toBe(true); + expect(await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.runId))) + .toMatchObject([{ eventType: "agent_observability.unavailable", payload: { reason: "unsupported_harness" } }]); + } + await appendHeartbeatRunEvent(db, { companyId, agentId, runId: f.runId, + eventType: "error", stream: "system", level: "error", message: diagnostic }); + await appendHeartbeatRunEvent(db, { companyId, agentId, runId: f.runId, eventType: "lifecycle", stream: "system", level: "info", + message: "task.run.measured", payload: { durableSettlement: true, durationMs: 2505, outcome: "failed", phase: "task.run.measured" } }); + await db.update(agents).set({ status: "running", runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }) + .where(eq(agents.id, agentId)); + const [capacity] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" }).returning(); + const heartbeat = heartbeatService(db); + f.wakeup.mockImplementation(heartbeat.wakeup); + const app = express(); + app.use(express.json()); + app.use((req, _res, next) => { req.actor = boardActor(companyId); next(); }); + app.use("/api", issueRoutes(db, createStorageService().storage, { chatRunRetries: f.service })); + app.use(errorHandler); + const [failed] = await readHeartbeatRunsById(f.runId); + if (mode === "observability_reason" || mode === "observability_revision" || mode === "observability_extra") { + const [warning] = await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, f.runId), eq(heartbeatRunEvents.seq, 1))); + await db.update(heartbeatRunEvents).set({ payload: { ...warning.payload, + ...(mode === "observability_reason" ? { reason: "unknown_preparation" } : mode === "observability_revision" ? { revision: null } : { providerWork: true }) } }) + .where(eq(heartbeatRunEvents.id, warning.id)); + } + if (mode === "observability_provenance") await db.update(heartbeatRunEvents).set({ sourceEventId: "untrusted-provider-warning" }) + .where(and(eq(heartbeatRunEvents.runId, f.runId), eq(heartbeatRunEvents.seq, 1))); + const validation = failed.resultJson!.workspaceValidation as object; + if (mode === "other_reason" || mode === "wrong_issue") await db.update(heartbeatRuns).set({ resultJson: { ...failed.resultJson, + workspaceValidation: { ...validation, ...(mode === "other_reason" ? { reason: "plugin_workspace_strategy_unsupported" } : { issueId: randomUUID() }) } } }) + .where(eq(heartbeatRuns.id, f.runId)); + if (mode === "provider_started" || mode === "result_extra") await db.update(heartbeatRuns).set({ resultJson: { ...failed.resultJson, + ...(mode === "provider_started" ? { executionRecovery: { kind: "bootstrap", providerWorkStarted: true } } : { unknownExecution: true }) } }) + .where(eq(heartbeatRuns.id, f.runId)); + if (mode === "profile") await db.update(heartbeatRuns).set({ runnerProfileJson: { ...failed.runnerProfileJson, nativeExecutionInput: {} } }).where(eq(heartbeatRuns.id, f.runId)); + if (mode === "usage") await db.update(heartbeatRuns).set({ usageJson: { ...failed.usageJson, tokens: 1 } }).where(eq(heartbeatRuns.id, f.runId)); + if (mode === "output") await db.update(heartbeatRuns).set({ stdoutExcerpt: "provider output" }).where(eq(heartbeatRuns.id, f.runId)); + if (mode === "process") await db.update(heartbeatRuns).set({ processPid: 999999 }).where(eq(heartbeatRuns.id, f.runId)); + if (mode === "controller") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60000) }).where(eq(heartbeatRuns.id, f.runId)); + if (mode === "active_control") adapterExecutionControls.set(f.runId, createAdapterExecutionControl()); + if (mode === "lease") await db.insert(environmentLeases).values({ companyId, heartbeatRunId: f.runId, issueId, + provider: "daytona", providerLeaseId: "contradictory-preparation-allocation", status: "active", leasePolicy: "ephemeral" }); + if (mode === "accepted_result") { + const contractId = randomUUID(), resultId = randomUUID(); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: {}, + canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" }); + await db.update(heartbeatRuns).set({ nativeIssueId: issueId, completionContractId: contractId }).where(eq(heartbeatRuns.id, f.runId)); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId: f.runId, completionContractId: contractId, + schemaStatus: "accepted", serverFingerprint: resultId, canonicalSha256: resultId, + resultJson: { terminal: { runTerminalState: "succeeded" } } }); + } + if (mode === "event_source") await db.update(heartbeatRunEvents).set({ sourceEventId: "provider-event" }).where(and(eq(heartbeatRunEvents.runId, f.runId), eq(heartbeatRunEvents.seq, 1))); + if (mode === "extra_event") await db.insert(heartbeatRunEvents).values({ companyId, agentId, runId: f.runId, + seq: 3, eventType: "lifecycle", stream: "system", level: "info", message: "unexpected provider preparation" }); + if (mode === "principal_revoked") await db.update(chatIdentityLinks).set({ status: "revoked", revokedAt: new Date() }).where(eq(chatIdentityLinks.endpointId, f.endpoint.id)); + if (mode === "contained") await expect(db.transaction(tx => f.service.prepareFailedChatRunRetry(tx, { + companyId, issueId, agentId, failedRunId: f.runId, initiatedByUserId: "owner-user", + }))).rejects.toMatchObject({ details: { code: "chat_failed_run_retry_not_authorized" } }); + const send = () => request(app).post(`/api/issues/${issueId}/retry`).send({ idempotencyKey: randomUUID(), observedRunId: f.runId }); + if (mode === "prior_denied_intent") { + const prior = await send().expect(202); + // Reproduce a terminal unadmitted authorization refusal, retaining the + // real HTTP intent/action receipts; the next normal click sees them. + await db.update(heartbeatRuns).set({ logBytes: 1 }).where(eq(heartbeatRuns.id, f.runId)); + await f.service.processFailedChatRunRetry(prior.body.chatActionId); + await db.update(heartbeatRuns).set({ logBytes: null }).where(eq(heartbeatRuns.id, f.runId)); + await reconcileTaskChatRetryReceipts(db); + expect((await readAgentWakeupRequestsById(prior.body.requestId))[0]).toMatchObject({ status: "failed", runId: null }); + expect((await db.select().from(chatActions).where(eq(chatActions.id, prior.body.chatActionId)))[0]).toMatchObject({ status: "failed" }); + } + const response = await send(); + if (["principal_revoked", "accepted_result"].includes(mode)) { + expect(response.status).toBe(409); + } else { + expect(response.status).toBe(202); + if (mode === "source_edited") await db.update(issueComments).set({ body: "Changed after Retry acceptance" }).where(eq(issueComments.id, String(f.action.payload.commentId))); + await f.service.processFailedChatRunRetry(response.body.chatActionId); + } + if (!["contained", "prior_denied_intent", "observability_warning"].includes(mode)) { + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0); + return; + } + const [wake] = await readAgentWakeupRequestsById(response.body.chatActionId); + expect(wake).toMatchObject({ status: "queued", runId: expect.any(String) }); + const [successor] = await readHeartbeatRunsById(wake.runId!); + expect(successor).toMatchObject({ retryOfRunId: f.runId, contextSnapshot: { + source: "chat:github", wakeCommentId: f.action.payload.commentId, taskRetryRequestId: response.body.requestId } }); + const realInitialize = runIdentityModule.initializeRunIdentity; + const initialized: string[] = []; + const probe = vi.spyOn(runIdentityModule, "initializeRunIdentity").mockImplementation(async (database, input) => { + const identity = await realInitialize(database, input); + if (input.companyId !== companyId) return identity; + expect(identity.messageId).toBe(f.action.payload.commentId); + initialized.push(input.runId); + throw new Error("workspace_retry_identity_probe_complete"); + }); + try { + await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, capacity.id)); + await db.update(agents).set({ status: "idle" }).where(eq(agents.id, agentId)); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + expect(initialized).toEqual([successor.id]); + expect((await readHeartbeatRunsById(successor.id))[0]).toMatchObject({ error: "workspace_retry_identity_probe_complete" }); + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(1); + } finally { probe.mockRestore(); await heartbeat.drainActiveRunExecutions(); } + } finally { + adapterExecutionControls.get(f.runId)?.finish(); adapterExecutionControls.delete(f.runId); + await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date() }).where(and( + eq(agentWakeupRequests.companyId, companyId), inArray(agentWakeupRequests.status, ["queued", "deferred_issue_execution"]))); + await f.service.shutdown(); + } + }); + + it("retries the saved board-message turn in a connected task after environment qualification failed before provider startup", async () => { + const f = await failedChatRetryFixture("github", "42", true, false, true); + try { + const companyId = f.fixture.companyId, agentId = f.fixture.assignedAgentId, issueId = f.issue.id; + await initializeRunIdentity(db, { companyId, runId: f.runId, issueId, responsibleUserId: "owner-user", + messageIds: [String(f.action.payload.commentId)], cause: "chat_message" }); + const [comment] = await db.insert(issueComments).values({ companyId, issueId, authorType: "user", + authorUserId: "owner-user", body: "Continue from saved work and review the current head." }).returning(); + await db.update(agents).set({ status: "running", runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }) + .where(eq(agents.id, agentId)); + const [capacity] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" }).returning(); + const heartbeat = heartbeatService(db); + const current = await heartbeat.wakeup(agentId, { source: "automation", triggerDetail: "system", + reason: "issue_reopened_via_comment", requestedByActorType: "user", requestedByActorId: "owner-user", + payload: { issueId, commentId: comment.id }, contextSnapshot: { issueId, taskKey: issueId, + source: "issue.comment.reopen", wakeCommentId: comment.id, wakeCommentIds: [comment.id] } }); + expect(current).toMatchObject({ status: "queued" }); + await initializeRunIdentity(db, { companyId, runId: current!.id, issueId, responsibleUserId: "owner-user", + parentRunId: f.runId, messageIds: [comment.id], cause: "issue_reopened_via_comment" }); + const diagnostic = 'Failed to acquire lease for environment "Managed computer" (sandbox): sandbox_snapshot_qualification_required: The snapshot qualification is missing or stale. Test this environment again before running Paperclip Runner.'; + await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), runtimeMode: "legacy", + errorCode: "setup_failed", error: diagnostic, sessionIdBefore: "prior-provider-session-hint", + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, + usageJson: { ledgerScope: { issueId, projectId: null } }, + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, stopReason: "adapter_failed" } }) + .where(eq(heartbeatRuns.id, current!.id)); + await db.update(agentWakeupRequests).set({ status: "failed", finishedAt: new Date() }).where(eq(agentWakeupRequests.id, current!.wakeupRequestId!)); + await db.insert(heartbeatRunEvents).values([ + { companyId, agentId, runId: current!.id, seq: 1, eventType: "agent_observability.unavailable", stream: "system", level: "warn", + message: "Company observability was not applied: this harness is not supported.", payload: { reason: "unsupported_harness", revision: randomUUID() } }, + { companyId, agentId, runId: current!.id, seq: 2, eventType: "error", stream: "system", level: "error", message: diagnostic }, + { companyId, agentId, runId: current!.id, seq: 3, eventType: "lifecycle", stream: "system", level: "info", + message: "task.run.measured", payload: { durableSettlement: true, durationMs: 18000, outcome: "failed", phase: "task.run.measured" } }, + ]); + await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, capacity.id)); + await db.update(issues).set({ status: "blocked", executionRunId: null }).where(eq(issues.id, issueId)); + await db.update(agents).set({ status: "idle" }).where(eq(agents.id, agentId)); + const [failed] = await readHeartbeatRunsById(current!.id); + await expect(chatRunSourceMessageId(db, failed, { provider: "github" })).resolves.toBe(f.action.payload.commentId); + const app = express(); + app.use(express.json()); + app.use((req, _res, next) => { req.actor = boardActor(companyId); next(); }); + app.use("/api", issueRoutes(db, createStorageService().storage, { chatRunRetries: f.service })); + app.use(errorHandler); + const denySource = async (apply: () => Promise, restore: () => Promise, stale = false) => { + try { + await apply(); + const denied = await request(app).post(`/api/issues/${issueId}/retry`).send({ idempotencyKey: randomUUID(), observedRunId: current!.id }).expect(stale ? 200 : 409); + if (stale) expect(denied.body).toMatchObject({ accepted: false, stale: true }); + expect(await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.reason, "task_retry")))).toHaveLength(0); + } finally { await restore(); } + }; + await denySource(() => db.update(issueComments).set({ body: "Edited after acceptance", updatedAt: new Date() }).where(eq(issueComments.id, comment.id)), + () => db.update(issueComments).set({ body: comment.body, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id))); + await denySource(() => db.update(issueComments).set({ authorUserId: "another-user" }).where(eq(issueComments.id, comment.id)), + () => db.update(issueComments).set({ authorUserId: comment.authorUserId, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id))); + await denySource(() => db.update(issueComments).set({ deletedAt: new Date() }).where(eq(issueComments.id, comment.id)), + () => db.update(issueComments).set({ deletedAt: null, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id))); + const [sourceWake] = await readAgentWakeupRequestsById(current!.wakeupRequestId!); + await denySource(() => db.update(agentWakeupRequests).set({ requestedByActorId: "another-user" }).where(eq(agentWakeupRequests.id, sourceWake.id)), + () => db.update(agentWakeupRequests).set({ requestedByActorId: sourceWake.requestedByActorId }).where(eq(agentWakeupRequests.id, sourceWake.id))); + for (const patch of [ + { responsibleUserId: "another-user" }, { agentId: f.fixture.replacementAgentId }, + { contextSnapshot: { ...failed.contextSnapshot, issueId: randomUUID() } }, + { activeIdentityContextId: null }, { sessionIdAfter: "provider-started" }, { processPid: 999999 }, + { controllerLeaseExpiresAt: new Date(Date.now() + 60000) }, { stdoutExcerpt: "provider output" }, + { resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: true } } }, + { runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" }, unexpectedExecution: true } }, + ]) { + const restore = Object.fromEntries(Object.keys(patch).map(key => [key, failed[key as keyof typeof failed]])); + await denySource(() => db.update(heartbeatRuns).set(patch).where(eq(heartbeatRuns.id, failed.id)), + () => db.update(heartbeatRuns).set(restore).where(eq(heartbeatRuns.id, failed.id)), "agentId" in patch); + } + const contractId = randomUUID(), resultId = randomUUID(); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: {}, + canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" }); + await denySource(async () => { + await db.update(heartbeatRuns).set({ nativeIssueId: issueId, completionContractId: contractId }).where(eq(heartbeatRuns.id, failed.id)); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId: failed.id, + completionContractId: contractId, schemaStatus: "accepted", serverFingerprint: resultId, canonicalSha256: resultId, + resultJson: { terminal: { runTerminalState: "succeeded" } } }); + }, async () => { + await db.delete(nativeRunResults).where(eq(nativeRunResults.id, resultId)); + await db.update(heartbeatRuns).set({ nativeIssueId: null, completionContractId: null }).where(eq(heartbeatRuns.id, failed.id)); + await db.delete(completionContracts).where(eq(completionContracts.id, contractId)); + }); + const [liveLease] = await db.insert(environmentLeases).values({ companyId, issueId, heartbeatRunId: failed.id, + provider: "daytona", providerLeaseId: "contradictory-board-startup", status: "active", leasePolicy: "ephemeral" }).returning(); + await denySource(async () => {}, () => db.delete(environmentLeases).where(eq(environmentLeases.id, liveLease.id))); + adapterExecutionControls.set(failed.id, createAdapterExecutionControl()); + await denySource(async () => {}, async () => { adapterExecutionControls.get(failed.id)?.finish(); adapterExecutionControls.delete(failed.id); }); + const idempotencyKey = randomUUID(); + const response = await request(app).post(`/api/issues/${issueId}/retry`).send({ idempotencyKey, observedRunId: current!.id }).expect(202); + expect(response.body).toMatchObject({ accepted: true, observedRunId: current!.id }); + expect(response.body.chatActionId).toBeUndefined(); + const [retry] = await readAgentWakeupRequestsById(response.body.requestId); + expect(retry).toMatchObject({ reason: "task_retry", requestedByActorId: "owner-user", + payload: { issueId, taskRetry: { sourceRunId: current!.id } } }); + expect(await db.select().from(chatActions).where(and(eq(chatActions.companyId, companyId), eq(chatActions.kind, "failed_run_retry")))).toHaveLength(0); + await db.update(agents).set({ status: "running" }).where(eq(agents.id, agentId)); + const [dispatchCapacity] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" }).returning(); + await db.update(issueComments).set({ body: "Changed after Retry acceptance", updatedAt: new Date() }).where(eq(issueComments.id, comment.id)); + await heartbeat.resumeExecutionWaitComments(); + expect((await readAgentWakeupRequestsById(retry.id))[0]).toMatchObject({ status: "deferred_issue_execution", runId: null }); + await db.update(issueComments).set({ body: comment.body, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id)); + await heartbeat.resumeExecutionWaitComments(); + const [successor] = (await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.contextSnapshot?.taskRetryRequestId === retry.id); + expect(successor).toMatchObject({ status: "queued", responsibleUserId: "owner-user", + contextSnapshot: { taskRetryRequestId: retry.id, previousRunId: current!.id } }); + expect((await readAgentWakeupRequestsById(retry.id))[0]).toMatchObject({ runId: successor.id }); + const realInitialize = runIdentityModule.initializeRunIdentity; + const initialized: string[] = []; + const probe = vi.spyOn(runIdentityModule, "initializeRunIdentity").mockImplementation(async (database, input) => { + if (input.runId !== successor.id) return realInitialize(database, input); + const [canonical] = await readAgentWakeupRequestsById(retry.id); + const [dispatch] = await readAgentWakeupRequestsById(successor.wakeupRequestId!); + const denyIdentity = async (apply: () => Promise, restore: () => Promise) => { + try { await apply(); await expect(realInitialize(database, input)).rejects.toThrow("This saved message can no longer be retried safely"); } + finally { await restore(); } + expect((await readHeartbeatRunsById(input.runId))[0].activeIdentityContextId).toBeNull(); + }; + await denyIdentity(() => db.update(agentWakeupRequests).set({ payload: { ...canonical.payload, + taskRetry: { ...(canonical.payload!.taskRetry as object), boardMessageOrigin: undefined } } }).where(eq(agentWakeupRequests.id, canonical.id)), + () => db.update(agentWakeupRequests).set({ payload: canonical.payload }).where(eq(agentWakeupRequests.id, canonical.id))); + await denyIdentity(() => db.update(agentWakeupRequests).set({ runId: failed.id }).where(eq(agentWakeupRequests.id, canonical.id)), + () => db.update(agentWakeupRequests).set({ runId: canonical.runId }).where(eq(agentWakeupRequests.id, canonical.id))); + await denyIdentity(() => db.update(agentWakeupRequests).set({ requestedByActorId: "another-user" }).where(eq(agentWakeupRequests.id, canonical.id)), + () => db.update(agentWakeupRequests).set({ requestedByActorId: canonical.requestedByActorId }).where(eq(agentWakeupRequests.id, canonical.id))); + await denyIdentity(() => db.update(agentWakeupRequests).set({ runId: failed.id }).where(eq(agentWakeupRequests.id, dispatch.id)), + () => db.update(agentWakeupRequests).set({ runId: dispatch.runId }).where(eq(agentWakeupRequests.id, dispatch.id))); + await denyIdentity(() => db.update(issueComments).set({ body: "Changed before provider dispatch", updatedAt: new Date() }).where(eq(issueComments.id, comment.id)), + () => db.update(issueComments).set({ body: comment.body, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id))); + await denyIdentity(() => db.update(heartbeatRuns).set({ contextSnapshot: { ...successor.contextSnapshot, taskId: randomUUID() } }).where(eq(heartbeatRuns.id, successor.id)), + () => db.update(heartbeatRuns).set({ contextSnapshot: successor.contextSnapshot }).where(eq(heartbeatRuns.id, successor.id))); + await denyIdentity(() => db.update(issues).set({ executionRunId: failed.id }).where(eq(issues.id, issueId)), + () => db.update(issues).set({ executionRunId: successor.id }).where(eq(issues.id, issueId))); + const identity = await realInitialize(database, input); + expect(identity).toMatchObject({ responsibleUserId: "owner-user", parentContextId: retry.payload!.taskRetry && + (retry.payload!.taskRetry as { boardMessageOrigin: { identityContextId: string } }).boardMessageOrigin.identityContextId }); + const [executing] = await readHeartbeatRunsById(input.runId); + await expect(chatRunSourceMessageId(db, executing, { provider: "github" })).resolves.toBe(f.action.payload.commentId); + initialized.push(input.runId); + // This fixture tests the real dispatch and identity boundary, without + // allocating a provider or pretending its task work has completed. + throw new Error("board_retry_identity_probe_complete"); + }); + try { + await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, dispatchCapacity.id)); + await db.update(agents).set({ status: "idle" }).where(eq(agents.id, agentId)); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + expect(initialized).toEqual([successor.id]); + expect((await readHeartbeatRunsById(successor.id))[0]).toMatchObject({ status: "failed", error: "board_retry_identity_probe_complete" }); + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.contextSnapshot?.taskRetryRequestId === retry.id)).toHaveLength(1); + expect((await readHeartbeatRunsById(current!.id))[0]).toMatchObject({ status: "failed", activeIdentityContextId: failed.activeIdentityContextId }); + } finally { probe.mockRestore(); await heartbeat.drainActiveRunExecutions(); } + const [firstRetry] = await readHeartbeatRunsById(successor.id); + const nextRequest = () => request(app).post(`/api/issues/${issueId}/retry`).send({ + idempotencyKey: randomUUID(), observedRunId: firstRetry.id }); + await db.update(heartbeatRuns).set({ sessionIdAfter: "contradictory-provider-session" }).where(eq(heartbeatRuns.id, firstRetry.id)); + await nextRequest().expect(409); + await db.update(heartbeatRuns).set({ sessionIdAfter: null }).where(eq(heartbeatRuns.id, firstRetry.id)); + await db.update(issueComments).set({ body: "Changed original accepted message", updatedAt: new Date() }).where(eq(issueComments.id, comment.id)); + await nextRequest().expect(409); + await db.update(issueComments).set({ body: comment.body, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id)); + const [firstIntent] = await readAgentWakeupRequestsById(retry.id); + await db.update(agentWakeupRequests).set({ payload: { ...firstIntent.payload, + taskRetry: { ...(firstIntent.payload!.taskRetry as object), sourceRunId: firstRetry.id } } }).where(eq(agentWakeupRequests.id, firstIntent.id)); + await db.update(heartbeatRuns).set({ contextSnapshot: { ...firstRetry.contextSnapshot, previousRunId: firstRetry.id } }).where(eq(heartbeatRuns.id, firstRetry.id)); + await nextRequest().expect(409); + await db.update(agentWakeupRequests).set({ payload: firstIntent.payload }).where(eq(agentWakeupRequests.id, firstIntent.id)); + await db.update(heartbeatRuns).set({ contextSnapshot: firstRetry.contextSnapshot }).where(eq(heartbeatRuns.id, firstRetry.id)); + const second = await nextRequest().expect(202); + expect(second.body).toMatchObject({ accepted: true, observedRunId: firstRetry.id }); + expect(second.body.chatActionId).toBeUndefined(); + const [secondIntent] = await readAgentWakeupRequestsById(second.body.requestId); + expect(secondIntent.payload).toMatchObject({ taskRetry: { sourceRunId: firstRetry.id, + boardMessageOrigin: { commentId: comment.id, identityContextId: firstRetry.activeIdentityContextId } } }); + const secondIdentities: string[] = []; + const secondProbe = vi.spyOn(runIdentityModule, "initializeRunIdentity").mockImplementation(async (database, input) => { + const identity = await realInitialize(database, input); + if (input.companyId === companyId) { + expect(identity).toMatchObject({ responsibleUserId: "owner-user", parentContextId: firstRetry.activeIdentityContextId }); + const [executing] = await readHeartbeatRunsById(input.runId); + await expect(chatRunSourceMessageId(db, executing, { provider: "github" })).resolves.toBe(f.action.payload.commentId); + secondIdentities.push(input.runId); + throw new Error("board_retry_identity_probe_complete"); + } + return identity; + }); + try { + await heartbeat.resumeExecutionWaitComments(); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + const [secondRun] = (await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.contextSnapshot?.taskRetryRequestId === secondIntent.id); + expect(secondRun).toMatchObject({ status: "failed", error: "board_retry_identity_probe_complete", responsibleUserId: "owner-user" }); + expect(secondIdentities).toEqual([secondRun.id]); + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.contextSnapshot?.taskRetryRequestId === secondIntent.id)).toHaveLength(1); + } finally { secondProbe.mockRestore(); await heartbeat.drainActiveRunExecutions(); } + + } finally { + await db.update(agentWakeupRequests).set({ status: "failed", finishedAt: new Date() }).where(and( + eq(agentWakeupRequests.companyId, f.fixture.companyId), eq(agentWakeupRequests.reason, "task_retry"), + eq(agentWakeupRequests.status, "deferred_issue_execution"))); + await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and( + eq(heartbeatRuns.companyId, f.fixture.companyId), eq(heartbeatRuns.status, "queued"))); + await f.service.shutdown(); + } + }); + + it("distinguishes an inherited GitHub requester from permission to replay a committed provider failure", async () => { + const f = await failedChatRetryFixture("github", "42", true, false, true); + try { + const companyId = f.fixture.companyId, agentId = f.fixture.assignedAgentId, issueId = f.issue.id; + await initializeRunIdentity(db, { companyId, runId: f.runId, issueId, responsibleUserId: "owner-user", + messageIds: [String(f.action.payload.commentId)], cause: "chat_message" }); + await db.update(agents).set({ status: "running", runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }) + .where(eq(agents.id, agentId)); + const [capacity] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" }).returning(); + const heartbeat = heartbeatService(db); + const resumed = await heartbeat.wakeup(agentId, { source: "on_demand", triggerDetail: "manual", + reason: "issue_tree_resumed", requestedByActorType: "user", requestedByActorId: "owner-user", + payload: { issueId }, contextSnapshot: { issueId, taskKey: f.issue.identifier, source: "issue.tree_resume" } }); + expect(resumed).toMatchObject({ status: "queued" }); + await initializeRunIdentity(db, { companyId, runId: resumed!.id, issueId, responsibleUserId: "owner-user", + parentRunId: f.runId, cause: "issue_tree_resumed" }); + const contractId = randomUUID(), resultId = randomUUID(); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: {}, + canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" }); + await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), runtimeMode: "native", + nativeIssueId: issueId, nativePhase: "committed", errorCode: "adapter_failed", error: "Native session failed", + completionContractId: contractId }).where(eq(heartbeatRuns.id, resumed!.id)); + await db.update(agentWakeupRequests).set({ status: "failed", finishedAt: new Date() }).where(eq(agentWakeupRequests.id, resumed!.wakeupRequestId!)); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId: resumed!.id, + completionContractId: contractId, schemaStatus: "accepted", serverFingerprint: resultId, canonicalSha256: resultId, + resultJson: { result: { reportedWorkDisposition: "needs_review", completionClaim: { objectiveSatisfied: false } }, + terminal: { runTerminalState: "failed", turnTerminalState: "failed" } } }); + await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId: resumed!.id, phase: "committed", resultId }); + const lease = { id: randomUUID(), companyId, heartbeatRunId: resumed!.id, provider: "daytona", providerLeaseId: "contained-resume-provider" }; + await db.insert(environmentLeases).values({ ...lease, issueId, status: "expired", leasePolicy: "ephemeral", + cleanupStatus: "success", releasedAt: new Date(), metadata: { remoteExecutionTermination: + remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }) } }); + await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, capacity.id)); + await db.update(issues).set({ status: "blocked", executionRunId: null }).where(eq(issues.id, issueId)); + await db.update(agents).set({ status: "idle" }).where(eq(agents.id, agentId)); + const [failedResume] = await readHeartbeatRunsById(resumed!.id); + expect(failedResume.contextSnapshot).toMatchObject({ source: "issue.tree_resume" }); + expect(failedResume.contextSnapshot).not.toHaveProperty("wakeCommentIds"); + await expect(chatRunSourceMessageId(db, failedResume, { provider: "github" })) + .resolves.toBe(f.action.payload.commentId); + const input = { db, companyId, issueId, actorId: "owner-user", actorSource: "cloud_tenant", + observedRunId: resumed!.id, idempotencyKey: randomUUID(), prepareChatRetry: f.service.prepareFailedChatRunRetry }; + const app = express(); + app.use(express.json()); + app.use((req, _res, next) => { req.actor = boardActor(companyId); next(); }); + app.use("/api", issueRoutes(db, createStorageService().storage, { chatRunRetries: f.service })); + app.use(errorHandler); + const denied = await request(app).post(`/api/issues/${issueId}/retry`).send({ + idempotencyKey: input.idempotencyKey, observedRunId: input.observedRunId, + }).expect(409); + expect(denied.body).toMatchObject({ error: "This attempt already recorded a result. Send a new message on the task to continue from the saved work.", + details: { code: "chat_failed_run_retry_not_authorized", reason: "accepted_result_without_new_input" } }); + // Isolate the later accepted-result guard using only owned fixture data. + // Resolving inherited provenance cannot by itself authorize replay. + await db.update(heartbeatRuns).set({ contextSnapshot: { ...failedResume.contextSnapshot, + source: "chat:github", wakeCommentId: f.action.payload.commentId, wakeCommentIds: [f.action.payload.commentId] } }) + .where(eq(heartbeatRuns.id, failedResume.id)); + try { + await expect(requestTaskRetry({ ...input, idempotencyKey: randomUUID() })).rejects.toMatchObject({ details: { + code: "chat_failed_run_retry_not_authorized", reason: "accepted_result_without_new_input" } }); + } finally { await db.update(heartbeatRuns).set({ contextSnapshot: failedResume.contextSnapshot }).where(eq(heartbeatRuns.id, failedResume.id)); } + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === resumed!.id)).toHaveLength(0); + } finally { await f.service.shutdown(); } + }); + + it.each([ + ...(["real_budget_ingress", "legacy_budget_ingress", "legacy_prior_response", "legacy_execution", "legacy_execution_log", "legacy_execution_lease", "legacy_execution_profile", "legacy_execution_event", "legacy_execution_controller", "legacy_execution_intent", "legacy_no_prior", "legacy_prior_late", "legacy_prior_unprocessed", "legacy_prior_delivery_unprocessed", "legacy_prior_numeric_app", "legacy_response_conflict", "legacy_prior_wrong_app", "legacy_prior_wrong_agent", "legacy_prior_unlinked", "legacy_prior_retargeted", "legacy_explicit_null", "legacy_explicit_empty", "legacy_explicit_malformed", "legacy_explicit_mismatch", "legacy_current_app_changed", "source_app_changed", "source_destination_changed", "accepted", "multiple_saved", "real_queue", "committed_succeeded", "committed_consumed", "committed_failure", "committed_failed_publication", "committed_unknown_publication", "committed_pending_publication", "committed_mixed_publication", "committed_consumed_publication", "committed_succeeded_publication", "committed_delayed_cleanup", "committed_unknown_cleanup", "committed_live_controller", "unreconciled", "source_edited", "source_claimed", "malformed_previous", "endpoint_paused", "principal_revoked", "still_budget_blocked"] as const) + .map(mode => ({ provider: "github" as const, mode })), + ])("retries the newer saved $provider request instead of the failed request after a budget hold: $mode", async ({ provider, mode }) => { + const legacyIngress = mode.startsWith("legacy_"); + const realBudgetIngress = mode === "real_budget_ingress" || legacyIngress; + const executionRevalidation = mode.startsWith("legacy_execution"); + if (legacyIngress) { + // The legacy intake producer retained requester authority but no App ID. + // A budget refusal never reaches the acknowledgement which could bind it. + const historical = vi.spyOn(githubTaskPreparation, "retainGitHubDeliveryAuthority").mockImplementation( + async (tx, endpoint, deliveryId, linkedUserId, taskUserId) => { + const guest = !linkedUserId; + await tx.update(chatDeliveries).set({ normalizedEvent: sql`${chatDeliveries.normalizedEvent} || ${JSON.stringify({ + githubAuthority: { guest, responsibleUserId: taskUserId, sponsorUserId: guest ? endpoint.sponsorUserId : null }, + })}::jsonb` }).where(eq(chatDeliveries.id, deliveryId)); + }); + onTestFinished(() => historical.mockRestore()); + } + const context = await failedChatRetryFixture(provider, realBudgetIngress ? "42" : undefined, true, false, realBudgetIngress); + try { + const companyId = context.fixture.companyId; + const [skill] = await db.insert(companySkills).values({ companyId, key: `company/${companyId}/saved-review`, + slug: "saved-review", name: "Saved review", markdown: "# Saved review" }).returning(); + if (provider === "github") { + const configured = await githubChatManagementService(db).configuration(context.endpoint.id, "owner-user"); + const configuration = { ...configured.configuration, defaults: { ...configured.configuration.defaults, + instructions: `Use [/saved-review](skill://${skill.id}).` } }; + if (realBudgetIngress) await githubChatManagementService(db).saveConfiguration(context.endpoint.id, + { expectedRevision: configured.revision, configuration }, "owner-user"); + else await db.insert(chatGitHubConfigurations).values({ companyId, endpointId: context.endpoint.id, + configuration, updatedByUserId: "owner-user", revision: 2 }); + } + if (legacyIngress) { + await githubResponseCommentService(db, context.providerFetch).tryAcknowledge(context.action.deliveryId!); + const [prior] = await db.select().from(chatActions).where(and( + eq(chatActions.deliveryId, context.action.deliveryId!), eq(chatActions.kind, "github_response_comment"))); + expect(prior).toMatchObject({ status: "processed", payload: { + botExternalId: (await context.service.get(context.endpoint.id)).botExternalId, agentId: context.fixture.assignedAgentId, + } }); + } + if (mode === "legacy_no_prior") await db.delete(chatActions).where(and( + eq(chatActions.deliveryId, context.action.deliveryId!), eq(chatActions.kind, "github_response_comment"))); + if (realBudgetIngress) { + await db.update(heartbeatRuns).set({ costAccountingPending: true }).where(eq(heartbeatRuns.id, context.runId)); + await db.insert(budgetPolicies).values({ companyId, scopeType: "agent", scopeId: context.fixture.assignedAgentId, + metric: "billed_cents", windowKind: "lifetime", amount: 100, notifyEnabled: false }); + context.wakeup.mockImplementation(heartbeatService(db).wakeup); + } else context.wakeup.mockImplementation(async (agentId, opts) => { + const request = opts.durableChatRequest!; + await request.authorize(db); + await db.insert(agentWakeupRequests).values({ id: request.id, companyId, + agentId, source: opts.source!, triggerDetail: opts.triggerDetail, reason: "budget.blocked", + requestedByActorType: request.requestedByActorType, requestedByActorId: request.requestedByActorId, + requestedAt: request.requestedAt, idempotencyKey: request.idempotencyKey, + payload: { ...opts.payload, executionWait: { scopeType: "agent", scopeId: agentId } }, + status: "skipped", finishedAt: new Date() }); + return null; + }); + if (realBudgetIngress) { + const endpoint = await context.service.get(context.endpoint.id); + attachFakeGitHubIssueCommentWebhook({ botUsername: endpoint.botUsername!, callbacks: context.runtime.configurations.get(endpoint.id)!.callbacks, + endpointId: endpoint.id, runtime: context.runtime.endpoints.get(endpoint.id)! }); + const ref = (await db.select().from(toolConnections).where(eq(toolConnections.id, endpoint.connectionId)))[0]! + .credentialSecretRefs.find(ref => ref.configPath === "credentials.webhookSecret")!; + const secret = await secretService(db).resolveSecretValue(companyId, ref.secretId, ref.versionSelector ?? "latest"); + const response = await context.service.handleWebhook(endpoint.publicId, "github", signedGitHubWebhookRequest({ + delivery: randomUUID(), event: "issue_comment", webhookSecret: secret, + payload: { action: "created", installation: { id: 2468 }, + repository: { id: 97531, full_name: "paperclipai/paperclip", owner: { id: 1357, login: "paperclipai" } }, + issue: { number: 791, pull_request: { url: "https://api.github.com/repos/paperclipai/paperclip/pulls/791" } }, + sender: { id: 42, login: "octocat", type: "User" }, + comment: { id: 990092, body: `@${endpoint.botUsername} review using the selected instructions`, + user: { id: 42, login: "octocat", type: "User" }, created_at: new Date().toISOString() } }, + })); + expect(response.status).toBeLessThan(300); + } else await deliverMessage({ callbacks: context.runtime.configurations.get(context.endpoint.id)!.callbacks, + endpointId: context.endpoint.id, provider, thread: context.thread.thread, + message: makeMessage({ id: provider === "telegram" ? `${context.thread.thread.channelId}:102` : "saved-after-budget-hold", text: "@maya review using the selected instructions", + userId: provider === "telegram" ? context.thread.thread.channelId : "U-SAFE-PROGRESS", mentioned: true }), + trigger: context.thread.thread.isDM ? "direct_message" : "mention" }); + if (mode === "multiple_saved") await deliverMessage({ + callbacks: context.runtime.configurations.get(context.endpoint.id)!.callbacks, + endpointId: context.endpoint.id, provider, thread: context.thread.thread, + message: makeMessage({ id: "second-saved-after-budget-hold", text: "@maya include the follow-up", + userId: "U-SAFE-PROGRESS", mentioned: true }), trigger: "mention" }); + const savedInputs = await db.select().from(chatActions).where(and( + eq(chatActions.companyId, companyId), eq(chatActions.conversationId, context.conversation.id), + eq(chatActions.kind, "inbound_wakeup"), sql`${chatActions.id} <> ${context.action.id}`)) + .orderBy(asc(chatActions.createdAt), asc(chatActions.id)); + const saved = savedInputs.at(-1)!; + const expectedComments = savedInputs.map(action => action.payload.commentId); + expect(saved, JSON.stringify((await db.select({ state: chatDeliveries.state, error: chatDeliveries.redactedError, event: chatDeliveries.eventKind }).from(chatDeliveries).where(eq(chatDeliveries.endpointId, context.endpoint.id))))).toMatchObject({ status: "failed", result: { receiptStatus: "skipped" } }); + expect(saved.payload.commentId).not.toBe(context.action.payload.commentId); + if (realBudgetIngress) { + expect((await readAgentWakeupRequestsById(saved.id))[0]).toMatchObject({ status: "skipped", reason: "budget.blocked", + runId: null, requestedByActorType: "user", requestedByActorId: "owner-user" }); + await db.update(budgetPolicies).set({ isActive: false }).where(eq(budgetPolicies.companyId, companyId)); + await budgetService(db).getInvocationBlock(companyId, context.fixture.assignedAgentId); + const lifecycle = configureAgentLifecycle(db, { requiredPluginIds: async () => [], + runPlugin: async () => "complete", runHost: async () => "complete" }); + try { await lifecycle.process(context.fixture.assignedAgentId); } finally { await lifecycle.stop(); } + expect((await db.select().from(agents).where(eq(agents.id, context.fixture.assignedAgentId)))[0].lifecycleHolds).not.toContain("budget"); + expect((await db.select().from(agents).where(eq(agents.id, context.fixture.assignedAgentId)))[0]).toMatchObject({status: "idle"}); + expect((await readHeartbeatRunsById(context.runId))[0]).toMatchObject({status: "failed"}); + } + const unadmitted = () => db.select({ id: issues.id }).from(issues).where(and(eq(issues.id, context.issue.id), + unadmittedChatWakeupCondition(issues.id, issues.companyId))); + expect(await unadmitted()).toHaveLength(1); + // The budget was cleared. Retry must reconsider this independent saved + // request through chat admission, not replay the older failed turn. + const held = mode === "unreconciled" || mode === "committed_delayed_cleanup" || mode === "committed_unknown_cleanup" || mode === "committed_live_controller"; + if (mode === "unreconciled" || mode.startsWith("committed_") || executionRevalidation) { + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: context.issue.id, + nativePhase: "terminal_failure", runnerProfileJson: {} }).where(eq(heartbeatRuns.id, context.runId)); + await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: context.issue.id, + kind: "active_run_watchdog", cause: "native_continuation_requires_reconciliation", status: "active", + ownerType: "board", fingerprint: randomUUID(), nextAction: "Reconcile the previous executor.", + evidence: { runId: context.runId, automaticRecovery: { replay: "blocked" } } }); + if (mode.startsWith("committed_") || executionRevalidation) { + const contractId = randomUUID(), resultId = randomUUID(); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId: context.issue.id, + revision: 1, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: {}, + canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" }); + await db.update(heartbeatRuns).set({ nativePhase: "committed", completionContractId: contractId }) + .where(eq(heartbeatRuns.id, context.runId)); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId: context.issue.id, + runId: context.runId, completionContractId: contractId, serverFingerprint: resultId, + schemaStatus: "accepted", canonicalSha256: resultId, + resultJson: { terminal: { runTerminalState: mode.startsWith("committed_succeeded") ? "succeeded" : "failed" } } }); + await db.insert(nativeRunFinalizations).values({ companyId, issueId: context.issue.id, + runId: context.runId, resultId, phase: "committed", + leaseOwner: mode === "committed_live_controller" ? "active-controller" : null }); + const lease = { id: randomUUID(), companyId, heartbeatRunId: context.runId, + provider: "daytona", providerLeaseId: "owned-saved-input-sandbox" }; + await db.insert(environmentLeases).values({ ...lease, issueId: context.issue.id, + status: "expired", cleanupStatus: "success", releasedAt: new Date(), leasePolicy: "ephemeral", + metadata: mode === "committed_unknown_cleanup" || mode === "committed_delayed_cleanup" ? {} : { + remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }) } }); + } + } + if (executionRevalidation || ["committed_failed_publication", "committed_unknown_publication", "committed_pending_publication", "committed_mixed_publication", "committed_consumed_publication", "committed_succeeded_publication"].includes(mode)) { + // A failed native run materializes its summary before delivery. Keep + // this historical output distinct from the newer user's request. + const [resultComment] = await db.insert(issueComments).values({ companyId, issueId: context.issue.id, + authorAgentId: context.fixture.assignedAgentId, createdByRunId: context.runId, + body: "The OpenCode run failed before it completed." }).returning(); + await db.insert(chatPublications).values({ companyId, endpointId: context.endpoint.id, + conversationId: context.conversation.id, issueId: context.issue.id, commentId: resultComment.id, + idempotencyKey: `run:${context.runId}:failed-result`, payload: { text: resultComment.body }, + state: executionRevalidation || ["committed_failed_publication", "committed_mixed_publication", "committed_consumed_publication", "committed_succeeded_publication"].includes(mode) ? "failed" + : mode === "committed_unknown_publication" ? "delivery_unknown" : "pending", + redactedError: "GitHub failure has no request binding" }); + if (mode === "committed_mixed_publication") await db.insert(chatPublications).values({ companyId, + endpointId: context.endpoint.id, conversationId: context.conversation.id, issueId: context.issue.id, + commentId: resultComment.id, idempotencyKey: `run:${context.runId}:uncertain-result`, + payload: { text: resultComment.body }, state: "delivery_unknown" }); + } + if (mode.startsWith("committed_consumed")) await db.update(heartbeatRuns).set({ contextSnapshot: { + ...((await readHeartbeatRunsById(context.runId))[0]).contextSnapshot, + wakeCommentId: saved.payload.commentId, wakeCommentIds: [saved.payload.commentId], + } }).where(eq(heartbeatRuns.id, context.runId)); + if (mode === "malformed_previous") await db.update(heartbeatRuns).set({ contextSnapshot: { + ...((await readHeartbeatRunsById(context.runId))[0]).contextSnapshot, wakeCommentIds: ["unbound"], + } }).where(eq(heartbeatRuns.id, context.runId)); + if (mode === "source_app_changed" || mode === "source_destination_changed") { + const [delivery] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.id, saved.deliveryId!)); + await db.update(chatDeliveries).set({ normalizedEvent: { ...delivery.normalizedEvent, + ...(mode === "source_app_changed" ? { githubAuthority: { ...(delivery.normalizedEvent.githubAuthority as object), botExternalId: "different-app" } } + : { resource: { ...(delivery.normalizedEvent.resource as object), providerResourceId: "another/repository" } }), + } }).where(eq(chatDeliveries.id, delivery.id)); + } + const previousPublications = await readChatPublicationsByEndpointId(context.endpoint.id); + const request = { db, companyId, issueId: context.issue.id, actorId: "owner-user", actorSource: "cloud_tenant", + observedRunId: context.runId, idempotencyKey: randomUUID(), prepareChatRetry: context.service.prepareFailedChatRunRetry }; + if (legacyIngress) { + const [delivery] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.id, saved.deliveryId!)); + expect(delivery.normalizedEvent.githubAuthority).not.toHaveProperty("botExternalId"); + expect(await db.select({ id: chatActions.id }).from(chatActions).where(and( + eq(chatActions.deliveryId, saved.deliveryId!), eq(chatActions.kind, "github_response_comment")))).toHaveLength(0); + const priorWhere = and(eq(chatActions.deliveryId, context.action.deliveryId!), eq(chatActions.kind, "github_response_comment")); + if (mode === "legacy_prior_late") await db.update(chatActions).set({ createdAt: new Date(delivery.createdAt.getTime() + 1) }).where(priorWhere); + if (mode === "legacy_prior_unprocessed") await db.update(chatActions).set({ status: "received" }).where(priorWhere); + if (mode === "legacy_prior_delivery_unprocessed") await db.update(chatDeliveries).set({ state: "failed" }) + .where(eq(chatDeliveries.id, context.action.deliveryId!)); + if (mode === "legacy_prior_numeric_app") await db.update(chatActions).set({ + payload: sql`jsonb_set(${chatActions.payload}, '{botExternalId}', ${JSON.stringify(Number((await context.service.get(context.endpoint.id)).botExternalId))}::jsonb)`, + }).where(priorWhere); + if (mode === "legacy_response_conflict") await db.insert(chatActions).values({ companyId, + endpointId: context.endpoint.id, conversationId: context.conversation.id, deliveryId: delivery.id, + principalId: delivery.principalId, kind: "github_response_comment", providerActionId: `github_response:${delivery.id}`, + status: "processed", payload: { botExternalId: "conflicting-app", agentId: context.fixture.assignedAgentId } }); + if (mode === "legacy_prior_wrong_app" || mode === "legacy_prior_wrong_agent") await db.update(chatActions).set({ + payload: sql`jsonb_set(${chatActions.payload}, ${mode === "legacy_prior_wrong_app" ? "{botExternalId}" : "{agentId}"}, '"another-identity"'::jsonb)`, + }).where(priorWhere); + if (mode === "legacy_prior_unlinked") await db.delete(chatMessageLinks).where(eq(chatMessageLinks.deliveryId, context.action.deliveryId!)); + if (mode === "legacy_prior_retargeted") await db.update(chatActions).set({ providerActionId: "github_response:unbound" }).where(priorWhere); + if (mode.startsWith("legacy_explicit_")) await db.update(chatDeliveries).set({ normalizedEvent: { + ...delivery.normalizedEvent, githubAuthority: { ...(delivery.normalizedEvent.githubAuthority as object), + botExternalId: mode === "legacy_explicit_null" ? null : mode === "legacy_explicit_empty" ? "" + : mode === "legacy_explicit_malformed" ? { appId: "untrusted" } : "another-app" }, + } }).where(eq(chatDeliveries.id, delivery.id)); + if (mode === "legacy_current_app_changed") await db.update(chatEndpoints).set({ botExternalId: randomUUID() }) + .where(eq(chatEndpoints.id, context.endpoint.id)); + if (mode !== "legacy_budget_ingress" && mode !== "legacy_prior_response" && !executionRevalidation) { + await expect(requestTaskRetry(request)).rejects.toMatchObject({ details: { + code: "chat_failed_run_retry_not_authorized", reason: "source_authorization", authorityReason: "app_binding", + } }); + expect((await readAgentWakeupRequestsById(saved.id))[0]).toMatchObject({ status: "skipped", runId: null }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual(previousPublications); + return; + } + } + if (mode === "source_app_changed" || mode === "source_destination_changed") { + await expect(requestTaskRetry(request)).rejects.toMatchObject({ details: { + code: "chat_failed_run_retry_not_authorized", reason: "source_authorization", + authorityReason: mode === "source_app_changed" ? "app_binding" : "destination_binding", + } }); + expect((await readAgentWakeupRequestsById(saved.id))[0]).toMatchObject({ status: "skipped", runId: null }); + expect((await readHeartbeatRunsById(context.runId))[0]).toMatchObject({ status: "failed" }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual(previousPublications); + return; + } + if (mode.startsWith("committed_succeeded") || mode.startsWith("committed_consumed") || mode === "malformed_previous" || mode === "committed_unknown_publication" || mode === "committed_pending_publication" || mode === "committed_mixed_publication") { + await expect(requestTaskRetry(request)).rejects.toMatchObject({ details: { + code: "chat_failed_run_retry_not_authorized", + ...(mode.endsWith("publication") ? { reason: mode === "committed_consumed_publication" || mode === "committed_succeeded_publication" + ? "accepted_result_without_new_input" : "prior_result_publication" } : {}), + } }); + expect(await unadmitted()).toHaveLength(1); + return; + } + const receipt = await requestTaskRetry(request); + expect(receipt).toMatchObject({ accepted: true }); + const [retry] = await db.select().from(chatActions).where(eq(chatActions.id, receipt.chatActionId!)); + expect(retry.payload).toMatchObject({ commentIds: expectedComments }); + expect(await db.select({ id: chatActions.id }).from(chatActions).where(and(eq(chatActions.id, retry.id), dueChatAction("failed_run_retry")))).toHaveLength(1); + expect((await readAgentWakeupRequestsById(saved.id))[0]).toMatchObject({ status: "skipped", runId: null }); + expect((await readHeartbeatRunsById(context.runId))[0]).toMatchObject({ status: "failed" }); + if (mode === "source_claimed") await db.update(agentWakeupRequests).set({ status: "queued", runId: context.runId }) + .where(eq(agentWakeupRequests.id, saved.id)); + if (mode === "source_edited") await db.update(issueComments).set({ body: "Edited after Retry" }) + .where(eq(issueComments.id, String(saved.payload.commentId))); + if (mode === "endpoint_paused") await db.update(chatEndpoints).set({ status: "paused" }) + .where(eq(chatEndpoints.id, context.endpoint.id)); + if (mode === "principal_revoked") await db.update(companyMemberships).set({ status: "suspended" }) + .where(and(eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalId, "owner-user"))); + if (realBudgetIngress || mode === "real_queue" || mode === "unreconciled" || mode.startsWith("committed_")) { + // Capacity prevents a real provider start while exercising production + // admission, persisted source identity, and queue ownership. + await db.update(agents).set({ adapterType: "paperclip_runner", status: "running", + runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }) + .where(eq(agents.id, context.fixture.assignedAgentId)); + await db.insert(heartbeatRuns).values({ companyId, agentId: context.fixture.assignedAgentId, + status: "running", startedAt: new Date() }); + context.wakeup.mockImplementation(heartbeatService(db).wakeup); + } else if (mode !== "still_budget_blocked") context.wakeup.mockImplementation(async () => ({ accepted: true })); + if (mode === "source_edited" || mode === "source_claimed" || mode === "endpoint_paused" || mode === "principal_revoked") { + expect(await context.service.processFailedChatRunRetry(receipt.chatActionId!)).toMatchObject({ status: "failed" }); + expect(await readAgentWakeupRequestsById(receipt.chatActionId!)).toHaveLength(0); + } else { + await context.service.processFailedChatRunRetry(receipt.chatActionId!); + const [admitted] = await readAgentWakeupRequestsById(receipt.chatActionId!); + expect(admitted, JSON.stringify({ action: (await db.select().from(chatActions).where(eq(chatActions.id, receipt.chatActionId!)))[0], calls: context.wakeup.mock.calls.length })).toMatchObject({ requestedByActorId: "owner-user", + payload: { wakeCommentId: saved.payload.commentId }, + status: mode === "still_budget_blocked" ? "skipped" : held ? "deferred_issue_execution" : "queued" }); + expect(context.wakeup.mock.calls.at(-1)![1].contextSnapshot).toMatchObject({ + source: `chat:${provider}`, wakeCommentId: saved.payload.commentId, + }); + + if (executionRevalidation) { + const heartbeat = heartbeatService(db); + const originalAuthorize = durableChatWakeup.authorizeFailedChatRunRetryWake; + const verified: string[] = []; + const identitySources: Array<{ runId: string; messageId: string | null }> = []; + const realInitialize = runIdentityModule.initializeRunIdentity; + const identityProof = vi.spyOn(runIdentityModule, "initializeRunIdentity").mockImplementation(async (database, input) => { + const identity = await realInitialize(database, input); + if (input.companyId === companyId && mode === "legacy_execution" && verified.length > 1) { + identitySources.push({ runId: input.runId, messageId: identity.messageId }); + // Exercise the actual initializer after the real Retry execution + // authority, without allocating a native provider for this probe. + throw new Error("certified_retry_identity_probe_complete"); + } + return identity; + }); + const executionProof = vi.spyOn(durableChatWakeup, "authorizeFailedChatRunRetryWake").mockImplementation(async (...args) => { + const input = args[2]; + const authorized = await originalAuthorize(...args); + if (input.companyId === companyId && input.phase === "execution") { + expect(authorized).toBe(true); + if (!verified.length && mode === "legacy_execution") { + // Exercise the actual dispatch authorization transaction after + // real admission; restore each contradiction before the next. + const tx = args[1]; + for (const changed of [{ runId: context.runId }, { companyId: randomUUID() }, + { agentId: randomUUID() }, { issueId: randomUUID() }]) { + await expect(originalAuthorize(args[0], tx, { ...input, ...changed })).rejects.toThrow(); + } + const [current] = await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId!)); + const [task] = await tx.select().from(issues).where(eq(issues.id, context.issue.id)); + const [wake] = await tx.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, input.wakeupRequestId!)); + const [coordinator] = await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, context.runId)); + const [nativeResult] = await tx.select().from(nativeRunResults).where(eq(nativeRunResults.runId, context.runId)); + const [lease] = await tx.select().from(environmentLeases).where(eq(environmentLeases.heartbeatRunId, context.runId)); + const [sourceComment] = await tx.select().from(issueComments).where(eq(issueComments.id, saved.payload.commentId as string)); + const [endpoint] = await tx.select().from(chatEndpoints).where(eq(chatEndpoints.id, context.endpoint.id)); + const resolved = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, companyId), + eq(issueRecoveryActions.sourceIssueId, context.issue.id), eq(issueRecoveryActions.status, "resolved"))); + const deny = async (mutate: () => Promise, restore: () => Promise) => { + await mutate(); + try { await expect(originalAuthorize(...args)).rejects.toThrow(); } + finally { await restore(); } + }; + await deny(() => tx.update(heartbeatRuns).set({ contextSnapshot: { ...current.contextSnapshot, issueId: randomUUID() } }).where(eq(heartbeatRuns.id, current.id)), + () => tx.update(heartbeatRuns).set({ contextSnapshot: current.contextSnapshot }).where(eq(heartbeatRuns.id, current.id))); + await deny(() => tx.update(heartbeatRuns).set({ contextSnapshot: { ...current.contextSnapshot, explicitUserContinuation: { + ...current.contextSnapshot!.explicitUserContinuation as object, commentId: randomUUID() } } }).where(eq(heartbeatRuns.id, current.id)), + () => tx.update(heartbeatRuns).set({ contextSnapshot: current.contextSnapshot }).where(eq(heartbeatRuns.id, current.id))); + await deny(() => tx.update(agentWakeupRequests).set({ runId: context.runId }).where(eq(agentWakeupRequests.id, wake.id)), + () => tx.update(agentWakeupRequests).set({ runId: wake.runId }).where(eq(agentWakeupRequests.id, wake.id))); + await deny(() => tx.update(issues).set({ executionRunId: context.runId }).where(eq(issues.id, task.id)), + () => tx.update(issues).set({ executionRunId: task.executionRunId }).where(eq(issues.id, task.id))); + await deny(() => tx.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, task.id)), + () => tx.update(issues).set({ assigneeAgentId: task.assigneeAgentId }).where(eq(issues.id, task.id))); + await deny(() => tx.update(issueComments).set({ body: "Changed authorized message" }).where(eq(issueComments.id, sourceComment.id)), + () => tx.update(issueComments).set({ body: sourceComment.body, updatedAt: sourceComment.updatedAt }).where(eq(issueComments.id, sourceComment.id))); + await deny(() => tx.update(chatEndpoints).set({ status: "paused" }).where(eq(chatEndpoints.id, endpoint.id)), + () => tx.update(chatEndpoints).set({ status: endpoint.status }).where(eq(chatEndpoints.id, endpoint.id))); + await deny(() => tx.update(agents).set({ status: "paused" }).where(eq(agents.id, input.agentId)), + () => tx.update(agents).set({ status: "running" }).where(eq(agents.id, input.agentId))); + await deny(() => tx.update(nativeRunFinalizations).set({ leaseOwner: "still-running" }).where(eq(nativeRunFinalizations.runId, coordinator.runId)), + () => tx.update(nativeRunFinalizations).set({ leaseOwner: coordinator.leaseOwner }).where(eq(nativeRunFinalizations.runId, coordinator.runId))); + await deny(async () => { adapterExecutionControls.set(context.runId, createAdapterExecutionControl()); }, + async () => { adapterExecutionControls.get(context.runId)?.finish(); adapterExecutionControls.delete(context.runId); }); + await deny(() => tx.update(agentWakeupRequests).set({ status: "running" }).where(eq(agentWakeupRequests.id, wake.id)), + () => tx.update(agentWakeupRequests).set({ status: wake.status }).where(eq(agentWakeupRequests.id, wake.id))); + await deny(() => tx.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60000) }).where(eq(heartbeatRuns.id, context.runId)), + () => tx.update(heartbeatRuns).set({ controllerLeaseExpiresAt: null }).where(eq(heartbeatRuns.id, context.runId))); + await deny(() => tx.update(environmentLeases).set({ metadata: {} }).where(eq(environmentLeases.id, lease.id)), + () => tx.update(environmentLeases).set({ metadata: lease.metadata }).where(eq(environmentLeases.id, lease.id))); + await deny(() => tx.update(nativeRunResults).set({ resultJson: { terminal: { runTerminalState: "succeeded" } } }).where(eq(nativeRunResults.id, nativeResult.id)), + () => tx.update(nativeRunResults).set({ resultJson: nativeResult.resultJson }).where(eq(nativeRunResults.id, nativeResult.id))); + for (const action of resolved) await deny(() => tx.update(issueRecoveryActions).set({ evidence: {} }).where(eq(issueRecoveryActions.id, action.id)), + () => tx.update(issueRecoveryActions).set({ evidence: action.evidence }).where(eq(issueRecoveryActions.id, action.id))); + } + verified.push(input.runId!); + if (mode === "legacy_execution" && verified.length > 1) return authorized; + // Stop after the real execution authority boundary. This produces + // the exact never-started bootstrap shape without a provider call. + throw conflict("The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state.", + { code: "chat_failed_run_retry_not_authorized" }); + } + return authorized; + }); + try { + await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and( + eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.status, "running"), isNull(heartbeatRuns.wakeupRequestId))); + await db.update(agents).set({ status: "idle" }).where(eq(agents.id, context.fixture.assignedAgentId)); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + const [firstAttempt] = await readHeartbeatRunsById(admitted.runId!); + expect(firstAttempt).toMatchObject({ status: "failed", errorCode: "chat_failed_run_retry_not_authorized", + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } }); + expect(verified).toEqual([firstAttempt.id]); + if (mode === "legacy_execution_log") await db.update(heartbeatRuns).set({ logBytes: 1 }).where(eq(heartbeatRuns.id, firstAttempt.id)); + if (mode === "legacy_execution_profile") await db.update(heartbeatRuns).set({ runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" }, unexpectedExecution: true } }).where(eq(heartbeatRuns.id, firstAttempt.id)); + if (mode === "legacy_execution_controller") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60000) }).where(eq(heartbeatRuns.id, firstAttempt.id)); + if (mode === "legacy_execution_event") await db.update(heartbeatRunEvents).set({ message: "different diagnostic" }).where(and(eq(heartbeatRunEvents.runId, firstAttempt.id), eq(heartbeatRunEvents.seq, 1))); + if (mode === "legacy_execution_intent") await db.update(agentWakeupRequests).set({ runId: context.runId }).where(eq(agentWakeupRequests.id, receipt.requestId)); + if (mode === "legacy_execution_lease") await db.insert(environmentLeases).values({ companyId, heartbeatRunId: firstAttempt.id, + issueId: context.issue.id, provider: "daytona", providerLeaseId: "unresolved-bootstrap-lease", status: "active", leasePolicy: "ephemeral" }); + const next = await requestTaskRetry({ ...request, observedRunId: firstAttempt.id, idempotencyKey: randomUUID() }); + expect(next).toMatchObject({ accepted: true }); + await context.service.processFailedChatRunRetry(next.chatActionId!); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + const [nextWake] = await readAgentWakeupRequestsById(next.chatActionId!); + if (mode !== "legacy_execution") { + expect(nextWake).toBeUndefined(); + expect(verified).toEqual([firstAttempt.id]); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual(previousPublications); + return; + } + expect(nextWake.runId).toBeTruthy(); + expect(nextWake.runId).not.toBe(firstAttempt.id); + const [nextRun] = await readHeartbeatRunsById(nextWake.runId!); + expect(nextRun).toMatchObject({ retryOfRunId: firstAttempt.id, wakeupRequestId: next.chatActionId, + contextSnapshot: { source: "chat:github", wakeCommentId: saved.payload.commentId, + wakeCommentIds: [saved.payload.commentId] } }); + expect(verified).toEqual([firstAttempt.id, nextRun.id]); + expect(identitySources).toEqual([{ runId: nextRun.id, messageId: saved.payload.commentId }]); + await expect(chatRunSourceMessageId(db, nextRun, { provider: "github" })).resolves.toBe(saved.payload.commentId); + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.contextSnapshot?.wakeCommentId === saved.payload.commentId)) + .toHaveLength(2); + expect((await readHeartbeatRunsById(context.runId))[0]).toMatchObject({ status: "failed" }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual(previousPublications); + return; + } finally { executionProof.mockRestore(); identityProof.mockRestore(); await heartbeat.drainActiveRunExecutions(); } + } + await context.service.processFailedChatRunRetry(receipt.chatActionId!); + expect(await unadmitted()).toHaveLength(mode === "still_budget_blocked" ? 1 : 0); + if (held) { + expect(admitted.runId).toBeNull(); + await reconcileTaskChatRetryReceipts(db); + await context.service.processFailedChatRunRetry(receipt.chatActionId!); + expect((await readAgentWakeupRequestsById(receipt.chatActionId!))[0]).toMatchObject({ + status: "deferred_issue_execution", runId: null }); + expect((await db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, context.issue.id))))) + .toEqual(expect.arrayContaining([expect.objectContaining({ status: "active" })])); + if (mode === "committed_delayed_cleanup") { + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.heartbeatRunId, context.runId)); + await db.update(environmentLeases).set({ metadata: { ...lease.metadata, + remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }) }, + }).where(eq(environmentLeases.id, lease.id)); + await reconcileTaskChatRetryReceipts(db); + await context.service.processFailedChatRunRetry(receipt.chatActionId!); + const [resumed] = await readAgentWakeupRequestsById(receipt.chatActionId!); + expect(resumed).toMatchObject({ id: admitted.id, status: "queued" }); + expect(resumed.runId).toBeTruthy(); + expect((await readHeartbeatRunsById(resumed.runId!))[0]).toMatchObject({ status: "queued", + contextSnapshot: { wakeCommentId: saved.payload.commentId } }); + await context.service.processFailedChatRunRetry(receipt.chatActionId!); + await reconcileTaskChatRetryReceipts(db); + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => + run.contextSnapshot?.taskRetryRequestId === receipt.requestId)).toHaveLength(1); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, context.runId)))[0]) + .toMatchObject({ phase: "committed" }); + } + } + if (mode === "real_queue" || mode === "committed_failure" || mode === "committed_failed_publication") { + expect(admitted.runId).toBeTruthy(); + expect((await readHeartbeatRunsById(admitted.runId!))[0]).toMatchObject({ status: "queued", + contextSnapshot: { source: `chat:${provider}`, wakeCommentId: saved.payload.commentId } }); + if (provider === "github") expect(await resolveRunScopedMentionedSkillKeys({ db, companyId, + issueId: context.issue.id, commentIds: [String(saved.payload.commentId)] })).toEqual([skill.key]); + await reconcileTaskChatRetryReceipts(db); + expect((await readAgentWakeupRequestsById(receipt.requestId!))[0]).toMatchObject({ + status: "coalesced", runId: admitted.runId }); + const calls = context.wakeup.mock.calls.length; + await db.update(chatActions).set({ status: "issued", result: null, updatedAt: new Date() }) + .where(eq(chatActions.id, receipt.chatActionId!)); + await context.service.processFailedChatRunRetry(receipt.chatActionId!); + expect(context.wakeup.mock.calls).toHaveLength(calls); + expect((await db.select().from(chatActions).where(eq(chatActions.id, receipt.chatActionId!)))[0]) + .toMatchObject({ status: "processed" }); + if (mode === "committed_failure") { + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, context.runId)))[0]) + .toMatchObject({ phase: "committed" }); + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, context.runId)))[0]) + .toMatchObject({ schemaStatus: "accepted", resultJson: { terminal: { runTerminalState: "failed" } } }); + } + } + } + expect((await readAgentWakeupRequestsById(saved.id))[0]).toMatchObject(mode === "source_claimed" + ? { status: "queued", runId: context.runId } : { status: "skipped", runId: null }); + expect((await readHeartbeatRunsById(context.runId))[0]).toMatchObject({ status: "failed" }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual(previousPublications); + } finally { await context.service.shutdown(); } + }); + it.each([ "settled", "distinct_account", @@ -62777,8 +63687,26 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { fields?: Record, ) => { if (!injected && fields?.status === agents.status) { - injected = true; - throw new Error("temporary source read unavailable"); + const selection = select(fields as Parameters[0]); + const from = selection.from.bind(selection); + vi.spyOn(selection, "from").mockImplementation((table) => { + const query = from(table); + const where = query.where.bind(query); + vi.spyOn(query, "where").mockImplementation((condition) => { + const scoped = where(condition); + const { params } = scoped.toSQL(); + // Maintenance may inspect another task's queued notice first. + // Fail only this final's own source-authorization read. + if (!injected && params.includes(context.fixture.companyId) && + params.includes(context.fixture.assignedAgentId)) { + injected = true; + throw new Error("temporary source read unavailable"); + } + return scoped; + }); + return query; + }); + return selection; } return fields ? select(fields as Parameters[0]) @@ -63379,6 +64307,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { linkedOriginalActor = false, githubUnavailableFile = false, upgradedGitHub = true, + currentGitHubPr = false, ) { const fixture = await seedCompany(); const configured = @@ -63419,8 +64348,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { }) : provider === "github" ? makeThread({ - channelId: "github:paperclipai/paperclip", - id: `github:paperclipai/paperclip:issue:${700 + Number(suffix)}`, + channelId: currentGitHubPr ? "paperclipai/paperclip" : "github:paperclipai/paperclip", + id: currentGitHubPr ? `github:paperclipai/paperclip:${700 + Number(suffix)}` : `github:paperclipai/paperclip:issue:${700 + Number(suffix)}`, name: "paperclipai/paperclip", }) : provider === "discord" @@ -63472,6 +64401,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { externalId: originalActorId, }); } + if (provider === "github" && currentGitHubPr) { + const config = await githubChatManagementService(db).configuration(endpoint.id, "owner-user"); + await db.insert(chatGitHubConfigurations).values({ companyId: fixture.companyId, endpointId: endpoint.id, + configuration: { ...config.configuration, toolsEnabled: true, defaults: { ...config.configuration.defaults, + invocation: "mentions_only", events: [] } }, updatedByUserId: "owner-user" }); + } const sourceMessage = makeMessage({ id: messageId, mentioned: !thread.thread.isDM, @@ -63482,6 +64417,9 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { externalActorId ?? (provider === "telegram" ? telegramChatId : "U-SAFE-PROGRESS"), }); + if (currentGitHubPr) sourceMessage.raw = { action: "created", installation: { id: 2468 }, + repository: { id: 97531, full_name: "paperclipai/paperclip" }, issue: { number: 791, pull_request: {} }, + sender: { id: 42, login: "octocat", type: "User" }, comment: { id: Number(messageId), body: sourceMessage.text, user: { id: 42 } } }; if (githubUnavailableFile) { const url = "https://github.com/user-attachments/files/31967808/private-current.txt"; diff --git a/server/src/services/chat-channels/failed-run-retry.ts b/server/src/services/chat-channels/failed-run-retry.ts index c18a15bdcb8..0792b140497 100644 --- a/server/src/services/chat-channels/failed-run-retry.ts +++ b/server/src/services/chat-channels/failed-run-retry.ts @@ -1,5 +1,8 @@ +import { safeChatWakeAuthorityReason, type ChatWakeAuthorityReason } from "./inbound/wakeup-authorization-error.js"; +import { adapterExecutionControls } from "../adapter-execution-control.js"; +import { admitExplicitNativeContinuation } from "../explicit-native-continuation.js"; import { continuingTeamsFilePublication, type ContinuingTeamsFileTransfer } from "./microsoft-teams/file-transfer-source.js"; -import { readTaskChatRetryIntent, taskChatRetryCanAwaitRecovery, taskChatRetryExecutionStopped } from "./task-retry-authority.js"; +import { readTaskChatRetryIntent, taskChatRetryCanAwaitRecovery, taskChatRetryExecutionStopped, taskChatRetryAdmittedContinuationStopped } from "./task-retry-authority.js"; import { notifyChatActionWork, notifyChatPublicationWork } from "../chat-work-notifications.js"; import { FAILED_RUN_RETRY_STALE_MS, dueChatAction } from "../chat-action-work.js"; import { readChatAuthorizationRun } from "./authorization-run-read.js"; @@ -125,10 +128,13 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti } = options; - const failedChatRetryDenied = () => + const failedChatRetryDenied = (reason?: "accepted_result_without_new_input" | + "prior_result_publication" | "prior_interaction" | "source_authorization", authorityReason?: ChatWakeAuthorityReason) => conflict( - "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state.", - { code: "chat_failed_run_retry_not_authorized" }, + reason === "accepted_result_without_new_input" + ? "This attempt already recorded a result. Send a new message on the task to continue from the saved work." + : "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state.", + { code: "chat_failed_run_retry_not_authorized", ...(reason ? { reason } : {}), ...(reason === "source_authorization" && authorityReason ? { authorityReason } : {}) }, ); class FailedChatRetryPublicationReadError extends Error { constructor() { @@ -197,24 +203,48 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti } } - /** The reviewed-chat ownership check runs before runtime/profile resolution - * or provider startup. Recognize only that exact persisted failure together - * with its sole system diagnostic and absence of execution evidence. A - * generic setup error (or a prior-session display hint) is not this proof. */ - async function isPreProviderReviewedChatFailure( + /** Certify known preparation boundaries from their server-owned failure + * contract and absence of execution evidence. Other workspace validations + * may run after acquisition and must not inherit this exception. A saved + * session display hint never establishes containment or resume authority. */ + async function isPreProviderChatFailure( tx: DbOrTransaction, run: typeof heartbeatRuns.$inferSelect, + allowWorkspacePreparation = false, ): Promise { - const diagnostic = "reviewed_chat_execution_binding_not_authorized"; + const retryDiagnostic = "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state."; + const failedRetry = run.errorCode === "chat_failed_run_retry_not_authorized" && run.error === retryDiagnostic; + const validation = run.resultJson?.workspaceValidation as Record | undefined; + const recovery = run.resultJson?.executionRecovery as Record | undefined; + const workspacePreparation = allowWorkspacePreparation && run.errorCode === "workspace_validation_failed" && + validation?.reason === "isolated_task_directory_binding_mismatch" && + typeof validation.issueId === "string" && validation.issueId === run.contextSnapshot?.issueId && + (run.issueId === null || run.issueId === validation.issueId) && + (run.contextSnapshot?.taskId === undefined || run.contextSnapshot.taskId === validation.issueId) && + Object.keys(validation).every(key => ["reason", "issueId"].includes(key)) && + recovery?.kind === "bootstrap" && recovery.providerWorkStarted === false && + Object.keys(recovery).every(key => ["kind", "providerWorkStarted"].includes(key)) && + run.resultJson?.stopReason === "adapter_failed" && + Object.keys(run.resultJson).every(key => ["executionRecovery", "workspaceValidation", "stopReason", + "effectiveTimeoutSec", "timeoutConfigured", "timeoutFired", "timeoutSource"].includes(key)); + const diagnostic = failedRetry ? retryDiagnostic : workspacePreparation ? run.error : "reviewed_chat_execution_binding_not_authorized"; + const dispatchOnlyProfile = failedRetry || workspacePreparation; if ( + adapterExecutionControls.has(run.id) || + (run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > new Date()) || run.runtimeMode !== "legacy" || - run.errorCode !== "setup_failed" || - run.error !== diagnostic || + (!failedRetry && !workspacePreparation && run.errorCode !== "setup_failed") || + run.status !== "failed" || !run.finishedAt || + (failedRetry && run.sessionIdBefore !== null) || + (dispatchOnlyProfile && JSON.stringify(run.runnerProfileJson) !== JSON.stringify({ adapterDispatch: { adapterType: "paperclip_runner" } })) || + !diagnostic || run.error !== diagnostic || + (workspacePreparation && run.usageJson !== null && + Object.keys(run.usageJson).some(key => key !== "ledgerScope")) || [ run.runtimeModeResolverVersion, run.runtimeModeReason, run.runtimeModeResolvedAt, - run.runnerProfileJson, + ...(dispatchOnlyProfile ? [] : [run.runnerProfileJson]), run.runnerInstanceId, run.nativeSessionId, run.nativeIssueId, @@ -237,7 +267,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti run.lastOutputAt, run.lastOutputStream, run.lastOutputBytes, - run.usageJson, + ...(workspacePreparation ? [] : [run.usageJson]), run.exitCode, run.signal, ].some((value) => value !== null) || @@ -254,14 +284,52 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti eq(heartbeatRunEvents.runId, run.id), ), ) - .limit(2) + .orderBy(asc(heartbeatRunEvents.seq)) + .limit(workspacePreparation ? 4 : 3) .for("share", { noWait: true }); - const event = events[0]; + let preparationOffset = 0; + const warning = events[0]; + if (workspacePreparation && warning?.eventType === "agent_observability.unavailable") { + // This known warning is written by configuration preparation, before + // workspace allocation. It is not evidence that a provider executed. + if (warning.seq !== 1 || warning.agentId !== run.agentId || warning.stream !== "system" || warning.level !== "warn" || + warning.message !== "Company observability was not applied: this harness is not supported." || + warning.payload?.reason !== "unsupported_harness" || (typeof warning.payload.revision !== "string" || !isUuidLike(warning.payload.revision)) || + Object.keys(warning.payload).some(key => !["reason", "revision"].includes(key)) || + warning.sourceInstanceId !== null || warning.sourceEventId !== null || warning.sourceSeq !== null || + warning.sourcePayloadSha256 !== null || warning.protocolSchemaVersion !== null) return false; + preparationOffset = 1; + } + const event = events[preparationOffset]; + const measured = events[preparationOffset + 1]; + if (failedRetry) { + const explicit = run.contextSnapshot?.explicitUserContinuation as Record | undefined; + const hint = run.contextSnapshot?.chatFailedRunRetry as Record | undefined; + const intentId = run.contextSnapshot?.taskRetryRequestId; + if (typeof intentId !== "string" || !run.retryOfRunId || !run.wakeupRequestId || + hint?.version !== 1 || hint.actionId !== run.wakeupRequestId || hint.failedRunId !== run.retryOfRunId || + explicit?.previousRunId !== run.retryOfRunId || explicit.taskRetryRequestId !== intentId || + explicit.commentId !== run.contextSnapshot?.wakeCommentId) return false; + const prior = await readTaskChatRetryIntent(tx as unknown as Db, { companyId: run.companyId, + issueId: String(run.contextSnapshot?.issueId), agentId: run.agentId, + sourceRunId: run.retryOfRunId, intentId, chatActionId: run.wakeupRequestId }); + if (!prior || prior.wake.status !== "coalesced" || prior.wake.runId !== run.id) return false; + } + if (failedRetry || workspacePreparation) { + if (!measured || measured.seq !== 2 + preparationOffset || measured.agentId !== run.agentId || + measured.eventType !== "lifecycle" || measured.stream !== "system" || + measured.message !== "task.run.measured" || measured.sourceInstanceId !== null || + measured.sourceEventId !== null || measured.sourceSeq !== null || measured.sourcePayloadSha256 !== null || + measured.protocolSchemaVersion !== null || typeof measured.payload?.durationMs !== "number" || + !Number.isFinite(measured.payload.durationMs) || measured.payload.durationMs < 0 || + measured.payload?.durableSettlement !== true || measured.payload?.outcome !== "failed" || + measured.payload?.phase !== "task.run.measured") return false; + } if ( - events.length !== 1 || + events.length !== (failedRetry || workspacePreparation ? 2 + preparationOffset : 1) || !event || event.agentId !== run.agentId || - event.seq !== 1 || + event.seq !== 1 + preparationOffset || event.eventType !== "error" || event.stream !== "system" || event.level !== "error" || @@ -553,10 +621,11 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti async function assertFailedNativeRetryState( tx: DbOrTransaction, - source: Pick, + source: Pick, companyId: string, taskRetryIntentId?: string, allowPendingRecovery = false, + executionRunId?: string, ) { const [run] = await tx .select() @@ -573,7 +642,25 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti if (taskRetryIntentId) { if (!await readTaskChatRetryIntent(tx as unknown as Db, { companyId, issueId: source.issueId, agentId: source.agentId, sourceRunId: run.id, intentId: taskRetryIntentId })) throw failedChatRetryDenied(); - if (await taskChatRetryExecutionStopped(tx as unknown as Db, run)) return; + const previousComments = new Set(source.taskRetryInput?.sourceActionIds.length + ? retryCommentIds(run.contextSnapshot ?? {}, source.taskRetryInput) : []); + const newInput = source.taskRetryInput?.sourceActionIds.length && source.requestedByActorType === "user" && + source.commentIds.length && source.commentIds.every(id => !previousComments.has(id)); + if (newInput) { + // Persisting independent input may wait behind an unknown executor. + // Promotion still requires the ordinary new-user-turn stop proof. + if (allowPendingRecovery) return; + if (executionRunId && await taskChatRetryAdmittedContinuationStopped(tx as unknown as Db, run, { + issueId: source.issueId, intentId: taskRetryIntentId, successorRunId: executionRunId, + commentId: source.commentIds.at(-1)!, + })) return; + if (await admitExplicitNativeContinuation({ db: tx as Db, companyId, issueId: source.issueId, + agentId: source.agentId, actorType: "user", actorId: source.requestedByActorId, + reason: "issue_commented", commentId: source.commentIds.at(-1)!, successorRunId: taskRetryIntentId, + dryRun: true })) return; + } + if (await taskChatRetryExecutionStopped(tx as unknown as Db, run) || + await isPreProviderChatFailure(tx, run, true)) return; if (allowPendingRecovery && await taskChatRetryCanAwaitRecovery( tx as unknown as Db, run, source.issueId, taskRetryIntentId, )) return; @@ -719,7 +806,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti "process_exit", "timeout", ].includes(failedRun.errorCode ?? "") || - (evidence.kind === "retry" && await isPreProviderReviewedChatFailure(tx, evidence.run)) + (evidence.kind === "retry" && await isPreProviderChatFailure(tx, evidence.run)) )) || (!input.publication && ["backlog", "done", "cancelled"].includes(issue.status)) @@ -732,6 +819,22 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti if (failedRun.runtimeMode === "native" && evidence.kind === "retry" && !input.taskRetryIntentId) await failedNativeRetryCoordinator(tx, evidence.run, issue.id); const context = failedRun.contextSnapshot ?? {}; + const selection = input.taskRetryInput && taskRetryInputSchema.safeParse(input.taskRetryInput); + if (selection && !selection.success) throw failedChatRetryDenied(); + const selectedInput = selection?.success ? selection.data : undefined; + if (input.taskRetryIntentId && !input.committedResponse && failedRun.status === "failed" && + failedRun.runtimeMode === "native" && failedRun.nativeIssueId === issue.id && + context.issueId === issue.id && context.source === "issue.tree_resume" && + !selectedInput?.sourceActionIds.length) { + // An inherited requester is not permission to replay a recorded result. + // Explain the existing refusal before raw chat-source validation hides + // the supported new-message path. This branch grants no continuation. + const [recorded] = await tx.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and( + eq(nativeRunResults.companyId, input.companyId), eq(nativeRunResults.issueId, issue.id), + eq(nativeRunResults.runId, failedRun.id), eq(nativeRunResults.schemaStatus, "accepted"), + )).limit(1).for("share", { noWait: true }); + if (recorded) throw failedChatRetryDenied("accepted_result_without_new_input"); + } const provider = CHAT_PROVIDERS.find( (value) => context.source === `chat:${value}`, ); @@ -742,13 +845,11 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti (context.taskKey !== issue.identifier && context.taskKey !== issue.id) ) throw failedChatRetryDenied(); - const selection = input.taskRetryInput && taskRetryInputSchema.safeParse(input.taskRetryInput); - if (selection && !selection.success) throw failedChatRetryDenied(); - const selectedInput = selection?.success ? selection.data : undefined; if (selectedInput && (!input.taskRetryIntentId || provider !== "github")) throw failedChatRetryDenied(); const reauthorization = input.taskRetryReauthorization ?? (selectedInput ? { intentId: input.taskRetryIntentId!, sourceRunId: failedRun.id, runtimeFence: selectedInput.runtimeFence, } : undefined); + if (reauthorization && selectedInput?.skippedActionIds) reauthorization.skippedActionIds = selectedInput.skippedActionIds; const pendingSources = selectedInput?.sourceActionIds.length ? await tx.select().from(chatActions).where(and( eq(chatActions.companyId, input.companyId), eq(chatActions.kind, "inbound_wakeup"), sql`${chatActions.payload}->>'issueId' = ${issue.id}`, @@ -859,6 +960,11 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti ), ) .limit(1); + const previousComments = new Set(pendingSources?.length ? retryCommentIds(context, selectedInput) : []); + const independentFailedInput = input.taskRetryIntentId && pendingSources?.length && + pendingSources.every(action => action.payload.requestedByActorType === "user") && + commentIds.every(id => !previousComments.has(id)) && failedRun.status === "failed" && + (accepted?.resultJson.terminal as Record | undefined)?.runTerminalState === "failed"; const [resultPublication] = await tx .select({ id: chatPublications.id }) .from(chatPublications) @@ -869,6 +975,10 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti input.committedResponse ? eq(chatPublications.state, "delivery_unknown") : undefined, + // A distinct admitted user request does not replay the failed + // result's output. Preserve that terminal publication as history; + // pending or uncertain delivery still owns its external effects. + independentFailedInput ? ne(chatPublications.state, "failed") : undefined, continuingUnknownPublicationId ? ne(chatPublications.id, continuingUnknownPublicationId) : undefined, @@ -884,13 +994,10 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti ), ) .limit(1); - if ( - (!input.committedResponse && accepted) || - resultPublication || - question || - answerDelivery - ) - throw failedChatRetryDenied(); + if (!input.committedResponse && accepted && !independentFailedInput) + throw failedChatRetryDenied(input.taskRetryIntentId ? "accepted_result_without_new_input" : undefined); + if (resultPublication) throw failedChatRetryDenied(input.taskRetryIntentId ? "prior_result_publication" : undefined); + if (question || answerDelivery) throw failedChatRetryDenied(input.taskRetryIntentId ? "prior_interaction" : undefined); if (context.chatFailedRunRetry && !pendingSources) { const hint = context.chatFailedRunRetry as Record; const [priorIntent] = await tx @@ -1069,7 +1176,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti const invalidated = new Set(lifecycleRows.map(row => lifecycleKey(row.target, row.context))); const batch = reauthorization ? await authorizeGitHubTaskRetryBatch(tx, ordered, { nonblocking: true, terminal: input.publication === true, taskRetry: reauthorization, - }).catch(error => { if (isExternalActionAuthorizationChange(error)) throw failedChatRetryDenied(); throw error; }) : null; + }).catch(error => { if (isExternalActionAuthorizationChange(error)) throw failedChatRetryDenied(input.taskRetryIntentId ? "source_authorization" : undefined, safeChatWakeAuthorityReason(error)); throw error; }) : null; const canonicalSources: Array> = []; let destination: Awaited> | null = null; @@ -1080,16 +1187,21 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti ...(reauthorization ? { taskRetry: reauthorization } : {}), }).catch((error) => { if (isExternalActionAuthorizationChange(error)) - throw failedChatRetryDenied(); + throw failedChatRetryDenied(input.taskRetryIntentId ? "source_authorization" : undefined, safeChatWakeAuthorityReason(error)); throw error; }); if (reauthorization && !reauthorization.runtimeFence) reauthorization.runtimeFence = destination.retryRuntimeFence; const delivery = deliveriesById.get(action.deliveryId!); const current = actionsById.get(action.id); const comment = commentsById.get(String(action.payload.commentId)); + const receipt = receiptsById.get(action.id); + const skippedInput = pendingSources && selectedInput?.skippedActionIds?.includes(action.id) && + receipt?.status === "skipped" && !receipt.runId && + ["budget.blocked", "agent.not_invokable"].includes(receipt.reason ?? "") && + action.payload.requestedByActorType === "user"; if ( !current || - current.status !== "processed" || + (current.status !== "processed" && !(skippedInput && current.status === "failed")) || inboundWakeScope(current) !== inboundWakeScope(action) || !delivery || delivery.state !== "processed" || @@ -1106,7 +1218,6 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti ) throw failedChatRetryDenied(); if (invalidated.has(lifecycleKey(delivery.providerEventId, lifecycleRuntimeFence(delivery)))) throw failedChatRetryDenied(); - const receipt = receiptsById.get(action.id); if (!receipt) throw failedChatRetryDenied(); assertFailedRetryReceipt( createDurableChatWakeupRequest({ @@ -1125,7 +1236,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti receipt, ); if (pendingSources) { - const direct = receipt.status === "deferred_issue_execution" && !receipt.runId + const direct = (receipt.status === "deferred_issue_execution" || skippedInput) && !receipt.runId && receipt.payload?.coalescedIntoWakeupRequestId === undefined; const owned = input.retryActionId && receipt.status === "coalesced" && receipt.payload?.coalescedIntoWakeupRequestId === input.retryActionId; @@ -1164,7 +1275,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti owner.agentId !== input.agentId || owner.requestedByActorType !== receipt.requestedByActorType || owner.requestedByActorId !== receipt.requestedByActorId)) || - ["skipped", "cancelled"].includes(receipt.status) + (!skippedInput && ["skipped", "cancelled"].includes(receipt.status)) ) throw failedChatRetryDenied(); canonicalSources.push({ @@ -1457,7 +1568,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti }); await assertFailedNativeRetryState(tx, source, input.companyId, typeof source.intent.payload.taskRetryIntentId === "string" ? source.intent.payload.taskRetryIntentId : undefined, - input.phase === "admission"); + input.phase === "admission", input.phase === "execution" ? input.runId : undefined); const context = input.contextSnapshot; const hint = context.chatFailedRunRetry as | Record diff --git a/server/src/services/chat-channels/github/review-wait.ts b/server/src/services/chat-channels/github/review-wait.ts index e34b3a576c7..9b688564be8 100644 --- a/server/src/services/chat-channels/github/review-wait.ts +++ b/server/src/services/chat-channels/github/review-wait.ts @@ -1,6 +1,8 @@ import { createHash } from "node:crypto"; -import { and, eq, or } from "drizzle-orm"; -import { chatActions, chatGitHubReviews, type Db } from "@paperclipai/db"; +import { HttpError } from "../../../errors.js"; +import { chatRunSourceMessageId } from "../run-origin.js"; +import { and, desc, eq, or, sql } from "drizzle-orm"; +import { chatActions, chatConversations, chatEndpointResources, chatEndpoints, chatGitHubReviews, chatMessageLinks, heartbeatRuns, toolConnections, type Db } from "@paperclipai/db"; export async function hasPublishedGitHubReview(db: Db, binding: { companyId: string; @@ -27,3 +29,69 @@ export async function hasPublishedGitHubReview(db: Db, binding: { )).limit(1); return !!publication; } + +type WaitRun = Pick; + +/** Passive review waits inherit the same accepted requester lineage as GitHub + * tools. This is not a grant to publish, run a monitor, or reuse attachments. */ +export async function readInheritedGitHubReviewWait(db: Db, run: WaitRun) { + if (!run.activeIdentityContextId || !run.nativeIssueId) return null; + let commentId: string | null; + try { + commentId = await chatRunSourceMessageId(db, run, { provider: "github" }); + } catch (error) { + if (error instanceof HttpError && error.status === 403) return { kind: "revoked" as const }; + throw error; + } + if (!commentId) return null; + const [review] = await db.select({ + id: chatGitHubReviews.id, endpointId: chatGitHubReviews.endpointId, + state: chatGitHubReviews.state, repository: chatGitHubReviews.repository, + repositoryId: chatGitHubReviews.repositoryId, headSha: chatGitHubReviews.headSha, + assessment: chatGitHubReviews.assessment, reviewRunId: chatGitHubReviews.runId, + connectionEnabled: toolConnections.enabled, connectionStatus: toolConnections.status, + reviewAgentId: heartbeatRuns.agentId, reviewCompanyId: heartbeatRuns.companyId, + resourceRepository: chatEndpointResources.providerResourceId, + resourceRepositoryId: sql`${chatEndpointResources.metadata}->>'providerRepositoryId'`, + pullNumber: chatGitHubReviews.pullNumber, thread: chatConversations.externalThreadId, + }).from(chatMessageLinks) + .innerJoin(chatConversations, and(eq(chatConversations.id, chatMessageLinks.conversationId), + eq(chatConversations.companyId, run.companyId), eq(chatConversations.issueId, run.nativeIssueId))) + .innerJoin(chatEndpoints, and(eq(chatEndpoints.id, chatConversations.endpointId), + eq(chatEndpoints.companyId, run.companyId), eq(chatEndpoints.provider, "github"), + eq(chatEndpoints.assignedAgentId, run.agentId))) + .leftJoin(toolConnections, and(eq(toolConnections.id, chatEndpoints.connectionId), + eq(toolConnections.companyId, run.companyId))) + .innerJoin(chatEndpointResources, and(eq(chatEndpointResources.id, chatConversations.resourceId), + eq(chatEndpointResources.companyId, run.companyId), eq(chatEndpointResources.endpointId, chatEndpoints.id))) + .innerJoin(chatGitHubReviews, and(eq(chatGitHubReviews.endpointId, chatEndpoints.id), + eq(chatGitHubReviews.companyId, run.companyId), eq(chatGitHubReviews.issueId, run.nativeIssueId))) + .leftJoin(heartbeatRuns, eq(heartbeatRuns.id, chatGitHubReviews.runId)) + .where(and(eq(chatMessageLinks.companyId, run.companyId), eq(chatMessageLinks.commentId, commentId), + eq(chatMessageLinks.direction, "inbound"))) + .orderBy(desc(chatGitHubReviews.createdAt), desc(chatGitHubReviews.id)).limit(1); + // A newer unreviewed head cannot inherit the prior head's publication. + if (!review) return null; + if (!review.connectionEnabled || review.connectionStatus !== "active" || review.state !== "completed" || + review.reviewCompanyId !== run.companyId || review.reviewAgentId !== run.agentId || + review.assessment?.reviewedCommit !== review.headSha || + review.resourceRepository.toLowerCase() !== review.repository.toLowerCase() || + review.resourceRepositoryId !== review.repositoryId || + review.thread.toLowerCase() !== `github:${review.repository}:${review.pullNumber}`.toLowerCase()) return { kind: "revoked" as const }; + const [publication] = await db.select({ id: chatActions.id }).from(chatActions).where(and( + eq(chatActions.companyId, run.companyId), eq(chatActions.endpointId, review.endpointId), + eq(chatActions.kind, "github_review_publication"), eq(chatActions.status, "processed"), + eq(chatActions.providerActionId, `github_publication:${run.nativeIssueId}:assessment:${createHash("sha256").update(JSON.stringify(review.id)).digest("hex")}`), + )).limit(1); + if (!publication) return { kind: "revoked" as const }; + // The caller must still verify live task ownership, destination and principal + // policy. This normalized source is used only by the passive-wait boundary. + return { kind: "published" as const, endpointId: review.endpointId, reviewId: review.id, reviewRunId: review.reviewRunId, + headSha: review.headSha, context: { + source: "chat:github", paperclipHarnessCheckedOut: true, + wakeCommentId: commentId, wakeCommentIds: [commentId], + paperclipWake: { checkedOutByHarness: true, externalChatProvider: "github", + issue: { id: run.nativeIssueId }, commentIds: [commentId] }, + } }; +} diff --git a/server/src/services/chat-channels/github/task-retry-authorization.ts b/server/src/services/chat-channels/github/task-retry-authorization.ts index 52e51f38ebf..d27edc958cf 100644 --- a/server/src/services/chat-channels/github/task-retry-authorization.ts +++ b/server/src/services/chat-channels/github/task-retry-authorization.ts @@ -1,6 +1,6 @@ -import { and, eq, inArray } from "drizzle-orm"; -import { chatActions, chatConversations, chatDeliveries, chatEndpointResources, chatMessageLinks, issueComments, toolConnections } from "@paperclipai/db"; -import { forbidden } from "../../../errors.js"; +import { and, eq, inArray, isNull, lt, sql } from "drizzle-orm"; +import { agentWakeupRequests, chatActions, chatConversations, chatDeliveries, chatEndpointResources, chatMessageLinks, issueComments, toolConnections } from "@paperclipai/db"; +import { chatWakeAuthorityDenied } from "../inbound/wakeup-authorization-error.js"; import type { DbOrTransaction, EndpointRow, LifecycleRuntimeFence } from "../runtime-types.js"; import { credentialFingerprint } from "../credential-fingerprint.js"; import { readTaskChatRetryIntent } from "../task-retry-authority.js"; @@ -9,8 +9,25 @@ export type TaskRetryWakeAuthority = { intentId: string; sourceRunId: string; runtimeFence?: LifecycleRuntimeFence; + skippedActionIds?: string[]; }; +async function skippedTaskRetryInputs(tx: DbOrTransaction, actions: Array, + authority: TaskRetryWakeAuthority) { + const first = actions[0]; + const ids = actions.filter(action => action.status === "failed" && action.payload.requestedByActorType === "user" && + authority.skippedActionIds?.includes(action.id)).map(action => action.id); + if (!first || !ids.length) return new Set(); + const rows = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.companyId, first.companyId), inArray(agentWakeupRequests.id, ids), + eq(agentWakeupRequests.agentId, String(first.payload.agentId)), eq(agentWakeupRequests.status, "skipped"), + isNull(agentWakeupRequests.runId), inArray(agentWakeupRequests.reason, ["budget.blocked", "agent.not_invokable"]), + eq(agentWakeupRequests.requestedByActorType, "user"), + eq(agentWakeupRequests.requestedByActorId, String(first.payload.requestedByActorId)), + )); + return new Set(rows.map(row => row.id)); +} + export function githubTaskRetryAppMatches(endpoint: EndpointRow, delivery: typeof chatDeliveries.$inferSelect, response: { payload: Record } | undefined) { const original = delivery.normalizedEvent.githubAuthority as { botExternalId?: unknown } | undefined; @@ -18,6 +35,37 @@ export function githubTaskRetryAppMatches(endpoint: EndpointRow, delivery: typeo : response?.payload.botExternalId === endpoint.botExternalId && response.payload.agentId === endpoint.assignedAgentId; } +/** Legacy intake did not persist an App ID when admission stopped before + * acknowledgement. An earlier request ledger proves this endpoint had already + * claimed the App: response payloads are server-owned and a non-null endpoint + * claim cannot be replaced by setup, reconnect, or lifecycle repair. This is + * identity evidence only; the earlier response is never replayed. */ +async function githubTaskRetryAppAuthorized(tx: DbOrTransaction, endpoint: EndpointRow, + delivery: typeof chatDeliveries.$inferSelect, response: { payload: Record } | undefined) { + if (githubTaskRetryAppMatches(endpoint, delivery, response)) return true; + const original = delivery.normalizedEvent.githubAuthority as { botExternalId?: unknown } | undefined; + // Never substitute historical evidence for an explicit conflicting binding. + if (original?.botExternalId !== undefined || response || !endpoint.botExternalId || !delivery.conversationId) return false; + const [prior] = await tx.select({ id: chatActions.id }).from(chatActions) + .innerJoin(chatDeliveries, and(eq(chatDeliveries.id, chatActions.deliveryId), + eq(chatDeliveries.companyId, chatActions.companyId), eq(chatDeliveries.endpointId, chatActions.endpointId), + eq(chatDeliveries.conversationId, chatActions.conversationId), eq(chatDeliveries.principalId, chatActions.principalId))) + .innerJoin(chatMessageLinks, and(eq(chatMessageLinks.deliveryId, chatDeliveries.id), + eq(chatMessageLinks.companyId, chatDeliveries.companyId), eq(chatMessageLinks.endpointId, chatDeliveries.endpointId), + eq(chatMessageLinks.conversationId, chatDeliveries.conversationId), eq(chatMessageLinks.direction, "inbound"))) + .where(and(eq(chatActions.companyId, endpoint.companyId), eq(chatActions.endpointId, endpoint.id), + eq(chatActions.conversationId, delivery.conversationId), eq(chatActions.kind, "github_response_comment"), + eq(chatActions.status, "processed"), eq(chatDeliveries.state, "processed"), + eq(chatActions.providerActionId, sql`'github_response:' || ${chatDeliveries.id}::text`), + sql`jsonb_typeof(${chatActions.payload}->'botExternalId') = 'string'`, + sql`${chatActions.payload}->>'botExternalId' = ${endpoint.botExternalId}`, + sql`jsonb_typeof(${chatActions.payload}->'agentId') = 'string'`, + sql`${chatActions.payload}->>'agentId' = ${endpoint.assignedAgentId}`, + lt(chatActions.createdAt, delivery.createdAt), lt(chatDeliveries.createdAt, delivery.createdAt))) + .limit(1); + return Boolean(prior); +} + export function githubTaskRetryDestinationMatches(delivery: typeof chatDeliveries.$inferSelect, conversation: typeof chatConversations.$inferSelect, resource: typeof chatEndpointResources.$inferSelect | null) { const originalResource = delivery.normalizedEvent.resource as { providerResourceId?: unknown } | undefined; @@ -33,27 +81,28 @@ export function githubTaskRetryDestinationMatches(delivery: typeof chatDeliverie export async function githubTaskRetryRuntimeFence(tx: DbOrTransaction, endpoint: EndpointRow, action: typeof chatActions.$inferSelect, delivery: typeof chatDeliveries.$inferSelect, authority: TaskRetryWakeAuthority): Promise { - const deny = () => forbidden("This accepted chat message is no longer authorized to start work", - { code: "chat_action_authorization_changed" }); + const deny = chatWakeAuthorityDenied; const retry = await readTaskChatRetryIntent(tx as Parameters[0], { companyId: action.companyId, issueId: String(action.payload.issueId), agentId: String(action.payload.agentId), sourceRunId: authority.sourceRunId, intentId: authority.intentId, }); - if (!retry || endpoint.provider !== "github" || !endpoint.botExternalId || action.kind !== "inbound_wakeup" - || action.status !== "processed" || delivery.state !== "processed") throw deny(); + const skipped = await skippedTaskRetryInputs(tx, [action], authority); + if (!retry) throw deny("retry_intent"); + if (endpoint.provider !== "github" || !endpoint.botExternalId || action.kind !== "inbound_wakeup" + || (action.status !== "processed" && !skipped.has(action.id)) || delivery.state !== "processed") throw deny("source_receipt"); const [response] = await tx.select({ payload: chatActions.payload }).from(chatActions).where(and( eq(chatActions.companyId, action.companyId), eq(chatActions.endpointId, endpoint.id), eq(chatActions.conversationId, action.conversationId!), eq(chatActions.deliveryId, delivery.id), eq(chatActions.kind, "github_response_comment"), )); - if (!githubTaskRetryAppMatches(endpoint, delivery, response)) throw deny(); + if (!(await githubTaskRetryAppAuthorized(tx, endpoint, delivery, response))) throw deny("app_binding"); const [connection] = await tx.select({ refs: toolConnections.credentialSecretRefs, enabled: toolConnections.enabled, status: toolConnections.status }).from(toolConnections).where(and( eq(toolConnections.companyId, endpoint.companyId), eq(toolConnections.id, endpoint.connectionId), )); - if (!connection?.enabled || connection.status !== "active") throw deny(); + if (!connection?.enabled || connection.status !== "active") throw deny("connection_access"); const generation = Number((endpoint.setup as { runtimeGeneration?: unknown }).runtimeGeneration ?? 0); - if (!Number.isSafeInteger(generation) || generation < 0) throw deny(); + if (!Number.isSafeInteger(generation) || generation < 0) throw deny("runtime_generation"); return authority.runtimeFence ?? { generation, credentialFingerprint: credentialFingerprint(connection.refs ?? []) }; } @@ -71,16 +120,17 @@ export function createGitHubTaskRetryBatchAuthority Promise) { async function authorizeGitHubTaskRetryBatch(tx: DbOrTransaction, actions: Array, notice: { nonblocking: true; terminal: boolean; taskRetry: TaskRetryWakeAuthority }) { - const deny = () => forbidden("This accepted chat message is no longer authorized to start work", - { code: "chat_action_authorization_changed" }); + const deny = chatWakeAuthorityDenied; const first = actions[0]; - if (!first) throw deny(); + if (!first) throw deny("batch_scope"); + const skipped = await skippedTaskRetryInputs(tx, actions, notice.taskRetry); const commonScope = (action: typeof chatActions.$inferSelect) => JSON.stringify([ - action.companyId, action.endpointId, action.conversationId, action.principalId, action.kind, action.status, + action.companyId, action.endpointId, action.conversationId, action.principalId, action.kind, + skipped.has(action.id) ? "processed" : action.status, ...["version", "issueId", "agentId", "sessionGeneration", "requestedByActorType", "requestedByActorId"] .map(key => action.payload[key]), ]); - if (actions.some(action => commonScope(action) !== commonScope(first))) throw deny(); + if (actions.some(action => commonScope(action) !== commonScope(first))) throw deny("batch_scope"); const deliveries = await tx.select().from(chatDeliveries).where(and(eq(chatDeliveries.companyId, first.companyId), inArray(chatDeliveries.id, actions.map(action => action.deliveryId!)))); const byDelivery = new Map(deliveries.map(delivery => [delivery.id, delivery])); @@ -102,21 +152,21 @@ export function createGitHubTaskRetryBatchAuthority(); for (const action of actions) { const delivery = byDelivery.get(action.deliveryId!); - if (!delivery || !linked.has(`${delivery.id}:${action.payload.commentId}`)) throw deny(); + if (!delivery || !linked.has(`${delivery.id}:${action.payload.commentId}`)) throw deny("batch_source_link"); // Reuse only the same live grant within this transaction. Automatic review // policies with different contexts still receive their own full check. const grant = JSON.stringify(delivery.normalizedEvent.githubAutomatic ?? delivery.normalizedEvent.githubIssue ?? null); let current = groups.get(grant); if (!current) { current = await authorize(tx, action, notice); - if (current.endpoint.provider !== "github") throw deny(); + if (current.endpoint.provider !== "github") throw deny("app_binding"); notice.taskRetry.runtimeFence ??= current.retryRuntimeFence; groups.set(grant, current); } if (delivery.state !== "processed" || delivery.endpointId !== current.endpoint.id || delivery.conversationId !== current.conversation.id || delivery.principalId !== action.principalId || - !githubTaskRetryAppMatches(current.endpoint, delivery, legacyByDelivery.get(delivery.id)) || - !githubTaskRetryDestinationMatches(delivery, current.conversation, current.resource)) throw deny(); + !(await githubTaskRetryAppAuthorized(tx, current.endpoint, delivery, legacyByDelivery.get(delivery.id))) || + !githubTaskRetryDestinationMatches(delivery, current.conversation, current.resource)) throw deny("destination_binding"); authorized.set(action.id, { ...current, delivery }); } return authorized; diff --git a/server/src/services/chat-channels/inbound/wakeup-authorization-error.test.ts b/server/src/services/chat-channels/inbound/wakeup-authorization-error.test.ts new file mode 100644 index 00000000000..2a4d4f2c0ae --- /dev/null +++ b/server/src/services/chat-channels/inbound/wakeup-authorization-error.test.ts @@ -0,0 +1,16 @@ +import { describe, expect, it } from "vitest"; +import { forbidden } from "../../../errors.js"; +import { chatWakeAuthorityDenied, safeChatWakeAuthorityReason } from "./wakeup-authorization-error.js"; + +describe("safe chat wake authority reason", () => { + it("retains only the stable category from a known authority refusal", () => { + expect(safeChatWakeAuthorityReason(chatWakeAuthorityDenied("source_receipt"))).toBe("source_receipt"); + }); + it.each([ + forbidden("private provider response", { code: "chat_action_authorization_changed", authorityReason: "private-source-payload" }), + forbidden("private provider response", { code: "another_error", authorityReason: "source_receipt" }), + { details: { code: "chat_action_authorization_changed", authorityReason: "source_receipt" } }, + ])("does not copy arbitrary error fields or untrusted error shapes", error => { + expect(safeChatWakeAuthorityReason(error)).toBeUndefined(); + }); +}); diff --git a/server/src/services/chat-channels/inbound/wakeup-authorization-error.ts b/server/src/services/chat-channels/inbound/wakeup-authorization-error.ts new file mode 100644 index 00000000000..8edf2730346 --- /dev/null +++ b/server/src/services/chat-channels/inbound/wakeup-authorization-error.ts @@ -0,0 +1,19 @@ +import { forbidden, HttpError } from "../../../errors.js"; + +// Stable categories only. Never surface stored provider payloads, identifiers, +// credential fingerprints, or arbitrary downstream error text to the board. +const reasons = ["source_shape", "task_assignment", "delivery_binding", "runtime_fence", + "conversation_binding", "destination_binding", "automatic_policy", "task_privacy", + "principal_access", "source_comment", "retry_intent", "source_receipt", "app_binding", + "connection_access", "runtime_generation", "batch_scope", "batch_source_link"] as const; +export type ChatWakeAuthorityReason = typeof reasons[number]; +export function chatWakeAuthorityDenied(authorityReason: ChatWakeAuthorityReason) { + return forbidden("This accepted chat message is no longer authorized to start work", + { code: "chat_action_authorization_changed", authorityReason }); +} +export function safeChatWakeAuthorityReason(error: unknown): ChatWakeAuthorityReason | undefined { + if (!(error instanceof HttpError) || !error.details || typeof error.details !== "object") return undefined; + const details = error.details as Record; + return details.code === "chat_action_authorization_changed" && reasons.includes(details.authorityReason as ChatWakeAuthorityReason) + ? details.authorityReason as ChatWakeAuthorityReason : undefined; +} diff --git a/server/src/services/chat-channels/inbound/wakeup-authorization.ts b/server/src/services/chat-channels/inbound/wakeup-authorization.ts index f4da4fa7cce..0d45fcb7606 100644 --- a/server/src/services/chat-channels/inbound/wakeup-authorization.ts +++ b/server/src/services/chat-channels/inbound/wakeup-authorization.ts @@ -1,3 +1,4 @@ +import { chatWakeAuthorityDenied } from "./wakeup-authorization-error.js"; import { nonDirectDestinationAllowed } from "./surfaces.js"; import { lifecycleRuntimeFence } from "./delivery-ledger.js"; import type { DbTransaction, DbOrTransaction } from "../runtime-types.js"; @@ -45,11 +46,7 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions notice?: { nonblocking: true; terminal: boolean; taskRetry?: TaskRetryWakeAuthority }, ) { const payload = action.payload; - const deny = () => - forbidden( - "This accepted chat message is no longer authorized to start work", - { code: "chat_action_authorization_changed" }, - ); + const deny = chatWakeAuthorityDenied; if ( payload.version !== 1 || typeof payload.issueId !== "string" || @@ -61,7 +58,7 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions !["user", "system"].includes(String(payload.requestedByActorType)) || typeof payload.requestedByActorId !== "string" ) - throw deny(); + throw deny("source_shape"); const issue = await tx .select() .from(issues) @@ -80,7 +77,7 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions (!notice?.terminal && ["backlog", "done", "cancelled"].includes(issue.status)) ) - throw deny(); + throw deny("task_assignment"); // The scheduler already owns the issue lock. Do not wait in the opposite // order behind an ingress transaction which owns the endpoint first. const endpoint = await tx @@ -117,13 +114,11 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions delivery.endpointId !== endpoint.id || delivery.conversationId !== action.conversationId || delivery.principalId !== action.principalId || - !currentFence || - !(await runtimeCallbackEndpoint(tx as unknown as DbTransaction, endpoint.id, currentFence, [ - "verifying", - "active", - ])) + !currentFence ) - throw deny(); + throw deny("delivery_binding"); + if (!(await runtimeCallbackEndpoint(tx as unknown as DbTransaction, endpoint.id, currentFence, ["verifying", "active"]))) + throw deny("runtime_fence"); const conversation = await tx .select() .from(chatConversations) @@ -143,7 +138,7 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions conversation.sessionGeneration !== payload.sessionGeneration || !["active", "waiting"].includes(conversation.state) ) - throw deny(); + throw deny("conversation_binding"); const resource = conversation.resourceId ? await tx .select() @@ -163,15 +158,15 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions endpoint, action.principalId, ); - if (notice?.taskRetry && !githubTaskRetryDestinationMatches(delivery, conversation, resource)) throw deny(); + if (notice?.taskRetry && !githubTaskRetryDestinationMatches(delivery, conversation, resource)) throw deny("destination_binding"); if (endpoint.provider === "github") { const automatic = await resolveGitHubWakeAuthority(tx, endpoint, delivery); - if (automatic.kind === "denied") throw deny(); + if (automatic.kind === "denied") throw deny("automatic_policy"); if (automatic.kind === "allowed") authorization.userId = automatic.userId; } if (!(await canActorReadIssuePrivacy(tx, authorization.userId ? { type: "board", userId: authorization.userId } - : { type: "none" }, issue))) throw deny(); + : { type: "none" }, issue))) throw deny("task_privacy"); const expectedUserId = payload.requestedByActorType === "user" ? payload.requestedByActorId @@ -185,7 +180,7 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions (payload.requestedByActorType === "system" && payload.requestedByActorId !== action.principalId) ) - throw deny(); + throw deny("principal_access"); const source = await tx .select({ authorUserId: issueComments.authorUserId, @@ -217,7 +212,7 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions (source.deletedAt && !notice?.terminal) || source.authorUserId !== expectedUserId ) - throw deny(); + throw deny("source_comment"); return { issue, delivery, endpoint, conversation, resource, ...(notice?.taskRetry && currentFence ? { retryRuntimeFence: currentFence } : {}) }; } diff --git a/server/src/services/chat-channels/task-retry-authority.ts b/server/src/services/chat-channels/task-retry-authority.ts index 340f2a67b54..8bb532d9c0e 100644 --- a/server/src/services/chat-channels/task-retry-authority.ts +++ b/server/src/services/chat-channels/task-retry-authority.ts @@ -1,5 +1,6 @@ import { and, eq, sql } from "drizzle-orm"; -import { agentWakeupRequests, environmentLeases, heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { agentWakeupRequests, environmentLeases, heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, issues, type Db } from "@paperclipai/db"; +import { buildExecutionContinuation } from "../execution-continuation.js"; import { readTaskRetryIntent } from "../task-retry.js"; import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, recordedLocalExecutionStopped } from "../native-local-process-stop.js"; import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js"; @@ -30,13 +31,29 @@ export async function readTaskChatRetryIntent(db: Db, input: { /** An unusable checkpoint does not prevent an isolated attempt. The old * executor must still be contained; a terminal database row is insufficient. */ export async function taskChatRetryExecutionStopped(db: Db, run: typeof heartbeatRuns.$inferSelect) { + return taskChatRetryExecutionContained(db, run, false); +} + +async function taskChatRetryExecutionContained(db: Db, run: typeof heartbeatRuns.$inferSelect, admittedFailure: boolean) { if (adapterExecutionControls.has(run.id) || (run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > new Date())) return false; const [coordinator] = await db.select().from(nativeRunFinalizations).where(and( eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id), )).for("share", { noWait: true }); - if (coordinator && (coordinator.leaseOwner || coordinator.resultId || coordinator.failureDetail?.successorRunId || + if (coordinator && (coordinator.leaseOwner || coordinator.failureDetail?.successorRunId || !["terminal_failure", "committed"].includes(coordinator.phase))) return false; + if (coordinator?.resultId) { + if (!admittedFailure || run.runtimeMode !== "native" || run.status !== "failed" || + run.nativePhase !== "committed" || !run.completionContractId || coordinator.phase !== "committed" || + coordinator.issueId !== run.nativeIssueId) return false; + const [failedResult] = await db.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and( + eq(nativeRunResults.companyId, run.companyId), eq(nativeRunResults.runId, run.id), + eq(nativeRunResults.issueId, run.nativeIssueId!), eq(nativeRunResults.id, coordinator.resultId), + eq(nativeRunResults.schemaStatus, "accepted"), eq(nativeRunResults.completionContractId, run.completionContractId), + sql`${nativeRunResults.resultJson}#>>'{terminal,runTerminalState}' = 'failed'`, + )).limit(1); + if (!failedResult) return false; + } const leases = await db.select().from(environmentLeases).where(and( eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id), )).for("share", { noWait: true }); @@ -46,6 +63,42 @@ export async function taskChatRetryExecutionStopped(db: Db, run: typeof heartbea await hasHistoricalSuspendedNativeSession(db, run); } +/** Dispatch revalidates the canonical admission receipt, not an admission which + * has already resolved its recovery holds. This exception certifies only the + * exact current successor; it never makes the failed result replayable. */ +export async function taskChatRetryAdmittedContinuationStopped(db: Db, run: typeof heartbeatRuns.$inferSelect, + input: { issueId: string; intentId: string; successorRunId: string; commentId: string }) { + if (run.nativeIssueId !== input.issueId) return false; + const retry = await readTaskChatRetryIntent(db, { companyId: run.companyId, issueId: input.issueId, + agentId: run.agentId, sourceRunId: run.id, intentId: input.intentId }); + if (!retry?.intent.chatActionId) return false; + const [task] = await db.select({ assignee: issues.assigneeAgentId, executionRunId: issues.executionRunId }) + .from(issues).where(and(eq(issues.companyId, run.companyId), eq(issues.id, input.issueId))); + const [successor] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, run.companyId), + eq(heartbeatRuns.id, input.successorRunId), eq(heartbeatRuns.agentId, run.agentId))).for("share", { noWait: true }); + const [receipt] = await db.select({ runId: agentWakeupRequests.runId, status: agentWakeupRequests.status }) + .from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, run.companyId), + eq(agentWakeupRequests.id, retry.intent.chatActionId), eq(agentWakeupRequests.agentId, run.agentId))); + const explicit = successor?.contextSnapshot?.explicitUserContinuation as Record | undefined; + if (!successor || task?.assignee !== run.agentId || task.executionRunId !== successor.id || + successor.contextSnapshot?.issueId !== input.issueId || + (successor.nativeIssueId !== null && successor.nativeIssueId !== input.issueId) || + successor.status !== "running" || successor.wakeupRequestId !== retry.intent.chatActionId || + successor.retryOfRunId !== run.id || receipt?.runId !== successor.id || receipt.status !== "claimed" || + successor.contextSnapshot?.taskRetryRequestId !== input.intentId || + explicit?.previousRunId !== run.id || explicit.commentId !== input.commentId || + explicit.taskRetryRequestId !== input.intentId || await getExecutionBlocker(db, run.companyId, input.issueId)) return false; + try { + await buildExecutionContinuation({ db, companyId: run.companyId, issueId: input.issueId, + agentId: run.agentId, runId: successor.id, context: successor.contextSnapshot!, summary: null, exposeLowTrustRaw: false }); + } catch (error) { + if (error instanceof Error && ["continuation_user_authorization_missing", "continuation_source_context_missing", + "continuation_task_ownership_changed"].includes(error.message)) return false; + throw error; + } + return taskChatRetryExecutionContained(db, run, true); +} + /** Admission may persist a wait for the exact recovery this user requested. * This grants no execution authority: promotion and execution still require * containment, and the queue retains the source's current execution blocker. */ diff --git a/server/src/services/chat-channels/task-retry-input.ts b/server/src/services/chat-channels/task-retry-input.ts index b5fb8a30233..200f77bda1c 100644 --- a/server/src/services/chat-channels/task-retry-input.ts +++ b/server/src/services/chat-channels/task-retry-input.ts @@ -1,5 +1,5 @@ import { z } from "zod"; -import { and, asc, eq, inArray, isNull, sql } from "drizzle-orm"; +import { and, asc, eq, gt, inArray, isNull, or, sql } from "drizzle-orm"; import { agentWakeupRequests, chatActions, chatConversations, chatEndpoints, heartbeatRuns, issues, type Db } from "@paperclipai/db"; import type { DbOrTransaction } from "./runtime-types.js"; @@ -8,15 +8,19 @@ export const taskRetryInputSchema = z.object({ // Truncating it strands older requests or makes the newer-input guard reject // every Retry. Dispatch already carries the full queue with bounded inline text. sourceActionIds: z.array(z.string().uuid()).refine(ids => new Set(ids).size === ids.length), + skippedActionIds: z.array(z.string().uuid()).optional(), runtimeFence: z.object({ generation: z.number().int().nonnegative(), credentialFingerprint: z.string().regex(/^[a-f0-9]{64}$/) }).optional(), -}); +}).refine(value => !value.skippedActionIds || ( + new Set(value.skippedActionIds).size === value.skippedActionIds.length && + value.skippedActionIds.every(id => value.sourceActionIds.includes(id)) +)); export type TaskRetryInput = z.infer; export async function selectTaskRetryInput(tx: DbOrTransaction, input: { companyId: string; issueId: string; agentId: string; failedRunId: string; }) { - const [conversation] = await tx.select({ id: chatConversations.id }).from(chatConversations) + const [conversation] = await tx.select({ id: chatConversations.id, runCreatedAt: heartbeatRuns.createdAt }).from(chatConversations) .innerJoin(chatEndpoints, eq(chatEndpoints.id, chatConversations.endpointId)) .innerJoin(chatActions, and(eq(chatActions.companyId, chatConversations.companyId), eq(chatActions.conversationId, chatConversations.id))) @@ -28,15 +32,20 @@ export async function selectTaskRetryInput(tx: DbOrTransaction, input: { eq(chatEndpoints.externalExecutionPolicy, "restricted"), )).limit(1); if (!conversation) return undefined; - const pending = await tx.select({ id: chatActions.id }).from(chatActions) + const pending = await tx.select({ id: chatActions.id, status: agentWakeupRequests.status }).from(chatActions) .innerJoin(agentWakeupRequests, and(eq(agentWakeupRequests.companyId, chatActions.companyId), eq(agentWakeupRequests.id, chatActions.id))).where(and( eq(chatActions.companyId, input.companyId), eq(chatActions.conversationId, conversation.id), - eq(chatActions.kind, "inbound_wakeup"), eq(chatActions.status, "processed"), - eq(agentWakeupRequests.agentId, input.agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"), + eq(chatActions.kind, "inbound_wakeup"), eq(agentWakeupRequests.agentId, input.agentId), + or(and(eq(chatActions.status, "processed"), eq(agentWakeupRequests.status, "deferred_issue_execution")), + and(eq(chatActions.status, "failed"), eq(agentWakeupRequests.status, "skipped"), + inArray(agentWakeupRequests.reason, ["budget.blocked", "agent.not_invokable"]), + sql`${chatActions.payload}->>'requestedByActorType' = 'user'`, + gt(chatActions.createdAt, conversation.runCreatedAt))), isNull(agentWakeupRequests.runId), )).orderBy(asc(chatActions.createdAt), asc(chatActions.id)).for("update", { of: agentWakeupRequests }); - return taskRetryInputSchema.parse({ sourceActionIds: pending.map(row => row.id) }); + return taskRetryInputSchema.parse({ sourceActionIds: pending.map(row => row.id), + skippedActionIds: pending.filter(row => row.status === "skipped").map(row => row.id) }); } export async function claimTaskRetryInput(tx: DbOrTransaction, input: { diff --git a/server/src/services/chat-run-publications.ts b/server/src/services/chat-run-publications.ts index 25ac411e74b..c3d3bc0d478 100644 --- a/server/src/services/chat-run-publications.ts +++ b/server/src/services/chat-run-publications.ts @@ -131,14 +131,18 @@ export async function resolveChatRunPresentationAuthorizationReason( /** A missing receipt is not proof that a provider write never arrived. */ export async function githubRunReplyState( db: Db, - input: { companyId: string; issueId: string; runId: string; endpointId: string }, + input: { companyId: string; issueId: string; runId: string; endpointId: string; sourceCommentId?: string; requireSettled?: boolean }, ): Promise<"none" | "unsettled" | "confirmed"> { const actions = await db .select({ status: chatActions.status, operation: sql`${chatActions.payload}->>'operation'`, reviewId: sql`${chatActions.payload}->>'reviewId'`, - result: chatActions.result, + result: sql | null>`case when ${chatActions.result} is null then null else jsonb_build_object( + 'reviewId', ${chatActions.result}->'reviewId', 'id', ${chatActions.result}->'id', 'url', ${chatActions.result}->'url', 'summaryUrl', ${chatActions.result}->'summaryUrl', + 'replyWriteStarted', ${chatActions.result}->'replyWriteStarted', 'retryable', ${chatActions.result}->'retryable', + 'code', ${chatActions.result}->'code', 'attempts', ${chatActions.result}->'attempts', + 'deliveryUnknown', ${chatActions.result}->'deliveryUnknown') end`, }) .from(chatActions) .innerJoin(chatConversations, and( @@ -161,12 +165,19 @@ export async function githubRunReplyState( sql`${chatActions.payload}->'session'->>'issueId' = ${input.issueId}`, sql`${chatActions.payload}->'session'->>'runId' = ${input.runId}`, sql`${chatActions.payload}->>'operation' in ('comment', 'formal_review', 'assessment')`, + input.sourceCommentId ? sql`exists (select 1 from chat_message_links source_link + where source_link.company_id = ${chatActions.companyId} and source_link.endpoint_id = ${chatActions.endpointId} + and source_link.conversation_id = ${chatActions.conversationId} and source_link.delivery_id = ${chatActions.deliveryId} + and source_link.direction = 'inbound' and source_link.comment_id = ${input.sourceCommentId}::uuid)` : undefined, )); - let unsettled = false; + let unsettled = false, confirmed = false; for (const action of actions) { + if (input.requireSettled && (action.status !== "processed" || action.result?.deliveryUnknown === true)) { + unsettled = true; + } // An assessment can publish findings before a later step fails or loses // authority. Those durable receipts still prove a reply was delivered. - if (action.operation === "assessment") { + if (action.operation === "assessment" && !input.requireSettled) { const [review] = await db .select({ receipts: chatGitHubReviews.publicationReceipts }) .from(chatGitHubReviews) @@ -181,11 +192,16 @@ export async function githubRunReplyState( } if (action.status === "processed") { if (action.operation === "comment" || action.operation === "formal_review") { - if (action.result?.id && action.result?.url) return "confirmed"; - unsettled = true; + if (action.result?.id && action.result?.url) { + if (!input.requireSettled) return "confirmed"; + confirmed = true; + } else unsettled = true; } if (action.operation === "assessment") { - if (action.result?.summaryUrl) return "confirmed"; + if (action.result?.summaryUrl) { + if (!input.requireSettled) return "confirmed"; + confirmed = true; + } else if (input.requireSettled) unsettled = true; } } else if (action.result?.replyWriteStarted === true || ["received", "processing"].includes(action.status) || (action.status === "failed" && (action.result?.retryable === true || action.result?.code === "publication_failed")) || @@ -194,7 +210,7 @@ export async function githubRunReplyState( unsettled = true; } } - return unsettled ? "unsettled" : "none"; + return unsettled ? "unsettled" : confirmed ? "confirmed" : "none"; } type ChatRunMilestoneCandidate = { diff --git a/server/src/services/durable-chat-wakeup.ts b/server/src/services/durable-chat-wakeup.ts index 9a689254e03..65a150f8a26 100644 --- a/server/src/services/durable-chat-wakeup.ts +++ b/server/src/services/durable-chat-wakeup.ts @@ -181,6 +181,19 @@ export function unadmittedChatWakeupCondition( and chat_admission.agent_id::text = chat_unadmitted.payload->>'agentId' and chat_admission.status <> 'skipped' ) + and not exists ( + select 1 from chat_actions chat_retry + join agent_wakeup_requests chat_retry_admission on chat_retry_admission.id = chat_retry.id + where chat_retry.company_id = chat_unadmitted.company_id + and chat_retry.endpoint_id = chat_unadmitted.endpoint_id + and chat_retry.conversation_id = chat_unadmitted.conversation_id + and chat_retry.kind = 'failed_run_retry' + and chat_retry.payload->'taskRetryInput'->'skippedActionIds' ? chat_unadmitted.id::text + and chat_retry.payload->>'issueId' = chat_unadmitted.payload->>'issueId' + and chat_retry_admission.company_id = chat_retry.company_id + and chat_retry_admission.agent_id::text = chat_retry.payload->>'agentId' + and chat_retry_admission.status <> 'skipped' + ) and (chat_unadmitted.status <> 'failed' or not exists ( select 1 from chat_actions chat_later join agent_wakeup_requests chat_later_admission on chat_later_admission.id = chat_later.id diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 8cda4456d6c..72a79b53199 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -2322,7 +2322,7 @@ export function heartbeatService( interactionId: readNonEmptyString(context.interactionId), issueId, messageIds: - run.retryOfRunId || context.retryOfRunId + (run.retryOfRunId || context.retryOfRunId) && !isFailedChatRunRetry ? [] : queuedCommentIdsFromRunContext(context).length ? queuedCommentIdsFromRunContext(context) diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index 7d337f06cfb..debcd1c18e0 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -1,4 +1,4 @@ -import { assertTaskRetryConversationAllowed, readTaskRetryIntent, retireSupersededTaskRetry, taskRetryMutationAllowed } from "../task-retry.js"; +import { assertTaskRetryConversationAllowed, readTaskRetryIntent, readTaskRetryBoardMessageOrigin, retireSupersededTaskRetry, taskRetryMutationAllowed } from "../task-retry.js"; import { insertIssueComment } from "../issue-comment-writes.js"; import { withExternalAdmissionGuard, assertNoExternalOverlap } from "../../modules/external-agents/index.js"; import { isAgentAwaitingSetup } from "../../modules/agent-lifecycle/index.js"; @@ -2195,6 +2195,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende if (executionReconciliationWake && !issueId) return null; const durableRequest = opts.durableChatRequest; let taskRetryActorId: string | null = null; + delete enrichedContextSnapshot.boardMessageRetry; if (opts.taskRetryRequestId) { const [receipt] = await db.select().from(agentWakeupRequests).where(and( eq(agentWakeupRequests.id, opts.taskRetryRequestId), eq(agentWakeupRequests.agentId, agentId), @@ -2213,6 +2214,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende enrichedContextSnapshot.forceFreshSession = true; enrichedContextSnapshot.previousRunId = retry.sourceRunId; enrichedContextSnapshot.taskRetryRequestId = receipt.id; + if (retry.boardMessageOrigin) enrichedContextSnapshot.boardMessageRetry = true; } @@ -2859,6 +2861,12 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende return { kind: "deferred" as const }; if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: agent.companyId, issueId, actorId: retry.actorId, actorSource: retry.actorSource })) return { kind: "deferred" as const }; + if (retry.boardMessageOrigin) { + const origin = await readTaskRetryBoardMessageOrigin(tx as unknown as Db, { companyId: agent.companyId, + issueId, agentId, sourceRunId: retry.sourceRunId, actorId: retry.actorId }); + if (!origin || origin.commentId !== retry.boardMessageOrigin.commentId || + origin.identityContextId !== retry.boardMessageOrigin.identityContextId) return { kind: "deferred" as const }; + } } if (opts.queuedCommentRequestId) { const ids = await undeliveredLegacyUserCommentIds(tx as unknown as Db, diff --git a/server/src/services/native-runtime/chat-attachment-reuse.ts b/server/src/services/native-runtime/chat-attachment-reuse.ts index 3d30cf42f74..cc7990e9b7a 100644 --- a/server/src/services/native-runtime/chat-attachment-reuse.ts +++ b/server/src/services/native-runtime/chat-attachment-reuse.ts @@ -19,6 +19,7 @@ import { issueComments, issues, issueWorkProducts, + runIdentityContexts, } from "@paperclipai/db"; import { @@ -32,8 +33,33 @@ import { loadConfig } from "../../config.js"; import { HttpError } from "../../errors.js"; import { voiceSessionStore, type VoiceTransaction } from "../voice/voice-session-store.js"; import { issueService } from "../issues.js"; +import { readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js"; import { boundExternalChatProvider } from "./external-chat-provider.js"; import { resolveExternalChatQuestionResponse } from "./external-chat-question-response.js"; +import { githubRunReplyState } from "../chat-run-publications.js"; + +// This boundary reads identity and wake authority, never launch renderings, +// saved provider profiles, or run output. Keep the nested wake proof narrow too. +const responseWaitContextKeys = ["source", "issueId", "wakeReason", "wakeCommentId", "wakeCommentIds", + "interactionId", "interactionKind", "interactionStatus", "sourceRunId", "sourceCommentId", + "externalChatContinuation", "paperclipExternalChatQuestionResponse", "paperclipHarnessCheckedOut", + "paperclipExternalChatExecutionBound"] as const; +const responseWaitRun = { + id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + nativeIssueId: heartbeatRuns.nativeIssueId, issueId: heartbeatRuns.issueId, status: heartbeatRuns.status, + responsibleUserId: heartbeatRuns.responsibleUserId, activeIdentityContextId: heartbeatRuns.activeIdentityContextId, + contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then (select coalesce(jsonb_object_agg(field.key, field.value), '{}'::jsonb) + from jsonb_each(${heartbeatRuns.contextSnapshot}) as field(key, value) + where field.key in (${sql.join(responseWaitContextKeys.map(key => sql`${key}`), sql`, `)})) + || jsonb_build_object('paperclipWake', jsonb_build_object( + 'checkedOutByHarness', ${heartbeatRuns.contextSnapshot}#>'{paperclipWake,checkedOutByHarness}', + 'externalChatExecutionBound', ${heartbeatRuns.contextSnapshot}#>'{paperclipWake,externalChatExecutionBound}', + 'externalChatProvider', ${heartbeatRuns.contextSnapshot}#>'{paperclipWake,externalChatProvider}', + 'issue', jsonb_build_object('id', ${heartbeatRuns.contextSnapshot}#>'{paperclipWake,issue,id}'), + 'commentIds', ${heartbeatRuns.contextSnapshot}#>'{paperclipWake,commentIds}')) + else null end`, +}; export const LIST_CHAT_ATTACHMENTS_TOOL_NAME = "list_chat_attachments"; export const REUSE_CHAT_ATTACHMENT_TOOL_NAME = "reuse_chat_attachment"; @@ -652,7 +678,7 @@ export async function resolveExternalChatResponseWaitAuthorizationInTransaction( : issueQuery.for("update").limit(1)); const runQuery = tx - .select() + .select(responseWaitRun) .from(heartbeatRuns) .where( and( @@ -684,12 +710,12 @@ export async function resolveExternalChatResponseWaitAuthorizationInTransaction( !answerContext ) return "revoked"; - const candidate = run - ? externalChatWaitCandidate( - answerContext?.authorizationContext ?? run.contextSnapshot, - binding, - ) - : null; + const directContext = answerContext?.authorizationContext ?? run?.contextSnapshot; + const inheritedReviewWait = run && !externalChatWaitCandidate(directContext, binding) + ? await readInheritedGitHubReviewWait(tx, run) : null; + if (inheritedReviewWait?.kind === "revoked") return "revoked"; + const authorizationContext = inheritedReviewWait?.context ?? directContext; + const candidate = run ? externalChatWaitCandidate(authorizationContext, binding) : null; if (!candidate) return "not_applicable"; const actorQuery = tx @@ -779,7 +805,7 @@ export async function resolveExternalChatResponseWaitAuthorizationInTransaction( await authorizeChatConversationForBoundRun( tx, binding, - run.contextSnapshot, + authorizationContext, lockMode, ); return "authorized"; @@ -811,6 +837,49 @@ export async function resolveExternalChatResponseWaitAuthorization(input: { ); } +const publishedGitHubReviewWaitKey = "github-review-response:"; + +export function isPublishedGitHubReviewCompletionContinuation(value: unknown): boolean { + const continuation = record(value); + return continuation.kind === "response_wake" && typeof continuation.idempotencyKey === "string" && + continuation.idempotencyKey.startsWith(publishedGitHubReviewWaitKey); +} + +/** Project a wait from the same saved publication and current authority used + * by finalization. Inherited waits need an accepted tree resume or a confirmed + * reply from this run to its exact current source. An arbitrary old review does + * not assign an unanswered new turn a wait. */ +export async function readPublishedGitHubReviewCompletionContinuationInTransaction( + tx: Db, binding: ChatReuseBinding, lockMode: AuthorizationLockMode = "read", +) { + const [run] = await tx.select(responseWaitRun).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, binding.runId), eq(heartbeatRuns.companyId, binding.companyId), + eq(heartbeatRuns.agentId, binding.agentId), eq(heartbeatRuns.nativeIssueId, binding.issueId), + eq(heartbeatRuns.runtimeMode, "native"), + )).limit(1); + if (!run || run.status !== "running") return null; + const wait = await readInheritedGitHubReviewWait(tx, run); + if (wait?.kind !== "published") return null; + if (wait.reviewRunId !== run.id) { + const [identity] = await tx.select({ cause: runIdentityContexts.cause, parentContextId: runIdentityContexts.parentContextId }) + .from(runIdentityContexts).where(and(eq(runIdentityContexts.id, run.activeIdentityContextId!), + eq(runIdentityContexts.companyId, run.companyId), eq(runIdentityContexts.runId, run.id), + eq(runIdentityContexts.status, "accepted"))).limit(1); + const resumed = run.contextSnapshot?.source === "issue.tree_resume" && identity?.cause === "issue_tree_resumed" && + Boolean(identity.parentContextId); + if (!resumed && !(run.contextSnapshot?.source === "chat:github" && + await githubRunReplyState(tx, { ...binding, endpointId: wait.endpointId, + sourceCommentId: wait.context.wakeCommentId, requireSettled: true }) === "confirmed")) return null; + } + if (await resolveExternalChatResponseWaitAuthorizationInTransaction(tx, binding, lockMode) !== "authorized") return null; + return { kind: "response_wake" as const, summary: "Wait for the next authorized GitHub response to the published review.", + idempotencyKey: `${publishedGitHubReviewWaitKey}${run.id}:${wait.reviewId}:${wait.headSha}` }; +} + +export async function readPublishedGitHubReviewCompletionContinuation(input: { db: Db; binding: ChatReuseBinding }) { + return input.db.transaction(tx => readPublishedGitHubReviewCompletionContinuationInTransaction(tx as unknown as Db, input.binding)); +} + async function sourceLineageExists( tx: Db, binding: ChatReuseBinding, diff --git a/server/src/services/native-runtime/external-chat-wait.integration.test.ts b/server/src/services/native-runtime/external-chat-wait.integration.test.ts index 70a6933e80a..50d7910bb96 100644 --- a/server/src/services/native-runtime/external-chat-wait.integration.test.ts +++ b/server/src/services/native-runtime/external-chat-wait.integration.test.ts @@ -1,3 +1,4 @@ +import { initializeRunIdentity } from "../run-identity.js"; import { createHash, randomUUID } from "node:crypto"; import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; @@ -36,12 +37,14 @@ import { issues, nativeRunFinalizations, nativeRunResults, + runIdentityContexts, statusDecisions, toolApplications, toolConnections, chatVoiceSessions, chatVoiceReplies, chatVoiceToolCalls, workspaceOperations, } from "@paperclipai/db"; +import { validatePrpStructuredRunResult } from "../../vendor/paperclip-runner/index.js"; import type { PrpStructuredRunResult, PrpTerminalState, @@ -50,7 +53,10 @@ import type { import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { finalizeNativeRun } from "./native-run-finalizer.js"; import { nativeCompletionFeedback } from "./native-completion-feedback.js"; -import { hasPublishedGitHubReview } from "../chat-channels/github/review-wait.js"; +import { hasPublishedGitHubReview, readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js"; +import * as chatWaitAuthority from "./chat-attachment-reuse.js"; +import * as publishedReviewAuthority from "../chat-channels/github/review-wait.js"; +import { toolAccessService } from "../tool-access.js"; import { PaperclipControlPlanePort } from "./paperclip-control-plane-port.js"; import { authorizeNativeChatReviewPresentation, @@ -114,6 +120,8 @@ describe("native external-chat response wait", () => { | "slack" = "telegram", attentionRequests: PrpStructuredRunResult["attentionRequests"] = [], resultOverrides: Partial = {}, + deferCompletion = false, + contractOverride?: { objective: string; criteria: Array<{ id: string; requirement: string }> }, ) { const companyId = randomUUID(); const agentId = randomUUID(); @@ -177,6 +185,7 @@ describe("native external-chat response wait", () => { criteria: [ { id: "response", requirement: "Return the requested photo" }, ], + ...contractOverride, }, canonicalSha256: contractSha256, createdByActorType: "system", @@ -384,7 +393,7 @@ describe("native external-chat response wait", () => { workAssessmentId: randomUUID(), statusDecisionId: randomUUID(), }; - await port.completeRun({ + if (!deferCompletion) await port.completeRun({ result, terminal, callerResultId: `wait-result-${runId}`, @@ -401,6 +410,7 @@ describe("native external-chat response wait", () => { .where(eq(heartbeatRuns.id, runId)); return { + port, result, terminal, agentId, companyId, conversationId, @@ -4385,7 +4395,14 @@ describe("native external-chat response wait", () => { [5, "done", "done", null, false], [3, "yielded", "in_review", "monitor", false], [3, "yielded", "in_review", "response_wake", true], - ] as const)("settles a published %s/5 review reporting %s in %s with %s, separate approval: %s", async (score, disposition, status, continuationKind, separateApproval) => { + [3, "yielded", "in_review", "response_wake", false, "tree_resume"], + [3, "yielded", "in_review", "response_wake", false, "continued_review"], + [3, "yielded", "in_review", "monitor", false, "continued_review"], + [3, "yielded", "in_progress", "response_wake", false, "new_head"], + ] as const)("settles a published %s/5 review reporting %s in %s with %s, separate approval: %s, resume: %s", async (score, disposition, status, continuationKind, separateApproval, resumeMode?: "tree_resume" | "continued_review" | "new_head") => { + const resumed = resumeMode !== undefined; + const savedPublication = resumeMode === "continued_review" || resumeMode === "new_head"; + const unreviewedHead = resumeMode === "new_head"; const fixture = await seedWaitTurn("github", [], { reportedWorkDisposition: disposition, summary: disposition === "done" ? "Review is 5/5 and required checks passed." : "Review published; waiting for the next PR update.", @@ -4394,7 +4411,7 @@ describe("native external-chat response wait", () => { }); await db.update(issues).set({ title: "Review the pull request", originKind: "chat_channel", createdByUserId: fixture.userId, responsibleUserId: fixture.userId }).where(eq(issues.id, fixture.issueId)); - const oldRequest = score === 3 ? await seedPriorCompletionReview(fixture) : null; + const oldRequest = score === 3 && !resumed ? await seedPriorCompletionReview(fixture) : null; if (continuationKind === "monitor") { await new PaperclipRunnerToolAuthority(db, fixture).execute({ tool: "set_task_monitor", callId: randomUUID(), arguments: { idempotencyKey: "review-monitor", monitor: { nextCheckAt: new Date(Date.now() + 60_000).toISOString(), notes: "Check PR status." } } }); @@ -4417,7 +4434,7 @@ describe("native external-chat response wait", () => { }).returning(); await expect(hasPublishedGitHubReview(db, fixture)).resolves.toBe(false); // A completed but unpublished head must not hide another published head from this run. - const [published] = await db.insert(chatGitHubReviews).values({ ...review!, id: randomUUID(), deliveryId: randomUUID(), headSha: "c".repeat(40) }).returning(); + const [published] = await db.insert(chatGitHubReviews).values({ ...review!, id: randomUUID(), deliveryId: randomUUID(), headSha: "c".repeat(40), assessment: { ...review!.assessment!, reviewedCommit: "c".repeat(40) }, createdAt: new Date(Date.now() + 10) }).returning(); await db.insert(chatActions).values({ companyId: fixture.companyId, endpointId: fixture.endpointId, kind: "github_review_publication", providerActionId: `github_publication:${fixture.issueId}:assessment:${createHash("sha256").update(JSON.stringify(published!.id)).digest("hex")}`, @@ -4433,8 +4450,76 @@ describe("native external-chat response wait", () => { providerActionId: `github_publication:${fixture.issueId}:assessment:${createHash("sha256").update(JSON.stringify(older!.id)).digest("hex")}`, }); } + let inheritedOriginalRunId: string | null = null; + if (resumed) { + await db.update(chatEndpointResources).set({ metadata: { providerRepositoryId: "1" } }).where(eq(chatEndpointResources.id, fixture.resourceId)); + await db.update(chatConversations).set({ externalThreadId: "github:paperclip/test-repository:1" }).where(eq(chatConversations.id, fixture.conversationId)); + // A tree resume inherits its requester through the accepted identity + // ledger, while its immediate wake correctly has no chat envelope. + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + const originalRunId = randomUUID(); + inheritedOriginalRunId = originalRunId; + await db.insert(heartbeatRuns).values({ ...run!, id: originalRunId, activeIdentityContextId: null, status: "succeeded", finishedAt: new Date() }); + const [origin] = await db.insert(runIdentityContexts).values({ + companyId: fixture.companyId, runId: originalRunId, revision: 1, + messageId: fixture.commentId, responsibleUserId: fixture.userId, + cause: "chat_message", correlationId: "dispatch", status: "accepted", + }).returning(); + const [continuation] = await db.insert(runIdentityContexts).values({ + companyId: fixture.companyId, runId: fixture.runId, revision: 1, + parentContextId: origin!.id, responsibleUserId: fixture.userId, + cause: "issue_tree_resumed", correlationId: "dispatch", status: "accepted", + }).returning(); + await db.update(heartbeatRuns).set({ activeIdentityContextId: continuation!.id, + responsibleUserId: fixture.userId, contextSnapshot: { + source: "issue.tree_resume", wakeReason: "issue_tree_resumed", + issueId: fixture.issueId, paperclipHarnessCheckedOut: true, + paperclipWake: { checkedOutByHarness: true, issue: { id: fixture.issueId }, commentIds: [] }, + }, + }).where(eq(heartbeatRuns.id, fixture.runId)); + } await expect(hasPublishedGitHubReview(db, fixture)).resolves.toBe(true); await expect(hasPublishedGitHubReview(db, { ...fixture, runId: randomUUID() })).resolves.toBe(false); + if (resumed) { + const [inheritedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + await expect(readInheritedGitHubReviewWait(db, inheritedRun!)).resolves.toMatchObject({ kind: "published" }); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.toBe("authorized"); + for (const binding of [ + { ...fixture, companyId: randomUUID() }, { ...fixture, issueId: randomUUID() }, + { ...fixture, agentId: randomUUID() }, + ]) await expect(resolveExternalChatResponseWaitAuthorization({ db, binding })).resolves.not.toBe("authorized"); + await db.update(chatActions).set({ status: "pending" }).where(eq(chatActions.companyId, fixture.companyId)); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.not.toBe("authorized"); + await db.update(chatActions).set({ status: "processed" }).where(eq(chatActions.companyId, fixture.companyId)); + await db.update(chatEndpointResources).set({ enabled: false }).where(eq(chatEndpointResources.endpointId, fixture.endpointId)); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.toBe("revoked"); + await db.update(chatEndpointResources).set({ enabled: true }).where(eq(chatEndpointResources.endpointId, fixture.endpointId)); + await db.update(heartbeatRuns).set({ responsibleUserId: "different-requester" }).where(eq(heartbeatRuns.id, fixture.runId)); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.not.toBe("authorized"); + await db.update(heartbeatRuns).set({ responsibleUserId: fixture.userId }).where(eq(heartbeatRuns.id, fixture.runId)); + await db.update(chatConversations).set({ externalThreadId: "github:paperclip/other-repository:1" }).where(eq(chatConversations.id, fixture.conversationId)); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.not.toBe("authorized"); + await db.update(chatConversations).set({ externalThreadId: "github:paperclip/test-repository:1" }).where(eq(chatConversations.id, fixture.conversationId)); + await db.update(chatEndpointResources).set({ metadata: { providerRepositoryId: "2" } }).where(eq(chatEndpointResources.id, fixture.resourceId)); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.not.toBe("authorized"); + await db.update(chatEndpointResources).set({ metadata: { providerRepositoryId: "1" } }).where(eq(chatEndpointResources.id, fixture.resourceId)); + const [newHead] = await db.insert(chatGitHubReviews).values({ ...review!, id: randomUUID(), + deliveryId: randomUUID(), runId: null, headSha: "d".repeat(40), state: "queued", createdAt: new Date(Date.now() + 1000), + }).returning(); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.not.toBe("authorized"); + await db.delete(chatGitHubReviews).where(eq(chatGitHubReviews.id, newHead!.id)); + if (savedPublication) { + await db.update(chatGitHubReviews).set({ runId: inheritedOriginalRunId }).where(eq(chatGitHubReviews.issueId, fixture.issueId)); + await expect(hasPublishedGitHubReview(db, fixture)).resolves.toBe(false); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.toBe("authorized"); + } + } + if (unreviewedHead) { + await db.insert(chatGitHubReviews).values({ ...review!, id: randomUUID(), runId: null, + deliveryId: randomUUID(), headSha: "e".repeat(40), state: "queued", createdAt: new Date(Date.now() + 1000), + }); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.toBe("revoked"); + } const [stored] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, fixture.runId)); await expect(nativeCompletionFeedback(db, fixture.runId, stored!.resultJson.result as PrpStructuredRunResult)).resolves.toContain("accepted"); if (oldRequest) { @@ -4468,6 +4553,15 @@ describe("native external-chat response wait", () => { if (oldRequest) expect(await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, oldRequest.id))) .toEqual([expect.objectContaining({ status: "cancelled", result: expect.objectContaining({ outcome: "withdrawn" }) })]); expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toEqual([]); + if (resumed) { + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toEqual([]); + expect(await db.select().from(chatActions).where(and(eq(chatActions.companyId, fixture.companyId), eq(chatActions.kind, "github_review_publication")))).toHaveLength(1); + if (unreviewedHead) { + expect((await db.select().from(statusDecisions).where(eq(statusDecisions.runId, fixture.runId)))[0]) + .toMatchObject({ reasonCode: "external_chat_response_wait_authorization_lost" }); + } + } if (status === "in_review" && continuationKind === "response_wake") { const attention = async () => (await issueService(db).listReviewAttention(fixture.companyId, [{ id: fixture.issueId, companyId: fixture.companyId, status }])).get(fixture.issueId)?.state; await db.update(chatEndpointResources).set({ enabled: false }).where(eq(chatEndpointResources.endpointId, fixture.endpointId)); @@ -4483,6 +4577,397 @@ describe("native external-chat response wait", () => { } }); + async function seedPublishedReviewCompletionTurn(contractOverride?: Parameters[4]) { + const fixture = await seedWaitTurn("github", [], {}, true, contractOverride); + await db.update(chatEndpointResources).set({ metadata: { providerRepositoryId: "1" } }) + .where(eq(chatEndpointResources.id, fixture.resourceId)); + await db.update(chatConversations).set({ externalThreadId: "github:paperclip/test-repository:1" }) + .where(eq(chatConversations.id, fixture.conversationId)); + const [identity] = await db.insert(runIdentityContexts).values({ companyId: fixture.companyId, + runId: fixture.runId, revision: 1, messageId: fixture.commentId, responsibleUserId: fixture.userId, + cause: "chat_message", correlationId: "dispatch", status: "accepted" }).returning(); + await db.update(heartbeatRuns).set({ activeIdentityContextId: identity!.id, responsibleUserId: fixture.userId }) + .where(eq(heartbeatRuns.id, fixture.runId)); + const [review] = await db.insert(chatGitHubReviews).values({ companyId: fixture.companyId, + endpointId: fixture.endpointId, issueId: fixture.issueId, runId: fixture.runId, + repositoryId: "1", repository: "paperclip/test-repository", pullNumber: 1, headSha: "a".repeat(40), + deliveryId: fixture.deliveryId, configurationRevision: 1, policySnapshot: defaultGitHubReviewPolicy(), + event: { event: "opened", deliveryId: fixture.deliveryId, repositoryId: "1", repository: "paperclip/test-repository", + pullNumber: 1, title: "Review", body: "", baseSha: "b".repeat(40), headSha: "a".repeat(40), + baseBranch: "master", author: { id: "1", login: "author", isBot: false }, sender: { id: "1", login: "author" }, draft: false, labels: [] }, + state: "completed", assessment: { complete: true, score: 3, reviewedCommit: "a".repeat(40), + summary: "Quantity is ignored", rationale: "Review complete", findings: [], + coverage: { reviewedPaths: [], omittedPaths: [], limitations: [] } }, checkId: "123" }).returning(); + await db.insert(chatActions).values({ companyId: fixture.companyId, endpointId: fixture.endpointId, + kind: "github_review_publication", status: "processed", payload: { operation: "assessment", reviewId: review!.id }, + providerActionId: `github_publication:${fixture.issueId}:assessment:${createHash("sha256").update(JSON.stringify(review!.id)).digest("hex")}` }); + return { ...fixture, review: review! }; + } + + it.each([false, true])("binds a low-score Retry wait to its accepted newer source after a predecessor failed before identity: %s", async acceptedSource => { + const objective = "Review the current head. A lower score stays In review with a passive response wake; only a merged PR or 5/5 with passing checks is Done."; + const fixture = await seedPublishedReviewCompletionTurn({ objective, + criteria: [{ id: "response", requirement: objective }] }); + await db.update(chatGitHubReviews).set({ policySnapshot: { ...fixture.review.policySnapshot, ratingThreshold: 5 }, + assessment: { ...fixture.review.assessment!, score: 2 } }).where(eq(chatGitHubReviews.id, fixture.review.id)); + const previousRunId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: previousRunId, companyId: fixture.companyId, agentId: fixture.agentId, + issueId: fixture.issueId, status: "failed", errorCode: "chat_failed_run_retry_not_authorized", + finishedAt: new Date(), contextSnapshot: { issueId: fixture.issueId, source: "chat:github" } }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + await db.update(heartbeatRuns).set({ activeIdentityContextId: null, retryOfRunId: previousRunId, + contextSnapshot: { ...run!.contextSnapshot, retryOfRunId: previousRunId, wakeReason: "retry_failed_run" } }) + .where(eq(heartbeatRuns.id, fixture.runId)); + await db.delete(runIdentityContexts).where(eq(runIdentityContexts.id, run!.activeIdentityContextId!)); + await initializeRunIdentity(db, { companyId: fixture.companyId, issueId: fixture.issueId, + runId: fixture.runId, responsibleUserId: fixture.userId, parentRunId: previousRunId, + messageIds: acceptedSource ? [fixture.commentId] : [], cause: "retry_failed_run" }); + const authority = new PaperclipRunnerToolAuthority(db, fixture); + const read = () => authority.execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} }); + const { resolveCompletionContinuation } = await import(new URL("../../../../packages/paperclip-runner/src/drivers/completion-continuation.ts", import.meta.url).href); + const report = { ...fixture.result, continuation: undefined, completionClaim: { + ...fixture.result.completionClaim, objectiveSatisfied: false, + criteria: [{ criterionId: "response", status: "unknown" as const, evidenceRefs: [] }], + remainingWork: [{ description: "Author must fix the quantity defect before this review can reach the passing score.", blocksCompletion: true }], + } }; + if (!acceptedSource) { + await expect(resolveCompletionContinuation(report, fixture.runId, read)).rejects.toThrow("no eligible saved monitor"); + expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0].status).not.toBe("done"); + return; + } + const resolved = await resolveCompletionContinuation(report, fixture.runId, read) as PrpStructuredRunResult; + expect(resolved).toMatchObject({ reportedWorkDisposition: "yielded", continuation: { kind: "response_wake" } }); + await expect(nativeCompletionFeedback(db, fixture.runId, resolved)).resolves.toContain("accepted"); + await fixture.port.completeRun({ result: resolved, terminal: fixture.terminal, callerResultId: `retry-wait-${fixture.runId}` }); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0]).toMatchObject({ status: "in_review" }); + expect((await db.select().from(statusDecisions).where(eq(statusDecisions.runId, fixture.runId)))[0]) + .toMatchObject({ reasonCode: "external_chat_response_waiting", toStatus: "in_review", decisionJson: { effects: [] } }); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toHaveLength(0); + expect(await db.select().from(chatActions).where(and(eq(chatActions.companyId, fixture.companyId), + eq(chatActions.kind, "github_review_publication")))).toHaveLength(1); + }); + + async function seedInheritedReviewCompletionTurn(resume: boolean) { + const fixture = await seedPublishedReviewCompletionTurn(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + const predecessorId = randomUUID(); + await db.insert(heartbeatRuns).values({ ...run!, id: predecessorId, activeIdentityContextId: null, + status: "succeeded", finishedAt: new Date() }); + await db.update(chatGitHubReviews).set({ runId: predecessorId }).where(eq(chatGitHubReviews.id, fixture.review.id)); + if (resume) { + const [origin] = await db.insert(runIdentityContexts).values({ companyId: fixture.companyId, + runId: predecessorId, revision: 1, messageId: fixture.commentId, responsibleUserId: fixture.userId, + cause: "chat_message", correlationId: "dispatch", status: "accepted" }).returning(); + await db.update(runIdentityContexts).set({ messageId: null, parentContextId: origin!.id, + cause: "issue_tree_resumed" }).where(eq(runIdentityContexts.id, run!.activeIdentityContextId!)); + await db.update(heartbeatRuns).set({ contextSnapshot: { source: "issue.tree_resume", wakeReason: "issue_tree_resumed", + issueId: fixture.issueId, paperclipHarnessCheckedOut: true, + paperclipWake: { checkedOutByHarness: true, issue: { id: fixture.issueId }, commentIds: [] } }, + }).where(eq(heartbeatRuns.id, fixture.runId)); + } + return { ...fixture, predecessorId }; + } + + it.each([undefined, null, "malformed-first"] as const)("derives an omitted inherited GitHub review wait after normal tree resume: %s", async absent => { + const fixture = await seedInheritedReviewCompletionTurn(true); + const authority = new PaperclipRunnerToolAuthority(db, fixture); + const read = () => authority.execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} }); + const { resolveCompletionContinuation } = await import(new URL("../../../../packages/paperclip-runner/src/drivers/completion-continuation.ts", import.meta.url).href); + const report = { ...fixture.result, continuation: absent === "malformed-first" ? "response_wake" : absent }; + if (absent === "malformed-first") { + expect(await resolveCompletionContinuation(report, fixture.runId, read)).toEqual(report); + expect(validatePrpStructuredRunResult(report).ok).toBe(false); + delete report.continuation; + } + const resolved = await resolveCompletionContinuation(report, fixture.runId, read) as PrpStructuredRunResult; + expect(resolved).toMatchObject({ reportedWorkDisposition: "yielded", continuation: { kind: "response_wake" } }); + await expect(nativeCompletionFeedback(db, fixture.runId, resolved)).resolves.toContain("accepted"); + await fixture.port.completeRun({ result: resolved, terminal: fixture.terminal, callerResultId: `wait-result-${fixture.runId}` }); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0]).toMatchObject({ status: "in_review" }); + expect((await db.select().from(statusDecisions).where(eq(statusDecisions.runId, fixture.runId)))[0]) + .toMatchObject({ reasonCode: "external_chat_response_waiting", decisionJson: { effects: [] } }); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toHaveLength(0); + expect(await db.select().from(chatActions).where(and(eq(chatActions.companyId, fixture.companyId), + eq(chatActions.kind, "github_review_publication")))).toHaveLength(1); + }); + + async function seedCurrentAuthorReply(fixture: Awaited>) { + const [reply] = await db.insert(chatActions).values({ companyId: fixture.companyId, + endpointId: fixture.endpointId, conversationId: fixture.conversationId, deliveryId: fixture.deliveryId, + kind: "github_review_publication", status: "processed", providerActionId: `github_publication:${fixture.issueId}:comment:${randomUUID()}`, + payload: { operation: "comment", session: { companyId: fixture.companyId, issueId: fixture.issueId, + runId: fixture.runId, agentId: fixture.agentId }, body: "Quantity is ignored by the current total calculation." }, + result: { id: "123", url: "https://github.com/paperclip/test-repository/pull/1#issuecomment-123" }, + }).returning(); + return reply!; + } + + it.each([undefined, null] as const)("derives an omitted inherited review wait only after this author reply is delivered: %s", async continuation => { + const fixture = await seedInheritedReviewCompletionTurn(false); + await seedCurrentAuthorReply(fixture); + const authority = new PaperclipRunnerToolAuthority(db, fixture); + const { resolveCompletionContinuation } = await import(new URL("../../../../packages/paperclip-runner/src/drivers/completion-continuation.ts", import.meta.url).href); + const resolved = await resolveCompletionContinuation({ ...fixture.result, continuation }, fixture.runId, + () => authority.execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} })) as PrpStructuredRunResult; + expect(resolved).toMatchObject({ continuation: { kind: "response_wake" } }); + await expect(nativeCompletionFeedback(db, fixture.runId, resolved)).resolves.toContain("accepted"); + await fixture.port.completeRun({ result: resolved, terminal: fixture.terminal, callerResultId: `wait-result-${fixture.runId}` }); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0]).toMatchObject({ status: "in_review" }); + expect((await db.select().from(statusDecisions).where(eq(statusDecisions.runId, fixture.runId)))[0]) + .toMatchObject({ reasonCode: "external_chat_response_waiting", decisionJson: { effects: [] } }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toHaveLength(0); + }); + + it.each(["missing_parent", "wrong_cause", "principal_revoked", "new_head", "pending_publication"] as const)( + "does not derive an inherited tree resume wait after %s", async gate => { + const fixture = await seedInheritedReviewCompletionTurn(true); + if (gate === "missing_parent" || gate === "wrong_cause") await db.update(runIdentityContexts) + .set(gate === "missing_parent" ? { parentContextId: null } : { cause: "chat_message" }) + .where(eq(runIdentityContexts.runId, fixture.runId)); + if (gate === "principal_revoked") await db.update(companyMemberships).set({ status: "suspended" }) + .where(eq(companyMemberships.companyId, fixture.companyId)); + if (gate === "new_head") await db.insert(chatGitHubReviews).values({ ...fixture.review, id: randomUUID(), + runId: null, deliveryId: randomUUID(), headSha: "c".repeat(40), state: "queued", assessment: null, + createdAt: new Date(Date.now() + 1000) }); + if (gate === "pending_publication") await db.update(chatActions).set({ status: "received" }) + .where(eq(chatActions.companyId, fixture.companyId)); + const context = new PaperclipRunnerToolAuthority(db, fixture) + .execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} }); + if (gate === "principal_revoked") await expect(context).rejects.toThrow("task is no longer available"); + else expect(await context).toMatchObject({ completionContinuation: null }); + }); + + it.each(["missing", "pending", "unknown", "mixed_unknown", "mixed_missing_assessment", "other_run", "other_source", "ack_only", "partial_assessment", "new_head", "revoked"] as const)( + "does not derive an inherited author reply wait after %s", async gate => { + const fixture = await seedInheritedReviewCompletionTurn(false); + const reply = await seedCurrentAuthorReply(fixture); + if (gate === "missing") await db.delete(chatActions).where(eq(chatActions.id, reply.id)); + if (gate === "pending" || gate === "unknown") await db.update(chatActions) + .set({ status: gate === "pending" ? "received" : "failed", result: { deliveryUnknown: true } }) + .where(eq(chatActions.id, reply.id)); + if (gate === "mixed_unknown") await db.insert(chatActions).values({ ...reply, id: randomUUID(), + providerActionId: `github_publication:${randomUUID()}`, status: "failed", result: { deliveryUnknown: true } }); + if (gate === "mixed_missing_assessment") await db.insert(chatActions).values({ ...reply, id: randomUUID(), + providerActionId: `github_publication:${randomUUID()}`, payload: { ...reply.payload, + operation: "assessment", reviewId: fixture.review.id }, result: { published: 1, total: 2 } }); + if (gate === "other_run") await db.update(chatActions).set({ payload: { ...reply.payload, + session: { companyId: fixture.companyId, issueId: fixture.issueId, runId: fixture.predecessorId, agentId: fixture.agentId } } }) + .where(eq(chatActions.id, reply.id)); + if (gate === "other_source") { + const [source] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.id, fixture.deliveryId)); + const [other] = await db.insert(chatDeliveries).values({ ...source!, id: randomUUID(), providerEventId: randomUUID(), deduplicationKey: randomUUID() }).returning(); + await db.update(chatActions).set({ deliveryId: other!.id }).where(eq(chatActions.id, reply.id)); + } + if (gate === "partial_assessment") { + const [partial] = await db.insert(chatGitHubReviews).values({ ...fixture.review, id: randomUUID(), + runId: fixture.runId, deliveryId: randomUUID(), createdAt: new Date(Date.now() - 86_400_000), publicationReceipts: { + partial: { id: "123", url: "https://github.com/paperclip/test-repository/pull/1#issuecomment-123", digest: "partial" }, + } }).returning(); + await db.update(chatActions).set({ status: "failed", payload: { ...reply.payload, + operation: "assessment", reviewId: partial!.id }, result: { code: "publication_failed" } }) + .where(eq(chatActions.id, reply.id)); + } + if (gate === "ack_only") await db.update(chatActions).set({ kind: "github_response_comment" }) + .where(eq(chatActions.id, reply.id)); + if (gate === "new_head") await db.insert(chatGitHubReviews).values({ ...fixture.review, id: randomUUID(), + runId: null, deliveryId: randomUUID(), headSha: "c".repeat(40), state: "queued", assessment: null, + createdAt: new Date(Date.now() + 1000) }); + if (gate === "revoked") await db.update(toolConnections).set({ enabled: false }) + .where(eq(toolConnections.companyId, fixture.companyId)); + const context = await new PaperclipRunnerToolAuthority(db, fixture) + .execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} }); + expect(context).toMatchObject({ completionContinuation: null }); + }); + + it.each(["current", "resume", "reply"] as const)("projects only bound authority while deriving a published GitHub completion wait: %s", async turn => { + const fixture = turn === "current" ? await seedPublishedReviewCompletionTurn() : await seedInheritedReviewCompletionTurn(turn === "resume"); + const reply = turn === "reply" ? await seedCurrentAuthorReply(fixture as Awaited>) : null; + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + const payload = "unused-saved-review-rendering".repeat(32768); + await db.update(heartbeatRuns).set({ runnerProfileJson: { retained: payload }, stdoutExcerpt: payload, stderrExcerpt: payload, + contextSnapshot: { ...run.contextSnapshot, executionContinuation: { savedInput: payload }, + paperclipTaskMarkdown: payload, paperclipWake: { ...(run.contextSnapshot!.paperclipWake as object), retainedPayload: payload } }, + }).where(eq(heartbeatRuns.id, fixture.runId)); + if (reply) await db.update(chatActions).set({ result: { ...reply.result, savedProviderOutput: payload } }).where(eq(chatActions.id, reply.id)); + const observed = createDb(temporary?.connectionString ?? externalTestDatabaseUrl!); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns, chatActions }, capture); + try { + const context = await new PaperclipRunnerToolAuthority(observed, fixture) + .execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} }); + expect(context).toMatchObject({ completionContinuation: { kind: "response_wake" } }); + capture.flush(); + const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(16 * 1024); + const actionReads = events.filter(event => event.table === "chat_actions" && event.operation === "select"); + expect(actionReads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(16 * 1024); + expect(JSON.stringify(context)).not.toContain(payload); + } finally { capture.flush(); await observed.$client.end(); } + }); + + it.each([undefined, null, "malformed-first"] as const)("derives an omitted published GitHub review wait through current task context: %s", async absent => { + const fixture = await seedPublishedReviewCompletionTurn(); + const authority = new PaperclipRunnerToolAuthority(db, fixture); + const read = () => authority.execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} }); + const { resolveCompletionContinuation } = await import(new URL("../../../../packages/paperclip-runner/src/drivers/completion-continuation.ts", import.meta.url).href); + const report = { ...fixture.result, continuation: absent === "malformed-first" ? "response_wake" : absent }; + if (absent === "malformed-first") { + // Explicit malformed input remains invalid; the next omitted report can + // recover from authoritative state without changing its disposition. + expect(await resolveCompletionContinuation(report, fixture.runId, read)).toEqual(report); + expect(validatePrpStructuredRunResult(report).ok).toBe(false); + delete report.continuation; + } + const resolved = await resolveCompletionContinuation(report, fixture.runId, read) as PrpStructuredRunResult; + expect(resolved).toMatchObject({ reportedWorkDisposition: "yielded", continuation: { kind: "response_wake" } }); + await expect(nativeCompletionFeedback(db, fixture.runId, resolved)).resolves.toContain("accepted"); + await fixture.port.completeRun({ result: resolved, terminal: fixture.terminal, callerResultId: `wait-result-${fixture.runId}` }); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0]).toMatchObject({ status: "in_review" }); + expect((await db.select().from(statusDecisions).where(eq(statusDecisions.runId, fixture.runId)))[0]) + .toMatchObject({ reasonCode: "external_chat_response_waiting", decisionJson: { effects: [] } }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toHaveLength(0); + }); + + it.each(["missing_publication", "pending_publication", "unknown_publication", "new_head", "other_run", "review_mismatch", "principal_revoked", "connection_revoked", "endpoint_revoked", "ownership_changed"] as const)( + "does not derive a published GitHub completion wait after %s", async gate => { + const fixture = await seedPublishedReviewCompletionTurn(); + if (gate === "missing_publication") await db.delete(chatActions).where(eq(chatActions.companyId, fixture.companyId)); + if (gate === "pending_publication" || gate === "unknown_publication") await db.update(chatActions) + .set({ status: gate === "pending_publication" ? "received" : "failed", result: { deliveryUnknown: true } }) + .where(eq(chatActions.companyId, fixture.companyId)); + if (gate === "new_head") await db.insert(chatGitHubReviews).values({ ...fixture.review, id: randomUUID(), + runId: null, deliveryId: randomUUID(), headSha: "c".repeat(40), state: "queued", assessment: null, + createdAt: new Date(Date.now() + 1000) }); + if (gate === "other_run") { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + const previousId = randomUUID(); + await db.insert(heartbeatRuns).values({ ...run!, id: previousId, activeIdentityContextId: null, + status: "succeeded", finishedAt: new Date() }); + await db.update(chatGitHubReviews).set({ runId: previousId }).where(eq(chatGitHubReviews.id, fixture.review.id)); + } + if (gate === "review_mismatch") await db.update(chatGitHubReviews).set({ assessment: { + ...fixture.review.assessment!, reviewedCommit: "f".repeat(40), + } }).where(eq(chatGitHubReviews.id, fixture.review.id)); + if (gate === "principal_revoked") await db.update(companyMemberships).set({ status: "suspended" }) + .where(eq(companyMemberships.companyId, fixture.companyId)); + if (gate === "connection_revoked") await db.update(toolConnections).set({ enabled: false }) + .where(eq(toolConnections.companyId, fixture.companyId)); + if (gate === "endpoint_revoked") await db.update(chatEndpoints).set({ status: "revoked" }) + .where(eq(chatEndpoints.id, fixture.endpointId)); + if (gate === "ownership_changed") await db.update(issues).set({ executionRunId: null }) + .where(eq(issues.id, fixture.issueId)); + const authority = new PaperclipRunnerToolAuthority(db, fixture); + const { resolveCompletionContinuation } = await import(new URL("../../../../packages/paperclip-runner/src/drivers/completion-continuation.ts", import.meta.url).href); + await expect(resolveCompletionContinuation({ ...fixture.result, continuation: undefined }, fixture.runId, + () => authority.execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} }))).rejects.toThrow(); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toHaveLength(0); + }, + ); + + it("serializes an actual connection disable against the final status committer proof", async () => { + const fixture = await seedPublishedReviewCompletionTurn(); + const authority = new PaperclipRunnerToolAuthority(db, fixture); + const { resolveCompletionContinuation } = await import(new URL("../../../../packages/paperclip-runner/src/drivers/completion-continuation.ts", import.meta.url).href); + const result = await resolveCompletionContinuation({ ...fixture.result, continuation: null }, fixture.runId, + () => authority.execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} })) as PrpStructuredRunResult; + await nativeCompletionFeedback(db, fixture.runId, result); + await fixture.port.completeRun({ result, terminal: fixture.terminal, callerResultId: `wait-result-${fixture.runId}` }); + const [connection] = await db.select().from(toolConnections).where(eq(toolConnections.companyId, fixture.companyId)); + const applicationName = `review-wait-race-${fixture.runId}`; + const writerDb = createDb(externalTestDatabaseUrl ?? temporary!.connectionString, { applicationName }); + let writer: Promise | undefined; + let writerFinished = false, serialized = false; + const readProof = publishedReviewAuthority.readInheritedGitHubReviewWait; + const readForCommit = chatWaitAuthority.readPublishedGitHubReviewCompletionContinuationInTransaction; + const commitGuard = vi.spyOn(chatWaitAuthority, "readPublishedGitHubReviewCompletionContinuationInTransaction") + .mockImplementationOnce(async (...args) => { + const [committer] = await args[0].execute(sql`select pg_backend_pid() as pid`); + const proofGuard = vi.spyOn(publishedReviewAuthority, "readInheritedGitHubReviewWait") + .mockImplementationOnce(async (...proofArgs) => { + const proof = await readProof(...proofArgs); + // This is the same maintained writer used by PATCH tool-connections. + // It does not borrow the task's issue lock to disable a chat connection. + writer = toolAccessService(writerDb).updateConnection(connection!.id, { enabled: false }) + .finally(() => { writerFinished = true; }); + await vi.waitFor(async () => { + const waiting = await db.execute(sql`select query from pg_stat_activity + where application_name = ${applicationName} and wait_event_type = 'Lock'`); + expect(writerFinished || waiting.length > 0).toBe(true); + if (waiting.length) expect(String(waiting[0]!.query)).toContain('update "tool_connections"'); + serialized = !writerFinished && waiting.length > 0; + if (serialized) { + const blockers = await db.execute(sql`select pg_blocking_pids(pid) as blockers from pg_stat_activity where application_name = ${applicationName} and wait_event_type = 'Lock'`); + expect(blockers[0]!.blockers).toContain(Number(committer!.pid)); + } + }); + return proof; + }); + try { return await readForCommit(...args); } + finally { proofGuard.mockRestore(); } + }); + try { + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + await writer; + expect(serialized).toBe(true); + expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0]).toMatchObject({ status: "in_review" }); + expect((await db.select().from(toolConnections).where(eq(toolConnections.id, connection!.id)))[0]).toMatchObject({ enabled: false }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toHaveLength(0); + } finally { commitGuard.mockRestore(); await Promise.allSettled(writer ? [writer] : []); await writerDb.$client.end({ timeout: 1 }); } + }); + + it.each((["current", "resume", "reply"] as const).flatMap(turn => + (["before_feedback", "before_finalization", "during_commit"] as const).map(phase => ({ turn, phase }))))( + "rechecks a projected GitHub completion wait $turn $phase", async ({ turn, phase }) => { + const fixture = turn === "current" ? await seedPublishedReviewCompletionTurn() : await seedInheritedReviewCompletionTurn(turn === "resume"); + const reply = turn === "reply" ? await seedCurrentAuthorReply(fixture as Awaited>) : null; + const authority = new PaperclipRunnerToolAuthority(db, fixture); + const { resolveCompletionContinuation } = await import(new URL("../../../../packages/paperclip-runner/src/drivers/completion-continuation.ts", import.meta.url).href); + const result = await resolveCompletionContinuation({ ...fixture.result, continuation: null }, fixture.runId, + () => authority.execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} })) as PrpStructuredRunResult; + const revoke = () => reply + ? db.update(chatActions).set({ status: "failed", result: { deliveryUnknown: true } }).where(eq(chatActions.id, reply.id)) + : turn === "resume" ? db.update(chatActions).set({ status: "received" }).where(eq(chatActions.companyId, fixture.companyId)) + : db.update(toolConnections).set({ enabled: false }).where(eq(toolConnections.companyId, fixture.companyId)); + if (phase === "before_feedback") { + await revoke(); + await expect(nativeCompletionFeedback(db, fixture.runId, result)).rejects.toThrow("no longer authorized"); + return; + } + await nativeCompletionFeedback(db, fixture.runId, result); + await fixture.port.completeRun({ result, terminal: fixture.terminal, callerResultId: `wait-result-${fixture.runId}` }); + let guard: ReturnType | undefined; + if (phase === "before_finalization") await revoke(); + if (phase === "during_commit") { + const read = chatWaitAuthority.readPublishedGitHubReviewCompletionContinuationInTransaction; + guard = vi.spyOn(chatWaitAuthority, "readPublishedGitHubReviewCompletionContinuationInTransaction") + .mockImplementation(async (...args) => { + if (reply) await args[0].update(chatActions).set({ status: "failed", result: { deliveryUnknown: true } }).where(eq(chatActions.id, reply.id)); + else if (turn === "resume") await args[0].update(chatActions).set({ status: "received" }).where(eq(chatActions.companyId, fixture.companyId)); + else await args[0].update(toolConnections).set({ enabled: false }).where(eq(toolConnections.companyId, fixture.companyId)); + return read(...args); + }); + } + try { + const finalized = await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0]).toMatchObject({ status: "in_progress" }); + const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, fixture.runId)); + if (phase === "during_commit") { + expect(finalized).toMatchObject({ phase: "retryable_failure" }); + expect(decisions).toHaveLength(0); + } else expect(decisions.at(-1)).toMatchObject({ reasonCode: "external_chat_response_wait_authorization_lost" }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toHaveLength(0); + } finally { guard?.mockRestore(); } + }, + ); + it("admits a real repository-access request to Mine until it is resolved", async () => { const fixture = await seedWaitTurn("github", [{ kind: "approval", ownerClass: "human", summary: "Grant repository access so I can read the diff." }], { reportedWorkDisposition: "needs_review", continuation: undefined, diff --git a/server/src/services/native-runtime/external-review-wait.ts b/server/src/services/native-runtime/external-review-wait.ts index 4a63ad6510d..b59aed390c3 100644 --- a/server/src/services/native-runtime/external-review-wait.ts +++ b/server/src/services/native-runtime/external-review-wait.ts @@ -1,5 +1,6 @@ import { and, eq, inArray, sql } from "drizzle-orm"; import { chatConversations, chatEndpoints, heartbeatRuns, issues, statusDecisions, type Db } from "@paperclipai/db"; +import { readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js"; import { authorizeChatConversationForBoundRun } from "./chat-attachment-reuse.js"; import { EXTERNAL_CHAT_QUESTION_RESPONSE_KEY } from "./external-chat-question-response.js"; @@ -30,6 +31,8 @@ export async function listExternalReviewWaits(db: Db, companyId: string, issueId runId: heartbeatRuns.id, agentId: heartbeatRuns.agentId, contextSnapshot: authorizationContext, + responsibleUserId: heartbeatRuns.responsibleUserId, + activeIdentityContextId: heartbeatRuns.activeIdentityContextId, }).from(issues) .innerJoin(statusDecisions, and( eq(statusDecisions.id, issues.lastStatusDecisionId), @@ -58,9 +61,14 @@ export async function listExternalReviewWaits(db: Db, companyId: string, issueId eq(chatEndpoints.status, "active"), )) .where(and(eq(issues.companyId, companyId), inArray(issues.id, issueIds), eq(issues.status, "in_review"))); - const waits = await Promise.all(candidates.map(async ({ runId, agentId, contextSnapshot, ...wait }) => { + const waits = await Promise.all(candidates.map(async ({ runId, agentId, contextSnapshot, responsibleUserId, activeIdentityContextId, ...wait }) => { try { - await authorizeChatConversationForBoundRun(db, { companyId, issueId: wait.issueId, runId, agentId }, contextSnapshot, "read"); + const inheritedWait = await readInheritedGitHubReviewWait(db, { + id: runId, companyId, agentId, responsibleUserId, activeIdentityContextId, + nativeIssueId: wait.issueId, issueId: wait.issueId, contextSnapshot, + }); + if (inheritedWait?.kind === "revoked") return null; + await authorizeChatConversationForBoundRun(db, { companyId, issueId: wait.issueId, runId, agentId }, inheritedWait?.context ?? contextSnapshot, "read"); return wait; } catch (error) { if (error instanceof Error && ["paperclip_runner_chat_attachment_binding_denied", diff --git a/server/src/services/native-runtime/native-completion-feedback.ts b/server/src/services/native-runtime/native-completion-feedback.ts index 407201931f3..9bb94b39c72 100644 --- a/server/src/services/native-runtime/native-completion-feedback.ts +++ b/server/src/services/native-runtime/native-completion-feedback.ts @@ -9,7 +9,7 @@ import { evaluateAgentInvokabilityFromDb } from "../agent-invokability.js"; import { issueService } from "../issues.js"; import { isConversation } from "../agent-conversations.js"; import { issueTreeControlService } from "../issue-tree-control.js"; -import { resolveExternalChatResponseWaitAuthorization } from "./chat-attachment-reuse.js"; +import { isPublishedGitHubReviewCompletionContinuation, readPublishedGitHubReviewCompletionContinuation, resolveExternalChatResponseWaitAuthorization } from "./chat-attachment-reuse.js"; import { getNativeReviewAssignment, readNativeReviewAssignmentContext } from "./native-review-participant.js"; import { and, eq, inArray, notInArray } from "drizzle-orm"; import { @@ -68,6 +68,12 @@ export async function nativeCompletionFeedback( ) .then((rows) => rows[0]); if (!issue) throw new Error("Completion task no longer exists."); + if (result.reportedWorkDisposition === "yielded" && isPublishedGitHubReviewCompletionContinuation(result.continuation)) { + const current = await readPublishedGitHubReviewCompletionContinuation({ db, + binding: { companyId: run.companyId, issueId: issue.id, runId, agentId: run.agentId } }); + if (!current || current.idempotencyKey !== result.continuation!.idempotencyKey) + throw new Error("The published GitHub review wait is no longer authorized for this run. Reassess current task state without repeating the published review."); + } const acceptedHandoff = await assertNativeCaseHandoffResult(db, runId, issue.id, result); if (acceptedHandoff) return "The persisted case handoff controls this turn. End the turn; the task remains open for its next agent."; const reviewContext = readNativeReviewAssignmentContext(run.contextSnapshot); diff --git a/server/src/services/native-runtime/native-run-finalizer.ts b/server/src/services/native-runtime/native-run-finalizer.ts index 5931662d7db..4ade80add27 100644 --- a/server/src/services/native-runtime/native-run-finalizer.ts +++ b/server/src/services/native-runtime/native-run-finalizer.ts @@ -3,7 +3,7 @@ import { notifyExecutionRunWork } from "../execution-control-notifications.js"; import { reconcileMuseNativeTerminalRun } from "./muse-terminal-reconciliation.js"; import { assertNativeCaseHandoffResult } from "./native-case-handoff.js"; import { scheduledIssueMonitorWait } from "../issue-monitors.js"; -import { hasPublishedGitHubReview } from "../chat-channels/github/review-wait.js"; +import { hasPublishedGitHubReview, readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js"; import { readNativeMonitorDelivery, readNativePreservedMonitorWait } from "./native-monitor-delivery.js"; import { isNativePlanWaitResult, readNativePlanWait } from "./native-plan-wait.js"; import { hasPendingNativeChildCompletion } from "./native-child-completion-delivery.js"; @@ -57,7 +57,7 @@ import { } from "./native-board-response-wait.js"; import { emitAgentTaskRun } from "../agent-task-run-telemetry.js"; import { isNewTerminalRunFailure, reportRunFailure } from "../run-failure-report.js"; -import { resolveExternalChatResponseWaitAuthorization } from "./chat-attachment-reuse.js"; +import { isPublishedGitHubReviewCompletionContinuation, readPublishedGitHubReviewCompletionContinuation, resolveExternalChatResponseWaitAuthorization } from "./chat-attachment-reuse.js"; import { authorizeNativeChatReviewPresentation, hasMaterializedNativeReviewResponse, @@ -1317,10 +1317,17 @@ export async function finalizeNativeRun(input: { .then((rows) => rows[0] ?? null); if (!authoritativeIssue) throw new Error("native_finalization_issue_missing"); + const publishedReviewContinuation = assessment.reportedDisposition === "yielded" && + isPublishedGitHubReviewCompletionContinuation(assessment.continuation) + ? await readPublishedGitHubReviewCompletionContinuation({ db: input.db, + binding: { companyId: run.companyId, issueId: authoritativeIssue.id, runId: run.id, agentId: run.agentId } }) + : null; const externalChatResponseWaitAuthorization = assessment.reportedDisposition === "yielded" && assessment.continuation?.kind === "response_wake" - ? await resolveExternalChatResponseWaitAuthorization({ + ? isPublishedGitHubReviewCompletionContinuation(assessment.continuation) + ? publishedReviewContinuation?.idempotencyKey === assessment.continuation.idempotencyKey ? "authorized" : "revoked" + : await resolveExternalChatResponseWaitAuthorization({ db: input.db, binding: { companyId: run.companyId, @@ -1452,7 +1459,7 @@ export async function finalizeNativeRun(input: { (externalChatResponseWaitAuthorization === "authorized" || monitorWaitAt !== null) ? await hasPublishedGitHubReview(input.db, { companyId: run.companyId, issueId: authoritativeIssue.id, runId: run.id, - }) : false, + }) || (await readInheritedGitHubReviewWait(input.db, run))?.kind === "published" : false, boardResponseWaitAuthorized: boardResponseWait !== null, boardResponseWaitOrigin: boardResponseWaitOrigin !== null, isConversation: isConversation(authoritativeIssue), @@ -1554,7 +1561,7 @@ export async function finalizeNativeRun(input: { : undefined, requireExternalChatResponseWaitAuthorization: decision.reasonCode === "external_chat_response_waiting" - ? { agentId: run.agentId } + ? { agentId: run.agentId, ...(publishedReviewContinuation ? { completionContinuationKey: publishedReviewContinuation.idempotencyKey } : {}) } : undefined, reviewResponsePresentation: decision.reasonCode === "governed_response_waiting" && diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index 31f65f12fcd..cc535813f77 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -103,6 +103,7 @@ import { type CurrentWakeCommentsBinding, } from "./current-wake-comments.js"; import { + readPublishedGitHubReviewCompletionContinuation, authorizeChatAttachmentReuse, authorizeChatConversationForBoundRun, LIST_CHAT_ATTACHMENTS_TOOL_DEFINITION, @@ -687,6 +688,7 @@ export class PaperclipRunnerToolAuthority { } : !this.binding.nativeReview && context.issue.workMode === "standard" && context.issue.executionRunId === this.binding.runId ? await pendingTaskRepositoryContinuation(this.db, { companyId: this.binding.companyId, issueId: this.binding.issueId, runId: this.binding.runId, workspaceId: context.issue.executionWorkspaceId }) + ?? await readPublishedGitHubReviewCompletionContinuation({ db: this.db, binding: this.binding }) : null, childTasks: visibleChildTasks.slice(0, 100).map(redactedTask), childTasksTruncated: childTasks.length > 100, diff --git a/server/src/services/native-runtime/status-decision-committer.ts b/server/src/services/native-runtime/status-decision-committer.ts index c7684448cbf..0e34904c5b3 100644 --- a/server/src/services/native-runtime/status-decision-committer.ts +++ b/server/src/services/native-runtime/status-decision-committer.ts @@ -1,5 +1,5 @@ import { scheduledIssueMonitorWait } from "../issue-monitors.js"; -import { hasPublishedGitHubReview } from "../chat-channels/github/review-wait.js"; +import { hasPublishedGitHubReview, readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js"; import { pendingNativeGovernance } from "./native-governance.js"; import { findSupersededExternalReviewRequests, supersedeExternalReviewRequestsInTransaction } from "./superseded-external-review-requests.js"; import { normalizePrpResultSignals, type PrpStructuredRunResult } from "../../vendor/paperclip-runner/index.js"; @@ -46,6 +46,7 @@ import { restoreNativeChatReviewPresentationInTransaction, } from "./native-chat-review-presentation.js"; import { + readPublishedGitHubReviewCompletionContinuationInTransaction, isExternalChatWaitAuthorizationContention, resolveExternalChatResponseWaitAuthorizationInTransaction, } from "./chat-attachment-reuse.js"; @@ -1561,7 +1562,7 @@ export async function commitNativeStatusDecision(input: { preMaterializedEffects?: NativeMaterializedStatusEffect[]; supersedesCommittedDecisionId?: string; supersededExternalReviewRequests?: { agentId: string; resultId: string; interactionIds: string[] }; - requireExternalChatResponseWaitAuthorization?: { agentId: string }; + requireExternalChatResponseWaitAuthorization?: { agentId: string; completionContinuationKey?: string }; requirePlanWaitSource?: NativePlanWaitSource; requireMonitorWait?: { agentId: string; nextCheckAt: string }; requireMonitorDelivery?: NativeMonitorDelivery; @@ -1658,8 +1659,23 @@ export async function commitNativeStatusDecision(input: { } if (reasonCode === "scheduled_monitor_waiting") { const expected = input.requireMonitorWait; - const publishedReviewWait = input.decision.statusAction === "in_review" && input.decision.toStatus === "in_review" && - await hasPublishedGitHubReview(tx as unknown as Db, input); + const inReview = input.decision.statusAction === "in_review" && input.decision.toStatus === "in_review"; + let publishedReviewWait = inReview && await hasPublishedGitHubReview(tx as unknown as Db, input); + if (expected && inReview && !publishedReviewWait) { + // Arbitration may reuse a previous publication. Prove that same source + // here, while retaining the independently saved monitor and its owner. + const [run] = await tx.select({ + id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + responsibleUserId: heartbeatRuns.responsibleUserId, + activeIdentityContextId: heartbeatRuns.activeIdentityContextId, + nativeIssueId: heartbeatRuns.nativeIssueId, issueId: heartbeatRuns.issueId, + }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.agentId, expected.agentId), + )).limit(1); + publishedReviewWait = !!run && (await readInheritedGitHubReviewWait(tx as unknown as Db, + { ...run, contextSnapshot: null }))?.kind === "published"; + } if (!expected || issue.workMode !== "standard" || issue.executionRunId !== input.runId || (input.decision.statusAction !== "preserve" && !publishedReviewWait) || input.decision.effects.length !== 1 || input.decision.effects[0]?.kind !== "release_checkout" || @@ -1731,6 +1747,13 @@ export async function commitNativeStatusDecision(input: { if (authorization !== "authorized") { throw new NativeStatusRaceError(); } + if (input.requireExternalChatResponseWaitAuthorization.completionContinuationKey) { + const current = await readPublishedGitHubReviewCompletionContinuationInTransaction(tx as unknown as Db, + { companyId: input.companyId, issueId: input.issueId, runId: input.runId, + agentId: input.requireExternalChatResponseWaitAuthorization.agentId }, "nonblocking"); + if (current?.idempotencyKey !== input.requireExternalChatResponseWaitAuthorization.completionContinuationKey) + throw new NativeStatusRaceError(); + } } const passiveBoardResponseWait = reasonCode === "board_response_waiting" || diff --git a/server/src/services/run-identity.ts b/server/src/services/run-identity.ts index 2f1e9e97d65..166b1ae8cac 100644 --- a/server/src/services/run-identity.ts +++ b/server/src/services/run-identity.ts @@ -1,3 +1,4 @@ +import { readTaskRetryIntent, readTaskRetryBoardMessageOrigin } from "./task-retry.js"; import { and, asc, desc, eq, getTableColumns, inArray, sql } from "drizzle-orm"; import { agentWakeupRequests, @@ -201,6 +202,39 @@ export async function initializeRunIdentity( return current!; } const operatorIdentity = await explicitOperatorRunIdentity(tx, run); + let retryOrigin: string | null = null; + const retryId = run.contextSnapshot?.taskRetryRequestId; + if (typeof retryId === "string") { + const [receipt] = await tx.select().from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.id, retryId), eq(agentWakeupRequests.companyId, input.companyId), + eq(agentWakeupRequests.agentId, run.agentId))); + const retry = readTaskRetryIntent(receipt?.payload); + if (run.contextSnapshot?.boardMessageRetry === true || retry?.boardMessageOrigin) { + if (!retry?.boardMessageOrigin) throw forbidden("This saved message can no longer be retried safely. Send a new message on the task to continue from saved work."); + const [dispatch] = run.wakeupRequestId ? await tx.select().from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.id, run.wakeupRequestId), eq(agentWakeupRequests.companyId, input.companyId), + eq(agentWakeupRequests.agentId, run.agentId), eq(agentWakeupRequests.runId, run.id), + eq(agentWakeupRequests.status, "claimed"), eq(agentWakeupRequests.reason, "task_retry"), + eq(agentWakeupRequests.idempotencyKey, `task-retry-dispatch:${receipt!.id}`), + eq(agentWakeupRequests.requestedByActorType, "user"), eq(agentWakeupRequests.requestedByActorId, retry.actorId))) : []; + const [task] = input.issueId ? await tx.select({ id: issues.id }).from(issues).where(and( + eq(issues.id, input.issueId), eq(issues.companyId, input.companyId), eq(issues.assigneeAgentId, run.agentId), + eq(issues.executionRunId, run.id))) : []; + const origin = input.issueId && run.status === "running" && task && dispatch && + [run.issueId, run.nativeIssueId, run.contextSnapshot?.issueId, run.contextSnapshot?.taskId] + .every(id => id === null || id === undefined || id === input.issueId) && receipt!.runId === run.id && receipt!.status === "coalesced" && + receipt!.reason === "task_retry" && receipt!.requestedByActorType === "user" && + run.contextSnapshot?.previousRunId === retry.sourceRunId && retry.generationId === receipt!.id && !retry.chatActionId && !retry.canonicalIntentId && + receipt!.payload?.issueId === input.issueId && retry.actorId === receipt!.requestedByActorId && + await readTaskRetryBoardMessageOrigin(tx as unknown as Db, { companyId: input.companyId, issueId: input.issueId, + agentId: run.agentId, sourceRunId: retry.sourceRunId, actorId: retry.actorId }); + if (!origin || origin.commentId !== retry.boardMessageOrigin.commentId || + origin.identityContextId !== retry.boardMessageOrigin.identityContextId) { + throw forbidden("This saved message can no longer be retried safely. Send a new message on the task to continue from saved work."); + } + retryOrigin = origin.identityContextId; + } + } const [parent] = input.parentRunId ? await tx .select() @@ -226,7 +260,7 @@ export async function initializeRunIdentity( ) : []; const parentId = operatorIdentity ? null : ( - interaction?.sourceIdentityContextId ?? input.parentContextId ?? parent?.activeIdentityContextId + retryOrigin ?? interaction?.sourceIdentityContextId ?? input.parentContextId ?? parent?.activeIdentityContextId ); const [origin] = parentId ? await tx diff --git a/server/src/services/task-retry.ts b/server/src/services/task-retry.ts index f1ec119386f..1d56d6d49b3 100644 --- a/server/src/services/task-retry.ts +++ b/server/src/services/task-retry.ts @@ -3,7 +3,7 @@ import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; import { z } from "zod"; import { randomUUID } from "node:crypto"; import { and, desc, eq, inArray, or, sql } from "drizzle-orm"; -import { agentWakeupRequests, chatActions, chatConversations, chatEndpoints, heartbeatRuns, issues, issueRecoveryActions, type Db } from "@paperclipai/db"; +import { agentWakeupRequests, chatActions, chatConversations, chatEndpoints, heartbeatRuns, issues, issueComments, runIdentityContexts, issueRecoveryActions, type Db } from "@paperclipai/db"; import { conflict, forbidden, notFound } from "../errors.js"; import { persistActivity } from "./activity-log.js"; import { createSystemRecovery, readSystemRecovery } from "./system-recovery-state.js"; @@ -13,6 +13,7 @@ import { notifyChatActionWork } from "./chat-work-notifications.js"; import { authorizationService } from "./authorization.js"; import { preserveNativeFailedWorkspace } from "./native-runtime/native-failed-workspace-recovery.js"; import { NativeWorkspaceFinalizationBusyError } from "./native-runtime/native-workspace-finalization-ownership.js"; +import { adapterExecutionControls } from "./adapter-execution-control.js"; import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js"; export async function taskRetryMutationAllowed(db: Db, input: { @@ -42,6 +43,7 @@ const taskRetryIntentSchema = z.object({ schema: z.literal("paperclip.task-retry.v1"), generationId: z.string().uuid(), sourceRunId: z.string().uuid(), actorId: z.string().min(1), actorSource: z.enum(["local_implicit", "session", "board_key", "cloud_tenant"]).default("session"), strategy: z.enum(["evaluate", "continue", "fresh_isolated_workspace", "finalize_result"]), chatActionId: z.string().uuid().optional(), canonicalIntentId: z.string().uuid().optional(), + boardMessageOrigin: z.object({ commentId: z.string().uuid(), identityContextId: z.string().uuid() }).optional(), }); export type TaskRetryIntent = z.infer; export function readTaskRetryIntent(payload: Record | null | undefined): TaskRetryIntent | null { @@ -49,6 +51,126 @@ export function readTaskRetryIntent(payload: Record | null | un return result.success ? result.data : null; } +/** A board message has its own accepted user identity, not an external inbound + * receipt. Certify its exact unchanged source and retained pre-provider evidence; + * this selects the canonical Retry owner and never substitutes chat authority. + * Normal queue containment and current permissions still govern dispatch. */ +export async function readTaskRetryBoardMessageOrigin(db: Db, input: { + companyId: string; issueId: string; agentId: string; sourceRunId: string; actorId: string; +}) { + return readBoardMessageRetrySource(db, input, new Set()); +} + +async function readBoardMessageRetrySource(db: Db, input: { + companyId: string; issueId: string; agentId: string; sourceRunId: string; actorId: string; +}, seen: Set): Promise<{ commentId: string; identityContextId: string } | null> { + if (seen.size >= 20 || seen.has(input.sourceRunId) || adapterExecutionControls.has(input.sourceRunId)) return null; + seen.add(input.sourceRunId); + const [task] = await db.select({ id: issues.id }).from(issues).where(and( + eq(issues.id, input.issueId), eq(issues.companyId, input.companyId), eq(issues.assigneeAgentId, input.agentId), + sql`${issues.status} not in ('done', 'cancelled')`)); + if (!task) return null; + const sourceConditions = [ + eq(heartbeatRuns.id, input.sourceRunId), eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.responsibleUserId, input.actorId), + eq(heartbeatRuns.status, "failed"), eq(heartbeatRuns.runtimeMode, "legacy"), eq(heartbeatRuns.errorCode, "setup_failed"), + sql`${heartbeatRuns.finishedAt} is not null`, + sql`(${heartbeatRuns.issueId} is null or ${heartbeatRuns.issueId} = ${input.issueId}::uuid)`, + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${input.issueId}`, + sql`(${heartbeatRuns.contextSnapshot}->>'taskId' is null or ${heartbeatRuns.contextSnapshot}->>'taskId' = ${input.issueId})`, + sql`${heartbeatRuns.resultJson}#>>'{executionRecovery,kind}' = 'bootstrap'`, + sql`${heartbeatRuns.resultJson}#>'{executionRecovery,providerWorkStarted}' = 'false'::jsonb`, + sql`${heartbeatRuns.resultJson}->>'stopReason' = 'adapter_failed'`, + sql`not exists (select 1 from jsonb_object_keys(case when jsonb_typeof(${heartbeatRuns.resultJson}) = 'object' + then ${heartbeatRuns.resultJson} else '{}'::jsonb end) key + where key not in ('effectiveTimeoutSec', 'executionRecovery', 'stopReason', 'timeoutConfigured', 'timeoutFired', 'timeoutSource'))`, + sql`${heartbeatRuns.runnerProfileJson}->'adapterDispatch' = '{"adapterType":"paperclip_runner"}'::jsonb`, + sql`not exists (select 1 from jsonb_object_keys(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' + then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) key + where key not in ('adapterDispatch', 'aiConnectionNonAssigneeCommentWake', 'chatControlRecoveryAdmission'))`, + ...[heartbeatRuns.nativeIssueId, heartbeatRuns.nativeSessionId, heartbeatRuns.nativePhase, + heartbeatRuns.runtimeModeResolvedAt, heartbeatRuns.runtimeModeResolverVersion, heartbeatRuns.runtimeModeReason, heartbeatRuns.driverKind, heartbeatRuns.driverVersion, + heartbeatRuns.completionContractId, heartbeatRuns.runnerInstanceId, heartbeatRuns.sessionIdAfter, + heartbeatRuns.externalRunId, heartbeatRuns.processPid, heartbeatRuns.processGroupId, heartbeatRuns.processStartedAt, + heartbeatRuns.logStore, heartbeatRuns.logRef, heartbeatRuns.logBytes, heartbeatRuns.logSha256, heartbeatRuns.stdoutExcerpt, + heartbeatRuns.stderrExcerpt, heartbeatRuns.lastOutputAt, heartbeatRuns.lastOutputBytes, + heartbeatRuns.lastOutputStream, heartbeatRuns.exitCode, heartbeatRuns.signal].map(column => sql`${column} is null`), + sql`${heartbeatRuns.lastOutputSeq} = 0`, sql`not ${heartbeatRuns.logCompressed}`, + sql`(${heartbeatRuns.usageJson} is null or ${heartbeatRuns.usageJson} - 'ledgerScope' = '{}'::jsonb)`, + sql`(${heartbeatRuns.controllerLeaseExpiresAt} is null or ${heartbeatRuns.controllerLeaseExpiresAt} <= now())`, + sql`not exists (select 1 from native_run_results where company_id = ${input.companyId}::uuid and run_id = ${input.sourceRunId}::uuid)`, + sql`not exists (select 1 from native_run_finalizations where company_id = ${input.companyId}::uuid and run_id = ${input.sourceRunId}::uuid)`, + sql`not exists (select 1 from environment_leases where company_id = ${input.companyId}::uuid and heartbeat_run_id = ${input.sourceRunId}::uuid)`, + sql`exists (select 1 from heartbeat_run_events where company_id = ${input.companyId}::uuid and run_id = ${input.sourceRunId}::uuid + and event_type = 'error' and stream = 'system' and source_event_id is null and message = ${heartbeatRuns.error})`, + sql`exists (select 1 from heartbeat_run_events where company_id = ${input.companyId}::uuid and run_id = ${input.sourceRunId}::uuid + and event_type = 'lifecycle' and stream = 'system' and source_event_id is null and message = 'task.run.measured' + and payload->>'durableSettlement' = 'true' and payload->>'outcome' = 'failed')`, + sql`not exists (select 1 from heartbeat_run_events where company_id = ${input.companyId}::uuid and run_id = ${input.sourceRunId}::uuid + and (stream <> 'system' or source_event_id is not null or source_instance_id is not null or source_seq is not null + or source_payload_sha256 is not null or protocol_schema_version is not null or event_type not in ('error', 'agent_observability.unavailable', 'lifecycle') + or (event_type = 'lifecycle' and (message <> 'task.run.measured' or payload->>'durableSettlement' is distinct from 'true')) + or (event_type = 'error' and message is distinct from ${heartbeatRuns.error})))`, + ]; + const [origin] = await db.select({ commentId: issueComments.id, identityContextId: runIdentityContexts.id }) + .from(heartbeatRuns) + .innerJoin(runIdentityContexts, and(eq(runIdentityContexts.id, heartbeatRuns.activeIdentityContextId), + eq(runIdentityContexts.companyId, input.companyId), eq(runIdentityContexts.runId, heartbeatRuns.id), + eq(runIdentityContexts.status, "accepted"), eq(runIdentityContexts.responsibleUserId, input.actorId))) + .innerJoin(issueComments, and(eq(issueComments.id, runIdentityContexts.messageId), + eq(issueComments.companyId, input.companyId), eq(issueComments.issueId, input.issueId), + eq(issueComments.authorType, "user"), eq(issueComments.authorUserId, input.actorId), + sql`${issueComments.createdByRunId} is null`, sql`${issueComments.deletedAt} is null`, + eq(issueComments.createdAt, issueComments.updatedAt), sql`nullif(trim(${issueComments.body}), '') is not null`)) + .innerJoin(agentWakeupRequests, and(eq(agentWakeupRequests.id, heartbeatRuns.wakeupRequestId), + eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.agentId, input.agentId), + eq(agentWakeupRequests.runId, heartbeatRuns.id), eq(agentWakeupRequests.requestedByActorType, "user"), + eq(agentWakeupRequests.requestedByActorId, input.actorId), eq(agentWakeupRequests.reason, "issue_reopened_via_comment"), eq(agentWakeupRequests.status, "failed"), + sql`${agentWakeupRequests.payload}->>'issueId' = ${input.issueId}`, + sql`${agentWakeupRequests.payload}->>'commentId' = ${issueComments.id}::text`)) + .where(and(...sourceConditions, + sql`${heartbeatRuns.contextSnapshot}->>'source' = 'issue.comment.reopen'`, + sql`${heartbeatRuns.contextSnapshot}->>'wakeCommentId' = ${issueComments.id}::text`, + sql`${heartbeatRuns.contextSnapshot}->'wakeCommentIds' = jsonb_build_array(${issueComments.id}::text)`, + )).limit(1); + if (origin) return origin; + // A repeated preparation failure must retain the same certified board + // source through its real coalesced intent and failed dispatch receipt. + // Every predecessor is checked again; cycles and long damaged chains fail closed. + const [successor] = await db.select({ identityContextId: heartbeatRuns.activeIdentityContextId, + retryId: sql`${heartbeatRuns.contextSnapshot}->'taskRetryRequestId'`, + previousRunId: sql`${heartbeatRuns.contextSnapshot}->'previousRunId'`, + dispatchKey: agentWakeupRequests.idempotencyKey }).from(heartbeatRuns) + .innerJoin(agentWakeupRequests, and(eq(agentWakeupRequests.id, heartbeatRuns.wakeupRequestId), + eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.agentId, input.agentId), + eq(agentWakeupRequests.runId, heartbeatRuns.id), eq(agentWakeupRequests.status, "failed"), + eq(agentWakeupRequests.reason, "task_retry"), eq(agentWakeupRequests.requestedByActorType, "user"), + eq(agentWakeupRequests.requestedByActorId, input.actorId))) + .where(and(...sourceConditions, sql`${heartbeatRuns.contextSnapshot}->'boardMessageRetry' = 'true'::jsonb`)).limit(1); + if (!successor?.identityContextId || !z.string().uuid().safeParse(successor.retryId).success || + successor.dispatchKey !== `task-retry-dispatch:${successor.retryId}`) return null; + const [receipt] = await db.select().from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.id, String(successor.retryId)), eq(agentWakeupRequests.companyId, input.companyId), + eq(agentWakeupRequests.agentId, input.agentId), eq(agentWakeupRequests.runId, input.sourceRunId), + eq(agentWakeupRequests.status, "coalesced"), eq(agentWakeupRequests.reason, "task_retry"), + eq(agentWakeupRequests.requestedByActorType, "user"), eq(agentWakeupRequests.requestedByActorId, input.actorId), + sql`${agentWakeupRequests.payload}->>'issueId' = ${input.issueId}`)); + const intent = readTaskRetryIntent(receipt?.payload); + if (!intent?.boardMessageOrigin || intent.generationId !== receipt!.id || intent.actorId !== input.actorId || + intent.chatActionId || intent.canonicalIntentId || intent.sourceRunId !== successor.previousRunId) return null; + const [identity] = await db.select({ id: runIdentityContexts.id }).from(runIdentityContexts).where(and( + eq(runIdentityContexts.id, successor.identityContextId), eq(runIdentityContexts.companyId, input.companyId), + eq(runIdentityContexts.runId, input.sourceRunId), eq(runIdentityContexts.responsibleUserId, input.actorId), + eq(runIdentityContexts.status, "accepted"), eq(runIdentityContexts.cause, "task_retry"), + eq(runIdentityContexts.parentContextId, intent.boardMessageOrigin.identityContextId), + sql`${runIdentityContexts.messageId} is null`)).limit(1); + if (!identity) return null; + const parent = await readBoardMessageRetrySource(db, { ...input, sourceRunId: intent.sourceRunId }, seen); + return parent && parent.commentId === intent.boardMessageOrigin.commentId && + parent.identityContextId === intent.boardMessageOrigin.identityContextId + ? { commentId: parent.commentId, identityContextId: identity.id } : null; +} + /** Caller holds the issue dispatch lock. Newer work supersedes an unadmitted * Retry; retiring its authority does not acknowledge or discard saved input. */ export async function retireSupersededTaskRetry(db: Db, input: { @@ -155,13 +277,22 @@ export async function requestTaskRetry(input: { .innerJoin(chatEndpoints, eq(chatEndpoints.id, chatConversations.endpointId)).where(and( eq(chatConversations.companyId, issue.companyId), eq(chatConversations.issueId, issue.id), eq(chatEndpoints.externalExecutionPolicy, "restricted"))).limit(1); + const boardMessageOrigin = chat && !active && !busy + ? await readTaskRetryBoardMessageOrigin(db, { companyId: issue.companyId, + issueId: issue.id, agentId: issue.assigneeAgentId, sourceRunId: latest.id, actorId: input.actorId }) : null; + if (boardMessageOrigin) intent.boardMessageOrigin = boardMessageOrigin; + if (chat && !active && !busy && !boardMessageOrigin && (latest.contextSnapshot?.source === "issue.comment.reopen" || latest.contextSnapshot?.boardMessageRetry === true) && + latest.responsibleUserId === input.actorId) { + throw conflict("This saved message cannot be retried safely. Send a new message on the task to continue from saved work.", + { code: "chat_failed_run_retry_not_authorized", reason: "saved_board_message_not_replayable" }); + } await tx.insert(agentWakeupRequests).values({ id, companyId: issue.companyId, agentId: issue.assigneeAgentId, source: "on_demand", triggerDetail: "manual", reason: "task_retry", idempotencyKey: key, requestedByActorType: "user", requestedByActorId: input.actorId, status: active || busy ? "coalesced" : "deferred_issue_execution", runId: active?.runId ?? (busy ? latest.id : null), payload: { issueId: issue.id, taskRetry: intent }, finishedAt: active || busy ? new Date() : null }); - if (chat && !active && !busy) { + if (chat && !boardMessageOrigin && !active && !busy) { if (!input.prepareChatRetry) throw conflict("Retry is unavailable for this connected conversation."); intent.chatActionId = (await input.prepareChatRetry(db, { companyId: issue.companyId, issueId: issue.id, agentId: issue.assigneeAgentId, failedRunId: latest.id, initiatedByUserId: input.actorId, retryIntentId: id })).actionId; diff --git a/ui/src/hooks/useTaskRetry.test.tsx b/ui/src/hooks/useTaskRetry.test.tsx index 1cbec6ee9fa..c47c9ce3ddc 100644 --- a/ui/src/hooks/useTaskRetry.test.tsx +++ b/ui/src/hooks/useTaskRetry.test.tsx @@ -5,12 +5,17 @@ import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import type { Issue } from "@paperclipai/shared"; import { useTaskRetry } from "./useTaskRetry"; +import { ExecutionBlockerNotice } from "../components/ExecutionBlockerNotice"; +import { ApiError } from "../api/client"; import { issuesApi } from "../api/issues"; vi.mock("../api/issues", () => ({ issuesApi: { retryTask: vi.fn() } })); const issue = { id: "task", interruption: { observedRunId: "failed-run" } } as Issue; function View({ task = issue }: { task?: Issue }) { const retry = useTaskRetry(task); - return <>{retry.optimistic && Thinking}{retry.takingLonger && Taking longer than usual…}; + return <>{retry.optimistic && Thinking}{retry.takingLonger && Taking longer than usual…}{retry.error && }; } describe("optimistic task Retry", () => { let container: HTMLDivElement, root: Root, client: QueryClient; @@ -20,6 +25,19 @@ describe("optimistic task Retry", () => { }); afterEach(async () => { await act(async () => root.unmount()); client.clear(); container.remove(); vi.useRealTimers(); }); async function render(task = issue) { await act(async () => root.render()); } + it("shows the saved-work continuation path after normal Retry is definitively denied", async () => { + const message = "This attempt already recorded a result. Send a new message on the task to continue from the saved work."; + vi.mocked(issuesApi.retryTask).mockRejectedValue(new ApiError(message, 409, + { error: message, details: { code: "chat_failed_run_retry_not_authorized", reason: "accepted_result_without_new_input" } })); + await render(); + await act(async () => container.querySelector("button")!.click()); + expect(issuesApi.retryTask).toHaveBeenCalledExactlyOnceWith("task", { + observedRunId: "failed-run", idempotencyKey: expect.any(String), + }); + expect(container.querySelector('[role="alert"]')?.textContent).toContain(message); + expect(container.textContent).not.toContain("Thinking"); + expect(container.textContent).not.toContain("Taking longer"); + }); it("paints Thinking before admission resolves and shares the same request across double clicks", async () => { let settle!: (value: Awaited>) => void; vi.mocked(issuesApi.retryTask).mockImplementation(() => new Promise(resolve => { settle = resolve; })); From db32bdf98b9910a0137ed8128ac297d2639dd242 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 26 Oct 2026 22:22:57 -0500 Subject: [PATCH 042/133] fix(github): recover review replay and preserve passive summaries (#17726) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit https://entire.com/gh/paperclipai/-/trails/8/changes/paperclip/8 ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - GitHub tools save review assessments and publish their confirmed receipts. > - A timeout before publication staging can leave an invocation with no saved publication. > - Exact replay must repeat only the authorized preflight and converge on one publication. > - PR requests also share an App-owned summary comment. Progress and maintenance must preserve its final content. > - A native turn can intentionally wait for an author without sending another reply. > - This pull request repairs unstaged replay and prevents queued or passive-wait notices from replacing a published assessment. ## Linked Issues or Issue Description Related repairs: Refs #16883, Refs #16284, Refs #16275 recover saved publication receipts. Refs #17183 covers unavailable approved replay output. Refs #17714 fixes a separate publication lock-order problem. **What happened?** Exact review retries can return null `begin_review` output or `github_publication_unavailable` after an upstream read times out before publication staging. Separately, response maintenance can replace a completed PR assessment with “Review queued…” or a generic missing-response notice after an authorized no-new-work reply. The latter run can correctly commit a passive wait while its external summary is still overwritten. **Expected behavior** Exact retries repeat only the authorized preflight. A saved or uncertain publication keeps its existing recovery owner. Queued acknowledgements preserve final App-owned summary content. A committed native passive wait settles its current acknowledgement without writing a response or treating an old summary as a new-run reply. Success before presentation remains held until a durable decision exists; exact current-run confirmed receipts can retire their placeholders. Genuine missing/error responses retain fallback handling. **Steps to reproduce** 1. Hold a GitHub metadata read until an authorized gateway call times out before publication staging. Repeat the exact request. 2. Publish a valid PR assessment. Send an ordinary author reply that asks for no new work and no duplicate comment. 3. Commit the native `response_wake` with a no-comment server presentation decision. Run response maintenance twice. 4. The base returns the invalid replay result or replaces the assessment with progress/fallback text. **Paperclip version or commit** Replay regression: `41c42895fa862c586ff841f8c50200d93ffc83a4`. Maintenance regression: the previous PR head `c7553d2325df53d98d9febff7023ade4b1e704eb`. Both use real PostgreSQL and simulated GitHub responses. The maintenance failure was also observed in normal hosted use. **Deployment mode** Source checkout on macOS arm64 with isolated PostgreSQL 18, followed by a dedicated QA qualification. No production rollout is included. ## What Changed - Route exact `begin_review` replay through its typed owner. Repeat unstaged preflight only with the complete request digest and no successful invocation result. - Preserve current policy, immutable assessments, unique action keys, receipt recovery, and provider leases. - Preserve a verified App/marker-owned final shared PR comment during acknowledgements. New canonical assessments can still replace it at the same or a new head. - Project only a bounded server-committed native wait decision. Settle the current main-PR acknowledgement without provider writes or a fabricated current-run reply receipt. - Hold succeeded native runs while presentation is undecided, including the committed-before-presentation window. An exact current-run confirmed receipt can still retire its placeholder. - Keep missing/error fallback and uncertain publication checks. Update the bot guide, execution semantics, maps, and real-database regressions. ## Verification - **Causal RED:** exact begin replay returned null; exact unstaged submit replay returned unavailable. The new real-database maintenance test then reproduced the exact generic summary overwrite. Its first repair also exposed an earlier queued-acknowledgement overwrite. Both publication boundaries are repaired. Four additional ordering regressions reproduced the generic PATCH while native presentation was undecided; the corrected tests hold until the same row settles. - **Local GREEN:** 196 complete GitHub workflow cases pass on an empty migrated PostgreSQL fixture. This includes queued/running preservation, both success-before-presentation windows, null/scalar/invalid decisions, exact-current versus predecessor receipts, durable acknowledgement settlement, repeated no-op maintenance, eventual genuine missing/error fallback, pending writes and same/new-head canonical replacement. Workspace typecheck, exact-head build, maps and whitespace checks pass. The real database respects the actual selected-column projection. - **Command:** `PAPERCLIP_TEST_DATABASE_URL= pnpm --filter @paperclipai/server exec vitest run src/__tests__/chat-channels.integration.test.ts -t 'GitHub agent review workflow'`. - **CI/review:** historical heads `c7553d2325df53d98d9febff7023ade4b1e704eb` and `4d00d3af6f2ccbda8f38a98f81d392d82dd3c59c` had Greptile 5/5. Final readiness head `2d267f1e4aa0b835a11aede97798fbd800d9cd96` passed independent source review with no actionable finding and fresh Greptile 5/5 with zero unresolved threads; all 54 check runs pass with 2 intended skips; Snyk is successful and the merge state is CLEAN. The dedicated QA destination acceptance also passed. - **Full-suite limitation:** the earlier `pnpm test:run` attempt produced no final summary. Its retained log reports failures in unchanged workspace-runtime tests and 15-second gateway-audit and stale-question timeouts. Their cause remains unknown. Do not treat the targeted results as a full local-suite pass. - **Live evidence and limits:** normal native OpenCode/DeepSeek review delivery published a useful assessment, inline finding and App check at the exact reviewed head. A separate ordinary author reply delivered a useful answer and settled. Those are ordinary delivery proofs, not live injected-timeout proofs. A later no-new-work reply exposed the maintenance overwrite. The final combined source `070e9c4ead9341cd497b23f533a91bf194475e79` includes the exact publication repair and passed normal review → ordinary author reply → quiet destination acceptance. A normal current-head review restored the legitimate assessment; the ordinary no-new-work reply then committed a passive wait without new GitHub tools. The same 1,436-character summary stayed identical while queued/running, after terminal settlement, and more than 3 minutes 50 seconds after successful cleanup. Review rows and receipts stayed unchanged, with no owner, live run or queued input. The timeout fault fixtures prove replay and single-publication safety. The historical first upstream timeout producer remains unknown. The live board reads do not expose the raw acknowledgement action row: its terminal cancellation and next-pass no-op are real-PostgreSQL fixture proofs, while the hosted checks prove actual destination stability. The hosted journey did not inject a timeout or pause the terminal/presentation writer window. Private destinations, raw receipts and internal task identifiers remain outside this PR. ## Risks - Original and repeated metadata reads can overlap. The immutable assessment, action uniqueness and publication lease must remain authoritative. - Missing complete request identity or a deleted completed receipt remains held. The gateway does not reconstruct receipts or relax approval. - Passive suppression requires actual native mode/committed phase, current source ownership and the server's no-comment decision. It applies only to the main PR review surface. Uncertain current writes stay held; other surfaces keep normal placeholder retirement. - There is no migration, credential change, dependency or new retry scheduler. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) in Codex assisted with diagnosis, implementation, tests, and source review. It used reasoning, repository tools and code execution. The context window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/connections/GITHUB-REVIEW-BOT.md | 25 +- doc/execution-semantics.md | 16 ++ maps/agent-access-map/chat-channels.md | 18 +- maps/feature-map/chat-channels.md | 19 +- .../chat-channels.integration.test.ts | 243 +++++++++++++++++- .../services/chat-github-response-comments.ts | 48 +++- server/src/services/tool-gateway.ts | 15 +- 7 files changed, 367 insertions(+), 17 deletions(-) diff --git a/doc/connections/GITHUB-REVIEW-BOT.md b/doc/connections/GITHUB-REVIEW-BOT.md index f63732c56a4..c3c2f036108 100644 --- a/doc/connections/GITHUB-REVIEW-BOT.md +++ b/doc/connections/GITHUB-REVIEW-BOT.md @@ -169,6 +169,19 @@ permissions. Ordinary discussion does not change a review score. Repeat review mentions and pushes continue the existing task; inline replies return to the task owning that thread. +A successful native execution can still be resolving its visible response. +Without a confirmed reply for that run, maintenance waits for its committed +presentation decision before choosing a completion fallback. It then uses the same saved request to settle the wait, +publish a genuine missing-response notice, or report a failed turn. + +A queued or running follow-up acknowledgement preserves an existing App-owned +final PR summary. A new validated assessment can replace it through the review +publication path. When an authorized native turn commits a passive response +wait with no new comment, maintenance settles that request's acknowledgement +without replacing the assessment with a generic completion notice. This does +not count the old summary as a reply from the new run. Missing or failed +responses still receive their normal fallback; uncertain writes remain held. + Event prompts supplement the agent's instructions. Repository content and PR prose are untrusted input and cannot change tool authority or publication policy. The execution records the configuration revision and event context used. @@ -249,9 +262,15 @@ the receipt. Receipt recovery compares the complete saved publication request, including long text and nested coverage entries. Shortened audit previews do not establish -request identity. Changed content or a missing durable publication returns a -conflict without dispatching another write. Exact retries read the current -receipt; the durable worker owns pending publication retries. +request identity. Changed content or a missing completed durable publication +returns a conflict without dispatching another write. Exact retries read the +current receipt; the durable worker owns pending publication retries. + +A timeout during metadata or file reads can occur before publication staging. +The same request can repeat this preflight with its complete request identity. +The saved assessment, unique publication key, and provider lease still prevent +repeat external writes. `begin_review` retries return the run's current review +result. They do not return a shortened audit preview. For a call that required human approval, the durable publication keeps that verified approval for its exact operation. An exact caller retry only reads the diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 17411564865..fe985686b95 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1258,6 +1258,22 @@ apply. Stream closure without a turn terminal is not proof of success. Event replay uses the existing source receipts and never repeats provider work merely to recover recorded output. +Native execution success can precede its durable presentation decision. GitHub +response maintenance without a confirmed current-run reply holds that request +until the native phase is committed and a valid presentation decision exists. +A confirmed current-run receipt can retire its placeholder. The same maintenance owner +then rechecks the settled outcome; elapsed time cannot turn undecided output +into a generic failure notice. Failed execution retains normal error fallback. + +GitHub main-PR response maintenance honors a committed native passive response wait +whose server presentation decision intentionally selects no comment. It settles +the current source's acknowledgement without a provider write or a fabricated +current-run reply receipt. Queued/running acknowledgements preserve an App-owned +final shared PR summary; a new canonical assessment may replace it. A generic +completion fallback must not overwrite that summary after an intentional wait. +Failure, genuinely missing response, and uncertain-publication handling retain +their existing independent checks. + If runnerd synthesizes a result when the provider stops, it publishes that result before the provider-turn terminal and publishes the run terminal last. The adapter can therefore retain the result while the matching turn still has diff --git a/maps/agent-access-map/chat-channels.md b/maps/agent-access-map/chat-channels.md index aaa0b34679b..b2d708898b3 100644 --- a/maps/agent-access-map/chat-channels.md +++ b/maps/agent-access-map/chat-channels.md @@ -115,8 +115,13 @@ GitHub publication recovery (2026-10-14): assigned-agent `submit_review`, `comme and `update_comment` calls reuse their run-, task-, endpoint-, and argument-bound publication on an exact retry. Identity uses the complete durable request rather than the shortened audit preview. A changed suffix, changed nested entry, or -missing publication cannot authorize another write. Exact retries only read the -receipt; the worker owns pending provider retries. Fresh gateway policy and provider authority remain +missing completed publication cannot authorize another write. Exact retries read an +existing receipt; the worker owns pending provider retries. A request interrupted +during metadata or file reads before staging can repeat that preflight with the +same full request identity. The immutable assessment, unique publication and +provider lease still serialize its external write. `begin_review` exact retries +resolve the run's current review and return a typed result instead of an audit +preview. Fresh gateway policy and provider authority remain required. The durable worker resumes transient failures after restart and stops at its retry cap, revoked authority, or a superseded head. A pending receipt is not task completion. Source: @@ -134,6 +139,15 @@ follow normal access checks. Provider-fixture coverage is not live GitHub proof. Native terminal reconciliation resolves the accepted run identity through the existing chat provenance resolver; an older run's reply cannot complete a newer request merely because both requests share the same PR comment. +A terminal native success does not authorize missing-response publication while +its native phase or durable presentation decision remains pending. Maintenance +rechecks the same request after authoritative settlement; failures keep their +separate fallback path. +Maintenance projects the bounded server-committed native passive-wait decision +only after current request ownership. It settles a no-comment acknowledgement +without authorizing a new send or counting an old shared receipt as a current-run +reply. Existing final App/marker-owned PR content survives progress acknowledgements; +new canonical review publication remains the writer for replacement assessments. Speko browser sessions use `POST /api/companies/{companyId}/voice-sessions`, with GET inspection/notification and POST `/{sessionId}/end` operations below diff --git a/maps/feature-map/chat-channels.md b/maps/feature-map/chat-channels.md index a5a490d0a33..087993e8536 100644 --- a/maps/feature-map/chat-channels.md +++ b/maps/feature-map/chat-channels.md @@ -140,6 +140,18 @@ contracts with synthetic providers; record a live journey separately. For a long-lived task, repeat more than 100 continuations and check that tool access remains available. Fail a later GitHub run after a final answer and check that the earlier answer remains intact. +After publishing a PR assessment, deliver an ordinary no-new-work author reply. +Hold native presentation after execution succeeds. Verify maintenance makes no +provider write while undecided, including repeated passes and the explicit +fallback path. Commit either a passive wait, genuine missing response, or failure +and verify that the same request settles once with the corresponding behavior. +Verify the actual shared summary stays unchanged during queued/running +acknowledgements and after the committed passive wait, including maintenance, +restart and cleanup. Confirm the per-request acknowledgement is terminal and a +second maintenance pass makes no write. Then request a real same-head and +changed-head assessment and confirm canonical publication can replace it. +Real-PostgreSQL provider fixtures cover this producer and missing/error/unknown +write safety; record hosted destination acceptance separately. ### `endpoint-management` @@ -174,7 +186,12 @@ write and verify one summary, finding, and App-scoped check. Retry the same tool call and verify it returns the saved receipt. Include text longer than 500 characters and coverage with more than 50 entries. Change only the text suffix or final coverage entry and verify rejection without another write. Remove the -durable publication and verify that a retry cannot dispatch a new write. +completed durable publication and verify that a retry cannot dispatch a new write. +Also interrupt metadata reads before publication staging, repeat the exact +`begin_review` and `submit_review` requests, and verify typed results and one +assessment publication. Let the original metadata read finish alongside its +retry and verify that both converge on the same publication. Changed requests +and revoked access must still stop preflight recovery. Leave a transient failure queued, restart the instance, and verify publication resumes at its saved deadline without another inbound message. Revoked access and permanent rejections must diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index f7b82550879..804405037a2 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -2830,7 +2830,138 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { )))[0]; return { ...f, service, head, assessment, provider, action, inline, checks, writes }; } - it("reuses one main GitHub PR comment across review requests", async () => { + it("preserves the published GitHub assessment during passive-response maintenance", async () => { + const f = await publicationRetryFixture(); + await f.service.execute(f.session, "submit_review", f.assessment, "published-before-passive-reply"); + await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.run.id)); + const reviewBefore = await readChatGitHubReviewsByEndpointId(f.endpoint.id); + const originalId = [...f.responseComments.keys()][0]; + const originalBody = f.responseComments.get(originalId)!.body; + const writesBefore = f.responseWrites.length; + const responses = githubResponseCommentService(db, f.providerFetch); + const thread = makeThread({ channelId: "paperclipai/paperclip", id: f.conversation.externalThreadId }).thread; + await deliverMessage({ callbacks: f.callbacks, endpointId: f.endpoint.id, provider: "github", thread, + trigger: "subscribed_message", message: makeMessage({ id: "41802", text: "No fix pushed; keep the existing review unchanged and wait for my reply.", userId: "42", userName: "octocat" }) }); + await responses.processPending(); + expect(f.responseComments.get(originalId)!.body).toBe(originalBody); + expect(f.responseWrites).toHaveLength(writesBefore); + const [next] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.assignedAgentId, + status: "running", runtimeMode: "native", contextSnapshot: await chatWakeContext({ + endpointId: f.endpoint.id, issueId: f.conversation.issueId, provider: "github", providerMessageId: "41802", + }), resultJson: { finalizationPhase: "committed", finalizationReasonCode: "external_chat_response_waiting", + nativeResult: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", + continuation: { kind: "response_wake", idempotencyKey: "passive-author-reply" } }, + presentationDecision: { schema: "paperclip.run_presentation_decision.v1", resolverVersion: "2", + activityDisposition: "collapse", chosenSource: "none", commentAction: "none", commentId: null, + reasonCodes: ["yielded_control_plane_wait"] } } }).returning(); + await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, f.conversation.issueId!)); + await responses.processPending(); + expect(f.responseComments.get(originalId)!.body).toBe(originalBody); + expect(f.responseWrites).toHaveLength(writesBefore); + await db.update(heartbeatRuns).set({ status: "succeeded", nativePhase: "committed" }).where(eq(heartbeatRuns.id, next.id)); + await responses.processPending(); + await responses.processPending(); + expect(f.responseComments.get(originalId)!.body).toBe(originalBody); + expect(await readChatGitHubReviewsByEndpointId(f.endpoint.id)).toEqual(reviewBefore); + expect([...f.responseComments.keys()]).toEqual([originalId]); + expect(await githubRunReplyState(db, { ...f.session, runId: next.id, endpointId: f.endpoint.id })).toBe("none"); + const [delivery] = await db.select().from(chatDeliveries).where(and(eq(chatDeliveries.endpointId, f.endpoint.id), + eq(chatDeliveries.providerEventId, `${thread.id}:41802`))); + const [response] = await db.select().from(chatActions).where(and(eq(chatActions.deliveryId, delivery.id), + eq(chatActions.kind, "github_response_comment"))); + expect(response).toMatchObject({ status: "cancelled", result: { final: true, code: "passive_response_wait" } }); + const writes = f.responseWrites.length; + const updatedAt = response.updatedAt; + await responses.processPending(); + expect(f.responseWrites).toHaveLength(writes); + expect((await readChatActionsById(response.id))[0].updatedAt).toEqual(updatedAt); + await expect(responses.failRun({ ...f.session, runId: next.id, endpointId: f.endpoint.id, + body: "This turn ended without publishing a final response." })).rejects.toMatchObject({ status: 403 }); + expect(f.responseWrites).toHaveLength(writes); + }); + it.each(["completed", "failed", "missing_decision", "comment_requested", "missing_semantic", "legacy", "uncommitted_native", "pending_publication", "missing_decision_then_missing", "uncommitted_then_failed", "malformed_null", "malformed_scalar", "malformed_schema", "malformed_action"] as const)( + "keeps GitHub response maintenance truthful for a new request: %s", async scenario => { + const pendingPresentation = scenario.startsWith("missing_decision") || scenario.startsWith("uncommitted") || scenario.startsWith("malformed"); + const f = await publicationRetryFixture(); + await f.service.execute(f.session, "submit_review", f.assessment, "review-before-new-request"); + await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.run.id)); + const originalId = [...f.responseComments.keys()][0]; + const originalBody = f.responseComments.get(originalId)!.body; + const thread = makeThread({ channelId: "paperclipai/paperclip", id: f.conversation.externalThreadId }).thread; + await deliverMessage({ callbacks: f.callbacks, endpointId: f.endpoint.id, provider: "github", thread, + trigger: "subscribed_message", message: makeMessage({ id: "41802", text: "Please handle this new request", userId: "42", userName: "octocat" }) }); + const [next] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.assignedAgentId, + status: scenario === "failed" ? "failed" : "succeeded", runtimeMode: scenario === "legacy" ? "legacy" : "native", + nativePhase: scenario.startsWith("uncommitted") ? "observed" : "committed", contextSnapshot: await chatWakeContext({ + endpointId: f.endpoint.id, issueId: f.conversation.issueId, provider: "github", providerMessageId: "41802", + }), resultJson: { finalizationPhase: "committed", finalizationReasonCode: "external_chat_response_waiting", + ...(scenario === "missing_semantic" ? {} : { nativeResult: { schema: "paperclip.run_result.v1", + reportedWorkDisposition: scenario === "completed" ? "completed" : "yielded", + continuation: { kind: "response_wake", idempotencyKey: "new-request-wait" } } }), + ...(scenario.startsWith("missing_decision") ? {} : { presentationDecision: scenario === "malformed_null" ? null : scenario === "malformed_scalar" ? "pending" : { + schema: scenario === "malformed_schema" ? "unknown" : "paperclip.run_presentation_decision.v1", chosenSource: "none", + commentAction: scenario === "malformed_action" ? "unknown" : scenario === "comment_requested" ? "create" : "none", reasonCodes: ["yielded_control_plane_wait"] } }) } }).returning(); + if (scenario === "pending_publication") await db.insert(chatActions).values({ companyId: f.companyId, + endpointId: f.endpoint.id, conversationId: f.conversation.id, kind: "github_review_publication", + providerActionId: `github_publication:${randomUUID()}`, status: "processing", + payload: { operation: "comment", session: { ...f.session, runId: next.id } } }); + const responses = githubResponseCommentService(db, f.providerFetch); + await responses.processPending(); + const [delivery] = await db.select().from(chatDeliveries).where(and(eq(chatDeliveries.endpointId, f.endpoint.id), + eq(chatDeliveries.providerEventId, `${thread.id}:41802`))); + const [response] = await db.select().from(chatActions).where(and(eq(chatActions.deliveryId, delivery.id), + eq(chatActions.kind, "github_response_comment"))); + if (scenario === "pending_publication") { + expect(f.responseComments.get(originalId)!.body).toBe(originalBody); + expect(response.result?.final).not.toBe(true); + expect(response).toMatchObject({ status: "failed", result: { retryable: true, code: "acknowledgement_unconfirmed" } }); + } else if (pendingPresentation) { + expect(f.responseComments.get(originalId)!.body).toBe(originalBody); + expect(response.result?.final).not.toBe(true); + const pendingWrites = f.responseWrites.length; + await responses.processPending(); + await expect(responses.failRun({ ...f.session, runId: next.id, endpointId: f.endpoint.id, + body: "This turn ended without publishing a final response." })).rejects.toMatchObject({ status: 409 }); + expect(f.responseWrites).toHaveLength(pendingWrites); + const failed = scenario === "uncommitted_then_failed"; + const missing = scenario === "missing_decision_then_missing"; + await db.update(heartbeatRuns).set({ status: failed ? "failed" : "succeeded", + nativePhase: failed ? "failed" : "committed", resultJson: { + ...next.resultJson, finalizationPhase: failed ? "failed" : "committed", + nativeResult: { schema: "paperclip.run_result.v1", reportedWorkDisposition: missing ? "completed" : "yielded", + continuation: missing ? null : { kind: "response_wake", idempotencyKey: "settled-request-wait" } }, + presentationDecision: { schema: "paperclip.run_presentation_decision.v1", chosenSource: "none", + commentAction: "none", reasonCodes: [missing ? "no_final_response" : "yielded_control_plane_wait"] }, + } }).where(eq(heartbeatRuns.id, next.id)); + await responses.processPending(); + if (failed || missing) expect(f.responseComments.get(originalId)!.body).toContain(failed + ? "This turn stopped before completing" : "This turn ended without publishing a final response"); + else expect(f.responseComments.get(originalId)!.body).toBe(originalBody); + expect((await readChatActionsById(response.id))[0]).toMatchObject({ + status: failed || missing ? "processed" : "cancelled", result: { final: true }, + }); + } else { + expect(f.responseComments.get(originalId)!.body).toContain(scenario === "failed" + ? "This turn stopped before completing" : "This turn ended without publishing a final response"); + expect(response).toMatchObject({ status: "processed", result: { final: true } }); + } + const writes = f.responseWrites.length; + await responses.processPending(); + expect(f.responseWrites).toHaveLength(writes); + }); + it("keeps per-request fallback outside the main GitHub PR review wait", async () => { + const f = await toolReplyFixture(); + await db.update(heartbeatRuns).set({ status: "succeeded", nativePhase: "committed", resultJson: { + finalizationPhase: "committed", finalizationReasonCode: "external_chat_response_waiting", + nativeResult: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", + continuation: { kind: "response_wake", idempotencyKey: "not-a-main-pr-review" } }, + presentationDecision: { schema: "paperclip.run_presentation_decision.v1", chosenSource: "none", + commentAction: "none", reasonCodes: ["yielded_control_plane_wait"] }, + } }).where(eq(heartbeatRuns.id, f.run.id)); + await githubResponseCommentService(db, f.providerFetch).processPending(); + expect([...f.responseComments.values()][0].body).toContain("without publishing a final response"); + }); + it.each(["same_head", "new_head"] as const)("reuses one main GitHub PR comment across review requests: %s", async headCase => { const f = await publicationRetryFixture(); await f.service.execute(f.session, "submit_review", f.assessment, "first-review"); const originalId = [...f.responseComments.keys()][0]; @@ -2848,8 +2979,13 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { endpointId: f.endpoint.id, issueId: f.conversation.issueId, provider: "github", providerMessageId: "41802", }) }).returning(); const session = { ...f.session, runId: next.id }; - await f.service.execute(session, "begin_review", { reviewedCommit: f.head }); - await f.service.execute(session, "submit_review", { ...f.assessment, summary: "Second assessment" }, "second-review"); + const currentHead = headCase === "new_head" ? "c".repeat(40) : f.head; + f.setSupplementalProviderFetch(async (input, init) => String(input).endsWith("/pulls/418") + ? Response.json({ number: 418, state: "open", draft: false, head: { sha: currentHead }, + base: { sha: "a".repeat(40), ref: "master" }, user: { id: 42, login: "octocat", type: "User" }, labels: [] }) + : f.provider(input, init)); + await f.service.execute(session, "begin_review", { reviewedCommit: currentHead }); + await f.service.execute(session, "submit_review", { ...f.assessment, reviewedCommit: currentHead, summary: "Second assessment" }, "second-review"); expect([...f.responseComments.keys()]).toEqual([originalId]); expect(f.responseComments.get(originalId)?.body).toContain("Second assessment"); expect(f.responseWrites.filter(write => write.method === "POST")).toHaveLength(1); @@ -3628,6 +3764,97 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { expect(classifications).toHaveLength(2); expect(classifications.map(c => c.violation)).toEqual([false, true]); expect(classifications.every(c => c.toolInvocationId === approval.invocationId && c.costEventId)).toBe(true); }); + it.each(["begin_review", "submit_review"] as const)("recovers %s after a timeout before publication staging", async name => { + const f = await publicationRetryFixture(); + let release!: () => void; + const blocked = new Promise(resolve => { release = resolve; }); + let metadataReads = 0; + f.setSupplementalProviderFetch(async (input, init) => { + if (String(input).endsWith("/pulls/418") && metadataReads++ === 0) { + await blocked; + throw new Error("Connection closed before PR metadata arrived"); + } + return f.provider(input, init); + }); + vi.stubGlobal("fetch", f.providerFetch); + onTestFinished(() => { release(); vi.unstubAllGlobals(); }); + const gateway = createToolGatewayService(db); + const session = await gateway.createSession({ companyId: f.companyId, agentId: f.assignedAgentId, runId: f.run.id }); + const tool = (await githubBotToolsForSession(db, f.session)).find(tool => tool.upstreamToolName === name)!; + const parameters = name === "begin_review" ? { reviewedCommit: f.head } : f.assessment; + const input = { sessionToken: session.token, tool: tool.name, parameters }; + await expect(gateway.executeTool({ ...input, timeoutMs: 50 })).rejects.toMatchObject({ reasonCode: "tool_timeout" }); + expect(await f.action()).toBeUndefined(); + release(); + const retry = await gateway.executeTool(input); + expect(retry).toMatchObject({ result: { data: name === "begin_review" + ? { reviewedCommit: f.head, state: "running" } + : { status: "processed", score: 3, conclusion: "neutral" } } }); + const writes = f.responseWrites.length; + expect(await gateway.executeTool(input)).toMatchObject({ invocationId: retry.invocationId, + result: { data: name === "begin_review" ? { reviewedCommit: f.head } : { status: "processed" } } }); + expect(f.responseWrites).toHaveLength(writes); + expect(f.responseComments.size).toBe(1); + }); + it("stages one assessment when the timed-out metadata read completes alongside its retry", async () => { + const f = await publicationRetryFixture(); + let release!: () => void; + const blocked = new Promise(resolve => { release = resolve; }); + let metadataReads = 0; + f.setSupplementalProviderFetch(async (input, init) => { + if (String(input).endsWith("/pulls/418") && metadataReads++ === 0) await blocked; + return f.provider(input, init); + }); + vi.stubGlobal("fetch", f.providerFetch); + onTestFinished(() => { release(); vi.unstubAllGlobals(); }); + const gateway = createToolGatewayService(db); + const session = await gateway.createSession({ companyId: f.companyId, agentId: f.assignedAgentId, runId: f.run.id }); + const tool = (await githubBotToolsForSession(db, f.session)).find(tool => tool.upstreamToolName === "submit_review")!; + const input = { sessionToken: session.token, tool: tool.name, parameters: f.assessment }; + await expect(gateway.executeTool({ ...input, timeoutMs: 50 })).rejects.toMatchObject({ reasonCode: "tool_timeout" }); + const retry = gateway.executeTool(input); + release(); + expect(await retry).toMatchObject({ result: { data: { actionId: expect.any(String) } } }); + await expect.poll(async () => (await f.action())?.status).toBe("processed"); + expect(await gateway.executeTool(input)).toMatchObject({ result: { data: { status: "processed" } } }); + const actions = await db.select().from(chatActions).where(and(eq(chatActions.endpointId, f.endpoint.id), + eq(chatActions.kind, "github_review_publication"))); + expect(actions).toHaveLength(1); + expect(actions[0].status).toBe("processed"); + expect(f.responseComments.size).toBe(1); + expect(f.responseWrites.map(write => write.method)).toEqual(["POST", "PATCH"]); + }); + it("rejects changed requests and revoked access before repeating an unstaged review", async () => { + const f = await publicationRetryFixture(); + let metadataReads = 0; + let unavailable = true; + f.setSupplementalProviderFetch(async (input, init) => { + if (String(input).endsWith("/pulls/418")) { + metadataReads++; + if (unavailable) throw new Error("Connection closed before PR metadata arrived"); + } + return f.provider(input, init); + }); + vi.stubGlobal("fetch", f.providerFetch); + onTestFinished(() => vi.unstubAllGlobals()); + const gateway = createToolGatewayService(db); + const session = await gateway.createSession({ companyId: f.companyId, agentId: f.assignedAgentId, runId: f.run.id }); + const tool = (await githubBotToolsForSession(db, f.session)).find(tool => tool.upstreamToolName === "submit_review")!; + const input = { sessionToken: session.token, tool: tool.name, parameters: f.assessment, idempotencyKey: "preflight-request" }; + await expect(gateway.executeTool(input)).rejects.toThrow("GitHub is temporarily unavailable"); + unavailable = false; + const failedReads = metadataReads; + await expect(gateway.executeTool({ ...input, parameters: { ...f.assessment, summary: "Changed assessment" } })) + .rejects.toMatchObject({ reasonCode: "github_publication_unavailable" }); + expect(await f.action()).toBeUndefined(); + expect(metadataReads).toBe(failedReads); + await db.insert(toolPolicies).values({ companyId: f.companyId, name: "Revoke review", policyType: "block", priority: 0, + selectors: { catalogEntryId: tool.catalogEntryId! } }); + await expect(gateway.executeTool(input)).rejects.toThrow(); + expect(await f.action()).toBeUndefined(); + expect(metadataReads).toBe(failedReads); + expect(f.responseWrites.map(write => write.method)).toEqual(["POST"]); + }); it("reads pending GitHub receipts without dispatch and refuses a missing publication", async () => { const f = await publicationRetryFixture(); let attempts = 0; @@ -3785,7 +4012,10 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { const f = await toolReplyFixture(); const [response] = await db.select().from(chatActions).where(and(eq(chatActions.endpointId, f.endpoint.id), eq(chatActions.kind, "github_response_comment"))); await db.update(chatActions).set({ status: "received", result: null }).where(eq(chatActions.id, response.id)); - await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.run.id)); + await db.update(heartbeatRuns).set({ status: "succeeded", nativePhase: "committed", resultJson: { presentationDecision: { + schema: "paperclip.run_presentation_decision.v1", commentAction: "none", chosenSource: "none", + reasonCodes: ["no_final_response"], + } } }).where(eq(heartbeatRuns.id, f.run.id)); await githubResponseCommentService(db, f.providerFetch).processPending(); expect(f.responseComments.size).toBe(1); expect([...f.responseComments.values()][0].body).toContain("without publishing a final response"); @@ -4052,7 +4282,10 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { }); it("does not invent a GitHub completion response when the agent sends no tool reply", async () => { const f = await toolReplyFixture(); - await db.update(heartbeatRuns).set({ status: "succeeded", resultJson: { presentationDecision: { authorizationReason: "internal_agent_write" } } }).where(eq(heartbeatRuns.id, f.run.id)); + await db.update(heartbeatRuns).set({ status: "succeeded", nativePhase: "committed", resultJson: { presentationDecision: { + schema: "paperclip.run_presentation_decision.v1", commentAction: "none", chosenSource: "none", + reasonCodes: ["no_final_response"], authorizationReason: "internal_agent_write", + } } }).where(eq(heartbeatRuns.id, f.run.id)); await expect(resolveChatRunPresentationAuthorizationReason(db, f.session)).resolves.toBe("internal_agent_write"); await enqueueChatRunMilestones(db); await f.service.processPendingPublications(); expect(f.runtime.endpoints.get(f.endpoint.id)?.posts).toEqual([]); diff --git a/server/src/services/chat-github-response-comments.ts b/server/src/services/chat-github-response-comments.ts index c032ade25ba..1c2b7383df2 100644 --- a/server/src/services/chat-github-response-comments.ts +++ b/server/src/services/chat-github-response-comments.ts @@ -35,6 +35,33 @@ const responseRunColumns = { issueId: heartbeatRuns.issueId, status: heartbeatRuns.status, updatedAt: heartbeatRuns.updatedAt, + // Terminal execution can precede native presentation commit. Until its + // durable decision exists, missing output is undecided rather than failure. + nativePresentationPending: sql`coalesce( + ${heartbeatRuns.status} = 'succeeded' + and ${heartbeatRuns.runtimeMode} = 'native' + and (${heartbeatRuns.nativePhase} is distinct from 'committed' + or ${heartbeatRuns.resultJson}->'presentationDecision'->>'schema' is distinct from 'paperclip.run_presentation_decision.v1' + or coalesce(${heartbeatRuns.resultJson}->'presentationDecision'->>'commentAction', '') not in ('none', 'create', 'reuse')), + false + )`, + // Project only the server-committed no-publication decision, never the + // potentially large native result or model-authored summary. + passiveResponseWait: sql`coalesce( + ${heartbeatRuns.status} = 'succeeded' + and ${heartbeatRuns.runtimeMode} = 'native' + and ${heartbeatRuns.nativePhase} = 'committed' + and ${heartbeatRuns.resultJson}->>'finalizationPhase' = 'committed' + and ${heartbeatRuns.resultJson}->>'finalizationReasonCode' = 'external_chat_response_waiting' + and ${heartbeatRuns.resultJson}->'nativeResult'->>'schema' = 'paperclip.run_result.v1' + and ${heartbeatRuns.resultJson}->'nativeResult'->>'reportedWorkDisposition' = 'yielded' + and ${heartbeatRuns.resultJson}->'nativeResult'->'continuation'->>'kind' = 'response_wake' + and ${heartbeatRuns.resultJson}->'presentationDecision'->>'schema' = 'paperclip.run_presentation_decision.v1' + and ${heartbeatRuns.resultJson}->'presentationDecision'->>'chosenSource' = 'none' + and ${heartbeatRuns.resultJson}->'presentationDecision'->>'commentAction' = 'none' + and ${heartbeatRuns.resultJson}->'presentationDecision'->'reasonCodes' @> '["yielded_control_plane_wait"]'::jsonb, + false + )`, contextSnapshot: sql>`jsonb_build_object( 'issueId', ${heartbeatRuns.contextSnapshot}->'issueId', 'taskId', ${heartbeatRuns.contextSnapshot}->'taskId', @@ -130,7 +157,7 @@ export async function writeGitHubResponseComment( const sameRequest = !shared || previous?.body?.includes(requestResponseMarker(source)) === true; const alreadyFinal = sameRequest && previous?.body?.includes("") === true; const body = `${projectSafeChatPublicationText(input.body)}\n\n${marker}${shared ? `\n${requestResponseMarker(source)}` : ""}\n${input.final ? "\n" : ""}`; - const preserved = previous && (previous.body === body || (input.acknowledgement && sameRequest) || (alreadyFinal && !input.final) || + const preserved = previous && (previous.body === body || (input.acknowledgement && (sameRequest || (shared && previous.body?.includes("")))) || (alreadyFinal && !input.final) || (input.preserveFinal && previous.body?.includes(""))) ? previous : null; const posted = preserved && (!shared || preserved.body?.includes(marker)) ? preserved : await api.request( previous ? `${item}/${previous.id}` : source.replyId @@ -145,7 +172,8 @@ export async function writeGitHubResponseComment( const receipt = { id: String(posted.id), url: posted.html_url }; await lease.commit(async tx => { if (shared) await tx.update(chatActions).set({ result: { - ...receipt, final: input.final || alreadyFinal, versionAt: versionAt.toISOString(), deliveryId: source.delivery.id, + ...receipt, final: input.final || alreadyFinal || Boolean(preserved?.body?.includes("")), + versionAt: versionAt.toISOString(), deliveryId: source.delivery.id, }, updatedAt: new Date() }).where(eq(chatActions.id, target.id)); await tx.update(chatActions).set({ status: "processed", result: { ...receipt, final: input.final || alreadyFinal, versionAt: versionAt.toISOString(), @@ -230,6 +258,20 @@ export function githubResponseCommentService(db: Db, fetchImpl = fetch) { companyId: source.endpoint.companyId, endpointId: source.endpoint.id, issueId: source.conversation.issueId!, runId: run.id }) : "none"; if (reply === "unsettled") throw conflict("GitHub reply is still being resolved"); + // A confirmed reply for this exact run can retire its placeholder. + // Otherwise terminal execution is not yet a missing-response decision. + if (ownsLatestRequest && run.nativePresentationPending && reply === "none") return; + if (isMainPullResponse(source) && ownsLatestRequest && run.passiveResponseWait && reply === "none") { + // A committed passive turn intentionally has no new response. + // Settle this request's acknowledgement without editing the shared + // PR assessment or inventing a current-run publication receipt. + await lease.commit(async tx => { await tx.update(chatActions).set({ + status: "cancelled", result: { ...existing.result, final: true, code: "passive_response_wait" }, + updatedAt: new Date(), + }).where(and(eq(chatActions.id, existing.id), eq(chatActions.companyId, source.endpoint.companyId), + sql`coalesce(${chatActions.result}->>'final', 'false') <> 'true'`)); }); + return; + } if (existing.result?.final === true) return existing.result.id ? { id: String(existing.result.id), url: String(existing.result.url) } : undefined; preserveFinal = reply === "confirmed"; onlyExisting = reply === "confirmed" && !existing.result?.id; @@ -374,6 +416,8 @@ export function githubResponseCommentService(db: Db, fetchImpl = fetch) { )).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1); if (latest?.id !== run.id) throw forbidden("A newer GitHub run owns the response"); if ((await responseForRequest())?.result?.final === true) throw forbidden("GitHub request already has a final response"); + if (current.nativePresentationPending) throw conflict("GitHub native response presentation is still being resolved"); + if (isMainPullResponse(source) && current.passiveResponseWait) throw forbidden("GitHub run intentionally yielded without a new response"); if (await githubRunReplyState(db, input) !== "none") throw conflict("GitHub reply is still being resolved"); }; receipt = await acknowledge(link.deliveryId, input.body, assertRun); diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index f5971156148..68abd9e014f 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -10808,7 +10808,7 @@ export function createToolGatewayService( : false; const gitHubPublicationReplay = recorded.replayed && tool.providerType === "paperclip_github_chat" && - ["comment", "update_comment", "submit_review"].includes(tool.upstreamToolName ?? "") && + ["begin_review", "comment", "update_comment", "submit_review"].includes(tool.upstreamToolName ?? "") && recorded.invocation.approvalState === "not_required"; const gitHubReplayPolicyAllowed = accessDecision.allowed; const gitHubReplayBindingMatches = @@ -10833,9 +10833,16 @@ export function createToolGatewayService( throw new ToolGatewayHttpError(409, error.message, "github_idempotency_conflict"); throw error; } - if (!receipt) - throw new ToolGatewayHttpError(409, "The saved GitHub publication is not available yet", "github_publication_unavailable"); - recoveredGitHubPublication = { content: JSON.stringify(receipt), data: receipt }; + if (receipt) { + recoveredGitHubPublication = { content: JSON.stringify(receipt), data: receipt }; + } else { + const canRepeatPreflight = tool.upstreamToolName === "begin_review" || + (recorded.invocation.status !== "succeeded" && !recorded.invocation.resultHash); + const exactRequest = recorded.invocation.argumentsHash === + toolRequestIdentityHash(effectiveParameters, options.toolActionSigningSecret); + if (!canRepeatPreflight || !exactRequest) + throw new ToolGatewayHttpError(409, "The saved GitHub publication is not available yet", "github_publication_unavailable"); + } } if (recorded.replayed && !retryingSlackRateLimit && !retryingGitHubPublication) { if (gitHubPublicationReplay && !gitHubReplayPolicyAllowed) From dcfc2acfa80f082fb59f527c9849bdb6eebb274a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 26 Oct 2026 22:23:05 -0500 Subject: [PATCH 043/133] Preserve complete bounded OpenCode MCP tool results (#17825) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit https://entire.com/gh/paperclipai/-/trails/34/changes/paperclip/34 ## Thinking Path > - Paperclip manages AI agents and their governed tools. > - Native OpenCode receives tool results through a Runner-owned MCP bridge. > - The bridge cut serialized results at 64 KiB, even inside a JSON string. > - A tool could finish while its saved OpenCode output lost files and receipts. > - Native app results already use bounded results and saved-result pages. > - This change preserves complete JSON within a bounded output limit. > - Agents can read the result without repeating the completed action. ## Linked Issues or Issue Description **What happened?** The OpenCode MCP bridge sliced serialized JSON at 65,536 characters. A large file-list result became invalid JSON in OpenCode's saved output. The tool execution itself had completed. **Expected behavior** The bridge must deliver complete bounded JSON. A delivery limit must produce an explicit error, rather than a partial success value. A repeated call identity must not execute the action again. **Steps to reproduce** 1. Return a structured result with 27 file entries and duplicated text/data representations larger than 64 KiB. 2. Call the tool through the actual HTTP MCP bridge. 3. Parse the returned text as JSON. The old source fails at position 65,536. Related: #17740 changed assigned-tool catalog delivery. This PR repairs result delivery and does not change the tool catalog or authority. ## What Changed - Replace JSON slicing with complete serialization and a 1 MiB UTF-8 output limit. - Bound serialization traversal, including escaped strings and many small fields. - Keep the existing native 500 KB result/page and saved-artifact contracts. - Report oversized delivery with a warning that the action may already have completed. Keep call-identity replay on the original execution. - Add real HTTP regressions and document the transport limit. ## Verification - Red before fix: the large-result HTTP regression fails with an unterminated JSON string at position 65,536. - 91 focused MCP, OpenCode driver and continuation assertions pass on Node 24.22.1. They cover escaped UTF-8 pages, oversized delivery, authentication, cancellation and duplicate-call execution. - Workspace typecheck and build pass. - Actual native Rust → fixed MCP bridge → OpenCode 1.18.34 → `openrouter/deepseek/deepseek-v4.1-flash-1017` passed with a public synthetic 27-file result. One assigned read produced a 281,375-byte saved JSON file. The model used Bash/Python to parse all filenames and both unpredictable receipt fields. One finish completed the turn. - The native probe used a synthetic gateway and a mocked completion acknowledgment. It does not prove hosted deployment, board finalization or a particular upstream model vendor. - The full default test attempt is failed/incomplete. It was stopped after repeated embedded PostgreSQL bootstrap failures and skipped database suites. Its 15 warm-workspace cases pass unchanged on isolated PostgreSQL 18. A controlled default-database rerun fails before the test body when embedded PostgreSQL cannot start after five attempts. The test source is byte-identical to the base commit. The failed attempt remains recorded. - The final test-only commit checks a 500 KB escaped producer page inside the actual controller text envelope. The production bridge hash still matches the native proof. Its 91 focused tests and package TypeScript check pass. - Current-head verification (`73d18ca2`): 53 CI checks passed and two conditional Storybook checks skipped. Greptile is 5/5 on this exact head with no review threads. The failed/incomplete full local attempt and the controlled passing suites above remain separate evidence. ## Risks - Larger complete results use more bounded serialization memory than the old truncated output. - A bridge result above 1 MiB fails delivery. Native app results and saved-result pages are bounded at 500 KB before this bridge. The existing artifact protocol handles larger results. - A failed delivery cannot prove whether an upstream action took effect. The error forbids repeating it, and identical call identities retain the original execution. - This change does not alter model routing, admitted tools, credentials or trust policy. Hosted rollout and incident recovery remain separate verification steps. ## Model Used - OpenAI `gpt-6.1-sol`, with reasoning, tool use and code execution. The session did not expose an authoritative context-window size. Native acceptance used the exact DeepSeek model named above. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the relevant local tests and they pass; the full local-suite environment limitation is documented above. - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- packages/paperclip-runner/README.md | 14 +++++ .../src/drivers/opencode/mcp-bridge.test.ts | 59 +++++++++++++++++++ .../src/drivers/opencode/mcp-bridge.ts | 38 +++++++++++- 3 files changed, 109 insertions(+), 2 deletions(-) diff --git a/packages/paperclip-runner/README.md b/packages/paperclip-runner/README.md index 51823ae1d55..488c97584b3 100644 --- a/packages/paperclip-runner/README.md +++ b/packages/paperclip-runner/README.md @@ -797,6 +797,20 @@ fit the compacted bound produces a diagnostic; the event stream continues. Tool output and error fields have reserved retention space and size records, so large input values cannot consume their output tails or make excerpts look complete. A separator split between reads does not count against the frame limit. +### OpenCode MCP tool results + +The Runner-owned MCP bridge sends complete JSON tool results up to 1 MiB of +UTF-8 output. It does not slice JSON text. This includes the native server's +bounded 500 KB app-tool results and saved-result pages. OpenCode can put a long +result in its own output file; that file must still contain the complete JSON. +The existing `read_tool_result` artifact protocol handles larger app results. + +Serialization has a bounded traversal budget and checks escaped UTF-8 output. +If delivery exceeds the limit, the bridge reports an error without a partial +success value. The action may already have completed. Reconcile its saved +result or receipt instead of repeating it. Replaying the same MCP call identity +reuses the original handler execution, including when delivery failed. + ## Preconfigured home Set `homeMode: "preconfigured"` in a `paperclip_runner` agent's adapter config diff --git a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts index f281fd2a55a..40c2fc68fd7 100644 --- a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts @@ -50,6 +50,65 @@ describe("OpenCode MCP bridge", () => { }); }); + it("delivers a complete large assigned-tool result and replays its receipt without repeating the action", async () => { + const files = Array.from({ length: 27 }, (_, index) => ({ + filename: `src/public-fixture-${index}.ts`, + patch: `@@ synthetic public fixture ${index} @@\n${"+ documented change\n".repeat(220)}`, + })); + const data = { items: files, page: 1, hasMore: false, receipt: "LAST_FILE_AND_RECEIPT_INTACT" }; + const result = { content: [{ type: "text", text: JSON.stringify(data) }], data }; + expect(JSON.stringify(result).length).toBeGreaterThan(111_877); + const handler = vi.fn(async () => result); + const bridge = await startOpenCodeMcpBridge({ + tools: [{ name: "paperclip_call_assigned_tool", inputSchema: { type: "object" } }], handler, + }); + bridges.push(bridge); + const request = { id: "large-result", method: "tools/call", params: { name: "paperclip_call_assigned_tool", arguments: {} } }; + for (let attempt = 0; attempt < 2; attempt++) { + const response = await (await rpc(bridge, request)).json() as { result: { content: Array<{ text: string }> } }; + expect(JSON.parse(response.result.content[0]!.text)).toEqual(result); + } + expect(handler).toHaveBeenCalledTimes(1); + }); + + it("preserves producer-sized pages with UTF-8 and heavily escaped nested data", async () => { + const chunk = '\"\\\n\t\u0000😀'; + const result = { responseText: { nextOffsetBytes: 300_000, totalBytes: 600_000 }, data: chunk.repeat(25_000) }; + expect(Buffer.byteLength(JSON.stringify(result))).toBeLessThanOrEqual(500_000); + expect(Buffer.byteLength(JSON.stringify(result))).toBeGreaterThan(64 * 1024); + // The driver returns a controller text item, not the raw app page. Its + // encoded JSON must survive a second layer of escaping at this boundary. + const controllerResult = { text: JSON.stringify(result), type: "inputText" }; + expect(Buffer.byteLength(JSON.stringify(controllerResult))).toBeGreaterThan(500_000); + const bridge = await startOpenCodeMcpBridge({ + tools: [{ name: "read_tool_result", inputSchema: { type: "object" } }], handler: async () => controllerResult, + }); + bridges.push(bridge); + const response = await (await rpc(bridge, { id: "page", method: "tools/call", params: { name: "read_tool_result", arguments: {} } })).json() as { result: { content: Array<{ text: string }> } }; + const delivered = JSON.parse(response.result.content[0]!.text) as typeof controllerResult; + expect(delivered).toEqual(controllerResult); + expect(JSON.parse(delivered.text)).toEqual(result); + }); + + it.each([ + ["UTF-8 scalar", () => ({ data: "😀".repeat(300_000) })], + ["escaped scalar", () => ({ data: "\u0000".repeat(200_000) })], + ["many fields", () => Array.from({ length: 600_000 }, () => 0)], + ])("rejects an oversized %s without a partial success or repeating the completed call", async (_name, makeResult) => { + const handler = vi.fn(async () => makeResult()); + const bridge = await startOpenCodeMcpBridge({ + tools: [{ name: "assigned.write", inputSchema: { type: "object" } }], handler, + }); + bridges.push(bridge); + const request = { id: "write-receipt", method: "tools/call", params: { name: "assigned.write", arguments: {} } }; + for (let attempt = 0; attempt < 2; attempt++) { + expect(await (await rpc(bridge, request)).json()).toMatchObject({ + result: { isError: true, content: [{ text: expect.stringContaining("may already have completed; do not repeat it") }] }, + }); + } + expect(handler).toHaveBeenCalledTimes(1); + }); + it("admits runner-private extension operations without advertising them to the model", async () => { const handler = vi.fn(async ({ tool }) => ({ decision: tool === "__paperclip_permission" ? "accept" : "decline" })); const bridge = await startOpenCodeMcpBridge({ diff --git a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts index cb1efd91922..af847094c40 100644 --- a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts +++ b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts @@ -44,6 +44,9 @@ export interface OpenCodeMcpBridge { const DEFAULT_TIMEOUT_MS = 30_000; const DEFAULT_MAX_BODY_BYTES = 1_048_576; +// Native app results/pages are capped at 500 KB before reaching this bridge. +// Leave room for their delivery envelope, but never cut a completed JSON value. +const MAX_RESULT_BYTES = 1_048_576; export function canonicalOpenCodeMcpToolName(name: string): string { const trimmed = name.trim(); @@ -348,8 +351,39 @@ function safeError(error: unknown): string { } function safeJson(value: unknown): string { - const serialized = JSON.stringify(value); - return (serialized ?? "null").slice(0, 64 * 1024); + const tooLarge = () => new Error( + "Tool result delivery exceeded the 1 MiB MCP limit. The action may already have completed; do not repeat it. Reconcile its saved result or receipt instead.", + ); + // Separator accounting is conservative (it includes the first entry too). + // Twice the output limit admits every result within the exact byte cap. + let budget = 2 * MAX_RESULT_BYTES; + let root = true; + const charge = (text: string) => { + // Check length before measuring UTF-8 so even a huge scalar is rejected + // without copying it. JSON escaping can expand the final value further. + if (text.length > budget) throw tooLarge(); + budget -= Buffer.byteLength(text); + if (budget < 0) throw tooLarge(); + }; + const serialized = JSON.stringify(value, function (key, entry: unknown) { + const first = root; + root = false; + const scalar = entry === null || typeof entry !== "object"; + const omitted = entry === undefined || typeof entry === "function" || typeof entry === "symbol"; + if (omitted && !Array.isArray(this)) return entry; + if (!first) { + charge(","); + if (!Array.isArray(this)) { charge(key); charge('"":'); } + } + if (typeof entry === "string" || entry instanceof String) { charge(String(entry)); charge('""'); } + else if (!scalar) charge("{}"); + else charge(omitted ? "null" : JSON.stringify(entry) ?? "null"); + return entry; + }) ?? "null"; + // The traversal budget bounds allocation even for many small fields. This + // exact check also accounts for quotes, control characters and UTF-8 bytes. + if (Buffer.byteLength(serialized) > MAX_RESULT_BYTES) throw tooLarge(); + return serialized; } function canonicalJson(value: unknown): string { From ea727dc3c2a14660635f30cdba09f7790ac5603d Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Mon, 26 Oct 2026 20:25:10 -0700 Subject: [PATCH 044/133] Preserve plugin session continuity through warm execution and checkpoint recovery (#17820) --- doc/plugins/PLUGIN_RUNNER_EXECUTION.md | 20 ++- doc/testing-native-remote-continuity.md | 14 ++ .../environment-run-orchestrator.test.ts | 14 ++ .../src/__tests__/environment-runtime.test.ts | 2 +- ...ronment-task-authority.integration.test.ts | 25 ++++ .../heartbeat-plugin-environment.test.ts | 11 +- .../__tests__/lifecycle.test.ts | 18 +++ server/src/services/agent-hiring.ts | 6 +- .../services/agent-runner-selection.test.ts | 20 ++- server/src/services/agent-runner-selection.ts | 29 +++- .../services/environment-run-orchestrator.ts | 3 +- .../src/services/environment-task-runtime.ts | 23 +++- server/src/services/heartbeat.ts | 1 + .../heartbeat/runtime-selection.test.ts | 20 +++ .../services/heartbeat/runtime-selection.ts | 7 +- .../heartbeat/workspace-preparation.test.ts | 2 + .../heartbeat/workspace-preparation.ts | 2 + .../native-session-executor.test.ts | 130 ++++++++++++++++-- .../native-runtime/native-session-executor.ts | 107 +++++++++++--- .../native-runtime/plugin-runner-task.test.ts | 35 +++-- .../native-runtime/plugin-runner-task.ts | 4 +- .../runner-environment-lifecycle.test.ts | 18 ++- .../services/runner-environment-lifecycle.ts | 4 +- 23 files changed, 433 insertions(+), 82 deletions(-) diff --git a/doc/plugins/PLUGIN_RUNNER_EXECUTION.md b/doc/plugins/PLUGIN_RUNNER_EXECUTION.md index d1e567873c8..ae04f0ab6d2 100644 --- a/doc/plugins/PLUGIN_RUNNER_EXECUTION.md +++ b/doc/plugins/PLUGIN_RUNNER_EXECUTION.md @@ -23,7 +23,7 @@ bounded `read_file` operation for attachments and deliverables. Core supports per-turn workspace sessions with Codex, OpenCode, and ACPX. OpenCode and ACPX require the existing qualified provider profiles. -Workspace-free providers are not supported here. Codex warm retention is opt-in through `warmTaskVersion: 1`; ACPX and OpenCode remain per-turn. +Workspace-free providers are not supported here. Drivers opt into warm retention through `warmTaskVersion: 1`. Core prefers warm execution for new task runs on capable drivers, including ACPX; existing provider admission still determines which harnesses the driver supports. The plugin owns project workspace preparation. Core does not prepare or reuse a Core Git worktree for a task-only environment. Selecting a plain task directory does not add redundant default configuration. @@ -164,13 +164,20 @@ The existing agent-independent `inspect` and `remove` operations continue to protect storage maintenance. Providers that do not declare finalization version 1 retain the inspection-based completion path. No stored binding migration is required. -## Codex warm task retention +## Warm task retention A driver declares `warmTaskVersion: 1`, implements `onEnvironmentResumeLease`, and returns `warmTaskVersion: 1` from `describe` only when its execution host -supports authorized run attachment. Select the agent's warm lifecycle or set -`runnerLifecycleMode: "warm"` and `runnerIdleTimeoutMs` on its plugin environment. -Core requires an explicitly selected account. Setup probes remain ephemeral. +supports authorized run attachment. Newly hired agents targeting such a driver +are configured for warm execution. Existing agents keep their explicit policy; +plugin runs without an explicit policy prefer warm execution. There is no +provider-specific warm gate or additional setting. Existing runs retain their +admitted lifecycle policy during restart recovery. The existing +`runnerIdleTimeoutMs` controls bounded retention. Fresh warm admission requires +a Core-managed credential identity before lease acquisition. Plugin-owned installed +accounts continue per-turn until such an identity is available; the saved warm +preference does not bypass account ownership checks. Drivers without warm +capability and setup probes remain ephemeral. Plugin task execution does not create a Core-managed GitHub broker. Warm reuse and restart recovery use the same broker-ownership rule as initial startup. @@ -208,4 +215,5 @@ Changes to account, credential generation, workspace, runtime or process policy retire the old owner before recovery. A new authority epoch archives the stopped checkpoint under its stable archive key and preserves provider session files. Warm status monitoring relies on PRP after startup and does not poll an idle -Launcher. General ACPX warm support is deliberately excluded. +Launcher. ACPX uses this task-provider contract; other execution transports retain +their provider-specific lifecycle policies. diff --git a/doc/testing-native-remote-continuity.md b/doc/testing-native-remote-continuity.md index 434b179e149..8c9577fdfbd 100644 --- a/doc/testing-native-remote-continuity.md +++ b/doc/testing-native-remote-continuity.md @@ -16,6 +16,20 @@ Each recovery case includes another healthy turn after the recovered turn and asserts that it does not start another runner or provider process. It does not qualify a live model or a sandbox provider API. +Per-turn plugin continuation uses a stopped checkpoint on a replacement lease. +The executor tests cover Claude and Codex, including reopening blocked work. +The lease-authority integration test uses PostgreSQL to check that checkpoint +discovery cannot launch work or bypass a termination receipt. A stale negative +availability flag requires a fresh provider status before the checkpoint can +be read for continuation. + +The executor can restore one quarantined journal in the same session scope. +It requires a terminal prior run, completed commands, a terminal event, and an +identity-matched suspended plugin checkpoint with no pending delivery or cleanup. +Multiple candidates, contradictory direct state, and unfinished work still fail +closed. Recovery preserves the journal and the provider's existing state owner. +Do not delete a quarantine or change database ownership to force continuation. + Use a disposable Linux container with Node.js, `sh`, `tar`, `base64`, `ps`, and `setsid`. Build `paperclip-runnerd` and `fake-codex-app-server` from the checkout's `packages/paperclip-runner/runner/Cargo.toml` for that container's architecture. diff --git a/server/src/__tests__/environment-run-orchestrator.test.ts b/server/src/__tests__/environment-run-orchestrator.test.ts index 85591e1070a..4329d374ac5 100644 --- a/server/src/__tests__/environment-run-orchestrator.test.ts +++ b/server/src/__tests__/environment-run-orchestrator.test.ts @@ -764,6 +764,20 @@ describe("environmentRunOrchestrator — realizeForRun", () => { describe("native runner lifecycle changes before lease acquisition", () => { + it.each([undefined, "account:generation"])("uses a warm plugin lease only with a managed credential identity: %s", async managedAiCredentialIdentity => { + const environment = { ...makeEnvironment("plugin"), config: { pluginKey: "fixture", driverKey: "tasks", driverConfig: {} } }; + mockGetEnvironment.mockResolvedValue(environment); + const acquireRunLease = vi.fn(async input => ({ environment: input.environment, + lease: makeLease({ leasePolicy: input.environment.config.runnerLifecycleMode === "warm" ? "reuse_by_environment" : "ephemeral" }), leaseContext: {} })); + const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: makeMockRuntime({ acquireRunLease }) }); + const result = await orchestrator.acquireForRun({ companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local", + adapterType: "paperclip_runner", adapterConfig: { lifecycleMode: "warm" }, managedAiCredentialIdentity, + issueId: "task-1", heartbeatRunId: "run-1", agentId: "agent-1", persistedExecutionWorkspace: null, executionWorkspaceSettings: null }); + expect(result.environment.config.runnerLifecycleMode).toBe(managedAiCredentialIdentity ? "warm" : "per_turn"); + expect(result.lease.leasePolicy).toBe(managedAiCredentialIdentity ? "reuse_by_environment" : "ephemeral"); + expect(environment.config).not.toHaveProperty("runnerLifecycleMode"); + }); + it("requests a reusable lease when an existing task switches from per-turn to warm", async () => { const environment = { ...makeEnvironment("sandbox"), config: { provider: "daytona", reuseLease: false } }; mockGetEnvironment.mockResolvedValue(environment); diff --git a/server/src/__tests__/environment-runtime.test.ts b/server/src/__tests__/environment-runtime.test.ts index b926be7e975..14b968a0ba5 100644 --- a/server/src/__tests__/environment-runtime.test.ts +++ b/server/src/__tests__/environment-runtime.test.ts @@ -824,7 +824,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { const cold = await runtime.acquireRunLease({ ...input, environment, heartbeatRunId: seeded.runId }); expect(cold.lease.leasePolicy).toBe("ephemeral"); await runtime.releaseRunLeases(seeded.runId, "released"); - const warmEnvironment = resolveRunnerEnvironmentForRun(environment, "paperclip_runner", { lifecycleMode: "warm" }); + const warmEnvironment = resolveRunnerEnvironmentForRun(environment, "paperclip_runner", { lifecycleMode: "warm" }, undefined, "selected-account:generation"); const warmRunId = randomUUID(); await db.insert(heartbeatRuns).values({ id: warmRunId, companyId: seeded.companyId, agentId: seeded.agentId, status: "running" }); const warm = await runtime.acquireRunLease({ ...input, environment: warmEnvironment, heartbeatRunId: warmRunId }); diff --git a/server/src/__tests__/environment-task-authority.integration.test.ts b/server/src/__tests__/environment-task-authority.integration.test.ts index 446c590012f..d01611e1972 100644 --- a/server/src/__tests__/environment-task-authority.integration.test.ts +++ b/server/src/__tests__/environment-task-authority.integration.test.ts @@ -59,16 +59,41 @@ databaseSuite("environment task lease authority", () => { await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, oldLeaseId)); await expect(authorize()).rejects.toThrow("not authorized"); await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, oldLeaseId)); + let sessionStateAvailable = false; + const operations: string[] = []; const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }), call: async (_id: string, _method: string, params: any) => ["read_state", "archive_state"].includes(params.operation.kind) ? { kind: "state", taskId: params.taskId, state: { schema: "paperclip.runner.durable.state.v1", lifecycle: "suspended", ...(params.operation.identity ?? params.operation.priorIdentity) } } + : params.operation.kind === "status" + ? (operations.push("status"), { kind: "status", taskId: params.taskId, phase: "completed", executionStopped: true, sessionStateAvailable }) : { kind: "accepted", taskId: params.taskId }, } as unknown as PluginWorkerManager; const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers }); const runner = { runnerId: "runner", leaseId: oldLeaseId, sessionId: "session", runId: nextRunId, turnId: "turn", itemId: "item", harness: "codex", protocolMin: 1, protocolMax: 3 }; const checkpointIdentity = { ...priorIdentity, turnId: "prior-turn", itemId: "prior-item" }; + const [source] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, oldLeaseId)); + await db.update(environmentLeases).set({ metadata: { ...source.metadata, runnerTaskSessionRetained: false } }).where(eq(environmentLeases.id, oldLeaseId)); + await expect(authorize()).rejects.toThrow("not authorized"); + await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).resolves.toBe("old-task"); + const inspect = () => runtime.task({ companyId, leaseId: nextLeaseId, operation: { + kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task", + } }); + await expect(inspect()).rejects.toThrow("not authorized"); + await expect(authorize()).rejects.toThrow("not authorized"); + // Discovery cannot launch, and cannot bypass the original termination receipt. + await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: { + kind: "submit", preparedLaunchId: "fixture", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task", + } })).rejects.toThrow("resume identity mismatch"); + await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, oldLeaseId)); + await expect(inspect()).rejects.toThrow("not authorized"); + expect(operations).toEqual(["status"]); + await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, oldLeaseId)); + sessionStateAvailable = true; + await expect(inspect()).resolves.toMatchObject({ kind: "state" }); + expect(operations).toEqual(["status", "status"]); + await expect(authorize()).resolves.toBe("old-task"); for (const operation of [ { kind: "read_state" as const, identity: checkpointIdentity, resumeFrom: "old-task" }, { kind: "archive_state" as const, priorIdentity: checkpointIdentity, archiveKey: "a".repeat(24), resumeFrom: "old-task" }, diff --git a/server/src/__tests__/heartbeat-plugin-environment.test.ts b/server/src/__tests__/heartbeat-plugin-environment.test.ts index 99c5abfa58b..f41e8aea8cf 100644 --- a/server/src/__tests__/heartbeat-plugin-environment.test.ts +++ b/server/src/__tests__/heartbeat-plugin-environment.test.ts @@ -93,7 +93,7 @@ describeEmbeddedPostgres("heartbeat plugin environments", () => { await stopDb?.(); }); - it("dispatches a plugin-only project without Core workspace files through the native heartbeat boundary", async () => { + it("dispatches a plugin-owned account per-turn without Core workspace files through the native heartbeat boundary", async () => { const companyId = randomUUID(), projectId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), sourceId = randomUUID(); const environmentId = randomUUID(), pluginId = randomUUID(), pluginKey = `test.tasks.${pluginId}`; const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-task-heartbeat-")); tempRoots.push(root); @@ -112,11 +112,11 @@ describeEmbeddedPostgres("heartbeat plugin environments", () => { capabilities: ["environment.drivers.register", "workspaces.manage", "events.subscribe"], resourceLifecycle: ["project"], workspaceProvider: { modes: ["shared_workspace", "isolated_workspace", "operator_branch"], strategies: ["project_primary", "git_worktree"] }, entrypoints: { worker: "worker.js" }, - environmentDrivers: [{ driverKey: "tasks", displayName: "Tasks", supportsTasks: true, configSchema: { type: "object" } }] } }); + environmentDrivers: [{ driverKey: "tasks", displayName: "Tasks", supportsTasks: true, warmTaskVersion: 1, configSchema: { type: "object" } }] } }); await db.insert(environments).values({ id: environmentId, companyId, name: "Tasks", driver: "plugin", status: "active", config: { pluginKey, driverKey: "tasks", driverConfig: {} } }); await db.insert(agents).values({ id: agentId, companyId, name: "Task agent", role: "engineer", status: "idle", - adapterType: "paperclip_runner", adapterConfig: { provider: "codex", model: "gpt-6.1-sol" }, runtimeConfig: {}, + adapterType: "paperclip_runner", adapterConfig: { provider: "codex", model: "gpt-6.1-sol", lifecycleMode: "warm" }, runtimeConfig: {}, defaultEnvironmentId: environmentId, permissions: {} }); await db.insert(issues).values({ id: issueId, companyId, projectId, projectWorkspaceId: sourceId, title: "Task-only dispatch", status: "in_progress", assigneeAgentId: agentId, issueNumber: 1, identifier: "TASK-1" }); @@ -129,14 +129,15 @@ describeEmbeddedPostgres("heartbeat plugin environments", () => { if (method === "environmentReleaseLease" || method === "environmentDestroyLease") return undefined; throw new Error(`Unexpected task provider call: ${method}`); }); - const workerManager = { isRunning: () => true, getWorker: () => ({ supportedMethods: ["workspaceOperation", "environmentTask", "environmentAcquireLease", "environmentReleaseLease", "environmentDestroyLease"] }), call } as unknown as PluginWorkerManager; + const workerManager = { isRunning: () => true, getWorker: () => ({ supportedMethods: ["workspaceOperation", "environmentTask", "environmentAcquireLease", "environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] }), call } as unknown as PluginWorkerManager; const entered = vi.fn(() => { throw new Error("synthetic_native_boundary_reached"); }); const heartbeat = heartbeatService(db, { pluginWorkerManager: workerManager, nativeSessionBackendFactory: entered }); heartbeats.push(heartbeat); const run = await heartbeat.wakeup(agentId, { source: "on_demand", triggerDetail: "manual", contextSnapshot: { issueId, projectId } }); expect(run).not.toBeNull(); await vi.waitFor(async () => { expect((await heartbeat.getRun(run!.id))?.status).toBe("failed"); }, { timeout: 10_000 }); - expect(entered).toHaveBeenCalledWith(expect.objectContaining({ binding: expect.objectContaining({ runId: run!.id, companyId, issueId }) })); + expect(entered).toHaveBeenCalledWith(expect.objectContaining({ binding: expect.objectContaining({ runId: run!.id, companyId, issueId }), + session: expect.objectContaining({ lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } }) })); expect((await heartbeat.getRun(run!.id))?.error).toContain("synthetic_native_boundary_reached"); await heartbeat.drainActiveRunExecutions(); expect(call).toHaveBeenCalledWith(pluginId, "environmentDestroyLease", expect.objectContaining({ providerLeaseId: "heartbeat-task-attempt" })); diff --git a/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts b/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts index d9b7dcf8e0c..8f494cc6fa7 100644 --- a/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts +++ b/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts @@ -48,6 +48,24 @@ const support = await getEmbeddedPostgresTestSupport(); async function hire(status = "idle") { return createAgentLifecycle(db).requestHire(companyId, { name: "Agent", status, adapterType: "process" }); } async function current(id: string) { return (await db.select().from(agents).where(eq(agents.id, id)))[0]!; } + it.each((["codex", "claude"] as const).flatMap(provider => ["direct", "setup", "approval"].map(path => [provider, path] as const)))("defaults a newly hired %s agent on a warm task environment to warm execution through %s", async (provider, path) => { + const pluginId = randomUUID(), environmentId = randomUUID(); + await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "warm-hire-fixture", version: "1.0.0", status: "ready", + manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true, warmTaskVersion: 1 }] } as never }); + await db.insert(environments).values({ id: environmentId, name: `Warm computer ${environmentId}`, driver: "plugin", + config: { pluginKey: pluginId, driverKey: "tasks", driverConfig: {} } }); + const adapterConfig = provider === "claude" ? { provider: "acpx", acpxAgent: "claude", model: "claude-sonnet-4-6", lifecycleMode: "per_turn" } + : { provider: "codex", model: "gpt-5.6-luna", lifecycleMode: "per_turn" }; + const lifecycle = createAgentLifecycle(db); + const input = { name: "Warm hire", adapterType: "paperclip_runner", adapterConfig, defaultEnvironmentId: environmentId }; + const draft = path === "setup" ? await lifecycle.requestHire(companyId, input, { setupOnly: true }) : null; + const hired = await lifecycle.requestHire(companyId, { ...input, ...(path === "approval" ? { status: "pending_approval" } : {}) }, draft ? { setupAgentId: draft.id } : undefined); + if (path === "approval") await lifecycle.approveHire(hired.id, "board"); + expect((await current(hired.id)).adapterConfig).toMatchObject({ ...adapterConfig, lifecycleMode: "warm" }); + await createAgentLifecycle(db).updateConfiguration(hired.id, { adapterConfig: { ...(await current(hired.id)).adapterConfig, lifecycleMode: "per_turn" } }); + expect((await current(hired.id)).adapterConfig.lifecycleMode).toBe("per_turn"); + }); + it("rejects an outdated configuration snapshot under the write lock", async () => { const agent = await hire(); const snapshot = await current(agent.id); diff --git a/server/src/services/agent-hiring.ts b/server/src/services/agent-hiring.ts index 00ed74c6ab3..8a08fa6052e 100644 --- a/server/src/services/agent-hiring.ts +++ b/server/src/services/agent-hiring.ts @@ -12,7 +12,7 @@ import { secretService, assertClaudeOAuthBindingInvariant } from "./secrets.js"; import { agentCredentialService } from "./agent-credentials.js"; import { agentIdentityService } from "./agent-identity.js"; import { initializePrimaryAgent } from "./primary-agent.js"; -import { resolveNewAgentRunnerForCompany } from "./agent-runner-selection.js"; +import { resolveAgentRunnerTaskDriverForCompany, resolveNewAgentRunnerForCompany } from "./agent-runner-selection.js"; export async function prepareAgentHire(db: Db, companyId: string, input: CreateAgentData, options?: CreateAgentOptions): Promise { const { runner, ...data } = input; @@ -44,6 +44,10 @@ export async function prepareAgentHire(db: Db, companyId: string, input: CreateA ? await secretService(db).normalizeAdapterConfigForPersistence(companyId, data.adapterConfig, { adapterType }) : {}; const adapterConfig = normalizePaperclipRunnerAdapterConfig(adapterType, rawAdapterConfig); + if (adapterType === "paperclip_runner" && !["openai_dot", "grok_bot", "muse"].includes(String(adapterConfig.provider)) && + (await resolveAgentRunnerTaskDriverForCompany(db, companyId, data.defaultEnvironmentId))?.warmTaskVersion === 1) { + adapterConfig.lifecycleMode = "warm"; + } await agentCredentialService(db).enforceClaudeOAuthBindingClaim(db, { companyId, consume: true, environmentId: data.defaultEnvironmentId ?? null, childAdapterConfig: adapterConfig, claudeLogin: options?.claudeLogin, diff --git a/server/src/services/agent-runner-selection.test.ts b/server/src/services/agent-runner-selection.test.ts index afc3949acdf..042c6084f45 100644 --- a/server/src/services/agent-runner-selection.test.ts +++ b/server/src/services/agent-runner-selection.test.ts @@ -12,7 +12,7 @@ vi.mock("./environment-config.js", async () => ({ })); vi.mock("./plugin-registry.js", () => ({ pluginRegistryService: () => ({ getByKey: state.pluginByKey }) })); vi.mock("@paperclipai/adapter-utils/ssh", () => ({ runSshCommand: state.ssh })); -import { AGENT_RUNNER_SSH_PLATFORM_COMMAND, agentRunnerAvailability, resolveAgentRunnerTargetForCompany, resolveNewAgentRunner, resolveNewAgentRunnerForCompany } from "./agent-runner-selection.js"; +import { AGENT_RUNNER_SSH_PLATFORM_COMMAND, agentRunnerAvailability, resolveAgentRunnerTaskDriverForCompany, resolveAgentRunnerTargetForCompany, resolveNewAgentRunner, resolveNewAgentRunnerForCompany } from "./agent-runner-selection.js"; import { buildCodexAcpConfig, resolveCodexExecutionEngine } from "@paperclipai/adapter-codex-local/server"; import { secretService } from "./secrets.js"; import * as localPlatform from "../lib/agent-runner-platform.js"; @@ -522,6 +522,24 @@ describe("Runner-only plugin environments", () => { expect(agentRunnerAvailability("gemini_local", target)).toEqual({ supportedRunners: ["legacy"], defaultRunner: "legacy" }); expect(state.ssh).not.toHaveBeenCalled(); }); + it("carries warm task capability from the selected driver without provider gates", async () => { + state.pluginByKey.mockResolvedValue(plugin({ supportsTasks: true, warmTaskVersion: 1 })); + expect(await resolveAgentRunnerTargetForCompany({} as never, "company", "computer")) + .toEqual({ driver: "plugin", runnerTasks: true, warmTaskVersion: 1 }); + expect(await resolveAgentRunnerTaskDriverForCompany({} as never, "company", "computer")) + .toMatchObject({ driverKey: "tasks", supportsTasks: true, warmTaskVersion: 1 }); + }); + + it.each(["ssh", "sandbox"])("does not qualify a %s runner while inspecting hire defaults", async driver => { + state.getEnvironment.mockResolvedValue({ ...pluginEnvironment, driver, config: { provider: "fake" } }); + expect(await resolveAgentRunnerTaskDriverForCompany({} as never, "company", "computer")).toBeUndefined(); + expect(state.ssh).not.toHaveBeenCalled(); + expect(state.sandboxQualification).not.toHaveBeenCalled(); + expect(state.pluginByKey).not.toHaveBeenCalled(); + if (driver === "sandbox") await expect(resolveAgentRunnerTargetForCompany({} as never, "company", "computer")) + .rejects.toThrow("The fake sandbox provider cannot execute runs."); + }); + it("saves a Claude agent on the plugin environment as Paperclip Runner", async () => { const saved = await resolveNewAgentRunnerForCompany({} as never, "company", { adapterType: "claude_local", defaultEnvironmentId: "computer" }); expect(saved).toMatchObject({ adapterType: "paperclip_runner", adapterConfig: { provider: "acpx", acpxAgent: "claude" } }); diff --git a/server/src/services/agent-runner-selection.ts b/server/src/services/agent-runner-selection.ts index 904e3c7aae3..77ecf01cfe7 100644 --- a/server/src/services/agent-runner-selection.ts +++ b/server/src/services/agent-runner-selection.ts @@ -15,6 +15,7 @@ export type RunnerTarget = { driver: string; platform?: string; architecture?: string; libc?: "glibc" | "musl" | "unknown"; glibcVersion?: string; /** A plugin environment whose driver declares Runner task execution. It runs agents only through Paperclip Runner. */ runnerTasks?: boolean; + warmTaskVersion?: 1; }; export const AGENT_RUNNER_SSH_PLATFORM_COMMAND = "uname -s; uname -m; if paperclip_glibc_version=$(getconf GNU_LIBC_VERSION 2>/dev/null) && [ -n \"$paperclip_glibc_version\" ]; then printf 'glibc\\n%s\\n' \"$paperclip_glibc_version\"; else case \"$(ldd --version 2>&1)\" in *musl*) printf 'musl\\n' ;; *) printf 'unknown\\n' ;; esac; fi"; export interface AgentRunnerSelection { @@ -293,7 +294,10 @@ export async function resolveAgentRunnerTargetForCompany(db: Db, companyId: stri const { environment, driver } = await resolveAgentRunnerEnvironmentForCompany(db, companyId, defaultEnvironmentId); let target: RunnerTarget = driver === "local" ? localAgentRunnerTarget() : { driver }; - if (driver === "plugin" && environment && await pluginEnvironmentRunsRunnerTasks(db, environment)) target = { driver, runnerTasks: true }; + if (driver === "plugin" && environment) { + const taskDriver = await pluginEnvironmentRunnerTaskDriver(db, environment); + if (taskDriver) target = { driver, runnerTasks: true, ...(taskDriver.warmTaskVersion === 1 ? { warmTaskVersion: 1 as const } : {}) }; + } if (driver === "sandbox" && environment) { const { readSandboxPlatformQualification } = await import("./environment-probe.js"); const qualified = await readSandboxPlatformQualification(db, environment, companyId); @@ -312,6 +316,18 @@ export async function resolveAgentRunnerTargetForCompany(db: Db, companyId: stri } export async function resolveAgentRunnerEnvironmentForCompany(db: Db, companyId: string, defaultEnvironmentId?: string | null) { + const resolved = await resolveAgentEnvironmentForCompany(db, companyId, defaultEnvironmentId); + if (resolved.environment?.driver === "sandbox" && resolved.environment.config?.provider === "fake") throw unprocessable("The fake sandbox provider cannot execute runs."); + return resolved; +} + +// Hiring defaults inspect capabilities without probing or qualifying an already selected runner. +export async function resolveAgentRunnerTaskDriverForCompany(db: Db, companyId: string, defaultEnvironmentId?: string | null) { + const { environment, driver } = await resolveAgentEnvironmentForCompany(db, companyId, defaultEnvironmentId); + return driver === "plugin" && environment ? pluginEnvironmentRunnerTaskDriver(db, environment) : undefined; +} + +async function resolveAgentEnvironmentForCompany(db: Db, companyId: string, defaultEnvironmentId?: string | null) { const { instanceSettingsService } = await import("./instance-settings.js"); const { environmentService } = await import("./environments.js"); const settings = await instanceSettingsService(db).get(); @@ -331,7 +347,6 @@ export async function resolveAgentRunnerEnvironmentForCompany(db: Db, companyId: const boundCompanyIds = await envs.listBoundCompanyIds(environment.id); if (boundCompanyIds.length > 0 && !boundCompanyIds.includes(companyId)) throw forbidden("The selected environment belongs to another company.", { code: "environment_company_mismatch" }); if (environment.status === "archived") throw unprocessable("Environment is archived."); - if (environment.driver === "sandbox" && environment.config?.provider === "fake") throw unprocessable("The fake sandbox provider cannot execute runs."); } return { environment, driver: environment?.driver ?? "local" }; } @@ -339,17 +354,17 @@ export async function resolveAgentRunnerEnvironmentForCompany(db: Db, companyId: // The plugin's manifest declaration decides whether its environment runs // Runner tasks. Run admission separately verifies a ready plugin and a live // worker, so a plugin that is down still selects the Runner and fails its run. -async function pluginEnvironmentRunsRunnerTasks(db: Db, environment: Parameters[0]): Promise { +async function pluginEnvironmentRunnerTaskDriver(db: Db, environment: Parameters[0]) { let config: { pluginKey: string; driverKey: string }; try { const parsed = parseEnvironmentDriverConfig(environment); - if (parsed.driver !== "plugin") return false; + if (parsed.driver !== "plugin") return undefined; config = parsed.config; } catch { - return false; + return undefined; } const { pluginRegistryService } = await import("./plugin-registry.js"); const plugin = await pluginRegistryService(db).getByKey(config.pluginKey); - return Boolean(plugin?.manifestJson.capabilities.includes("environment.drivers.register") - && plugin.manifestJson.environmentDrivers?.some(driver => driver.driverKey === config.driverKey && driver.supportsTasks === true)); + return plugin?.manifestJson.capabilities.includes("environment.drivers.register") + ? plugin.manifestJson.environmentDrivers?.find(driver => driver.driverKey === config.driverKey && driver.supportsTasks === true) : undefined; } diff --git a/server/src/services/environment-run-orchestrator.ts b/server/src/services/environment-run-orchestrator.ts index caa024c79a4..065108ee6b8 100644 --- a/server/src/services/environment-run-orchestrator.ts +++ b/server/src/services/environment-run-orchestrator.ts @@ -273,6 +273,7 @@ export function environmentRunOrchestrator( adapterType: string; adapterConfig?: Record; admittedLifecycleMode?: "warm" | "per_turn"; + managedAiCredentialIdentity?: string; issueId: string | null; heartbeatRunId: string; agentId: string; @@ -292,7 +293,7 @@ export function environmentRunOrchestrator( }); const environment = resolveRunnerEnvironmentForRun( - selectedEnvironment, input.adapterType, input.adapterConfig, input.admittedLifecycleMode, + selectedEnvironment, input.adapterType, input.adapterConfig, input.admittedLifecycleMode, input.managedAiCredentialIdentity, ); if (input.recoverPluginTask && environment.driver === "plugin" && input.adapterType === "paperclip_runner") { diff --git a/server/src/services/environment-task-runtime.ts b/server/src/services/environment-task-runtime.ts index e58491b01fa..020cbca36d6 100644 --- a/server/src/services/environment-task-runtime.ts +++ b/server/src/services/environment-task-runtime.ts @@ -85,10 +85,11 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage throw new Error("Plugin-provided Runner execution identity mismatch"); } if (runner && resumeFrom) { - const sources = await db.select({ runId: environmentLeases.heartbeatRunId, metadata: environmentLeases.metadata }).from(environmentLeases).where(and( + const inspectCheckpoint = operation.kind === "read_state"; + const sources = await db.select({ id: environmentLeases.id, runId: environmentLeases.heartbeatRunId, metadata: environmentLeases.metadata }).from(environmentLeases).where(and( eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.provider, lease.provider!), eq(environmentLeases.providerLeaseId, resumeFrom), - sql`${environmentLeases.metadata}->>'runnerTaskSessionRetained' = 'true'`, + inspectCheckpoint ? undefined : sql`${environmentLeases.metadata}->>'runnerTaskSessionRetained' = 'true'`, checkpointIdentity ? eq(environmentLeases.heartbeatRunId, checkpointIdentity.runId) : undefined, )).limit(2); const priorRunId = sources.length === 1 ? sources[0]!.runId : null; @@ -98,7 +99,20 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage companyId: input.companyId, leaseId: lease.id, priorIdentity: { runnerInstanceId: runner.runnerId, environmentLeaseId: runner.leaseId, runId: priorRunId, normalizedSessionId: runner.sessionId }, + inspectCheckpoint, })) throw new Error("Plugin Runner resume identity mismatch"); + // Older providers misclassified suspended checkpoints. A fresh provider + // attestation may repair availability, but never supplies stop/owner authority. + if (inspectCheckpoint && sources[0]!.metadata?.runnerTaskSessionRetained !== true) { + const status = await executeEnvironmentTask(db, workers, { companyId: input.companyId, + leaseId: sources[0]!.id, operation: { kind: "status" } }); + if (status.kind !== "status" || status.executionStopped !== true || status.sessionStateAvailable !== true || + resumeFrom !== await resolvePluginTaskResume(db, { companyId: input.companyId, leaseId: lease.id, + priorIdentity: { runnerInstanceId: runner.runnerId, environmentLeaseId: runner.leaseId, + runId: priorRunId, normalizedSessionId: runner.sessionId } })) { + throw new Error("Plugin Runner retained session is not authorized"); + } + } } if (metadata.driver !== "plugin" || typeof metadata.pluginId !== "string" || typeof metadata.driverKey !== "string") { throw new Error("Plugin-provided Runner execution lease has no pinned plugin driver"); @@ -380,6 +394,8 @@ export async function settleDispatchedEnvironmentTask(db: Db, workers: PluginWor export async function resolvePluginTaskResume(db: Db, input: { companyId: string; leaseId: string; priorIdentity: { runnerInstanceId: string; environmentLeaseId: string; runId: string; normalizedSessionId: string }; + /** Read-only discovery; launching still requires an attested retained checkpoint. */ + inspectCheckpoint?: boolean; }): Promise { const rows = await db.select({ lease: environmentLeases, run: heartbeatRuns }).from(environmentLeases) .innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), eq(heartbeatRuns.companyId, input.companyId))) @@ -389,7 +405,8 @@ export async function resolvePluginTaskResume(db: Db, input: { // A retained PRP lease is logical; later provider tasks have different physical lease IDs. const candidates = rows.filter(row => { const identity = row.lease.metadata?.runnerTaskIdentity as Record | undefined; - return row.lease.id !== input.leaseId && row.run.id === input.priorIdentity.runId && row.lease.metadata?.runnerTaskSessionRetained === true && + return row.lease.id !== input.leaseId && row.run.id === input.priorIdentity.runId && + (input.inspectCheckpoint || row.lease.metadata?.runnerTaskSessionRetained === true) && identity?.runnerId === input.priorIdentity.runnerInstanceId && identity.leaseId === input.priorIdentity.environmentLeaseId && identity.sessionId === input.priorIdentity.normalizedSessionId && identity.runId === input.priorIdentity.runId && !rows.some(successor => successor.lease.id !== input.leaseId && diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 72a79b53199..59d2bd04a61 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -2713,6 +2713,7 @@ export function heartbeatService( mergedConfig, configSnapshot, secretManifest, + managedAiCredentialIdentity: managedAiRuntime?.identity, }, reuse: { requestedShouldReuseExisting, diff --git a/server/src/services/heartbeat/runtime-selection.test.ts b/server/src/services/heartbeat/runtime-selection.test.ts index f3ce34c7759..737f69beffa 100644 --- a/server/src/services/heartbeat/runtime-selection.test.ts +++ b/server/src/services/heartbeat/runtime-selection.test.ts @@ -311,6 +311,26 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () = expect(await coordinators(run.id)).toHaveLength(1); }); + it.each(["api_key", "subscription"] as const)("keeps managed Claude %s task execution warm", async method => { + const { input, run, issue } = await fixture(); + const adapterConfig = { provider: "acpx", acpxAgent: "claude", model: "claude-sonnet-4-6", lifecycleMode: "warm" }; + input.agent = { ...input.agent, adapterConfig }; + await db.update(agents).set({ adapterConfig }).where(eq(agents.id, input.agent.id)); + input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({ persisted: input.run, enabled: true, runtimeConfig: {}, adapterConfig, + agent: input.agent, issue, target: null, workspaceId: null }); + input.workspace.executionTarget = { ...sandbox, runnerTask: vi.fn(), reusableLeaseConfigured: true, + effectiveCapabilities: { reusableLeases: true } as never, runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 } }; + const attribution = { connectionId: randomUUID(), grantId: randomUUID(), provider: "anthropic" as const, method, + mode: "responsible_user" as const, responsibleUserId: "task-owner" }; + input.config.managedAiRuntime = { sessionIdentity: "account:epoch-1", identity: "account:generation-1", + config: { env: {}, managedAiConnection: { ...attribution, identity: "account:generation-1", sessionIdentity: "account:epoch-1" } }, + attribution, accountName: "Saved account", accountOwnerUserId: "task-owner", home: instructionRoot, cleanup: async () => {} }; + const selected = await selectHeartbeatRuntime(db, input); + if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native Claude selection"); + expect(selected.nativeExecution.session.lifecyclePolicy).toEqual({ mode: "warm", idleTimeoutMs: 300_000 }); + expect((await read(run.id)).runnerProfileJson).toMatchObject({ nativeExecutionInput: { session: { lifecyclePolicy: { mode: "warm" } } } }); + }); + it("preserves only server-owned admission and dispatch evidence when recording legacy selection", async () => { const { input, run } = await fixture("process"); const admission = { version: 1, kind: "fixture" }, dispatch = { entered: true }; diff --git a/server/src/services/heartbeat/runtime-selection.ts b/server/src/services/heartbeat/runtime-selection.ts index 05a7b039a2f..032f3875580 100644 --- a/server/src/services/heartbeat/runtime-selection.ts +++ b/server/src/services/heartbeat/runtime-selection.ts @@ -506,11 +506,10 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele executionTarget.transport === "sandbox" ? (executionTarget.runnerLifecyclePolicy ?? null) : null; - // Codex refreshes its durable auth home; Copilot owns a fenced home - // and an explicitly bound token. Both retire on credential generation - // changes. Other managed harnesses require per-turn cleanup. + // Task providers own credential lifetime; process-backed managed homes retain their provider-specific cleanup policy. const supportsManagedWarmSession = agent.adapterType === "paperclip_runner" && - supportsManagedNativeWarmSession({ ...nativeRuntimeResolution.profile, acpxAgent: parseObject(agent.adapterConfig).acpxAgent }, managedAiRuntime?.attribution); + ((executionTarget?.kind === "remote" && executionTarget.transport === "sandbox" && !!executionTarget.runnerTask) || + supportsManagedNativeWarmSession({ ...nativeRuntimeResolution.profile, acpxAgent: parseObject(agent.adapterConfig).acpxAgent }, managedAiRuntime?.attribution)); const effectiveLifecyclePolicy = persistedNativeExecutionInput?.session.lifecyclePolicy ?? (["openai_dot_mcp", "grok_bot_mcp", "muse_external"].includes(nativeRuntimeResolution.profile.backend) || managedAiRuntime && !supportsManagedWarmSession ? { mode: "per_turn" as const, idleTimeoutMs: null } diff --git a/server/src/services/heartbeat/workspace-preparation.test.ts b/server/src/services/heartbeat/workspace-preparation.test.ts index eb0f0e2abc3..d3c9944abd5 100644 --- a/server/src/services/heartbeat/workspace-preparation.test.ts +++ b/server/src/services/heartbeat/workspace-preparation.test.ts @@ -456,10 +456,12 @@ describe.skipIf(!support.supported)("heartbeat workspace preparation boundary", it("persists the workspace and issue binding before acquiring its environment", async () => { const { input, source, issue, envOrchestrator } = await fixture(); + input.config.managedAiCredentialIdentity = "account:generation"; const acquire = envOrchestrator.acquireForRun; vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => { expect((await readIssue(issue.id)).executionWorkspaceId).toBe(args.persistedExecutionWorkspace?.id); expect((await readRun(input.run.id)).contextSnapshot?.executionWorkspaceId).toBe(args.persistedExecutionWorkspace?.id); + expect(args.managedAiCredentialIdentity).toBe("account:generation"); return acquire(args); }); const result = await prepareHeartbeatWorkspace(db, input); diff --git a/server/src/services/heartbeat/workspace-preparation.ts b/server/src/services/heartbeat/workspace-preparation.ts index f56bde29125..d81b24cd919 100644 --- a/server/src/services/heartbeat/workspace-preparation.ts +++ b/server/src/services/heartbeat/workspace-preparation.ts @@ -129,6 +129,7 @@ export interface HeartbeatWorkspacePreparationInput { mergedConfig: Record; configSnapshot: WorkspaceMetadataInput["configSnapshot"]; secretManifest: WorkspaceMetadataInput["secretManifest"]; + managedAiCredentialIdentity?: string; }; reuse: { requestedShouldReuseExisting: boolean; @@ -1121,6 +1122,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace adapterType: agent.adapterType, adapterConfig: parseObject(agent.adapterConfig), admittedLifecycleMode: persistedNativeExecutionInput?.session.lifecyclePolicy.mode, + managedAiCredentialIdentity: input.config.managedAiCredentialIdentity, runnerTaskProjectIds:[...new Set([executionWorkspace.projectId,...resolvedWorkspace.runnerTaskProjectIds ?? [], ...(executionWorkspace.additionalWorkspaces ?? []).map(workspace=>workspace.projectId)] .filter((id):id is string=>Boolean(id)))], diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index de8105859b1..30d8792a64d 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -11128,13 +11128,15 @@ describe("runnerd provider runtime wiring", () => { await rm(isolatedStateDirectory, { recursive: true, force: true }); }); - it.each(["unchanged", "changed", "runtime-context", "retirement-failed", "stale-owner", "active-owner", "missing-owner", "foreign-environment", "plugin-replaced", "plugin-managed-github"] as const)("rotates an owned warm remote runner before checking changed configuration: %s", async scenario => { + it.each((["codex", "claude"] as const).flatMap(provider => (["unchanged", "changed", "runtime-context", "retirement-failed", "stale-owner", "active-owner", "missing-owner", "foreign-environment", "plugin-replaced", "plugin-managed-github"] as const).filter(scenario => provider === "codex" || scenario.startsWith("plugin-")).map(scenario => [provider, scenario] as const)))("rotates an owned warm %s remote runner before checking changed configuration: %s", async (provider, scenario) => { const plugin = scenario === "plugin-replaced" || scenario === "plugin-managed-github"; const reusable = scenario === "unchanged" || scenario === "plugin-managed-github"; - const first = { ...scopedNativeExecution(`warm-admission-${scenario}`), + const { permissionPolicy: _permissionPolicy, ...claudeProfile } = QUALIFIED_ACPX_PROFILES.claude; + const first = { ...scopedNativeExecution(`warm-admission-${provider}-${scenario}`), schema: "paperclip.native-execution-input.v5" as const, - provider: { kind: "codex" as const, model: null, approvalPolicy: "never" as const }, + provider: provider === "claude" ? { kind: "acpx" as const, agent: "claude" as const, model: "claude-sonnet-4-6", permissionMode: "approve-all" as const, profile: claudeProfile } : { kind: "codex" as const, model: null, approvalPolicy: "never" as const }, executionMode: "default" as const, planningContext: null, runtimeContext: nativeRuntimeContextFixture() }; + if (provider === "claude") first.session.driverKind = "acpx_runtime"; first.session.lifecyclePolicy = { mode: "warm", idleTimeoutMs: 60_000 }; const next = { ...first, binding: { ...first.binding, runId: `${first.binding.runId}-next` }, session: { ...first.session, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: ["unchanged", "runtime-context", "stale-owner", "missing-owner", "plugin-replaced", "plugin-managed-github"].includes(scenario) ? 60_000 : 90_000 } } }; @@ -11161,7 +11163,7 @@ describe("runnerd provider runtime wiring", () => { if(plugin) { await mkdir(home,{recursive:true});await writeFile(join(home,"auth.json"),JSON.stringify({OPENAI_API_KEY:"synthetic-fixture"}),{mode:0o600}); } - const pluginOptions=plugin ? {runnerIngressAuthorized:true,managedGitHub:scenario === "plugin-managed-github",managedAiCredentialHome:home, + const pluginOptions=plugin ? {runnerIngressAuthorized:true,runnerEnvironment:provider === "claude" ? {ANTHROPIC_API_KEY:"synthetic-fixture"} : {},managedGitHub:scenario === "plugin-managed-github",managedAiCredentialHome:home, managedAiCredentialIdentity:"revision",managedAiAccountIdentity:"account"} : {}; const resume=scenario === "plugin-replaced" ? vi.spyOn(environmentTaskRuntime,"resolvePluginTaskResume").mockResolvedValue("task-first") : undefined; const createOwner = pluginTaskRuntime.createPluginRunnerTaskOwner; @@ -11862,6 +11864,103 @@ describe("runnerd provider runtime wiring", () => { } }); + it.each(["claude", "codex"].flatMap(provider => (["per_turn", "warm"] as const).flatMap(policy => ["canonical", "local-receipt", "local-receipt-missing-terminal", "local-receipt-pending-command", "quarantined", "empty-runner", "empty-runner-quarantined", "nonempty-runner", "wrong-identity", "active", "foreign-scope", "no-stop-proof", "not-suspended", "pending-outbox", "pending-command", "missing-terminal", "ambiguous", "contradictory"].map(mode => [provider, policy, mode] as const))))( + "recovers a blocked %s %s plugin checkpoint on a replacement lease: %s", async (provider, policy, mode) => { + const { permissionPolicy: _permissionPolicy, ...claudeProfile } = QUALIFIED_ACPX_PROFILES.claude; + const first = { ...scopedNativeExecution(`plugin-${provider}-${policy}-${mode}`), + schema: "paperclip.native-execution-input.v5", executionMode: "default", planningContext: null, runtimeContext: nativeRuntimeContextFixture(), + ...(provider === "claude" ? { provider: { kind: "acpx", agent: "claude", model: "claude-sonnet-4-6", permissionMode: "approve-all", profile: claudeProfile } } : { provider: { kind: "codex", model: null, approvalPolicy: "never" } }), + session: { ...execution.session, driverKind: provider === "claude" ? "acpx_runtime" : "codex_app_server", + normalizedSessionId: `plugin-${provider}-${policy}-${mode}`, lifecyclePolicy: policy === "warm" ? { mode: "warm", idleTimeoutMs: 60_000 } : { mode: "per_turn", idleTimeoutMs: null } }, + } as NativeExecutionInput; + parseNativeExecutionInput(first); + const next = { ...first, binding: { ...first.binding, runId: `${first.binding.runId}-reopened` } }; + const identity = { runId: first.binding.runId, normalizedSessionId: first.session.normalizedSessionId!, + runnerInstanceId: "plugin-runner", environmentLeaseId: "old-lease", turnId: "old-turn", itemId: "old-item" }; + const scoped = await initializeScopedRunnerRoot(first, identity.runnerInstanceId); + await mkdir(join(scoped, "control-plane"), { recursive: true }); + const journal = JSON.stringify({ ...durableControlPlaneState(identity), commands: [{ status: mode.endsWith("pending-command") ? "pending" : "completed" }], + committedEvents: mode.endsWith("missing-terminal") ? [] : [{ eventType: "run.terminal", envelope: { ...identity, payload: { reportedWorkDisposition: "blocked" } } }] }); + await writeFile(join(scoped, "control-plane/control-plane-state.json"), journal); + if (mode.startsWith("empty-runner") || mode === "nonempty-runner") { + await mkdir(join(scoped, "runner")); + if (mode === "nonempty-runner") await writeFile(join(scoped, "runner", "unknown.json"), "{}"); + } + if (mode.startsWith("local-receipt")) await writeFile(join(scoped, "plugin-runner-state.json"), JSON.stringify(durableRunnerState(identity, "suspended"))); + let root = scoped; + if (["quarantined", "empty-runner-quarantined", "ambiguous"].includes(mode)) { + await mkdir(join(isolatedStateDirectory, "quarantine"), { recursive: true }); + root = join(isolatedStateDirectory, "quarantine", `${scoped.split("/").at(-1)}.identity_indeterminate.1`); + await rename(scoped, root); + if (mode === "ambiguous") await cp(root, `${root}-second`, { recursive: true }); + } + if (mode === "contradictory") { + await mkdir(join(root, "runner")); + await writeFile(join(root, "runner/runner-state.json"), "{"); + } + const db = leaseDb(next); + const ordinarySelect = db.select.bind(db); + db.select = ((fields?: Record) => fields?.status === heartbeatRuns.status && fields?.runnerProfileJson === heartbeatRuns.runnerProfileJson + ? priorNativeRunDb(mode === "foreign-scope" ? { ...first, binding: { ...first.binding, agentId: "foreign" } } : first, + mode === "active" ? "running" : "succeeded").select(fields as never) + : ordinarySelect(fields as never)) as Db["select"]; + const throughExecutor = policy === "warm" && ["canonical", "local-receipt", "pending-outbox"].includes(mode); + let inspected = false; + const resume = vi.spyOn(environmentTaskRuntime, "resolvePluginTaskResume").mockImplementation(async (_db, request) => { + if (request.inspectCheckpoint) inspected = true; + else if (throughExecutor && !inspected) throw new Error("stale negative checkpoint availability"); + if (mode === "no-stop-proof") throw new Error("Plugin Runner retained session is not authorized"); + return "old-task"; + }); + const task = vi.fn(async (operation: any) => { + environmentTaskOperationSchema.parse(operation); + if (operation.kind === "read_state") return { kind: "state", taskId: "replacement-task", state: { + ...durableRunnerState(identity, mode === "not-suspended" ? "ready" : "suspended"), + outbox: mode === "pending-outbox" ? [{}] : [], pendingTerminalDelivery: null, + ...(mode === "wrong-identity" ? { itemId: "foreign" } : {}), + } }; + return { kind: "description", taskId: "replacement-task", launchPreparationVersion: 1, projectAuthCheckVersion: 1, warmTaskVersion: 1, + runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` } }; + }); + const owner = vi.spyOn(pluginTaskRuntime, "createPluginRunnerTaskOwner"); + state.createBackend.mockClear(); + const home = join(isolatedStateDirectory, "checkpoint-account"); + await mkdir(home, { recursive: true }); + await writeFile(join(home, "auth.json"), JSON.stringify({ OPENAI_API_KEY: "synthetic-fixture" }), { mode: 0o600 }); + try { + const launch = { db, execution: next, runnerInstanceId: "replacement-runner", runnerIngressAuthorized: true, + managedAiCredentialHome: home, managedAiCredentialIdentity: "revision", managedAiAccountIdentity: "account", + runnerEnvironment: { ANTHROPIC_API_KEY: "synthetic-fixture" }, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".", leaseId: "replacement-lease", runnerTask: task, + sandboxLeaseAcquisition: { outcome: "replacement", providerLeaseId: "replacement-task" } } as never }; + if (throughExecutor) { + const validated = validatePrpStructuredRunResult({ schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary: "done", + completionClaim: { contractRevision: next.completionContract.contract.revision, objectiveSatisfied: true, criteria: [], remainingWork: [] }, + evidence: [], verification: [], artifacts: [], attentionRequests: [] }); + if (!validated.ok) throw new Error("Invalid completion fixture"); + state.execute.mockResolvedValueOnce({ result: validated.result, terminal: { runTerminalState: "succeeded" }, turnId: "turn", + normalizedSessionId: next.session.normalizedSessionId, providerSessionId: "original-provider-history", driverKind: next.session.driverKind, + driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null }); + } + const backend = throughExecutor ? executePaperclipNativeSession({ ...launch, useRunnerd: true }) : createRunnerdBackend(launch); + if (["canonical", "local-receipt", "quarantined", "empty-runner", "empty-runner-quarantined"].includes(mode)) { + await expect(backend).resolves.toBeDefined(); + expect(state.createBackend).toHaveBeenCalledOnce(); + expect(owner).toHaveBeenCalledWith(expect.objectContaining({ resumeFrom: "old-task", priorIdentity: identity })); + expect(resume).toHaveBeenCalledWith(db, expect.objectContaining({ leaseId: "replacement-lease", priorIdentity: identity, inspectCheckpoint: true })); + expect(JSON.parse(await readFile(join(scoped, "plugin-runner-state.json"), "utf8"))).toMatchObject({ ...identity, lifecycle: "suspended" }); + expect(await readFile(join(scoped, "control-plane/control-plane-state.json"), "utf8")).toBe(journal); + } else { + await expect(backend).rejects.toThrow(); + expect(state.createBackend).not.toHaveBeenCalled(); + if (["local-receipt-missing-terminal", "local-receipt-pending-command", "active", "foreign-scope", "pending-command", "missing-terminal", "ambiguous", "contradictory", "nonempty-runner", "no-stop-proof"].includes(mode)) { + expect(task).not.toHaveBeenCalled(); + } + if (mode === "ambiguous") expect(await readFile(join(root, "control-plane/control-plane-state.json"), "utf8")).toBe(journal); + } + } finally { owner.mockRestore(); resume.mockRestore(); } + }); + it.each(["verified", "wrong-state", "no-stop-proof", "disposition-retry", "disposition-wrong-state", "disposition-no-stop-proof"] as const)("admits only an authorized same-run plugin checkpoint: %s", async mode => { const current = scopedNativeExecution(`plugin-crash-${mode}`); const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!, @@ -11969,10 +12068,11 @@ describe("runnerd provider runtime wiring", () => { } finally { owner.mockRestore(); } }); - it.each(["codex", "fresh", "missing-capability", "missing-account", "acpx"] as const)("negotiates task-only warm execution: %s", async mode => { + it.each(["codex", "fresh", "missing-capability", "missing-account", "acpx", "installed-codex", "installed-claude"] as const)("negotiates task-only execution: %s", async mode => { const current = scopedNativeExecution(`plugin-warm-${mode}`); - current.session.lifecyclePolicy = {mode:"warm",idleTimeoutMs:300000}; - if(mode==="acpx") current.provider={kind:"acpx",agent:"claude",model:"fixture",permissionMode:"deny-all"} as never; + const installed = mode.startsWith("installed-"); + current.session.lifecyclePolicy = installed ? {mode:"per_turn",idleTimeoutMs:null} : {mode:"warm",idleTimeoutMs:300000}; + if(mode==="acpx" || mode==="installed-claude") current.provider={kind:"acpx",agent:"claude",model:"fixture",permissionMode:"deny-all"} as never; const home=await mkdtemp(join(tmpdir(),"plugin-warm-account-")); await writeFile(join(home,"auth.json"),JSON.stringify({OPENAI_API_KEY:"synthetic-fixture"}),{mode:0o600}); const owner=vi.spyOn(pluginTaskRuntime,"createPluginRunnerTaskOwner"); @@ -11983,20 +12083,22 @@ describe("runnerd provider runtime wiring", () => { state.createBackend.mockClear();state.createTransport.mockClear(); try { const backend=createRunnerdBackend({db:leaseDb(current),execution:current,runnerInstanceId:"runner",runnerIngressAuthorized:true, - managedAiCredentialHome:home,managedAiCredentialIdentity:mode==="missing-account" ? undefined : "revision",managedAiAccountIdentity:"account", + runnerEnvironment:{ANTHROPIC_API_KEY:"synthetic-fixture"},managedAiCredentialHome:home,managedAiCredentialIdentity:mode==="missing-account" || installed ? undefined : "revision",managedAiAccountIdentity:installed ? undefined : "account", runnerExecutionTarget:{kind:"remote",transport:"sandbox",leaseId:"lease",remoteCwd:".",runnerTask:task, ...(mode==="fresh" ? {sandboxLeaseAcquisition:{outcome:"resumed",providerLeaseId:"old-task"}} : {})}}); - if(mode!=="codex" && mode!=="fresh") {await expect(backend).rejects.toThrow(mode==="acpx" ? "Codex workspace" : "capability or account");return;} + if(mode==="missing-capability" || mode==="missing-account") {await expect(backend).rejects.toThrow("capability or account");return;} await backend; if(mode==="fresh") expect(task.mock.calls.map(([operation])=>operation.kind)).toEqual(["stop","status","complete","describe"]); state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); - expect(state.createTransport.mock.calls.at(-1)![0]).toMatchObject({lifecyclePolicy:{mode:"warm",idleTimeoutMs:300000}}); - expect(owner.mock.calls.at(-1)![0]).toMatchObject({warmTaskVersion:1,accountIdentity:"account",onCredentials:undefined}); + expect(state.createTransport.mock.calls.at(-1)![0]).toMatchObject({lifecyclePolicy:current.session.lifecyclePolicy}); + expect(owner.mock.calls.at(-1)![0]).toMatchObject({warmTaskVersion:1,onCredentials:undefined}); + expect(owner.mock.calls.at(-1)![0].accountIdentity).toBe(installed ? undefined : "account"); } finally {owner.mockRestore();await rm(home,{recursive:true,force:true});} }); - it.each([true,false])("reconnects task-only warm recovery under the saved identity before reuse or retirement: %s", async reuseProcess => { - const current=scopedNativeExecution(`plugin-restart-${reuseProcess}`); + it.each((["codex", "claude"] as const).flatMap(provider => [true, false].map(reuseProcess => [provider, reuseProcess] as const)))("reconnects task-only %s warm recovery under the saved identity before reuse or retirement: %s", async (provider, reuseProcess) => { + const current=scopedNativeExecution(`plugin-restart-${provider}-${reuseProcess}`); + if(provider === "claude") current.provider={kind:"acpx",agent:"claude",model:"fixture",permissionMode:"deny-all"} as never; current.session.lifecyclePolicy={mode:"warm",idleTimeoutMs:300000}; const identity={runnerInstanceId:"old-runner",environmentLeaseId:"old-lease",runId:"old-run", normalizedSessionId:current.session.normalizedSessionId!,turnId:"old-turn",itemId:"old-item"}; @@ -12008,7 +12110,7 @@ describe("runnerd provider runtime wiring", () => { state.createBackend.mockClear();state.createTransport.mockClear(); try { await createRunnerdBackend({db:leaseDb(current),execution:current,runnerInstanceId:"new-runner",runnerIngressAuthorized:true, - managedAiCredentialHome:home,managedAiCredentialIdentity:reuseProcess ? "old-revision" : "new-revision",managedAiAccountIdentity:"account", + runnerEnvironment:{ANTHROPIC_API_KEY:"synthetic-fixture"},managedAiCredentialHome:home,managedAiCredentialIdentity:reuseProcess ? "old-revision" : "new-revision",managedAiAccountIdentity:"account", managedAiCredentialWriteback:{} as never, remoteWarmRecovery:{identity,reuseProcess,previousExecution:current,checkpoint:{} as never, remote:{providerLeaseId:"saved-task",remoteCwd:"."},revalidate:async()=>{}}, diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 4817a631bf9..15f54f4a5bb 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -2062,29 +2062,34 @@ function savePluginRunnerCheckpoint(root: string, state: Record renameSync(temporary, path); } +function pluginRunnerdAuthorityLifecycle( + root: string, + identity: RunnerdDurableIdentity, +): "absent" | "suspended" | "not_suspended" | "indeterminate" { + try { + const state = record(JSON.parse(readBoundedNativeFile(resolve(root, "plugin-runner-state.json"), + NATIVE_RUNNER_STATE_MAX_BYTES, "runner_state_too_large").toString("utf8"))); + if (state.schema !== RUNNERD_STATE_SCHEMA || !RUNNERD_STATE_LIFECYCLES.has(String(state.lifecycle)) + || !Object.entries(identity).every(([key, value]) => state[key] === value)) return "indeterminate"; + return state.lifecycle === "suspended" ? "suspended" : "not_suspended"; + } catch (error) { return (error as NodeJS.ErrnoException).code === "ENOENT" ? "absent" : "indeterminate"; } +} + function runnerdAuthorityLifecycle( root: string, identity: RunnerdDurableIdentity, ): "absent" | "suspended" | "not_suspended" | "indeterminate" { const runnerRoot = resolve(root, "runner"); - if (!existsSync(runnerRoot)) { - try { - const state = record(JSON.parse(readBoundedNativeFile(resolve(root, "plugin-runner-state.json"), - NATIVE_RUNNER_STATE_MAX_BYTES, "runner_state_too_large").toString("utf8"))); - if (state.schema !== RUNNERD_STATE_SCHEMA || !RUNNERD_STATE_LIFECYCLES.has(String(state.lifecycle)) - || !Object.entries(identity).every(([key, value]) => state[key] === value)) return "indeterminate"; - return state.lifecycle === "suspended" ? "suspended" : "not_suspended"; - } catch (error) { return (error as NodeJS.ErrnoException).code === "ENOENT" ? "absent" : "indeterminate"; } - } + if (!existsSync(runnerRoot)) return pluginRunnerdAuthorityLifecycle(root, identity); if (!isSafeNativeStateDirectory(runnerRoot)) return "indeterminate"; const statePath = resolve(runnerRoot, "runner-state.json"); if (!existsSync(statePath)) { try { // Older remote transports created this controller-side placeholder even // though runner state was owned by the sandbox. It carries no authority, - // so a verified failover backup may be consulted. Any non-empty direct + // so a verified plugin receipt or failover backup may be consulted. Any non-empty direct // directory remains indeterminate and therefore blocks fallback. - return readdirSync(runnerRoot).length === 0 ? "absent" : "indeterminate"; + return readdirSync(runnerRoot).length === 0 ? pluginRunnerdAuthorityLifecycle(root, identity) : "indeterminate"; } catch { return "indeterminate"; } @@ -4894,6 +4899,72 @@ async function verifyWarmTransitionRestart(input: { return verified; } +async function recoverPluginRunnerCheckpoint(input: { + db: Db; + execution: NativeExecutionInput; + runnerExecutionTarget?: AdapterExecutionTarget | null; +}, scoped: string): Promise { + const target = input.runnerExecutionTarget; + if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runnerTask || !target.leaseId || + target.sandboxLeaseAcquisition?.outcome === "resumed") return; + let root = scoped; + const quarantined = !existsSync(scoped); + const deny = (): never => { throw new Error("runner_state_identity_mismatch: plugin_checkpoint_recovery_unproven"); }; + if (quarantined) { + const quarantine = resolve(runnerdStateBase(), "quarantine"); + if (!existsSync(quarantine)) return; + if (!isSafeNativeStateDirectory(quarantine)) return deny(); + const candidates = readdirSync(quarantine).filter(name => name.startsWith(`${basename(scoped)}.identity_indeterminate.`)); + if (candidates.length === 0) return; + if (candidates.length !== 1) return deny(); + root = resolve(quarantine, candidates[0]!); + } + if (!existsSync(root)) return; + if (!isSafeNativeStateDirectory(root)) return deny(); + if (hasRetainedWarmTransitionEvidence(root)) { + if (quarantined) return deny(); + return; + } + const identity = readRunnerdDurableIdentity(root); + if (!durableIdentityMatchesSession(identity, input.execution) || identity.runId === input.execution.binding.runId) { + if (quarantined) return deny(); + return; + } + const direct = runnerdAuthorityLifecycle(root, identity); + const verifiedPluginReceipt = direct === "suspended" && + !existsSync(resolve(root, "runner", "runner-state.json")) && pluginRunnerdAuthorityLifecycle(root, identity) === "suspended"; + // Never hide contradictory direct authority behind a remote checkpoint. + if (direct !== "absent" && !(quarantined && direct === "suspended") && !verifiedPluginReceipt) { + if (quarantined) return deny(); + return; + } + await assertCleanupActivationCommitted(input.db, root, input.execution); + const prior = await verifyPriorRunnerdStateForSessionScope({ ...input, root, identity, allowVerifiedBackup: true }); + if (prior !== "terminal_state_indeterminate" && !((quarantined || verifiedPluginReceipt) && prior === "verified")) { + if (quarantined) return deny(); + return; + } + const control = record(JSON.parse(readBoundedNativeFile(resolve(root, "control-plane", "control-plane-state.json"), + NATIVE_CONTROL_PLANE_STATE_MAX_BYTES, "runner_durable_identity_too_large").toString("utf8"))); + const terminal = record(Array.isArray(control.committedEvents) ? control.committedEvents.at(-1) : null); + if (!Array.isArray(control.commands) || control.commands.some(command => record(command).status !== "completed") || + terminal.eventType !== "run.terminal" || record(terminal.envelope).runId !== identity.runId || + record(terminal.envelope).normalizedSessionId !== identity.normalizedSessionId) return deny(); + const resumeFrom = await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId, + leaseId: target.leaseId, priorIdentity: identity, inspectCheckpoint: true }); + const checkpoint = environmentTaskResultSchema.parse(await target.runnerTask({ kind: "read_state", identity, resumeFrom })); + if (checkpoint.kind !== "state" || checkpoint.state.lifecycle !== "suspended" || + !Array.isArray(checkpoint.state.outbox) || checkpoint.state.outbox.length !== 0 || + checkpoint.state.pendingTerminalDelivery != null || checkpoint.state.pendingProviderCleanup != null || + checkpoint.state.warmTransition != null) return deny(); + savePluginRunnerCheckpoint(root, checkpoint.state, identity); + if (await verifyPriorRunnerdStateForSessionScope({ ...input, root, identity, allowVerifiedBackup: true }) !== "verified") return deny(); + if (quarantined) { + if (existsSync(scoped)) return deny(); + renameSync(root, scoped); + } +} + async function migrateRunnerdStateRootForExecution(input: { db: Db; execution: NativeExecutionInput; @@ -4913,6 +4984,9 @@ async function migrateRunnerdStateRootForExecution(input: { issueId: input.execution.binding.issueId, stateKey: basename(scoped), }); + if (!input.restartRecovery && !input.retainedWarmOwnerRunId) { + await recoverPluginRunnerCheckpoint(input, scoped); + } if ( input.allowLocalRecovery && !input.retainedWarmOwnerRunId && @@ -8000,10 +8074,7 @@ async function executePaperclipNativeSessionWithinScope( // Inspection can replace a stopped retained task before deciding between adoption and checkpoint recovery. await taskTarget.runnerTask({kind:"describe"}); if (taskTarget.sandboxLeaseAcquisition?.outcome !== "resumed") { - const resumeFrom=await resolvePluginTaskResume(input.db,{companyId:input.execution.binding.companyId,leaseId:taskTarget.leaseId,priorIdentity:identity}); - const checkpoint=environmentTaskResultSchema.parse(await taskTarget.runnerTask({kind:"read_state",identity,resumeFrom})); - if(checkpoint.kind!=="state") throw new Error("Plugin Runner checkpoint unavailable"); - savePluginRunnerCheckpoint(root,checkpoint.state,identity); + await recoverPluginRunnerCheckpoint(input, root); } } if (durableIdentityMatchesSession(identity, input.execution) && identity.runId !== input.execution.binding.runId && @@ -11802,8 +11873,8 @@ async function createRunnerdBackendWithinSessionClaim( const remoteTarget = target.kind === "remote" ? target : null; const pluginTask = remoteTarget?.transport === "sandbox" ? remoteTarget.runnerTask : undefined; if (pluginTask && input.runnerIngressAuthorized !== true) throw new Error("runner_ingress_unavailable"); - if (pluginTask && (input.execution.schema === "paperclip.native-execution-input.v6" || input.execution.schema === "paperclip.native-execution-input.v8" || (input.execution.session.lifecyclePolicy.mode === "warm" && input.execution.provider.kind !== "codex"))) { - throw new Error("Plugin warm Runner tasks require a Codex workspace session"); + if (pluginTask && (input.execution.schema === "paperclip.native-execution-input.v6" || input.execution.schema === "paperclip.native-execution-input.v8")) { + throw new Error("Plugin Runner tasks require a workspace session"); } const remoteCommandRunner = remoteTarget ? remoteTarget.transport === "ssh" @@ -11992,7 +12063,7 @@ async function createRunnerdBackendWithinSessionClaim( .where(and(eq(environmentLeases.id, remoteTarget.leaseId!), eq(environmentLeases.companyId, input.execution.binding.companyId))); }, onDispatched: input.onTaskDispatched, }; - if (input.execution.session.lifecyclePolicy.mode === "warm" && (!credentials.accountIdentity || description.warmTaskVersion !== 1)) throw new Error("Plugin Codex warm task capability or account unavailable"); + if (input.execution.session.lifecyclePolicy.mode === "warm" && (!credentials.accountIdentity || description.warmTaskVersion !== 1)) throw new Error("Plugin warm task capability or account unavailable"); const owner = retainedOwner ?? createPluginRunnerTaskOwner(ownerInput); if (retainedOwner) retainedOwner.rebind(ownerInput); if (description.workspaceFileReadVersion === 1) { diff --git a/server/src/services/native-runtime/plugin-runner-task.test.ts b/server/src/services/native-runtime/plugin-runner-task.test.ts index 27f7f4f5364..a1f4bae2d98 100644 --- a/server/src/services/native-runtime/plugin-runner-task.test.ts +++ b/server/src/services/native-runtime/plugin-runner-task.test.ts @@ -23,8 +23,11 @@ const requirements: RunnerProviderLaunchRequirements = { settings: {}, approvalPolicy: "never", readOnly: false, preparedConversation: false, includeCollaborationModeInstructions: false, }; -async function prepare(owner: ReturnType) { - await owner.driver.prepareLaunch!(requirements); +function launchRequirements(harness: string): RunnerProviderLaunchRequirements { + return harness === "claude" ? { ...requirements, provider: "acpx", settings: { acpxAgent: "claude" } } : requirements; +} +async function prepare(owner: ReturnType, current = requirements) { + await owner.driver.prepareLaunch!(current); } function provider() { let stopped = false; @@ -323,14 +326,14 @@ describe("plugin task workspace file transfer", () => { }); }); -it("reuses a warm Codex process across run authorities without idle status polling", async () => { +it.each(["codex", "claude"])("reuses a warm %s process across run authorities without idle status polling", async harness => { const {task} = provider(); ingress.connect.mockReturnValue({ready:Promise.resolve(),close:async()=>{}}); const retained = vi.fn(async()=>{}); - const input = {task,projectIds:[],harness:"codex",projectAuthCheckVersion:1 as const,warmTaskVersion:1 as const, + const input = {task,projectIds:[],harness,projectAuthCheckVersion:1 as const,warmTaskVersion:1 as const, accountIdentity:"account",credentialIdentity:"revision",pollMs:1,onSessionRetained:retained}; const owner = createPluginRunnerTaskOwner(input); - await prepare(owner); + await prepare(owner, launchRequirements(harness)); const registration = await owner.registration({} as never,"run"); const spec = launchSpec(); spec.nativeRunner!.lifecyclePolicy = {mode:"warm",idleTimeoutMs:300000}; const handle = owner.launcher(spec); registration.activate(); await registration.ready(); @@ -345,7 +348,7 @@ it("reuses a warm Codex process across run authorities without idle status polli owner.rebind({...input,task:nextTask}); const nextRegistration=await owner.registration({} as never,"next-run"); const nextIdentity={...requirements.identity,runId:"next-run",turnId:"next-turn",itemId:"next-item"}; - await owner.driver.prepareLaunch!({...requirements,identity:nextIdentity}); + await owner.driver.prepareLaunch!({...launchRequirements(harness),identity:nextIdentity}); expect(nextTask.mock.calls[0][0]).toMatchObject({kind:"prepare",priorIdentity:requirements.identity,accountIdentity:"account"}); expect(nextTask.mock.calls.map(([op])=>op.kind)).toEqual(["prepare","check_project_auth"]); await registration.release(); nextRegistration.activate(); await nextRegistration.ready(); @@ -355,13 +358,13 @@ it("reuses a warm Codex process across run authorities without idle status polli expect(nextTask.mock.calls.filter(([op])=>op.kind==="stop")).toHaveLength(1); }); -it("adopts a verified retained Codex task without submitting a replacement", async () => { +it.each(["codex", "claude"])("adopts a verified retained %s task without submitting a replacement", async harness => { const {task}=provider(); ingress.connect.mockReturnValue({ready:Promise.resolve(),close:async()=>{}}); - const owner=createPluginRunnerTaskOwner({task,projectIds:[],harness:"codex",projectAuthCheckVersion:1, + const owner=createPluginRunnerTaskOwner({task,projectIds:[],harness,projectAuthCheckVersion:1, warmTaskVersion:1,accountIdentity:"account",adoptIdentity:requirements.identity,pollMs:1}); const recovered = {configurationId:"saved",requirementsDigest:`sha256:${"a".repeat(64)}`,provider:{}}; - await expect(owner.driver.prepareLaunch!(requirements,recovered)).resolves.toEqual(recovered); + await expect(owner.driver.prepareLaunch!(launchRequirements(harness),recovered)).resolves.toEqual(recovered); expect(task.mock.calls.map(([op])=>op.kind)).toEqual(["check_project_auth"]); expect(task.mock.calls[0][0]).toMatchObject({configurationId:"saved",projectIds:[]}); expect(await owner.driver.adoptExistingRunner!.isAlive()).toBe(true); @@ -372,28 +375,30 @@ it("adopts a verified retained Codex task without submitting a replacement", asy expect(task.mock.calls.some(([op])=>op.kind==="complete")).toBe(true); }); -it("denies retained process adoption when current project authentication is revoked",async()=>{ +it.each(["codex", "claude"])("denies retained %s process adoption when current project authentication is revoked",async harness=>{ const {task}=provider(); task.mockImplementation(async op=>op.kind === "check_project_auth" ? {kind:"project_auth",taskId:"task",configurationId:op.configurationId,provider:op.provider,allowed:false} as never : {kind:"accepted",taskId:"task"} as never); - const owner=createPluginRunnerTaskOwner({task,projectIds:["project"],harness:"codex",projectAuthCheckVersion:1, + const owner=createPluginRunnerTaskOwner({task,projectIds:["project"],harness,projectAuthCheckVersion:1, warmTaskVersion:1,accountIdentity:"account",adoptIdentity:requirements.identity}); - await expect(owner.driver.prepareLaunch!(requirements,{configurationId:"saved",requirementsDigest:`sha256:${"a".repeat(64)}`,provider:{}})) + await expect(owner.driver.prepareLaunch!(launchRequirements(harness),{configurationId:"saved",requirementsDigest:`sha256:${"a".repeat(64)}`,provider:{}})) .rejects.toThrow("project authentication conflicts"); expect(task.mock.calls.map(([op])=>op.kind)).toEqual(["check_project_auth"]); }); -it("refuses warm account or credential generation changes", async()=>{ +it.each(["codex", "claude"])("refuses warm %s account, harness, project, or credential generation changes", async harness=>{ const {task}=provider(); - const input={task,projectIds:[],harness:"codex",projectAuthCheckVersion:1 as const,warmTaskVersion:1 as const, + const input={task,projectIds:[],harness,projectAuthCheckVersion:1 as const,warmTaskVersion:1 as const, accountIdentity:"account",credentialIdentity:"revision",pollMs:1}; const owner=createPluginRunnerTaskOwner(input); - await prepare(owner); + await prepare(owner, launchRequirements(harness)); const spec=launchSpec();spec.nativeRunner!.lifecyclePolicy={mode:"warm",idleTimeoutMs:300000}; const process=owner.launcher(spec); expect(()=>owner.rebind({...input,accountIdentity:"other"})).toThrow("cannot be rebound"); expect(()=>owner.rebind({...input,credentialIdentity:"rotated"})).toThrow("cannot be rebound"); + expect(()=>owner.rebind({...input,harness:harness === "codex" ? "claude" : "codex"})).toThrow("cannot be rebound"); + expect(()=>owner.rebind({...input,projectIds:["foreign"]})).toThrow("cannot be rebound"); const registration=await owner.registration({} as never,"run"); await registration.checkpoint("unsettled"); await process.completion; }); diff --git a/server/src/services/native-runtime/plugin-runner-task.ts b/server/src/services/native-runtime/plugin-runner-task.ts index 8cbb01a9f4c..ca5e6f87cde 100644 --- a/server/src/services/native-runtime/plugin-runner-task.ts +++ b/server/src/services/native-runtime/plugin-runner-task.ts @@ -128,7 +128,7 @@ export function createPluginRunnerTaskOwner(input: { if (launched || released) throw new Error("Plugin Runner replacement requires a new task lease"); const native = spec.nativeRunner; if (!preparedLaunchId) throw new Error("Plugin Runner launch requires preparation"); - if (!native || (native.lifecyclePolicy?.mode !== "per_turn" && !(native.lifecyclePolicy?.mode === "warm" && input.harness === "codex" && input.accountIdentity && input.warmTaskVersion === 1)) + if (!native || (native.lifecyclePolicy?.mode !== "per_turn" && !(native.lifecyclePolicy?.mode === "warm" && input.accountIdentity && input.warmTaskVersion === 1)) || !preparedIdentity || Object.entries(preparedIdentity).some(([key, value]) => native.identity[key as keyof DurableRecoveryIdentity] !== value)) { throw new Error("Plugin Runner requires a structured per-turn native launch"); } @@ -295,7 +295,7 @@ export function createPluginRunnerTaskOwner(input: { }; return { launcher, registration, driver, readWorkspaceFile, rebind(next: typeof input) { - if (!launched || stopRequested || stoppedState || detached || next.harness !== "codex" || + if (!launched || stopRequested || stoppedState || detached || next.harness !== input.harness || input.accountIdentity !== next.accountIdentity || input.credentialIdentity !== next.credentialIdentity || JSON.stringify(input.projectIds) !== JSON.stringify(next.projectIds)) throw new Error("Plugin warm task cannot be rebound"); input = next; diff --git a/server/src/services/runner-environment-lifecycle.test.ts b/server/src/services/runner-environment-lifecycle.test.ts index d6aaeb0d9bf..41aaac73124 100644 --- a/server/src/services/runner-environment-lifecycle.test.ts +++ b/server/src/services/runner-environment-lifecycle.test.ts @@ -1,8 +1,20 @@ import {expect,it} from "vitest"; import {resolveRunnerEnvironmentForRun} from "./runner-environment-lifecycle.js"; -it("opts Codex plugin tasks into warm retention without opting ACPX in",()=>{ + +it.each([undefined,"codex","acpx","opencode"])("defaults plugin Runner execution to warm without provider gates: %s",provider=>{ const environment={driver:"plugin" as const,config:{pluginKey:"fixture",driverKey:"tasks",driverConfig:{}}}; - expect(resolveRunnerEnvironmentForRun(environment,"paperclip_runner",{provider:"codex",lifecycleMode:"warm",idleTimeoutMs:1234})).toMatchObject({config:{runnerLifecycleMode:"warm",runnerIdleTimeoutMs:1234}}); - expect(resolveRunnerEnvironmentForRun(environment,"paperclip_runner",{provider:"acpx",lifecycleMode:"warm"})).toMatchObject({config:{runnerLifecycleMode:"per_turn"}}); + expect(resolveRunnerEnvironmentForRun(environment,"paperclip_runner",{provider,idleTimeoutMs:1234},undefined,"selected-account:generation")).toMatchObject({config:{runnerLifecycleMode:"warm",runnerIdleTimeoutMs:1234}}); expect(environment.config).not.toHaveProperty("runnerLifecycleMode"); }); +it("keeps plugin-owned credentials per-turn without changing admitted recovery",()=>{ + const environment={driver:"plugin" as const,config:{pluginKey:"fixture",driverKey:"tasks",driverConfig:{},runnerLifecycleMode:"warm"}}; + expect(resolveRunnerEnvironmentForRun(environment,"paperclip_runner",{lifecycleMode:"warm"})).toMatchObject({config:{runnerLifecycleMode:"per_turn"}}); + expect(resolveRunnerEnvironmentForRun(environment,"paperclip_runner",{},"warm")).toMatchObject({config:{runnerLifecycleMode:"warm"}}); +}); +it("preserves explicit per-turn settings and the admitted recovery policy",()=>{ + const environment={driver:"plugin" as const,config:{pluginKey:"fixture",driverKey:"tasks",driverConfig:{}}}; + expect(resolveRunnerEnvironmentForRun(environment,"paperclip_runner",{lifecycleMode:"per_turn"},undefined,"account:generation")).toMatchObject({config:{runnerLifecycleMode:"per_turn"}}); + expect(resolveRunnerEnvironmentForRun(environment,"paperclip_runner",{},"per_turn")).toMatchObject({config:{runnerLifecycleMode:"per_turn"}}); + expect(resolveRunnerEnvironmentForRun({...environment,config:{...environment.config,runnerLifecycleMode:"per_turn"}},"paperclip_runner",{lifecycleMode:"warm"},undefined,"account:generation")).toMatchObject({config:{runnerLifecycleMode:"per_turn"}}); + expect(resolveRunnerEnvironmentForRun(environment,"claude_local",{})).toBe(environment); +}); diff --git a/server/src/services/runner-environment-lifecycle.ts b/server/src/services/runner-environment-lifecycle.ts index b2908196eb7..52cfb782aa0 100644 --- a/server/src/services/runner-environment-lifecycle.ts +++ b/server/src/services/runner-environment-lifecycle.ts @@ -10,14 +10,16 @@ export function resolveRunnerEnvironmentForRun = {}, admittedLifecycleMode?: "warm" | "per_turn", + managedAiCredentialIdentity?: string, ): T { if (adapterType !== "paperclip_runner" || !["sandbox", "plugin"].includes(environment.driver)) return environment; const config = environment.config ?? {}; const lifecycleMode = admittedLifecycleMode ?? (config.runnerLifecycleMode === "warm" || config.runnerLifecycleMode === "per_turn" ? config.runnerLifecycleMode : adapterConfig.lifecycleMode); + // Warm tasks need a managed credential identity; recovery keeps its admitted policy. if (environment.driver === "plugin") return { ...environment, config: { ...config, - runnerLifecycleMode: lifecycleMode === "warm" && (adapterConfig.provider ?? "codex") === "codex" ? "warm" : "per_turn", + runnerLifecycleMode: admittedLifecycleMode ?? (managedAiCredentialIdentity && lifecycleMode !== "per_turn" ? "warm" : "per_turn"), runnerIdleTimeoutMs: resolvePaperclipRunnerIdleTimeoutMs(config.runnerIdleTimeoutMs ?? adapterConfig.idleTimeoutMs), } }; if (lifecycleMode !== "warm") return environment; From a02be1efc393b4d18245df0c1b7603fcc899b8fe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=A6=A8=E5=86=89?= Date: Tue, 27 Oct 2026 11:48:14 +0800 Subject: [PATCH 045/133] fix(continuation): degrade a stale interruptedRunId hint instead of failing the wake (#14807) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A wake builds its continuation envelope from a chain of resume-source run ids: explicit user resume, retry chain, previous run, then the interruptedRunId hint > - interruptedRunId is a best-effort hint written when a run is interrupted, and it is not cleared when an issue is reassigned to another task > - When the hint names another task's run (or a deleted row), the envelope build threw continuation_source_context_missing, so the new assignee's wake could never continue the work > - This pull request degrades a stale interruptedRunId hint to "no resume source" when it is the only source tier and does not resolve to this task, while explicit user resume and retry chains stay fail-closed > - The benefit is that cross-issue reassigns no longer brick an issue's continuation wakes ## Linked Issues or Issue Description Fixes #14723 ## What Changed - `server/src/services/execution-continuation.ts`: when `interruptedRunId` is the only resume-source tier and the referenced run is missing or belongs to another issue, continue without a resume source instead of throwing `continuation_source_context_missing` - The degraded envelope reports `trigger.sourceRunId: null` and no longer re-echoes the stale `interruptedRunId` into the new envelope (so the next wake does not inherit the poisoned hint) - Explicit user resume (`explicitUserContinuation.previousRunId`), `retryOfRunId`, and `previousRunId` tiers keep the existing fail-closed throw (`continuation_user_authorization_missing` for explicit user resume) - `server/src/services/execution-continuation.test.ts`: the "rejects handoff history from a different task" test is rewritten to the intended degraded behavior, plus a deleted-row case and a fail-closed control test for explicit user resume ## Verification - `cd server && npx vitest run src/services/execution-continuation.test.ts` — 35/35 pass (33 prior tests plus 2 new; 1 existing test rewritten to the new intended behavior) - `npx tsc --noEmit` reports no errors in the touched files - `git diff --check` is clean Manual reasoning check: the degrade branch only fires when `interruptedRunId` is the sole defined tier (`explicitUserSource`, `retryOfRunId`, `previousRunId` all null) and the run does not resolve to this task. Every other tier keeps the previous throw, covered by the new control test. ## Risks - Behavioral change: a wake that previously failed with `continuation_source_context_missing` for a stale interrupted-only hint now continues without resume history. This is the intended fix per the issue; the fail-closed invariant is preserved for all user-driven and retry source tiers - The `interruptedRunId` no longer propagates into the envelope when stale, so downstream consumers no longer see the stale id (they would previously only see it when the envelope build succeeded, which it never did for this case) ## Model Used Codex (OpenAI Codex CLI, GPT-5 based coding agent, repository-aware tool use, ~1M context window, extended reasoning), assisted by three read-only explorer sub-agents for issue triage and duplicate checking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: 馨冉 --- .../execution-continuation-provenance.test.ts | 13 ++++++- .../services/execution-continuation.test.ts | 37 ++++++++++++++++--- server/src/services/execution-continuation.ts | 31 +++++++++++++--- 3 files changed, 69 insertions(+), 12 deletions(-) diff --git a/server/src/services/execution-continuation-provenance.test.ts b/server/src/services/execution-continuation-provenance.test.ts index fd294b88ffd..ce9a12edd6b 100644 --- a/server/src/services/execution-continuation-provenance.test.ts +++ b/server/src/services/execution-continuation-provenance.test.ts @@ -98,11 +98,22 @@ const support = await getEmbeddedPostgresTestSupport(); expect(JSON.stringify(envelope)).not.toContain("Unrelated"); }); - it.each(["retryOfRunId", "previousRunId", "interruptedRunId"])("rejects explicit %s history from another task", async (key) => { + it.each(["retryOfRunId", "previousRunId"])("rejects explicit %s history from another task", async (key) => { const f = await fixture(); await expect(f.build({ [key]: f.producerRunId })).rejects.toThrow("continuation_source_context_missing"); }); + it("degrades a stale interruptedRunId hint from another task and keeps producer attribution", async () => { + // interruptedRunId is a best-effort hint, not explicit user history: when + // it names another task's run the wake continues without that resume + // source, and an interaction wake falls back to its producer. + const f = await fixture(); + const envelope = await f.build({ interruptedRunId: f.producerRunId }); + expect(envelope.trigger.sourceRunId).toBe(f.producerRunId); + expect(envelope.completedWork).toBeNull(); + expect(envelope).not.toHaveProperty("interruptedRunId"); + }); + it("retains the explicit user continuation authorization requirement", async () => { const f = await fixture(); await expect(f.build({ explicitUserContinuation: { previousRunId: f.producerRunId } })) diff --git a/server/src/services/execution-continuation.test.ts b/server/src/services/execution-continuation.test.ts index 8f027562972..834bf515ee7 100644 --- a/server/src/services/execution-continuation.test.ts +++ b/server/src/services/execution-continuation.test.ts @@ -193,14 +193,41 @@ const support = await getEmbeddedPostgresTestSupport(); } }); - it("rejects handoff history from a different task", async () => { + it("degrades a stale interrupted-run hint from a different task instead of failing the wake", async () => { + // Cross-issue reassigns can leave interruptedRunId pointing at another + // task's run. The hint is best-effort: the wake must continue without a + // resume source instead of throwing continuation_source_context_missing. const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: randomUUID() } }).where(eq(heartbeatRuns.id, runId)); try { - await expectMissingContinuationContext( - () => buildExecutionContinuation({ db, companyId, issueId, agentId, - context: { interruptedRunId: runId }, summary: null, exposeLowTrustRaw: false }), - ); + const envelope = await buildExecutionContinuation({ db, companyId, issueId, agentId, + context: { interruptedRunId: runId, wakeReason: "issue_assigned" }, summary: null, exposeLowTrustRaw: false }); + expect(envelope.trigger.sourceRunId).toBeNull(); + expect(envelope).not.toHaveProperty("interruptedRunId"); + expect(envelope.objective).toBe("Focus the Gmail summary on launch decisions."); + } finally { + await db.update(heartbeatRuns).set({ contextSnapshot: source.contextSnapshot }).where(eq(heartbeatRuns.id, runId)); + } + }); + + it("degrades an interrupted-run hint whose run row no longer exists", async () => { + const missingRunId = randomUUID(); + const envelope = await buildExecutionContinuation({ db, companyId, issueId, agentId, + context: { interruptedRunId: missingRunId, wakeReason: "issue_assigned" }, summary: null, exposeLowTrustRaw: false }); + expect(envelope.trigger.sourceRunId).toBeNull(); + expect(envelope).not.toHaveProperty("interruptedRunId"); + }); + + it("still rejects an explicit user resume from a different task", async () => { + // Fail-closed stays in force for explicit user resume history; only the + // best-effort interruptedRunId hint degrades. + const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: randomUUID() } }).where(eq(heartbeatRuns.id, runId)); + try { + await expect( + buildExecutionContinuation({ db, companyId, issueId, agentId, + context: { explicitUserContinuation: { previousRunId: runId }, wakeReason: "user_resume" }, summary: null, exposeLowTrustRaw: false }), + ).rejects.toThrow("continuation_user_authorization_missing"); } finally { await db.update(heartbeatRuns).set({ contextSnapshot: source.contextSnapshot }).where(eq(heartbeatRuns.id, runId)); } diff --git a/server/src/services/execution-continuation.ts b/server/src/services/execution-continuation.ts index 53f1f479817..c44ce6ee325 100644 --- a/server/src/services/execution-continuation.ts +++ b/server/src/services/execution-continuation.ts @@ -197,25 +197,44 @@ export async function buildExecutionContinuation(input: { ); const explicitContinuation = object(input.context.explicitUserContinuation); const explicitUserSource = string(explicitContinuation.previousRunId); + const interruptedSource = string(input.context.interruptedRunId); const resumeSourceRunId = explicitUserSource ?? string(input.context.retryOfRunId) ?? string(input.context.previousRunId) ?? - string(input.context.interruptedRunId); + interruptedSource; const producerRunId = triggerInteraction?.sourceRunId ?? null; - const sourceRunId = resumeSourceRunId ?? producerRunId; + const requestedSourceRunId = resumeSourceRunId ?? producerRunId; const loadRun = async (id: string) => (await db .select({ context: heartbeatRuns.contextSnapshot, result: heartbeatRuns.resultJson }) .from(heartbeatRuns) .where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, id))) )[0]; - const candidate = sourceRunId ? await loadRun(sourceRunId) : null; + const candidate = requestedSourceRunId ? await loadRun(requestedSourceRunId) : null; // An interaction producer is provenance. Explicit resume history must still // belong to this task, and only task-scoped content can enter the envelope. const sourceRun = object(candidate?.context).issueId === issueId ? candidate : null; - if ((sourceRunId && !candidate) || (resumeSourceRunId && !sourceRun)) + // interruptedRunId is a best-effort hint written when a run is interrupted. + // After a cross-issue reassign it can name another task's run or a deleted + // row. Unlike an explicit user resume or a retry chain, a hint that does + // not resolve to this task must not fail the whole wake: degrade to "no + // resume source" and continue. + const staleInterruptedSource = + interruptedSource !== null + && resumeSourceRunId === interruptedSource + && explicitUserSource === null + && string(input.context.retryOfRunId) === null + && string(input.context.previousRunId) === null + && sourceRun === null; + // A degraded hint falls back to the interaction producer so interaction + // wakes keep their source attribution. + const sourceRunId = staleInterruptedSource ? (producerRunId ?? null) : requestedSourceRunId; + if ( + !staleInterruptedSource + && ((requestedSourceRunId && !candidate) || (resumeSourceRunId && !sourceRun)) + ) throw new Error(explicitUserSource ? "continuation_user_authorization_missing" : "continuation_source_context_missing"); - const producer = producerRunId === sourceRunId ? candidate + const producer = producerRunId === requestedSourceRunId ? candidate : producerRunId ? await loadRun(producerRunId) : null; if (producerRunId && !producer) throw new Error("continuation_source_context_missing"); const producerIssueId = string(object(producer?.context).issueId); @@ -431,7 +450,7 @@ export async function buildExecutionContinuation(input: { if (!predecessor || !authorization || explicitUserSource !== sourceRunId) throw new Error("continuation_user_authorization_missing"); } - const interruptedRunId = explicitUserSource ?? string(input.context.interruptedRunId) ?? (lastTerminal && lastTerminal.status !== "succeeded" && + const interruptedRunId = explicitUserSource ?? (staleInterruptedSource ? undefined : interruptedSource) ?? (lastTerminal && lastTerminal.status !== "succeeded" && (hasConversationContinuationPolicy(lastTerminal.result) || lastTerminal.status === "interrupted" || lastTerminal.errorCode === "process_lost") ? lastTerminal.id : undefined); From 8153061d71f75b66a531aa393f6c3a6e49493819 Mon Sep 17 00:00:00 2001 From: mdc2122 Date: Mon, 26 Oct 2026 22:49:07 -0500 Subject: [PATCH 046/133] fix(pi-local): strip inherited server PAPERCLIP_* env from pi_local processes (#14503) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The `pi_local` adapter spawns the Pi CLI as a child process for agent runs, environment tests, and model discovery > - Each of these three paths builds the child env with `{ ...process.env, ...env }`, so the child inherits the full server environment > - The server environment can hold server-only `PAPERCLIP_*` secrets and config paths, such as `PAPERCLIP_AGENT_JWT_SECRET`, `PAPERCLIP_TOOL_ACTION_SIGNING_SECRET`, and `PAPERCLIP_SECRETS_MASTER_KEY_FILE` > - `@paperclipai/adapter-utils` already has `sanitizeInheritedPaperclipEnv`, and the shared `runChildProcess` path uses it for its base env. `pi_local` passes its pre-merged env as `opts.env`, which `runChildProcess` spreads over that sanitized base, so the inherited server values come back > - This pull request routes the three `pi_local` merges through `sanitizeInheritedPaperclipEnv` > - The benefit is that `pi_local` agent processes no longer receive server secrets. This is a security hardening change. ## Linked Issues or Issue Description No issue exists. This is a bug report in the in-PR format. **What happened?** A `pi_local` agent process receives the inherited `PAPERCLIP_*` variables of the Paperclip server process. An env dump from inside a `pi_local` run showed `PAPERCLIP_AGENT_JWT_SECRET`, `PAPERCLIP_TOOL_ACTION_SIGNING_SECRET`, `PAPERCLIP_SECRETS_MASTER_KEY_FILE`, and server config path variables. The agent process can read these values. **Expected behavior** The `pi_local` child env drops inherited server `PAPERCLIP_*` variables, the same as the shared `runChildProcess` path. Only `PAPERCLIP_RUNTIME_API_URL`, `PAPERCLIP_LISTEN_HOST`, and `PAPERCLIP_LISTEN_PORT` stay inherited. The run-scoped variables that the adapter sets (for example `PAPERCLIP_AGENT_ID` and `PAPERCLIP_API_KEY`) and the adapter-configured env still reach the child. **Steps to reproduce** 1. Start the Paperclip server with `PAPERCLIP_AGENT_JWT_SECRET` set in its environment. 2. Run a `pi_local` agent whose command writes `process.env` to a file. 3. Read the file. `PAPERCLIP_AGENT_JWT_SECRET` is present. **Agent adapter(s) involved** `pi_local` only. **Additional context** Other CLI adapters appear to merge `process.env` into the child env in a similar way (for example in some `execute.ts` and `test.ts` files). This pull request does not change them. It keeps the scope to `pi_local`. Related: #3030 and #12870 also touch inherited agent env. This pull request is independent of #14479. ## What Changed - `packages/adapters/pi-local/src/server/execute.ts`: the run env and the runtime env that the adapter logs now start from `sanitizeInheritedPaperclipEnv(process.env)` instead of `process.env`. - `packages/adapters/pi-local/src/server/test.ts`: the environment test env now starts from `sanitizeInheritedPaperclipEnv(process.env)`. - `packages/adapters/pi-local/src/server/models.ts`: the model discovery env (`pi --list-models`) now starts from `sanitizeInheritedPaperclipEnv(process.env)`. - `server/src/__tests__/pi-local-execute.test.ts`: add a regression test. The test sets three server secrets on `process.env`, runs `execute` with a fake Pi command that dumps its env, and checks that the secrets are absent. It also checks that the adapter-configured env and the run-scoped `PAPERCLIP_AGENT_ID` and `PAPERCLIP_API_KEY` are present. ## Verification - `cd server && npx vitest run src/__tests__/pi-local-execute.test.ts src/__tests__/pi-local-adapter-environment.test.ts src/__tests__/pi-local-skill-sync.test.ts`: 7 tests pass. - The new test fails without the source change (the secret assertion fails) and passes with it. - `npx vitest run packages/adapters/pi-local`: 42 tests pass. - `pnpm --filter @paperclipai/adapter-pi-local typecheck`: pass. - `cd server && tsc --noEmit` (after `pnpm --filter @paperclipai/plugin-sdk ensure-build-deps`): pass. - Follow-up coverage observes real child environments from direct model discovery, agent execution, and `testEnvironment`. Each path omits inherited secrets and retains the permitted runtime/listen variables and configured env. Removing either the model-discovery or environment-probe sanitizer independently makes the regression fail. All 49 scoped tests and both direct typechecks pass. ## Risks - Low risk. A `pi_local` setup that relied on an inherited server `PAPERCLIP_*` variable (other than the three kept keys) no longer gets it. Such a value can still be set in the adapter `env` config. This is the same behavior as the shared `runChildProcess` path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI GPT-6.1 Sol, exact runtime model ID `gpt6sol-cliproxy/gpt-6.1-sol`, with tool use and code execution. Context-window and thinking-level metadata are not exposed. The prior model and human-review disclosure was not substantiated by available runtime metadata. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: mdc2122 <259970464+mdc2122@users.noreply.github.com> --- .../adapters/pi-local/src/server/execute.ts | 5 +- .../adapters/pi-local/src/server/models.ts | 4 +- packages/adapters/pi-local/src/server/test.ts | 3 +- server/src/__tests__/pi-local-execute.test.ts | 91 ++++++++++++++++++- 4 files changed, 97 insertions(+), 6 deletions(-) diff --git a/packages/adapters/pi-local/src/server/execute.ts b/packages/adapters/pi-local/src/server/execute.ts index bfe07590b83..744a79f5afe 100644 --- a/packages/adapters/pi-local/src/server/execute.ts +++ b/packages/adapters/pi-local/src/server/execute.ts @@ -53,6 +53,7 @@ import { DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE, DEFAULT_PAPERCLIP_CONVERSATION_PROMPT_TEMPLATE, runChildProcess, + sanitizeInheritedPaperclipEnv, } from "@paperclipai/adapter-utils/server-utils"; import { shellQuote } from "@paperclipai/adapter-utils/ssh"; import { isPiUnknownSessionError, parsePiJsonl, createPiJsonlParser } from "./parse.js"; @@ -356,7 +357,7 @@ export async function execute(ctx: AdapterExecutionContext): Promise injectedSkillKeys.has(entry.key) && entry.source.length > 0) .map((entry) => path.join(entry.source, "bin")); - const mergedEnv = ensurePathInEnv({ ...process.env, ...env }); + const mergedEnv = ensurePathInEnv({ ...sanitizeInheritedPaperclipEnv(process.env), ...env }); const pathKey = typeof mergedEnv.Path === "string" && mergedEnv.Path.length > 0 && !mergedEnv.PATH ? "Path" @@ -508,7 +509,7 @@ export async function execute(ctx: AdapterExecutionContext): Promise typeof entry[1] === "string", ), ), diff --git a/packages/adapters/pi-local/src/server/models.ts b/packages/adapters/pi-local/src/server/models.ts index 06b1a803474..f8aae197f21 100644 --- a/packages/adapters/pi-local/src/server/models.ts +++ b/packages/adapters/pi-local/src/server/models.ts @@ -1,6 +1,6 @@ import { createHash } from "node:crypto"; import type { AdapterModel } from "@paperclipai/adapter-utils"; -import { asString, runChildProcess } from "@paperclipai/adapter-utils/server-utils"; +import { asString, runChildProcess, sanitizeInheritedPaperclipEnv } from "@paperclipai/adapter-utils/server-utils"; const MODELS_CACHE_TTL_MS = 60_000; @@ -108,7 +108,7 @@ export async function discoverPiModels(input: { const command = resolvePiCommand(input.command); const cwd = asString(input.cwd, process.cwd()); const env = normalizeEnv(input.env); - const runtimeEnv = normalizeEnv({ ...process.env, ...env }); + const runtimeEnv = normalizeEnv({ ...sanitizeInheritedPaperclipEnv(process.env), ...env }); const result = await runChildProcess( `pi-models-${Date.now()}-${Math.random().toString(16).slice(2)}`, diff --git a/packages/adapters/pi-local/src/server/test.ts b/packages/adapters/pi-local/src/server/test.ts index 782699ce7fa..370a8c3d51d 100644 --- a/packages/adapters/pi-local/src/server/test.ts +++ b/packages/adapters/pi-local/src/server/test.ts @@ -7,6 +7,7 @@ import { asString, parseObject, ensurePathInEnv, + sanitizeInheritedPaperclipEnv, } from "@paperclipai/adapter-utils/server-utils"; import { asStringArray, @@ -125,7 +126,7 @@ export async function testEnvironment( for (const [key, value] of Object.entries(envConfig)) { if (typeof value === "string") env[key] = value; } - const runtimeEnv = normalizeEnv(ensurePathInEnv({ ...process.env, ...env })); + const runtimeEnv = normalizeEnv(ensurePathInEnv({ ...sanitizeInheritedPaperclipEnv(process.env), ...env })); const cwdInvalid = checks.some((check) => check.code === "pi_cwd_invalid"); if (cwdInvalid) { diff --git a/server/src/__tests__/pi-local-execute.test.ts b/server/src/__tests__/pi-local-execute.test.ts index 265a26800f3..a11bb83f2aa 100644 --- a/server/src/__tests__/pi-local-execute.test.ts +++ b/server/src/__tests__/pi-local-execute.test.ts @@ -2,7 +2,7 @@ import { describe, expect, it } from "vitest"; import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; -import { execute } from "@paperclipai/adapter-pi-local/server"; +import { discoverPiModels, execute, testEnvironment } from "@paperclipai/adapter-pi-local/server"; async function writeFakePiCommand(commandPath: string): Promise { const script = `#!/usr/bin/env node @@ -207,4 +207,93 @@ describe("pi_local execute", () => { await fs.rm(root, { recursive: true, force: true }); } }); + + it("isolates server secrets during execution, model discovery, and environment probes", async () => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-pi-env-")); + const workspace = path.join(root, "workspace"); + const commandPath = path.join(root, "pi"); + const envDumpPath = path.join(root, "captured-env.json"); + await fs.mkdir(workspace, { recursive: true }); + await fs.writeFile(commandPath, `#!/usr/bin/env node +require("node:fs").appendFileSync(${JSON.stringify(envDumpPath)}, JSON.stringify({ args: process.argv.slice(2), env: process.env }) + "\\n"); +if (process.argv.includes("--list-models")) { + console.log("provider model"); + console.log("google gemini-3-flash-preview"); + process.exit(0); +} +console.log(JSON.stringify({ type: "agent_start" })); +console.log(JSON.stringify({ type: "turn_end", message: { role: "assistant", content: [{ type: "text", text: "hello" }] }, toolResults: [] })); +console.log(JSON.stringify({ type: "agent_end", messages: [] })); +`, "utf8"); + await fs.chmod(commandPath, 0o755); + + const serverSecrets = { + PAPERCLIP_AGENT_JWT_SECRET: "server-jwt-secret", + PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: "server-tool-action-secret", + PAPERCLIP_SECRETS_MASTER_KEY_FILE: "/server/master.key", + }; + const permittedEnv = { + PAPERCLIP_RUNTIME_API_URL: "https://runtime.example.test", + PAPERCLIP_LISTEN_HOST: "127.0.0.1", + PAPERCLIP_LISTEN_PORT: "3100", + }; + const inheritedEnv = { ...serverSecrets, ...permittedEnv }; + const previous = { HOME: process.env.HOME, ...Object.fromEntries(Object.keys(inheritedEnv).map((k) => [k, process.env[k]])) }; + process.env.HOME = root; + Object.assign(process.env, inheritedEnv); + + try { + // Discover directly so execute/testEnvironment cannot pre-sanitize this input. + await discoverPiModels({ command: commandPath, cwd: workspace, env: { AGENT_SETTING: "kept" } }); + await execute({ + runId: "run-pi-env-leak", + agent: { id: "agent-env", companyId: "company-env", name: "Pi Agent", adapterType: "pi_local", adapterConfig: {} }, + runtime: { sessionId: null, sessionParams: null, sessionDisplayId: null, taskKey: null }, + config: { + command: commandPath, + cwd: workspace, + model: "google/gemini-3-flash-preview", + promptTemplate: "Keep working.", + env: { AGENT_SETTING: "kept" }, + }, + context: {}, + authToken: "run-jwt-token", + onLog: async () => {}, + }); + + const diagnostics = await testEnvironment({ + companyId: "company-env", + adapterType: "pi_local", + config: { + command: commandPath, + cwd: workspace, + model: "google/gemini-3-flash-preview", + env: { AGENT_SETTING: "kept" }, + }, + }); + expect(diagnostics.status).toBe("pass"); + + const captures = (await fs.readFile(envDumpPath, "utf8")).trim().split("\n").map( + (line) => JSON.parse(line) as { args: string[]; env: Record }, + ); + const discovery = captures.find(({ args }) => args.includes("--list-models")); + const execution = captures.find(({ env }) => env.PAPERCLIP_AGENT_ID === "agent-env"); + const probe = captures.find(({ args }) => args.includes("Respond with hello.")); + expect(discovery).toBeDefined(); + expect(execution).toBeDefined(); + expect(probe).toBeDefined(); + for (const capture of [discovery!, execution!, probe!]) { + for (const key of Object.keys(serverSecrets)) expect(capture.env).not.toHaveProperty(key); + expect(capture.env.AGENT_SETTING).toBe("kept"); + expect(capture.env).toMatchObject(permittedEnv); + } + expect(execution!.env.PAPERCLIP_API_KEY).toBe("run-jwt-token"); + } finally { + for (const [key, value] of Object.entries(previous)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + await fs.rm(root, { recursive: true, force: true }); + } + }); }); From b46d1ab77bbd2628c13c69e0d67f2602bb90dc87 Mon Sep 17 00:00:00 2001 From: conreo Date: Tue, 27 Oct 2026 10:52:20 +0700 Subject: [PATCH 047/133] feat(ui): request a standalone window so the app can be installed (#13461) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - The Paperclip board is used from phones, and `ui/public/site.webmanifest` is what a browser reads when someone adds the board to a home screen. > - That manifest asks for `display: "browser"`. Chrome installs a web app only when `display` is `fullscreen`, `standalone` or `minimal-ui`; with `browser` it answers `manifest-display-not-supported` and adds a shortcut rather than an app, so every launch keeps the address bar. > - The user interface already implements the no-browser-chrome case. `StandaloneBrowserControls` renders Refresh, Share and Open in Browser, and it is rendered only when `mobile && isChromelessDisplayMode()`. `ui/src/lib/pwa-display-mode.ts` exists for no other purpose than to detect `standalone`, `fullscreen` and `window-controls-overlay`. > - None of those modes can occur on a mobile launch while the manifest says `browser`, so that component and its fallback controls are unreachable in practice. > - That makes the manifest value look like the outlier rather than the intent, so the change is one manifest field plus the test that pins it. ## Linked Issues or Issue Description Refs #2139 — "enh: Officially support Install as Progressive Web App", which asks for a supported home-screen install. This pull request is the installability half of that request; it does not touch notifications. **Subsystem affected** The web app manifest and the unit test that guards it: `ui/public/site.webmanifest` and `ui/src/lib/pwa-install-mode.test.ts`. No server, plugin, or database code. **Problem or motivation** Chrome's installability criteria require `display` to be one of `fullscreen`, `standalone` or `minimal-ui`. With `display: "browser"` Chrome reports `manifest-display-not-supported`, offers only a shortcut, and the launch keeps browser controls, so the board never behaves like an installed application even though the manifest, service worker, and icons all exist. The same value also makes existing code unreachable: the in-app replacement controls for a chromeless launch (`StandaloneBrowserControls`, gated on `mobile && isChromelessDisplayMode()`) cannot be triggered from a mobile home-screen launch, so the app carries a fallback that nothing can exercise. **Proposed solution** - Set `display: "standalone"` in `ui/public/site.webmanifest`. - Update `ui/src/lib/pwa-install-mode.test.ts`, which asserted `display === "browser"`. The assertion now pins `standalone` and a comment records why, so the next reader does not have to reconstruct this reasoning. The assertions that keep the legacy `mobile-web-app-capable` and the Apple metas out of `index.html` are unchanged. `standalone` rather than `fullscreen` for two reasons. It keeps the Android status bar showing the clock and battery, and it avoids a known Android behaviour of `fullscreen` launches: opening the soft keyboard triggers a fullscreen transition and a visible flicker ([Google issue 392925620](https://issuetracker.google.com/issues/392925620), [Stack Overflow 71606355](https://stackoverflow.com/questions/71606355/pwa-does-not-resize-window-when-virtual-keyboard-is-active-on-fullscreen-display)), which `standalone` does not do. **Alternatives considered** - `fullscreen`. Also installable, and it additionally hides the Android status bar. Rejected as the default for that reason; it is a one-word change if preferred. - Adding the legacy `mobile-web-app-capable` meta. Rejected: Chrome reads the manifest, so the meta is redundant. - Adding the Apple `apple-mobile-web-app-capable` metas for iOS. Rejected for this change: it is iOS-specific, and iOS is not the platform whose installability check this fixes. - Changing `orientation`, which #5564 reports. Rejected as out of scope; this pull request stays on `display` only. **Roadmap alignment** `ROADMAP.md` does not plan this work. Its only mention of install is the skills store. **Additional context** - #7792 attempted a standalone PWA earlier and was closed during triage. It bundled a deploy-pipeline change and a heartbeat change with the manifest change, and its title carried an internal ticket id. This pull request is deliberately the manifest change alone. - #5564 reports that the manifest's `orientation: any` ignores the Android rotation lock. Related, not addressed here. - The keyboard flicker above was reported against a board installed with `fullscreen`, which is one of the two reasons this pull request asks for `standalone` instead of the more immersive mode. ## What Changed - `ui/public/site.webmanifest`: `display` is now `"standalone"` instead of `"browser"`. - `ui/src/lib/pwa-install-mode.test.ts`: the expectation that pinned `browser` now pins `standalone`, with the reasoning in a comment; the meta assertions are unchanged. No runtime code, no dependency, and no data change. The icons the installability check also requires (`android-chrome-192x192.png`, `android-chrome-512x512.png`) are already present and unchanged. ## Verification - `vitest run src/lib/pwa-install-mode.test.ts src/lib/pwa-display-mode.test.ts` in `ui`: 2 files, 7 tests passed (the same 2 + 5 the sibling plugin-event change reports). - The manifest parses as JSON, keeps `display: "standalone"`, and keeps 192 and 512 pixel icons. - The only place in the repository that asserts the manifest's `display` value is the test updated here; the other references to `site.webmanifest` are request routing and HTTP log policy and are unaffected. - CI on the previous head of this branch had two failures unrelated to this change: an e2e hover-opacity assertion on a sidebar star button (`Star Zeta`: expected `1`, received `0`) and `TypeError: Cannot read properties of undefined (reading 'logicalEffectCount')` in the runner verify job. Both are in subsystems this change cannot reach; CI is re-running on this commit. ## Risks - Low. One manifest field and one test expectation; no runtime code changes. - The value applies to new installations. An app already installed with the previous value keeps its installed configuration until it is reinstalled, so nobody loses a working install. - iOS launches are unchanged: Safari does not take `display` from the manifest for home-screen launches, and this change does not add the Apple metas. - Nobody depends on the address bar being present in an installed launch; the in-app controls that replace it (`StandaloneBrowserControls`) are already implemented and become reachable. ## Model Used - Provider: DeepSeek. Model ID: `deepseek-v4-flash`. The model ran as a tool-using coding agent (shell, file edits, headless browser automation) inside a coding harness. The harness did not report a context-window size or a reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and issues and linked them above (#2139, #7792, #5564) - [x] I have either (a) linked existing issues with `Fixes:` / `Closes` / `Refs #` OR (b) described the issue in-PR following the relevant issue template — both, see above - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run the affected tests locally and they pass (2 files, 7 tests) - [x] I have added or updated tests where applicable — the guard test is updated, and it is the only test that pins this value - [x] I have updated relevant documentation to reflect my changes — no user-facing document describes the manifest; the updated test now records the reasoning in place - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green — CI is re-running on this commit after the unrelated failures described under Verification - [ ] Greptile is 5/5 — awaiting the review run on this commit - [x] I will address all Greptile and reviewer comments before requesting merge --- ui/public/site.webmanifest | 2 +- ui/src/lib/pwa-install-mode.test.ts | 12 ++++++++++-- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/ui/public/site.webmanifest b/ui/public/site.webmanifest index ea5a8bc9f5e..907f6293e42 100644 --- a/ui/public/site.webmanifest +++ b/ui/public/site.webmanifest @@ -5,7 +5,7 @@ "description": "AI-powered project management and agent coordination platform", "start_url": "/", "scope": "/", - "display": "browser", + "display": "standalone", "orientation": "any", "theme_color": "#18181b", "background_color": "#18181b", diff --git a/ui/src/lib/pwa-install-mode.test.ts b/ui/src/lib/pwa-install-mode.test.ts index 56078526174..902d7dc356a 100644 --- a/ui/src/lib/pwa-install-mode.test.ts +++ b/ui/src/lib/pwa-install-mode.test.ts @@ -6,13 +6,21 @@ import { describe, expect, it } from "vitest"; const uiRoot = resolve(fileURLToPath(new URL("../..", import.meta.url))); describe("PWA install mode", () => { - it("opens home-screen launches with browser controls visible", () => { + it("requests a chromeless window so the in-app browser controls apply", () => { const manifest = JSON.parse(readFileSync(resolve(uiRoot, "public/site.webmanifest"), "utf8")) as { display?: string; }; const html = readFileSync(resolve(uiRoot, "index.html"), "utf8"); - expect(manifest.display).toBe("browser"); + // `browser` makes the app uninstallable: Chrome reports + // `manifest-display-not-supported` and only offers a shortcut. `standalone` + // is the installable mode, and it is the mode the app is already written for + // — StandaloneBrowserControls renders Refresh / Share / Open in Browser only + // when the launch is chromeless, which nothing could trigger before this. + expect(manifest.display).toBe("standalone"); + // The legacy `mobile-web-app-capable` meta is unnecessary (Chrome reads the + // manifest) and the Apple metas stay out: iOS is not an install target for + // this change, because Safari needs those metas rather than the manifest. expect(html).not.toContain('name="mobile-web-app-capable"'); expect(html).not.toContain('name="apple-mobile-web-app-capable"'); expect(html).not.toContain('name="apple-mobile-web-app-status-bar-style"'); From 61ecc93033d2dc37ac22ef6b36d5080c0b5978f9 Mon Sep 17 00:00:00 2001 From: Andrii Furmanets Date: Tue, 27 Oct 2026 05:52:47 +0200 Subject: [PATCH 048/133] fix: count a recovered shutdown interrupt as recovered (#15031) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The dashboard success rate counts recovered runs as successes > - A run is recovered when a later retry in its chain succeeds > - The summary puts `failed` and `timed_out` into recovered > - An `interrupted` run, including `server_shutdown_interrupted`, falls through to other > - The success rate stays down after the retry succeeds > - This pull request counts a recovered interrupt in recovered, not in other > - The benefit is that a shutdown that was retried successfully no longer looks like a failed day ## Linked Issues or Issue Description No open issue covers this. I searched open and closed issues and pull requests. **What happened?** `dashboardService.summary` puts only `failed` and `timed_out` into the recovered bucket. An `interrupted` run, including `server_shutdown_interrupted`, goes to other even when `recovered` is set. The success rate uses `succeeded + recovered`, so the day stays low after a successful retry. **Expected behavior** When the recovered flag is set, count `interrupted` in recovered. An interrupt with no successful retry stays in other. **Steps to reproduce** 1. Record an interrupted run with error code `server_shutdown_interrupted`. 2. Record a succeeded retry of that run. 3. Open the dashboard summary for that day. 4. Observe that the interrupt is in other and the success rate does not include it. **Paperclip version or commit** master at 215586d127e97c9301d86e769a39a15c13298ca2 **Deployment mode** Local dev (pnpm dev) ## What Changed - When a run is recovered, count status `interrupted` in the recovered bucket. - Leave an unrecovered interrupt in other. - Add a service test for a recovered shutdown and a stranded shutdown. ## Verification - `pnpm exec vitest run src/__tests__/dashboard-service.test.ts --config vitest.config.ts` in `server` - Result: 1 file passed, 5 tests passed ## Risks - Low risk. Unrecovered interrupts stay in other. Failed and timed-out runs keep the same buckets. ## Model Used - No model runtime is part of this change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --- packages/shared/src/types/dashboard.ts | 8 +-- .../src/__tests__/dashboard-service.test.ts | 50 +++++++++++++++++++ server/src/services/dashboard.ts | 13 +++-- 3 files changed, 63 insertions(+), 8 deletions(-) diff --git a/packages/shared/src/types/dashboard.ts b/packages/shared/src/types/dashboard.ts index 977b124ce18..cdeaebdb777 100644 --- a/packages/shared/src/types/dashboard.ts +++ b/packages/shared/src/types/dashboard.ts @@ -7,10 +7,10 @@ export interface DashboardRunActivityDay { */ failed: number; /** - * Runs that terminated in a failure state (failed/timed_out) but whose retry - * chain eventually succeeded — e.g. restart-killed runs that recovered. Kept - * out of `failed` so the headline failure count reflects true, unrecovered - * failures. + * Runs that terminated as failed, timed_out, or interrupted (shutdown) but + * whose retry chain eventually succeeded — e.g. restart-killed runs and + * shutdown interrupts that recovered. Kept out of `failed` so the headline + * failure count reflects true, unrecovered failures. */ recovered: number; other: number; diff --git a/server/src/__tests__/dashboard-service.test.ts b/server/src/__tests__/dashboard-service.test.ts index 8642844ce5d..71b6ff9cd24 100644 --- a/server/src/__tests__/dashboard-service.test.ts +++ b/server/src/__tests__/dashboard-service.test.ts @@ -274,4 +274,54 @@ describeEmbeddedPostgres("dashboard service", () => { // process_lost kills that recovered must not leak into the failed breakdown. expect(bucket?.failedByErrorCode.process_lost).toBeUndefined(); }); + + it("counts a recovered shutdown interrupt toward recovered, not other", async () => { + const companyId = randomUUID(); + const agentId = randomUUID(); + const day = utcDay(-1); + const shutdown = randomUUID(); + const shutdownRetry = randomUUID(); + const stranded = randomUUID(); + + await db.insert(companies).values({ + id: companyId, + name: "Paperclip", + issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`, + requireBoardApprovalForNewAgents: false, + }); + await db.insert(agents).values({ + id: agentId, + companyId, + name: "CodexCoder", + role: "engineer", + status: "running", + adapterType: "codex_local", + adapterConfig: {}, + runtimeConfig: {}, + permissions: {}, + }); + + const base = { + companyId, + agentId, + invocationSource: "assignment" as const, + createdAt: day, + }; + await db.insert(heartbeatRuns).values([ + { ...base, id: shutdown, status: "interrupted", errorCode: "server_shutdown_interrupted" }, + { ...base, id: shutdownRetry, status: "succeeded", retryOfRunId: shutdown }, + { ...base, id: stranded, status: "interrupted", errorCode: "server_shutdown_interrupted" }, + ]); + + const summary = await dashboardService(db).summary(companyId); + const bucket = summary.runActivity.find((entry) => entry.date === utcDateKey(day)); + expect(bucket).toMatchObject({ + succeeded: 1, + recovered: 1, + failed: 0, + other: 1, + total: 3, + failedByErrorCode: {}, + }); + }); }); diff --git a/server/src/services/dashboard.ts b/server/src/services/dashboard.ts index 9da5d08aa4f..afd7ee25c58 100644 --- a/server/src/services/dashboard.ts +++ b/server/src/services/dashboard.ts @@ -103,9 +103,10 @@ export function dashboardService(db: Db) { const monthSpendCents = Number(monthSpend); // Per-day run breakdown. A run is "recovered" when its retry chain later // succeeded (recovered_runs = all ancestors of a succeeded retry), so a - // restart-killed run whose retry succeeded is pulled out of the headline - // failed count. error_code is carried through so a failure spike can be - // attributed to an error class (e.g. process_lost, provider_quota). + // restart-killed or shutdown-interrupted run whose retry succeeded is + // pulled out of the headline failure count. error_code is carried through + // so a failure spike can be attributed to an error class (e.g. process_lost, + // provider_quota). // Both recursive arms are bounded to the chart window: a retry is always // created after the run it retries, so ancestors of an out-of-window // child are themselves out of window and invisible to the membership @@ -166,7 +167,11 @@ export function dashboardService(db: Db) { const recovered = row.recovered === true || row.recovered === "t" || row.recovered === "true"; if (status === "succeeded") { bucket.succeeded += count; - } else if (status === "failed" || status === "timed_out") { + } else if ( + status === "failed" || + status === "timed_out" || + (recovered && status === "interrupted") + ) { if (recovered) { bucket.recovered += count; } else { From 05218571c314b172fd221f78fde89296eb46bf90 Mon Sep 17 00:00:00 2001 From: fengqing Date: Tue, 27 Oct 2026 11:53:47 +0800 Subject: [PATCH 049/133] fix(watchdogs): reject incomplete subtree scans (#15063) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task watchdogs verify stopped issue subtrees and restore valid work paths. > - A watchdog must read the full subtree before it can prove that all paths stopped. > - The current query stops after 100 levels and does not report omitted children. > - A running child beyond this limit can be absent from both review dispatch and mutation checks. > - This pull request reports an incomplete scan and blocks both decisions. > - Complete subtrees keep their current behavior and stop fingerprints. ## Linked Issues or Issue Description Fixes: #15061 Related PRs: #11327 changes subtree loading but keeps the same depth limit without a completeness result. #10331 changes mutation freshness after watchdog writes. Neither patch covers this boundary. This patch does not take code from those proposals. ## What Changed - Keep the existing 100-level traversal limit. - Check for an eligible child beyond the boundary in the same SQL statement as the subtree read. - Apply the same company, visibility, harness, and watchdog-origin filters to this check. - Return `incomplete` without a stop fingerprint when the scan omits eligible children. - Skip review dispatch and reject watchdog mutation revalidation for an incomplete scan. - Report incomplete scans through their own count and watched issue IDs. Log each newly incomplete issue at startup or during periodic reconciliation. Suppress unchanged results. Allow a new warning after the issue recovers and becomes incomplete again. - Add real PostgreSQL regressions for dispatch, a previously valid mutation scope, the complete boundary leaf, and excluded children. - Document the incomplete-scan behavior. ## Verification - Base: `569c7203aa24b95440682983ce7940ba1d4247bd`. Head: `06f68199656f979ef5eb4897851f0fbcbca592d3`. - Before the fix, two real PostgreSQL regressions fail. The service dispatches a review for a partial subtree. It also accepts a prior scope after a running depth-100 child appears. The other 19 scheduler checks pass. - On the exact final head, `pnpm -r typecheck` and `pnpm build` pass on isolated Ubuntu with Node 24.19.0, pnpm 9.15.4, and runner Rust 1.97.1. - All 75 related tests pass. Run `pnpm --filter @paperclipai/server exec vitest run src/__tests__/task-watchdogs-classifier.test.ts src/__tests__/task-watchdogs-scheduler.test.ts src/__tests__/server-startup-feedback-export.test.ts src/__tests__/issue-watchdogs-routes.test.ts --maxWorkers=1`. - Full `test:run` coverage passes across 29 logical groups: all official stable-runner lanes plus supplemental service and script tests. The selection covers all 900 server source test files without duplicate or omitted selections. It also includes UI, CLI, and other workspace projects. - A separate validation tree applies both this PR and #15067 to upstream `994d6edcdd4e15d5f9cc5cf8c135ac599104b86a` without conflicts. Type checks, the build, and all 98 related tests across five files pass. Both public PR heads remain unchanged. - Four full server lanes required test-process environment corrections. The host DNS search suffix resolved a `.invalid` fixture to an address. Ancestor project authentication settings affected hire fixtures. The old path-isolation tool also failed test imports with `EFAULT`. The exact base has the same failures: 8 failed and 128 passed assertions without isolation, then three failed imports with the old tool. All 136 checks pass on both the base and this head with the corrected isolation. All four original full lanes pass with the corrected tool, a resolver without a search suffix, empty ancestor project settings, and the standard Node path. No assertion or wait limit changed. The host configuration remains unchanged. - Earlier interrupted Ubuntu and Docker runs are retained and are not counted as passes. This exact-head Ubuntu run passes the original chat, port-conflict, and serialized lanes. The causes of the earlier host-loss and timing failures are not fully established. - The startup import regression in an earlier revision is fixed by a diagnostics module with no database dependencies. All 21 startup checks pass. - Exact-head CI, including the added scans after marking the PR ready: 54 successful checks and 2 skipped checks. Greptile: 5/5, no outstanding findings or follow-ups. Both review threads are resolved. - The final audit rechecked issue authors, assignees, comments, 206 open watchdog search results, root functions and fields, source history, and related diffs. The latest upstream is `994d6edcdd4e15d5f9cc5cf8c135ac599104b86a`. Its new patches do not cover this root. No competing root repair was found within the audited scope. The issue has no assignee or new comment. - No model provider, production database, or user instance was used. This verifies service behavior, not a full HTTP authorization bypass. ## Risks - A subtree with eligible children beyond 100 levels now waits instead of running a review. This is a deliberate safety boundary. It remains blocked until the scan can include the full subtree. - The boundary check adds a child-existence lookup to the existing recursive query. - No schema migration is required. Complete subtrees retain their fingerprint format. - This does not add unlimited traversal or close the separate race between a subtree snapshot and later source mutations. - The configuration lifecycle defect in #15060 is handled by separate PR #15067. This branch does not include its schema or configuration changes. ## Model Used - OpenAI Codex, based on GPT-6, assisted with source analysis, implementation, and test execution. The exact serving model ID and context window are not exposed in this session. The agent used reasoning, shell tools, GitHub CLI, and real PostgreSQL tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --- doc/TASK-WATCHDOG.md | 2 +- .../task-watchdogs-classifier.test.ts | 43 ++++++++++- .../task-watchdogs-scheduler.test.ts | 75 +++++++++++++++++++ server/src/index.ts | 6 ++ server/src/routes/issues.ts | 4 + .../src/services/task-watchdog-diagnostics.ts | 16 ++++ server/src/services/task-watchdogs.ts | 52 +++++++++++-- 7 files changed, 191 insertions(+), 7 deletions(-) create mode 100644 server/src/services/task-watchdog-diagnostics.ts diff --git a/doc/TASK-WATCHDOG.md b/doc/TASK-WATCHDOG.md index c638a6bc0cf..99f2bbaad56 100644 --- a/doc/TASK-WATCHDOG.md +++ b/doc/TASK-WATCHDOG.md @@ -84,7 +84,7 @@ Paperclip runs a watchdog reconciliation tick at server startup, at the end of e For each active watchdog the tick: -1. **Walks the watched subtree.** Starts at the configured issue and follows `parent_id` downward, excluding every issue whose `originKind = 'task_watchdog'` and everything below it. This excludes the watchdog's own review tasks so it cannot trigger itself. +1. **Walks the watched subtree.** Starts at the configured issue and follows `parent_id` downward, excluding every issue whose `originKind = 'task_watchdog'` and everything below it. This excludes the watchdog's own review tasks so it cannot trigger itself. The scan includes at most 100 levels, with the watched issue at level 1. If an eligible child remains beyond that limit, the result is **incomplete**. Paperclip skips review dispatch and rejects watchdog mutation revalidation until it can read the full subtree. A leaf at level 100 does not by itself make the scan incomplete. Reconciliation reports these cases separately through `incomplete` and `incompleteIssueIds`. Startup and periodic reconciliation warn when a watched issue first becomes incomplete. The warning contains the newly affected issue IDs. An unchanged incomplete issue does not produce repeated warnings. Once a successful scan no longer reports that issue as incomplete, a later incomplete scan can warn again. 2. **Checks for live paths.** If any included issue has a live run (`queued`, `running`, `scheduled_retry`), a queued wake request, or a scheduled retry, the subtree is **live** and the watchdog does not fire. 3. **Computes a stop fingerprint.** A SHA-256 hash over the stopped leaves' identifiers, statuses, blockers, pending interactions, and the current watchdog configuration. The configuration is part of the fingerprint, so changing the agent or instructions invalidates the previously reviewed state. 4. **Compares against `lastReviewedFingerprint`.** Match → suppress (the watchdog already saw this exact stopped state). New → proceed. diff --git a/server/src/__tests__/task-watchdogs-classifier.test.ts b/server/src/__tests__/task-watchdogs-classifier.test.ts index b44d0c76c52..16ef800c088 100644 --- a/server/src/__tests__/task-watchdogs-classifier.test.ts +++ b/server/src/__tests__/task-watchdogs-classifier.test.ts @@ -1,6 +1,8 @@ -import { describe, expect, it } from "vitest"; +import { describe, expect, it, vi } from "vitest"; import { classifyTaskWatchdogSubtree, type TaskWatchdogClassifierIssue } from "../services/task-watchdogs.ts"; +import { createIncompleteTaskWatchdogScanReporter } from "../services/task-watchdog-diagnostics.ts"; + const companyId = "company-1"; const sourceId = "source-1"; const childId = "child-1"; @@ -35,6 +37,45 @@ function classify(overrides: Partial { + it("reports an unchanged incomplete scan once across startup and a day of periodic ticks", () => { + const warn = vi.fn(); + const report = createIncompleteTaskWatchdogScanReporter(warn); + report({ incomplete: 2, incompleteIssueIds: [sourceId, childId] }, "startup"); + for (let tick = 0; tick < 2880; tick += 1) { + report({ incomplete: 2, incompleteIssueIds: [childId, sourceId] }, "periodic"); + } + expect(warn).toHaveBeenCalledTimes(1); + expect(warn).toHaveBeenCalledWith( + { incomplete: 2, newIncompleteIssueIds: [sourceId, childId] }, + "startup task-watchdog reconciliation found incomplete subtree scans", + ); + }); + + it("reports new incomplete issues and reports a resolved issue if it becomes incomplete again", () => { + const warn = vi.fn(); + const report = createIncompleteTaskWatchdogScanReporter(warn); + report({ incomplete: 1, incompleteIssueIds: [sourceId] }, "startup"); + report({ incomplete: 2, incompleteIssueIds: [sourceId, childId] }, "periodic"); + expect(warn).toHaveBeenLastCalledWith( + { incomplete: 2, newIncompleteIssueIds: [childId] }, + "periodic task-watchdog reconciliation found incomplete subtree scans", + ); + report({ incomplete: 0, incompleteIssueIds: [] }, "periodic"); + expect(warn).toHaveBeenCalledTimes(2); + report({ incomplete: 1, incompleteIssueIds: [sourceId] }, "periodic"); + expect(warn).toHaveBeenCalledTimes(3); + expect(warn).toHaveBeenLastCalledWith( + { incomplete: 1, newIncompleteIssueIds: [sourceId] }, + "periodic task-watchdog reconciliation found incomplete subtree scans", + ); + }); + + it("does not produce a stopped fingerprint from an incomplete scan", () => { + const result = classify({ subtreeIncomplete: true }); + expect(result.state).toBe("incomplete"); + expect(result).not.toHaveProperty("stopFingerprint"); + }); + it("suppresses watchdog wakeups while watched subtree work has a live path", () => { const result = classify({ issues: [ diff --git a/server/src/__tests__/task-watchdogs-scheduler.test.ts b/server/src/__tests__/task-watchdogs-scheduler.test.ts index 1124c163eb2..e5fbf969ab4 100644 --- a/server/src/__tests__/task-watchdogs-scheduler.test.ts +++ b/server/src/__tests__/task-watchdogs-scheduler.test.ts @@ -437,6 +437,81 @@ describeEmbeddedPostgres("task watchdog scheduler", () => { expect(wakes).toHaveLength(1); }); + async function seedDeepSubtree() { + const companyId = await seedCompany(); + const sourceId = await seedIssue(companyId, { status: "blocked", issueNumber: 1000 }); + let boundaryId = sourceId; + for (let depth = 1; depth < 100; depth += 1) { + boundaryId = await seedIssue(companyId, { + parentId: boundaryId, status: "blocked", identifier: `DEEP-${depth}`, issueNumber: 1000 + depth, + }); + } + const agentId = await seedAgent(companyId); + const watchdog = await seedWatchdog(companyId, sourceId, agentId); + return { companyId, sourceId, boundaryId, agentId, watchdog, ...createService() }; + } + + it("rejects an incomplete subtree before dispatching a stopped review", async () => { + const x = await seedDeepSubtree(); + const omittedId = await seedIssue(x.companyId, { + parentId: x.boundaryId, status: "blocked", issueNumber: 1100, + }); + await db.insert(heartbeatRuns).values({ + companyId: x.companyId, agentId: x.agentId, status: "running", + contextSnapshot: { issueId: omittedId }, + }); + expect(await x.service.reconcileTaskWatchdogs({ companyId: x.companyId })) + .toMatchObject({ checked: 1, triggered: 0, incomplete: 1, incompleteIssueIds: [x.sourceId], skipped: 0 }); + expect(x.wakes).toHaveLength(0); + const [persisted] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.id, x.watchdog.id)); + expect(persisted!.watchdogIssueId).toBeNull(); + expect(persisted!.lastObservedFingerprint).toBeNull(); + }); + + it("allows a complete boundary leaf but revokes its scope when an omitted child appears", async () => { + const x = await seedDeepSubtree(); + expect((await x.service.reconcileTaskWatchdogs({ companyId: x.companyId })).triggered).toBe(1); + const [persisted] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.id, x.watchdog.id)); + const scope = { + kind: "watchdog" as const, watchdogId: x.watchdog.id, companyId: x.companyId, + watchedIssueId: x.sourceId, stopFingerprint: persisted!.lastObservedFingerprint, + }; + expect((await x.service.revalidateMutationScope(scope)).allowed).toBe(true); + const omittedId = await seedIssue(x.companyId, { + parentId: x.boundaryId, status: "blocked", issueNumber: 1100, + }); + await db.insert(heartbeatRuns).values({ + companyId: x.companyId, agentId: x.agentId, status: "running", + contextSnapshot: { issueId: omittedId }, + }); + const incomplete = await x.service.revalidateMutationScope(scope); + expect(incomplete.allowed).toBe(false); + expect(incomplete.classification?.state).toBe("incomplete"); + expect(incomplete.reason).toContain("incomplete"); + expect(await x.service.reconcileForIssueAndAncestors(x.companyId, x.sourceId)) + .toMatchObject({ triggered: 0, incomplete: 1, incompleteIssueIds: [x.sourceId], skipped: 0 }); + expect(x.wakes).toHaveLength(1); + await db.update(issues).set({ parentId: x.sourceId }).where(eq(issues.id, omittedId)); + const visible = await x.service.revalidateMutationScope(scope); + expect(visible.allowed).toBe(false); + expect(visible.classification?.state).toBe("live"); + }); + + it("does not treat excluded boundary children as a truncated watched subtree", async () => { + const x = await seedDeepSubtree(); + const hiddenId = await seedIssue(x.companyId, { parentId: x.boundaryId, issueNumber: 1100 }); + await db.update(issues).set({ hiddenAt: new Date() }).where(eq(issues.id, hiddenId)); + const harnessId = await seedIssue(x.companyId, { parentId: x.boundaryId, issueNumber: 1103 }); + await db.update(issues).set({ harnessKind: "test" }).where(eq(issues.id, harnessId)); + await seedIssue(x.companyId, { + parentId: x.boundaryId, issueNumber: 1101, originKind: "task_watchdog", originId: x.boundaryId, + }); + const otherCompanyId = await seedCompany(); + await seedIssue(otherCompanyId, { parentId: x.boundaryId, issueNumber: 1102 }); + expect((await x.service.reconcileTaskWatchdogs({ companyId: x.companyId })).triggered).toBe(1); + expect(x.wakes).toHaveLength(1); + }); + it("reconciles ancestor watchdogs for a descendant issue mutation", async () => { const companyId = await seedCompany(); const sourceId = await seedIssue(companyId, { identifier: "WDOG-ANCESTOR", status: "done" }); diff --git a/server/src/index.ts b/server/src/index.ts index b4e814959e1..23b13375283 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -85,6 +85,7 @@ import detectPort from "detect-port"; import { createApp } from "./app.js"; import { loadConfig } from "./config.js"; import { logger } from "./middleware/logger.js"; +import { createIncompleteTaskWatchdogScanReporter } from "./services/task-watchdog-diagnostics.js"; import { setStartupRecoveryPhase } from "./startup-recovery-state.js"; import { StartupRefusalError, @@ -927,6 +928,9 @@ async function startServerWithDatabaseTeardown( } }; const pluginWorkerManager = createPluginWorkerManager(); + const reportIncompleteWatchdogScans = createIncompleteTaskWatchdogScanReporter( + (details, message) => logger.warn(details, message), + ); const heartbeat = config.heartbeatSchedulerEnabled ? heartbeatService(db as any, { pluginWorkerManager }) : null; @@ -1733,6 +1737,7 @@ async function startServerWithDatabaseTeardown( } const taskWatchdogsReconciled = await heartbeat.reconcileTaskWatchdogs(); + reportIncompleteWatchdogScans(taskWatchdogsReconciled, "startup"); if (taskWatchdogsReconciled.triggered > 0) { logger.warn( { ...taskWatchdogsReconciled }, @@ -1940,6 +1945,7 @@ async function startServerWithDatabaseTeardown( }) .then(async () => { const reconciled = await heartbeat.reconcileTaskWatchdogs(); + reportIncompleteWatchdogScans(reconciled, "periodic"); if (reconciled.triggered > 0) { logger.warn({ ...reconciled }, "periodic task-watchdog reconciliation triggered watchdog work"); } diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index 2035e1099da..dccefa9a355 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -528,6 +528,8 @@ function noopTaskWatchdogService(): TaskWatchdogService { live: 0, pendingFirstRun: 0, alreadyReviewed: 0, + incomplete: 0, + incompleteIssueIds: [], skipped: 0, watchdogIssueIds: [], }), @@ -535,6 +537,8 @@ function noopTaskWatchdogService(): TaskWatchdogService { checked: 0, triggered: 0, pendingFirstRun: 0, + incomplete: 0, + incompleteIssueIds: [], skipped: 0, watchdogIssueIds: [], }), diff --git a/server/src/services/task-watchdog-diagnostics.ts b/server/src/services/task-watchdog-diagnostics.ts new file mode 100644 index 00000000000..ea08eebe2b4 --- /dev/null +++ b/server/src/services/task-watchdog-diagnostics.ts @@ -0,0 +1,16 @@ +export function createIncompleteTaskWatchdogScanReporter( + warn: (details: { incomplete: number; newIncompleteIssueIds: string[] }, message: string) => void, +) { + let previousIncompleteIssueIds = new Set(); + return (result: { incomplete: number; incompleteIssueIds: string[] }, phase: "startup" | "periodic") => { + const currentIssueIds = new Set(result.incompleteIssueIds); + const newIncompleteIssueIds = [...currentIssueIds].filter((id) => !previousIncompleteIssueIds.has(id)); + previousIncompleteIssueIds = currentIssueIds; + if (newIncompleteIssueIds.length === 0) return; + warn( + { incomplete: result.incomplete, newIncompleteIssueIds }, + `${phase} task-watchdog reconciliation found incomplete subtree scans`, + ); + }; +} + diff --git a/server/src/services/task-watchdogs.ts b/server/src/services/task-watchdogs.ts index e9627cf74bb..cf8d92d39df 100644 --- a/server/src/services/task-watchdogs.ts +++ b/server/src/services/task-watchdogs.ts @@ -149,6 +149,11 @@ type TaskWatchdogPendingInteractionsByIssueId = Record>; export type TaskWatchdogClassifierResult = + | { + state: "incomplete"; + reason: string; + includedIssueIds: string[]; + } | { state: "not_applicable"; reason: string; @@ -188,6 +193,7 @@ export type TaskWatchdogClassifierResult = export type TaskWatchdogClassifierInput = { watchdog: TaskWatchdogClassifierConfig; issues: TaskWatchdogClassifierIssue[]; + subtreeIncomplete?: boolean; activeRuns?: TaskWatchdogClassifierPath[]; queuedWakeRequests?: TaskWatchdogClassifierPath[]; blockers?: TaskWatchdogClassifierRelation[]; @@ -382,6 +388,15 @@ export function classifyTaskWatchdogSubtree(input: TaskWatchdogClassifierInput): }; } + // A partial scan cannot prove that every execution path has stopped. + if (input.subtreeIncomplete) { + return { + state: "incomplete", + reason: "Watched subtree scan is incomplete because it exceeds the traversal depth limit.", + includedIssueIds: [...issuesById.keys()], + }; + } + const childrenByParentId = new Map(); for (const issue of input.issues) { if (issue.companyId !== input.watchdog.companyId || !issue.parentId) continue; @@ -927,20 +942,35 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) assignee_user_id AS "assigneeUserId", origin_kind AS "originKind", updated_at AS "updatedAt", - created_at AS "createdAt" + created_at AS "createdAt", + EXISTS ( + SELECT 1 + FROM watched_issues boundary + JOIN issues child ON child.parent_id = boundary.id + WHERE boundary.depth = ${TASK_WATCHDOG_SUBTREE_MAX_DEPTH - 1} + AND child.company_id = ${companyId} + AND child.hidden_at IS NULL + AND child.harness_kind IS NULL + AND child.origin_kind <> ${TASK_WATCHDOG_ORIGIN_KIND} + ) AS "subtreeIncomplete" FROM watched_issues `); - return (Array.isArray(rows) ? rows : []) as TaskWatchdogClassifierIssue[]; + const issueRows = (Array.isArray(rows) ? rows : []) as Array< + TaskWatchdogClassifierIssue & { subtreeIncomplete: boolean } + >; + return { issues: issueRows, incomplete: issueRows.some((row) => row.subtreeIncomplete) }; } async function collectClassifierInput(companyId: string, watchdog: IssueWatchdogRow) { - const issueRows = await loadWatchdogSubtreeIssues(companyId, watchdog.issueId); + const subtree = await loadWatchdogSubtreeIssues(companyId, watchdog.issueId); + const issueRows = subtree.issues; const subtreeIssueIds = issueRows.map((issue) => issue.id); - if (subtreeIssueIds.length === 0) { + if (subtreeIssueIds.length === 0 || subtree.incomplete) { return { watchdog: summarizeIssueWatchdog(watchdog), - issues: [], + issues: issueRows, + subtreeIncomplete: subtree.incomplete, activeRuns: [], queuedWakeRequests: [], blockers: [], @@ -1665,6 +1695,8 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) allowed: false as const, reason: classification.state === "stopped" ? "Task-watchdog review is stale because the watched subtree stop fingerprint changed; refresh the source state before mutating it." + : classification.state === "incomplete" + ? "Task-watchdog review cannot authorize mutations because the watched subtree scan is incomplete." : "Task-watchdog review is stale because the watched subtree now has a live, waiting, already-reviewed, or not-applicable path; refresh the source state before mutating it.", classification, }; @@ -1763,6 +1795,8 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) live: 0, pendingFirstRun: 0, alreadyReviewed: 0, + incomplete: 0, + incompleteIssueIds: [] as string[], skipped: 0, watchdogIssueIds: [] as string[], }; @@ -1782,6 +1816,9 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) result.pendingFirstRun += 1; } else if (evaluated.state === "already_reviewed") { result.alreadyReviewed += 1; + } else if (evaluated.state === "incomplete") { + result.incomplete += 1; + result.incompleteIssueIds.push(row.issueId); } else { result.skipped += 1; } @@ -1799,6 +1836,8 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) checked: 0, triggered: 0, pendingFirstRun: 0, + incomplete: 0, + incompleteIssueIds: [] as string[], skipped: 0, watchdogIssueIds: [] as string[], }; @@ -1810,6 +1849,9 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) result.watchdogIssueIds.push(evaluated.watchdogIssueId); } else if (evaluated.state === "pending_first_run") { result.pendingFirstRun += 1; + } else if (evaluated.state === "incomplete") { + result.incomplete += 1; + result.incompleteIssueIds.push(row.issueId); } else if ( evaluated.state === "watchdog_review_open" || evaluated.state === "watchdog_live" || From 585d37d4f25e26689c60cbb191fdfc5cfd1c60b4 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 26 Oct 2026 22:54:18 -0500 Subject: [PATCH 050/133] [codex] Preserve AWS secret provider error classification (#10946) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source control plane operators use to manage AI-agent companies. > - Operators can store managed secret values in AWS Secrets Manager provider vaults. > - The AWS JSON gateway already classifies safe provider failures such as access denial, expired credentials, throttling, and invalid requests. > - The outer provider wrapper was re-normalizing those typed failures, which collapsed an AWS access denial into a generic provider error before the service could produce actionable guidance. > - Missing runtime credentials also need to be recognized as an access problem so operators receive the correct recovery path. > - This pull request preserves typed provider errors, expands credential-related classification, and clarifies the safe credential/IAM remediation text. > - The benefit is that failed AWS secret creation tells operators whether to restore runtime credentials or grant the required capability without exposing AWS account or principal details. ## Linked Issues or Issue Description No public issue was found for this exact remaining server-side classification gap. Bug report: - What happened: an AWS Secrets Manager access-denied response could be classified correctly by the JSON gateway and then collapsed to `provider_error` by the outer provider wrapper; missing runtime credentials were also reported generically. - Expected behavior: safe typed AWS failures should retain their classification, and credential resolution failures should surface as access-denied guidance. - Steps to reproduce: configure an AWS provider vault with invalid or missing runtime credentials, or without `secretsmanager:CreateSecret`, then create a Paperclip-managed secret. - Paperclip version/commit: current `master` before this PR. - Deployment mode: Paperclip server with an AWS Secrets Manager provider vault. Related prior work: - Refs #9161 - Refs #9645 ## What Changed - Preserve `SecretProviderClientError` instances returned by the AWS JSON gateway instead of wrapping them as generic provider failures. - Classify missing, expired, invalid, or incorrectly signed AWS runtime credentials as access-denied failures. - Clarify sanitized remediation guidance to cover both credential resolution and `secretsmanager:CreateSecret` authorization. - Add regression coverage for gateway access-denied responses, missing runtime credentials, and the service-level actionable message. ## Verification - `pnpm exec vitest run server/src/__tests__/aws-secrets-manager-provider.test.ts server/src/__tests__/secrets-service.test.ts` — changed provider tests passed; the service file was skipped in the first no-DB focused run. - `pnpm --filter @paperclipai/server typecheck` — passed. - `pnpm -r typecheck` — passed after a frozen-lockfile dependency refresh. - `pnpm build` — passed. - `pnpm test:run` — 314 files / 3,443 tests passed; one unrelated existing test failed in `plugin-orchestration-apis.test.ts` because it observed `heartbeat.scheduling_suppressed` instead of `issue_commented`. The same failure reproduced in isolation and none of this PR's four changed files participate in that test. - `git diff --check public-gh/master...HEAD` — passed. ## Risks Low risk. The change affects only error classification and sanitized failure guidance for AWS Secrets Manager operations. Successful secret writes, schemas, migrations, API shapes, and stored secret material are unchanged. The regression tests assert safe classifications while keeping raw AWS account and principal details out of service responses. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-5 family (exact runtime build and context-window size were not exposed), with reasoning, repository editing, shell execution, Git, GitHub, and test execution enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (not applicable: no documented contract changed) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com> --- .../aws-secrets-manager-provider.test.ts | 96 +++++++++++++++++++ server/src/__tests__/secrets-service.test.ts | 3 + .../secrets/aws-secrets-manager-provider.ts | 9 +- server/src/services/secrets.ts | 2 +- 4 files changed, 108 insertions(+), 2 deletions(-) diff --git a/server/src/__tests__/aws-secrets-manager-provider.test.ts b/server/src/__tests__/aws-secrets-manager-provider.test.ts index c2fc4cb6e17..df6aae4285a 100644 --- a/server/src/__tests__/aws-secrets-manager-provider.test.ts +++ b/server/src/__tests__/aws-secrets-manager-provider.test.ts @@ -231,6 +231,102 @@ describe("awsSecretsManagerProvider", () => { expect(init?.signal).toBeInstanceOf(AbortSignal); }); + it("preserves classified AWS create errors returned by the JSON gateway", async () => { + delete process.env.AWS_PROFILE; + delete process.env.AWS_DEFAULT_PROFILE; + delete process.env.AWS_CONFIG_FILE; + delete process.env.AWS_SHARED_CREDENTIALS_FILE; + delete process.env.AWS_SDK_LOAD_CONFIG; + process.env.AWS_ACCESS_KEY_ID = "AKIA_CREATE_ERROR_TEST"; + process.env.AWS_SECRET_ACCESS_KEY = "test-secret-key"; + + vi.spyOn(globalThis, "fetch").mockResolvedValue( + new Response( + JSON.stringify({ + __type: "AccessDeniedException", + message: + "User: arn:aws:sts::123456789012:assumed-role/prod/Paperclip is not authorized to perform secretsmanager:CreateSecret", + }), + { status: 400 }, + ), + ); + const provider = createAwsSecretsManagerProvider({ + config: { + region: "us-west-1", + endpoint: "https://secretsmanager.us-west-1.amazonaws.com", + deploymentId: "prod", + prefix: "paperclip", + kmsKeyId: null, + environmentTag: "production", + providerOwnerTag: "paperclip", + deleteRecoveryWindowDays: 30, + }, + }); + + await expect( + provider.createSecret({ + value: "super-secret-value", + context: { + companyId: "company-1", + secretKey: "openai-api-key", + secretName: "OpenAI API Key", + version: 1, + }, + }), + ).rejects.toMatchObject({ + code: "access_denied", + status: 403, + operation: "CreateSecret", + message: "AWS Secrets Manager denied the request. Check IAM permissions for this provider vault.", + rawMessage: expect.stringContaining("AccessDeniedException"), + }); + }); + + it("classifies missing runtime AWS credentials as access denied", async () => { + const provider = createAwsSecretsManagerProvider({ + config: { + region: "us-east-1", + endpoint: "https://secretsmanager.us-east-1.amazonaws.com", + deploymentId: "prod-use1", + prefix: "paperclip", + kmsKeyId: null, + environmentTag: "production", + providerOwnerTag: "paperclip", + deleteRecoveryWindowDays: 30, + }, + gateway: { + async createSecret() { + throw new Error("CredentialsProviderError: Could not load credentials from any providers"); + }, + async putSecretValue() { + throw new Error("not used"); + }, + async getSecretValue() { + throw new Error("not used"); + }, + async deleteSecret() { + throw new Error("not used"); + }, + }, + }); + + await expect( + provider.createSecret({ + value: "super-secret-value", + context: { + companyId: "company-1", + secretKey: "openai-api-key", + secretName: "OpenAI API Key", + version: 1, + }, + }), + ).rejects.toMatchObject({ + code: "access_denied", + status: 403, + rawMessage: "CredentialsProviderError: Could not load credentials from any providers", + }); + }); + it("creates new AWS secret versions against a namespace-valid existing secret reference", async () => { const calls: Array<{ op: string; input: Record }> = []; const provider = createAwsSecretsManagerProvider({ diff --git a/server/src/__tests__/secrets-service.test.ts b/server/src/__tests__/secrets-service.test.ts index e570ec2a6ee..8ad763f14e3 100644 --- a/server/src/__tests__/secrets-service.test.ts +++ b/server/src/__tests__/secrets-service.test.ts @@ -3409,6 +3409,9 @@ describeEmbeddedPostgres("secretService", () => { requiredCapability: "secretsmanager:CreateSecret", }, }); + expect((thrown as HttpError).details?.actionableMessage).toContain( + "Ensure the Paperclip server runtime can resolve valid AWS credentials", + ); expect(JSON.stringify(thrown)).not.toContain("arn:aws"); expect(JSON.stringify(thrown)).not.toContain("123456789012"); expect(thrown instanceof Error ? thrown.message : String(thrown)).not.toContain("arn:aws"); diff --git a/server/src/secrets/aws-secrets-manager-provider.ts b/server/src/secrets/aws-secrets-manager-provider.ts index 87090c86c53..4717217cf38 100644 --- a/server/src/secrets/aws-secrets-manager-provider.ts +++ b/server/src/secrets/aws-secrets-manager-provider.ts @@ -840,7 +840,11 @@ function asAwsSecretsManagerMaterial(value: StoredSecretVersionMaterial): AwsSec function classifyAwsProviderError(message: string): SecretProviderClientErrorCode { if (/ResourceExistsException|AlreadyExists/i.test(message)) return "conflict"; if (/ResourceNotFoundException|NotFound/i.test(message)) return "not_found"; - if (/AccessDeniedException|AccessDenied|UnrecognizedClientException|InvalidClientTokenId|not authorized/i.test(message)) { + if ( + /AccessDeniedException|AccessDenied|CredentialsProviderError|Could not load credentials|credential provider chain|UnrecognizedClientException|InvalidClientTokenId|ExpiredTokenException|InvalidSignatureException|SignatureDoesNotMatch|not authorized/i.test( + message, + ) + ) { return "access_denied"; } if (/Throttl|TooManyRequests|RequestLimitExceeded|Rate exceeded/i.test(message)) return "throttled"; @@ -870,6 +874,9 @@ function awsProviderSafeMessage(code: SecretProviderClientErrorCode): string { } function normalizeAwsError(operation: string, error: unknown): never { + if (error instanceof SecretProviderClientError) { + throw error; + } const rawMessage = error instanceof Error ? error.message : String(error); const code = classifyAwsProviderError(rawMessage); throw new SecretProviderClientError({ diff --git a/server/src/services/secrets.ts b/server/src/services/secrets.ts index 800c0e2fbdd..0fc88ed5421 100644 --- a/server/src/services/secrets.ts +++ b/server/src/services/secrets.ts @@ -505,7 +505,7 @@ function safeRemoteProviderErrorDetails( if (context.operation === "secret.create") { details.requiredCapability = "secretsmanager:CreateSecret"; details.actionableMessage = - "AWS managed secret creation needs secretsmanager:CreateSecret in the selected region for this provider vault. If the vault config uses a KMS key, the runtime credentials also need KMS write permissions for that key."; + "Ensure the Paperclip server runtime can resolve valid AWS credentials, then grant them secretsmanager:CreateSecret in the selected region for this provider vault. If the vault config uses a KMS key, those credentials also need KMS write permissions for that key."; details.safeAlternative = "If the secret already exists in AWS, link it as an external reference instead of creating a Paperclip-managed value."; } else if (context.operation === "secret.rotate") { From 2d72f2c4c0b2f119db3b94ea75bd8ee8b70ed959 Mon Sep 17 00:00:00 2001 From: "commitperclip[bot]" <284812584+commitperclip[bot]@users.noreply.github.com> Date: Mon, 26 Oct 2026 20:55:37 -0700 Subject: [PATCH 051/133] fix: Serialize ACPX session record saves per record to stop overlapping renames (EPERM on Windows) (#16925) > [!NOTE] > Opened by Prospector's automated issue-fix pipeline. A maintainer reviews and decides; nothing here merges on its own. ## Thinking Path - Issue #15346 reports a defect in paperclipai/paperclip. - Root cause, as the fixing agent read it: ACPX saves one session record from several writers (live checkpoint timer, turn end, finalize, close) with no coordination; each save renames a temp file onto the same record file, and on Windows overlapping renames onto one file fail with EPERM. All of these saves go through Paperclip's session-store wrapper, which forwarded them concurrently. - This pull request: Serialize ACPX session record saves per record to stop overlapping renames (EPERM on Windows) ## What Changed - `packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.ts`: Every ACPX save in Paperclip passes through this wrapper, so it now puts all saves of one record id in a single queue (a promise chain shared by all wrappers in the process). Saves of one record never overlap and keep their order; a failed save still returns its own error and does not block the next one; saves of different records still run in parallel. ## Verification - Reproducing test(s), added in this PR: `packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.test.ts` - They fail on the base commit and pass with the fix, each twice in a fresh sandbox. - 3 independent attempts wrote 3 reproductions; this fix passes every one of them. - The repository's compile command passes with the fix applied. - 3 existing test file(s) near the change pass. - The full test suite passes except for the 246 file(s) that already fail on the base commit. ## Risks - Written by an automated pipeline, not a person: review it as you would an outside contribution. - Risk tier of the touched paths: 2. - The scope reviewer found it also changes, beyond the report: save() called when no earlier save of that record is pending - The scope reviewer found it also changes, beyond the report: persistedStore.save throwing synchronously - The scope reviewer found it also changes, beyond the report: Records with missing/undefined acpxRecordId - The scope reviewer found it also changes, beyond the report: save() for different record ids ## Model Used - opus, sonnet (Anthropic Claude, via Claude Code), writing and checking the change in an isolated sandbox. ## Linked Issues Fixes #15346 Other pull requests that name #15346: - #15370 (closed): fix(acpx): serialize session file saves and clean failed temporary writes ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with 'Fixes: #' / 'Closes #' / 'Refs #' OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub '#NNN' / 'github.com/paperclipai/paperclip' URLs) - [ ] My branch name describes the change (e.g. 'docs/...', 'fix/...') and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com> --- .../ephemeral-session-environment.test.ts | 41 +++++++++++++++++++ .../ephemeral-session-environment.ts | 19 ++++++++- 2 files changed, 59 insertions(+), 1 deletion(-) diff --git a/packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.test.ts b/packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.test.ts index 40d13e7b403..ef0e834aa8b 100644 --- a/packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.test.ts +++ b/packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.test.ts @@ -162,4 +162,45 @@ describe("ephemeral ACPX session environment", () => { const input = freeze(record()); await expect(wrapper.save(input)).rejects.toBe(failure); expect(input.acpx?.session_options?.env).toEqual(currentEnvironment); }); + + it("never overlaps two saves of one record, even across wrappers, and keeps their order", async () => { + let active = 0, maxActive = 0; + const order: string[] = []; + const failure = new Error("fixture rename failure"); + const backing = stubStore(undefined); + backing.save.mockImplementation(async (saved) => { + active += 1; maxActive = Math.max(maxActive, active); + await new Promise(resolve => setTimeout(resolve, 5)); + order.push(saved.lastUsedAt); + active -= 1; + if (saved.lastUsedAt === "checkpoint") throw failure; + }); + const turnStore = createEphemeralSessionEnvironmentStore(backing, currentEnvironment); + const closeStore = createEphemeralSessionEnvironmentStore(backing, currentEnvironment); + const results = await Promise.allSettled([ + turnStore.save({ ...record(), lastUsedAt: "checkpoint" }), + turnStore.save({ ...record(), lastUsedAt: "turn-end" }), + closeStore.save({ ...record(), lastUsedAt: "close" }), + ]); + expect(maxActive).toBe(1); + expect(order).toEqual(["checkpoint", "turn-end", "close"]); + expect(results.map(result => result.status)).toEqual(["rejected", "fulfilled", "fulfilled"]); + expect((results[0] as PromiseRejectedResult).reason).toBe(failure); + }); + + it("does not serialize saves of different records", async () => { + let active = 0, maxActive = 0; + const backing = stubStore(undefined); + backing.save.mockImplementation(async () => { + active += 1; maxActive = Math.max(maxActive, active); + await new Promise(resolve => setTimeout(resolve, 5)); + active -= 1; + }); + const wrapper = createEphemeralSessionEnvironmentStore(backing, currentEnvironment); + await Promise.all([ + wrapper.save({ ...record(), acpxRecordId: "fixture-record-a" }), + wrapper.save({ ...record(), acpxRecordId: "fixture-record-b" }), + ]); + expect(maxActive).toBe(2); + }); }); diff --git a/packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.ts b/packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.ts index 3bd59942ec9..a4d068f9f6c 100644 --- a/packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.ts +++ b/packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.ts @@ -1,5 +1,22 @@ import type { AcpSessionStore } from "acpx/runtime"; +// ACPX saves one record from several writers (live checkpoint, turn end, +// finalize, close) with no lock. Each save renames a temp file onto the record +// file, and on Windows two overlapping renames onto one file fail with EPERM. +// Chain every save of one record so they never overlap. +const pendingSaves = new Map>(); + +function saveInOrder(recordId: string, save: () => Promise): Promise { + const previous = pendingSaves.get(recordId) ?? Promise.resolve(); + const current = previous.catch(() => undefined).then(save); + const settled = current.catch(() => undefined); + pendingSaves.set(recordId, settled); + void settled.then(() => { + if (pendingSaves.get(recordId) === settled) pendingSaves.delete(recordId); + }); + return current; +} + /** Keep the current run's environment in memory, outside persisted conversation state. */ export function createEphemeralSessionEnvironmentStore( persistedStore: AcpSessionStore, @@ -26,7 +43,7 @@ export function createEphemeralSessionEnvironmentStore( const { env: _environment, ...sessionOptions } = record.acpx.session_options; persisted.acpx = { ...record.acpx, session_options: sessionOptions }; } - return persistedStore.save(persisted); + return saveInOrder(record.acpxRecordId, () => persistedStore.save(persisted)); }, }; } From 4da876ab0b07ac8c1a00f7a911586cce4e0fa705 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Mon, 26 Oct 2026 21:08:00 -0700 Subject: [PATCH 052/133] fix(native): always advertise run-scoped readers to provider threads (#16995) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs use a Paperclip tool authority to declare tools to the provider thread. Codex runs through runnerd. > - An external-chat wake with a truncated comment batch requires the agent to call `read_current_wake_comments`. A terminal fence fails the run if the agent did not read every bound comment. > - Codex `thread/resume` has no `dynamicTools` parameter. A resumed thread keeps the tool catalog from its first `thread/start`. > - The tool authority declared `read_current_wake_comments` and `read_chat_attachment` only when the current run had a binding. The tool-contract fingerprint already said that both tools are "always advertised", so the code and the contract did not agree. > - When a truncated chat wake resumed a thread that started on an ordinary turn, the agent could not see the reader. The agent blocked, and the fence failed the run with `native_current_wake_comments_unread`. The task then stayed blocked for the board. > - This pull request declares both readers on every turn and rotates threads that started with the old catalog. > - The benefit: truncated GitHub, Slack, and other chat wakes can read their full comment batch on resumed threads. They no longer fail at the fence. ## Linked Issues or Issue Description No public issue exists. This is the bug description, in the fields of `bug_report.yml`. **What happened?** A native Codex run got a GitHub chat wake with a truncated comment batch. The prompt told the agent to call `read_current_wake_comments`. The agent searched for the tool and did not find it. It reported a block (`required_wake_reader_unavailable`). The server sent the reader in the run's authorized tool list, but the provider thread had resumed. The fence then failed the run with `native_current_wake_comments_unread`, and the task became blocked. This happened 6 times on staging for the same task. **Expected behavior** A resumed thread can call `read_current_wake_comments` and `read_chat_attachment` on any turn that has a binding. The run then completes normally. **Steps to reproduce** 1. Start a native Codex session on a task with an ordinary wake. Do not use a truncated chat wake. 2. Send an external-chat message that makes the wake comment batch truncated (`fallbackFetchNeeded: true`). 3. The run resumes the same Codex thread. The model cannot see `read_current_wake_comments`. The run fails with `native_current_wake_comments_unread`. **Paperclip version or commit** Hosted staging release `fc0d390d`. The code is the same on `master` at `3bbc210`. **Deployment mode** Hosted (staging). The native runner uses the `codex_app_server` backend through runnerd in an external sandbox. **Agent adapter(s) involved** Codex (native `paperclip_runner`). Related history: the existing retained-fingerprint table already lists a "conditional current-wake reader" rotation. This change fixes a regression of that contract. I searched open and closed PRs for `current_wake_comments` and "wake comments" and found no duplicate fix. ## What Changed - `PaperclipRunnerToolAuthority.definitions()` now always declares `read_current_wake_comments` and `read_chat_attachment`. Execution stays fail-closed: without the server-verified binding or scope, a call still fails with `paperclip_runner_tool_not_advertised` or `paperclip_runner_chat_attachment_read_scope_unavailable`. - The native tool-contract fingerprint goes from v17 to v18, with the two policy values set to `.v2`. Checkpoints from deployed v17 threads, which may lack the readers, start a fresh provider thread one time with full task context. - New regression test: the catalog for an ordinary turn is equal to the catalog for a truncated chat turn, and it contains both readers. - New resume tests: the deployed v17 local and remote fingerprints do not rebind. The existing test that asserted conditional advertisement now asserts the stable catalog and the fail-closed execution. ## Verification - Red before the fix: `npx vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts -t "advertises only real bindings|keeps run-scoped reader declarations stable"` had 2 failures. The unbound catalog had 48 tools and did not include either reader. - Green after the fix: `paperclip-runner-tool-authority.test.ts`, `native-session-resume.test.ts`, `current-wake-comments.test.ts`, `chat-attachment-read.test.ts`, `native-execution-input.test.ts`, and `file-delivery-bridges.test.ts` all pass. - `native-session-executor.test.ts`: 764 pass and 1 fails. The failure (`compares export process compatibility … (changed: false)`, `db.transaction is not a function`) also happens on unchanged `main`, so this change did not cause it. - Consequential assumption: Codex does not refresh dynamic tools on resume. Evidence: `codex app-server generate-json-schema` for codex-cli 0.160.0 shows `ThreadStartParams.dynamicTools`, but `ThreadResumeParams` has no `dynamicTools` field. The failed staging run also showed it: the server listed `read_current_wake_comments` as authorized, but the model's tool search returned no match after `thread/resume`. - Not run locally: the server `tsc --noEmit` step (the sandbox has a 4 GB memory limit and the process was killed) and `dot-runner.test.ts` (`cargo` is not installed). CI runs both. - Not run: a live end-to-end chat wake against a real Codex provider. That needs a staging deployment. ## Risks - The fingerprint bump makes every native session that resumes from a v17 checkpoint start a fresh provider thread one time. Paperclip rebuilds the full task context, but the provider-side conversation history is not carried over. This is the same tradeoff as earlier fingerprint bumps. - Two more tool declarations use a small part of the Codex dynamic-tool budget. The assigned-MCP fit check already includes these definitions. - Follow-up, not in this PR: the native Codex driver reports `toolRefreshOnResume: true`, and the runnerd fake app server accepts `dynamicTools` on `thread/resume`. Real codex-cli 0.160.0 has no such field. Connection-tool continuations that depend on resume refresh need a separate check. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) from Anthropic, running as a Paperclip agent with tool use (shell, file edit, and test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 --- .../native-session-resume.test.ts | 18 +++++++++ .../native-runtime/native-session-resume.ts | 6 +-- .../paperclip-runner-tool-authority.test.ts | 40 +++++++++++++++++-- .../paperclip-runner-tool-authority.ts | 9 +++-- 4 files changed, 62 insertions(+), 11 deletions(-) diff --git a/server/src/services/native-runtime/native-session-resume.test.ts b/server/src/services/native-runtime/native-session-resume.test.ts index c0f274b08e4..932260c5640 100644 --- a/server/src/services/native-runtime/native-session-resume.test.ts +++ b/server/src/services/native-runtime/native-session-resume.test.ts @@ -1773,6 +1773,13 @@ describe("rebindNativeSessionCheckpoint", () => { retainedFingerprint: "sha256:f12fe3b1bf5d63b7954d8ab16f57764874806ab49acff7c6582c4de007f4ed4d", }, + { + contract: "regressed conditional run-scoped readers", + // Deployed v17 threads were opened without read_current_wake_comments + // and read_chat_attachment on ordinary turns; resume cannot add them. + retainedFingerprint: + "sha256:dc495475fb6a4bd034c6dbb2bbe0cf52e7ffc1635838889dc5f7d7251273f119", + }, { contract: "task-bound human-input description", // Deployed v9 still advertises the generic mock-task question description. @@ -1876,6 +1883,17 @@ describe("rebindNativeSessionCheckpoint", () => { executionTargetKind: "remote", }), ).not.toBeNull(); + // Deployed v17 remote threads were opened without the run-scoped readers. + expect( + rebindNativeSessionCheckpoint({ + previousRun: previousRun({ + nativeToolContractFingerprint: + "sha256:96dde421aab01f847468d1a8f12863d32dfdb3038a9aca61e1e6bdc81022d5a1", + }), + currentExecution: execution(currentRunId), + executionTargetKind: "remote", + }), + ).toBeNull(); }); diff --git a/server/src/services/native-runtime/native-session-resume.ts b/server/src/services/native-runtime/native-session-resume.ts index e3ea827cbdf..a271601ce0b 100644 --- a/server/src/services/native-runtime/native-session-resume.ts +++ b/server/src/services/native-runtime/native-session-resume.ts @@ -34,17 +34,17 @@ export function nativeToolContractFingerprintForTarget( return `sha256:${createHash("sha256") .update( JSON.stringify({ - schema: "paperclip.native-tool-contract.v17", + schema: "paperclip.native-tool-contract.v18", executionTargetKind, advertisementPolicy: { // Direct provider threads retain declarations from thread/start. // Keep this explicit so changing a tool from conditional to stable // advertisement rotates checkpoints even when its schema is unchanged. - readCurrentWakeComments: "always_advertised_binding_gated.v1", + readCurrentWakeComments: "always_advertised_binding_gated.v2", historicalChatAttachments: "always_advertised_conversation_binding_gated.v1", registerDeliverable: "verified_local_or_remote_workspace.v2", - readChatAttachment: "always_advertised_run_scope_local_staging.v1", + readChatAttachment: "always_advertised_run_scope_local_staging.v2", structuredHumanInput: "always_advertised_run_issue_agent_binding_gated_current_task_description.v2", semanticCompletion: "finish_accessible_deliverable_evidence.v4", diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts index 74524c733f3..247f15532a3 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts @@ -28,7 +28,8 @@ import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority. import { PaperclipRunnerSemanticAuthority } from "./runner-semantic-authority.js"; import { createAssignedMcpTools } from "./assigned-mcp-tools.js"; import type { ToolGatewayService } from "../tool-gateway.js"; -import { READ_CURRENT_WAKE_COMMENTS_TOOL_NAME } from "./current-wake-comments.js"; +import { CURRENT_WAKE_COMMENTS_BINDING_SCHEMA, READ_CURRENT_WAKE_COMMENTS_TOOL_NAME } from "./current-wake-comments.js"; +import { NativeChatAttachmentReadScope, READ_CHAT_ATTACHMENT_TOOL_NAME } from "./chat-attachment-read.js"; import { buildLowTrustSourceTrust, buildPromotedSourceTrust, LOW_TRUST_QUARANTINED_BODY } from "../source-trust.js"; import { CAPABILITY_SEMANTIC_TOOL_CATALOG, runnerCodexDynamicToolsFit } from "../../vendor/paperclip-runner/index.js"; @@ -119,12 +120,10 @@ describe("PaperclipRunnerToolAuthority", () => { issueId, runId, }); - expect(authority.definitions()).toHaveLength(48); + expect(authority.definitions()).toHaveLength(50); expect(authority.definitions().map(tool => tool.name)).toEqual(expect.arrayContaining([ "get_workspace", "list_workspaces", "select_workspace", "prepare_repository", ])); - expect(authority.definitions().map(tool => tool.name)).not.toContain("read_chat_attachment"); - expect(authority.definitions().map(tool => tool.name)).not.toContain("read_current_wake_comments"); const questions = authority.definitions().find(tool => tool.name === "request_human_input")!; expect(questions.description).toContain("ask only the next unanswered question"); expect(questions.description).toContain("Never fabricate answers"); @@ -197,6 +196,39 @@ describe("PaperclipRunnerToolAuthority", () => { arguments: {}, }), ).rejects.toThrow("paperclip_runner_tool_not_advertised"); + await expect( + authority.execute({ + tool: READ_CHAT_ATTACHMENT_TOOL_NAME, + callId: "attachment-read-without-run-scope", + arguments: { sourceCommentId: randomUUID(), attachmentId: randomUUID() }, + }), + ).rejects.toThrow("paperclip_runner_chat_attachment_read_scope_unavailable"); + }); + + it("keeps run-scoped reader declarations stable for resumed provider threads", () => { + // Codex thread/resume has no dynamicTools parameter: a resumed thread keeps + // the catalog from its original thread/start. A truncated chat wake that + // resumes a thread opened on an ordinary turn still needs both readers. + const ordinary = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId }); + const chatBinding = { companyId, agentId, issueId, runId }; + const truncatedChat = new PaperclipRunnerToolAuthority(db, { + ...chatBinding, + currentWakeComments: { + schema: CURRENT_WAKE_COMMENTS_BINDING_SCHEMA, + ...chatBinding, + provider: "github", + commentIds: [randomUUID()], + attachmentOmissions: [], + bindingDigest: "binding-digest", + }, + chatAttachmentReadScope: new NativeChatAttachmentReadScope({ + db, binding: chatBinding, workspaceRoot: "/workspace", executionTargetKind: "local", + }), + }); + expect(ordinary.definitions().map(tool => tool.name)).toEqual(expect.arrayContaining([ + READ_CURRENT_WAKE_COMMENTS_TOOL_NAME, READ_CHAT_ATTACHMENT_TOOL_NAME, + ])); + expect(ordinary.definitions()).toEqual(truncatedChat.definitions()); }); it("reads current tool authority without transferring saved profiles or output", async () => { diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index cc535813f77..aa6863205fb 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -315,12 +315,13 @@ export class PaperclipRunnerToolAuthority { : descriptor.inputSchema, })); // Keep the provider session's direct tool catalog stable across ordinary - // and truncated external-chat turns. Execution still fails closed unless - // this exact run carries a server-verified current-wake binding. - if (this.binding.currentWakeComments) definitions.push(READ_CURRENT_WAKE_COMMENTS_TOOL_DEFINITION); + // and truncated external-chat turns: Codex thread/resume cannot add tools, + // so a reader missing at thread/start stays missing. Execution still fails + // closed unless this exact run carries the server-verified binding/scope. + definitions.push(READ_CURRENT_WAKE_COMMENTS_TOOL_DEFINITION); definitions.push(LIST_CHAT_ATTACHMENTS_TOOL_DEFINITION); definitions.push(REUSE_CHAT_ATTACHMENT_TOOL_DEFINITION); - if (this.binding.chatAttachmentReadScope) definitions.push(READ_CHAT_ATTACHMENT_TOOL_DEFINITION); + definitions.push(READ_CHAT_ATTACHMENT_TOOL_DEFINITION); if (this.binding.taskAttachmentRead) definitions.push(...TASK_ATTACHMENT_DEFINITIONS); definitions.push(...runnerBridgeDefinitions({ workspace: !this.binding.firstPartyOnly && !!this.binding.workspaceRoot && this.binding.workspaceBridge === true, skills: !!this.binding.runtimeContext, api: runnerApiToolsEnabled(this.binding.companyId, this.binding.apiToolsEnabled), mode: workMode })); const connectionTools = this.binding.firstPartyOnly ? [] : [...RUNTIME_CONNECTION_TOOL_DEFINITIONS, From 3e7c3ee9a97538215454054bcaf3891ecefbd44f Mon Sep 17 00:00:00 2001 From: Tonio Date: Mon, 26 Oct 2026 21:08:49 -0700 Subject: [PATCH 053/133] fix(ui): fall back to the icon face when an Artifacts V1 preview fails to load (#17260) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents produce artifacts: documents, files, pictures, video and links. The Artifacts V1 presentation (`enableArtifactsV1`, #17228) shows them as prints, sheets and rows on the Artifacts page, in the panel's Artifacts tab and in the task thread > - A picture is printed from its attachment's content path. When that content cannot be loaded (on staging, image work products whose attachment returns 404), the browser draws its broken-picture glyph inside the bordered print and inside the 40px row thumbnail > - The presentation already has a face for a file with nothing to show: the icon sheet (the faint glyph and the type label) and the row's document icon. A picture that cannot be loaded should take that face, not a browser glyph > - This pull request listens for the element's own `error` event in the V1 primitives and falls back to the icon face on all three surfaces: the stack front and the single tile on the Artifacts page, and the panel and thread rows > - The benefit is that a missing or unreadable file reads as a file of its type everywhere, with no fetch to find out, and nothing changes for a picture that loads ## Linked Issues or Issue Description No public issue exists. This section follows the bug report template. **What happened?** With Artifacts V1 on, an image work product whose attachment content returns 404 ("Attachment not found") shows the browser's broken-image glyph inside the bordered print on the Artifacts page (the stack front and the single tile) and inside the 40px thumbnail in the panel and thread rows. **What did you expect to happen?** The artifact shows the icon face a file without a preview gets: the icon sheet with the faint glyph and the type label on the Artifacts page, and the document icon with the type label in the rows. **Steps to reproduce** 1. Turn on Settings > Experimental > Artifacts V1. 2. Register an image work product whose attachment content returns 404, or delete the attachment behind one. 3. Open the Artifacts page, the task's Artifacts tab and the task thread. **Relevant subsystem** The web UI (`ui/src/components/artifacts/v1/`). ## What Changed - `ArtifactPrint`: a new `usePreviewFailure(source)` hook beside `useImageAspect`, and an `onMediaError` prop wired to the `error` event of the print's `` and `