From 01b51dc0e5e660cce77750a1ad9e511f1ec13b02 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Fri, 7 Aug 2026 07:23:56 -0700 Subject: [PATCH 001/201] fix(runtime): stop Live badge and Working shimmer after task teardown (#10985) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The task list and the chat views show a Live badge and a Working shimmer for an issue that has an active run. > - A finished task kept the Live badge and the Working shimmer after the run ended and the sandbox stopped. > - The user interface reads run liveness from the `heartbeat_runs.status` row. The run finalizer writes the terminal status in a step that is separate from the agent `status=done` update. When the sandbox or the run process stops between the two steps, `heartbeat_runs.status` stays `running` forever. > - A run row that stays `running` makes a finished task look perpetually Live, and the user interface has no guard for an issue that already reached a terminal status. > - This pull request closes the invariant "environment lease released implies the run is terminal" on the server, and adds a user interface guard that suppresses live state for a terminal issue. > - The benefit is that a finished task stops showing Live and Working, both at the source (the run row) and at the surface (the badge and the shimmer). ## Linked Issues or Issue Description **Bug description** - A completed task kept the Live badge and the Working shimmer after its run ended and the sandbox was torn down. **Steps to reproduce** - Run an agent task to completion. Let the sandbox tear down while the run finalizer is between the `status=done` update and the terminal run-status write. - Open the task list or the chat view for the finished task. **Expected behavior** - A finished task shows no Live badge and no Working shimmer. **Actual behavior (before this change)** - The finished task showed the Live badge and the Working shimmer because its `heartbeat_runs.status` row stayed `running`. This pull request supersedes the two separate pull requests #10954 (frontend) and #10955 (backend). It carries all of their changes for the same race. ## What Changed Server: - Run teardown terminalizes a still-running or still-queued run before it releases the environment lease. It writes `succeeded` when the issue already reached `done`, `cancelled` when the issue is `cancelled`, and `interrupted` otherwise. It never overwrites a status that another path already made terminal. - The recovery stale-lock sweep terminalizes an orphaned running run to `interrupted` after it confirms the process and the sandbox are both gone. It requires recorded process metadata, so it never terminalizes a live run, a queued run, or a scheduled retry. - Each terminal transition writes a run event. - The stale-lock sweep continues and clears the lock when the audit write fails. It logs the failure loudly. - New server tests cover both invariants. User interface: - A shared guard suppresses the Live badge and the Working shimmer when the issue status is terminal. - The guard keeps non-terminal `queued` and `running` issues live. - The guard prefers the newest issue live-status snapshot. - New user interface tests cover the guard and the snapshot preference. ## Verification Server: - `pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && tsc --noEmit` in `server/` — 0 errors. - `pnpm --filter server test heartbeat-run-lease-release-terminalization.test.ts recovery-stale-issue-lock-sweep.test.ts` — 12 tests pass. User interface: - `pnpm --filter @paperclipai/ui typecheck` — 0 errors. - `pnpm exec vitest run ui/src/lib/liveIssueIds.test.ts ui/src/lib/issue-chat-messages.test.ts` — 40 tests pass. ## Risks - Low risk. The server change only forces a still-live run row to a terminal status when the lease releases or when the recovery sweep confirms the process is dead. It never overwrites an existing terminal status, and it guards the recovery path with process metadata to avoid terminalizing a live run. - The user interface change is additive. The guard only suppresses live state for a terminal issue and keeps queued and running issues live. - No database migration. No change to any external endpoint. ## Model Used - Claude Opus 4.8 (`claude-opus-4-8`), extended thinking, tool use, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- ...-run-lease-release-terminalization.test.ts | 212 ++++++++++ .../recovery-stale-issue-lock-sweep.test.ts | 367 ++++++++++++++++++ server/src/services/heartbeat.ts | 102 ++++- server/src/services/recovery/service.ts | 218 ++++++++++- ui/src/components/IssueChatThread.tsx | 12 +- ui/src/components/TaskChatThread.tsx | 8 +- ui/src/lib/issue-chat-messages.test.ts | 38 ++ ui/src/lib/issue-chat-messages.ts | 11 +- ui/src/lib/liveIssueIds.test.ts | 83 ++++ ui/src/lib/liveIssueIds.ts | 62 ++- ui/src/pages/ExecutionWorkspaceDetail.tsx | 2 +- ui/src/pages/Inbox.tsx | 10 +- ui/src/pages/IssueDetail.tsx | 5 +- ui/src/pages/Issues.tsx | 3 +- ui/src/pages/ProjectDetail.tsx | 6 +- ui/src/pages/Routines.tsx | 2 +- ui/src/plugins/bridge-init.ts | 3 +- 17 files changed, 1113 insertions(+), 31 deletions(-) create mode 100644 server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts diff --git a/server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts b/server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts new file mode 100644 index 00000000000..4cda968220c --- /dev/null +++ b/server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts @@ -0,0 +1,212 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { + agents, + companies, + createDb, + heartbeatRunEvents, + heartbeatRuns, + issues, +} from "@paperclipai/db"; +import { + getEmbeddedPostgresTestSupport, + startEmbeddedPostgresTestDatabase, +} from "./helpers/embedded-postgres.js"; + +const mockTelemetryClient = vi.hoisted(() => ({ track: vi.fn() })); +vi.mock("../telemetry.ts", () => ({ getTelemetryClient: () => mockTelemetryClient })); + +import { heartbeatService } from "../services/heartbeat.ts"; + +const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; + +if (!embeddedPostgresSupport.supported) { + console.warn( + `Skipping embedded Postgres lease-release terminalization tests on this host: ${ + embeddedPostgresSupport.reason ?? "unsupported environment" + }`, + ); +} + +describeEmbeddedPostgres("heartbeat terminalizeRunOnLeaseRelease", () => { + let db!: ReturnType; + let tempDb: Awaited> | null = null; + + beforeAll(async () => { + tempDb = await startEmbeddedPostgresTestDatabase("paperclip-lease-release-terminal-"); + db = createDb(tempDb.connectionString); + }, 20_000); + + afterEach(async () => { + await db.delete(heartbeatRunEvents); + await db.delete(issues); + await db.delete(heartbeatRuns); + await db.delete(agents); + await db.delete(companies); + }); + + afterAll(async () => { + await tempDb?.cleanup(); + }); + + async function seed(input: { issueStatus: string; runStatus: string }) { + const companyId = randomUUID(); + const agentId = randomUUID(); + const issueId = randomUUID(); + const runId = randomUUID(); + + await db.insert(companies).values({ + id: companyId, + name: "Paperclip", + issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`, + requireBoardApprovalForNewAgents: false, + }); + await db.insert(agents).values({ + id: agentId, + companyId, + name: "Coder", + role: "engineer", + status: "active", + adapterType: "codex_local", + adapterConfig: {}, + runtimeConfig: {}, + permissions: {}, + }); + await db.insert(issues).values({ + id: issueId, + companyId, + title: "Terminalize on lease release", + status: input.issueStatus, + priority: "high", + assigneeAgentId: agentId, + }); + await db.insert(heartbeatRuns).values({ + id: runId, + companyId, + agentId, + status: input.runStatus, + invocationSource: "manual", + startedAt: new Date(), + contextSnapshot: { issueId }, + }); + + const run = await db + .select() + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runId)) + .then((rows) => rows[0]!); + + return { companyId, agentId, issueId, runId, run }; + } + + it("forces a still-running run to succeeded when the issue already reached done", async () => { + // This reproduces the defect: the agent PATCHed the issue to done, but the + // teardown released the environment lease before the run-terminal write. + const { issueId, runId, run } = await seed({ issueStatus: "done", runStatus: "running" }); + + const heartbeat = heartbeatService(db); + const terminal = await heartbeat.terminalizeRunOnLeaseRelease(run); + + expect(terminal.status).toBe("succeeded"); + + const runStatus = await db + .select({ status: heartbeatRuns.status }) + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runId)) + .then((rows) => rows[0]?.status); + expect(runStatus).toBe("succeeded"); + + const issueStatus = await db + .select({ status: issues.status }) + .from(issues) + .where(eq(issues.id, issueId)) + .then((rows) => rows[0]?.status); + expect(issueStatus).toBe("done"); + + const event = await db + .select({ message: heartbeatRunEvents.message, payload: heartbeatRunEvents.payload }) + .from(heartbeatRunEvents) + .where(eq(heartbeatRunEvents.runId, runId)) + .then((rows) => rows[0]); + expect(event?.message).toContain("lease release"); + expect((event?.payload as { terminalStatus?: string } | null)?.terminalStatus).toBe("succeeded"); + }); + + it("forces a still-running run to interrupted when the issue is not terminal", async () => { + const { runId, run } = await seed({ issueStatus: "in_progress", runStatus: "running" }); + + const heartbeat = heartbeatService(db); + const terminal = await heartbeat.terminalizeRunOnLeaseRelease(run); + + expect(terminal.status).toBe("interrupted"); + + const row = await db + .select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode }) + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runId)) + .then((rows) => rows[0]); + expect(row?.status).toBe("interrupted"); + expect(row?.errorCode).toBe("lease_released_before_terminal"); + }); + + it("forces a still-queued run to interrupted when the lease releases before it starts", async () => { + // A queued run holds a lease but never reached "running". The teardown + // released the lease, so the run must not stay queued and show a phantom + // live run. A running-only update would miss it. + const { runId, run } = await seed({ issueStatus: "in_progress", runStatus: "queued" }); + + const heartbeat = heartbeatService(db); + const terminal = await heartbeat.terminalizeRunOnLeaseRelease(run); + + expect(terminal.status).toBe("interrupted"); + + const row = await db + .select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode }) + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runId)) + .then((rows) => rows[0]); + expect(row?.status).toBe("interrupted"); + expect(row?.errorCode).toBe("lease_released_before_terminal"); + }); + + it("forces a still-queued run to succeeded when the issue already reached done", async () => { + const { runId, run } = await seed({ issueStatus: "done", runStatus: "queued" }); + + const heartbeat = heartbeatService(db); + const terminal = await heartbeat.terminalizeRunOnLeaseRelease(run); + + expect(terminal.status).toBe("succeeded"); + + const runStatus = await db + .select({ status: heartbeatRuns.status }) + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runId)) + .then((rows) => rows[0]?.status); + expect(runStatus).toBe("succeeded"); + }); + + it("keeps an already-terminal run authoritative and records no new event", async () => { + const { runId, run } = await seed({ issueStatus: "done", runStatus: "failed" }); + + const heartbeat = heartbeatService(db); + const terminal = await heartbeat.terminalizeRunOnLeaseRelease(run); + + expect(terminal.status).toBe("failed"); + + const runStatus = await db + .select({ status: heartbeatRuns.status }) + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runId)) + .then((rows) => rows[0]?.status); + expect(runStatus).toBe("failed"); + + const eventCount = await db + .select({ id: heartbeatRunEvents.id }) + .from(heartbeatRunEvents) + .where(eq(heartbeatRunEvents.runId, runId)) + .then((rows) => rows.length); + expect(eventCount).toBe(0); + }); +}); diff --git a/server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts b/server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts index de6baf9016b..718dde80c50 100644 --- a/server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts +++ b/server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts @@ -6,6 +6,7 @@ import { agents, companies, createDb, + heartbeatRunEvents, heartbeatRuns, issueComments, issueRelations, @@ -46,6 +47,7 @@ describeEmbeddedPostgres("recovery sweepStaleIssueLocks", () => { await db.delete(issueRelations); await db.delete(activityLog); await db.delete(issues); + await db.delete(heartbeatRunEvents); await db.delete(heartbeatRuns); await db.delete(agents); await db.delete(companies); @@ -223,4 +225,369 @@ describeEmbeddedPostgres("recovery sweepStaleIssueLocks", () => { expect(first.cleared).toBe(1); expect(second.cleared).toBe(0); }); + + it("terminalizes an orphaned running run whose process is gone, then clears the lock", async () => { + const { companyId, agentId, runningRunId } = await seed(); + // The run recorded a pid, but the process and its sandbox are gone. A pid + // this large never maps to a live process, so isPidAlive returns false. + // The issue is not terminal, so only the process-death authority applies. + await db + .update(heartbeatRuns) + .set({ processPid: 2_000_000_000 }) + .where(eq(heartbeatRuns.id, runningRunId)); + const issueId = randomUUID(); + await db.insert(issues).values({ + id: issueId, + companyId, + title: "Orphaned running run — terminalize then clear", + status: "in_progress", + priority: "high", + assigneeAgentId: agentId, + checkoutRunId: runningRunId, + executionRunId: runningRunId, + executionLockedAt: new Date(), + }); + + const heartbeat = heartbeatService(db); + const result = await heartbeat.sweepStaleIssueLocks(); + + expect(result.terminalizedRunIds).toEqual([runningRunId]); + expect(result.cleared).toBe(1); + + const run = await db + .select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode }) + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runningRunId)) + .then((rows) => rows[0]); + // Process died, outcome unknown, so the backstop uses "interrupted". + expect(run?.status).toBe("interrupted"); + expect(run?.errorCode).toBe("orphaned_running_run"); + + const lock = await db + .select({ checkoutRunId: issues.checkoutRunId, executionRunId: issues.executionRunId }) + .from(issues) + .where(eq(issues.id, issueId)) + .then((rows) => rows[0]); + expect(lock).toEqual({ checkoutRunId: null, executionRunId: null }); + + const event = await db + .select({ message: heartbeatRunEvents.message }) + .from(heartbeatRunEvents) + .where(eq(heartbeatRunEvents.runId, runningRunId)) + .then((rows) => rows[0]); + expect(event?.message).toContain("process and sandbox gone"); + }); + + it("terminalizes a running run whose issue is terminal, even while the process stays alive (reuse-lease path)", async () => { + // Reuse Lease ON stops the sandbox but keeps the server process alive, so + // the in-memory handle and the recorded pid can both persist. The + // process-death authority misses this case. The issue-terminal authority + // catches it: the issue reached "done" while the run row stayed "running". + const { companyId, agentId, runningRunId } = await seed(); + // process.pid is the live test process, so isPidAlive returns true. + await db + .update(heartbeatRuns) + .set({ processPid: process.pid }) + .where(eq(heartbeatRuns.id, runningRunId)); + const issueId = randomUUID(); + await db.insert(issues).values({ + id: issueId, + companyId, + title: "Reused sandbox stopped — issue done, run still running", + status: "done", + priority: "high", + assigneeAgentId: agentId, + checkoutRunId: runningRunId, + executionRunId: runningRunId, + executionLockedAt: new Date(), + }); + + const heartbeat = heartbeatService(db); + const result = await heartbeat.sweepStaleIssueLocks(); + + expect(result.terminalizedRunIds).toEqual([runningRunId]); + expect(result.cleared).toBe(1); + + const run = await db + .select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode }) + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runningRunId)) + .then((rows) => rows[0]); + // The issue is "done", so the terminal run status is "succeeded". A + // succeeded run carries no error code. + expect(run?.status).toBe("succeeded"); + expect(run?.errorCode).toBeNull(); + + const lock = await db + .select({ checkoutRunId: issues.checkoutRunId, executionRunId: issues.executionRunId }) + .from(issues) + .where(eq(issues.id, issueId)) + .then((rows) => rows[0]); + expect(lock).toEqual({ checkoutRunId: null, executionRunId: null }); + + const event = await db + .select({ message: heartbeatRunEvents.message }) + .from(heartbeatRunEvents) + .where(eq(heartbeatRunEvents.runId, runningRunId)) + .then((rows) => rows[0]); + expect(event?.message).toContain("issue reached a terminal status"); + }); + + it("terminalizes a running run to cancelled when its issue is cancelled (reuse-lease path)", async () => { + const { companyId, agentId, runningRunId } = await seed(); + await db + .update(heartbeatRuns) + .set({ processPid: process.pid }) + .where(eq(heartbeatRuns.id, runningRunId)); + const issueId = randomUUID(); + await db.insert(issues).values({ + id: issueId, + companyId, + title: "Reused sandbox stopped — issue cancelled, run still running", + status: "cancelled", + priority: "high", + assigneeAgentId: agentId, + checkoutRunId: runningRunId, + executionRunId: runningRunId, + executionLockedAt: new Date(), + }); + + const heartbeat = heartbeatService(db); + const result = await heartbeat.sweepStaleIssueLocks(); + + expect(result.terminalizedRunIds).toEqual([runningRunId]); + + const runStatus = await db + .select({ status: heartbeatRuns.status }) + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runningRunId)) + .then((rows) => rows[0]?.status); + expect(runStatus).toBe("cancelled"); + }); + + it("does not terminalize a running run whose process is alive and whose issue is not terminal", async () => { + const { companyId, agentId, runningRunId } = await seed(); + // process.pid is the live test process, so isPidAlive returns true. + await db + .update(heartbeatRuns) + .set({ processPid: process.pid }) + .where(eq(heartbeatRuns.id, runningRunId)); + const issueId = randomUUID(); + await db.insert(issues).values({ + id: issueId, + companyId, + title: "Live run — preserve", + status: "in_progress", + priority: "high", + assigneeAgentId: agentId, + checkoutRunId: runningRunId, + executionRunId: runningRunId, + executionLockedAt: new Date(), + }); + + const heartbeat = heartbeatService(db); + const result = await heartbeat.sweepStaleIssueLocks(); + + expect(result.terminalizedRunIds).toEqual([]); + expect(result.cleared).toBe(0); + + const runStatus = await db + .select({ status: heartbeatRuns.status }) + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runningRunId)) + .then((rows) => rows[0]?.status); + expect(runStatus).toBe("running"); + }); + + it("does not terminalize a live run that a terminal issue and an active issue both reference", async () => { + // A stale lock on a terminal issue and the real lock on an active issue can + // point at the same running run. The terminal reference alone must not + // terminalize the run, because the run is still live for the active issue. + const { companyId, agentId, runningRunId } = await seed(); + // process.pid is the live test process, so isPidAlive returns true. + await db + .update(heartbeatRuns) + .set({ processPid: process.pid }) + .where(eq(heartbeatRuns.id, runningRunId)); + + const terminalIssueId = randomUUID(); + const activeIssueId = randomUUID(); + await db.insert(issues).values([ + { + id: terminalIssueId, + companyId, + title: "Terminal issue holds a stale lock on the shared run", + status: "done", + priority: "high", + assigneeAgentId: agentId, + checkoutRunId: runningRunId, + executionRunId: null, + }, + { + id: activeIssueId, + companyId, + title: "Active issue owns the live shared run", + status: "in_progress", + priority: "high", + assigneeAgentId: agentId, + checkoutRunId: runningRunId, + executionRunId: runningRunId, + executionLockedAt: new Date(), + }, + ]); + + const heartbeat = heartbeatService(db); + const result = await heartbeat.sweepStaleIssueLocks(); + + // The run stays live, so the sweep terminalizes nothing and clears nothing. + expect(result.terminalizedRunIds).toEqual([]); + expect(result.cleared).toBe(0); + + const runStatus = await db + .select({ status: heartbeatRuns.status }) + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runningRunId)) + .then((rows) => rows[0]?.status); + expect(runStatus).toBe("running"); + + const activeLock = await db + .select({ checkoutRunId: issues.checkoutRunId, executionRunId: issues.executionRunId }) + .from(issues) + .where(eq(issues.id, activeIssueId)) + .then((rows) => rows[0]); + expect(activeLock).toEqual({ checkoutRunId: runningRunId, executionRunId: runningRunId }); + }); + + it("does not terminalize a shared live run when its context snapshot names the terminal issue", async () => { + // The run context snapshot names the terminal issue. The context-snapshot + // fallback in terminalizeOrphanedRunningRun could read that terminal status + // and terminalize the run. An active issue still owns the run, so the sweep + // must suppress the fallback and keep the run live. + const { companyId, agentId, runningRunId } = await seed(); + // process.pid is the live test process, so isPidAlive returns true. + await db + .update(heartbeatRuns) + .set({ processPid: process.pid }) + .where(eq(heartbeatRuns.id, runningRunId)); + + const terminalIssueId = randomUUID(); + const activeIssueId = randomUUID(); + // The run context snapshot names the terminal issue. This is the path the + // shared-run guard must still block. + await db + .update(heartbeatRuns) + .set({ contextSnapshot: { issueId: terminalIssueId } }) + .where(eq(heartbeatRuns.id, runningRunId)); + await db.insert(issues).values([ + { + id: terminalIssueId, + companyId, + title: "Terminal issue named in the run context snapshot", + status: "done", + priority: "high", + assigneeAgentId: agentId, + checkoutRunId: runningRunId, + executionRunId: null, + }, + { + id: activeIssueId, + companyId, + title: "Active issue owns the live shared run", + status: "in_progress", + priority: "high", + assigneeAgentId: agentId, + checkoutRunId: runningRunId, + executionRunId: runningRunId, + executionLockedAt: new Date(), + }, + ]); + + const heartbeat = heartbeatService(db); + const result = await heartbeat.sweepStaleIssueLocks(); + + // The run stays live, so the sweep terminalizes nothing and clears nothing. + expect(result.terminalizedRunIds).toEqual([]); + expect(result.cleared).toBe(0); + + const runStatus = await db + .select({ status: heartbeatRuns.status }) + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runningRunId)) + .then((rows) => rows[0]?.status); + expect(runStatus).toBe("running"); + + const activeLock = await db + .select({ checkoutRunId: issues.checkoutRunId, executionRunId: issues.executionRunId }) + .from(issues) + .where(eq(issues.id, activeIssueId)) + .then((rows) => rows[0]); + expect(activeLock).toEqual({ checkoutRunId: runningRunId, executionRunId: runningRunId }); + }); + + it("still clears the lock when the audit write fails after terminalization", async () => { + const { companyId, agentId, runningRunId } = await seed(); + // The run recorded a pid that never maps to a live process, so the sweep + // decides to terminalize it. The issue is not terminal, so the + // process-death authority drives the terminalization here. + await db + .update(heartbeatRuns) + .set({ processPid: 2_000_000_000 }) + .where(eq(heartbeatRuns.id, runningRunId)); + const issueId = randomUUID(); + await db.insert(issues).values({ + id: issueId, + companyId, + title: "Audit write fails — still clear the lock", + status: "in_progress", + priority: "high", + assigneeAgentId: agentId, + checkoutRunId: runningRunId, + executionRunId: runningRunId, + executionLockedAt: new Date(), + }); + + // Make only the audit-event insert fail. The run update commits the + // terminal status first, so the audit write is best-effort. The sweep must + // catch the failure and still clear the lock. + const realInsert = db.insert.bind(db); + const insertSpy = vi.spyOn(db, "insert").mockImplementation((table) => { + if (table === heartbeatRunEvents) { + throw new Error("simulated audit write failure"); + } + return realInsert(table); + }); + + try { + const heartbeat = heartbeatService(db); + const result = await heartbeat.sweepStaleIssueLocks(); + + expect(result.terminalizedRunIds).toEqual([runningRunId]); + expect(result.cleared).toBe(1); + } finally { + insertSpy.mockRestore(); + } + + // The run reached its terminal status even though the audit write failed. + const runStatus = await db + .select({ status: heartbeatRuns.status }) + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runningRunId)) + .then((rows) => rows[0]?.status); + expect(runStatus).toBe("interrupted"); + + // The sweep cleared the lock in the same pass. + const lock = await db + .select({ checkoutRunId: issues.checkoutRunId, executionRunId: issues.executionRunId }) + .from(issues) + .where(eq(issues.id, issueId)) + .then((rows) => rows[0]); + expect(lock).toEqual({ checkoutRunId: null, executionRunId: null }); + + // The audit write failed, so no run event exists for this run. + const events = await db + .select({ id: heartbeatRunEvents.id }) + .from(heartbeatRunEvents) + .where(eq(heartbeatRunEvents.runId, runningRunId)); + expect(events).toEqual([]); + }); }); diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 6941b33ba8f..11e9e247255 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -8825,11 +8825,25 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) runId: string, status: string, patch?: Partial, + ) { + return setRunStatusFromLive(runId, status, ["running"], patch); + } + + // Move a run to a new status only when its current status is one of + // `fromStatuses`. The compare-and-set is a single conditional update, so a + // concurrent path can win the race. When this update matches nothing, the + // function reads the current row and reports updated=false, so the caller can + // keep the terminal outcome that another path already wrote. + async function setRunStatusFromLive( + runId: string, + status: string, + fromStatuses: string[], + patch?: Partial, ) { const updated = await db .update(heartbeatRuns) .set({ status, ...patch, updatedAt: new Date() }) - .where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running"))) + .where(and(eq(heartbeatRuns.id, runId), inArray(heartbeatRuns.status, fromStatuses))) .returning() .then((rows) => rows[0] ?? null); @@ -8855,6 +8869,75 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) return { run: current, updated: false as const }; } + // Invariant: when a run releases its environment lease, the run row must be + // terminal. The finalizer writes the terminal status in a step that is + // separate from the agent status=done PATCH. If the sandbox or the run + // process stops between the two steps, heartbeat_runs.status stays "running". + // The UI reads liveness from that row, so a finished task shows "Live" + // forever. This function closes the gap in the run teardown: when the run is + // still running or queued, it forces a terminal status before the lease is + // released. It never overwrites a status that another path already made + // terminal. + async function terminalizeRunOnLeaseRelease( + run: typeof heartbeatRuns.$inferSelect, + ): Promise { + if (isHeartbeatRunTerminalStatus(run.status)) return run; + if (run.status !== "running" && run.status !== "queued") return run; + + // Choose the terminal status that reflects the true outcome. When the issue + // already reached a terminal status, the run reached its goal, so use the + // matching terminal run status. Otherwise the teardown cut the run short, + // so use "interrupted". + const issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId); + let terminalStatus: "succeeded" | "cancelled" | "interrupted" = "interrupted"; + if (issueId) { + const issueStatus = await db + .select({ status: issues.status }) + .from(issues) + .where(eq(issues.id, issueId)) + .then((rows) => rows[0]?.status ?? null); + if (issueStatus === "done") terminalStatus = "succeeded"; + else if (issueStatus === "cancelled") terminalStatus = "cancelled"; + } + + const message = + `run terminalized on environment lease release: heartbeat_runs.status was still ${run.status} at teardown`; + // Match both "running" and "queued". A queued run has released its lease but + // never reached "running", so a running-only update would miss it and leave + // a phantom live run behind. + const write = await setRunStatusFromLive(run.id, terminalStatus, ["running", "queued"], { + finishedAt: run.finishedAt ?? new Date(), + error: run.error ?? (terminalStatus === "interrupted" ? message : null), + errorCode: run.errorCode ?? (terminalStatus === "interrupted" ? "lease_released_before_terminal" : null), + }); + if (!write.updated) { + // Another path already finalized the run. Keep that terminal outcome. + return write.run ?? run; + } + + const terminalRun = write.run; + if (terminalRun) { + await appendRunEvent(terminalRun, await nextRunEventSeq(terminalRun.id), { + eventType: "lifecycle", + stream: "system", + level: terminalStatus === "interrupted" ? "warn" : "info", + message, + payload: { + previousStatus: run.status, + terminalStatus, + reason: "environment_lease_release", + ...(issueId ? { issueId } : {}), + }, + }).catch((eventErr) => { + logger.warn( + { err: eventErr, runId: run.id }, + "failed to append run event for lease-release terminalization", + ); + }); + } + return terminalRun ?? run; + } + function publishRunLifecyclePluginEvent(run: typeof heartbeatRuns.$inferSelect) { const eventType = run.status === "running" @@ -16193,7 +16276,20 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) } } } finally { - const latestRun = await getRun(run.id).catch(() => null); + let latestRun = await getRun(run.id).catch(() => null); + // Close the invariant "environment lease released implies the run is + // terminal". When the teardown reaches this point with the run still + // running or queued, force a terminal status before the lease is + // released, so the UI never shows a finished task as "Live". + if (latestRun) { + latestRun = await terminalizeRunOnLeaseRelease(latestRun).catch((terminalizeErr) => { + logger.error( + { err: terminalizeErr, runId: run.id }, + "failed to terminalize run before environment lease release", + ); + return latestRun; + }); + } await releaseEnvironmentLeasesForRun({ runId: run.id, companyId: run.companyId, @@ -18935,6 +19031,8 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) reconcileStrandedAssignedIssues, + terminalizeRunOnLeaseRelease, + sweepStaleIssueLocks, buildIssueGraphLivenessAutoRecoveryPreview, diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index f9545fa66b6..c0f2a5e0326 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -5477,22 +5477,178 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) return Math.max(1, Math.floor(asNumber(raw, fallback))); } + // Backstop reconciler: terminalizes a "running" run that can no longer reach a + // terminal status on its own. The run finalizer writes the terminal status in + // a step that is separate from the agent status=done PATCH. When the teardown + // stops between the two steps, heartbeat_runs.status stays "running" forever. + // The UI reads liveness from that row, so the task shows "Live" forever. This + // function forces the run to a terminal status and records a run event, so the + // state is auditable. It never overwrites a status that another path already + // made terminal. + // + // Two independent authorities terminalize the run. Either one is enough: + // + // - Issue-terminal authority: the run's issue already reached a terminal + // status (done or cancelled), but the run row is still "running". A healthy + // run always terminalizes its own row before or just after the issue reaches + // a terminal status, so a lasting "running" row under a terminal issue is + // orphaned. This authority does not depend on process death. It is the only + // authority that catches the reuse-lease path: the release stops the sandbox + // but keeps the server process alive, so the in-memory handle and the + // recorded pid can both persist. + // - Process-death authority: the run has no in-memory handle and its recorded + // process and process group are both gone. This catches a hard server crash + // that skipped the graceful teardown, even when the issue is not terminal. + async function terminalizeOrphanedRunningRun( + run: typeof heartbeatRuns.$inferSelect, + options?: { + // The terminal run status implied by a referencing issue. The caller + // passes it when it already knows the issue that holds the run in a lock + // column. It maps issue "done" to "succeeded" and issue "cancelled" to + // "cancelled". A null value means the referencing issue is not terminal. + referencingIssueTerminalStatus?: "succeeded" | "cancelled" | null; + // True when an active (non-terminal) issue still holds this run in a lock + // column. The run is live for that active issue, so the caller forbids the + // issue-terminal authority. This flag also suppresses the context-snapshot + // fallback below. Without it, a terminal issue named in the run context + // snapshot would still terminalize the shared run and defeat the guard. + runReferencedByActiveIssue?: boolean; + }, + ): Promise<{ terminalized: boolean; status: string }> { + // Act only on a run in "running" status. A "queued" run has no process yet, + // and a "scheduled_retry" run has no process on purpose because it waits to + // retry. Neither is orphaned, so this function must not terminalize them. + if (run.status !== "running") return { terminalized: false, status: run.status }; + + const pid = run.processPid ?? null; + const processGroupId = run.processGroupId ?? null; + + // Issue-terminal authority. When the run's issue is terminal, the run row is + // orphaned regardless of process or handle state. Prefer the referencing + // issue status that the caller passed, because a lock column is the direct + // link from the stuck "Live" issue to this run. Fall back to the issue id in + // the run context snapshot when the caller passed nothing. Skip the fallback + // when an active issue still references the run. The run is live for that + // active issue, so a terminal issue named in the context snapshot must not + // terminalize it. + let issueTerminalStatus: "succeeded" | "cancelled" | null = + options?.referencingIssueTerminalStatus ?? null; + const issueId = issueIdFromRunContext(run.contextSnapshot); + if (!issueTerminalStatus && !options?.runReferencedByActiveIssue && issueId) { + const issueStatus = await db + .select({ status: issues.status }) + .from(issues) + .where(eq(issues.id, issueId)) + .then((rows) => rows[0]?.status ?? null); + if (issueStatus === "done") issueTerminalStatus = "succeeded"; + else if (issueStatus === "cancelled") issueTerminalStatus = "cancelled"; + } + + // Process-death authority. The run is live only when a process still backs + // it. Check the in-memory handle first, then the recorded pid and process + // group. Require recorded process metadata, so this authority never fires on + // a run that has not yet stored its pid. + let processGone = false; + if (!runningProcesses.get(run.id)) { + if (typeof pid === "number" || typeof processGroupId === "number") { + const processAlive = + (typeof pid === "number" && isPidAlive(pid)) || + (typeof processGroupId === "number" && isProcessGroupAlive(processGroupId)); + processGone = !processAlive; + } + } + + // Neither authority applies. The run is still live, so leave it alone. + if (!issueTerminalStatus && !processGone) { + return { terminalized: false, status: run.status }; + } + + const authority = issueTerminalStatus ? "issue_terminal" : "process_gone"; + const terminalStatus = issueTerminalStatus ?? "interrupted"; + const errorCode = issueTerminalStatus + ? "orphaned_running_run_issue_terminal" + : "orphaned_running_run"; + const message = + authority === "issue_terminal" + ? "run terminalized by recovery backstop: issue reached a terminal status while heartbeat_runs.status stayed live" + : "run terminalized by recovery backstop: process and sandbox gone while heartbeat_runs.status stayed live"; + + const now = new Date(); + const updated = await db + .update(heartbeatRuns) + .set({ + status: terminalStatus, + finishedAt: run.finishedAt ?? now, + error: run.error ?? (terminalStatus === "interrupted" ? message : null), + errorCode: run.errorCode ?? (terminalStatus === "interrupted" ? errorCode : null), + updatedAt: now, + }) + .where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "running"))) + .returning() + .then((rows) => rows[0] ?? null); + if (!updated) { + // Another path finalized the run between the read and this write. Keep + // that terminal outcome authoritative. + const [current] = await db + .select({ status: heartbeatRuns.status }) + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, run.id)); + return { terminalized: false, status: current?.status ?? run.status }; + } + + runningProcesses.delete(run.id); + // The run update above already committed the terminal status. The audit + // event is best-effort: if the insert fails, the caller must still treat + // the run as terminalized and clear the lock in the same sweep. So catch + // the failure, log it, and continue. A thrown error here would abort the + // sweep and leave the stale lock in place. + try { + await appendRecoveryRunEvent(updated, { + level: "warn", + message, + payload: { + source: "recovery.sweep_stale_issue_locks", + authority, + previousStatus: run.status, + terminalStatus, + ...(issueId ? { issueId } : {}), + pid, + processGroupId, + }, + }); + } catch (error) { + logger.error( + { err: error, runId: run.id, previousStatus: run.status }, + "failed to append recovery run event after terminalizing orphaned run; run stays terminal and the sweep clears the lock", + ); + } + logger.warn( + { runId: run.id, authority, previousStatus: run.status, terminalStatus, issueId, pid, processGroupId }, + "terminalized orphaned running heartbeat run in stale-lock sweep", + ); + return { terminalized: true, status: updated.status }; + } + // Backstop sweeper: clears stale lock columns on issues whose checkoutRunId // or executionRunId points at a heartbeat_runs row that is either missing or // in a terminal status. Provides self-heal for stale locks that fell outside // releaseIssueExecutionAndPromote / clearCheckoutRunIfTerminal / adoption. - // Idempotent and safe: clears at most one row's worth of lock columns per - // candidate, and only when the referenced run row is unambiguously terminal. + // Before it evaluates cleanability, it terminalizes any referenced run that + // still claims to be live but can no longer reach a terminal status on its + // own, so a stuck "running" run can no longer block the sweep. Idempotent and + // safe: clears at most one row's worth of lock columns per candidate. async function sweepStaleIssueLocks() { const result = { cleared: 0, issueIds: [] as string[], + terminalizedRunIds: [] as string[], }; const candidates = await db .select({ id: issues.id, companyId: issues.companyId, + status: issues.status, checkoutRunId: issues.checkoutRunId, executionRunId: issues.executionRunId, }) @@ -5511,13 +5667,61 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) const runRows = referencedRunIds.length > 0 ? await db - .select({ id: heartbeatRuns.id, status: heartbeatRuns.status }) + .select() .from(heartbeatRuns) .where(inArray(heartbeatRuns.id, referencedRunIds)) : []; const runStatusById = new Map(); for (const row of runRows) runStatusById.set(row.id, row.status); + // Collect the runs that a non-terminal issue still references. Such a run is + // the live run of an active issue. A different, terminal issue can also hold + // the same run id in a stale lock column. The terminal reference alone must + // not terminalize a run that an active issue still owns, so exclude these + // runs from the issue-terminal authority below. + const runIdsReferencedByActiveIssue = new Set(); + for (const issue of candidates) { + if (issue.status === "done" || issue.status === "cancelled") continue; + for (const runId of [issue.checkoutRunId, issue.executionRunId]) { + if (runId) runIdsReferencedByActiveIssue.add(runId); + } + } + + // Map each referenced run to the terminal run status implied by its + // referencing issue. When a terminal issue still holds the run in a lock + // column, that run is orphaned: the issue is the stuck "Live" task the UI + // shows. A "done" issue implies "succeeded"; a "cancelled" issue implies + // "cancelled". Skip a run that an active issue also references, because that + // run is still live for the active issue. + const issueTerminalStatusByRunId = new Map(); + for (const issue of candidates) { + const implied = + issue.status === "done" + ? "succeeded" + : issue.status === "cancelled" + ? "cancelled" + : null; + if (!implied) continue; + for (const runId of [issue.checkoutRunId, issue.executionRunId]) { + if (runId && !runIdsReferencedByActiveIssue.has(runId)) { + issueTerminalStatusByRunId.set(runId, implied); + } + } + } + + // Pre-pass: terminalize any referenced run that still claims to be live but + // can no longer reach a terminal status on its own. This lets the sweep + // clear the lock in the same pass instead of waiting for the run to reach a + // terminal status by another route. + for (const row of runRows) { + const outcome = await terminalizeOrphanedRunningRun(row, { + referencingIssueTerminalStatus: issueTerminalStatusByRunId.get(row.id) ?? null, + runReferencedByActiveIssue: runIdsReferencedByActiveIssue.has(row.id), + }); + runStatusById.set(row.id, outcome.status); + if (outcome.terminalized) result.terminalizedRunIds.push(row.id); + } + const isCleanable = (runId: string | null) => { if (!runId) return true; const status = runStatusById.get(runId); @@ -5576,9 +5780,13 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) }); } - if (result.cleared > 0) { + if (result.cleared > 0 || result.terminalizedRunIds.length > 0) { logger.warn( - { cleared: result.cleared, issueIds: result.issueIds }, + { + cleared: result.cleared, + issueIds: result.issueIds, + terminalizedRunIds: result.terminalizedRunIds, + }, "swept stale issue lock columns", ); } diff --git a/ui/src/components/IssueChatThread.tsx b/ui/src/components/IssueChatThread.tsx index c6af475e502..729b64bd36b 100644 --- a/ui/src/components/IssueChatThread.tsx +++ b/ui/src/components/IssueChatThread.tsx @@ -71,6 +71,7 @@ import type { SuggestTasksInteraction, } from "../lib/issue-thread-interactions"; import { buildIssueThreadInteractionSummary, isIssueThreadInteraction } from "../lib/issue-thread-interactions"; +import { isLiveIssueRun } from "../lib/liveIssueIds"; import { resolveIssueChatTranscriptRuns } from "../lib/issueChatTranscriptRuns"; import { formatTimelineWorkspaceLabel, @@ -4438,9 +4439,10 @@ export function IssueChatThread({ const displayLiveRuns = useMemo(() => { const deduped = new Map(); for (const run of liveRuns) { + if (!isLiveIssueRun(run, issueStatus)) continue; deduped.set(run.id, run); } - if (activeRun) { + if (activeRun && isLiveIssueRun(activeRun, issueStatus)) { deduped.set(activeRun.id, { id: activeRun.id, status: activeRun.status, @@ -4471,7 +4473,7 @@ export function IssueChatThread({ }); } return [...deduped.values()].sort((a, b) => new Date(a.createdAt).getTime() - new Date(b.createdAt).getTime()); - }, [activeRun, liveRuns]); + }, [activeRun, issueStatus, liveRuns]); const transcriptRuns = useMemo(() => { return resolveIssueChatTranscriptRuns({ linkedRuns, @@ -4489,8 +4491,8 @@ export function IssueChatThread({ return ids; }, [displayLiveRuns]); const hasActiveRun = useMemo( - () => displayLiveRuns.some((run) => run.status === "running") || activeRun?.status === "running", - [displayLiveRuns, activeRun], + () => displayLiveRuns.some((run) => run.status === "running"), + [displayLiveRuns], ); // Real-time view of the handoff: a run that starts after the issue payload // was fetched must quiet the missing-disposition warnings without waiting @@ -4536,6 +4538,7 @@ export function IssueChatThread({ agentMap, currentUserId, userLabelMap, + issueStatus, }), [ comments, @@ -4552,6 +4555,7 @@ export function IssueChatThread({ agentMap, currentUserId, userLabelMap, + issueStatus, ], ); const stableMessagesRef = useRef([]); diff --git a/ui/src/components/TaskChatThread.tsx b/ui/src/components/TaskChatThread.tsx index 739da913434..27bfa3fb348 100644 --- a/ui/src/components/TaskChatThread.tsx +++ b/ui/src/components/TaskChatThread.tsx @@ -32,6 +32,7 @@ import { useSidebar } from "@/context/SidebarContext"; import { cn } from "@/lib/utils"; import { useIssuePlanDocument } from "@/hooks/useIssuePlanDocument"; import { latestSameRunHandoffTimestamp, type IssueChatComment } from "@/lib/issue-chat-messages"; +import { isLiveIssueRun, isTerminalIssueStatus } from "@/lib/liveIssueIds"; import { workModeInEffectAt } from "@/lib/issue-timeline-events"; import { workModeMetaFor } from "@/lib/work-mode-meta"; @@ -184,9 +185,10 @@ export function TaskChatThread(props: TaskChatThreadProps) { // The single in-flight run whose turn we stream live (non-terminal). const liveRun = useMemo(() => { - if (activeRun && !isTerminalRunStatus(activeRun.status)) return activeRun; - return (liveRuns ?? []).find((r) => !isTerminalRunStatus(r.status)) ?? null; - }, [activeRun, liveRuns]); + if (isTerminalIssueStatus(issueStatus)) return null; + if (activeRun && isLiveIssueRun(activeRun, issueStatus)) return activeRun; + return (liveRuns ?? []).find((r) => isLiveIssueRun(r, issueStatus)) ?? null; + }, [activeRun, issueStatus, liveRuns]); // Runs observed non-terminal while mounted: their turns ANIMATE the fold when // they settle. Runs already terminal at mount collapse instantly. diff --git a/ui/src/lib/issue-chat-messages.test.ts b/ui/src/lib/issue-chat-messages.test.ts index 365b985b7bf..169da08f5f9 100644 --- a/ui/src/lib/issue-chat-messages.test.ts +++ b/ui/src/lib/issue-chat-messages.test.ts @@ -624,6 +624,44 @@ describe("buildIssueChatMessages", () => { }); }); + it("suppresses live-run Working messages for terminal issues", () => { + const liveRun: LiveRunForIssue = { + id: "run-live-terminal", + status: "running", + invocationSource: "manual", + triggerDetail: null, + startedAt: "2026-04-06T12:04:00.000Z", + finishedAt: null, + createdAt: "2026-04-06T12:04:00.000Z", + agentId: "agent-1", + agentName: "CodexCoder", + adapterType: "codex_local", + }; + + const terminalMessages = buildIssueChatMessages({ + comments: [], + timelineEvents: [], + linkedRuns: [], + liveRuns: [liveRun], + issueStatus: "done", + currentUserId: "user-1", + }); + const liveMessages = buildIssueChatMessages({ + comments: [], + timelineEvents: [], + linkedRuns: [], + liveRuns: [liveRun], + issueStatus: "in_progress", + currentUserId: "user-1", + }); + + expect(terminalMessages.find((message) => message.id === "run-assistant:run-live-terminal")).toBeUndefined(); + expect(liveMessages.find((message) => message.id === "run-assistant:run-live-terminal")).toMatchObject({ + status: { type: "running" }, + metadata: { custom: { waitingText: "Working..." } }, + }); + }); + it("merges thread interactions into the same chronological feed as comments and runs", () => { const messages = buildIssueChatMessages({ comments: [ diff --git a/ui/src/lib/issue-chat-messages.ts b/ui/src/lib/issue-chat-messages.ts index 72462011834..84ccc7cb171 100644 --- a/ui/src/lib/issue-chat-messages.ts +++ b/ui/src/lib/issue-chat-messages.ts @@ -16,6 +16,7 @@ import { type IssueThreadInteraction, } from "./issue-thread-interactions"; import type { IssueTimelineEvent } from "./issue-timeline-events"; +import { isLiveIssueRun } from "./liveIssueIds"; import { summarizeNotice, } from "./transcriptPresentation"; @@ -954,13 +955,15 @@ export function buildAssistantPartsFromTranscript(entries: readonly IssueChatTra function normalizeLiveRuns( liveRuns: readonly LiveRunForIssue[], activeRun: ActiveRunForIssue | null | undefined, - issueId?: string, + issueId: string | undefined, + issueStatus: string | null | undefined, ) { const deduped = new Map(); for (const run of liveRuns) { + if (!isLiveIssueRun(run, issueStatus)) continue; deduped.set(run.id, run); } - if (activeRun) { + if (activeRun && isLiveIssueRun(activeRun, issueStatus)) { deduped.set(activeRun.id, { id: activeRun.id, status: activeRun.status, @@ -1056,6 +1059,7 @@ export function buildIssueChatMessages(args: { agentMap?: Map; currentUserId?: string | null; userLabelMap?: ReadonlyMap | null; + issueStatus?: string | null; }) { const { comments, @@ -1073,6 +1077,7 @@ export function buildIssueChatMessages(args: { agentMap, currentUserId, userLabelMap, + issueStatus, } = args; const orderedMessages: MessageWithOrder[] = []; @@ -1139,7 +1144,7 @@ export function buildIssueChatMessages(args: { }); } - for (const run of normalizeLiveRuns(liveRuns, activeRun, issueId)) { + for (const run of normalizeLiveRuns(liveRuns, activeRun, issueId, issueStatus)) { orderedMessages.push({ createdAtMs: toTimestamp(run.startedAt ?? run.createdAt), order: 3, diff --git a/ui/src/lib/liveIssueIds.test.ts b/ui/src/lib/liveIssueIds.test.ts index aa9a652b8d6..06bd8e025c3 100644 --- a/ui/src/lib/liveIssueIds.test.ts +++ b/ui/src/lib/liveIssueIds.test.ts @@ -2,6 +2,23 @@ import { describe, expect, it } from "vitest"; import type { LiveRunForIssue } from "../api/heartbeats"; import { collectLiveIssueIds, collectSubtreeLiveCounts } from "./liveIssueIds"; +function liveRun(overrides: Partial): LiveRunForIssue { + return { + id: "run", + status: "running", + invocationSource: "scheduler", + triggerDetail: null, + startedAt: "2026-04-20T10:00:00.000Z", + finishedAt: null, + createdAt: "2026-04-20T10:00:00.000Z", + agentId: "agent", + agentName: "Agent", + adapterType: "codex_local", + issueId: "issue", + ...overrides, + }; +} + describe("collectLiveIssueIds", () => { it("keeps only runs linked to issues", () => { const liveRuns: LiveRunForIssue[] = [ @@ -74,6 +91,72 @@ describe("collectLiveIssueIds", () => { expect([...collectLiveIssueIds(liveRuns)]).toEqual(["issue-1", "issue-2"]); }); + + it("suppresses live ids for terminal issues while keeping non-terminal issues live", () => { + const liveRuns: LiveRunForIssue[] = [ + { + id: "run-terminal", + status: "running", + invocationSource: "scheduler", + triggerDetail: null, + startedAt: "2026-04-20T10:00:00.000Z", + finishedAt: null, + createdAt: "2026-04-20T10:00:00.000Z", + agentId: "agent-1", + agentName: "Coder", + adapterType: "codex_local", + issueId: "issue-done", + }, + { + id: "run-live", + status: "queued", + invocationSource: "scheduler", + triggerDetail: null, + startedAt: null, + finishedAt: null, + createdAt: "2026-04-20T10:01:00.000Z", + agentId: "agent-2", + agentName: "Builder", + adapterType: "codex_local", + issueId: "issue-open", + }, + ]; + + expect([...collectLiveIssueIds(liveRuns, [ + { id: "issue-done", status: "done" }, + { id: "issue-open", status: "in_progress" }, + ])]).toEqual(["issue-open"]); + }); + + it("keeps newer terminal snapshots authoritative when stale non-terminal snapshots appear later", () => { + const liveRuns: LiveRunForIssue[] = [ + liveRun({ id: "run-done", issueId: "issue-done", status: "running" }), + liveRun({ id: "run-cancelled", issueId: "issue-cancelled", status: "queued" }), + liveRun({ id: "run-open", issueId: "issue-open", status: "running" }), + ]; + + expect([...collectLiveIssueIds(liveRuns, [ + { id: "issue-done", status: "done", updatedAt: "2026-04-20T10:02:00.000Z" }, + { id: "issue-cancelled", status: "cancelled", updatedAt: "2026-04-20T10:02:00.000Z" }, + { id: "issue-open", status: "in_progress", updatedAt: "2026-04-20T10:02:00.000Z" }, + { id: "issue-done", status: "in_progress", updatedAt: "2026-04-20T10:01:00.000Z" }, + { id: "issue-cancelled", status: "todo", updatedAt: "2026-04-20T10:01:00.000Z" }, + ])]).toEqual(["issue-open"]); + }); + + it("allows a newer non-terminal snapshot to reopen an issue with a stale terminal snapshot", () => { + const liveRuns: LiveRunForIssue[] = [ + liveRun({ id: "run-reopened", issueId: "issue-reopened", status: "running" }), + liveRun({ id: "run-terminal", issueId: "issue-terminal", status: "queued" }), + ]; + + expect([...collectLiveIssueIds(liveRuns, [ + { id: "issue-reopened", status: "done", updatedAt: "2026-04-20T10:01:00.000Z" }, + { id: "issue-terminal", status: "in_progress", updatedAt: "2026-04-20T10:01:00.000Z" }, + { id: "issue-reopened", status: "in_progress", updatedAt: "2026-04-20T10:02:00.000Z" }, + { id: "issue-terminal", status: "done", updatedAt: "2026-04-20T10:02:00.000Z" }, + ])]).toEqual(["issue-reopened"]); + }); }); describe("collectSubtreeLiveCounts", () => { diff --git a/ui/src/lib/liveIssueIds.ts b/ui/src/lib/liveIssueIds.ts index 07f4a75222e..a8c83ff9de7 100644 --- a/ui/src/lib/liveIssueIds.ts +++ b/ui/src/lib/liveIssueIds.ts @@ -1,13 +1,71 @@ +import type { IssueStatus } from "@paperclipai/shared"; import type { LiveRunForIssue } from "../api/heartbeats"; function isLiveRunStatus(status: string): boolean { return status === "queued" || status === "running"; } -export function collectLiveIssueIds(liveRuns: readonly LiveRunForIssue[] | null | undefined): Set { +const TERMINAL_ISSUE_STATUSES = new Set(["done", "cancelled"]); + +export function isTerminalIssueStatus(status: string | null | undefined): status is IssueStatus { + return TERMINAL_ISSUE_STATUSES.has(status as IssueStatus); +} + +export function isLiveIssueRun( + run: Pick, + issueStatus?: string | null, +): boolean { + return isLiveRunStatus(run.status) && !isTerminalIssueStatus(issueStatus); +} + +export interface LiveIssueStatusNode { + id: string; + status: IssueStatus | string; + updatedAt?: Date | string | number | null; +} + +function collectIssueStatusById(issues: readonly LiveIssueStatusNode[] | null | undefined): Map { + const snapshotByIssueId = new Map(); + for (const issue of issues ?? []) { + const candidate = { + status: issue.status, + updatedAtMs: issueUpdatedAtMs(issue.updatedAt), + }; + const existing = snapshotByIssueId.get(issue.id); + if (!existing || shouldReplaceIssueStatusSnapshot(existing, candidate)) snapshotByIssueId.set(issue.id, candidate); + } + return new Map([...snapshotByIssueId].map(([issueId, snapshot]) => [issueId, snapshot.status])); +} + +function issueUpdatedAtMs(updatedAt: LiveIssueStatusNode["updatedAt"]): number | null { + if (updatedAt === null || updatedAt === undefined) return null; + const timestamp = updatedAt instanceof Date ? updatedAt.getTime() : new Date(updatedAt).getTime(); + return Number.isFinite(timestamp) ? timestamp : null; +} + +function shouldReplaceIssueStatusSnapshot( + existing: { status: string; updatedAtMs: number | null }, + candidate: { status: string; updatedAtMs: number | null }, +): boolean { + if (candidate.updatedAtMs !== null && existing.updatedAtMs !== null) { + if (candidate.updatedAtMs !== existing.updatedAtMs) return candidate.updatedAtMs > existing.updatedAtMs; + } else if (candidate.updatedAtMs !== null) { + return true; + } else if (existing.updatedAtMs !== null) { + return false; + } + + return !isTerminalIssueStatus(existing.status) && isTerminalIssueStatus(candidate.status); +} + +export function collectLiveIssueIds( + liveRuns: readonly LiveRunForIssue[] | null | undefined, + issues?: readonly LiveIssueStatusNode[] | null, +): Set { const ids = new Set(); + const statusByIssueId = collectIssueStatusById(issues); for (const run of liveRuns ?? []) { - if (run.issueId && isLiveRunStatus(run.status)) ids.add(run.issueId); + if (run.issueId && isLiveIssueRun(run, statusByIssueId.get(run.issueId))) ids.add(run.issueId); } return ids; } diff --git a/ui/src/pages/ExecutionWorkspaceDetail.tsx b/ui/src/pages/ExecutionWorkspaceDetail.tsx index fa2b5a3abab..2cc119a5422 100644 --- a/ui/src/pages/ExecutionWorkspaceDetail.tsx +++ b/ui/src/pages/ExecutionWorkspaceDetail.tsx @@ -542,7 +542,7 @@ function ExecutionWorkspaceIssuesList({ }); usePublishSharedQueryData(sharedLiveRuns, liveRuns, liveRunsUpdatedAt); - const liveIssueIds = useMemo(() => collectLiveIssueIds(liveRuns), [liveRuns]); + const liveIssueIds = useMemo(() => collectLiveIssueIds(liveRuns, issues), [issues, liveRuns]); const updateIssue = useMutation({ mutationFn: ({ id, data }: { id: string; data: Record }) => issuesApi.update(id, data), diff --git a/ui/src/pages/Inbox.tsx b/ui/src/pages/Inbox.tsx index c86873fe70d..e9abc46966e 100644 --- a/ui/src/pages/Inbox.tsx +++ b/ui/src/pages/Inbox.tsx @@ -945,7 +945,6 @@ export function Inbox() { refetchInterval: sharedLiveRuns.refetchInterval, }); usePublishSharedQueryData(sharedLiveRuns, liveRuns, liveRunsUpdatedAt); - const liveIssueIds = useMemo(() => collectLiveIssueIds(liveRuns), [liveRuns]); const { data: companyMembers } = useQuery({ queryKey: queryKeys.access.companyUserDirectory(selectedCompanyId!), queryFn: () => accessApi.listUserDirectory(selectedCompanyId!), @@ -1001,6 +1000,15 @@ export function Inbox() { enabled: shouldUseIssueSearchSupplement, placeholderData: (previousData) => previousData, }); + const liveIssueIds = useMemo( + () => collectLiveIssueIds(liveRuns, [ + ...(issues ?? []), + ...mineIssuesRaw, + ...touchedIssuesRaw, + ...remoteIssueSearchResults, + ]), + [issues, liveRuns, mineIssuesRaw, remoteIssueSearchResults, touchedIssuesRaw], + ); const inboxIssueIdsForExternalObjectSummaries = useMemo(() => { const issueIds = new Set(); for (const issue of mineIssues) issueIds.add(issue.id); diff --git a/ui/src/pages/IssueDetail.tsx b/ui/src/pages/IssueDetail.tsx index 1c632986e40..f599c75365b 100644 --- a/ui/src/pages/IssueDetail.tsx +++ b/ui/src/pages/IssueDetail.tsx @@ -1980,7 +1980,10 @@ export function IssueDetail() { }, [issue?.id, rawChildIssues], ); - const liveIssueIds = useMemo(() => collectLiveIssueIds(companyLiveRuns), [companyLiveRuns]); + const liveIssueIds = useMemo( + () => collectLiveIssueIds(companyLiveRuns, issue ? [issue, ...childIssues] : childIssues), + [childIssues, companyLiveRuns, issue], + ); const issuePanelKey = useMemo( () => buildIssuePropertiesPanelKey(issue ?? null, childIssues), [childIssues, issue], diff --git a/ui/src/pages/Issues.tsx b/ui/src/pages/Issues.tsx index 0a2ce34f521..02bcddd5f18 100644 --- a/ui/src/pages/Issues.tsx +++ b/ui/src/pages/Issues.tsx @@ -115,8 +115,6 @@ export function Issues() { }); usePublishSharedQueryData(sharedLiveRuns, liveRuns, liveRunsUpdatedAt); - const liveIssueIds = useMemo(() => collectLiveIssueIds(liveRuns), [liveRuns]); - const issueLinkState = useMemo( () => createIssueDetailLocationState( @@ -169,6 +167,7 @@ export function Issues() { }); const issues = useMemo(() => mergeIssuePagesStable(issuePages?.pages ?? []) as Issue[], [issuePages]); + const liveIssueIds = useMemo(() => collectLiveIssueIds(liveRuns, issues), [issues, liveRuns]); const hasMoreServerIssues = syncedSearch.trim().length === 0 && hasNextPage === true; const loadMoreServerIssues = useCallback(() => { diff --git a/ui/src/pages/ProjectDetail.tsx b/ui/src/pages/ProjectDetail.tsx index 5a0fbe47db8..7041a4a7926 100644 --- a/ui/src/pages/ProjectDetail.tsx +++ b/ui/src/pages/ProjectDetail.tsx @@ -259,13 +259,12 @@ function ProjectIssuesList({ projectId, companyId }: { projectId: string; compan enabled: !!companyId, }); - const liveIssueIds = useMemo(() => collectLiveIssueIds(liveRuns), [liveRuns]); - const { data: issues, isLoading, error } = useQuery({ queryKey: queryKeys.issues.listByProject(companyId, projectId), queryFn: () => issuesApi.list(companyId, { projectId }), enabled: !!companyId, }); + const liveIssueIds = useMemo(() => collectLiveIssueIds(liveRuns, issues), [issues, liveRuns]); const updateIssue = useMutation({ mutationFn: ({ id, data }: { id: string; data: Record }) => @@ -330,13 +329,12 @@ function ProjectPluginOperationsList({ refetchInterval: sharedLiveRuns.refetchInterval, }); usePublishSharedQueryData(sharedLiveRuns, liveRuns, liveRunsUpdatedAt); - const liveIssueIds = useMemo(() => collectLiveIssueIds(liveRuns), [liveRuns]); - const { data: issues, isLoading, error } = useQuery({ queryKey: queryKeys.issues.listPluginOperationsByProject(companyId, projectId, originKindPrefix), queryFn: () => issuesApi.list(companyId, { projectId, originKindPrefix }), enabled: !!companyId && !!projectId, }); + const liveIssueIds = useMemo(() => collectLiveIssueIds(liveRuns, issues), [issues, liveRuns]); const updateIssue = useMutation({ mutationFn: ({ id, data }: { id: string; data: Record }) => diff --git a/ui/src/pages/Routines.tsx b/ui/src/pages/Routines.tsx index 213396320ed..4ec6f105038 100644 --- a/ui/src/pages/Routines.tsx +++ b/ui/src/pages/Routines.tsx @@ -655,7 +655,7 @@ export function Routines() { () => new Map((routineFolders?.folders ?? []).map((folder) => [folder.id, folder])), [routineFolders], ); - const liveIssueIds = useMemo(() => collectLiveIssueIds(liveRuns), [liveRuns]); + const liveIssueIds = useMemo(() => collectLiveIssueIds(liveRuns, routineExecutionIssues), [liveRuns, routineExecutionIssues]); const visibleRoutines = useMemo( () => (routines ?? []).filter((routine) => routine.status !== "archived"), [routines], diff --git a/ui/src/plugins/bridge-init.ts b/ui/src/plugins/bridge-init.ts index 819287024df..81800a041c4 100644 --- a/ui/src/plugins/bridge-init.ts +++ b/ui/src/plugins/bridge-init.ts @@ -289,13 +289,12 @@ function PluginSdkIssuesList({ refetchInterval: sharedLiveRuns.refetchInterval, }); usePublishSharedQueryData(sharedLiveRuns, liveRuns, liveRunsUpdatedAt); - const liveIssueIds = useMemo(() => collectLiveIssueIds(liveRuns), [liveRuns]); - const { data: issues, isLoading, error } = useQuery({ queryKey: issuesQueryKey, queryFn: () => issuesApi.list(companyId!, issueFilters), enabled: !!companyId, }); + const liveIssueIds = useMemo(() => collectLiveIssueIds(liveRuns, issues), [issues, liveRuns]); const updateIssue = useMutation({ mutationFn: ({ id, data }: { id: string; data: Record }) => From 9abb600e721059ec2d36aff73ed940ace4a98da8 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 7 Aug 2026 10:52:04 -0500 Subject: [PATCH 002/201] docs(connections): MCP-direct/DCR playbook section + Notion dry-run appendix (#11030) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Apps give agents governed access to external tools through catalog connectors. > - The connector playbook is the repeatable template for adding such connectors. > - Notion just shipped as the first MCP-direct connector with RFC 7591 dynamic client registration (#11009). > - The playbook had no guidance for MCP-direct connections, OAuth discovery, or DCR. > - This pull request documents that path and encodes three mandatory standards into the connector template. > - It also adds a Notion dry-run appendix recorded from the live probe and the shipped implementation. > - The benefit is that the next MCP-direct connector follows a recorded, verified path. > [!IMPORTANT] > **Depends on #11009.** This PR documents the Notion MCP connector that ships in #11009 (RFC 7591 DCR, discovery-first endpoints, `redirectConstraints` enforcement, refresh-rotation hardening). Reviewing this doc against master before #11009 merges will show the documented behavior as "not implemented" — that is expected. Draft until #11009 lands, then re-review. ## What Changed `doc/connections/CONNECTOR-PLAYBOOK.md` only (+364 lines, no code): - New **"MCP-Direct Connections"** section: RFC 9728/8414 endpoint discovery chain, an **RFC 7591 dynamic client registration** subsection (public client, PKCE S256, env-client precedence, persist-and-reuse), and a **redirect-URI constraints** subsection (`https-or-loopback-http`, fail-fast wizard error). - Three mandatory documentation standards encoded into the connector template itself: a service-involvement statement (DCR providers need neither Paperclip ID nor Paperclip Connect — instance-local per the PAP-14828 spec §10 item 8.4; cloud and self-hosted use the same path), a required **Connection Flow** section (sequence diagram + exact authorize/token/registration/callback endpoints), and a required **Administrator Setup** section. - A **Notion dry-run appendix** mirroring the Linear appendix, recorded from the live PAP-16649 probe: verified request sequence, redirect-URI probe results table, sequence diagram (mermaid), shipped manifest sketch, representative tool risk classes, admin setup (nothing to register), governance defaults, and validation hooks. ## Verification - Docs-only change; no code paths affected. `git diff --stat` shows exactly one file. - Every endpoint, error code, and constraint in the appendix was checked against the shipped implementation on the #11009 head: `server/src/services/tool-access.ts` (`assertOAuthRedirectConstraints`, DCR registration metadata, refresh serialization), `server/src/routes/tool-access.ts` (`POST /api/tools/oauth/:connectionId/start`, `GET /api/tools/oauth/callback`), and `packages/shared/src/app-definitions/notion.json` (`redirectConstraints: "https-or-loopback-http"`). - The request log mirrors the live probe record from PAP-16649 (2026-08-06/07), not vendor docs alone. - Mermaid source renders cleanly (rendered PNG attached to PAP-16653). ## Risks - Low: documentation only. Main risk is doc/implementation drift if #11009 changes before merging — mitigated by the dependency note above and re-review after #11009 lands. - The template changes add mandatory sections for future connector proposals; existing proposals are not retroactively invalidated. ## Model Used Claude Fable 5 (claude-fable-5). ## Linked Issues or Issue Description PAP-16653 (parent PAP-16637 P5). Companion catalog package landed in paperclip-content (`integrations/catalog/platforms/notion/areas/mcp/`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 --- doc/connections/CONNECTOR-PLAYBOOK.md | 34 +++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 5 deletions(-) diff --git a/doc/connections/CONNECTOR-PLAYBOOK.md b/doc/connections/CONNECTOR-PLAYBOOK.md index d1f94027983..28b57f519da 100644 --- a/doc/connections/CONNECTOR-PLAYBOOK.md +++ b/doc/connections/CONNECTOR-PLAYBOOK.md @@ -199,10 +199,31 @@ most; the broker resolves endpoints at connect time: dynamic registration — `registration_endpoint`. The broker implements this in `discoverOAuthEndpoints` -(`server/src/services/tool-access.ts`); connections with transport -`mcp_remote` and `auth: "oauth"` prefer discovered endpoints over manifest -hints. Keep manifest hints current anyway so reviewers can read the expected -endpoints without running discovery. +(`server/src/services/tool-access.ts`), but discovery is **not** +unconditional. `oauthEndpointsForConnection` resolves endpoints in this +order: + +1. If the manifest's method `defaults` ship a **complete** pair + (`authorizationEndpoint` **and** `tokenEndpoint`), those are used + unconditionally. `discoverOAuthEndpoints` never runs in this case, so + endpoints stored on the connection's own OAuth config and 401 challenge + hints are **not consulted at all**. +2. Otherwise, for `mcp_remote` connections, the broker calls + `discoverOAuthEndpoints`, which first checks endpoints already stored on + the connection's own OAuth config (falling back field-by-field to the 401 + challenge hints); a complete stored/hinted pair is used as-is — no + `.well-known` fetch. +3. Only when neither of the above yields a complete pair does the broker run + the RFC 9728 → RFC 8414 discovery chain above. + +Consequence: complete manifest endpoint hints are **authoritative, not +hints** — they override even endpoints that an earlier discovery persisted +on the connection, and if they go stale the broker keeps using them. For +discovery-capable vendors, ship only `serverUrl` in `defaults` (as +`notion.json` does) so the broker discovers fresh endpoints at connect +time; add explicit `authorizationEndpoint`/`tokenEndpoint` only for vendors +that do not publish RFC 9728/8414 metadata, and then own keeping them +current. ### Dynamic client registration (RFC 7591) @@ -497,7 +518,10 @@ alone. - Transport: `mcp_remote` - Endpoint: `https://mcp.notion.com/mcp` (Streamable HTTP; `/sse` fallback exists) - Auth mode: OAuth, endpoints resolved by discovery (RFC 9728 → RFC 8414), - public client via RFC 7591 DCR with PKCE S256 mandatory. + public client via RFC 7591 DCR with PKCE S256 mandatory. Discovery runs + because `notion.json` deliberately ships only `serverUrl` — no + `authorizationEndpoint`/`tokenEndpoint` hints, which would otherwise take + precedence and be used verbatim (see "MCP-Direct Connections" above). - Ownership modes: `dcr` (default, zero setup) and `customer` (env-registered classic integration via `PAPERCLIP_TOOL_OAUTH_NOTION_CLIENT_ID/_SECRET`, which always wins when set). From 42c73562c5c18cb4e01d60f3d89295ff13992a36 Mon Sep 17 00:00:00 2001 From: Michael Nguyen <13559011+nguyenm7@users.noreply.github.com> Date: Fri, 7 Aug 2026 09:06:36 -0700 Subject: [PATCH 003/201] fix(heartbeat): backfill projectWorkspaceId when restoring a reused execution workspace (#10171) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each run executes inside a persisted **execution workspace** (a row in `execution_workspaces`) that is either freshly created or **restored/reused** across runs of the same issue > - Before adapter launch, a guard rejects a restored workspace whose `projectWorkspaceId` is null while the issue resolves a concrete project workspace (`persisted_workspace_missing_project_workspace_id`) — a safety check against binding a run to a workspace with no project-workspace link > - The reuse/**restore** path updated the existing row (cwd, branch, status, metadata…) but **never set `projectWorkspaceId`**, so a row persisted with a null value stayed null on every restore > - Result: for an issue that resolves a project workspace, the guard fires, `reuse_existing` re-selects and re-binds the *same* stale null row on the next attempt, and the run crash-loops forever with no self-heal > - This pull request backfills `projectWorkspaceId` during restore (prefer the existing binding, fall back to the resolved one) so the row heals on first reuse and the guard stops firing > - The benefit is that reused workspaces created before their project had a primary project workspace self-repair on next use instead of crash-looping, while genuine mismatches are still surfaced by the guard ## Linked Issues or Issue Description No public GitHub issue exists — describing the bug inline per the bug report template (`.github/ISSUE_TEMPLATE/bug_report.yml`): ### What happened? In `heartbeatService`, the execution-workspace reuse/restore branch calls `executionWorkspacesSvc.update(reusableExistingExecutionWorkspace.id, { … })` without a `projectWorkspaceId` field. Only the sibling CREATE branch sets `projectWorkspaceId`. So an execution workspace that was persisted with a null `projectWorkspaceId` (e.g. created before its project had a primary project workspace) is never backfilled on restore. When such a workspace is later reused for a run whose issue resolves a concrete project workspace, the pre-launch guard throws `persisted_workspace_missing_project_workspace_id`, the run fails, and `reuse_existing` re-binds the identical stale row on the next attempt — an unbounded crash-loop with no self-heal. ### Expected behavior On restore, the reused workspace's `projectWorkspaceId` is backfilled from the resolved project workspace when it is currently null, so the guard passes and the run launches. An existing non-null binding is never overwritten (a genuine mismatch is still surfaced by the separate `project_workspace_mismatch` guard). ### Steps to reproduce 1. Have an `execution_workspaces` row with `project_workspace_id = NULL` that is eligible for reuse. 2. Give its project a primary project workspace (so the issue now resolves a concrete `projectWorkspaceId`). 3. Dispatch a run for an issue in that project that reuses the workspace. The restore `update()` leaves `project_workspace_id` null, the launch guard throws `persisted_workspace_missing_project_workspace_id`, and every subsequent reuse re-binds the same null row and fails identically. ### Paperclip version or commit `master` (branched from `14f20be92`); reproduced on a live self-hosted instance. ### Deployment mode Self-hosted, embedded Postgres, local adapters. ## What Changed - New exported pure helper `reconcileReusedExecutionWorkspaceProjectWorkspaceId(existing, resolved)` in `server/src/services/heartbeat.ts`, returning `existing ?? resolved ?? null`. It prefers an existing binding (never nulls out a good value or silently rebinds a genuine mismatch — the guard still surfaces those), backfills a null binding from the resolved value, and stays null when neither is present. - Wire the helper into the reuse/restore `executionWorkspacesSvc.update(...)` call so the restored row's `projectWorkspaceId` is set to `reconcileReusedExecutionWorkspaceProjectWorkspaceId(reusableExistingExecutionWorkspace.projectWorkspaceId, resolvedProjectWorkspaceId)`. The CREATE branch already set `projectWorkspaceId`; this brings the restore branch to parity. ## Verification - Added 3-case unit coverage in `server/src/__tests__/heartbeat-workspace-session.test.ts` for the helper: (a) backfills a null existing binding from the resolved value, (b) never overwrites an existing binding even when a resolved value is present, (c) returns null when both existing and resolved are absent (null and undefined inputs). - Confirmed the `update()` patch type accepts the field: `executionWorkspacesSvc.update` takes `Partial`, and `projectWorkspaceId` is a column on that table; both `reusableExistingExecutionWorkspace.projectWorkspaceId` and `resolvedProjectWorkspaceId` are `string | null`, matching the helper's `string | null | undefined` params / `string | null` return. - Live-instance exposure check (embedded Postgres): 354 `execution_workspaces` rows carry a null `project_workspace_id`; all of them belong to projects with **no** project workspace, so `expectedProjectWorkspaceId` currently resolves null and the guard does not fire today. The fix is durable heal-on-reuse protection for the moment any such project gains a primary project workspace (or a null row is reused for an issue that resolves one). - CI (full pnpm workspace install) runs the authoritative test + typecheck for this change on this PR. ## Risks - Low risk; scoped to the execution-workspace restore path, no schema or API change. - The helper only ever *adds* a `projectWorkspaceId` where the row had none; it never overwrites an existing binding, so it cannot mask a real `project_workspace_mismatch` (that guard still runs after). - Complementary to (not overlapping with) #10130, which escalates a terminal `workspace_validation_failed` run to `blocked` from the recovery side; this PR prevents the guard from firing on reuse in the first place. Neither depends on the other. ## Model Used Claude Opus 4.8 (`claude-opus-4-8`), extended thinking, with tool use / code execution (repo edit, embedded-Postgres exposure query, unit-logic verification). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above (searched open PRs touching heartbeat / execution-workspace / reuse; only #10130 is related, and it is complementary) - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no user-facing docs affected) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (in progress) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (pending) - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Paperclip --- .../heartbeat-workspace-session.test.ts | 27 +++++++++++++++++++ server/src/services/heartbeat.ts | 24 +++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/server/src/__tests__/heartbeat-workspace-session.test.ts b/server/src/__tests__/heartbeat-workspace-session.test.ts index 26734c687dd..a9693828626 100644 --- a/server/src/__tests__/heartbeat-workspace-session.test.ts +++ b/server/src/__tests__/heartbeat-workspace-session.test.ts @@ -25,6 +25,7 @@ import { prioritizeProjectWorkspaceCandidatesForRun, parseSessionCompactionPolicy, provisionExecutionWorkspaceForFreshnessDecision, + reconcileReusedExecutionWorkspaceProjectWorkspaceId, resolveExecutionWorkspaceConfigFreshness, resolveExecutionWorkspaceReuseRequestForIssue, resolveExecutionWorkspaceReuseProvisioningPolicy, @@ -2884,3 +2885,29 @@ describe("isWorkspaceSyncConflictFailure", () => { expect(isWorkspaceSyncConflictFailure("")).toBe(false); }); }); + +describe("reconcileReusedExecutionWorkspaceProjectWorkspaceId", () => { + it("backfills a null existing binding from the resolved value", () => { + expect( + reconcileReusedExecutionWorkspaceProjectWorkspaceId(null, "resolved-workspace"), + ).toBe("resolved-workspace"); + }); + + it("never overwrites an existing binding, even when a resolved value is present", () => { + expect( + reconcileReusedExecutionWorkspaceProjectWorkspaceId("existing-workspace", "resolved-workspace"), + ).toBe("existing-workspace"); + }); + + it("returns null when both existing and resolved are absent", () => { + expect(reconcileReusedExecutionWorkspaceProjectWorkspaceId(null, null)).toBeNull(); + expect(reconcileReusedExecutionWorkspaceProjectWorkspaceId(undefined, undefined)).toBeNull(); + expect(reconcileReusedExecutionWorkspaceProjectWorkspaceId(null, undefined)).toBeNull(); + }); + + it("backfills when existing is undefined and resolved is present", () => { + expect( + reconcileReusedExecutionWorkspaceProjectWorkspaceId(undefined, "resolved-workspace"), + ).toBe("resolved-workspace"); + }); +}); diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 11e9e247255..f9eddcbd5b8 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1979,6 +1979,26 @@ export async function assertPushCapabilityCheckoutValid(input: { ); } +/** + * Reconcile the `projectWorkspaceId` for a reused execution workspace. + * + * A `reuse_existing` workspace can have been persisted with a null + * `projectWorkspaceId` (e.g. it was created before its project had a primary + * project workspace). When we later restore it for a run whose issue now + * expects a concrete project workspace, backfill the column so the launch + * guard (`persisted_workspace_missing_project_workspace_id`) stops rejecting + * it on every requeue — otherwise `reuse_existing` re-binds the same stale + * record forever and the run crash-loops. Prefer the existing binding when + * present so we never null out a good value or silently rebind a genuine + * mismatch (which the guard still surfaces). + */ +export function reconcileReusedExecutionWorkspaceProjectWorkspaceId( + existingProjectWorkspaceId: string | null | undefined, + resolvedProjectWorkspaceId: string | null | undefined, +): string | null { + return existingProjectWorkspaceId ?? resolvedProjectWorkspaceId ?? null; +} + export async function assertGitSensitiveAdapterWorkspaceValid(input: { adapterType: string; agentId: string; @@ -14578,6 +14598,10 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) status: "active", lastUsedAt: new Date(), metadata: nextExecutionWorkspaceMetadata, + projectWorkspaceId: reconcileReusedExecutionWorkspaceProjectWorkspaceId( + reusableExistingExecutionWorkspace.projectWorkspaceId, + resolvedProjectWorkspaceId, + ), }) : resolvedProjectId ? await executionWorkspacesSvc.create({ From b18b0fc39bb35b8596e6f2418a8100317e3ab5fe Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 7 Aug 2026 14:26:40 -0500 Subject: [PATCH 004/201] feat: refine app connections and legacy worktree startup (#11040) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Apps UI manages app discovery and app connections. > - The managed worktree runtime starts agent work in repository worktrees. > - The Apps routes do not match the main discovery flow, and the connections view lacks a delete action. > - Legacy managed worktrees can also start before their pending seed operation runs. > - This pull request makes app discovery the main Apps route and makes connection management explicit. > - It also seeds legacy managed worktrees before runtime startup and makes the CLI read the repository-local config. > - The benefit is a clearer Apps workflow and a safer managed-worktree startup path. ## Linked Issues or Issue Description **What existing behavior does this improve?** This improves the Apps navigation, app connection management, managed git-worktree startup, and CLI worktree selection. **Subsystem affected** Cross-cutting. The change affects `ui/`, `server/`, `cli/`, and development documentation. **Current behavior** The `/apps` route opens the connections list while discovery uses a nested route. The connections list has no delete action. Some legacy managed worktrees can start runtime work before their pending seed operation runs. The CLI can also read an ambient Paperclip config instead of the repository-local config. **Proposed behavior** The `/apps` route opens Browse, and `/apps/connections` opens the connection list. Users can delete a connection after confirmation. Runtime startup seeds legacy managed worktrees when required. The CLI resolves the current worktree from the repository-local `.paperclip/config.json` file. **Reason and benefit** Users can discover apps from the canonical Apps route and can manage existing connections from a dedicated route. Legacy worktrees receive their required repository content before agent runtime starts. CLI worktree selection stays scoped to the current repository. **Breaking changes** The `/apps` and `/apps/browse` route behavior changes. Old Browse links redirect to `/apps`. The change does not modify an API schema or database schema. ## What Changed - Make Browse the canonical `/apps` page and move the connection list to `/apps/connections`. - Align Apps navigation, redirects, attention links, empty states, and connection actions with the new routes. - Add connection deletion with confirmation and clear failure feedback. - Seed legacy managed git worktrees before runtime startup when their seed status is pending. - Read the CLI worktree selection from the repository-local Paperclip config. - Update focused UI, server, CLI, and development documentation coverage. ## Verification - Ran 202 focused UI, server, and CLI tests. All tests passed. - Ran `pnpm -r typecheck`. All projects passed. - Ran `pnpm build`. All projects built successfully. - Ran `pnpm test:run`. The server and UI stages passed 7,168 tests. The CLI stage found one environment-sensitive secrets test because this workspace injects static AWS credentials. The isolated CLI file passed all 8 tests after those injected variables were unset. - Ran `pnpm check:token-gates`. It reports 12 existing color-token violations in the unchanged `PaperclipOrbit3D.tsx` file from the target branch. This pull request does not modify that file. - Ran focused regression coverage for repository-root CLI config resolution and connection deletion state. All tests and affected package typechecks passed. - Collected all 27 tests in the six changed Playwright specifications successfully. - GitHub Actions passed every latest-head CI gate, including all three e2e shards and the aggregate `e2e` and `verify` jobs. - Greptile reviewed the final commit at 5/5 with zero unresolved threads. ## Risks - Existing bookmarks for `/apps/browse` redirect to `/apps`. - Connection deletion changes visible connection state and requires user confirmation. - The legacy seed path runs only for managed git worktrees with pending seed state. Tests cover the startup condition. - The rebase preserves the target branch's direct OAuth policy for the Notion connection flow. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex with the GPT-5 model family assisted this change. The agent used reasoning, repository tools, code execution, and test execution. The runtime does not expose the exact model snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- cli/src/__tests__/worktree.test.ts | 44 ++++++ cli/src/commands/worktree.ts | 21 +-- doc/DEVELOPING.md | 1 + .../src/__tests__/workspace-runtime.test.ts | 36 +++++ server/src/services/workspace-runtime.ts | 32 +++- tests/e2e/app-not-connected.spec.ts | 9 +- tests/e2e/applications-crud.spec.ts | 8 +- tests/e2e/apps-dark-mode-shots.spec.ts | 9 +- tests/e2e/apps-prosumer-mcp-flow.spec.ts | 8 +- tests/e2e/mcp-user-stories.spec.ts | 2 +- tests/e2e/smoke-lab.spec.ts | 4 +- ui/src/App.test.tsx | 10 ++ ui/src/App.tsx | 15 +- .../components/AppConnectionSidebar.test.tsx | 8 +- ui/src/components/AppConnectionSidebar.tsx | 2 +- ui/src/components/AppsSidebar.test.tsx | 4 +- ui/src/components/AppsSidebar.tsx | 4 +- ui/src/components/Layout.test.tsx | 2 +- ui/src/pages/apps/AppDetail.tsx | 6 +- ui/src/pages/apps/AppNotConnected.tsx | 6 +- ui/src/pages/apps/AppsConnect.tsx | 6 +- ui/src/pages/apps/Browse.tsx | 3 +- ui/src/pages/apps/Connections.test.tsx | 137 ++++++++++++++++++ ui/src/pages/apps/Connections.tsx | 130 +++++++++++++++-- ui/src/pages/apps/gateways/AppsSubNav.tsx | 2 +- ui/src/pages/tools/PasteConfigTab.test.tsx | 2 +- ui/src/pages/tools/PasteConfigTab.tsx | 2 +- ui/src/pages/tools/ToolsAccess.test.tsx | 12 +- ui/src/pages/tools/ToolsAccess.tsx | 2 +- .../pages/tools/profiles/WizardToolsStep.tsx | 2 +- 30 files changed, 452 insertions(+), 77 deletions(-) diff --git a/cli/src/__tests__/worktree.test.ts b/cli/src/__tests__/worktree.test.ts index ce063abfd79..dab7f78bfd9 100644 --- a/cli/src/__tests__/worktree.test.ts +++ b/cli/src/__tests__/worktree.test.ts @@ -31,6 +31,7 @@ import { resolveWorktreeReseedTargetPaths, resolveGitWorktreeAddArgs, resolvePnpmInstallInvocation, + resolveCurrentWorktreeEndpoint, resolveWorktreeSeedBackupEngine, resolveWorktreeMakeTargetPath, worktreeRepairCommand, @@ -167,6 +168,49 @@ function buildSourceConfig(): PaperclipConfig { } describe("worktree helpers", () => { + it("uses the repo-local config for the current worktree", () => { + const targetRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-current-worktree-")); + try { + const localConfig = path.join(targetRoot, ".paperclip", "config.json"); + fs.mkdirSync(path.dirname(localConfig), { recursive: true }); + fs.writeFileSync(localConfig, "{}\n"); + process.env.PAPERCLIP_CONFIG = "/tmp/ambient-paperclip/config.json"; + process.chdir(targetRoot); + + expect(resolveCurrentWorktreeEndpoint()).toMatchObject({ + rootPath: targetRoot, + configPath: localConfig, + isCurrent: true, + }); + } finally { + process.chdir(ORIGINAL_CWD); + fs.rmSync(targetRoot, { recursive: true, force: true }); + } + }); + + it("uses the repository config from a nested working directory", () => { + const targetRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-current-worktree-nested-")); + try { + execFileSync("git", ["init", "-q"], { cwd: targetRoot }); + const nestedDirectory = path.join(targetRoot, "packages", "example", "src"); + const localConfig = path.join(targetRoot, ".paperclip", "config.json"); + fs.mkdirSync(nestedDirectory, { recursive: true }); + fs.mkdirSync(path.dirname(localConfig), { recursive: true }); + fs.writeFileSync(localConfig, "{}\n"); + process.env.PAPERCLIP_CONFIG = "/tmp/ambient-paperclip/config.json"; + process.chdir(nestedDirectory); + + expect(resolveCurrentWorktreeEndpoint()).toMatchObject({ + rootPath: targetRoot, + configPath: localConfig, + isCurrent: true, + }); + } finally { + process.chdir(ORIGINAL_CWD); + fs.rmSync(targetRoot, { recursive: true, force: true }); + } + }); + it("sanitizes instance ids", () => { expect(sanitizeWorktreeInstanceId("feature/worktree-support")).toBe("feature-worktree-support"); expect(sanitizeWorktreeInstanceId(" ")).toBe("worktree"); diff --git a/cli/src/commands/worktree.ts b/cli/src/commands/worktree.ts index 676ee7d52ff..fe07f3ce618 100644 --- a/cli/src/commands/worktree.ts +++ b/cli/src/commands/worktree.ts @@ -2218,10 +2218,13 @@ async function closeDb(db: ClosableDb): Promise { await db.$client?.end?.({ timeout: 5 }).catch(() => undefined); } -function resolveCurrentEndpoint(): ResolvedWorktreeEndpoint { +export function resolveCurrentWorktreeEndpoint(): ResolvedWorktreeEndpoint { + const cwd = path.resolve(process.cwd()); + const rootPath = detectGitWorkspaceInfo(cwd)?.root ?? cwd; + const localConfigPath = path.join(rootPath, ".paperclip", "config.json"); return { - rootPath: path.resolve(process.cwd()), - configPath: resolveConfigPath(), + rootPath, + configPath: existsSync(localConfigPath) ? localConfigPath : resolveConfigPath(), label: "current", isCurrent: true, }; @@ -2233,7 +2236,7 @@ function resolveAttachmentLookupStorages(input: { }): ConfiguredStorage[] { const orderedConfigPaths = [ input.sourceEndpoint.configPath, - resolveCurrentEndpoint().configPath, + resolveCurrentWorktreeEndpoint().configPath, input.targetEndpoint.configPath, ...toMergeSourceChoices(process.cwd()) .filter((choice) => choice.hasPaperclipConfig) @@ -2804,7 +2807,7 @@ export async function worktreeListCommand(opts: WorktreeListOptions): Promise { const excluded = excludeWorktreePath ? path.resolve(excludeWorktreePath) : null; - const currentEndpoint = resolveCurrentEndpoint(); + const currentEndpoint = resolveCurrentWorktreeEndpoint(); const choices = toMergeSourceChoices(process.cwd()) .filter((choice) => choice.hasPaperclipConfig || choice.isCurrent) .filter((choice) => path.resolve(choice.worktree) !== excluded) @@ -3295,7 +3298,7 @@ export async function worktreeMergeHistoryCommand(sourceArg: string | undefined, const targetEndpoint = opts.to ? resolveWorktreeEndpointFromSelector(opts.to, { allowCurrent: true }) - : resolveCurrentEndpoint(); + : resolveCurrentWorktreeEndpoint(); const sourceEndpoint = opts.from ? resolveWorktreeEndpointFromSelector(opts.from, { allowCurrent: true }) : sourceArg @@ -3400,7 +3403,7 @@ async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise { const targetEndpoint = opts.to ? resolveWorktreeEndpointFromSelector(opts.to, { allowCurrent: true }) - : resolveCurrentEndpoint(); + : resolveCurrentWorktreeEndpoint(); const source = resolveWorktreeReseedSource(opts); if (path.resolve(source.configPath) === path.resolve(targetEndpoint.configPath)) { diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 79af9ef36a0..ac53ac1b4f3 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -471,6 +471,7 @@ The default `worktree init` still seeds eagerly and writes `seed-complete` immed - `pnpm paperclipai worktree ensure-seeded` performs the deferred seed **exactly once**. It is lock-guarded and idempotent: a present `seed-complete` marker or a missing `seed-pending` marker short-circuits it, so it is safe to call repeatedly and from concurrent processes. It reads the source instance from the `seed-pending` marker unless you pass `--from-config`. - `paperclipai run` calls `ensureWorktreeSeeded` automatically before doctor/boot, so `run` transparently seeds a lean worktree on first launch. +- Managed git-worktree runtime startup also runs `scripts/provision-worktree-runtime.sh` automatically when a legacy workspace policy has no explicit runtime provision command and the worktree is still `seed-pending`. An explicitly configured runtime provision command always takes precedence. - Worktrees created before lazy seeding shipped have neither marker; they are treated as already-seeded for backward compatibility (never re-cloned). **Seed-pending guard.** `pnpm dev` (the dev-runner) refuses to boot a worktree whose database is still `seed-pending` and points you at the fix: diff --git a/server/src/__tests__/workspace-runtime.test.ts b/server/src/__tests__/workspace-runtime.test.ts index d9726d495a8..cf052cb4372 100644 --- a/server/src/__tests__/workspace-runtime.test.ts +++ b/server/src/__tests__/workspace-runtime.test.ts @@ -37,6 +37,7 @@ import { refreshRemoteTrackingBaseRef, releaseRuntimeServicesForRun, resetRuntimeServicesForTests, + resolveRuntimeProvisionCommand, resolveWorkspaceRuntimeReadinessTimeoutSec, resolveShell, sanitizeRuntimeServiceBaseEnv, @@ -380,6 +381,41 @@ describe("sanitizeRuntimeServiceBaseEnv", () => { }); }); +describe("resolveRuntimeProvisionCommand", () => { + it("backfills deferred seeding for legacy managed git worktrees", async () => { + const baseCwd = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-provision-")); + const cwd = path.join(baseCwd, "worktree"); + try { + await fs.mkdir(path.join(baseCwd, "scripts"), { recursive: true }); + await fs.writeFile( + path.join(baseCwd, "scripts", "provision-worktree-runtime.sh"), + "#!/usr/bin/env bash\n", + ); + await fs.mkdir(path.join(cwd, ".paperclip"), { recursive: true }); + await fs.writeFile(path.join(cwd, ".paperclip", "seed-pending"), "{}\n"); + const workspace = { + ...buildWorkspace(cwd), + baseCwd, + strategy: "git_worktree" as const, + worktreePath: cwd, + }; + + expect(resolveRuntimeProvisionCommand({ config: {}, workspace })).toBe( + "bash ./scripts/provision-worktree-runtime.sh", + ); + expect(resolveRuntimeProvisionCommand({ + config: { runtimeProvisionCommand: "./custom-provision.sh" }, + workspace, + })).toBe("./custom-provision.sh"); + + await fs.writeFile(path.join(cwd, ".paperclip", "seed-complete"), "{}\n"); + expect(resolveRuntimeProvisionCommand({ config: {}, workspace })).toBe(""); + } finally { + await fs.rm(baseCwd, { recursive: true, force: true }); + } + }); +}); + describe("refreshRemoteTrackingBaseRef git auth", () => { it("offers the remote URL to the provider and keeps ambient behavior when it returns null", async () => { const { remotePath, repoRoot } = await createClonedRepoWithRemote(); diff --git a/server/src/services/workspace-runtime.ts b/server/src/services/workspace-runtime.ts index 81a18e5be2b..cb96c5f9d4d 100644 --- a/server/src/services/workspace-runtime.ts +++ b/server/src/services/workspace-runtime.ts @@ -4109,6 +4109,34 @@ function readRuntimeProvisionCommand(config: Record) { ).trim(); } +export function resolveRuntimeProvisionCommand(input: { + config: Record; + workspace: RealizedExecutionWorkspace; +}) { + const configuredCommand = readRuntimeProvisionCommand(input.config); + if (configuredCommand) return configuredCommand; + + if (input.workspace.strategy !== "git_worktree") return ""; + + const stateDir = path.join(input.workspace.cwd, ".paperclip"); + const pendingMarker = path.join(stateDir, "seed-pending"); + const completeMarker = path.join(stateDir, "seed-complete"); + const provisionScript = path.join( + input.workspace.baseCwd, + "scripts", + "provision-worktree-runtime.sh", + ); + if ( + !existsSync(pendingMarker) + || existsSync(completeMarker) + || !existsSync(provisionScript) + ) { + return ""; + } + + return "bash ./scripts/provision-worktree-runtime.sh"; +} + function runtimeProvisionWorkspaceKey(input: StartLocalRuntimeServiceInput) { return input.executionWorkspaceId ? `execution-workspace:${input.executionWorkspaceId}` @@ -4794,7 +4822,7 @@ export async function ensureRuntimeServicesForRun(input: { }); const acquiredServiceIds: string[] = []; const refs: RuntimeServiceRef[] = []; - const runtimeProvisionCommand = readRuntimeProvisionCommand(input.config); + const runtimeProvisionCommand = resolveRuntimeProvisionCommand(input); const provisionCoordinator = createRuntimeProvisionCoordinator(); runtimeServiceLeasesByRun.set(input.runId, acquiredServiceIds); @@ -5007,7 +5035,7 @@ export async function startRuntimeServicesForWorkspaceControl( serviceStates: readConfiguredServiceStates(input.config), }); const invocationId = input.invocationId ?? randomUUID(); - const runtimeProvisionCommand = readRuntimeProvisionCommand(input.config); + const runtimeProvisionCommand = resolveRuntimeProvisionCommand(input); const provisionCoordinator = createRuntimeProvisionCoordinator(); if (rawServices.length === 0 || !input.db || (!input.executionWorkspaceId && !input.workspace.workspaceId)) { diff --git a/tests/e2e/app-not-connected.spec.ts b/tests/e2e/app-not-connected.spec.ts index fac657543d5..e795d03f414 100644 --- a/tests/e2e/app-not-connected.spec.ts +++ b/tests/e2e/app-not-connected.spec.ts @@ -86,7 +86,7 @@ test.describe.serial("not-connected app page", () => { applicationId = body.application.id as string; // Archive the connection (Remove app), then resurrect the application so - // it shows on /apps as "Not connected" — the state in Dotta's screenshot. + // it shows on /apps/connections as "Not connected" — the state in Dotta's screenshot. const archive = await request.delete(`/api/tool-connections/${connectionId}`); expect(archive.ok(), `archive failed ${archive.status()}: ${await archive.text()}`).toBe(true); const revive = await request.patch(`/api/tool-applications/${applicationId}`, { @@ -100,7 +100,7 @@ test.describe.serial("not-connected app page", () => { }); test("not-connected row opens the app page, not the generic wizard", async ({ page }) => { - await page.goto(`/${seed.prefix}/apps`); + await page.goto(`/${seed.prefix}/apps/connections`); const row = page.locator("tbody tr", { hasText: "Bla" }); await expect(row).toBeVisible({ timeout: 30_000 }); await expect(row.getByText("Not connected")).toBeVisible(); @@ -147,7 +147,7 @@ test.describe.serial("not-connected app page", () => { await page.goto(`/${seed.prefix}/apps/app/${applicationId}`); await expect(page).toHaveURL(new RegExp(`/${seed.prefix}/apps/${connectionId}/setup$`), { timeout: 20_000 }); - await page.goto(`/${seed.prefix}/apps`); + await page.goto(`/${seed.prefix}/apps/connections`); const row = page.locator("tbody tr", { hasText: "Bla" }); await expect(row).toBeVisible({ timeout: 30_000 }); await expect(row.getByRole("button", { name: /Open|Review/ })).toBeVisible(); @@ -173,8 +173,9 @@ test.describe.serial("not-connected app page", () => { await page.getByRole("button", { name: "Remove app", exact: true }).click(); await page.screenshot({ path: `${SCREENSHOT_DIR}/apps-nav-w6-04-app-page-danger.png`, fullPage: true }); await page.getByRole("button", { name: "Yes, remove it" }).click(); - await expect(page).toHaveURL(new RegExp(`/${seed.prefix}/apps$`), { timeout: 20_000 }); + await expect(page).toHaveURL(new RegExp(`/${seed.prefix}/apps/connections$`), { timeout: 20_000 }); await expect(page.getByText("App removed").first()).toBeVisible({ timeout: 20_000 }); + await expect(page.getByRole("heading", { name: "Connections" })).toBeVisible(); await expect(page.locator("tbody tr", { hasText: "Doomed app" })).toHaveCount(0); }); }); diff --git a/tests/e2e/applications-crud.spec.ts b/tests/e2e/applications-crud.spec.ts index 4abe456f966..073ab69c71b 100644 --- a/tests/e2e/applications-crud.spec.ts +++ b/tests/e2e/applications-crud.spec.ts @@ -57,7 +57,7 @@ async function createConnection( } async function gotoApps(page: Page, prefix: string) { - await page.goto(`/${prefix}/apps`); + await page.goto(`/${prefix}/apps/connections`); await expect(page.getByRole("heading", { name: "Connections" })).toBeVisible({ timeout: 30_000 }); } @@ -143,8 +143,9 @@ test.describe.serial("applications lifecycle", () => { await expect(page.getByRole("button", { name: "Yes, remove it" })).toBeVisible(); await page.screenshot({ path: `${SCREENSHOT_DIR}/applications-crud-current-remove-connected.png`, fullPage: true }); await page.getByRole("button", { name: "Yes, remove it" }).click(); - await expect(page).toHaveURL(new RegExp(`/${seed.prefix}/apps$`), { timeout: 20_000 }); + await expect(page).toHaveURL(new RegExp(`/${seed.prefix}/apps/connections$`), { timeout: 20_000 }); await expect(page.getByText("App removed").first()).toBeVisible({ timeout: 20_000 }); + await expect(page.getByRole("heading", { name: "Connections" })).toBeVisible(); await expect(page.locator("tbody tr", { hasText: renamed })).toHaveCount(0); }); @@ -158,8 +159,9 @@ test.describe.serial("applications lifecycle", () => { await page.getByRole("button", { name: "Remove app", exact: true }).click(); await page.screenshot({ path: `${SCREENSHOT_DIR}/applications-crud-current-remove-not-connected.png`, fullPage: true }); await page.getByRole("button", { name: "Yes, remove it" }).click(); - await expect(page).toHaveURL(new RegExp(`/${seed.prefix}/apps$`), { timeout: 20_000 }); + await expect(page).toHaveURL(new RegExp(`/${seed.prefix}/apps/connections$`), { timeout: 20_000 }); await expect(page.getByText("App removed").first()).toBeVisible({ timeout: 20_000 }); + await expect(page.getByRole("heading", { name: "Connections" })).toBeVisible(); await expect(page.locator("tbody tr", { hasText: cleanAppName })).toHaveCount(0); }); }); diff --git a/tests/e2e/apps-dark-mode-shots.spec.ts b/tests/e2e/apps-dark-mode-shots.spec.ts index cc2d789fdc4..18937694a4f 100644 --- a/tests/e2e/apps-dark-mode-shots.spec.ts +++ b/tests/e2e/apps-dark-mode-shots.spec.ts @@ -127,7 +127,7 @@ test.describe.serial("dark-mode Apps surfaces", () => { test("apps list dark mode with attention banner", async ({ page }) => { await forceDark(page); - await page.goto(`/${seed.prefix}/apps`); + await page.goto(`/${seed.prefix}/apps/connections`); await expect(page.getByRole("heading", { name: "Connections" })).toBeVisible({ timeout: 30_000 }); await expect(page.getByText(/needs attention/i).first()).toBeVisible({ timeout: 30_000 }); await page.screenshot({ path: `${SCREENSHOT_DIR}/apps-nav-01-apps-dark.png`, fullPage: true }); @@ -135,7 +135,7 @@ test.describe.serial("dark-mode Apps surfaces", () => { test("attention banner dark mode", async ({ page }) => { await forceDark(page); - await page.goto(`/${seed.prefix}/apps`); + await page.goto(`/${seed.prefix}/apps/connections`); await expect(page.getByRole("heading", { name: "Connections" })).toBeVisible({ timeout: 30_000 }); await expect(page.getByText(/app needs attention/i).first()).toBeVisible({ timeout: 30_000 }); await page.screenshot({ path: `${SCREENSHOT_DIR}/apps-nav-02-attention-dark.png`, fullPage: true }); @@ -166,7 +166,7 @@ test.describe.serial("dark-mode Apps surfaces", () => { await expect(page.locator('a[href$="/apps/advanced/audit"]', { hasText: "Activity" })).toBeVisible(); await expect(page.getByRole("link", { name: "Applications", exact: true })).toHaveCount(0); // Apps section lives in the same sidebar now. - await expect(page.locator('a[href$="/apps"]', { hasText: "Connections" })).toBeVisible(); + await expect(page.locator('a[href$="/apps/connections"]', { hasText: "Connections" })).toBeVisible(); await page.screenshot({ path: `${SCREENSHOT_DIR}/apps-nav-05-developer-overview-dark.png`, fullPage: true }); }); @@ -183,8 +183,9 @@ test.describe.serial("dark-mode Apps surfaces", () => { await page.getByRole("button", { name: "Remove app", exact: true }).click(); await page.screenshot({ path: `${SCREENSHOT_DIR}/apps-nav-06-danger-zone-dark.png`, fullPage: true }); await page.getByRole("button", { name: "Yes, remove it" }).click(); - await expect(page).toHaveURL(new RegExp(`/${seed.prefix}/apps$`), { timeout: 20_000 }); + await expect(page).toHaveURL(new RegExp(`/${seed.prefix}/apps/connections$`), { timeout: 20_000 }); await expect(page.getByText("App removed").first()).toBeVisible({ timeout: 20_000 }); + await expect(page.getByRole("heading", { name: "Connections" })).toBeVisible(); await page.screenshot({ path: `${SCREENSHOT_DIR}/apps-nav-07-after-remove-dark.png`, fullPage: true }); }); }); diff --git a/tests/e2e/apps-prosumer-mcp-flow.spec.ts b/tests/e2e/apps-prosumer-mcp-flow.spec.ts index 6ab439592c2..aa1cd892131 100644 --- a/tests/e2e/apps-prosumer-mcp-flow.spec.ts +++ b/tests/e2e/apps-prosumer-mcp-flow.spec.ts @@ -113,11 +113,11 @@ async function startMockMcp(options: { expectedHeader?: string } = {}): Promise< // ---- Helpers ---------------------------------------------------------------- async function gotoApps(page: Page, prefix: string) { - await page.goto(`/${prefix}/apps`); + await page.goto(`/${prefix}/apps/connections`); } async function gotoConnect(page: Page, prefix: string) { - await page.goto(`/${prefix}/apps/browse`); + await page.goto(`/${prefix}/apps`); await expect(page.getByRole("heading", { name: "Browse" })).toBeVisible({ timeout: 30_000 }); await page.getByRole("button", { name: /Connect your own tool/i }).click(); } @@ -127,7 +127,7 @@ async function gotoAdvanced(page: Page, prefix: string) { } async function gotoNeedsAttention(page: Page, prefix: string) { - await page.goto(`/${prefix}/apps`); + await page.goto(`/${prefix}/apps/connections`); } // ---- Tests ------------------------------------------------------------------ @@ -212,7 +212,7 @@ test.describe.serial("prosumer MCP flow prosumer MCP flow", () => { // Verify the mock saw a tools/list call from the catalog refresh. expect(mock.captures.some((c) => c.method === "tools/list")).toBe(true); - // The new connection should show up on /apps. + // The new connection should show up on /apps/connections. await gotoApps(page, seed.prefix); await expect(page.getByRole("heading", { name: "Connections" })).toBeVisible({ timeout: 15_000 }); await page.screenshot({ path: `${SCREENSHOT_DIR}/prosumer-mcp-06-apps-list.png`, fullPage: true }); diff --git a/tests/e2e/mcp-user-stories.spec.ts b/tests/e2e/mcp-user-stories.spec.ts index 4e1de84d34f..7e5d4600cd6 100644 --- a/tests/e2e/mcp-user-stories.spec.ts +++ b/tests/e2e/mcp-user-stories.spec.ts @@ -403,7 +403,7 @@ test.describe.serial("MCP prod Phase 5a user-story harness", () => { const health = await request.post(`/api/tool-connections/${connectionId}/health-check`); expect(health.status()).toBe(502); - await page.goto(`/${seed.prefix}/apps`); + await page.goto(`/${seed.prefix}/apps/connections`); await expect(page.getByRole("heading", { name: "Connections" })).toBeVisible({ timeout: 30_000 }); await screenshot(page, "US-8", "01-needs-attention"); diff --git a/tests/e2e/smoke-lab.spec.ts b/tests/e2e/smoke-lab.spec.ts index 09ae358164b..1b2fed21b3c 100644 --- a/tests/e2e/smoke-lab.spec.ts +++ b/tests/e2e/smoke-lab.spec.ts @@ -164,7 +164,7 @@ async function navigateForEvidence(page: Page, seed: Seed, connectionId: string, return; } if (scenario.uiEntryPath === "attention") { - await page.goto(`/${seed.prefix}/apps`); + await page.goto(`/${seed.prefix}/apps/connections`); await expect(page.getByRole("heading", { name: "Connections" })).toBeVisible({ timeout: 20_000 }); return; } @@ -391,7 +391,7 @@ test.describe.serial("Smoke Lab scenario catalog mirror", () => { await request.post(`/api/tool-connections/${connection.id}/catalog/refresh`), ); expect(refresh.quarantinedCount).toBeGreaterThan(0); - await page.goto(`/${seed.prefix}/apps`); + await page.goto(`/${seed.prefix}/apps/connections`); return `Catalog refresh quarantined ${refresh.quarantinedCount} changed entries.`; }); diff --git a/ui/src/App.test.tsx b/ui/src/App.test.tsx index 9f7b3cd32d6..f1d6fb1c2e5 100644 --- a/ui/src/App.test.tsx +++ b/ui/src/App.test.tsx @@ -242,3 +242,13 @@ describe("Skill Studio routes", () => { expect(createIndexes[1]).toBeLessThan(detailIndexes[1]!); }); }); + +describe("Apps routes", () => { + it("uses browse as the Apps landing page and gives connections a canonical URL", () => { + expect(appSource).toContain('} />'); + expect(appSource).toContain('} />'); + expect(appSource).toContain('} />'); + expect(appSource).toContain('} />'); + expect(appSource).toContain('} />'); + }); +}); diff --git a/ui/src/App.tsx b/ui/src/App.tsx index 0e309f4fa20..3a5c3873232 100644 --- a/ui/src/App.tsx +++ b/ui/src/App.tsx @@ -122,14 +122,15 @@ function boardRoutes() { } /> } /> }> - } /> - } /> + } /> + } /> + } /> } /> - } /> - } /> + } /> + } /> } /> {/* Needs attention folded into Connections (PAP-13254); keep legacy links working. */} - } /> + } /> } /> } /> } /> @@ -309,7 +310,7 @@ function boardRoutes() { function AppsConnectEntryRoute() { const location = useLocation(); const searchParams = new URLSearchParams(location.search); - return canEnterAppsConnect(searchParams) ? : ; + return canEnterAppsConnect(searchParams) ? : ; } function InboxRootRedirect() { @@ -403,7 +404,7 @@ function LegacyToolsRedirect() { function legacyToolsRedirectTarget(tab?: string) { if (!tab) return "/apps/advanced/profiles"; - if (tab === "applications" || tab === "connections" || tab === "overview" || tab === "examples") return "/apps"; + if (tab === "applications" || tab === "connections" || tab === "overview" || tab === "examples") return "/apps/connections"; return `/apps/advanced/${tab}`; } diff --git a/ui/src/components/AppConnectionSidebar.test.tsx b/ui/src/components/AppConnectionSidebar.test.tsx index 68fe3f7c9e6..db0cb79dfb9 100644 --- a/ui/src/components/AppConnectionSidebar.test.tsx +++ b/ui/src/components/AppConnectionSidebar.test.tsx @@ -167,7 +167,7 @@ describe("AppConnectionSidebar", () => { it("renders a back link and the connected app tabs (including Test)", async () => { await renderSidebar(); - expect(container.querySelector('a[href="/apps"]')?.textContent).toContain("All apps"); + expect(container.querySelector('a[href="/apps/connections"]')?.textContent).toContain("All apps"); expect(container.textContent).toContain("GitHub"); expect(container.querySelectorAll("[data-to]").length).toBe(6); expect(sidebarNavItemMock).toHaveBeenCalledWith(expect.objectContaining({ to: "/apps/conn-1/setup", label: "Setup", end: true })); @@ -190,7 +190,7 @@ describe("AppConnectionSidebar", () => { await renderSidebar(); - expect(container.querySelector('a[href="/apps"]')?.textContent).toContain("All apps"); + expect(container.querySelector('a[href="/apps/connections"]')?.textContent).toContain("All apps"); expect(container.textContent).toContain("GitHub"); expect(mockToolsApi.getConnection).not.toHaveBeenCalled(); expect(sidebarNavItemMock).toHaveBeenCalledWith(expect.objectContaining({ to: "/apps/app/app-1/setup", label: "Setup", end: true })); @@ -212,7 +212,7 @@ describe("AppConnectionSidebar", () => { await renderSidebar(); expect(container.textContent).toContain("App"); - expect(container.querySelector('a[href="/apps"]')?.textContent).toContain("All apps"); + expect(container.querySelector('a[href="/apps/connections"]')?.textContent).toContain("All apps"); expect(container.querySelectorAll("[data-to]").length).toBe(6); }); @@ -225,7 +225,7 @@ describe("AppConnectionSidebar", () => { await renderSidebar(); expect(container.textContent).toContain("App"); - expect(container.querySelector('a[href="/apps"]')?.textContent).toContain("All apps"); + expect(container.querySelector('a[href="/apps/connections"]')?.textContent).toContain("All apps"); expect(container.querySelectorAll("[data-to]").length).toBe(5); }); }); diff --git a/ui/src/components/AppConnectionSidebar.tsx b/ui/src/components/AppConnectionSidebar.tsx index adc50eef97f..406ce39b034 100644 --- a/ui/src/components/AppConnectionSidebar.tsx +++ b/ui/src/components/AppConnectionSidebar.tsx @@ -83,7 +83,7 @@ export function AppDetailSidebar(props: AppDetailSidebarProps) {