From 081dacba1d44e5b2b357d7f5105adda82f99e450 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 08:23:49 -0600 Subject: [PATCH] perf(workspaces): inspect only due workspaces and wait for a holding run's release (#21617) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The terminal workspace sweep stores one deadline per workspace: cooldown end, compaction, cleanup retry, Git idle. > - When any deadline was due, the worker ran a full rotation over every open candidate workspace. That costs a task tree read and more for each one. > - A workspace held by a `running` run kept a past deadline, so the 5-minute deadline rotation came back for it forever. A run that never finishes would keep the instance awake indefinitely. > - This pull request makes a due deadline inspect only the due workspaces. A held workspace waits for its run instead of for a timer. > - The benefit is that an idle instance does work only for workspaces whose deadline really arrived. The cost no longer grows with the number of open workspaces. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** How the terminal workspace worker (`server/src/services/terminal-workspace-work.ts`) acts on due deadlines and on workspaces held by an active run. **Current behavior** `hasPending` reads `nextTerminalWorkspaceSweepAt`. When the earliest deadline is past, the next run starts a full keyset rotation, 50 workspaces per page with 1 second between pages, over every candidate. A workspace whose issue still has a `queued` or `running` run is skipped, but the archive deadline set earlier in the same pass stays. It is already past, so the rotation repeats every 5 minutes. **Proposed behavior** - When the run fires for a due deadline, the worker takes the due workspace ids from the service (`takeDueTerminalWorkspaceTargets`) and runs a targeted inspection of only those workspaces. The service first clears the deadlines of the targeted workspaces; the inspection sets fresh ones. - If more than 50 workspaces are due, or a due deadline belongs to the isolated source, nested clone or task directory work, the worker still runs a full rotation. Startup still runs a full rotation. - A workspace skipped because a run holds it drops its deadline. - `writeRunStatus` (`server/src/services/heartbeat/run-lifecycle.ts`) publishes `issue:` to the terminal workspace queue when a run reaches a terminal status. That targeted inspection reaches the workspace the run held. - **Backstop.** About 200 code paths write `heartbeat_runs` directly (reapers, recovery terminalization, bulk cancels), and they do not pass through `writeRunStatus`. So the worker keeps a full rotation as a backstop, armed only by activity, like the recovery backstop from #21552: - Any terminal-workspace hint arms it. - Once armed, a full rotation runs at most once per hour (`TERMINAL_WORKSPACE_BACKSTOP_MS`) after the previous full rotation. - An idle instance with no hint schedules nothing. - A missed release hint therefore delays archiving by at most one hour after activity, without a restart. - The hour is a compromise. On master today, the 5-minute rotation re-reads every candidate. On the hosted instance that is about 1,000 finished remote workspaces, roughly 3,000 statements per lap. The hourly backstop re-reads them at most once per hour while there is activity, about 1 statement per second averaged. **Reason and benefit** Background work must be driven by deadlines and change notifications, and its cost must not grow with the backlog. Every guarantee keeps a named trigger: - cooldown end, compaction, retry backoff and Git idle each have a stored deadline, and a due deadline inspects exactly that workspace; - the release of a holding run publishes a hint; - task status changes and workspace writes still publish hints; - startup still reconciles everything. ## What Changed - `server/src/services/execution-workspaces.ts`: - New `takeDueTerminalWorkspaceTargets()`. - A targeted inspection clears the deadlines of its targets before it inspects them. - A workspace skipped for an active run drops its deadline. - `server/src/services/terminal-workspace-work.ts`: a due run uses the due targets instead of a full rotation, unless they overflow or a non-workspace deadline is due. A due run that finds nothing due does nothing. Any hint arms the hourly backstop rotation. - `server/src/services/heartbeat/run-lifecycle.ts`: a terminal run status hints the run's task to the terminal workspace queue. - `server/src/__tests__/terminal-workspace-deadlines.test.ts`: four new tests on real PostgreSQL. They use the real coordinator and worker on a fake clock. - `server/src/__tests__/terminal-workspace-idle.test.ts` (from #21616): its idle window now bounds the number of sweeps instead of requiring zero targeted sweeps, because a due deadline is now a targeted inspection. ## Verification - `inspects only the workspace whose cooldown ends, however many trees stay open`: 60 workspaces with open task trees and 1 done workspace whose 1-day cooldown ends 1 hour later. - Before the change: after startup, every due deadline starts a full rotation (27 full sweeps in 2 hours). - After the change: no full rotation; the done workspace is inspected alone (`checked: 1`) once its cooldown ends; fewer than 200 statements in 2 hours. - `waits for the release of the run that holds a workspace instead of rechecking it`: a done workspace held by a `running` run. - Before the change: a full rotation every 5 minutes (22 sweeps in 2 hours). - After the change: 0 statements in 2 idle hours. When the run is set to `succeeded` through the lifecycle, the workspace is inspected (`checked: 1`, `skippedActiveRun: 0`). - `leaves a thousand finished remote workspaces unread at idle`. This is the hosted state: 1,000 `active` remote workspaces with a location, a committed snapshot and a task done 5 days ago, and zero runs. - After startup and 2 idle hours: no full rotation. No remote workspace is read again; the only inspections are of the one due local workspace. Statements stay within 10 per inspection. - On master, every due deadline re-reads all 1,000. - `archives a workspace whose run ended outside the run lifecycle within the activity-armed backstop`: - A workspace held by a `running` run. The run is set to `failed` by a direct `UPDATE`, so no release hint is published, and one unrelated hint arrives. - The workspace is archived within one hour and one minute, without a restart. - With the backstop scheduling disabled, it stays `active`. - `npx vitest run src/__tests__/terminal-workspace-deadlines.test.ts src/__tests__/terminal-workspace-work.test.ts src/__tests__/execution-workspaces-service.test.ts src/__tests__/isolated-task-directory-retention.test.ts src/__tests__/git-workspace-expiration.test.ts src/__tests__/workspace-delivery-cleanup.integration.test.ts src/__tests__/nested-clone-migration.test.ts src/__tests__/execution-workspace-compaction.test.ts src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 333 tests pass. - `pnpm check:module-boundaries` passes. Local fixture with #21616 applied as the base, 30 idle minutes on a fake clock, real PostgreSQL. Each fixture holds open trees, a stranded `running` holder, cooldown, done local workspaces with missing files, closed remote workspaces with and without a committed snapshot: | Workspaces | Before: sweeps | Before: statements | After: sweeps | After: statements | | --- | --- | --- | --- | --- | | 140 | 20 (full rotations) | 1,371 | 21 (targeted only) | 866 | | 420 | 84 (full rotations) | 6,597 | 33 (targeted only) | 3,272 | After the change, no sweep reads the task tree of a workspace whose tree is still open (`skippedNonTerminalTree` falls from 1,440 to 0 at 420 workspaces). What remains comes only from workspaces with a due retry or inspection backoff, so it grows with the number of stuck workspaces, not open ones. Per inspected workspace a pass issues the same 3–6 statements as before; batching them per page is a separate pull request. ## Risks - **A run that leaves `running` without `writeRunStatus`** does not hint. The activity-armed backstop catches it within one hour of activity, and startup reconciles. This pull request does not decide that a stranded run is gone; it waits for a status change. - **A workspace whose deadline is reset to "now"** (a reopen-pending flag within its grace period, task directory budget overflow) is inspected alone on each due run. The inspection backoff in `nextTerminalWorkspaceSweepAt` still spaces these runs. - **Workspaces without a stored deadline** (for example an undelivered workspace whose deadline was deleted) are inspected again only when a hint or the startup rotation reaches them. Before this change they were also reached by the rotation that a different workspace's deadline started. ## Open decision **Should a terminal workspace whose local files are gone be archived, or should it wait?** This pull request does not change the behaviour; it leaves the decision to the owner. Today such a workspace fails its close-readiness inspection and is reported as `skippedUndelivered`. A non-remote workspace keeps its past deadline, so the 5-minute inspection backoff re-inspects it again and again. That costs about 3–4 statements per stuck workspace every 5 minutes (from the local fixture: 255 inspections, 866 statements), and it never ends while the files stay missing. The cause is typically a replaced container. The two options: - **Archive it.** Nothing remains on disk to keep. - **Store a "files missing" state.** It is then inspected again only when a write announces a change. Either option removes this cost. Archiving also removes the row from later rotations. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../terminal-workspace-deadlines.test.ts | 177 ++++++++++++++++++ .../__tests__/terminal-workspace-idle.test.ts | 2 +- server/src/services/execution-workspaces.ts | 10 + .../src/services/heartbeat/run-lifecycle.ts | 3 + .../src/services/terminal-workspace-work.ts | 33 +++- 5 files changed, 221 insertions(+), 4 deletions(-) create mode 100644 server/src/__tests__/terminal-workspace-deadlines.test.ts diff --git a/server/src/__tests__/terminal-workspace-deadlines.test.ts b/server/src/__tests__/terminal-workspace-deadlines.test.ts new file mode 100644 index 00000000000..e1d70a53efb --- /dev/null +++ b/server/src/__tests__/terminal-workspace-deadlines.test.ts @@ -0,0 +1,177 @@ +import { randomUUID } from "node:crypto"; +import { eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, companies, createDb, executionWorkspaceSnapshots, executionWorkspaces, heartbeatRuns, issues, projects, type Db } from "@paperclipai/db"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { executionWorkspaceService } from "../services/execution-workspaces.js"; +import { createHeartbeatLifecycle, type HeartbeatLifecycleDependencies } from "../services/heartbeat/run-lifecycle.js"; +import { notifyTerminalWorkspaceWork, registerTerminalWorkspaceWork, TERMINAL_WORKSPACE_BACKSTOP_MS, TERMINAL_WORKSPACE_RETRY_MS } from "../services/terminal-workspace-work.js"; +import { meterDatabaseStatements } from "./helpers/database-statement-meter.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; + +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_TERMINAL_WORKSPACE_DEADLINE_TEST_DATABASE_URL"); +const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport(); +const HOUR = 3_600_000; +const realSetTimeout = globalThis.setTimeout; +const realSleep = (ms: number) => new Promise(resolve => realSetTimeout(resolve, ms)); + +(support.supported ? describe : describe.skip)("terminal workspace deadlines", () => { + let temporary: Awaited> | undefined; + let db: Db; + let meter: ReturnType; + beforeAll(async () => { + if (!external) temporary = await startEmbeddedPostgresTestDatabase("terminal-workspace-deadlines-"); + const url = external ?? temporary!.connectionString; + db = createDb(url); + db.$client.options.onnotice = () => {}; + meter = meterDatabaseStatements(db, url); + }, 120_000); + afterEach(async () => { vi.useRealTimers(); await db.execute(sql`truncate companies cascade`); }); + afterAll(async () => { meter?.uninstall(); await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + + async function seed(input: { open: number; cooldownEndsInMs?: number; heldByRunningRun?: boolean }) { + const companyId = randomUUID(), projectId = randomUUID(), agentId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Deadlines", issuePrefix: `D${companyId.slice(0, 6)}` }); + await db.insert(projects).values({ id: projectId, companyId, name: "Project", status: "in_progress" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Agent", role: "engineer", status: "idle", adapterType: "process" }); + const old = new Date(Date.now() - 48 * HOUR); + const workspace = async (status: string, completedAt: Date | null, runId: string | null) => { + const id = randomUUID(), issueId = randomUUID(); + await db.insert(executionWorkspaces).values({ id, companyId, projectId, mode: "shared_workspace", strategyType: "project_primary", + name: id, status: "active", providerType: "local_fs", cwd: `/missing/${id}`, createdAt: old, updatedAt: old }); + await db.insert(issues).values({ id: issueId, companyId, projectId, title: status, status, executionWorkspaceId: id, + executionRunId: runId, completedAt, updatedAt: old }); + await db.update(executionWorkspaces).set({ sourceIssueId: issueId, updatedAt: old }).where(eq(executionWorkspaces.id, id)); + return { id, issueId }; + }; + for (let index = 0; index < input.open; index++) await workspace("in_progress", null, null); + let runId: string | null = null; + if (input.heldByRunningRun) { + runId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running" }); + } + const target = await workspace("done", new Date(Date.now() - 24 * HOUR + (input.cooldownEndsInMs ?? -HOUR)), runId); + if (runId) await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: target.issueId } }).where(eq(heartbeatRuns.id, runId)); + return { companyId, runId, target }; + } + + function start(options: Parameters[1] = {}) { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); + const service = executionWorkspaceService(db, { workspaceReaperCooldownDays: 1, workspaceCompactionDelayMs: HOUR, + terminalWorkspaceInspectionBackoffMs: TERMINAL_WORKSPACE_RETRY_MS, ...options }); + const sweeps: Array<{ targets: string[] | null; checked: number; skippedCooldown: number; skippedActiveRun: number }> = []; + let inFlight = 0; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: () => {} }); + const worker = registerTerminalWorkspaceWork(coordinator, { ...service, + sweepTerminalWorkspaces: async (limit, signal, options) => { + inFlight += 1; + const result = await service.sweepTerminalWorkspaces(limit, signal, options).finally(() => { inFlight -= 1; }); + sweeps.push({ targets: options?.targets ? [...options.targets.workspaceIds, ...options.targets.issueIds] : null, + checked: result.checked, skippedCooldown: (result as { skippedCooldown: number }).skippedCooldown, + skippedActiveRun: (result as { skippedActiveRun: number }).skippedActiveRun }); + return result; + } }, () => true); + const advance = async (ms: number) => { + for (let elapsed = 0; elapsed < ms; elapsed += 10_000) { + await vi.advanceTimersByTimeAsync(10_000); + do { await realSleep(10); await vi.advanceTimersByTimeAsync(1); } while (inFlight > 0); + } + }; + const settle = async (operation: Promise) => { + let done = false; + const result = operation.finally(() => { done = true; }); + while (!done) { await realSleep(5); await vi.advanceTimersByTimeAsync(1); } + return result; + }; + return { sweeps, coordinator, worker, advance, settle, stop: async () => { vi.useRealTimers(); await worker.stop(); await coordinator.stop(); } }; + } + + it("inspects only the workspace whose cooldown ends, however many trees stay open", async () => { + const f = await seed({ open: 60, cooldownEndsInMs: HOUR }); + const work = start(); + try { + await work.advance(5 * 60_000); + const startup = work.sweeps.length; + expect(startup).toBeGreaterThan(0); + expect(work.sweeps.at(-1)).toMatchObject({ targets: null }); + const before = meter.read().statements; + await work.advance(2 * HOUR); + const window = work.sweeps.slice(startup); + expect(window.filter(sweep => sweep.targets === null)).toEqual([]); + expect(window.some(sweep => sweep.targets?.includes(f.target.id) && sweep.checked === 1 && sweep.skippedCooldown === 0)).toBe(true); + expect(window.reduce((sum, sweep) => sum + sweep.checked, 0)).toBeLessThanOrEqual(window.length); + expect(meter.read().statements - before).toBeLessThan(200); + } finally { await work.stop(); } + }, 300_000); + + it("leaves a thousand finished remote workspaces unread at idle", async () => { + const f = await seed({ open: 0, cooldownEndsInMs: 30 * 60_000 }); + const [target] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.target.id)); + const old = new Date(Date.now() - 5 * 24 * HOUR); + const rows = Array.from({ length: 1000 }, () => ({ workspaceId: randomUUID(), issueId: randomUUID(), snapshotId: randomUUID() })); + for (let index = 0; index < rows.length; index += 250) { + const page = rows.slice(index, index + 250); + await db.insert(executionWorkspaces).values(page.map(row => ({ id: row.workspaceId, companyId: f.companyId, projectId: target!.projectId, + mode: "isolated_workspace", strategyType: "git_worktree", name: row.workspaceId, status: "active", providerType: "remote", + createdAt: old, updatedAt: old, metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1, + snapshotId: row.snapshotId, source: { kind: "empty" }, writerRunId: null, dirty: false, + location: { environmentId: randomUUID(), leaseId: randomUUID(), providerLeaseId: `sandbox-${row.workspaceId}`, remoteCwd: "/workspace" } } } }))); + await db.insert(issues).values(page.map(row => ({ id: row.issueId, companyId: f.companyId, projectId: target!.projectId, title: "done", + status: "done", executionWorkspaceId: row.workspaceId, completedAt: old, updatedAt: old }))); + await db.insert(executionWorkspaceSnapshots).values(page.map(row => ({ id: row.snapshotId, companyId: f.companyId, + executionWorkspaceId: row.workspaceId, sourceOperationId: randomUUID(), generation: 1, epoch: 1, status: "committed" as const }))); + for (const row of page) await db.update(executionWorkspaces).set({ sourceIssueId: row.issueId, updatedAt: old }) + .where(eq(executionWorkspaces.id, row.workspaceId)); + } + const work = start(); + try { + await work.advance(5 * 60_000); + const startup = work.sweeps.length; + expect(work.sweeps.reduce((sum, sweep) => sum + sweep.checked, 0)).toBe(1001); + const before = meter.read().statements; + await work.advance(2 * HOUR); + const window = work.sweeps.slice(startup); + expect(window.filter(sweep => sweep.targets === null)).toEqual([]); + expect(window.every(sweep => sweep.targets?.length === 1 && sweep.targets[0] === f.target.id && sweep.checked === 1)).toBe(true); + expect(meter.read().statements - before).toBeLessThan(10 * Math.max(window.length, 1)); + } finally { await work.stop(); } + }, 300_000); + + it("archives a workspace whose run ended outside the run lifecycle within the activity-armed backstop", async () => { + const f = await seed({ open: 10, heldByRunningRun: true }); + const work = start({ inspectGitCloseReadiness: async workspace => ({ statusInspectionSucceeded: true, warnings: [], git: { + repoRoot: null, workspacePath: workspace.cwd, branchName: null, baseRef: null, hasDirtyTrackedFiles: false, hasUntrackedFiles: false, + dirtyEntryCount: 0, untrackedEntryCount: 0, aheadCount: 0, behindCount: 0, isMergedIntoBase: true, createdByRuntime: false } }) }); + try { + await work.advance(5 * 60_000); + expect(work.sweeps.at(-1)).toMatchObject({ targets: null, skippedActiveRun: 1 }); + await work.settle(db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId!))); + await work.settle(db.transaction(tx => notifyTerminalWorkspaceWork(tx, `workspace:${randomUUID()}`))); + const releasedAt = Date.now(); + await work.advance(TERMINAL_WORKSPACE_BACKSTOP_MS + 60_000); + const [row] = await work.settle(db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.target.id))); + expect(row!.status).toBe("archived"); + expect(row!.closedAt!.getTime() - releasedAt).toBeLessThanOrEqual(TERMINAL_WORKSPACE_BACKSTOP_MS + 60_000); + } finally { await work.stop(); } + }, 300_000); + + it("waits for the release of the run that holds a workspace instead of rechecking it", async () => { + const f = await seed({ open: 10, heldByRunningRun: true }); + const work = start(); + try { + await work.advance(5 * 60_000); + expect(work.sweeps.at(-1)).toMatchObject({ targets: null, skippedActiveRun: 1 }); + const startup = work.sweeps.length; + const before = meter.read().statements; + await work.advance(2 * HOUR); + expect(work.sweeps.slice(startup)).toEqual([]); + expect(meter.read().statements - before).toBe(0); + vi.useRealTimers(); + await createHeartbeatLifecycle(db, { emitTerminalAgentTaskRun: vi.fn() } as unknown as HeartbeatLifecycleDependencies) + .setRunStatus(f.runId!, "succeeded", { finishedAt: new Date() }); + await vi.waitFor(() => expect(work.sweeps.slice(startup).some(sweep => sweep.targets?.includes(f.target.issueId) + && sweep.checked === 1 && sweep.skippedActiveRun === 0)).toBe(true), { timeout: 20_000 }); + } finally { await work.stop(); } + }, 300_000); +}); diff --git a/server/src/__tests__/terminal-workspace-idle.test.ts b/server/src/__tests__/terminal-workspace-idle.test.ts index c1bef7fc449..5abda8fe837 100644 --- a/server/src/__tests__/terminal-workspace-idle.test.ts +++ b/server/src/__tests__/terminal-workspace-idle.test.ts @@ -118,7 +118,7 @@ const realSleep = (ms: number) => new Promise(resolve => realSetTimeout(resolve, vi.useRealTimers(); const window = sweeps.slice(before.sweeps); expect(destroyCalls.length).toBe(before.destroys); - expect(window.filter(sweep => sweep.targeted)).toEqual([]); + expect(window.length).toBeLessThanOrEqual(12); expect(meter.read().statements - before.statements).toBeLessThan(5_000); for (const id of remote) { const [row] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, id)); diff --git a/server/src/services/execution-workspaces.ts b/server/src/services/execution-workspaces.ts index 6ec8deb869e..7eeb921e750 100644 --- a/server/src/services/execution-workspaces.ts +++ b/server/src/services/execution-workspaces.ts @@ -3660,6 +3660,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic compacted: 0, compactionFailed: 0, }; + if (targets) for (const id of [...targets.workspaceIds, ...candidates.map(candidate => candidate.workspace.id)]) terminalWorkspaceDeadlines.delete(id); const readinessByWorkingTree = new Map>(); const backOffTerminalWorkspace = async (snapshot: ExecutionWorkspaceRow, failure: TerminalWorkspaceFailure) => { const [target] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, snapshot.id)); @@ -3914,6 +3915,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic if (await workspaceHasActiveRun(workspace)) { if (reopenPending) result.skippedReopened += 1; else result.skippedActiveRun += 1; + terminalWorkspaceDeadlines.delete(workspace.id); continue; } if (remoteWorkspace && isClosedExecutionWorkspaceStatus(workspace.status)) { @@ -4154,6 +4156,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic } if (await workspaceHasActiveRun(workspace)) { result.skippedActiveRun += 1; + terminalWorkspaceDeadlines.delete(workspace.id); continue; } result.eligible += 1; @@ -4364,6 +4367,13 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic } }, + takeDueTerminalWorkspaceTargets: (): { workspaceIds: string[]; rotation: boolean } => { + const at = now().getTime(); + const due = (deadlines: Map) => [...deadlines].filter(([, deadline]) => deadline <= at).map(([id]) => id); + const rotation = due(isolatedRehomeDeadlines).length + due(nestedCloneMigrationDeadlines).length + due(taskDirectoryDeadlines).length > 0; + return { workspaceIds: due(terminalWorkspaceDeadlines), rotation }; + }, + nextTerminalWorkspaceSweepAt: async () => { if (terminalWorkspaceDeadlines.size === 0 && isolatedRehomeDeadlines.size === 0 && nestedCloneMigrationDeadlines.size === 0 && taskDirectoryDeadlines.size === 0) return null; diff --git a/server/src/services/heartbeat/run-lifecycle.ts b/server/src/services/heartbeat/run-lifecycle.ts index e4cfaa222cc..103dfe8f709 100644 --- a/server/src/services/heartbeat/run-lifecycle.ts +++ b/server/src/services/heartbeat/run-lifecycle.ts @@ -1,3 +1,4 @@ +import { notifyTerminalWorkspaceWork } from "../terminal-workspace-work.js"; import { settleReattachRun } from "./run-admission.js"; import { readNativeStartupCleanup } from "../native-runtime/native-startup-cleanup.js"; import { isContextCompactionRun } from "../context-compaction.js"; @@ -1122,6 +1123,8 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl if (previous && isHeartbeatRunTerminalStatus(status)) await notifyRetainedNativeSessionCleanup(tx, { companyId: previous.companyId, runId: previous.id }); if (isHeartbeatRunTerminalStatus(status)) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, companyId); + const runIssueId = previous ? previous.nativeIssueId ?? previous.issueId ?? readNonEmptyString(parseObject(previous.contextSnapshot).issueId) : null; + if (previous && isHeartbeatRunTerminalStatus(status) && runIssueId) await notifyTerminalWorkspaceWork(tx, `issue:${runIssueId}`); await notifyExecutionRunWork(tx, previous ? { companyId: previous.companyId, runId: previous.id, issueId: previous.nativeIssueId ?? previous.issueId ?? readNonEmptyString(parseObject(previous.contextSnapshot).issueId) ?? undefined } : undefined); if (previous) await notifyChatRunMilestones(tx, { companyId: previous.companyId, runId: previous.id, diff --git a/server/src/services/terminal-workspace-work.ts b/server/src/services/terminal-workspace-work.ts index d7248aae517..b9845bdbb5d 100644 --- a/server/src/services/terminal-workspace-work.ts +++ b/server/src/services/terminal-workspace-work.ts @@ -3,6 +3,7 @@ import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notificatio export const TERMINAL_WORKSPACE_RETRY_MS = 5 * 60 * 1000; export const MAX_TERMINAL_WORKSPACE_TARGETS = 50; +export const TERMINAL_WORKSPACE_BACKSTOP_MS = 60 * 60 * 1000; export function notifyTerminalWorkspaceWork(transaction: object, hint?: string) { return notifyDeliveryWork(transaction, DELIVERY_QUEUES.terminalWorkspaces, hint); @@ -36,6 +37,7 @@ export function registerTerminalWorkspaceWork( nextTerminalWorkspaceSweepAt(): Promise; hasTerminalWorkspaceSweepContinuation(): boolean; terminalWorkspaceWorkSettled?(hints?: ReadonlySet, options?: { targeted?: boolean }): void; + takeDueTerminalWorkspaceTargets?(): { workspaceIds: string[]; rotation: boolean }; }, canRun: () => boolean, ) { @@ -49,10 +51,14 @@ export function registerTerminalWorkspaceWork( let rotationRequested = false; let dueAt: number | null = null; let lastRotationPageAt = 0; + let backstopArmed = false; + let lastFullRotationAt = Date.now(); + let continuing = false; const targetWorkspaceIds = new Set(), targetIssueIds = new Set(); const worker = coordinator.register(DELIVERY_QUEUES.terminalWorkspaces, { retryMs: TERMINAL_WORKSPACE_RETRY_MS, onSettled: hints => { + backstopArmed = true; const targets = terminalWorkspaceTargetsFromHints(hints); for (const id of targets?.workspaceIds ?? []) targetWorkspaceIds.add(id); for (const id of targets?.issueIds ?? []) targetIssueIds.add(id); @@ -70,12 +76,26 @@ export function registerTerminalWorkspaceWork( next = Date.now() + TERMINAL_WORKSPACE_RETRY_MS; const startup = !startupSweepStarted; startupSweepStarted = true; + let dueRotation = due; + if (due && !startup && service.takeDueTerminalWorkspaceTargets) { + const dueTargets = service.takeDueTerminalWorkspaceTargets(); + dueRotation = dueTargets.rotation; + for (const id of dueTargets.workspaceIds) targetWorkspaceIds.add(id); + if (targetWorkspaceIds.size + targetIssueIds.size > MAX_TERMINAL_WORKSPACE_TARGETS) { + dueRotation = true; + targetWorkspaceIds.clear(); targetIssueIds.clear(); + } + } const options = { compact: !startup, allowRemoteDeliveryChecks: !startup }; const targeted = targetWorkspaceIds.size + targetIssueIds.size > 0; - if (startup || due || rotationRequested || !targeted + const backstopDue = backstopArmed && Date.now() >= lastFullRotationAt + TERMINAL_WORKSPACE_BACKSTOP_MS; + if (due && service.takeDueTerminalWorkspaceTargets && !startup && !dueRotation && !targeted && !rotationRequested && !backstopDue + && !service.hasTerminalWorkspaceSweepContinuation()) return { checked: 0 }; + if (startup || dueRotation || rotationRequested || backstopDue || !targeted || (service.hasTerminalWorkspaceSweepContinuation() && Date.now() >= lastRotationPageAt + 1000)) { rotationRequested = false; lastRotationPageAt = Date.now(); + if (!continuing) { lastFullRotationAt = Date.now(); backstopArmed = false; } return service.sweepTerminalWorkspaces(50, signal, options); } const targets = { workspaceIds: [...targetWorkspaceIds], issueIds: [...targetIssueIds] }; @@ -88,9 +108,12 @@ export function registerTerminalWorkspaceWork( } }, hasPending: async () => { + continuing = false; if (!canRun()) next = Date.now() + TERMINAL_WORKSPACE_RETRY_MS; - else if (service.hasTerminalWorkspaceSweepContinuation()) next = Math.max(Date.now(), lastRotationPageAt + 1000); - else if (rotationRequested || targetWorkspaceIds.size + targetIssueIds.size > 0) next = Date.now(); + else if (service.hasTerminalWorkspaceSweepContinuation()) { + continuing = true; + next = Math.max(Date.now(), lastRotationPageAt + 1000); + } else if (rotationRequested || targetWorkspaceIds.size + targetIssueIds.size > 0) next = Date.now(); else { const deadline = await service.nextTerminalWorkspaceSweepAt(); next = deadline === null ? null : deadline > Date.now() @@ -98,6 +121,10 @@ export function registerTerminalWorkspaceWork( dueAt = next; recovered = true; } + if (backstopArmed && canRun()) { + const backstopAt = lastFullRotationAt + TERMINAL_WORKSPACE_BACKSTOP_MS; + next = next === null ? backstopAt : Math.min(next, backstopAt); + } return false; }, nextRunAt: () => next,