perf(workspaces): inspect only due workspaces and wait for a holding run's release (#21617)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep stores one deadline per workspace:
cooldown end, compaction, cleanup retry, Git idle.
> - When any deadline was due, the worker ran a full rotation over every
open candidate workspace. That costs a task tree read and more for each
one.
> - A workspace held by a `running` run kept a past deadline, so the
5-minute deadline rotation came back for it forever. A run that never
finishes would keep the instance awake indefinitely.
> - This pull request makes a due deadline inspect only the due
workspaces. A held workspace waits for its run instead of for a timer.
> - The benefit is that an idle instance does work only for workspaces
whose deadline really arrived. The cost no longer grows with the number
of open workspaces.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

How the terminal workspace worker
(`server/src/services/terminal-workspace-work.ts`) acts on due deadlines
and on workspaces held by an active run.

**Current behavior**

`hasPending` reads `nextTerminalWorkspaceSweepAt`. When the earliest
deadline is past, the next run starts a full keyset rotation, 50
workspaces per page with 1 second between pages, over every candidate. A
workspace whose issue still has a `queued` or `running` run is skipped,
but the archive deadline set earlier in the same pass stays. It is
already past, so the rotation repeats every 5 minutes.

**Proposed behavior**

- When the run fires for a due deadline, the worker takes the due
workspace ids from the service (`takeDueTerminalWorkspaceTargets`) and
runs a targeted inspection of only those workspaces. The service first
clears the deadlines of the targeted workspaces; the inspection sets
fresh ones.
- If more than 50 workspaces are due, or a due deadline belongs to the
isolated source, nested clone or task directory work, the worker still
runs a full rotation. Startup still runs a full rotation.
- A workspace skipped because a run holds it drops its deadline.
- `writeRunStatus` (`server/src/services/heartbeat/run-lifecycle.ts`)
publishes `issue:<runIssueId>` to the terminal workspace queue when a
run reaches a terminal status. That targeted inspection reaches the
workspace the run held.
- **Backstop.** About 200 code paths write `heartbeat_runs` directly
(reapers, recovery terminalization, bulk cancels), and they do not pass
through `writeRunStatus`. So the worker keeps a full rotation as a
backstop, armed only by activity, like the recovery backstop from
#21552:
  - Any terminal-workspace hint arms it.
- Once armed, a full rotation runs at most once per hour
(`TERMINAL_WORKSPACE_BACKSTOP_MS`) after the previous full rotation.
  - An idle instance with no hint schedules nothing.
- A missed release hint therefore delays archiving by at most one hour
after activity, without a restart.
- The hour is a compromise. On master today, the 5-minute rotation
re-reads every candidate. On the hosted instance that is about 1,000
finished remote workspaces, roughly 3,000 statements per lap. The hourly
backstop re-reads them at most once per hour while there is activity,
about 1 statement per second averaged.

**Reason and benefit**

Background work must be driven by deadlines and change notifications,
and its cost must not grow with the backlog. Every guarantee keeps a
named trigger:
- cooldown end, compaction, retry backoff and Git idle each have a
stored deadline, and a due deadline inspects exactly that workspace;
- the release of a holding run publishes a hint;
- task status changes and workspace writes still publish hints;
- startup still reconciles everything.

## What Changed

- `server/src/services/execution-workspaces.ts`:
  - New `takeDueTerminalWorkspaceTargets()`.
- A targeted inspection clears the deadlines of its targets before it
inspects them.
  - A workspace skipped for an active run drops its deadline.
- `server/src/services/terminal-workspace-work.ts`: a due run uses the
due targets instead of a full rotation, unless they overflow or a
non-workspace deadline is due. A due run that finds nothing due does
nothing. Any hint arms the hourly backstop rotation.
- `server/src/services/heartbeat/run-lifecycle.ts`: a terminal run
status hints the run's task to the terminal workspace queue.
- `server/src/__tests__/terminal-workspace-deadlines.test.ts`: four new
tests on real PostgreSQL. They use the real coordinator and worker on a
fake clock.
- `server/src/__tests__/terminal-workspace-idle.test.ts` (from #21616):
its idle window now bounds the number of sweeps instead of requiring
zero targeted sweeps, because a due deadline is now a targeted
inspection.

## Verification

- `inspects only the workspace whose cooldown ends, however many trees
stay open`: 60 workspaces with open task trees and 1 done workspace
whose 1-day cooldown ends 1 hour later.
- Before the change: after startup, every due deadline starts a full
rotation (27 full sweeps in 2 hours).
- After the change: no full rotation; the done workspace is inspected
alone (`checked: 1`) once its cooldown ends; fewer than 200 statements
in 2 hours.
- `waits for the release of the run that holds a workspace instead of
rechecking it`: a done workspace held by a `running` run.
- Before the change: a full rotation every 5 minutes (22 sweeps in 2
hours).
- After the change: 0 statements in 2 idle hours. When the run is set to
`succeeded` through the lifecycle, the workspace is inspected (`checked:
1`, `skippedActiveRun: 0`).
- `leaves a thousand finished remote workspaces unread at idle`. This is
the hosted state: 1,000 `active` remote workspaces with a location, a
committed snapshot and a task done 5 days ago, and zero runs.
- After startup and 2 idle hours: no full rotation. No remote workspace
is read again; the only inspections are of the one due local workspace.
Statements stay within 10 per inspection.
  - On master, every due deadline re-reads all 1,000.
- `archives a workspace whose run ended outside the run lifecycle within
the activity-armed backstop`:
- A workspace held by a `running` run. The run is set to `failed` by a
direct `UPDATE`, so no release hint is published, and one unrelated hint
arrives.
- The workspace is archived within one hour and one minute, without a
restart.
  - With the backstop scheduling disabled, it stays `active`.
- `npx vitest run src/__tests__/terminal-workspace-deadlines.test.ts
src/__tests__/terminal-workspace-work.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/git-workspace-expiration.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/nested-clone-migration.test.ts
src/__tests__/execution-workspace-compaction.test.ts
src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 333
tests pass.
- `pnpm check:module-boundaries` passes.

Local fixture with #21616 applied as the base, 30 idle minutes on a fake
clock, real PostgreSQL. Each fixture holds open trees, a stranded
`running` holder, cooldown, done local workspaces with missing files,
closed remote workspaces with and without a committed snapshot:

| Workspaces | Before: sweeps | Before: statements | After: sweeps |
After: statements |
| --- | --- | --- | --- | --- |
| 140 | 20 (full rotations) | 1,371 | 21 (targeted only) | 866 |
| 420 | 84 (full rotations) | 6,597 | 33 (targeted only) | 3,272 |

After the change, no sweep reads the task tree of a workspace whose tree
is still open (`skippedNonTerminalTree` falls from 1,440 to 0 at 420
workspaces). What remains comes only from workspaces with a due retry or
inspection backoff, so it grows with the number of stuck workspaces, not
open ones. Per inspected workspace a pass issues the same 3–6 statements
as before; batching them per page is a separate pull request.

## Risks

- **A run that leaves `running` without `writeRunStatus`** does not
hint. The activity-armed backstop catches it within one hour of
activity, and startup reconciles. This pull request does not decide that
a stranded run is gone; it waits for a status change.
- **A workspace whose deadline is reset to "now"** (a reopen-pending
flag within its grace period, task directory budget overflow) is
inspected alone on each due run. The inspection backoff in
`nextTerminalWorkspaceSweepAt` still spaces these runs.
- **Workspaces without a stored deadline** (for example an undelivered
workspace whose deadline was deleted) are inspected again only when a
hint or the startup rotation reaches them. Before this change they were
also reached by the rotation that a different workspace's deadline
started.

## Open decision

**Should a terminal workspace whose local files are gone be archived, or
should it wait?** This pull request does not change the behaviour; it
leaves the decision to the owner.

Today such a workspace fails its close-readiness inspection and is
reported as `skippedUndelivered`. A non-remote workspace keeps its past
deadline, so the 5-minute inspection backoff re-inspects it again and
again. That costs about 3–4 statements per stuck workspace every 5
minutes (from the local fixture: 255 inspections, 866 statements), and
it never ends while the files stay missing. The cause is typically a
replaced container.

The two options:
- **Archive it.** Nothing remains on disk to keep.
- **Store a "files missing" state.** It is then inspected again only
when a write announces a change.

Either option removes this cost. Archiving also removes the row from
later rotations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-14 08:23:49 -06:00
1 parent 20ccf03676
commit 081dacba1d
5 files changed
+221 -4

No files matched your search

@@ -0,0 +1,177 @@
import { randomUUID } from "node:crypto";
import { eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, companies, createDb, executionWorkspaceSnapshots, executionWorkspaces, heartbeatRuns, issues, projects, type Db } from "@paperclipai/db";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { executionWorkspaceService } from "../services/execution-workspaces.js";
import { createHeartbeatLifecycle, type HeartbeatLifecycleDependencies } from "../services/heartbeat/run-lifecycle.js";
import { notifyTerminalWorkspaceWork, registerTerminalWorkspaceWork, TERMINAL_WORKSPACE_BACKSTOP_MS, TERMINAL_WORKSPACE_RETRY_MS } from "../services/terminal-workspace-work.js";
import { meterDatabaseStatements } from "./helpers/database-statement-meter.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_TERMINAL_WORKSPACE_DEADLINE_TEST_DATABASE_URL");
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
const HOUR = 3_600_000;
const realSetTimeout = globalThis.setTimeout;
const realSleep = (ms: number) => new Promise(resolve => realSetTimeout(resolve, ms));
(support.supported ? describe : describe.skip)("terminal workspace deadlines", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: Db;
let meter: ReturnType<typeof meterDatabaseStatements>;
beforeAll(async () => {
if (!external) temporary = await startEmbeddedPostgresTestDatabase("terminal-workspace-deadlines-");
const url = external ?? temporary!.connectionString;
db = createDb(url);
db.$client.options.onnotice = () => {};
meter = meterDatabaseStatements(db, url);
}, 120_000);
afterEach(async () => { vi.useRealTimers(); await db.execute(sql`truncate companies cascade`); });
afterAll(async () => { meter?.uninstall(); await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
async function seed(input: { open: number; cooldownEndsInMs?: number; heldByRunningRun?: boolean }) {
const companyId = randomUUID(), projectId = randomUUID(), agentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Deadlines", issuePrefix: `D${companyId.slice(0, 6)}` });
await db.insert(projects).values({ id: projectId, companyId, name: "Project", status: "in_progress" });
await db.insert(agents).values({ id: agentId, companyId, name: "Agent", role: "engineer", status: "idle", adapterType: "process" });
const old = new Date(Date.now() - 48 * HOUR);
const workspace = async (status: string, completedAt: Date | null, runId: string | null) => {
const id = randomUUID(), issueId = randomUUID();
await db.insert(executionWorkspaces).values({ id, companyId, projectId, mode: "shared_workspace", strategyType: "project_primary",
name: id, status: "active", providerType: "local_fs", cwd: `/missing/${id}`, createdAt: old, updatedAt: old });
await db.insert(issues).values({ id: issueId, companyId, projectId, title: status, status, executionWorkspaceId: id,
executionRunId: runId, completedAt, updatedAt: old });
await db.update(executionWorkspaces).set({ sourceIssueId: issueId, updatedAt: old }).where(eq(executionWorkspaces.id, id));
return { id, issueId };
};
for (let index = 0; index < input.open; index++) await workspace("in_progress", null, null);
let runId: string | null = null;
if (input.heldByRunningRun) {
runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running" });
}
const target = await workspace("done", new Date(Date.now() - 24 * HOUR + (input.cooldownEndsInMs ?? -HOUR)), runId);
if (runId) await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: target.issueId } }).where(eq(heartbeatRuns.id, runId));
return { companyId, runId, target };
}
function start(options: Parameters<typeof executionWorkspaceService>[1] = {}) {
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
const service = executionWorkspaceService(db, { workspaceReaperCooldownDays: 1, workspaceCompactionDelayMs: HOUR,
terminalWorkspaceInspectionBackoffMs: TERMINAL_WORKSPACE_RETRY_MS, ...options });
const sweeps: Array<{ targets: string[] | null; checked: number; skippedCooldown: number; skippedActiveRun: number }> = [];
let inFlight = 0;
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: () => {} });
const worker = registerTerminalWorkspaceWork(coordinator, { ...service,
sweepTerminalWorkspaces: async (limit, signal, options) => {
inFlight += 1;
const result = await service.sweepTerminalWorkspaces(limit, signal, options).finally(() => { inFlight -= 1; });
sweeps.push({ targets: options?.targets ? [...options.targets.workspaceIds, ...options.targets.issueIds] : null,
checked: result.checked, skippedCooldown: (result as { skippedCooldown: number }).skippedCooldown,
skippedActiveRun: (result as { skippedActiveRun: number }).skippedActiveRun });
return result;
} }, () => true);
const advance = async (ms: number) => {
for (let elapsed = 0; elapsed < ms; elapsed += 10_000) {
await vi.advanceTimersByTimeAsync(10_000);
do { await realSleep(10); await vi.advanceTimersByTimeAsync(1); } while (inFlight > 0);
}
};
const settle = async <T>(operation: Promise<T>) => {
let done = false;
const result = operation.finally(() => { done = true; });
while (!done) { await realSleep(5); await vi.advanceTimersByTimeAsync(1); }
return result;
};
return { sweeps, coordinator, worker, advance, settle, stop: async () => { vi.useRealTimers(); await worker.stop(); await coordinator.stop(); } };
}
it("inspects only the workspace whose cooldown ends, however many trees stay open", async () => {
const f = await seed({ open: 60, cooldownEndsInMs: HOUR });
const work = start();
try {
await work.advance(5 * 60_000);
const startup = work.sweeps.length;
expect(startup).toBeGreaterThan(0);
expect(work.sweeps.at(-1)).toMatchObject({ targets: null });
const before = meter.read().statements;
await work.advance(2 * HOUR);
const window = work.sweeps.slice(startup);
expect(window.filter(sweep => sweep.targets === null)).toEqual([]);
expect(window.some(sweep => sweep.targets?.includes(f.target.id) && sweep.checked === 1 && sweep.skippedCooldown === 0)).toBe(true);
expect(window.reduce((sum, sweep) => sum + sweep.checked, 0)).toBeLessThanOrEqual(window.length);
expect(meter.read().statements - before).toBeLessThan(200);
} finally { await work.stop(); }
}, 300_000);
it("leaves a thousand finished remote workspaces unread at idle", async () => {
const f = await seed({ open: 0, cooldownEndsInMs: 30 * 60_000 });
const [target] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.target.id));
const old = new Date(Date.now() - 5 * 24 * HOUR);
const rows = Array.from({ length: 1000 }, () => ({ workspaceId: randomUUID(), issueId: randomUUID(), snapshotId: randomUUID() }));
for (let index = 0; index < rows.length; index += 250) {
const page = rows.slice(index, index + 250);
await db.insert(executionWorkspaces).values(page.map(row => ({ id: row.workspaceId, companyId: f.companyId, projectId: target!.projectId,
mode: "isolated_workspace", strategyType: "git_worktree", name: row.workspaceId, status: "active", providerType: "remote",
createdAt: old, updatedAt: old, metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1,
snapshotId: row.snapshotId, source: { kind: "empty" }, writerRunId: null, dirty: false,
location: { environmentId: randomUUID(), leaseId: randomUUID(), providerLeaseId: `sandbox-${row.workspaceId}`, remoteCwd: "/workspace" } } } })));
await db.insert(issues).values(page.map(row => ({ id: row.issueId, companyId: f.companyId, projectId: target!.projectId, title: "done",
status: "done", executionWorkspaceId: row.workspaceId, completedAt: old, updatedAt: old })));
await db.insert(executionWorkspaceSnapshots).values(page.map(row => ({ id: row.snapshotId, companyId: f.companyId,
executionWorkspaceId: row.workspaceId, sourceOperationId: randomUUID(), generation: 1, epoch: 1, status: "committed" as const })));
for (const row of page) await db.update(executionWorkspaces).set({ sourceIssueId: row.issueId, updatedAt: old })
.where(eq(executionWorkspaces.id, row.workspaceId));
}
const work = start();
try {
await work.advance(5 * 60_000);
const startup = work.sweeps.length;
expect(work.sweeps.reduce((sum, sweep) => sum + sweep.checked, 0)).toBe(1001);
const before = meter.read().statements;
await work.advance(2 * HOUR);
const window = work.sweeps.slice(startup);
expect(window.filter(sweep => sweep.targets === null)).toEqual([]);
expect(window.every(sweep => sweep.targets?.length === 1 && sweep.targets[0] === f.target.id && sweep.checked === 1)).toBe(true);
expect(meter.read().statements - before).toBeLessThan(10 * Math.max(window.length, 1));
} finally { await work.stop(); }
}, 300_000);
it("archives a workspace whose run ended outside the run lifecycle within the activity-armed backstop", async () => {
const f = await seed({ open: 10, heldByRunningRun: true });
const work = start({ inspectGitCloseReadiness: async workspace => ({ statusInspectionSucceeded: true, warnings: [], git: {
repoRoot: null, workspacePath: workspace.cwd, branchName: null, baseRef: null, hasDirtyTrackedFiles: false, hasUntrackedFiles: false,
dirtyEntryCount: 0, untrackedEntryCount: 0, aheadCount: 0, behindCount: 0, isMergedIntoBase: true, createdByRuntime: false } }) });
try {
await work.advance(5 * 60_000);
expect(work.sweeps.at(-1)).toMatchObject({ targets: null, skippedActiveRun: 1 });
await work.settle(db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId!)));
await work.settle(db.transaction(tx => notifyTerminalWorkspaceWork(tx, `workspace:${randomUUID()}`)));
const releasedAt = Date.now();
await work.advance(TERMINAL_WORKSPACE_BACKSTOP_MS + 60_000);
const [row] = await work.settle(db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.target.id)));
expect(row!.status).toBe("archived");
expect(row!.closedAt!.getTime() - releasedAt).toBeLessThanOrEqual(TERMINAL_WORKSPACE_BACKSTOP_MS + 60_000);
} finally { await work.stop(); }
}, 300_000);
it("waits for the release of the run that holds a workspace instead of rechecking it", async () => {
const f = await seed({ open: 10, heldByRunningRun: true });
const work = start();
try {
await work.advance(5 * 60_000);
expect(work.sweeps.at(-1)).toMatchObject({ targets: null, skippedActiveRun: 1 });
const startup = work.sweeps.length;
const before = meter.read().statements;
await work.advance(2 * HOUR);
expect(work.sweeps.slice(startup)).toEqual([]);
expect(meter.read().statements - before).toBe(0);
vi.useRealTimers();
await createHeartbeatLifecycle(db, { emitTerminalAgentTaskRun: vi.fn() } as unknown as HeartbeatLifecycleDependencies)
.setRunStatus(f.runId!, "succeeded", { finishedAt: new Date() });
await vi.waitFor(() => expect(work.sweeps.slice(startup).some(sweep => sweep.targets?.includes(f.target.issueId)
&& sweep.checked === 1 && sweep.skippedActiveRun === 0)).toBe(true), { timeout: 20_000 });
} finally { await work.stop(); }
}, 300_000);
});
@@ -118,7 +118,7 @@ const realSleep = (ms: number) => new Promise(resolve => realSetTimeout(resolve,
vi.useRealTimers();
const window = sweeps.slice(before.sweeps);
expect(destroyCalls.length).toBe(before.destroys);
expect(window.filter(sweep => sweep.targeted)).toEqual([]);
expect(window.length).toBeLessThanOrEqual(12);
expect(meter.read().statements - before.statements).toBeLessThan(5_000);
for (const id of remote) {
const [row] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, id));
@@ -3660,6 +3660,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
compacted: 0,
compactionFailed: 0,
};
if (targets) for (const id of [...targets.workspaceIds, ...candidates.map(candidate => candidate.workspace.id)]) terminalWorkspaceDeadlines.delete(id);
const readinessByWorkingTree = new Map<string, ReturnType<typeof inspectWorkspace>>();
const backOffTerminalWorkspace = async (snapshot: ExecutionWorkspaceRow, failure: TerminalWorkspaceFailure) => {
const [target] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, snapshot.id));
@@ -3914,6 +3915,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
if (await workspaceHasActiveRun(workspace)) {
if (reopenPending) result.skippedReopened += 1;
else result.skippedActiveRun += 1;
terminalWorkspaceDeadlines.delete(workspace.id);
continue;
}
if (remoteWorkspace && isClosedExecutionWorkspaceStatus(workspace.status)) {
@@ -4154,6 +4156,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
}
if (await workspaceHasActiveRun(workspace)) {
result.skippedActiveRun += 1;
terminalWorkspaceDeadlines.delete(workspace.id);
continue;
}
result.eligible += 1;
@@ -4364,6 +4367,13 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
}
},
takeDueTerminalWorkspaceTargets: (): { workspaceIds: string[]; rotation: boolean } => {
const at = now().getTime();
const due = (deadlines: Map<string, number>) => [...deadlines].filter(([, deadline]) => deadline <= at).map(([id]) => id);
const rotation = due(isolatedRehomeDeadlines).length + due(nestedCloneMigrationDeadlines).length + due(taskDirectoryDeadlines).length > 0;
return { workspaceIds: due(terminalWorkspaceDeadlines), rotation };
},
nextTerminalWorkspaceSweepAt: async () => {
if (terminalWorkspaceDeadlines.size === 0 && isolatedRehomeDeadlines.size === 0 && nestedCloneMigrationDeadlines.size === 0
&& taskDirectoryDeadlines.size === 0) return null;
@@ -1,3 +1,4 @@
import { notifyTerminalWorkspaceWork } from "../terminal-workspace-work.js";
import { settleReattachRun } from "./run-admission.js";
import { readNativeStartupCleanup } from "../native-runtime/native-startup-cleanup.js";
import { isContextCompactionRun } from "../context-compaction.js";
@@ -1122,6 +1123,8 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
if (previous && isHeartbeatRunTerminalStatus(status))
await notifyRetainedNativeSessionCleanup(tx, { companyId: previous.companyId, runId: previous.id });
if (isHeartbeatRunTerminalStatus(status)) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, companyId);
const runIssueId = previous ? previous.nativeIssueId ?? previous.issueId ?? readNonEmptyString(parseObject(previous.contextSnapshot).issueId) : null;
if (previous && isHeartbeatRunTerminalStatus(status) && runIssueId) await notifyTerminalWorkspaceWork(tx, `issue:${runIssueId}`);
await notifyExecutionRunWork(tx, previous ? { companyId: previous.companyId, runId: previous.id,
issueId: previous.nativeIssueId ?? previous.issueId ?? readNonEmptyString(parseObject(previous.contextSnapshot).issueId) ?? undefined } : undefined);
if (previous) await notifyChatRunMilestones(tx, { companyId: previous.companyId, runId: previous.id,
+30 -3
View File
@@ -3,6 +3,7 @@ import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notificatio
export const TERMINAL_WORKSPACE_RETRY_MS = 5 * 60 * 1000;
export const MAX_TERMINAL_WORKSPACE_TARGETS = 50;
export const TERMINAL_WORKSPACE_BACKSTOP_MS = 60 * 60 * 1000;
export function notifyTerminalWorkspaceWork(transaction: object, hint?: string) {
return notifyDeliveryWork(transaction, DELIVERY_QUEUES.terminalWorkspaces, hint);
@@ -36,6 +37,7 @@ export function registerTerminalWorkspaceWork(
nextTerminalWorkspaceSweepAt(): Promise<number | null>;
hasTerminalWorkspaceSweepContinuation(): boolean;
terminalWorkspaceWorkSettled?(hints?: ReadonlySet<string>, options?: { targeted?: boolean }): void;
takeDueTerminalWorkspaceTargets?(): { workspaceIds: string[]; rotation: boolean };
},
canRun: () => boolean,
) {
@@ -49,10 +51,14 @@ export function registerTerminalWorkspaceWork(
let rotationRequested = false;
let dueAt: number | null = null;
let lastRotationPageAt = 0;
let backstopArmed = false;
let lastFullRotationAt = Date.now();
let continuing = false;
const targetWorkspaceIds = new Set<string>(), targetIssueIds = new Set<string>();
const worker = coordinator.register(DELIVERY_QUEUES.terminalWorkspaces, {
retryMs: TERMINAL_WORKSPACE_RETRY_MS,
onSettled: hints => {
backstopArmed = true;
const targets = terminalWorkspaceTargetsFromHints(hints);
for (const id of targets?.workspaceIds ?? []) targetWorkspaceIds.add(id);
for (const id of targets?.issueIds ?? []) targetIssueIds.add(id);
@@ -70,12 +76,26 @@ export function registerTerminalWorkspaceWork(
next = Date.now() + TERMINAL_WORKSPACE_RETRY_MS;
const startup = !startupSweepStarted;
startupSweepStarted = true;
let dueRotation = due;
if (due && !startup && service.takeDueTerminalWorkspaceTargets) {
const dueTargets = service.takeDueTerminalWorkspaceTargets();
dueRotation = dueTargets.rotation;
for (const id of dueTargets.workspaceIds) targetWorkspaceIds.add(id);
if (targetWorkspaceIds.size + targetIssueIds.size > MAX_TERMINAL_WORKSPACE_TARGETS) {
dueRotation = true;
targetWorkspaceIds.clear(); targetIssueIds.clear();
}
}
const options = { compact: !startup, allowRemoteDeliveryChecks: !startup };
const targeted = targetWorkspaceIds.size + targetIssueIds.size > 0;
if (startup || due || rotationRequested || !targeted
const backstopDue = backstopArmed && Date.now() >= lastFullRotationAt + TERMINAL_WORKSPACE_BACKSTOP_MS;
if (due && service.takeDueTerminalWorkspaceTargets && !startup && !dueRotation && !targeted && !rotationRequested && !backstopDue
&& !service.hasTerminalWorkspaceSweepContinuation()) return { checked: 0 };
if (startup || dueRotation || rotationRequested || backstopDue || !targeted
|| (service.hasTerminalWorkspaceSweepContinuation() && Date.now() >= lastRotationPageAt + 1000)) {
rotationRequested = false;
lastRotationPageAt = Date.now();
if (!continuing) { lastFullRotationAt = Date.now(); backstopArmed = false; }
return service.sweepTerminalWorkspaces(50, signal, options);
}
const targets = { workspaceIds: [...targetWorkspaceIds], issueIds: [...targetIssueIds] };
@@ -88,9 +108,12 @@ export function registerTerminalWorkspaceWork(
}
},
hasPending: async () => {
continuing = false;
if (!canRun()) next = Date.now() + TERMINAL_WORKSPACE_RETRY_MS;
else if (service.hasTerminalWorkspaceSweepContinuation()) next = Math.max(Date.now(), lastRotationPageAt + 1000);
else if (rotationRequested || targetWorkspaceIds.size + targetIssueIds.size > 0) next = Date.now();
else if (service.hasTerminalWorkspaceSweepContinuation()) {
continuing = true;
next = Math.max(Date.now(), lastRotationPageAt + 1000);
} else if (rotationRequested || targetWorkspaceIds.size + targetIssueIds.size > 0) next = Date.now();
else {
const deadline = await service.nextTerminalWorkspaceSweepAt();
next = deadline === null ? null : deadline > Date.now()
@@ -98,6 +121,10 @@ export function registerTerminalWorkspaceWork(
dueAt = next;
recovered = true;
}
if (backstopArmed && canRun()) {
const backstopAt = lastFullRotationAt + TERMINAL_WORKSPACE_BACKSTOP_MS;
next = next === null ? backstopAt : Math.min(next, backstopAt);
}
return false;
},
nextRunAt: () => next,