mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
perf(workspaces): inspect only due workspaces and wait for a holding run's release (#21617)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The terminal workspace sweep stores one deadline per workspace: cooldown end, compaction, cleanup retry, Git idle. > - When any deadline was due, the worker ran a full rotation over every open candidate workspace. That costs a task tree read and more for each one. > - A workspace held by a `running` run kept a past deadline, so the 5-minute deadline rotation came back for it forever. A run that never finishes would keep the instance awake indefinitely. > - This pull request makes a due deadline inspect only the due workspaces. A held workspace waits for its run instead of for a timer. > - The benefit is that an idle instance does work only for workspaces whose deadline really arrived. The cost no longer grows with the number of open workspaces. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** How the terminal workspace worker (`server/src/services/terminal-workspace-work.ts`) acts on due deadlines and on workspaces held by an active run. **Current behavior** `hasPending` reads `nextTerminalWorkspaceSweepAt`. When the earliest deadline is past, the next run starts a full keyset rotation, 50 workspaces per page with 1 second between pages, over every candidate. A workspace whose issue still has a `queued` or `running` run is skipped, but the archive deadline set earlier in the same pass stays. It is already past, so the rotation repeats every 5 minutes. **Proposed behavior** - When the run fires for a due deadline, the worker takes the due workspace ids from the service (`takeDueTerminalWorkspaceTargets`) and runs a targeted inspection of only those workspaces. The service first clears the deadlines of the targeted workspaces; the inspection sets fresh ones. - If more than 50 workspaces are due, or a due deadline belongs to the isolated source, nested clone or task directory work, the worker still runs a full rotation. Startup still runs a full rotation. - A workspace skipped because a run holds it drops its deadline. - `writeRunStatus` (`server/src/services/heartbeat/run-lifecycle.ts`) publishes `issue:<runIssueId>` to the terminal workspace queue when a run reaches a terminal status. That targeted inspection reaches the workspace the run held. - **Backstop.** About 200 code paths write `heartbeat_runs` directly (reapers, recovery terminalization, bulk cancels), and they do not pass through `writeRunStatus`. So the worker keeps a full rotation as a backstop, armed only by activity, like the recovery backstop from #21552: - Any terminal-workspace hint arms it. - Once armed, a full rotation runs at most once per hour (`TERMINAL_WORKSPACE_BACKSTOP_MS`) after the previous full rotation. - An idle instance with no hint schedules nothing. - A missed release hint therefore delays archiving by at most one hour after activity, without a restart. - The hour is a compromise. On master today, the 5-minute rotation re-reads every candidate. On the hosted instance that is about 1,000 finished remote workspaces, roughly 3,000 statements per lap. The hourly backstop re-reads them at most once per hour while there is activity, about 1 statement per second averaged. **Reason and benefit** Background work must be driven by deadlines and change notifications, and its cost must not grow with the backlog. Every guarantee keeps a named trigger: - cooldown end, compaction, retry backoff and Git idle each have a stored deadline, and a due deadline inspects exactly that workspace; - the release of a holding run publishes a hint; - task status changes and workspace writes still publish hints; - startup still reconciles everything. ## What Changed - `server/src/services/execution-workspaces.ts`: - New `takeDueTerminalWorkspaceTargets()`. - A targeted inspection clears the deadlines of its targets before it inspects them. - A workspace skipped for an active run drops its deadline. - `server/src/services/terminal-workspace-work.ts`: a due run uses the due targets instead of a full rotation, unless they overflow or a non-workspace deadline is due. A due run that finds nothing due does nothing. Any hint arms the hourly backstop rotation. - `server/src/services/heartbeat/run-lifecycle.ts`: a terminal run status hints the run's task to the terminal workspace queue. - `server/src/__tests__/terminal-workspace-deadlines.test.ts`: four new tests on real PostgreSQL. They use the real coordinator and worker on a fake clock. - `server/src/__tests__/terminal-workspace-idle.test.ts` (from #21616): its idle window now bounds the number of sweeps instead of requiring zero targeted sweeps, because a due deadline is now a targeted inspection. ## Verification - `inspects only the workspace whose cooldown ends, however many trees stay open`: 60 workspaces with open task trees and 1 done workspace whose 1-day cooldown ends 1 hour later. - Before the change: after startup, every due deadline starts a full rotation (27 full sweeps in 2 hours). - After the change: no full rotation; the done workspace is inspected alone (`checked: 1`) once its cooldown ends; fewer than 200 statements in 2 hours. - `waits for the release of the run that holds a workspace instead of rechecking it`: a done workspace held by a `running` run. - Before the change: a full rotation every 5 minutes (22 sweeps in 2 hours). - After the change: 0 statements in 2 idle hours. When the run is set to `succeeded` through the lifecycle, the workspace is inspected (`checked: 1`, `skippedActiveRun: 0`). - `leaves a thousand finished remote workspaces unread at idle`. This is the hosted state: 1,000 `active` remote workspaces with a location, a committed snapshot and a task done 5 days ago, and zero runs. - After startup and 2 idle hours: no full rotation. No remote workspace is read again; the only inspections are of the one due local workspace. Statements stay within 10 per inspection. - On master, every due deadline re-reads all 1,000. - `archives a workspace whose run ended outside the run lifecycle within the activity-armed backstop`: - A workspace held by a `running` run. The run is set to `failed` by a direct `UPDATE`, so no release hint is published, and one unrelated hint arrives. - The workspace is archived within one hour and one minute, without a restart. - With the backstop scheduling disabled, it stays `active`. - `npx vitest run src/__tests__/terminal-workspace-deadlines.test.ts src/__tests__/terminal-workspace-work.test.ts src/__tests__/execution-workspaces-service.test.ts src/__tests__/isolated-task-directory-retention.test.ts src/__tests__/git-workspace-expiration.test.ts src/__tests__/workspace-delivery-cleanup.integration.test.ts src/__tests__/nested-clone-migration.test.ts src/__tests__/execution-workspace-compaction.test.ts src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 333 tests pass. - `pnpm check:module-boundaries` passes. Local fixture with #21616 applied as the base, 30 idle minutes on a fake clock, real PostgreSQL. Each fixture holds open trees, a stranded `running` holder, cooldown, done local workspaces with missing files, closed remote workspaces with and without a committed snapshot: | Workspaces | Before: sweeps | Before: statements | After: sweeps | After: statements | | --- | --- | --- | --- | --- | | 140 | 20 (full rotations) | 1,371 | 21 (targeted only) | 866 | | 420 | 84 (full rotations) | 6,597 | 33 (targeted only) | 3,272 | After the change, no sweep reads the task tree of a workspace whose tree is still open (`skippedNonTerminalTree` falls from 1,440 to 0 at 420 workspaces). What remains comes only from workspaces with a due retry or inspection backoff, so it grows with the number of stuck workspaces, not open ones. Per inspected workspace a pass issues the same 3–6 statements as before; batching them per page is a separate pull request. ## Risks - **A run that leaves `running` without `writeRunStatus`** does not hint. The activity-armed backstop catches it within one hour of activity, and startup reconciles. This pull request does not decide that a stranded run is gone; it waits for a status change. - **A workspace whose deadline is reset to "now"** (a reopen-pending flag within its grace period, task directory budget overflow) is inspected alone on each due run. The inspection backoff in `nextTerminalWorkspaceSweepAt` still spaces these runs. - **Workspaces without a stored deadline** (for example an undelivered workspace whose deadline was deleted) are inspected again only when a hint or the startup rotation reaches them. Before this change they were also reached by the rotation that a different workspace's deadline started. ## Open decision **Should a terminal workspace whose local files are gone be archived, or should it wait?** This pull request does not change the behaviour; it leaves the decision to the owner. Today such a workspace fails its close-readiness inspection and is reported as `skippedUndelivered`. A non-remote workspace keeps its past deadline, so the 5-minute inspection backoff re-inspects it again and again. That costs about 3–4 statements per stuck workspace every 5 minutes (from the local fixture: 255 inspections, 866 statements), and it never ends while the files stay missing. The cause is typically a replaced container. The two options: - **Archive it.** Nothing remains on disk to keep. - **Store a "files missing" state.** It is then inspected again only when a write announces a change. Either option removes this cost. Archiving also removes the row from later rotations. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
20ccf03676
commit
081dacba1d
5 files changed
+221
-4
No files matched your search
@@ -0,0 +1,177 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, companies, createDb, executionWorkspaceSnapshots, executionWorkspaces, heartbeatRuns, issues, projects, type Db } from "@paperclipai/db";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { executionWorkspaceService } from "../services/execution-workspaces.js";
|
||||
import { createHeartbeatLifecycle, type HeartbeatLifecycleDependencies } from "../services/heartbeat/run-lifecycle.js";
|
||||
import { notifyTerminalWorkspaceWork, registerTerminalWorkspaceWork, TERMINAL_WORKSPACE_BACKSTOP_MS, TERMINAL_WORKSPACE_RETRY_MS } from "../services/terminal-workspace-work.js";
|
||||
import { meterDatabaseStatements } from "./helpers/database-statement-meter.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_TERMINAL_WORKSPACE_DEADLINE_TEST_DATABASE_URL");
|
||||
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
const HOUR = 3_600_000;
|
||||
const realSetTimeout = globalThis.setTimeout;
|
||||
const realSleep = (ms: number) => new Promise(resolve => realSetTimeout(resolve, ms));
|
||||
|
||||
(support.supported ? describe : describe.skip)("terminal workspace deadlines", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: Db;
|
||||
let meter: ReturnType<typeof meterDatabaseStatements>;
|
||||
beforeAll(async () => {
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("terminal-workspace-deadlines-");
|
||||
const url = external ?? temporary!.connectionString;
|
||||
db = createDb(url);
|
||||
db.$client.options.onnotice = () => {};
|
||||
meter = meterDatabaseStatements(db, url);
|
||||
}, 120_000);
|
||||
afterEach(async () => { vi.useRealTimers(); await db.execute(sql`truncate companies cascade`); });
|
||||
afterAll(async () => { meter?.uninstall(); await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
|
||||
|
||||
async function seed(input: { open: number; cooldownEndsInMs?: number; heldByRunningRun?: boolean }) {
|
||||
const companyId = randomUUID(), projectId = randomUUID(), agentId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Deadlines", issuePrefix: `D${companyId.slice(0, 6)}` });
|
||||
await db.insert(projects).values({ id: projectId, companyId, name: "Project", status: "in_progress" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Agent", role: "engineer", status: "idle", adapterType: "process" });
|
||||
const old = new Date(Date.now() - 48 * HOUR);
|
||||
const workspace = async (status: string, completedAt: Date | null, runId: string | null) => {
|
||||
const id = randomUUID(), issueId = randomUUID();
|
||||
await db.insert(executionWorkspaces).values({ id, companyId, projectId, mode: "shared_workspace", strategyType: "project_primary",
|
||||
name: id, status: "active", providerType: "local_fs", cwd: `/missing/${id}`, createdAt: old, updatedAt: old });
|
||||
await db.insert(issues).values({ id: issueId, companyId, projectId, title: status, status, executionWorkspaceId: id,
|
||||
executionRunId: runId, completedAt, updatedAt: old });
|
||||
await db.update(executionWorkspaces).set({ sourceIssueId: issueId, updatedAt: old }).where(eq(executionWorkspaces.id, id));
|
||||
return { id, issueId };
|
||||
};
|
||||
for (let index = 0; index < input.open; index++) await workspace("in_progress", null, null);
|
||||
let runId: string | null = null;
|
||||
if (input.heldByRunningRun) {
|
||||
runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running" });
|
||||
}
|
||||
const target = await workspace("done", new Date(Date.now() - 24 * HOUR + (input.cooldownEndsInMs ?? -HOUR)), runId);
|
||||
if (runId) await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: target.issueId } }).where(eq(heartbeatRuns.id, runId));
|
||||
return { companyId, runId, target };
|
||||
}
|
||||
|
||||
function start(options: Parameters<typeof executionWorkspaceService>[1] = {}) {
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
|
||||
const service = executionWorkspaceService(db, { workspaceReaperCooldownDays: 1, workspaceCompactionDelayMs: HOUR,
|
||||
terminalWorkspaceInspectionBackoffMs: TERMINAL_WORKSPACE_RETRY_MS, ...options });
|
||||
const sweeps: Array<{ targets: string[] | null; checked: number; skippedCooldown: number; skippedActiveRun: number }> = [];
|
||||
let inFlight = 0;
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: () => {} });
|
||||
const worker = registerTerminalWorkspaceWork(coordinator, { ...service,
|
||||
sweepTerminalWorkspaces: async (limit, signal, options) => {
|
||||
inFlight += 1;
|
||||
const result = await service.sweepTerminalWorkspaces(limit, signal, options).finally(() => { inFlight -= 1; });
|
||||
sweeps.push({ targets: options?.targets ? [...options.targets.workspaceIds, ...options.targets.issueIds] : null,
|
||||
checked: result.checked, skippedCooldown: (result as { skippedCooldown: number }).skippedCooldown,
|
||||
skippedActiveRun: (result as { skippedActiveRun: number }).skippedActiveRun });
|
||||
return result;
|
||||
} }, () => true);
|
||||
const advance = async (ms: number) => {
|
||||
for (let elapsed = 0; elapsed < ms; elapsed += 10_000) {
|
||||
await vi.advanceTimersByTimeAsync(10_000);
|
||||
do { await realSleep(10); await vi.advanceTimersByTimeAsync(1); } while (inFlight > 0);
|
||||
}
|
||||
};
|
||||
const settle = async <T>(operation: Promise<T>) => {
|
||||
let done = false;
|
||||
const result = operation.finally(() => { done = true; });
|
||||
while (!done) { await realSleep(5); await vi.advanceTimersByTimeAsync(1); }
|
||||
return result;
|
||||
};
|
||||
return { sweeps, coordinator, worker, advance, settle, stop: async () => { vi.useRealTimers(); await worker.stop(); await coordinator.stop(); } };
|
||||
}
|
||||
|
||||
it("inspects only the workspace whose cooldown ends, however many trees stay open", async () => {
|
||||
const f = await seed({ open: 60, cooldownEndsInMs: HOUR });
|
||||
const work = start();
|
||||
try {
|
||||
await work.advance(5 * 60_000);
|
||||
const startup = work.sweeps.length;
|
||||
expect(startup).toBeGreaterThan(0);
|
||||
expect(work.sweeps.at(-1)).toMatchObject({ targets: null });
|
||||
const before = meter.read().statements;
|
||||
await work.advance(2 * HOUR);
|
||||
const window = work.sweeps.slice(startup);
|
||||
expect(window.filter(sweep => sweep.targets === null)).toEqual([]);
|
||||
expect(window.some(sweep => sweep.targets?.includes(f.target.id) && sweep.checked === 1 && sweep.skippedCooldown === 0)).toBe(true);
|
||||
expect(window.reduce((sum, sweep) => sum + sweep.checked, 0)).toBeLessThanOrEqual(window.length);
|
||||
expect(meter.read().statements - before).toBeLessThan(200);
|
||||
} finally { await work.stop(); }
|
||||
}, 300_000);
|
||||
|
||||
it("leaves a thousand finished remote workspaces unread at idle", async () => {
|
||||
const f = await seed({ open: 0, cooldownEndsInMs: 30 * 60_000 });
|
||||
const [target] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.target.id));
|
||||
const old = new Date(Date.now() - 5 * 24 * HOUR);
|
||||
const rows = Array.from({ length: 1000 }, () => ({ workspaceId: randomUUID(), issueId: randomUUID(), snapshotId: randomUUID() }));
|
||||
for (let index = 0; index < rows.length; index += 250) {
|
||||
const page = rows.slice(index, index + 250);
|
||||
await db.insert(executionWorkspaces).values(page.map(row => ({ id: row.workspaceId, companyId: f.companyId, projectId: target!.projectId,
|
||||
mode: "isolated_workspace", strategyType: "git_worktree", name: row.workspaceId, status: "active", providerType: "remote",
|
||||
createdAt: old, updatedAt: old, metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1,
|
||||
snapshotId: row.snapshotId, source: { kind: "empty" }, writerRunId: null, dirty: false,
|
||||
location: { environmentId: randomUUID(), leaseId: randomUUID(), providerLeaseId: `sandbox-${row.workspaceId}`, remoteCwd: "/workspace" } } } })));
|
||||
await db.insert(issues).values(page.map(row => ({ id: row.issueId, companyId: f.companyId, projectId: target!.projectId, title: "done",
|
||||
status: "done", executionWorkspaceId: row.workspaceId, completedAt: old, updatedAt: old })));
|
||||
await db.insert(executionWorkspaceSnapshots).values(page.map(row => ({ id: row.snapshotId, companyId: f.companyId,
|
||||
executionWorkspaceId: row.workspaceId, sourceOperationId: randomUUID(), generation: 1, epoch: 1, status: "committed" as const })));
|
||||
for (const row of page) await db.update(executionWorkspaces).set({ sourceIssueId: row.issueId, updatedAt: old })
|
||||
.where(eq(executionWorkspaces.id, row.workspaceId));
|
||||
}
|
||||
const work = start();
|
||||
try {
|
||||
await work.advance(5 * 60_000);
|
||||
const startup = work.sweeps.length;
|
||||
expect(work.sweeps.reduce((sum, sweep) => sum + sweep.checked, 0)).toBe(1001);
|
||||
const before = meter.read().statements;
|
||||
await work.advance(2 * HOUR);
|
||||
const window = work.sweeps.slice(startup);
|
||||
expect(window.filter(sweep => sweep.targets === null)).toEqual([]);
|
||||
expect(window.every(sweep => sweep.targets?.length === 1 && sweep.targets[0] === f.target.id && sweep.checked === 1)).toBe(true);
|
||||
expect(meter.read().statements - before).toBeLessThan(10 * Math.max(window.length, 1));
|
||||
} finally { await work.stop(); }
|
||||
}, 300_000);
|
||||
|
||||
it("archives a workspace whose run ended outside the run lifecycle within the activity-armed backstop", async () => {
|
||||
const f = await seed({ open: 10, heldByRunningRun: true });
|
||||
const work = start({ inspectGitCloseReadiness: async workspace => ({ statusInspectionSucceeded: true, warnings: [], git: {
|
||||
repoRoot: null, workspacePath: workspace.cwd, branchName: null, baseRef: null, hasDirtyTrackedFiles: false, hasUntrackedFiles: false,
|
||||
dirtyEntryCount: 0, untrackedEntryCount: 0, aheadCount: 0, behindCount: 0, isMergedIntoBase: true, createdByRuntime: false } }) });
|
||||
try {
|
||||
await work.advance(5 * 60_000);
|
||||
expect(work.sweeps.at(-1)).toMatchObject({ targets: null, skippedActiveRun: 1 });
|
||||
await work.settle(db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId!)));
|
||||
await work.settle(db.transaction(tx => notifyTerminalWorkspaceWork(tx, `workspace:${randomUUID()}`)));
|
||||
const releasedAt = Date.now();
|
||||
await work.advance(TERMINAL_WORKSPACE_BACKSTOP_MS + 60_000);
|
||||
const [row] = await work.settle(db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.target.id)));
|
||||
expect(row!.status).toBe("archived");
|
||||
expect(row!.closedAt!.getTime() - releasedAt).toBeLessThanOrEqual(TERMINAL_WORKSPACE_BACKSTOP_MS + 60_000);
|
||||
} finally { await work.stop(); }
|
||||
}, 300_000);
|
||||
|
||||
it("waits for the release of the run that holds a workspace instead of rechecking it", async () => {
|
||||
const f = await seed({ open: 10, heldByRunningRun: true });
|
||||
const work = start();
|
||||
try {
|
||||
await work.advance(5 * 60_000);
|
||||
expect(work.sweeps.at(-1)).toMatchObject({ targets: null, skippedActiveRun: 1 });
|
||||
const startup = work.sweeps.length;
|
||||
const before = meter.read().statements;
|
||||
await work.advance(2 * HOUR);
|
||||
expect(work.sweeps.slice(startup)).toEqual([]);
|
||||
expect(meter.read().statements - before).toBe(0);
|
||||
vi.useRealTimers();
|
||||
await createHeartbeatLifecycle(db, { emitTerminalAgentTaskRun: vi.fn() } as unknown as HeartbeatLifecycleDependencies)
|
||||
.setRunStatus(f.runId!, "succeeded", { finishedAt: new Date() });
|
||||
await vi.waitFor(() => expect(work.sweeps.slice(startup).some(sweep => sweep.targets?.includes(f.target.issueId)
|
||||
&& sweep.checked === 1 && sweep.skippedActiveRun === 0)).toBe(true), { timeout: 20_000 });
|
||||
} finally { await work.stop(); }
|
||||
}, 300_000);
|
||||
});
|
||||
@@ -118,7 +118,7 @@ const realSleep = (ms: number) => new Promise(resolve => realSetTimeout(resolve,
|
||||
vi.useRealTimers();
|
||||
const window = sweeps.slice(before.sweeps);
|
||||
expect(destroyCalls.length).toBe(before.destroys);
|
||||
expect(window.filter(sweep => sweep.targeted)).toEqual([]);
|
||||
expect(window.length).toBeLessThanOrEqual(12);
|
||||
expect(meter.read().statements - before.statements).toBeLessThan(5_000);
|
||||
for (const id of remote) {
|
||||
const [row] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, id));
|
||||
|
||||
@@ -3660,6 +3660,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
compacted: 0,
|
||||
compactionFailed: 0,
|
||||
};
|
||||
if (targets) for (const id of [...targets.workspaceIds, ...candidates.map(candidate => candidate.workspace.id)]) terminalWorkspaceDeadlines.delete(id);
|
||||
const readinessByWorkingTree = new Map<string, ReturnType<typeof inspectWorkspace>>();
|
||||
const backOffTerminalWorkspace = async (snapshot: ExecutionWorkspaceRow, failure: TerminalWorkspaceFailure) => {
|
||||
const [target] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, snapshot.id));
|
||||
@@ -3914,6 +3915,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
if (await workspaceHasActiveRun(workspace)) {
|
||||
if (reopenPending) result.skippedReopened += 1;
|
||||
else result.skippedActiveRun += 1;
|
||||
terminalWorkspaceDeadlines.delete(workspace.id);
|
||||
continue;
|
||||
}
|
||||
if (remoteWorkspace && isClosedExecutionWorkspaceStatus(workspace.status)) {
|
||||
@@ -4154,6 +4156,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
}
|
||||
if (await workspaceHasActiveRun(workspace)) {
|
||||
result.skippedActiveRun += 1;
|
||||
terminalWorkspaceDeadlines.delete(workspace.id);
|
||||
continue;
|
||||
}
|
||||
result.eligible += 1;
|
||||
@@ -4364,6 +4367,13 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
}
|
||||
},
|
||||
|
||||
takeDueTerminalWorkspaceTargets: (): { workspaceIds: string[]; rotation: boolean } => {
|
||||
const at = now().getTime();
|
||||
const due = (deadlines: Map<string, number>) => [...deadlines].filter(([, deadline]) => deadline <= at).map(([id]) => id);
|
||||
const rotation = due(isolatedRehomeDeadlines).length + due(nestedCloneMigrationDeadlines).length + due(taskDirectoryDeadlines).length > 0;
|
||||
return { workspaceIds: due(terminalWorkspaceDeadlines), rotation };
|
||||
},
|
||||
|
||||
nextTerminalWorkspaceSweepAt: async () => {
|
||||
if (terminalWorkspaceDeadlines.size === 0 && isolatedRehomeDeadlines.size === 0 && nestedCloneMigrationDeadlines.size === 0
|
||||
&& taskDirectoryDeadlines.size === 0) return null;
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyTerminalWorkspaceWork } from "../terminal-workspace-work.js";
|
||||
import { settleReattachRun } from "./run-admission.js";
|
||||
import { readNativeStartupCleanup } from "../native-runtime/native-startup-cleanup.js";
|
||||
import { isContextCompactionRun } from "../context-compaction.js";
|
||||
@@ -1122,6 +1123,8 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
|
||||
if (previous && isHeartbeatRunTerminalStatus(status))
|
||||
await notifyRetainedNativeSessionCleanup(tx, { companyId: previous.companyId, runId: previous.id });
|
||||
if (isHeartbeatRunTerminalStatus(status)) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, companyId);
|
||||
const runIssueId = previous ? previous.nativeIssueId ?? previous.issueId ?? readNonEmptyString(parseObject(previous.contextSnapshot).issueId) : null;
|
||||
if (previous && isHeartbeatRunTerminalStatus(status) && runIssueId) await notifyTerminalWorkspaceWork(tx, `issue:${runIssueId}`);
|
||||
await notifyExecutionRunWork(tx, previous ? { companyId: previous.companyId, runId: previous.id,
|
||||
issueId: previous.nativeIssueId ?? previous.issueId ?? readNonEmptyString(parseObject(previous.contextSnapshot).issueId) ?? undefined } : undefined);
|
||||
if (previous) await notifyChatRunMilestones(tx, { companyId: previous.companyId, runId: previous.id,
|
||||
|
||||
@@ -3,6 +3,7 @@ import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notificatio
|
||||
|
||||
export const TERMINAL_WORKSPACE_RETRY_MS = 5 * 60 * 1000;
|
||||
export const MAX_TERMINAL_WORKSPACE_TARGETS = 50;
|
||||
export const TERMINAL_WORKSPACE_BACKSTOP_MS = 60 * 60 * 1000;
|
||||
|
||||
export function notifyTerminalWorkspaceWork(transaction: object, hint?: string) {
|
||||
return notifyDeliveryWork(transaction, DELIVERY_QUEUES.terminalWorkspaces, hint);
|
||||
@@ -36,6 +37,7 @@ export function registerTerminalWorkspaceWork(
|
||||
nextTerminalWorkspaceSweepAt(): Promise<number | null>;
|
||||
hasTerminalWorkspaceSweepContinuation(): boolean;
|
||||
terminalWorkspaceWorkSettled?(hints?: ReadonlySet<string>, options?: { targeted?: boolean }): void;
|
||||
takeDueTerminalWorkspaceTargets?(): { workspaceIds: string[]; rotation: boolean };
|
||||
},
|
||||
canRun: () => boolean,
|
||||
) {
|
||||
@@ -49,10 +51,14 @@ export function registerTerminalWorkspaceWork(
|
||||
let rotationRequested = false;
|
||||
let dueAt: number | null = null;
|
||||
let lastRotationPageAt = 0;
|
||||
let backstopArmed = false;
|
||||
let lastFullRotationAt = Date.now();
|
||||
let continuing = false;
|
||||
const targetWorkspaceIds = new Set<string>(), targetIssueIds = new Set<string>();
|
||||
const worker = coordinator.register(DELIVERY_QUEUES.terminalWorkspaces, {
|
||||
retryMs: TERMINAL_WORKSPACE_RETRY_MS,
|
||||
onSettled: hints => {
|
||||
backstopArmed = true;
|
||||
const targets = terminalWorkspaceTargetsFromHints(hints);
|
||||
for (const id of targets?.workspaceIds ?? []) targetWorkspaceIds.add(id);
|
||||
for (const id of targets?.issueIds ?? []) targetIssueIds.add(id);
|
||||
@@ -70,12 +76,26 @@ export function registerTerminalWorkspaceWork(
|
||||
next = Date.now() + TERMINAL_WORKSPACE_RETRY_MS;
|
||||
const startup = !startupSweepStarted;
|
||||
startupSweepStarted = true;
|
||||
let dueRotation = due;
|
||||
if (due && !startup && service.takeDueTerminalWorkspaceTargets) {
|
||||
const dueTargets = service.takeDueTerminalWorkspaceTargets();
|
||||
dueRotation = dueTargets.rotation;
|
||||
for (const id of dueTargets.workspaceIds) targetWorkspaceIds.add(id);
|
||||
if (targetWorkspaceIds.size + targetIssueIds.size > MAX_TERMINAL_WORKSPACE_TARGETS) {
|
||||
dueRotation = true;
|
||||
targetWorkspaceIds.clear(); targetIssueIds.clear();
|
||||
}
|
||||
}
|
||||
const options = { compact: !startup, allowRemoteDeliveryChecks: !startup };
|
||||
const targeted = targetWorkspaceIds.size + targetIssueIds.size > 0;
|
||||
if (startup || due || rotationRequested || !targeted
|
||||
const backstopDue = backstopArmed && Date.now() >= lastFullRotationAt + TERMINAL_WORKSPACE_BACKSTOP_MS;
|
||||
if (due && service.takeDueTerminalWorkspaceTargets && !startup && !dueRotation && !targeted && !rotationRequested && !backstopDue
|
||||
&& !service.hasTerminalWorkspaceSweepContinuation()) return { checked: 0 };
|
||||
if (startup || dueRotation || rotationRequested || backstopDue || !targeted
|
||||
|| (service.hasTerminalWorkspaceSweepContinuation() && Date.now() >= lastRotationPageAt + 1000)) {
|
||||
rotationRequested = false;
|
||||
lastRotationPageAt = Date.now();
|
||||
if (!continuing) { lastFullRotationAt = Date.now(); backstopArmed = false; }
|
||||
return service.sweepTerminalWorkspaces(50, signal, options);
|
||||
}
|
||||
const targets = { workspaceIds: [...targetWorkspaceIds], issueIds: [...targetIssueIds] };
|
||||
@@ -88,9 +108,12 @@ export function registerTerminalWorkspaceWork(
|
||||
}
|
||||
},
|
||||
hasPending: async () => {
|
||||
continuing = false;
|
||||
if (!canRun()) next = Date.now() + TERMINAL_WORKSPACE_RETRY_MS;
|
||||
else if (service.hasTerminalWorkspaceSweepContinuation()) next = Math.max(Date.now(), lastRotationPageAt + 1000);
|
||||
else if (rotationRequested || targetWorkspaceIds.size + targetIssueIds.size > 0) next = Date.now();
|
||||
else if (service.hasTerminalWorkspaceSweepContinuation()) {
|
||||
continuing = true;
|
||||
next = Math.max(Date.now(), lastRotationPageAt + 1000);
|
||||
} else if (rotationRequested || targetWorkspaceIds.size + targetIssueIds.size > 0) next = Date.now();
|
||||
else {
|
||||
const deadline = await service.nextTerminalWorkspaceSweepAt();
|
||||
next = deadline === null ? null : deadline > Date.now()
|
||||
@@ -98,6 +121,10 @@ export function registerTerminalWorkspaceWork(
|
||||
dueAt = next;
|
||||
recovered = true;
|
||||
}
|
||||
if (backstopArmed && canRun()) {
|
||||
const backstopAt = lastFullRotationAt + TERMINAL_WORKSPACE_BACKSTOP_MS;
|
||||
next = next === null ? backstopAt : Math.min(next, backstopAt);
|
||||
}
|
||||
return false;
|
||||
},
|
||||
nextRunAt: () => next,
|
||||
|
||||
Reference in new issue
Block a user