mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-18 23:08:26 +01:00
perf(workspaces): hold kept plugin workspaces instead of re-inspecting them every sweep (#22054)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Workspace storage plugins hold task workspaces, and Core's terminal workspace sweep removes them after their tasks finish > - The sweep inspects each candidate workspace through the provider. A provider inspection can start a sandbox or call a remote API, so each one has a real cost > - When an inspection fails, the sweep backs off. When an inspection succeeds but keeps the workspace (local data, or work that is not merged), there is no backoff, so the sweep inspects the same workspace again on every pass, forever > - Kept workspaces also kept a past-due sweep deadline, so targeted sweeps came back to them every few minutes, not only once per hourly rotation > - With many closed workspaces, this repeated work grows without limit and gives the same answer each time > - This pull request records a hold after an inspection keeps a provider workspace, and skips the provider inspection while nothing that could change the answer has changed > - The benefit is far fewer provider operations for kept workspaces, with no delay for workspaces that can be removed ## Linked Issues or Issue Description **Subsystem affected** Server: the terminal workspace sweep for workspace storage plugins (`server/src/services/execution-workspaces.ts`). **Problem or motivation** The terminal workspace sweep inspects every kept plugin workspace again on every pass. For a workspace kept for local data, the answer cannot change until something changes the workspace. For a clean, unmerged workspace, the answer changes only when the remote changes. Each repeated inspection is a provider operation, which can start a sandbox or call a remote API. **Proposed solution** After an inspection keeps a plugin workspace, Core records a hold. A workspace kept for local data is not inspected again until it or its task tree changes, or a change notice names it. A clean, unmerged workspace is checked again after 1 hour, then on an interval that doubles up to 24 hours. Workspaces that can be removed are removed as before. **Alternatives considered** An in-memory hold only. It is lost on restart, and the existing in-memory inspection cache holds only 256 entries, so a large set of kept workspaces would be inspected again after every restart. ## What Changed - `server/src/services/terminal-workspace-retry.ts`: add the `terminalInspectionHold` workspace metadata record (generation, fingerprint, reason, HEAD, check count, check time, next check time). The write is fenced on the lifecycle generation. Add the recheck delay: 1 hour, doubling, capped at 24 hours. - `server/src/services/execution-workspaces.ts` (`sweepTerminalWorkspaces`): - Before it inspects a plugin workspace, the sweep compares a fingerprint of the inputs that can change the answer: the row's `updatedAt`, status, provider ref, repository, branch, base ref, provider binding, delivery candidate, and the task tree's terminal state. The lifecycle generation scopes the record. While the hold applies, the sweep skips the inspection and counts the workspace as `skippedBackoff`. - After an inspection keeps the workspace, the sweep records a `local_data` hold (dirty or untracked files, or local-only data; no recheck time) or an `unmerged` hold (growing recheck interval). A new inspected HEAD, a new fingerprint, or a change notice restarts the interval. - A kept workspace now gets a sweep deadline at its recheck time, or none. Before, it kept its past-due archive deadline, so targeted sweeps came back to it every few minutes. - A provider refresh notice (`plugin-workspaces`: activation, upgrade, capability change, company re-enablement) ends every hold, because it can change how any plugin workspace inspects. - A `workspace:` or `issue:` change notice ends the holds that it names (the workspace, or a workspace whose task tree or linked tasks contain the issue). A notice without a scope ends every hold. The next sweep visit clears an ended hold from the row, so the reset survives a restart. The in-memory notice entries are pruned after the next full rotation. - When an inspection finds the workspace removable, the sweep clears the hold and continues to removal as before. - The hold applies only to plugin workspaces. Local Git worktree inspections are cheap, and a person can change those folders directly, so they keep the current behavior. - Core owns the new metadata key. Workspace metadata updates keep it, like `terminalCleanupRetry`. - `doc/plugins/PLUGIN_RUNNER_EXECUTION.md`: describe the hold for provider authors. - Optional item not done: `removePluginWorkspace` still inspects again before it removes. That inspection runs right before a destructive operation, passes the recovery PR hint to the provider, and checks `remotelyRecoverable`, which the sweep's inspection result does not carry. Skipping it is not clearly safe. ## Verification - `npx vitest run src/__tests__/plugin-workspace-terminal-hold.integration.test.ts` (new, embedded Postgres): - Local-data hold: no inspection after 30 days; a workspace notice, an issue notice, a generation bump, a metadata update, a new task completion time, and a provider refresh notice each cause exactly one new inspection; no sweep deadline is left. The provider refresh step fails without its fix (6 inspections, not 7). - Unmerged backoff: recheck delays are 1, 2, 4, 8, 16, 24, 24 hours; a new HEAD resets to 1 hour; a workspace notice resets to 1 hour. - Removable workspaces: a clean merged workspace is removed on its first sweep (inspection count unchanged at 2); a held dirty workspace is removed on the next sweep after a notice, and the hold is cleared; a closed held workspace is removed after an unscoped notice. - Failure backoff: delays stay 5 and 10 minutes, and a failed inspection does not record a hold. - Red check: the five new tests fail against the unchanged service and pass with the change. - Related suites pass locally (10 files, 204 tests): `execution-workspaces-service`, `plugin-workspaces.integration`, `terminal-workspace-deadlines`, `terminal-workspace-idle`, `terminal-workspace-page-batch`, `terminal-workspace-rotation-limits`, `terminal-workspace-work`, `workspace-delivery-cleanup.integration`, `plugin-workspace-terminal-retry.integration`, and the new file. - A scratch check confirmed the old deadline behavior: after a sweep kept a workspace, the old code left it due with a past deadline. The new code leaves no deadline. - `plugin-workspace-terminal-retry.integration.test.ts`: one step now sends a workspace notice before it checks that a new failure restarts the backoff at 1, because the preceding successful inspection now holds the dirty workspace. - `cd server && npx tsc --noEmit -p .` passes. ## Risks - A held `local_data` workspace is inspected again only after a change. If a change notice is lost (for example a restart between the commit and the notice) and the change does not touch the workspace row, its generation, binding, status, delivery candidate, or task tree terminal state, the workspace stays held until one of those changes. Runs end with an `issue:` notice and new task completion times, so the normal paths are covered. - An `unmerged` workspace that becomes merged on the remote waits until its next recheck (at most 24 hours), not the next hourly rotation. - Each sweep page now runs one extra query for the linked tasks of held workspaces. Each kept workspace gets one metadata write per inspection. - No schema migration. The record is in existing workspace metadata. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3a41e54c51
commit
960a2b781e
5 files changed
+382
-11
No files matched your search
@@ -0,0 +1,201 @@
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { createDb, executionWorkspaces, heartbeatRuns, issues } from "@paperclipai/db";
|
||||
import { createPluginWorkspaceFixture } from "./helpers/plugin-workspaces.js";
|
||||
import { preparePluginWorkspace, type WorkspaceWorkers } from "../services/plugin-workspaces.js";
|
||||
import { executionWorkspaceService } from "../services/execution-workspaces.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const MINUTE = 60_000;
|
||||
const HOUR = 60 * MINUTE;
|
||||
|
||||
type Inspection = { headSha: string; hasDirtyTrackedFiles: boolean; hasUntrackedFiles: boolean; hasLocalOnlyData: boolean;
|
||||
isMergedIntoBase: boolean | null; remotelyRecoverable: boolean };
|
||||
type Hold = { reason: string; checks: number; checkedAt: string; recheckAt: string | null; headSha: string | null };
|
||||
|
||||
(support.supported ? describe : describe.skip)("plugin workspace terminal inspection hold", () => {
|
||||
let db: ReturnType<typeof createDb>, stopDb: (() => Promise<void>) | undefined;
|
||||
beforeAll(async () => {
|
||||
const started = await startEmbeddedPostgresTestDatabase("plugin-workspace-terminal-hold");
|
||||
db = createDb(started.connectionString, { maxConnections: 1 });
|
||||
stopDb = started.cleanup;
|
||||
});
|
||||
afterAll(async () => {
|
||||
await db?.$client.end();
|
||||
await stopDb?.();
|
||||
});
|
||||
|
||||
async function setup(inspection: Partial<Inspection>) {
|
||||
const f = await createPluginWorkspaceFixture(db);
|
||||
const state = { inspection: { headSha: "a".repeat(40), hasDirtyTrackedFiles: false, hasUntrackedFiles: false, hasLocalOnlyData: false,
|
||||
isMergedIntoBase: true, remotelyRecoverable: true, ...inspection } as Inspection, failInspect: false };
|
||||
const workers = { ...f.workers, call: async (...args: Parameters<WorkspaceWorkers["call"]>) => {
|
||||
const params = args[2] as { operation: { kind: string } };
|
||||
if (params.operation.kind === "inspect" && state.failInspect) {
|
||||
f.calls.push(params);
|
||||
throw new Error("provider outage");
|
||||
}
|
||||
const result = await f.workers.call(...args);
|
||||
return params.operation.kind === "inspect" ? { ...(result as object), ...state.inspection } : result;
|
||||
} } as WorkspaceWorkers;
|
||||
const workspace = (await preparePluginWorkspace(db, workers, f.input))!;
|
||||
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.runId));
|
||||
let clock = Date.now();
|
||||
const svc = executionWorkspaceService(db, { pluginWorkerManager: workers, workspaceReaperCooldownDays: 0, now: () => new Date(clock) });
|
||||
const read = async () => (await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace.id)))[0]!;
|
||||
const hold = async () => (await read()).metadata?.terminalInspectionHold as Hold | undefined;
|
||||
const inspections = () => f.calls.filter(call => call.executionWorkspaceId === workspace.id && call.operation.kind === "inspect").length;
|
||||
const removals = () => f.calls.filter(call => call.executionWorkspaceId === workspace.id && call.operation.kind === "remove").length;
|
||||
const sweep = () => svc.sweepTerminalWorkspaces(100);
|
||||
const due = () => svc.takeDueTerminalWorkspaceTargets().workspaceIds.includes(workspace.id);
|
||||
return { f, state, svc, workspace, read, hold, inspections, removals, sweep, due,
|
||||
get clock() { return clock; }, set clock(value: number) { clock = value; } };
|
||||
}
|
||||
|
||||
it("holds a workspace kept for local data until the workspace, its tasks, or a hint changes it", async () => {
|
||||
const s = await setup({ hasLocalOnlyData: true });
|
||||
await s.sweep();
|
||||
expect(s.inspections()).toBe(1);
|
||||
expect((await s.read()).status).toBe("active");
|
||||
expect(await s.hold()).toMatchObject({ reason: "local_data", checks: 1, recheckAt: null });
|
||||
expect(s.due()).toBe(false);
|
||||
|
||||
s.clock += 30 * 24 * HOUR;
|
||||
expect((await s.sweep()).skippedBackoff).toBeGreaterThanOrEqual(1);
|
||||
expect(s.inspections()).toBe(1);
|
||||
expect(s.due()).toBe(false);
|
||||
|
||||
s.svc.terminalWorkspaceWorkSettled(new Set([`workspace:${s.workspace.id}`]), { targeted: true });
|
||||
s.clock += 1;
|
||||
await s.sweep();
|
||||
expect(s.inspections()).toBe(2);
|
||||
await s.sweep();
|
||||
expect(s.inspections()).toBe(2);
|
||||
|
||||
s.svc.terminalWorkspaceWorkSettled(new Set([`issue:${s.f.issueId}`]), { targeted: true });
|
||||
s.clock += 1;
|
||||
await s.sweep();
|
||||
expect(s.inspections()).toBe(3);
|
||||
|
||||
await db.update(executionWorkspaces).set({ metadata: sql`jsonb_set(${executionWorkspaces.metadata}, '{lifecycleGeneration}',
|
||||
to_jsonb(coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) + 1))` })
|
||||
.where(eq(executionWorkspaces.id, s.workspace.id));
|
||||
await s.sweep();
|
||||
expect(s.inspections()).toBe(4);
|
||||
|
||||
await executionWorkspaceService(db).update(s.workspace.id, { metadata: { note: "operator edit" } });
|
||||
expect(await s.hold()).toMatchObject({ reason: "local_data" });
|
||||
await s.sweep();
|
||||
expect(s.inspections()).toBe(5);
|
||||
|
||||
await db.update(issues).set({ completedAt: new Date(s.clock) }).where(eq(issues.id, s.f.issueId));
|
||||
await s.sweep();
|
||||
expect(s.inspections()).toBe(6);
|
||||
await s.sweep();
|
||||
expect(s.inspections()).toBe(6);
|
||||
|
||||
s.svc.terminalWorkspaceWorkSettled(new Set(["plugin-workspaces"]), { targeted: true });
|
||||
s.clock += 1;
|
||||
await s.sweep();
|
||||
expect(s.inspections()).toBe(7);
|
||||
await s.sweep();
|
||||
expect(s.inspections()).toBe(7);
|
||||
expect(s.removals()).toBe(0);
|
||||
});
|
||||
|
||||
it("re-checks a clean unmerged workspace on a doubling interval and resets it on a new head or a hint", async () => {
|
||||
const s = await setup({ isMergedIntoBase: false, remotelyRecoverable: false });
|
||||
const start = s.clock;
|
||||
await s.sweep();
|
||||
expect(await s.hold()).toMatchObject({ reason: "unmerged", checks: 1, recheckAt: new Date(start + HOUR).toISOString() });
|
||||
|
||||
s.clock = start + 59 * MINUTE;
|
||||
await s.sweep();
|
||||
expect(s.inspections()).toBe(1);
|
||||
expect(s.due()).toBe(false);
|
||||
|
||||
const delays: number[] = [];
|
||||
for (let check = 0; check < 7; check++) {
|
||||
const recheckAt = Date.parse((await s.hold())!.recheckAt!);
|
||||
delays.push((recheckAt - s.clock) / HOUR);
|
||||
s.clock = recheckAt;
|
||||
expect(s.due()).toBe(true);
|
||||
await s.sweep();
|
||||
}
|
||||
expect(s.inspections()).toBe(8);
|
||||
expect(delays.slice(1)).toEqual([2, 4, 8, 16, 24, 24]);
|
||||
expect(await s.hold()).toMatchObject({ checks: 8, recheckAt: new Date(s.clock + 24 * HOUR).toISOString() });
|
||||
|
||||
s.state.inspection.headSha = "b".repeat(40);
|
||||
s.clock = Date.parse((await s.hold())!.recheckAt!);
|
||||
await s.sweep();
|
||||
expect(await s.hold()).toMatchObject({ checks: 1, headSha: "b".repeat(40), recheckAt: new Date(s.clock + HOUR).toISOString() });
|
||||
|
||||
s.clock = Date.parse((await s.hold())!.recheckAt!);
|
||||
await s.sweep();
|
||||
expect(await s.hold()).toMatchObject({ checks: 2 });
|
||||
s.clock += 10 * MINUTE;
|
||||
s.svc.terminalWorkspaceWorkSettled(new Set([`workspace:${s.workspace.id}`]), { targeted: true });
|
||||
s.clock += 1;
|
||||
await s.sweep();
|
||||
expect(s.inspections()).toBe(11);
|
||||
expect(await s.hold()).toMatchObject({ checks: 1, recheckAt: new Date(s.clock + HOUR).toISOString() });
|
||||
});
|
||||
|
||||
it("removes a workspace as soon as it is removable, including one that was held", async () => {
|
||||
const merged = await setup({});
|
||||
await merged.sweep();
|
||||
expect(merged.inspections()).toBe(2);
|
||||
expect(merged.removals()).toBe(1);
|
||||
expect(await merged.read()).toMatchObject({ status: "archived", metadata: { workspaceProviderRemoved: true } });
|
||||
|
||||
const s = await setup({ hasDirtyTrackedFiles: true });
|
||||
await s.sweep();
|
||||
expect(await s.hold()).toMatchObject({ reason: "local_data" });
|
||||
s.state.inspection.hasDirtyTrackedFiles = false;
|
||||
s.svc.terminalWorkspaceWorkSettled(new Set([`workspace:${s.workspace.id}`]), { targeted: true });
|
||||
s.clock += 1;
|
||||
await s.sweep();
|
||||
expect(s.removals()).toBe(1);
|
||||
const removed = await s.read();
|
||||
expect(removed).toMatchObject({ status: "archived", metadata: { workspaceProviderRemoved: true } });
|
||||
expect(removed.metadata).not.toHaveProperty("terminalInspectionHold");
|
||||
});
|
||||
|
||||
it("holds a closed workspace with local data and removes it after an unscoped change notice", async () => {
|
||||
const s = await setup({});
|
||||
await db.update(executionWorkspaces).set({ status: "archived", closedAt: new Date() }).where(eq(executionWorkspaces.id, s.workspace.id));
|
||||
s.state.inspection.hasUntrackedFiles = true;
|
||||
await s.sweep();
|
||||
expect(await s.hold()).toMatchObject({ reason: "local_data" });
|
||||
await s.sweep();
|
||||
expect(s.inspections()).toBe(1);
|
||||
s.state.inspection.hasUntrackedFiles = false;
|
||||
s.svc.terminalWorkspaceWorkSettled();
|
||||
s.clock += 1;
|
||||
await s.sweep();
|
||||
expect(s.removals()).toBe(1);
|
||||
expect(await s.read()).toMatchObject({ metadata: { workspaceProviderRemoved: true } });
|
||||
});
|
||||
|
||||
it("keeps the inspection failure backoff and does not hold a workspace it could not inspect", async () => {
|
||||
const s = await setup({ hasLocalOnlyData: true });
|
||||
s.state.failInspect = true;
|
||||
await s.sweep();
|
||||
expect(await s.hold()).toBeUndefined();
|
||||
expect((await s.read()).metadata?.terminalCleanupRetry).toMatchObject({ failures: 1,
|
||||
retryAt: new Date(s.clock + 5 * MINUTE).toISOString() });
|
||||
s.clock += 5 * MINUTE;
|
||||
await s.sweep();
|
||||
expect((await s.read()).metadata?.terminalCleanupRetry).toMatchObject({ failures: 2,
|
||||
retryAt: new Date(s.clock + 10 * MINUTE).toISOString() });
|
||||
s.state.failInspect = false;
|
||||
s.clock += 10 * MINUTE;
|
||||
await s.sweep();
|
||||
expect(s.inspections()).toBe(3);
|
||||
expect((await s.read()).metadata).not.toHaveProperty("terminalCleanupRetry");
|
||||
expect(await s.hold()).toMatchObject({ reason: "local_data", checks: 1 });
|
||||
});
|
||||
});
|
||||
@@ -204,6 +204,8 @@ function providerRefusal() {
|
||||
await svc.sweepTerminalWorkspaces(100);
|
||||
expect(s.callsFor(workspace.id, "inspect")).toHaveLength(4);
|
||||
expect(await retryAfter()).toBeUndefined();
|
||||
// The successful inspection kept the dirty workspace on hold; a change notice ends the hold.
|
||||
svc.terminalWorkspaceWorkSettled(new Set([`workspace:${workspace.id}`]), { targeted: true });
|
||||
s.failures.inspect = true;
|
||||
clock += MINUTE;
|
||||
await svc.sweepTerminalWorkspaces(100);
|
||||
|
||||
@@ -39,10 +39,12 @@ import { authorizationService, executionWorkspaceReadSqlCondition, projectReadSq
|
||||
import type { SQL } from "drizzle-orm";
|
||||
import { nativeSessionProviderEvidence } from "./native-runtime/native-session-resume.js";
|
||||
import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
|
||||
import { notifyTerminalWorkspaceWork, type TerminalWorkspaceTargets } from "./terminal-workspace-work.js";
|
||||
import { notifyTerminalWorkspaceWork, TERMINAL_WORKSPACE_PLUGIN_HINT, type TerminalWorkspaceTargets } from "./terminal-workspace-work.js";
|
||||
import {
|
||||
clearTerminalWorkspaceRetry, readTerminalWorkspaceRetry, recordTerminalWorkspaceFailure, terminalWorkspaceFailure,
|
||||
TERMINAL_WORKSPACE_RETRY_KEY, type TerminalWorkspaceFailure,
|
||||
clearTerminalWorkspaceHold, readTerminalWorkspaceHold, recordTerminalWorkspaceHold, terminalWorkspaceRecheckDelayMs,
|
||||
TERMINAL_WORKSPACE_HOLD_KEY, type TerminalWorkspaceHold,
|
||||
} from "./terminal-workspace-retry.js";
|
||||
import {
|
||||
deleteWorkspaceCompactionTrash,
|
||||
@@ -269,6 +271,19 @@ export function clearMetadataReopenPendingConsumption(
|
||||
return next;
|
||||
}
|
||||
|
||||
/** What a kept provider workspace's inspection depends on, apart from its lifecycle generation and the remote. */
|
||||
function terminalHoldFingerprint(workspace: typeof executionWorkspaces.$inferSelect,
|
||||
issueTree: Array<{ id: string; status: string; completedAt: Date | null; cancelledAt: Date | null; updatedAt: Date }>,
|
||||
deliveryCandidate: { operationId: string; headSha: string; destinationRef: string } | null) {
|
||||
return createHash("sha256").update(JSON.stringify([
|
||||
workspace.updatedAt.toISOString(), workspace.status, workspace.providerRef, workspace.repoUrl, workspace.branchName, workspace.baseRef,
|
||||
readWorkspaceProviderBinding(workspace.metadata) ?? null,
|
||||
deliveryCandidate && [deliveryCandidate.operationId, deliveryCandidate.headSha, deliveryCandidate.destinationRef],
|
||||
issueTree.map(issue => [issue.id, issue.status, issueTerminalTimestamp(issue)?.toISOString() ?? null] as const)
|
||||
.sort((a, b) => a[0].localeCompare(b[0])),
|
||||
])).digest("hex");
|
||||
}
|
||||
|
||||
function terminalWorkspaceTargetFilter(targets: TerminalWorkspaceTargets) {
|
||||
const ids = (values: string[]) => sql.join(values.map(id => sql`${id}::uuid`), sql`, `);
|
||||
const byWorkspace = targets.workspaceIds.length ? sql`${executionWorkspaces.id} in (${ids(targets.workspaceIds)})` : undefined;
|
||||
@@ -314,7 +329,8 @@ function preserveWorkspaceMetadataForPatch(
|
||||
}
|
||||
}
|
||||
}
|
||||
const coreOwnedKeys = ["workspaceProviderFresh", "runnerTaskFreshWorkspace", "runnerTaskWorkspaceOwner", TERMINAL_WORKSPACE_RETRY_KEY] as const;
|
||||
const coreOwnedKeys = ["workspaceProviderFresh", "runnerTaskFreshWorkspace", "runnerTaskWorkspaceOwner", TERMINAL_WORKSPACE_RETRY_KEY,
|
||||
TERMINAL_WORKSPACE_HOLD_KEY] as const;
|
||||
if (patch.metadata !== undefined && (patch.metadata !== null || coreOwnedKeys.some(key => current.metadata?.[key] !== undefined))) {
|
||||
nextPatch.metadata = { ...(nextPatch.metadata ?? patch.metadata) };
|
||||
for (const key of coreOwnedKeys) {
|
||||
@@ -1563,6 +1579,7 @@ export async function releaseAncestorWorkspaceBinding(tx: DbTransaction, input:
|
||||
// move and the guard finish in one short transaction.
|
||||
const WORKSPACE_COMPACTION_TRASH_STALE_MS = 60 * 60 * 1000;
|
||||
const WORKSPACE_COMPACTION_TRASH_PURGE_INTERVAL_MS = 30 * 60 * 1000;
|
||||
const TERMINAL_HOLD_RESET_LIMIT = 65_536;
|
||||
|
||||
export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServiceOptions = {}) {
|
||||
const destroyRemoteLease: DestroyRemoteWorkspaceLease = input => opts.destroyRemoteWorkspaceLease
|
||||
@@ -1937,6 +1954,13 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
// when a sweep reaches the end of the candidate set.
|
||||
let terminalSweepCursor: { updatedAt: string; id: string } | null = null;
|
||||
const terminalInspections = new Map<string, { revision: string; issueIds: Set<string>; nextAt: number; parked?: boolean }>();
|
||||
// A `workspace:` or `issue:` hint ends the persisted inspection holds it names
|
||||
// that were checked at or before the hint. A sweep visit clears such a hold
|
||||
// from the row, so an entry is dropped once a full rotation that started after
|
||||
// it completes. A hint without a scope ends every hold.
|
||||
const terminalHoldResets = new Map<string, number>();
|
||||
let terminalHoldResetAllAt = Number.NEGATIVE_INFINITY;
|
||||
let terminalHoldRotationStartedAt: number | null = null;
|
||||
const terminalWorkspaceDeadlines = new Map<string, number>();
|
||||
const isolatedRehomeDeadlines = new Map<string, number>();
|
||||
const isolatedRehomeNotBefore = new Map<string, number>();
|
||||
@@ -2279,6 +2303,21 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
return trees;
|
||||
}
|
||||
|
||||
async function readPageLinkedIssueIds(workspaces: Array<Pick<ExecutionWorkspaceRow, "id" | "companyId">>) {
|
||||
const linked = new Map<string, string[]>();
|
||||
if (workspaces.length === 0) return linked;
|
||||
const rows = await db.select({ id: issues.id, workspaceId: issues.executionWorkspaceId }).from(issues).where(and(
|
||||
inArray(issues.companyId, [...new Set(workspaces.map(workspace => workspace.companyId))]),
|
||||
inArray(issues.executionWorkspaceId, workspaces.map(workspace => workspace.id))));
|
||||
for (const row of rows) {
|
||||
if (!row.workspaceId) continue;
|
||||
const ids = linked.get(row.workspaceId);
|
||||
if (ids) ids.push(row.id);
|
||||
else linked.set(row.workspaceId, [row.id]);
|
||||
}
|
||||
return linked;
|
||||
}
|
||||
|
||||
async function readPageActiveRunWorkspaces(workspaces: Array<Pick<ExecutionWorkspaceRow, "id" | "companyId" | "sourceIssueId">>) {
|
||||
if (workspaces.length === 0) return new Set<string>();
|
||||
const pairs = workspaces.map(workspace => ({ id: workspace.id, companyId: workspace.companyId, sourceIssueId: workspace.sourceIssueId }));
|
||||
@@ -3655,6 +3694,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
// rotation cover a finite set, so the cursor always reaches a short page.
|
||||
if (!targets && !cursor) {
|
||||
terminalSweepBoundary = now();
|
||||
terminalHoldRotationStartedAt = terminalSweepBoundary.getTime();
|
||||
terminalWorkspaceDeadlines.clear();
|
||||
isolatedRehomeDeadlines.clear();
|
||||
nestedCloneMigrationDeadlines.clear();
|
||||
@@ -3697,7 +3737,9 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
// Advance the cursor to this page's last row. A short page means the scan
|
||||
// reached the end of the bounded candidate set, so reset the cursor and
|
||||
// the bound to start a new rotation on the next sweep.
|
||||
if (!targets && candidates.length < limit) {
|
||||
const rotationCompleted = !targets && candidates.length < limit;
|
||||
const completedRotationStartedAt = terminalHoldRotationStartedAt;
|
||||
if (rotationCompleted) {
|
||||
terminalSweepCursor = null;
|
||||
terminalSweepBoundary = null;
|
||||
} else if (!targets) {
|
||||
@@ -3723,8 +3765,24 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
if (targets) for (const id of [...targets.workspaceIds, ...candidates.map(candidate => candidate.workspace.id)]) terminalWorkspaceDeadlines.delete(id);
|
||||
const readinessByWorkingTree = new Map<string, ReturnType<typeof inspectWorkspace>>();
|
||||
const pageWorkspaces = candidates.map(candidate => candidate.workspace);
|
||||
const [pageIssueTrees, pageActiveRunWorkspaces] = await Promise.all([
|
||||
readPageIssueTrees(pageWorkspaces), readPageActiveRunWorkspaces(pageWorkspaces)]);
|
||||
const [pageIssueTrees, pageActiveRunWorkspaces, pageLinkedIssueIds] = await Promise.all([
|
||||
readPageIssueTrees(pageWorkspaces), readPageActiveRunWorkspaces(pageWorkspaces),
|
||||
readPageLinkedIssueIds(pageWorkspaces.filter(workspace => workspace.metadata?.[TERMINAL_WORKSPACE_HOLD_KEY] !== undefined))]);
|
||||
// Clear a persisted hold that a hint or a malformed record has ended, so
|
||||
// the reset survives this process and the hint entry can be pruned.
|
||||
const releaseEndedTerminalHold = async (target: ExecutionWorkspaceRow) => {
|
||||
const hold = target.metadata?.[TERMINAL_WORKSPACE_HOLD_KEY] as { checkedAt?: unknown } | undefined;
|
||||
if (hold === undefined) return target;
|
||||
const checkedAt = typeof hold?.checkedAt === "string" ? Date.parse(hold.checkedAt) : Number.NaN;
|
||||
const issueIds = [...(pageIssueTrees.get(target.id) ?? []).map(issue => issue.id), ...(pageLinkedIssueIds.get(target.id) ?? [])];
|
||||
const resetAt = Math.max(terminalHoldResetAllAt, terminalHoldResets.get(`workspace:${target.id}`) ?? Number.NEGATIVE_INFINITY,
|
||||
...issueIds.map(id => terminalHoldResets.get(`issue:${id}`) ?? Number.NEGATIVE_INFINITY));
|
||||
if (Number.isFinite(checkedAt) && resetAt < checkedAt) return target;
|
||||
await clearTerminalWorkspaceHold(db, target.id);
|
||||
const metadata = { ...target.metadata };
|
||||
delete metadata[TERMINAL_WORKSPACE_HOLD_KEY];
|
||||
return { ...target, metadata };
|
||||
};
|
||||
const backOffTerminalWorkspace = async (snapshot: ExecutionWorkspaceRow, failure: TerminalWorkspaceFailure) => {
|
||||
const [target] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, snapshot.id));
|
||||
if (!target) return;
|
||||
@@ -3736,6 +3794,11 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
logger.warn({ event: "execution_workspace.terminal_cleanup_retry", executionWorkspaceId: target.id, sourceIssueId: target.sourceIssueId,
|
||||
failures: retry.failures, retryAt: retry.retryAt, failure: retry.failure }, "terminal workspace cleanup failed; a later sweep retries");
|
||||
};
|
||||
const setTerminalWorkspaceDeadline = (workspaceId: string, recheckAt: number | null, compactionAt: number | null) => {
|
||||
const nextAt = Math.min(recheckAt ?? Number.POSITIVE_INFINITY, compactionAt ?? Number.POSITIVE_INFINITY);
|
||||
if (Number.isFinite(nextAt)) terminalWorkspaceDeadlines.set(workspaceId, nextAt);
|
||||
else terminalWorkspaceDeadlines.delete(workspaceId);
|
||||
};
|
||||
const settleTerminalWorkspaceRetry = async (target: ExecutionWorkspaceRow) => {
|
||||
if (target.metadata?.[TERMINAL_WORKSPACE_RETRY_KEY] !== undefined) await clearTerminalWorkspaceRetry(db, target.id);
|
||||
};
|
||||
@@ -3769,8 +3832,8 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
};
|
||||
|
||||
for (const candidate of candidates) {
|
||||
let workspace = candidate.workspace;
|
||||
signal?.throwIfAborted();
|
||||
let workspace = await releaseEndedTerminalHold(candidate.workspace);
|
||||
// Trash deletion is part of compaction, so a sweep that skips compaction
|
||||
// also leaves leftover trash for a later sweep.
|
||||
if (options.compact !== false) await purgeCompactionTrashIfDue(workspace);
|
||||
@@ -4021,6 +4084,21 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
result.skippedBackoff += 1;
|
||||
continue;
|
||||
}
|
||||
// Each inspection of a provider workspace is a provider operation. Skip
|
||||
// it while an earlier inspection kept the workspace and nothing that
|
||||
// could change that answer has changed.
|
||||
const pendingCompactionAt = compactionPending && compactedTerminalAt.get(workspace.id) !== latestTerminalAt ? compactionDueAt : null;
|
||||
const holdGeneration = readExecutionWorkspaceLifecycleGeneration(workspace.metadata);
|
||||
const holdFingerprint = workspace.providerType === "plugin" && !reopenPending
|
||||
&& !workspace.metadata?.workspaceProviderRemoval && !workspace.metadata?.workspaceProviderRestoration
|
||||
? terminalHoldFingerprint(workspace, issueTree, deliveryCandidate) : null;
|
||||
const priorHold = holdFingerprint ? readTerminalWorkspaceHold(workspace.metadata, holdGeneration) : null;
|
||||
if (priorHold && priorHold.fingerprint === holdFingerprint
|
||||
&& (priorHold.recheckAt === null || Date.parse(priorHold.recheckAt) > now().getTime())) {
|
||||
setTerminalWorkspaceDeadline(workspace.id, priorHold.recheckAt === null ? null : Date.parse(priorHold.recheckAt), pendingCompactionAt);
|
||||
result.skippedBackoff += 1;
|
||||
continue;
|
||||
}
|
||||
if (opts.terminalWorkspaceInspectionBackoffMs) {
|
||||
const revision = createHash("sha256").update(JSON.stringify([
|
||||
workspace.updatedAt, workspace.metadata,
|
||||
@@ -4094,6 +4172,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
? JSON.stringify([workspace.companyId, workspacePath, workspace.baseRef, workspace.branchName, createdByRuntime])
|
||||
: `workspace:${workspace.id}`;
|
||||
let inspection = readinessByWorkingTree.get(readinessKey);
|
||||
const inspectionStartedAt = now().getTime();
|
||||
if (!inspection) {
|
||||
inspection = inspectWorkspace(toExecutionWorkspace(workspace), recoveryPullRequest);
|
||||
readinessByWorkingTree.set(readinessKey, inspection);
|
||||
@@ -4134,18 +4213,33 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
result.skippedNonTerminalTree += 1;
|
||||
continue;
|
||||
}
|
||||
if (assessment.workspaceDirty) {
|
||||
const keepReason = assessment.workspaceDirty ? "local_data" as const
|
||||
: assessment.deliveryState !== "merged_via_pr" && assessment.deliveryState !== "merged_by_ancestry" ? "unmerged" as const : null;
|
||||
if (keepReason && holdFingerprint) {
|
||||
const headSha = git?.headSha ?? null;
|
||||
const checks = priorHold?.reason === keepReason && priorHold.fingerprint === holdFingerprint && priorHold.headSha === headSha
|
||||
? priorHold.checks + 1 : 1;
|
||||
const recheckAt = keepReason === "unmerged" ? inspectionStartedAt + terminalWorkspaceRecheckDelayMs(checks) : null;
|
||||
const hold: TerminalWorkspaceHold = { generation: holdGeneration, fingerprint: holdFingerprint, reason: keepReason, headSha, checks,
|
||||
checkedAt: new Date(inspectionStartedAt).toISOString(), recheckAt: recheckAt === null ? null : new Date(recheckAt).toISOString() };
|
||||
await recordTerminalWorkspaceHold(db, workspace.id, hold);
|
||||
setTerminalWorkspaceDeadline(workspace.id, recheckAt, pendingCompactionAt);
|
||||
}
|
||||
if (keepReason === "local_data") {
|
||||
result.skippedUndelivered += 1;
|
||||
continue;
|
||||
}
|
||||
if (
|
||||
assessment.deliveryState !== "merged_via_pr"
|
||||
&& assessment.deliveryState !== "merged_by_ancestry"
|
||||
) {
|
||||
if (keepReason === "unmerged") {
|
||||
if (remoteWorkspace) terminalWorkspaceDeadlines.delete(workspace.id);
|
||||
result.skippedUndelivered += 1;
|
||||
continue;
|
||||
}
|
||||
if (workspace.metadata?.[TERMINAL_WORKSPACE_HOLD_KEY] !== undefined) {
|
||||
await clearTerminalWorkspaceHold(db, workspace.id);
|
||||
const metadata = { ...workspace.metadata };
|
||||
delete metadata[TERMINAL_WORKSPACE_HOLD_KEY];
|
||||
workspace = { ...workspace, metadata };
|
||||
}
|
||||
// Hold the archive during the cooldown window. The anchor is the most
|
||||
// recent terminal timestamp across the issue tree. A person can reopen
|
||||
// the work inside this window. A cooldown of 0 disables the check, so the
|
||||
@@ -4392,6 +4486,10 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
});
|
||||
}
|
||||
}
|
||||
if (rotationCompleted && completedRotationStartedAt !== null) {
|
||||
for (const [key, at] of terminalHoldResets) if (at < completedRotationStartedAt) terminalHoldResets.delete(key);
|
||||
if (terminalHoldResetAllAt < completedRotationStartedAt) terminalHoldResetAllAt = Number.NEGATIVE_INFINITY;
|
||||
}
|
||||
return result;
|
||||
} catch (error) {
|
||||
if (!options.targets) {
|
||||
@@ -4423,6 +4521,20 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
else for (const [id, skip] of taskDirectoryRowSkips) if (`issue:${skip.issueId}` === hint) taskDirectoryRowSkips.delete(id);
|
||||
}
|
||||
}
|
||||
// A provider refresh (activation, upgrade, capability change, company
|
||||
// re-enablement) can change how any plugin workspace inspects, and holds
|
||||
// apply only to plugin workspaces, so it ends every hold.
|
||||
const resetAt = now().getTime();
|
||||
if (!hints || hints.has(TERMINAL_WORKSPACE_PLUGIN_HINT)) terminalHoldResetAllAt = resetAt;
|
||||
for (const hint of hints ?? []) {
|
||||
if (!hint.startsWith("workspace:") && !hint.startsWith("issue:")) continue;
|
||||
if (terminalHoldResets.size >= TERMINAL_HOLD_RESET_LIMIT) {
|
||||
terminalHoldResets.clear();
|
||||
terminalHoldResetAllAt = resetAt;
|
||||
break;
|
||||
}
|
||||
terminalHoldResets.set(hint, resetAt);
|
||||
}
|
||||
if (!hints) { terminalInspections.clear(); return; }
|
||||
for (const [id, inspection] of terminalInspections) {
|
||||
if (hints.has(`workspace:${id}`) || [...inspection.issueIds].some(issueId => hints.has(`issue:${issueId}`))) {
|
||||
|
||||
@@ -48,3 +48,46 @@ export async function clearTerminalWorkspaceRetry(db: Db, workspaceId: string) {
|
||||
.set({ metadata: sql`${executionWorkspaces.metadata} - ${TERMINAL_WORKSPACE_RETRY_KEY}::text` })
|
||||
.where(and(eq(executionWorkspaces.id, workspaceId), sql`${executionWorkspaces.metadata} -> ${TERMINAL_WORKSPACE_RETRY_KEY}::text IS NOT NULL`));
|
||||
}
|
||||
|
||||
// A provider workspace that a sweep inspected and kept. `local_data` holds until
|
||||
// the workspace or its task tree changes; `unmerged` re-checks on a growing
|
||||
// interval because the remote can change without touching the workspace.
|
||||
export const TERMINAL_WORKSPACE_HOLD_KEY = "terminalInspectionHold";
|
||||
export const TERMINAL_WORKSPACE_RECHECK_MS = 60 * 60 * 1000;
|
||||
export const TERMINAL_WORKSPACE_RECHECK_MAX_MS = 24 * 60 * 60 * 1000;
|
||||
|
||||
export type TerminalWorkspaceHold = {
|
||||
generation: number;
|
||||
fingerprint: string;
|
||||
reason: "local_data" | "unmerged";
|
||||
headSha: string | null;
|
||||
checks: number;
|
||||
checkedAt: string;
|
||||
recheckAt: string | null;
|
||||
};
|
||||
|
||||
export function terminalWorkspaceRecheckDelayMs(checks: number) {
|
||||
return Math.min(TERMINAL_WORKSPACE_RECHECK_MS * 2 ** Math.min(Math.max(checks - 1, 0), 16), TERMINAL_WORKSPACE_RECHECK_MAX_MS);
|
||||
}
|
||||
|
||||
export function readTerminalWorkspaceHold(metadata: Record<string, unknown> | null | undefined, generation: number) {
|
||||
const hold = metadata?.[TERMINAL_WORKSPACE_HOLD_KEY] as Partial<TerminalWorkspaceHold> | undefined;
|
||||
return hold && typeof hold === "object" && hold.generation === generation && typeof hold.fingerprint === "string"
|
||||
&& (hold.reason === "local_data" || hold.reason === "unmerged") && Number.isInteger(hold.checks)
|
||||
&& typeof hold.checkedAt === "string" && (hold.recheckAt === null || typeof hold.recheckAt === "string")
|
||||
? hold as TerminalWorkspaceHold : null;
|
||||
}
|
||||
|
||||
/** Records the hold only while the workspace is still in the inspected lifecycle generation. */
|
||||
export async function recordTerminalWorkspaceHold(db: Db, workspaceId: string, hold: TerminalWorkspaceHold) {
|
||||
await db.update(executionWorkspaces)
|
||||
.set({ metadata: sql`coalesce(${executionWorkspaces.metadata}, '{}'::jsonb) || jsonb_build_object(${TERMINAL_WORKSPACE_HOLD_KEY}::text, ${JSON.stringify(hold)}::jsonb)` })
|
||||
.where(and(eq(executionWorkspaces.id, workspaceId),
|
||||
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${hold.generation}`));
|
||||
}
|
||||
|
||||
export async function clearTerminalWorkspaceHold(db: Db, workspaceId: string) {
|
||||
await db.update(executionWorkspaces)
|
||||
.set({ metadata: sql`${executionWorkspaces.metadata} - ${TERMINAL_WORKSPACE_HOLD_KEY}::text` })
|
||||
.where(and(eq(executionWorkspaces.id, workspaceId), sql`${executionWorkspaces.metadata} -> ${TERMINAL_WORKSPACE_HOLD_KEY}::text IS NOT NULL`));
|
||||
}
|
||||
Reference in new issue
Block a user