perf(workspaces): start a full terminal workspace pass only at startup and on the hourly backstop (#21630)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep walks every candidate workspace in a
full pass. On staging that is about 4,000 finished remote workspaces,
and each one costs a persistence probe and a work product read.
> - #21617 limited full passes to startup and an hourly backstop that
only activity arms. But three other paths in the worker still started a
full pass at once:
>   - a wake whose hints do not parse to workspace or task targets;
>   - more than 50 pending targets;
>   - any run that finds no targets.
> - Across one staging deploy the probe count rose by about 2.8 full
passes in 26 minutes. At 1,000 concurrent agents, more than 50
run-finish hints between passes would be normal, so the overflow path
alone would bring back passes proportional to history.
> - This pull request works through targets in batches of 50 and turns
hintless wakes into the backstop. It gives the one caller that needs a
broad look, a workspace provider plugin becoming ready, its own hint
that inspects only plugin workspaces.
> - The benefit is that a full pass happens only at startup and at most
once per backstop hour, whatever the activity.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On staging, the hourly steady state holds (one full pass per hour, no
probes between). But across a deploy the probe count rose by 11,351 in
under 26 minutes, about 2.8 full passes over the roughly 4,000 finished
remote workspaces.

**Steps to reproduce**

The new test file
`server/src/__tests__/terminal-workspace-rotation-limits.test.ts` runs
the real coordinator and worker against real PostgreSQL on a fake clock,
with finished remote workspaces that cannot be archived (the staging
state). These are the counts on master. The numbers are workspaces
checked by full passes; a pass over N workspaces checks N:

| Scenario | Full-pass checks on master | Expected |
| --- | --- | --- |
| A burst of 120 run-finish `issue:` hints, 300 workspaces | 600 (2
extra passes) | 0 |
| One hintless wake and one unparseable hint, 120 workspaces, 30 minutes
| 1,490 (about 12 passes) | 0, then 1 at the backstop |
| A workspace provider plugin becomes ready (hintless notify from
`plugin-registry`), 123 workspaces | 123 (1 pass) | 0; only the 3 plugin
workspaces |
| Restart: old instance drains while hints arrive, new instance starts
and gets the plugin hint and 120 run-finish hints, 202 workspaces, 30
minutes | 2,424 (12 passes) | 202 (the startup pass) |

**Cause of the passes across a deploy:**
- The new instance runs its startup pass.
- Each workspace provider plugin that turns `ready` sends a hintless
notification (`plugin-registry.ts`), which started a full pass.
- Run recovery at startup finishes many runs, and their `issue:` hints
overflow 50, which started another full pass.
- Any later wake without targets also started a full pass, including the
coordinator's 5-minute retry timer. In the fixture this produced two
back-to-back passes every 5 minutes.
- The draining old instance starts nothing, because the worker's
`canRun` is false during drain. The test confirms this.

**Expected behavior**

A full pass runs only at startup and when the activity-armed backstop is
due, at most once per hour. Targets are worked through in targeted
batches.

## What Changed

- `server/src/services/terminal-workspace-work.ts`:
- **Pending targets.** Workspace and task targets go into pending sets
with no cap. Each run inspects at most 50 of them, and the rest stay
queued for the next run, which is scheduled at once. A failed targeted
run puts its targets back.
  - **Hintless or unparseable wakes** only arm the hourly backstop.
  - **No targets, nothing due:** the run does nothing.
- **Due deadlines:** the due workspaces are queued as targets, including
isolated re-home and nested clone deadlines keyed by a workspace. A due
non-workspace deadline (the isolated source orphan sweep, task directory
work) runs a targeted sweep for its prelude and arms the backstop.
- **A failed plugin workspace listing** restores the plugin refresh
flag, so the coordinator's retry lists again.
- **Full passes** happen only at startup, when the backstop is due, and
on continuation pages of a pass already under way.
- **New hint `plugin-workspaces`** (`TERMINAL_WORKSPACE_PLUGIN_HINT`):
it queues the ids of plugin-provider candidate workspaces from
`listPluginTerminalWorkspaceIds`.
- The worker now always tells the service `targeted: true`, so a hint
during a pass no longer requests a rescan lap.
- `server/src/services/execution-workspaces.ts`:
- New `listPluginTerminalWorkspaceIds` reads the plugin-provider
candidates.
- A targeted sweep reads all matches of its at most 50 targets, with no
page limit and no rescan request.
- A failed targeted sweep no longer requests a rescan lap, because the
worker re-queues its targets.
- `server/src/services/plugin-registry.ts`: the two workspace-provider
notifications send `plugin-workspaces` instead of no hint.
- `server/src/__tests__/terminal-workspace-work.test.ts`: a new test,
`retries the plugin workspace listing after it fails instead of waiting
for the backstop`, fails without the retry fix. Three existing tests now
describe the new behaviour:
  - unscoped commits go to the backstop;
  - more than 50 hinted workspaces are inspected in batches of 50 and 1;
- a due deadline on a service without targeting still runs a full pass.
- `server/src/__tests__/terminal-workspace-rotation-limits.test.ts`:
four new tests, the scenarios above.

The startup pass stays. The 1,000-workspace test in
`terminal-workspace-deadlines.test.ts` shows that durable deadlines keep
an idle instance quiet after startup. But only the startup pass
reconciles workspaces whose hints were lost while no instance was
running, so removing it would need a stored "last full pass" marker
first.

## Verification

- The four new tests fail on master with the counts in the table and
pass after the change:
- 0 full-pass checks for the burst, worked as targeted batches of at
most 50 (120 checks in total);
- 0 full-pass checks for 30 minutes after hintless and unparseable
wakes, then 1 pass at the backstop;
  - only the 3 plugin workspaces are inspected after the plugin hint;
- exactly 1 full pass across the restart: none on the draining instance,
only the startup pass on the new one.
- Prompt archiving and the missed-hint backstop keep their existing
tests in `terminal-workspace-deadlines.test.ts`:
  - the release of a holding run inspects its workspace at once;
- a run made terminal outside `writeRunStatus` is archived within 1 hour
1 minute;
  - 1,000 finished remote workspaces stay unread at idle.
- `npx vitest run` in `server/` on `terminal-workspace-rotation-limits`,
`terminal-workspace-work`, `terminal-workspace-deadlines`,
`terminal-workspace-idle`, `terminal-workspace-page-batch`,
`execution-workspaces-service`, `plugin-workspaces.integration`,
`workspace-operations-reconciliation`, `execution-workspace-reopen`,
`isolated-task-directory-retention`, `git-workspace-expiration` and
`workspace-delivery-cleanup.integration`.
- `pnpm check:module-boundaries` passes.

## Risks

- **A notification that relied on an immediate broad pass now waits up
to an hour.** The only caller I found that sent such a notification on
purpose is the workspace provider readiness in `plugin-registry.ts`, and
it now has its own targeted hint. Other hintless notifications arm the
backstop.
- **The isolated source orphan sweep** runs only at the start of a full
pass. Its due deadline now arms the backstop instead of starting a pass
at once. This only matters when the isolated source re-home feature is
enabled.
- **A targeted sweep has no page limit.** Its at most 50 targets bound
the matched workspaces to those tasks, their ancestors and their
execution workspaces.
- **A burst of N hints costs N/50 targeted runs, back to back.** The
cost scales with activity, not with history.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-14 12:19:43 -06:00
1 parent 890dcbdd66
commit a18479bbba
5 files changed
+263 -55

No files matched your search

@@ -0,0 +1,172 @@
import { randomUUID } from "node:crypto";
import { eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { companies, createDb, executionWorkspaceSnapshots, executionWorkspaces, issues, projects, type Db } from "@paperclipai/db";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../services/delivery-work-notifications.js";
import { executionWorkspaceService } from "../services/execution-workspaces.js";
import {
notifyTerminalWorkspaceWork, registerTerminalWorkspaceWork, TERMINAL_WORKSPACE_BACKSTOP_MS, TERMINAL_WORKSPACE_PLUGIN_HINT,
TERMINAL_WORKSPACE_RETRY_MS,
} from "../services/terminal-workspace-work.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_TERMINAL_WORKSPACE_ROTATION_TEST_DATABASE_URL");
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
const HOUR = 3_600_000;
const realSetTimeout = globalThis.setTimeout;
const realSleep = (ms: number) => new Promise(resolve => realSetTimeout(resolve, ms));
(support.supported ? describe : describe.skip)("terminal workspace full rotation limits", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: Db;
beforeAll(async () => {
if (!external) temporary = await startEmbeddedPostgresTestDatabase("terminal-workspace-rotation-");
db = createDb(external ?? temporary!.connectionString);
db.$client.options.onnotice = () => {};
}, 120_000);
afterEach(async () => { vi.useRealTimers(); await db.execute(sql`truncate companies cascade`); });
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
async function seed(input: { remote: number; plugin?: number }) {
const companyId = randomUUID(), projectId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Rotation", issuePrefix: `R${companyId.slice(0, 6)}` });
await db.insert(projects).values({ id: projectId, companyId, name: "Project", status: "in_progress" });
const old = new Date(Date.now() - 5 * 24 * HOUR);
const remote = Array.from({ length: input.remote }, () => ({ workspaceId: randomUUID(), issueId: randomUUID(), snapshotId: randomUUID() }));
const plugin = Array.from({ length: input.plugin ?? 0 }, () => ({ workspaceId: randomUUID(), issueId: randomUUID() }));
for (let index = 0; index < remote.length; index += 250) {
const page = remote.slice(index, index + 250);
await db.insert(executionWorkspaces).values(page.map(row => ({ id: row.workspaceId, companyId, projectId, mode: "isolated_workspace",
strategyType: "git_worktree", name: row.workspaceId, status: "active", providerType: "remote", createdAt: old, updatedAt: old,
metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: row.snapshotId, source: { kind: "empty" },
writerRunId: null, dirty: false, location: { environmentId: randomUUID(), leaseId: randomUUID(),
providerLeaseId: `sandbox-${row.workspaceId}`, remoteCwd: "/workspace" } } } })));
await db.insert(executionWorkspaceSnapshots).values(page.map(row => ({ id: row.snapshotId, companyId, executionWorkspaceId: row.workspaceId,
sourceOperationId: randomUUID(), generation: 1, epoch: 1, status: "committed" as const })));
}
if (plugin.length) await db.insert(executionWorkspaces).values(plugin.map(row => ({ id: row.workspaceId, companyId, projectId,
mode: "isolated_workspace", strategyType: "git_worktree", name: row.workspaceId, status: "active", providerType: "plugin",
createdAt: old, updatedAt: old })));
const all = [...remote, ...plugin];
for (let index = 0; index < all.length; index += 250) {
const page = all.slice(index, index + 250);
await db.insert(issues).values(page.map(row => ({ id: row.issueId, companyId, projectId, title: "done", status: "done",
executionWorkspaceId: row.workspaceId, completedAt: old, updatedAt: old })));
}
await db.execute(sql`update execution_workspaces set source_issue_id = issues.id, updated_at = ${old.toISOString()}::timestamptz
from issues where issues.execution_workspace_id = execution_workspaces.id and issues.company_id = ${companyId}::uuid`);
return { companyId, remote, plugin };
}
function start(canRun: () => boolean = () => true) {
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
const service = executionWorkspaceService(db, { workspaceReaperCooldownDays: 1, terminalWorkspaceInspectionBackoffMs: TERMINAL_WORKSPACE_RETRY_MS });
const sweeps: Array<{ targeted: boolean; checked: number; workspaceIds: number }> = [];
let inFlight = 0;
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun, onError: () => {} });
const worker = registerTerminalWorkspaceWork(coordinator, { ...service,
sweepTerminalWorkspaces: async (limit, signal, options) => {
inFlight += 1;
const result = await service.sweepTerminalWorkspaces(limit, signal, options).finally(() => { inFlight -= 1; });
sweeps.push({ targeted: Boolean(options?.targets), checked: result.checked,
workspaceIds: (options?.targets?.workspaceIds.length ?? 0) + (options?.targets?.issueIds.length ?? 0) });
return result;
} }, canRun);
const advance = async (ms: number) => {
for (let elapsed = 0; elapsed < ms; elapsed += 10_000) {
await vi.advanceTimersByTimeAsync(10_000);
do { await realSleep(10); await vi.advanceTimersByTimeAsync(1); } while (inFlight > 0);
}
};
const settle = async <T>(operation: Promise<T>) => {
let done = false;
const result = operation.finally(() => { done = true; });
while (!done) { await realSleep(5); await vi.advanceTimersByTimeAsync(1); }
return result;
};
const fullChecked = (from = 0) => sweeps.slice(from).filter(sweep => !sweep.targeted).reduce((sum, sweep) => sum + sweep.checked, 0);
return { sweeps, advance, settle, fullChecked, stop: async () => { vi.useRealTimers(); await worker.stop(); await coordinator.stop(); } };
}
it("works through a burst of run-finish hints in targeted batches without a full pass", async () => {
const f = await seed({ remote: 300 });
const work = start();
try {
await work.advance(60_000);
expect(work.fullChecked()).toBe(300);
const mark = work.sweeps.length;
const burst = f.remote.slice(0, 120);
for (let index = 0; index < burst.length; index += 40) {
await work.settle(db.transaction(async tx => {
for (const row of burst.slice(index, index + 40)) await notifyTerminalWorkspaceWork(tx, `issue:${row.issueId}`);
}));
}
await work.advance(5 * 60_000);
const window = work.sweeps.slice(mark);
expect(work.fullChecked(mark)).toBe(0);
expect(window.every(sweep => sweep.targeted && sweep.workspaceIds <= 50)).toBe(true);
expect(window.reduce((sum, sweep) => sum + sweep.checked, 0)).toBe(120);
} finally { await work.stop(); }
}, 300_000);
it("arms the hourly backstop for a hint without a target instead of starting a pass", async () => {
await seed({ remote: 120 });
const work = start();
try {
await work.advance(60_000);
const mark = work.sweeps.length;
await work.settle(db.transaction(tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.terminalWorkspaces)));
await work.settle(db.transaction(tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.terminalWorkspaces, "not-a-target")));
await work.advance(30 * 60_000);
expect(work.fullChecked(mark)).toBe(0);
await work.advance(TERMINAL_WORKSPACE_BACKSTOP_MS);
expect(work.fullChecked(mark)).toBe(120);
} finally { await work.stop(); }
}, 300_000);
it("inspects plugin workspaces when a workspace provider becomes ready without a full pass", async () => {
await seed({ remote: 120, plugin: 3 });
const work = start();
try {
await work.advance(60_000);
const mark = work.sweeps.length;
await work.settle(db.transaction(tx => notifyTerminalWorkspaceWork(tx, TERMINAL_WORKSPACE_PLUGIN_HINT)));
await work.advance(60_000);
expect(work.fullChecked(mark)).toBe(0);
expect(work.sweeps.slice(mark).reduce((sum, sweep) => sum + sweep.checked, 0)).toBe(3);
} finally { await work.stop(); }
}, 300_000);
it("runs one full pass across a restart: none while the old instance drains and one at the new start", async () => {
const f = await seed({ remote: 200, plugin: 2 });
let draining = false;
const old = start(() => !draining);
try {
await old.advance(60_000);
const mark = old.sweeps.length;
draining = true;
for (let index = 0; index < 120; index += 40) {
await old.settle(db.transaction(async tx => {
for (const row of f.remote.slice(index, index + 40)) await notifyTerminalWorkspaceWork(tx, `issue:${row.issueId}`);
}));
}
await old.settle(db.transaction(tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.terminalWorkspaces)));
await old.advance(5 * 60_000);
expect(old.sweeps.slice(mark)).toEqual([]);
} finally { await old.stop(); }
const fresh = start();
try {
await fresh.advance(10_000);
await fresh.settle(db.transaction(tx => notifyTerminalWorkspaceWork(tx, TERMINAL_WORKSPACE_PLUGIN_HINT)));
for (let index = 0; index < 120; index += 40) {
await fresh.settle(db.transaction(async tx => {
for (const row of f.remote.slice(index, index + 40)) await notifyTerminalWorkspaceWork(tx, `issue:${row.issueId}`);
}));
}
await fresh.advance(30 * 60_000);
expect(fresh.fullChecked()).toBe(202);
} finally { await fresh.stop(); }
}, 300_000);
});
@@ -1,7 +1,7 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { notifyTerminalWorkspaceWork, registerTerminalWorkspaceWork, TERMINAL_WORKSPACE_RETRY_MS, type TerminalWorkspaceTargets } from "../services/terminal-workspace-work.js";
import { notifyTerminalWorkspaceWork, registerTerminalWorkspaceWork, TERMINAL_WORKSPACE_BACKSTOP_MS, TERMINAL_WORKSPACE_PLUGIN_HINT, TERMINAL_WORKSPACE_RETRY_MS, type TerminalWorkspaceTargets } from "../services/terminal-workspace-work.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
const coordinators: ReturnType<typeof createDeliveryWorkCoordinator>[] = [];
@@ -88,18 +88,21 @@ describe("terminal workspace work", () => {
} finally { release?.(); }
});
it("restores the cooldown deadline, lets commits preempt it, and parks after cleanup", async () => {
it("restores the cooldown deadline, leaves unscoped commits to the hourly backstop, and parks after cleanup", async () => {
const s = setup(), due = Date.now() + 86_400_000;
s.service.nextTerminalWorkspaceSweepAt.mockResolvedValue(due);
const worker = s.start();
await worker.ready;
expect(worker.hasUnrecoveredWork()).toBe(false);
await vi.advanceTimersByTimeAsync(3_600_000);
await vi.advanceTimersByTimeAsync(30 * 60_000);
expect(s.service.sweepTerminalWorkspaces).toHaveBeenCalledTimes(1);
await s.owner.transaction(tx => notifyTerminalWorkspaceWork(tx));
await s.owner.transaction(tx => notifyTerminalWorkspaceWork(tx));
await vi.advanceTimersByTimeAsync(1);
expect(s.service.sweepTerminalWorkspaces).toHaveBeenCalledTimes(1);
await vi.advanceTimersByTimeAsync(TERMINAL_WORKSPACE_BACKSTOP_MS);
expect(s.service.sweepTerminalWorkspaces).toHaveBeenCalledTimes(2);
expect(s.service.sweepTerminalWorkspaces.mock.calls[1]?.[2]?.targets).toBeUndefined();
expect(s.coordinator.nextWakeAt()).toBe(due);
s.service.nextTerminalWorkspaceSweepAt.mockResolvedValue(null);
await vi.advanceTimersByTimeAsync(due - Date.now());
@@ -107,7 +110,7 @@ describe("terminal workspace work", () => {
expect(vi.getTimerCount()).toBe(0);
});
it("inspects only hinted workspaces after a commit, and keeps unscoped commits and deadlines on the full rotation", async () => {
it("inspects only hinted workspaces after a commit, and leaves an unscoped commit to the backstop", async () => {
const s = setup(), due = Date.now() + 60_000;
s.service.nextTerminalWorkspaceSweepAt.mockResolvedValue(due);
await s.start().ready;
@@ -121,21 +124,34 @@ describe("terminal workspace work", () => {
expect(s.service.sweepTerminalWorkspaces.mock.calls[1]?.[2]).toMatchObject({ targets: { workspaceIds: [workspaceId], issueIds: [issueId] } });
await s.owner.transaction(tx => notifyTerminalWorkspaceWork(tx));
await vi.advanceTimersByTimeAsync(1);
expect(s.service.sweepTerminalWorkspaces).toHaveBeenCalledTimes(2);
await vi.advanceTimersByTimeAsync(due - Date.now());
expect(s.service.sweepTerminalWorkspaces).toHaveBeenCalledTimes(3);
expect(s.service.sweepTerminalWorkspaces.mock.calls[2]?.[2]?.targets).toBeUndefined();
await vi.advanceTimersByTimeAsync(due - Date.now());
expect(s.service.sweepTerminalWorkspaces).toHaveBeenCalledTimes(4);
expect(s.service.sweepTerminalWorkspaces.mock.calls[3]?.[2]?.targets).toBeUndefined();
});
it("falls back to the full rotation when more workspaces are hinted than one inspection reads", async () => {
it("retries the plugin workspace listing after it fails instead of waiting for the backstop", async () => {
const s = setup();
const pluginWorkspaceId = "6b1f9c2e-8a4d-4f3b-9c1e-2d7a5b8e0f13";
const listPluginTerminalWorkspaceIds = vi.fn<() => Promise<string[]>>()
.mockRejectedValueOnce(new Error("database unavailable")).mockResolvedValue([pluginWorkspaceId]);
await registerTerminalWorkspaceWork(s.coordinator, { ...s.service, listPluginTerminalWorkspaceIds }, s.enabled).ready;
await s.owner.transaction(tx => notifyTerminalWorkspaceWork(tx, TERMINAL_WORKSPACE_PLUGIN_HINT));
await vi.advanceTimersByTimeAsync(1);
expect(listPluginTerminalWorkspaceIds).toHaveBeenCalledTimes(1);
await vi.advanceTimersByTimeAsync(TERMINAL_WORKSPACE_RETRY_MS);
expect(listPluginTerminalWorkspaceIds).toHaveBeenCalledTimes(2);
expect(s.service.sweepTerminalWorkspaces.mock.calls.at(-1)?.[2]?.targets).toEqual({ workspaceIds: [pluginWorkspaceId], issueIds: [] });
});
it("works through more hinted workspaces than one inspection reads in targeted batches", async () => {
const s = setup();
await s.start().ready;
const ids = Array.from({ length: 51 }, (_, index) => `00000000-0000-4000-8000-${String(index).padStart(12, "0")}`);
await s.owner.transaction(async tx => { for (const id of ids) await notifyTerminalWorkspaceWork(tx, `workspace:${id}`); });
await vi.advanceTimersByTimeAsync(1);
expect(s.service.sweepTerminalWorkspaces).toHaveBeenCalledTimes(2);
expect(s.service.sweepTerminalWorkspaces.mock.calls[1]?.[2]?.targets).toBeUndefined();
await vi.advanceTimersByTimeAsync(10);
expect(s.service.sweepTerminalWorkspaces).toHaveBeenCalledTimes(3);
expect(s.service.sweepTerminalWorkspaces.mock.calls.slice(1).map(call => call[2]?.targets?.workspaceIds.length)).toEqual([50, 1]);
});
it("keeps the pause between rotation pages while hinted workspaces wait", async () => {
+21 -9
View File
@@ -39,7 +39,7 @@ import { authorizationService, executionWorkspaceReadSqlCondition, projectReadSq
import type { SQL } from "drizzle-orm";
import { nativeSessionProviderEvidence } from "./native-runtime/native-session-resume.js";
import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
import { notifyTerminalWorkspaceWork, terminalWorkspaceTargetsFromHints, type TerminalWorkspaceTargets } from "./terminal-workspace-work.js";
import { notifyTerminalWorkspaceWork, type TerminalWorkspaceTargets } from "./terminal-workspace-work.js";
import {
clearTerminalWorkspaceRetry, readTerminalWorkspaceRetry, recordTerminalWorkspaceFailure, terminalWorkspaceFailure,
TERMINAL_WORKSPACE_RETRY_KEY, type TerminalWorkspaceFailure,
@@ -3688,16 +3688,15 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
: undefined;
const scanFilter = targets ? and(baseCandidateFilter, terminalWorkspaceTargetFilter(targets))
: and(baseCandidateFilter, boundaryFilter, cursorFilter);
const candidates = await db
const candidateQuery = db
.select({ workspace: executionWorkspaces, cursorAt: sql<string>`${executionWorkspaces.updatedAt}::text` })
.from(executionWorkspaces)
.where(scanFilter)
.orderBy(asc(executionWorkspaces.updatedAt), asc(executionWorkspaces.id))
.limit(limit);
.orderBy(asc(executionWorkspaces.updatedAt), asc(executionWorkspaces.id));
const candidates = await (targets ? candidateQuery : candidateQuery.limit(limit));
// Advance the cursor to this page's last row. A short page means the scan
// reached the end of the bounded candidate set, so reset the cursor and
// the bound to start a new rotation on the next sweep.
if (targets && candidates.length >= limit) terminalSweepRescanRequested = true;
if (!targets && candidates.length < limit) {
terminalSweepCursor = null;
terminalSweepBoundary = null;
@@ -4398,8 +4397,8 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
if (!options.targets) {
terminalSweepCursor = previousCursor;
terminalSweepBoundary = previousBoundary;
terminalSweepRescanRequested = true;
}
terminalSweepRescanRequested = true;
throw error;
} finally {
terminalSweepInProgress = false;
@@ -4409,7 +4408,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
hasTerminalWorkspaceSweepContinuation: () => terminalSweepCursor !== null || terminalSweepRescanRequested,
terminalWorkspaceWorkSettled: (hints?: ReadonlySet<string>, options: { targeted?: boolean } = {}) => {
if ((terminalSweepInProgress || terminalSweepCursor !== null) && !(options.targeted && terminalWorkspaceTargetsFromHints(hints))) {
if ((terminalSweepInProgress || terminalSweepCursor !== null) && !options.targeted) {
terminalSweepRescanRequested = true;
}
pullRequestStateCache.clear();
@@ -4432,11 +4431,24 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
}
},
listPluginTerminalWorkspaceIds: async () => {
const rows = await db.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(
eq(executionWorkspaces.providerType, "plugin"),
sql<boolean>`${executionWorkspaces.sourceIssueId} IS NOT NULL`,
or(and(inArray(executionWorkspaces.status, ["active", "idle", "in_review"]), isNull(executionWorkspaces.closedAt)),
and(inArray(executionWorkspaces.status, ["archived", "cleanup_failed"]),
sql`(${executionWorkspaces.metadata}->>'workspaceProviderRemoved') IS DISTINCT FROM 'true'`)),
));
return rows.map(row => row.id);
},
takeDueTerminalWorkspaceTargets: (): { workspaceIds: string[]; rotation: boolean } => {
const at = now().getTime();
const due = (deadlines: Map<string, number>) => [...deadlines].filter(([, deadline]) => deadline <= at).map(([id]) => id);
const rotation = due(isolatedRehomeDeadlines).length + due(nestedCloneMigrationDeadlines).length + due(taskDirectoryDeadlines).length > 0;
return { workspaceIds: due(terminalWorkspaceDeadlines), rotation };
const workspaceKey = (key: string) => /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(key);
const migrations = [...due(isolatedRehomeDeadlines), ...due(nestedCloneMigrationDeadlines)];
const rotation = due(taskDirectoryDeadlines).length > 0 || migrations.some(key => !workspaceKey(key));
return { workspaceIds: [...new Set([...due(terminalWorkspaceDeadlines), ...migrations.filter(workspaceKey)])], rotation };
},
nextTerminalWorkspaceSweepAt: async () => {
+3 -2
View File
@@ -32,6 +32,7 @@ import type {
} from "@paperclipai/shared";
import { conflict, notFound } from "../errors.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import { TERMINAL_WORKSPACE_PLUGIN_HINT } from "./terminal-workspace-work.js";
function sameBundledPluginRegistration(existing: typeof plugins.$inferSelect, packageName: string, manifest: PaperclipPluginManifestV1): boolean {
return existing.pluginKey === manifest.id && existing.packageName === packageName && existing.version === manifest.version && isDeepStrictEqual(existing.manifestJson, manifest);
@@ -92,7 +93,7 @@ export function pluginRegistryService(db: Db) {
async function writeWorkspaceProviderChange<T>(reconsiderWorkspaces: boolean, write: (database: Db) => Promise<T>): Promise<T> {
return db.transaction(async tx => {
await notifyExternalObjectRefresh(tx);
if (reconsiderWorkspaces) await notifyDeliveryWork(tx, DELIVERY_QUEUES.terminalWorkspaces);
if (reconsiderWorkspaces) await notifyDeliveryWork(tx, DELIVERY_QUEUES.terminalWorkspaces, TERMINAL_WORKSPACE_PLUGIN_HINT);
return write(tx as unknown as Db);
});
}
@@ -517,7 +518,7 @@ export function pluginRegistryService(db: Db) {
.then((rows) => rows[0] ?? null);
if (plugin.manifestJson.workspaceProvider && existing?.enabled === false && input.enabled === true) {
await notifyDeliveryWork(db, DELIVERY_QUEUES.terminalWorkspaces);
await notifyDeliveryWork(db, DELIVERY_QUEUES.terminalWorkspaces, TERMINAL_WORKSPACE_PLUGIN_HINT);
}
if (existing) {
+40 -33
View File
@@ -4,6 +4,7 @@ import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notificatio
export const TERMINAL_WORKSPACE_RETRY_MS = 5 * 60 * 1000;
export const MAX_TERMINAL_WORKSPACE_TARGETS = 50;
export const TERMINAL_WORKSPACE_BACKSTOP_MS = 60 * 60 * 1000;
export const TERMINAL_WORKSPACE_PLUGIN_HINT = "plugin-workspaces";
export function notifyTerminalWorkspaceWork(transaction: object, hint?: string) {
return notifyDeliveryWork(transaction, DELIVERY_QUEUES.terminalWorkspaces, hint);
@@ -38,6 +39,7 @@ export function registerTerminalWorkspaceWork(
hasTerminalWorkspaceSweepContinuation(): boolean;
terminalWorkspaceWorkSettled?(hints?: ReadonlySet<string>, options?: { targeted?: boolean }): void;
takeDueTerminalWorkspaceTargets?(): { workspaceIds: string[]; rotation: boolean };
listPluginTerminalWorkspaceIds?(): Promise<string[]>;
},
canRun: () => boolean,
) {
@@ -48,26 +50,31 @@ export function registerTerminalWorkspaceWork(
// can take seconds per workspace, so the first sweep skips it and later
// sweeps do it.
let startupSweepStarted = false;
let rotationRequested = false;
let dueAt: number | null = null;
let lastRotationPageAt = 0;
let backstopArmed = false;
let lastFullRotationAt = Date.now();
let continuing = false;
const targetWorkspaceIds = new Set<string>(), targetIssueIds = new Set<string>();
let pluginRefresh = false;
const pendingWorkspaceIds = new Set<string>(), pendingIssueIds = new Set<string>();
const pendingCount = () => pendingWorkspaceIds.size + pendingIssueIds.size;
const takeTargets = (): TerminalWorkspaceTargets => {
const workspaceIds = [...pendingWorkspaceIds].slice(0, MAX_TERMINAL_WORKSPACE_TARGETS);
for (const id of workspaceIds) pendingWorkspaceIds.delete(id);
const issueIds = [...pendingIssueIds].slice(0, MAX_TERMINAL_WORKSPACE_TARGETS - workspaceIds.length);
for (const id of issueIds) pendingIssueIds.delete(id);
return { workspaceIds, issueIds };
};
const worker = coordinator.register(DELIVERY_QUEUES.terminalWorkspaces, {
retryMs: TERMINAL_WORKSPACE_RETRY_MS,
onSettled: hints => {
backstopArmed = true;
const targets = terminalWorkspaceTargetsFromHints(hints);
for (const id of targets?.workspaceIds ?? []) targetWorkspaceIds.add(id);
for (const id of targets?.issueIds ?? []) targetIssueIds.add(id);
const overflow = targetWorkspaceIds.size + targetIssueIds.size > MAX_TERMINAL_WORKSPACE_TARGETS;
service.terminalWorkspaceWorkSettled?.(hints, { targeted: Boolean(targets) && !overflow });
if (!targets || overflow) {
rotationRequested = true;
targetWorkspaceIds.clear(); targetIssueIds.clear();
}
if (hints?.has(TERMINAL_WORKSPACE_PLUGIN_HINT)) pluginRefresh = true;
const scoped = hints ? new Set([...hints].filter(hint => hint !== TERMINAL_WORKSPACE_PLUGIN_HINT)) : undefined;
const targets = terminalWorkspaceTargetsFromHints(scoped);
for (const id of targets?.workspaceIds ?? []) pendingWorkspaceIds.add(id);
for (const id of targets?.issueIds ?? []) pendingIssueIds.add(id);
service.terminalWorkspaceWorkSettled?.(hints, { targeted: true });
},
run: async signal => {
recovered = false;
@@ -76,44 +83,44 @@ export function registerTerminalWorkspaceWork(
next = Date.now() + TERMINAL_WORKSPACE_RETRY_MS;
const startup = !startupSweepStarted;
startupSweepStarted = true;
let dueRotation = due;
const options = { compact: !startup, allowRemoteDeliveryChecks: !startup };
let prelude = false;
const legacyDue = due && !startup && !service.takeDueTerminalWorkspaceTargets;
if (due && !startup && service.takeDueTerminalWorkspaceTargets) {
const dueTargets = service.takeDueTerminalWorkspaceTargets();
dueRotation = dueTargets.rotation;
for (const id of dueTargets.workspaceIds) targetWorkspaceIds.add(id);
if (targetWorkspaceIds.size + targetIssueIds.size > MAX_TERMINAL_WORKSPACE_TARGETS) {
dueRotation = true;
targetWorkspaceIds.clear(); targetIssueIds.clear();
for (const id of dueTargets.workspaceIds) pendingWorkspaceIds.add(id);
if (dueTargets.rotation) { prelude = true; backstopArmed = true; }
}
if (pluginRefresh && !startup && service.listPluginTerminalWorkspaceIds) {
pluginRefresh = false;
try {
for (const id of await service.listPluginTerminalWorkspaceIds()) pendingWorkspaceIds.add(id);
} catch (error) {
pluginRefresh = true;
throw error;
}
}
const options = { compact: !startup, allowRemoteDeliveryChecks: !startup };
const targeted = targetWorkspaceIds.size + targetIssueIds.size > 0;
const backstopDue = backstopArmed && Date.now() >= lastFullRotationAt + TERMINAL_WORKSPACE_BACKSTOP_MS;
if (due && service.takeDueTerminalWorkspaceTargets && !startup && !dueRotation && !targeted && !rotationRequested && !backstopDue
&& !service.hasTerminalWorkspaceSweepContinuation()) return { checked: 0 };
if (startup || dueRotation || rotationRequested || backstopDue || !targeted
|| (service.hasTerminalWorkspaceSweepContinuation() && Date.now() >= lastRotationPageAt + 1000)) {
rotationRequested = false;
if (startup || backstopDue || legacyDue || (continuing && Date.now() >= lastRotationPageAt + 1000)) {
lastRotationPageAt = Date.now();
if (!continuing) { lastFullRotationAt = Date.now(); backstopArmed = false; }
if (!continuing) { lastFullRotationAt = Date.now(); backstopArmed = false; pluginRefresh = false; }
return service.sweepTerminalWorkspaces(50, signal, options);
}
const targets = { workspaceIds: [...targetWorkspaceIds], issueIds: [...targetIssueIds] };
targetWorkspaceIds.clear(); targetIssueIds.clear();
if (pendingCount() === 0 && !prelude) return { checked: 0 };
const targets = takeTargets();
try {
return await service.sweepTerminalWorkspaces(50, signal, { ...options, targets });
} catch (error) {
rotationRequested = true;
for (const id of targets.workspaceIds) pendingWorkspaceIds.add(id);
for (const id of targets.issueIds) pendingIssueIds.add(id);
throw error;
}
},
hasPending: async () => {
continuing = false;
if (canRun()) continuing = service.hasTerminalWorkspaceSweepContinuation();
if (!canRun()) next = Date.now() + TERMINAL_WORKSPACE_RETRY_MS;
else if (service.hasTerminalWorkspaceSweepContinuation()) {
continuing = true;
next = Math.max(Date.now(), lastRotationPageAt + 1000);
} else if (rotationRequested || targetWorkspaceIds.size + targetIssueIds.size > 0) next = Date.now();
else if (continuing) next = Math.max(Date.now(), lastRotationPageAt + 1000);
else if (pendingCount() > 0 || pluginRefresh) next = Date.now();
else {
const deadline = await service.nextTerminalWorkspaceSweepAt();
next = deadline === null ? null : deadline > Date.now()