mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-27 11:20:42 +01:00
fix(recovery): wake retained workspace preservation after Retry (#17741)
## Thinking Path > - Paperclip lets people manage AI agents and their saved work. > - A stopped run can owe workspace preservation before saved input can continue. > - Retry saved a recovery request but did not wake the worker that owns preservation. > - Older recovery records could also take the bounded scan slots before a due request. > - A second Retry could change the revision while the physical owner was still saving files. > - This pull request wakes a separate worker after commit, selects due work first, and keeps the active owner's revision. > - Saved work can progress without replaying the stopped provider turn or blocking other finalization. ## Linked Issues or Issue Description **What happened?** An operator approved a saved design and selected Retry after native workspace finalization failed. The API accepted one canonical request. The source remained scheduled at zero preservation attempts for more than 30 minutes. No new provider turn started. A local regression also found that repeated Retry during export invalidated the physical owner's completion revision. **Expected behavior** Committed Retry must wake bounded source preservation. Due recovery must progress ahead of passive cleanup obligations and unrelated history. Duplicate requests must keep one physical attempt. Saved input must still pass normal admission after verified recovery. **Steps to reproduce** 1. Retain a terminal run with an exhausted workspace recovery and saved authorized input. 2. Let the delivery coordinator become quiet, then request Retry. 3. Observe the recovery state and physical preservation receipt. Repeat Retry while the export is held. **Paperclip version or commit** Observed on served commit `4b89eec644d6e1a99d58926e77ae58927c3fa87b`. Local baseline: `a1f77821b9cd5c9c846fdf7a0fa5cc620ad2b6bb`. Related work: Refs #17053, #17267, #17094. This change repairs explicit Retry delivery. It is separate from the run instruction projection in #17724 and historical unsafe-export discovery in #17687. ## What Changed - Notify a dedicated workspace recovery duty from the transaction that saves Retry, recovery admission, adoption, or recovery progress. - Reconstruct work and saved retry deadlines at startup. Fill each free export slot independently, track physical work for idle drain, and join the dedicated worker's exports on shutdown. - Drain one bounded pass through completed cleanup and saved-input obligations. Restart that pass after committed changes without repeatedly rechecking paused input. - Share source eligibility between deadline selection and candidate consumption. Keep resolved files-only obligations and ignore unrelated resolved or cancelled history. - Select due unfinished preservation before passive historical records. Rotate past source locks held by other workers and retain the indexed source lookup. - Bind the canonical Retry intent without changing an active physical attempt's generation or revision. Preserve that binding across later progress writes. - Add real producer, physical preservation, startup, deadline, idle, and independent finalization regressions. Document the delivery contract. ## Verification - Red controls: Retry without a notification stayed scheduled; duplicate Retry invalidated the active export; whole-batch waiting stranded a third Retry; a scan without cursor rotation starved a source behind two actual advisory locks. Removing unfinished-first selection left HTTP Retry scheduled behind five real recovered sources still owing cleanup. The fixture explicitly verifies their eligibility with the worker's actual query filter. - Further red controls: the previous worker visits only two of three eligible completed sources. Disabling the committed-change cursor reset misses a new earlier source arriving while two callbacks are held. Both pass with the final worker. - Green: the public Retry HTTP route, registered production worker, real file copyback, duplicate canonical receipts, and public recovery receipt reload passed on an owned local fixture. Its agent is paused to keep this files-only check from starting a provider turn. - Full workspace recovery suite passed 56 tests, including the foreign-lock regression. - Execution-control lifecycle suite passed 24 tests. It covers idle SQL, stored deadlines, cancelled and resolved history, independent finalization, and shutdown. One, three, and seven paused completed sources are each visited in a bounded pass, then stay deferred without a successor or SQL during 90 seconds of fake-clock idle time. A committed earlier source is evaluated when it arrives during two held callbacks. - Both GitHub and Telegram `automatic_running` continuation regressions passed with the new canonical intent binding. - `pnpm -r typecheck` passed. - `pnpm build` passed. - `node scripts/check-maps.mjs` and `node scripts/check-module-boundaries.mjs` passed. - Local `pnpm test:run` could not complete: embedded PostgreSQL repeatedly failed to start. The owned run was stopped after this setup failure repeated. Latest-head CI supplies broad coverage. - Earlier setup failures included a localhost sandbox denial and loss of the owned disposable Docker database. These were separate from the product red controls above. - Local database tests use an explicitly named disposable PostgreSQL `_test` database. That opt-in database is reset before the suite. Default embedded test behavior is unchanged. - Exact-head CI is green at `ef7dcf1ba1060864c9d216c33b51f4d9f8f57fbd`; Greptile and Ponytail both rate it 5/5, with no unresolved review threads. Earlier CI caught the indexed discovery assertion and two exact chat intent-binding regressions; all three were fixed and pass locally and in CI. A separate chat shard lost contact with its self-hosted runner without a product assertion result. A later shared runner shutdown interrupted several jobs; one infrastructure rerun passed before the final test-and-worker follow-up triggered a fresh green run. - No staging deployment is part of this PR. The original approval remains saved; deployed recovery still needs acceptance after rollout. ## Risks - More recovery writes now wake a bounded maintenance duty. Completed history must not keep it active. Idle and deadline tests cover this boundary. - Resolved files-only obligations must remain eligible without granting replay. The startup regression checks that path. - Provider stop proof, copyback ownership, source binding, retry budgets, and normal execution admission remain required. There is no schema or public API change. ## Model Used - OpenAI Codex based on GPT-6, with code execution and browser tools. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
338bf75761
commit
0373f9f261
13 files changed
+527
-63
No files matched your search
@@ -4,7 +4,7 @@ import { localAiLoginService } from "../services/local-ai-login.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { agents, companies, createDb, heartbeatRuns, issues, nativeRunFinalizations, applyPendingMigrations } from "@paperclipai/db";
|
||||
import { agents, agentWakeupRequests, companies, createDb, heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, applyPendingMigrations } from "@paperclipai/db";
|
||||
import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
@@ -16,7 +16,13 @@ import { terminalizeLegacyExecution } from "../services/legacy-execution-recover
|
||||
import { reconcileAbandonedExecutionControl } from "../services/execution-control-reconciliation.js";
|
||||
import { issueRecoveryActionService } from "../services/issue-recovery-actions.js";
|
||||
import { markExecutionReconciliation, deliverReconciledExecutions, settleUnrecoverableExecutions } from "../services/execution-recovery-resolution.js";
|
||||
import { nextWorkspaceRecoveryAt } from "../services/system-recovery.js";
|
||||
import { createSystemRecovery } from "../services/system-recovery-state.js";
|
||||
import { retainedWorkspaceCandidates } from "../services/workspace-recovery.js";
|
||||
import { registerWorkspaceRecoveryWork } from "../services/workspace-recovery-work.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { requestTaskRetry } from "../services/task-retry.js";
|
||||
|
||||
const coordinators: ReturnType<typeof createDeliveryWorkCoordinator>[] = [];
|
||||
afterEach(async () => { for (const worker of coordinators.splice(0)) await worker.stop(); vi.useRealTimers(); });
|
||||
@@ -94,13 +100,13 @@ describe("execution-control lifecycle", () => {
|
||||
expect(run).toHaveBeenCalledTimes(1); expect(next).toHaveBeenCalledTimes(1);
|
||||
await c.stop(); expect(c.nextWakeAt()).toBeNull();
|
||||
});
|
||||
it("keeps other obligations responsive while one is waiting, and awaits shutdown", async () => {
|
||||
it.each([DELIVERY_QUEUES.executionReplacement, DELIVERY_QUEUES.workspaceRecovery])("keeps finalization responsive while %s is waiting, and awaits shutdown", async heldQueue => {
|
||||
vi.useFakeTimers(); const owner = fakeOwner(); const c = coordinator(owner);
|
||||
let release!: () => void; const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
const status = vi.fn(async () => {});
|
||||
const started = registerExecutionControlWork(c, [
|
||||
{ queue: DELIVERY_QUEUES.executionReplacement, run: () => gate, next: async () => null },
|
||||
{ queue: DELIVERY_QUEUES.executionStatus, run: status, next: async () => null },
|
||||
{ queue: heldQueue, run: () => gate, next: async () => null },
|
||||
{ queue: DELIVERY_QUEUES.executionFinalization, run: status, next: async () => null },
|
||||
]);
|
||||
await vi.advanceTimersByTimeAsync(1); expect(status).toHaveBeenCalledTimes(1);
|
||||
let stopped = false; const stopping = c.stop().then(() => { stopped = true; });
|
||||
@@ -120,6 +126,7 @@ describe.skipIf(!support.supported)("execution-control durable work", () => {
|
||||
if (external) await applyPendingMigrations(external);
|
||||
else temporary = await startEmbeddedPostgresTestDatabase("execution-sleep-");
|
||||
db = createDb(external ?? temporary!.connectionString, { maxConnections: 6 });
|
||||
await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.05)`)));
|
||||
}, 90000);
|
||||
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
|
||||
beforeEach(async () => {
|
||||
@@ -145,22 +152,140 @@ describe.skipIf(!support.supported)("execution-control durable work", () => {
|
||||
await settle(); expect(c.nextWakeAt()).toBeNull(); const before = calls;
|
||||
await vi.advanceTimersByTimeAsync(90_000); expect(calls).toBe(before);
|
||||
}
|
||||
async function runRow(status = "running") {
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status, runtimeMode: "legacy", contextSnapshot: { issueId } }).returning();
|
||||
async function runRow(status = "running", sourceIssueId = issueId) {
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status, runtimeMode: "legacy", contextSnapshot: { issueId: sourceIssueId } }).returning();
|
||||
return run!;
|
||||
}
|
||||
it("stops all six pending/deadline scans with empty queues and completed history", async () => {
|
||||
async function pausedRecoveredSource(actionId = randomUUID()) {
|
||||
const sourceIssueId = randomUUID();
|
||||
await db.insert(issues).values({ id: sourceIssueId, companyId, title: "Recovered execution",
|
||||
status: "in_progress", assigneeAgentId: agentId });
|
||||
const run = await runRow("failed", sourceIssueId);
|
||||
await db.update(heartbeatRuns).set({ resultJson: { startupCancellation: { beforeNativeSelection: true } } })
|
||||
.where(eq(heartbeatRuns.id, run.id));
|
||||
await db.update(agents).set({ status: "paused" }).where(eq(agents.id, agentId));
|
||||
await db.insert(issueRecoveryActions).values({ id: actionId, companyId, sourceIssueId,
|
||||
kind: "active_run_watchdog", ownerType: "system", cause: "native_workspace_sync_out_retry_exhausted",
|
||||
fingerprint: run.id, nextAction: "Saved work recovered", evidence: { runId: run.id,
|
||||
systemRecovery: { ...createSystemRecovery(run.id), state: "complete" },
|
||||
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: run.id,
|
||||
status: "recovered", stage: "saving_receipts", attempt: 1, attemptLimit: 6, requestIds: [] } } });
|
||||
const retry = await requestTaskRetry({ db, companyId, issueId: sourceIssueId, actorId: "board",
|
||||
observedRunId: run.id, idempotencyKey: randomUUID() });
|
||||
expect(retry.accepted).toBe(true);
|
||||
return { run, retry };
|
||||
}
|
||||
it("stops all pending/deadline scans with empty queues and completed history", async () => {
|
||||
await runRow("succeeded");
|
||||
const schedule = executionControlSchedule(observed);
|
||||
const c = coordinator(db, undefined, error => { errors.push(error); });
|
||||
const topics = [DELIVERY_QUEUES.executionFinalization, DELIVERY_QUEUES.executionReplacement, DELIVERY_QUEUES.executionContinuation,
|
||||
const topics = [DELIVERY_QUEUES.executionFinalization, DELIVERY_QUEUES.workspaceRecovery, DELIVERY_QUEUES.executionReplacement, DELIVERY_QUEUES.executionContinuation,
|
||||
DELIVERY_QUEUES.executionStatus, DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.localLoginCleanup];
|
||||
const runs = [() => reconcileAbandonedExecutionControl(observed), () => reconcileSafeNativeReplacements(observed),
|
||||
const runs = [() => reconcileAbandonedExecutionControl(observed), async () => {}, () => reconcileSafeNativeReplacements(observed),
|
||||
() => deliverReconciledExecutions(observed, vi.fn(async () => null)), () => deliverExecutionStatuses(observed),
|
||||
() => settleUnrecoverableExecutions(observed), () => localAiLoginService(observed).reapExpired()];
|
||||
await registerExecutionControlWork(c, Object.values(schedule).map((next, i) => ({ queue: topics[i]!, run: runs[i]!, next }))).ready;
|
||||
await quiet(c);
|
||||
});
|
||||
it("schedules only a current unfinished workspace recovery and becomes quiet with retained history", async () => {
|
||||
const run = await runRow("failed");
|
||||
const state = createSystemRecovery(run.id);
|
||||
const evidence = { runId: run.id, systemRecovery: state,
|
||||
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: run.id,
|
||||
status: "queued", stage: "verifying_stop", attempt: 0, attemptLimit: 6, requestIds: [] } };
|
||||
const [action] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId,
|
||||
kind: "active_run_watchdog", ownerType: "system", cause: "native_workspace_sync_out_retry_exhausted",
|
||||
fingerprint: run.id, nextAction: "Save work", evidence }).returning();
|
||||
expect(await nextWorkspaceRecoveryAt(db)).toBe(0);
|
||||
const due = Date.now() + 60_000;
|
||||
await db.update(issueRecoveryActions).set({ evidence: { ...evidence, systemRecovery: { ...state, nextAttemptAt: due } } })
|
||||
.where(eq(issueRecoveryActions.id, action!.id));
|
||||
expect(await executionControlSchedule(db).workspaceRecovery()).toBe(due);
|
||||
for (const status of ["cancelled", "resolved"] as const) {
|
||||
await db.update(issueRecoveryActions).set({ status }).where(eq(issueRecoveryActions.id, action!.id));
|
||||
expect(await nextWorkspaceRecoveryAt(db)).toBeNull();
|
||||
expect(await retainedWorkspaceCandidates(db)).toEqual([]);
|
||||
}
|
||||
for (const system of [{ ...state, sourceRunId: randomUUID() }, { ...state, state: "complete" }, { ...state, state: "exhausted" }]) {
|
||||
await db.update(issueRecoveryActions).set({ status: "active", evidence: { ...evidence, systemRecovery: system } })
|
||||
.where(eq(issueRecoveryActions.id, action!.id));
|
||||
expect(await nextWorkspaceRecoveryAt(db)).toBeNull();
|
||||
}
|
||||
await db.update(issueRecoveryActions).set({ evidence, cause: "agent_unreachable" })
|
||||
.where(eq(issueRecoveryActions.id, action!.id));
|
||||
expect(await nextWorkspaceRecoveryAt(db)).toBeNull();
|
||||
await db.update(issueRecoveryActions).set({ cause: "native_workspace_sync_out_retry_exhausted",
|
||||
evidence: { ...evidence, workspaceRecovery: { ...evidence.workspaceRecovery, runId: randomUUID() } } })
|
||||
.where(eq(issueRecoveryActions.id, action!.id));
|
||||
expect(await nextWorkspaceRecoveryAt(db)).toBeNull();
|
||||
expect(await retainedWorkspaceCandidates(db)).toEqual([]);
|
||||
await db.update(issueRecoveryActions).set({ status: "cancelled", evidence: { ...evidence,
|
||||
systemRecovery: { ...state, state: "running" } } }).where(eq(issueRecoveryActions.id, action!.id));
|
||||
const c = coordinator(db, undefined, error => { errors.push(error); });
|
||||
const worker = registerWorkspaceRecoveryWork(observed, c,
|
||||
{} as Parameters<typeof registerWorkspaceRecoveryWork>[2], async () => {});
|
||||
try { await worker.ready; await quiet(c); } finally { await worker.stop(); }
|
||||
});
|
||||
|
||||
it.each([1, 3, 7])("visits all %i paused Retry sources once and becomes quiet after file recovery", async count => {
|
||||
const sources = [];
|
||||
for (let index = 0; index < count; index++) sources.push(await pausedRecoveredSource());
|
||||
const resume = vi.fn(heartbeatService(observed).resumeRemoteStopComments);
|
||||
const c = coordinator(db, undefined, error => { errors.push(error); });
|
||||
const worker = registerWorkspaceRecoveryWork(observed, c,
|
||||
{} as Parameters<typeof registerWorkspaceRecoveryWork>[2], resume);
|
||||
try {
|
||||
await worker.ready;
|
||||
for (let pass = 0; pass < 10 && resume.mock.calls.length < count; pass++) await settle();
|
||||
expect(resume).toHaveBeenCalledTimes(count);
|
||||
await quiet(c);
|
||||
for (const result of resume.mock.results) {
|
||||
await expect(result.value).rejects.toThrow("Agent is not invokable in its current state");
|
||||
}
|
||||
for (const { retry } of sources) {
|
||||
const [saved] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.requestId!));
|
||||
expect(saved).toMatchObject({ status: "deferred_issue_execution", runId: null });
|
||||
}
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId))).toHaveLength(count);
|
||||
expect((await db.select().from(agents).where(eq(agents.id, agentId)))[0]?.status).toBe("paused");
|
||||
} finally { await c.stop(); await worker.stop(); }
|
||||
});
|
||||
|
||||
it("visits a newly committed earlier Retry during a busy recovered-source pass, then becomes quiet", async () => {
|
||||
const sources = [];
|
||||
for (let index = 0; index < 3; index++) {
|
||||
sources.push(await pausedRecoveredSource(`ffffffff-000${index}-4000-8000-${randomUUID().slice(-12)}`));
|
||||
}
|
||||
let release!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
const heartbeat = heartbeatService(observed);
|
||||
const resume = vi.fn(async (run: Parameters<typeof heartbeat.resumeRemoteStopComments>[0]) => {
|
||||
await gate;
|
||||
return heartbeat.resumeRemoteStopComments(run);
|
||||
});
|
||||
const c = coordinator(db, undefined, error => { errors.push(error); });
|
||||
const worker = registerWorkspaceRecoveryWork(observed, c,
|
||||
{} as Parameters<typeof registerWorkspaceRecoveryWork>[2], resume);
|
||||
try {
|
||||
await worker.ready;
|
||||
await vi.waitFor(() => expect(resume).toHaveBeenCalledTimes(2));
|
||||
const earlier = await pausedRecoveredSource(`00000000-0000-4000-8000-${randomUUID().slice(-12)}`);
|
||||
sources.push(earlier);
|
||||
release();
|
||||
for (let pass = 0; pass < 10 && sources.some(({ run }) =>
|
||||
!resume.mock.calls.some(([visited]) => visited.id === run.id)); pass++) await settle();
|
||||
expect(resume.mock.calls.map(([run]) => run.id)).toContain(earlier.run.id);
|
||||
await quiet(c);
|
||||
for (const { run, retry } of sources) {
|
||||
expect(resume.mock.calls.map(([visited]) => visited.id)).toContain(run.id);
|
||||
const [saved] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.requestId!));
|
||||
expect(saved).toMatchObject({ status: "deferred_issue_execution", runId: null });
|
||||
}
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId))).toHaveLength(4);
|
||||
expect((await db.select().from(agents).where(eq(agents.id, agentId)))[0]?.status).toBe("paused");
|
||||
} finally { release(); await c.stop(); await worker.stop(); }
|
||||
});
|
||||
|
||||
it("bounds real producer bursts with another company's completed history present", async () => {
|
||||
const otherCompany = randomUUID(), otherAgent = randomUUID();
|
||||
await db.insert(companies).values({ id: otherCompany, name: "Other tenant", issuePrefix: "OTH" });
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { registerWorkspaceRecoveryWork } from "./services/workspace-recovery-work.js";
|
||||
import { registerEnvironmentCleanupWork } from "./services/environment-cleanup-work.js";
|
||||
import { loginCleanupSchedule } from "./services/login-cleanup-state.js";
|
||||
import { registerLoginCleanupWork } from "./services/login-cleanup-work.js";
|
||||
@@ -1342,6 +1343,9 @@ async function startServerWithDatabaseTeardown(
|
||||
});
|
||||
const deliveryWork = app.locals.deliveryWork as ReturnType<typeof createDeliveryWorkCoordinator>;
|
||||
const executionSchedule = executionControlSchedule(db);
|
||||
const workspaceRecoveryRuntime = environmentRuntimeService(db, { pluginWorkerManager });
|
||||
const workspaceRecoveryWork = registerWorkspaceRecoveryWork(db, deliveryWork, workspaceRecoveryRuntime,
|
||||
accountingHeartbeat.resumeRemoteStopComments, () => !heartbeatSchedulerStopped);
|
||||
registerExecutionControlWork(deliveryWork, [
|
||||
{ queue: DELIVERY_QUEUES.executionFinalization, run: () => reconcileAbandonedExecutionControl(db), next: executionSchedule.finalization },
|
||||
{ queue: DELIVERY_QUEUES.executionStatus, run: () => deliverExecutionStatuses(db), next: executionSchedule.status },
|
||||
@@ -2103,6 +2107,7 @@ async function startServerWithDatabaseTeardown(
|
||||
await skillSourceSync.stop();
|
||||
unsubscribeChatCompletions();
|
||||
await deliveryWork.stop();
|
||||
await workspaceRecoveryWork.stop();
|
||||
unregisterTerminalWorkspaceProbe?.();
|
||||
unsubscribeNativeWorkspaceSyncRetention?.();
|
||||
if (heartbeatSchedulerInterval) {
|
||||
|
||||
@@ -3,6 +3,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
/** Topics for existing durable queues; register intent before their writes. */
|
||||
export const DELIVERY_QUEUES = {
|
||||
executionFinalization: "execution-finalization",
|
||||
workspaceRecovery: "workspace-recovery",
|
||||
executionReplacement: "execution-replacement",
|
||||
executionContinuation: "execution-continuation",
|
||||
executionStatus: "execution-status",
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { nextWorkspaceRecoveryAt } from "./system-recovery.js";
|
||||
import { and, asc, eq, isNotNull, sql } from "drizzle-orm";
|
||||
import { adapterAuthSessions, heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
||||
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
||||
@@ -12,6 +13,7 @@ export function registerExecutionControlWork(
|
||||
queue: DeliveryQueue;
|
||||
run: () => Promise<unknown>;
|
||||
next: () => Promise<number | null>;
|
||||
onSettled?: () => void;
|
||||
}>,
|
||||
canRun: () => boolean = () => true,
|
||||
) {
|
||||
@@ -20,10 +22,10 @@ export function registerExecutionControlWork(
|
||||
let reconciled = 0;
|
||||
let next: number | null = null;
|
||||
let earliestRunAt = 0;
|
||||
return coordinator.register(duty.queue, {
|
||||
const worker = coordinator.register(duty.queue, {
|
||||
retryMs: EXECUTION_RECONCILIATION_INTERVAL_MS,
|
||||
pausePassiveMaintenance: true,
|
||||
onSettled: () => { requested += 1; },
|
||||
onSettled: () => { requested += 1; duty.onSettled?.(); },
|
||||
run: async () => {
|
||||
if (!canRun()) { requested += 1; next = null; return; }
|
||||
const now = Date.now();
|
||||
@@ -41,8 +43,9 @@ export function registerExecutionControlWork(
|
||||
hasPending: async () => reconciled !== requested,
|
||||
nextRunAt: () => next,
|
||||
});
|
||||
return { ...worker, wake: () => { requested += 1; worker.wake(); } };
|
||||
});
|
||||
return { ready: Promise.all(workers.map(worker => worker.ready)) };
|
||||
return { ready: Promise.all(workers.map(worker => worker.ready)), wake: () => { for (const worker of workers) worker.wake(); } };
|
||||
}
|
||||
|
||||
export function executionControlSchedule(db: Db) {
|
||||
@@ -63,6 +66,10 @@ export function executionControlSchedule(db: Db) {
|
||||
]);
|
||||
return deadline([native[0]?.at, legacy[0]?.at]);
|
||||
},
|
||||
async workspaceRecovery() {
|
||||
const at = await nextWorkspaceRecoveryAt(db);
|
||||
return at === null ? null : deadline([new Date(at)]);
|
||||
},
|
||||
async replacement() {
|
||||
const rows = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
||||
.innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
|
||||
|
||||
@@ -1,10 +1,17 @@
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { issueComments, issueRecoveryActions, type Db } from "@paperclipai/db";
|
||||
import { heartbeatRuns, issueComments, issueRecoveryActions, type Db } from "@paperclipai/db";
|
||||
import { writeExecutionControl } from "./execution-control-notifications.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
import { createSystemRecovery, readSystemRecovery, type SystemRecoveryState } from "./system-recovery-state.js";
|
||||
import type { WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js";
|
||||
import { workspaceRecoverySourceEligible, type WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js";
|
||||
|
||||
export async function adoptSystemRecovery(db: Db, source: WorkspaceRecoverySourceIdentity) {
|
||||
return db.transaction(async tx => {
|
||||
const [existing] = await tx.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions)
|
||||
.where(and(eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, source.actionId),
|
||||
eq(issueRecoveryActions.sourceIssueId, source.issueId))).for("update");
|
||||
if (!existing || existing.evidence.systemRecovery) return;
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
|
||||
const state = createSystemRecovery(source.runId);
|
||||
// Atomic, once per source. An exhausted previous execution budget is history;
|
||||
// adoption grants only the explicitly versioned maintenance capability.
|
||||
@@ -49,17 +56,22 @@ export async function updateSystemRecovery(db: Db, source: WorkspaceRecoverySour
|
||||
stage: next.phase === "verify_execution" ? "verifying_stop" : next.phase === "preserve_files" ? "restoring_files" : "saving_receipts",
|
||||
attempt: next.attempts, attemptLimit: 6, retryable: true, retryAt: next.nextAttemptAt ?? undefined,
|
||||
};
|
||||
const [updated] = await db.update(issueRecoveryActions).set({
|
||||
ownerType: "system", ownerAgentId: null, ownerUserId: null,
|
||||
evidence: sql`${issueRecoveryActions.evidence} || jsonb_build_object('systemRecovery', ${JSON.stringify(next)}::jsonb,
|
||||
'workspaceRecovery', ${JSON.stringify(compatibility)}::jsonb || jsonb_build_object('requestIds',
|
||||
coalesce(${issueRecoveryActions.evidence}#>'{workspaceRecovery,requestIds}', '[]'::jsonb)))`,
|
||||
updatedAt: new Date(),
|
||||
}).where(and(eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, source.actionId),
|
||||
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}' = ${previous.generationId}`,
|
||||
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,revision}' = ${String(previous.revision)}`))
|
||||
.returning({ id: issueRecoveryActions.id });
|
||||
return Boolean(updated);
|
||||
return writeExecutionControl(db, DELIVERY_QUEUES.workspaceRecovery, async tx => {
|
||||
const [updated] = await tx.update(issueRecoveryActions).set({
|
||||
ownerType: "system", ownerAgentId: null, ownerUserId: null,
|
||||
evidence: sql`${issueRecoveryActions.evidence} || jsonb_build_object('systemRecovery', ${JSON.stringify(next)}::jsonb ||
|
||||
${next.generationId === previous.generationId && next.intentId === previous.intentId
|
||||
? sql`jsonb_build_object('intentId', ${issueRecoveryActions.evidence}#>'{systemRecovery,intentId}')`
|
||||
: sql`'{}'::jsonb`},
|
||||
'workspaceRecovery', ${JSON.stringify(compatibility)}::jsonb || jsonb_build_object('requestIds',
|
||||
coalesce(${issueRecoveryActions.evidence}#>'{workspaceRecovery,requestIds}', '[]'::jsonb)))`,
|
||||
updatedAt: new Date(),
|
||||
}).where(and(eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, source.actionId),
|
||||
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}' = ${previous.generationId}`,
|
||||
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,revision}' = ${String(previous.revision)}`))
|
||||
.returning({ id: issueRecoveryActions.id });
|
||||
return Boolean(updated);
|
||||
});
|
||||
}
|
||||
|
||||
export async function loadSystemRecovery(db: Db, source: WorkspaceRecoverySourceIdentity) {
|
||||
@@ -68,3 +80,19 @@ export async function loadSystemRecovery(db: Db, source: WorkspaceRecoverySource
|
||||
eq(issueRecoveryActions.id, source.actionId)));
|
||||
return readSystemRecovery(row?.evidence);
|
||||
}
|
||||
|
||||
export async function nextWorkspaceRecoveryAt(db: Db): Promise<number | null> {
|
||||
const rows = await db.select({ at: sql<number>`case
|
||||
when jsonb_typeof(${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}') = 'number'
|
||||
then (${issueRecoveryActions.evidence}#>>'{systemRecovery,nextAttemptAt}')::bigint else 0 end` })
|
||||
.from(issueRecoveryActions).innerJoin(heartbeatRuns, and(
|
||||
eq(heartbeatRuns.companyId, issueRecoveryActions.companyId),
|
||||
eq(heartbeatRuns.id, sql`case when ${issueRecoveryActions.evidence}->>'runId' ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
|
||||
then (${issueRecoveryActions.evidence}->>'runId')::uuid end`),
|
||||
sql`${heartbeatRuns.status} in ('failed', 'interrupted', 'timed_out', 'cancelled', 'succeeded')`,
|
||||
)).where(and(
|
||||
workspaceRecoverySourceEligible(),
|
||||
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')`,
|
||||
)).orderBy(sql`1`).limit(1);
|
||||
return rows.length ? Number(rows[0].at) : null;
|
||||
}
|
||||
@@ -1,3 +1,5 @@
|
||||
import { writeExecutionControl } from "./execution-control-notifications.js";
|
||||
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
|
||||
import { z } from "zod";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, desc, eq, inArray, or, sql } from "drizzle-orm";
|
||||
@@ -73,7 +75,7 @@ export async function requestTaskRetry(input: {
|
||||
prepareChatRetry?: (tx: Db, scope: { companyId: string; issueId: string; agentId: string;
|
||||
failedRunId: string; initiatedByUserId: string; retryIntentId?: string }) => Promise<{ actionId: string }>;
|
||||
}) {
|
||||
return input.db.transaction(async tx => {
|
||||
return writeExecutionControl(input.db, DELIVERY_QUEUES.workspaceRecovery, async tx => {
|
||||
const db = tx as unknown as Db;
|
||||
const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, input.companyId),
|
||||
eq(issues.id, input.issueId))).for("update");
|
||||
@@ -154,6 +156,11 @@ export async function requestTaskRetry(input: {
|
||||
for (const action of records) {
|
||||
const previous = readSystemRecovery(action.evidence);
|
||||
if (!previous || previous.state === "complete") continue;
|
||||
if (previous.state === "running") {
|
||||
await updateSystemRecovery(db, { companyId: issue.companyId, issueId: issue.id, runId: latest.id,
|
||||
actionId: action.id }, previous, { ...previous, intentId: active?.id ?? id });
|
||||
continue;
|
||||
}
|
||||
// Active repair keeps its attempt and cooldown. An exhausted generation
|
||||
// gets a new budget only because this click is a new durable intent.
|
||||
const next = previous.state === "exhausted"
|
||||
@@ -163,8 +170,7 @@ export async function requestTaskRetry(input: {
|
||||
: { ...previous, revision: previous.revision + 1, intentId: active?.id ?? id,
|
||||
// Retry may advance our backoff, never a provider's cooldown and
|
||||
// never an operation whose physical owner is still running.
|
||||
nextAttemptAt: previous.state === "running" ? previous.nextAttemptAt
|
||||
: Math.max(Date.now(), previous.providerCooldownAt ?? 0) };
|
||||
nextAttemptAt: Math.max(Date.now(), previous.providerCooldownAt ?? 0) };
|
||||
if (next.generationId !== previous.generationId) await persistActivity(db, {
|
||||
companyId: issue.companyId, actorType: "system", actorId: "task-retry", entityType: "issue", entityId: issue.id,
|
||||
action: "issue.recovery_generation_replaced", runId: latest.id,
|
||||
|
||||
@@ -10,6 +10,29 @@ export const workspaceExportCauses = ["native_workspace_sync_out_unrecoverable",
|
||||
export const terminalWorkspaceRunStates = ["failed", "interrupted", "timed_out", "cancelled", "succeeded"];
|
||||
export type WorkspaceRecoverySourceIdentity = { companyId: string; issueId: string; runId: string; actionId: string };
|
||||
|
||||
export function workspaceRecoverySourceEligible(includeCompleted = false) {
|
||||
const actionEligible = sql`(${issueRecoveryActions.status} in ('active', 'escalated') or
|
||||
(${issueRecoveryActions.status} = 'resolved' and (
|
||||
(${issueRecoveryActions.evidence}#>>'{automaticRecovery,policy}' = 'preserve_without_replay_v1'
|
||||
and ${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'blocked'
|
||||
and ${issueRecoveryActions.evidence}#>>'{automaticRecovery,runId}' = ${heartbeatRuns.id}::text) or
|
||||
(${issueRecoveryActions.evidence}#>>'{historicalWorkspaceSource,schema}' = 'paperclip.historical-workspace-source.v1'
|
||||
and ${issueRecoveryActions.evidence}#>>'{historicalWorkspaceSource,runId}' = ${heartbeatRuns.id}::text))))`;
|
||||
return and(
|
||||
sql`${issueRecoveryActions.evidence} ? 'systemRecovery'`,
|
||||
includeCompleted ? sql`(${actionEligible} or ${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete')` : actionEligible,
|
||||
sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.issueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') = ${issueRecoveryActions.sourceIssueId}::text`,
|
||||
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,sourceRunId}' = ${heartbeatRuns.id}::text`,
|
||||
sql`${issueRecoveryActions.evidence}#>>'{workspaceRecovery,runId}' = ${heartbeatRuns.id}::text`,
|
||||
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,policyVersion}' = '1'`,
|
||||
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,schema}' = 'paperclip.system-recovery.v1'`,
|
||||
sql`${issueRecoveryActions.evidence}#>>'{workspaceRecovery,schema}' = 'paperclip.workspace-recovery.v1'`,
|
||||
sql`(${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'
|
||||
or ${issueRecoveryActions.evidence}->'historicalWorkspaceSource'->>'schema' = 'paperclip.historical-workspace-source.v1'
|
||||
or ${issueRecoveryActions.cause} in ('native_workspace_sync_out_unrecoverable', 'native_workspace_sync_out_retry_exhausted'))`,
|
||||
);
|
||||
}
|
||||
|
||||
/** The preservation owner may reopen only its exact stop-only allocation.
|
||||
* This does not release the source's ordinary execution or destruction fence. */
|
||||
export async function ownedWorkspaceRecoveryLease(db: Db, input: WorkspaceRecoverySourceIdentity & {
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
|
||||
import { executionControlSchedule, registerExecutionControlWork } from "./execution-control-work.js";
|
||||
import type { EnvironmentRuntimeService } from "./environment-runtime.js";
|
||||
import { reconcileRetainedWorkspaces } from "./workspace-recovery.js";
|
||||
|
||||
export function registerWorkspaceRecoveryWork(db: Db, coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
|
||||
runtime: EnvironmentRuntimeService, onRecovered: Parameters<typeof reconcileRetainedWorkspaces>[2],
|
||||
canRun: () => boolean = () => true) {
|
||||
const pending = new Set<Promise<void>>();
|
||||
let stopped = false;
|
||||
let scan: Promise<void> | undefined;
|
||||
let wakeWhileBusy = false;
|
||||
let passiveSweep: { cursor?: string } = {};
|
||||
const schedule = executionControlSchedule(db);
|
||||
const worker = registerExecutionControlWork(coordinator, [{
|
||||
queue: DELIVERY_QUEUES.workspaceRecovery,
|
||||
onSettled: () => { passiveSweep = {}; if (pending.size >= 2) wakeWhileBusy = true; },
|
||||
run: () => scan = (async () => {
|
||||
const work = await reconcileRetainedWorkspaces(db, runtime, onRecovered, true, continueSweep => {
|
||||
if (!stopped && (continueSweep || wakeWhileBusy)) { wakeWhileBusy = false; worker.wake(); }
|
||||
}, passiveSweep);
|
||||
for (const job of work ?? []) {
|
||||
pending.add(job);
|
||||
const settled = () => { pending.delete(job); };
|
||||
void job.then(settled, settled);
|
||||
}
|
||||
})(),
|
||||
next: () => pending.size >= 2 ? Promise.resolve(null) : schedule.workspaceRecovery(),
|
||||
}], () => !stopped && canRun());
|
||||
return { ready: worker.ready, async stop() {
|
||||
stopped = true;
|
||||
await scan?.catch(() => {});
|
||||
await Promise.all(pending);
|
||||
} };
|
||||
}
|
||||
@@ -20,21 +20,29 @@ import { generateKeyPairSync, randomUUID, sign } from "node:crypto";
|
||||
import { cp, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { and, eq, inArray, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, approvals, budgetPolicies, costEvents, completionContracts, issueApprovals, issueTreeHolds, issueTreeHoldMembers, companies, createDb, environmentLeases, environments, heartbeatRuns, issueComments, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, agentWakeupRequests, workspaceOperations } from "@paperclipai/db";
|
||||
import { agents, approvals, budgetPolicies, costEvents, completionContracts, issueApprovals, issueTreeHolds, issueTreeHoldMembers, companies, createDb, applyPendingMigrations, environmentLeases, environments, heartbeatRuns, issueComments, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, agentWakeupRequests, workspaceOperations } from "@paperclipai/db";
|
||||
import { resumePreservedResultFinalization } from "./preserved-result-finalization.js";
|
||||
import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
||||
import type { EnvironmentLease } from "@paperclipai/shared";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
|
||||
import { prepareNativeWorkspaceSync } from "./native-runtime/native-workspace-sync.js";
|
||||
import { nativeWorkspaceFinalizationLockKey } from "./native-runtime/native-workspace-finalization-ownership.js";
|
||||
import { remoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
import { prepareStoppedSandboxForRecovery } from "./sandbox-stop-and-retain.js";
|
||||
import { recoverRetainedWorkspace, requestWorkspaceRecovery, scheduleRecoveredWorkspaceCleanup, reconcileRetainedWorkspaces, retainedWorkspaceCandidates } from "./workspace-recovery.js";
|
||||
import * as activity from "./activity-log.js";
|
||||
import { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
||||
import { registerWorkspaceRecoveryWork } from "./workspace-recovery-work.js";
|
||||
import { heartbeatService } from "./heartbeat.js";
|
||||
import { hasVerifiedWorkspaceRepair, readWorkspaceRecovery } from "./workspace-recovery-receipt.js";
|
||||
import { getExecutionBlocker } from "./execution-blocker.js";
|
||||
import { issueRoutes } from "../routes/issues.js";
|
||||
import { errorHandler } from "../middleware/index.js";
|
||||
import * as workerProof from "./native-runtime/native-workspace-platform-proof.js";
|
||||
const targets = vi.hoisted(() => new Map<string, AdapterSandboxExecutionTarget>());
|
||||
vi.mock("./environment-execution-target.js", async importOriginal => {
|
||||
@@ -51,7 +59,13 @@ describe("automatic retained workspace recovery", () => {
|
||||
beforeAll(async () => {
|
||||
root = await mkdtemp(path.join(os.tmpdir(), "workspace-recovery-"));
|
||||
process.env.PAPERCLIP_HOME = path.join(root, "home"); process.env.PAPERCLIP_INSTANCE_ID = "recovery";
|
||||
temporary = await startEmbeddedPostgresTestDatabase("workspace-recovery-"); db = createDb(temporary.connectionString);
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_RECOVERY_TEST_DATABASE_URL");
|
||||
if (external) {
|
||||
await applyPendingMigrations(external);
|
||||
temporary = { connectionString: external, cleanup: async () => {} };
|
||||
} else temporary = await startEmbeddedPostgresTestDatabase("workspace-recovery-");
|
||||
db = createDb(temporary.connectionString);
|
||||
if (external) await db.execute(sql`TRUNCATE TABLE companies CASCADE`);
|
||||
}, 30_000);
|
||||
afterAll(async () => {
|
||||
await db?.$client.end(); await temporary?.cleanup(); await rm(root, { recursive: true, force: true });
|
||||
@@ -123,6 +137,178 @@ describe("automatic retained workspace recovery", () => {
|
||||
const action = async (id: string) => (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)))[0]!;
|
||||
const run = async (id: string) => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0]!;
|
||||
|
||||
it("wakes physical preservation from HTTP Retry before passive history without replaying the provider", async () => {
|
||||
for (let index = 0; index < 5; index++) {
|
||||
const old = await seed();
|
||||
await recoverRetainedWorkspace(old.input);
|
||||
await db.update(issueRecoveryActions).set({
|
||||
id: `00000000-000${index}-4000-8000-${randomUUID().slice(-12)}`,
|
||||
}).where(eq(issueRecoveryActions.id, old.actionId));
|
||||
}
|
||||
const f = await seed();
|
||||
await adoptSystemRecovery(db, f);
|
||||
const before = await action(f.actionId);
|
||||
const exhausted = failSystemRecovery(readSystemRecovery(before.evidence)!,
|
||||
new SystemRecoveryFailure("source_binding_invalid", "service_required"));
|
||||
await db.update(issueRecoveryActions).set({ cause: "native_workspace_sync_out_retry_exhausted",
|
||||
evidence: { ...before.evidence, systemRecovery: exhausted } })
|
||||
.where(eq(issueRecoveryActions.id, f.actionId));
|
||||
const errors = vi.fn();
|
||||
const competingCleanup = await retainedWorkspaceCandidates(db, undefined, false, true);
|
||||
expect(competingCleanup).toHaveLength(5);
|
||||
expect(competingCleanup.every(row => row.action.id.startsWith("00000000"))).toBe(true);
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: errors });
|
||||
const recovered = vi.fn(async () => {});
|
||||
const worker = registerWorkspaceRecoveryWork(db, coordinator, f.environmentRuntime, recovered);
|
||||
try {
|
||||
await worker.ready;
|
||||
await db.update(agents).set({ status: "paused" }).where(eq(agents.companyId, f.companyId));
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => { req.actor = { type: "board", source: "local_implicit", userId: "board" }; next(); });
|
||||
app.use("/api", issueRoutes(db, {} as Parameters<typeof issueRoutes>[1]));
|
||||
app.use(errorHandler);
|
||||
const body = { idempotencyKey: randomUUID(), observedRunId: f.runId };
|
||||
const retry = await request(app).post(`/api/issues/${f.issueId}/retry`).send(body).expect(202);
|
||||
const repeated = await request(app).post(`/api/issues/${f.issueId}/retry`).send(body).expect(202);
|
||||
const coalesced = await request(app).post(`/api/issues/${f.issueId}/retry`)
|
||||
.send({ ...body, idempotencyKey: randomUUID() }).expect(202);
|
||||
expect(repeated.body.requestId).toBe(retry.body.requestId);
|
||||
expect(coalesced.body.requestId).toBe(retry.body.requestId);
|
||||
await expect.poll(async () => readSystemRecovery((await action(f.actionId)).evidence)?.state,
|
||||
{ timeout: 20_000 }).toBe("complete");
|
||||
expect(await readFile(path.join(f.host, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
|
||||
expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.companyId, f.companyId))).toHaveLength(0);
|
||||
const reloaded = await request(app).get(`/api/issues/${f.issueId}/recovery-actions`).expect(200);
|
||||
expect(JSON.stringify(reloaded.body)).toContain('"state":"complete"');
|
||||
expect(errors).not.toHaveBeenCalled();
|
||||
} finally { await coordinator.stop(); await worker.stop(); }
|
||||
}, 30_000);
|
||||
|
||||
it("reconstructs a resolved files-only preservation obligation at startup without replay", async () => {
|
||||
const f = await seed("legacy");
|
||||
await adoptSystemRecovery(db, f);
|
||||
const before = await action(f.actionId);
|
||||
const automaticRecovery = { policy: "preserve_without_replay_v1", runId: f.runId, replay: "blocked" };
|
||||
await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "blocked",
|
||||
evidence: { ...before.evidence, automaticRecovery } }).where(eq(issueRecoveryActions.id, f.actionId));
|
||||
const errors = vi.fn();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: errors });
|
||||
const worker = registerWorkspaceRecoveryWork(db, coordinator, f.environmentRuntime, async () => {});
|
||||
try {
|
||||
await worker.ready;
|
||||
await expect.poll(async () => readSystemRecovery((await action(f.actionId)).evidence)?.state,
|
||||
{ timeout: 20_000 }).toBe("complete");
|
||||
expect((await action(f.actionId)).evidence.automaticRecovery).toEqual(automaticRecovery);
|
||||
expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0);
|
||||
expect(errors).not.toHaveBeenCalled();
|
||||
} finally { await coordinator.stop(); await worker.stop(); }
|
||||
}, 30_000);
|
||||
|
||||
it("fills a free export slot while another export is held and joins both at shutdown", async () => {
|
||||
const held = await seed(), fast = await seed();
|
||||
const fixtures = [held, fast];
|
||||
const runtimeFor = (runId: string) => fixtures.find(fixture => fixture.runId === runId)!.environmentRuntime;
|
||||
const runtime = { ...held.environmentRuntime,
|
||||
resumeWorkspaceRecoveryLease: (input: Parameters<typeof held.environmentRuntime.resumeWorkspaceRecoveryLease>[0]) =>
|
||||
runtimeFor(input.source.runId).resumeWorkspaceRecoveryLease(input),
|
||||
retryPendingSandboxTeardown: (input: Parameters<typeof held.environmentRuntime.retryPendingSandboxTeardown>[0]) =>
|
||||
runtimeFor(input.lease.heartbeatRunId!).retryPendingSandboxTeardown(input),
|
||||
};
|
||||
let active = 0, maximum = 0, release!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
const countExport = (fixture: typeof held, pause = false) => {
|
||||
const resume = fixture.runtime.resumeRunLease.getMockImplementation()!;
|
||||
fixture.runtime.resumeRunLease.mockImplementation(async input => {
|
||||
active += 1; maximum = Math.max(maximum, active);
|
||||
try { if (pause) await gate; return await resume(input); } finally { active -= 1; }
|
||||
});
|
||||
};
|
||||
countExport(held, true); countExport(fast);
|
||||
const errors = vi.fn();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: errors });
|
||||
const worker = registerWorkspaceRecoveryWork(db, coordinator, runtime, async () => {});
|
||||
try {
|
||||
await expect.poll(() => held.runtime.resumeRunLease.mock.calls.length, { timeout: 20_000 }).toBe(1);
|
||||
await expect.poll(async () => readSystemRecovery((await action(fast.actionId)).evidence)?.state,
|
||||
{ timeout: 20_000 }).toBe("complete");
|
||||
const third = await seed(); fixtures.push(third); countExport(third);
|
||||
await requestTaskRetry({ db, companyId: third.companyId, issueId: third.issueId, actorId: "board",
|
||||
observedRunId: third.runId, idempotencyKey: randomUUID() });
|
||||
await expect.poll(async () => readSystemRecovery((await action(third.actionId)).evidence)?.state,
|
||||
{ timeout: 20_000 }).toBe("complete");
|
||||
expect(readSystemRecovery((await action(held.actionId)).evidence)?.state).toBe("running");
|
||||
expect(maximum).toBeLessThanOrEqual(2);
|
||||
expect(await readFile(path.join(third.host, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, third.companyId))).toHaveLength(1);
|
||||
let stopped = false;
|
||||
const stopping = Promise.all([coordinator.stop(), worker.stop()]).then(() => { stopped = true; });
|
||||
await Promise.resolve(); expect(stopped).toBe(false);
|
||||
release(); await stopping;
|
||||
expect(readSystemRecovery((await action(held.actionId)).evidence)?.state).toBe("complete");
|
||||
expect(errors).not.toHaveBeenCalled();
|
||||
} finally { release(); await coordinator.stop(); await worker.stop(); }
|
||||
}, 60_000);
|
||||
|
||||
it("rotates past due sources whose physical locks belong to other workers", async () => {
|
||||
const blocked = [await seed(), await seed()], available = await seed();
|
||||
for (const [index, fixture] of [...blocked, available].entries()) {
|
||||
await adoptSystemRecovery(db, fixture);
|
||||
const id = `00000001-000${index}-4000-8000-${randomUUID().slice(-12)}` as const;
|
||||
await db.update(issueRecoveryActions).set({ id }).where(eq(issueRecoveryActions.id, fixture.actionId));
|
||||
fixture.actionId = id; fixture.input.actionId = id;
|
||||
}
|
||||
let unlock!: () => void;
|
||||
const gate = new Promise<void>(resolve => { unlock = resolve; });
|
||||
const held = blocked.map(fixture => {
|
||||
let ready!: () => void;
|
||||
const acquired = new Promise<void>(resolve => { ready = resolve; });
|
||||
const lock = db.transaction(async tx => {
|
||||
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${nativeWorkspaceFinalizationLockKey(fixture.companyId, fixture.runId)}, 0))`);
|
||||
ready(); await gate;
|
||||
});
|
||||
return { acquired, lock };
|
||||
});
|
||||
try {
|
||||
await Promise.all(held.map(item => item.acquired));
|
||||
await Promise.all(await reconcileRetainedWorkspaces(db, available.environmentRuntime, async () => {}, true) ?? []);
|
||||
expect(blocked.every(fixture => fixture.runtime.resumeRunLease.mock.calls.length === 0)).toBe(true);
|
||||
await Promise.all(await reconcileRetainedWorkspaces(db, available.environmentRuntime, async () => {}, true) ?? []);
|
||||
expect(readSystemRecovery((await action(available.actionId)).evidence)?.state).toBe("complete");
|
||||
expect(await readFile(path.join(available.host, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(blocked.every(fixture => fixture.runtime.resumeRunLease.mock.calls.length === 0)).toBe(true);
|
||||
} finally { unlock(); await Promise.all(held.map(item => item.lock)); }
|
||||
}, 30_000);
|
||||
|
||||
it("coalesces Retry while preservation owns its physical operation", async () => {
|
||||
const f = await seed();
|
||||
await adoptSystemRecovery(db, f);
|
||||
const resume = f.runtime.resumeRunLease.getMockImplementation()!;
|
||||
let release!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
f.runtime.resumeRunLease.mockImplementation(async input => { await gate; return resume(input); });
|
||||
const preserving = recoverRetainedWorkspace(f.input);
|
||||
let requestId: string | null | undefined;
|
||||
try {
|
||||
await expect.poll(() => f.runtime.resumeRunLease.mock.calls.length).toBe(1);
|
||||
const request = { db, companyId: f.companyId, issueId: f.issueId, actorId: "board",
|
||||
idempotencyKey: randomUUID(), observedRunId: f.runId };
|
||||
const first = await requestTaskRetry(request);
|
||||
requestId = first.requestId;
|
||||
const second = await requestTaskRetry({ ...request, idempotencyKey: randomUUID() });
|
||||
expect(second.requestId).toBe(first.requestId);
|
||||
} finally { release(); await preserving; }
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)?.intentId).toBe(requestId);
|
||||
expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1);
|
||||
expect(await readFile(path.join(f.host, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
|
||||
}, 30_000);
|
||||
|
||||
it("adopts only the exact historical infrastructure notice, preserving prose and user-action notices", async () => {
|
||||
const f = await seed("legacy", false);
|
||||
const metadata = { version: 1 as const, sourceRunId: f.runId,
|
||||
@@ -285,7 +471,7 @@ describe("automatic retained workspace recovery", () => {
|
||||
expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.companyId, f.companyId))).toHaveLength(0);
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0);
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce();
|
||||
expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1);
|
||||
expect(await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.companyId, f.companyId), eq(workspaceOperations.phase, "workspace_recovery")))).toHaveLength(1);
|
||||
}, 30_000);
|
||||
|
||||
@@ -304,9 +490,9 @@ describe("automatic retained workspace recovery", () => {
|
||||
expect(readWorkspaceRecovery((await action(f.actionId)).evidence)?.status).toBe("recovered");
|
||||
expect(await readFile(path.join(f.host, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(await readFile(path.join(f.host, "newer.txt"), "utf8")).toBe("newer local work\n");
|
||||
expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce();
|
||||
expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1);
|
||||
await recoverRetainedWorkspace({ ...f.input, environmentRuntime: runtime });
|
||||
expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce();
|
||||
expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0);
|
||||
}, 30_000);
|
||||
@@ -405,7 +591,7 @@ describe("automatic retained workspace recovery", () => {
|
||||
f.runtime.resumeRunLease.mockImplementationOnce(async () => { ready(); await gate; return { providerLeaseId: f.providerLeaseId, metadata: { remoteCwd: f.remote } }; });
|
||||
const first = recoverRetainedWorkspace(f.input);
|
||||
await started;
|
||||
try { await recoverRetainedWorkspace(f.input); expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce(); }
|
||||
try { await recoverRetainedWorkspace(f.input); expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1); }
|
||||
finally { release(); await first; }
|
||||
expect(readWorkspaceRecovery((await action(f.actionId)).evidence)?.status).toBe("recovered");
|
||||
}, 30_000);
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
import { beginIdleTrackedWork } from "./task-admission.js";
|
||||
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { resumePreservedResultFinalization } from "./preserved-result-finalization.js";
|
||||
import { and, asc, eq, gt, inArray, ne, or, sql } from "drizzle-orm";
|
||||
@@ -19,7 +21,7 @@ import { classifyNativeWorkspaceFailure } from "./native-runtime/native-workspac
|
||||
|
||||
import { adoptSystemRecovery, loadSystemRecovery, updateSystemRecovery } from "./system-recovery.js";
|
||||
import { createSystemRecovery, failSystemRecovery, readSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure, type SystemRecoveryState } from "./system-recovery-state.js";
|
||||
import { inspectWorkspaceRecoverySource } from "./workspace-recovery-source.js";
|
||||
import { inspectWorkspaceRecoverySource, workspaceRecoverySourceEligible } from "./workspace-recovery-source.js";
|
||||
import { preserveWorkspaceSnapshot } from "./workspace-recovery-snapshot.js";
|
||||
import { adoptHistoricalWorkspaceRecoveries } from "./workspace-recovery-backfill.js";
|
||||
|
||||
@@ -48,12 +50,13 @@ export async function requestWorkspaceRecovery(input: Source & { db: Db; idempot
|
||||
requestIds: [...(prior?.requestIds ?? []), input.idempotencyKey] };
|
||||
const previous = readSystemRecovery(action.evidence);
|
||||
const active = previous && ["scheduled", "running", "awaiting_evidence"].includes(previous.state);
|
||||
const system = active ? { ...previous, revision: previous.revision + 1,
|
||||
nextAttemptAt: previous.state === "running" ? previous.nextAttemptAt : Math.max(Date.now(), previous.providerCooldownAt ?? 0) } : { ...createSystemRecovery(run.id),
|
||||
const system = previous?.state === "running" ? previous : active ? { ...previous, revision: previous.revision + 1,
|
||||
nextAttemptAt: Math.max(Date.now(), previous.providerCooldownAt ?? 0) } : { ...createSystemRecovery(run.id),
|
||||
revision: (previous?.revision ?? -1) + 1,
|
||||
preservationOperationIds: previous?.preservationOperationIds ?? [],
|
||||
providerCooldownAt: previous?.providerCooldownAt ?? null,
|
||||
nextAttemptAt: previous?.providerCooldownAt ?? null };
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
|
||||
await tx.update(issueRecoveryActions).set({ ownerType: "system", ownerAgentId: null, ownerUserId: null,
|
||||
evidence: { ...action.evidence, workspaceRecovery: state, systemRecovery: system }, updatedAt: new Date() })
|
||||
.where(eq(issueRecoveryActions.id, action.id));
|
||||
@@ -286,6 +289,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
|
||||
|
||||
const cursors = new WeakMap<Db, string>();
|
||||
const workInFlight = new WeakMap<Db, Set<string>>();
|
||||
const unfinishedCursors = new WeakMap<Db, string>();
|
||||
|
||||
/** Destruction is a later cleanup obligation, authorized only by a matching
|
||||
* saved-file receipt and the original recorded disposition. */
|
||||
@@ -309,29 +313,31 @@ export async function scheduleRecoveredWorkspaceCleanup(db: Db, run: typeof hear
|
||||
sql`${environmentLeases.metadata} = ${JSON.stringify(lease.metadata)}::jsonb`,
|
||||
sql`${environmentLeases.metadata}->>'workspaceRecoveryCleanup' is null`)));
|
||||
}
|
||||
export function retainedWorkspaceCandidates(db: Db, cursor?: string) {
|
||||
export function retainedWorkspaceCandidates(db: Db, cursor?: string, unfinishedOnly = false, obligationsOnly = false) {
|
||||
const cleanup = sql`(${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,resourceDisposition}' = 'destroy' and exists (
|
||||
select 1 from ${environmentLeases} where ${environmentLeases.companyId} = ${heartbeatRuns.companyId}
|
||||
and ${environmentLeases.heartbeatRunId} = ${heartbeatRuns.id}
|
||||
and ${environmentLeases.id}::text = ${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,leaseId}'
|
||||
and ${environmentLeases.status} = 'released' and ${environmentLeases.cleanupStatus} = 'success'
|
||||
and ${environmentLeases.metadata}->'workspaceRecoveryCleanup' is null))`;
|
||||
const waitingInput = sql`exists (select 1 from ${agentWakeupRequests}
|
||||
where ${agentWakeupRequests.companyId} = ${issueRecoveryActions.companyId}
|
||||
and ${agentWakeupRequests.status} = 'deferred_issue_execution' and ${agentWakeupRequests.requestedByActorType} = 'user'
|
||||
and ${agentWakeupRequests.payload}->>'issueId' = ${issueRecoveryActions.sourceIssueId}::text)`;
|
||||
return db.select({ action: issueRecoveryActions, run: heartbeatRuns }).from(issueRecoveryActions)
|
||||
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, issueRecoveryActions.companyId), eq(heartbeatRuns.id,
|
||||
sql`case when ${issueRecoveryActions.evidence}->>'runId' ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
|
||||
then (${issueRecoveryActions.evidence}->>'runId')::uuid end`)))
|
||||
.where(and(inArray(heartbeatRuns.status, terminal),
|
||||
sql`(${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'
|
||||
or ${issueRecoveryActions.evidence}->'historicalWorkspaceSource'->>'schema' = 'paperclip.historical-workspace-source.v1'
|
||||
or ${issueRecoveryActions.cause} in ('native_workspace_sync_out_unrecoverable', 'native_workspace_sync_out_retry_exhausted'))`,
|
||||
sql`${issueRecoveryActions.evidence} ? 'systemRecovery'`,
|
||||
workspaceRecoverySourceEligible(true),
|
||||
unfinishedOnly ? sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')` : undefined,
|
||||
obligationsOnly ? sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')
|
||||
or (${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete' and ${cleanup}) or ${waitingInput})` : undefined,
|
||||
sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence', 'complete', 'exhausted')
|
||||
and (${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' = 'null'::jsonb
|
||||
or ${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' <= to_jsonb(${Date.now()}::bigint)))`,
|
||||
sql`(${issueRecoveryActions.evidence}#>>'{workspaceRecovery,status}' is distinct from 'recovered' or
|
||||
(${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,resourceDisposition}' = 'destroy' and exists (
|
||||
select 1 from ${environmentLeases} where ${environmentLeases.companyId} = ${heartbeatRuns.companyId}
|
||||
and ${environmentLeases.heartbeatRunId} = ${heartbeatRuns.id}
|
||||
and ${environmentLeases.id}::text = ${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,leaseId}'
|
||||
and ${environmentLeases.status} = 'released' and ${environmentLeases.cleanupStatus} = 'success'
|
||||
and ${environmentLeases.metadata}->'workspaceRecoveryCleanup' is null)) or exists (
|
||||
select 1 from ${agentWakeupRequests} where ${agentWakeupRequests.companyId} = ${issueRecoveryActions.companyId}
|
||||
and ${agentWakeupRequests.status} = 'deferred_issue_execution' and ${agentWakeupRequests.requestedByActorType} = 'user'
|
||||
and ${agentWakeupRequests.payload}->>'issueId' = ${issueRecoveryActions.sourceIssueId}::text))`,
|
||||
${cleanup} or ${waitingInput})`,
|
||||
cursor ? gt(issueRecoveryActions.id, cursor) : undefined,
|
||||
)).orderBy(asc(issueRecoveryActions.id)).limit(5);
|
||||
}
|
||||
@@ -339,19 +345,29 @@ export function retainedWorkspaceCandidates(db: Db, cursor?: string) {
|
||||
/** Bounded discovery shares the periodic recovery lifecycle. A repaired source
|
||||
* still owes cleanup even without saved input or after task reconciliation. */
|
||||
export async function reconcileRetainedWorkspaces(db: Db, runtime: EnvironmentRuntimeService,
|
||||
onRecovered: (run: typeof heartbeatRuns.$inferSelect) => Promise<void>) {
|
||||
onRecovered: (run: typeof heartbeatRuns.$inferSelect) => Promise<void>, obligationsOnly = false,
|
||||
onFinished?: (continueSweep: boolean) => void, passiveSweep?: { cursor?: string }) {
|
||||
await adoptHistoricalWorkspaceRecoveries(db);
|
||||
let pending = workInFlight.get(db);
|
||||
if (!pending) { pending = new Set(); workInFlight.set(db, pending); }
|
||||
if (pending.size >= 2) return;
|
||||
let rows = await retainedWorkspaceCandidates(db, cursors.get(db));
|
||||
if (!rows.length) rows = await retainedWorkspaceCandidates(db);
|
||||
let rows = await retainedWorkspaceCandidates(db, unfinishedCursors.get(db), true);
|
||||
if (!rows.length) rows = await retainedWorkspaceCandidates(db, undefined, true);
|
||||
const unfinished = rows.length > 0;
|
||||
if (!rows.length) rows = await retainedWorkspaceCandidates(db, passiveSweep ? passiveSweep.cursor : cursors.get(db), false, obligationsOnly);
|
||||
if (!rows.length && !passiveSweep) rows = await retainedWorkspaceCandidates(db, undefined, false, obligationsOnly);
|
||||
const work: Promise<void>[] = [];
|
||||
let hasUnvisited = false;
|
||||
for (const { action, run } of rows) {
|
||||
if (pending.size >= 2) break;
|
||||
cursors.set(db, action.id);
|
||||
if (pending.size >= 2) { hasUnvisited = true; break; }
|
||||
if (unfinished) unfinishedCursors.set(db, action.id);
|
||||
else if (passiveSweep) passiveSweep.cursor = action.id;
|
||||
else cursors.set(db, action.id);
|
||||
if (pending.has(run.id)) continue;
|
||||
pending.add(run.id);
|
||||
void (async () => { try {
|
||||
const finish = beginIdleTrackedWork();
|
||||
const physical = ["scheduled", "running", "awaiting_evidence"].includes(readSystemRecovery(action.evidence)?.state ?? "");
|
||||
work.push((async () => { try {
|
||||
await recoverRetainedWorkspace({ db, environmentRuntime: runtime, companyId: action.companyId, issueId: action.sourceIssueId, runId: run.id, actionId: action.id });
|
||||
const [updated] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id));
|
||||
const recovered = readWorkspaceRecovery(updated?.evidence ?? null)?.status === "recovered";
|
||||
@@ -370,6 +386,7 @@ export async function reconcileRetainedWorkspaces(db: Db, runtime: EnvironmentRu
|
||||
}
|
||||
}
|
||||
} catch { logger.warn({ runId: run.id }, "Retained workspace remains pending"); }
|
||||
finally { pending.delete(run.id); } })();
|
||||
finally { pending.delete(run.id); finish(); onFinished?.(physical || hasUnvisited); } })());
|
||||
}
|
||||
return work;
|
||||
}
|
||||
Reference in new issue
Block a user