mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 04:21:24 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Recovery tells a waiting parent or dependent when a child or blocker stalls on the board (`issue_blocker_stalled`) > - A legacy child that an older release bound to its parent's remote workspace failed its save, and its save recovery was exhausted. Its ended run still holds the shared workspace, days past the one-hour holder cutoff > - Workspace reservation refuses a run on such a hold and fails setup with "The workspace still has an unfinished writer or save operation." Every stall wake of the parent created a run that could only fail this way > - The stall sweep treated each failed run as an undelivered wake and retried it every ten minutes, with no backoff and no limit. On staging this made 72 failed runs (58 of 60 failures since a deploy), in pairs every ten minutes > - This pull request makes the stall sweep hold a wake while the waiter's workspace hold refuses a run, and bounds retries of a wake whose run fails > - The benefit is that a held workspace no longer produces a failed run every ten minutes, and the wake still arrives once the hold settles ## Linked Issues or Issue Description **What happened?** A parent task was woken by `issue_blocker_stalled` twice every ten minutes. Each run failed in setup (`setup_failed`, `executionRecovery.kind=bootstrap`, `providerWorkStarted=false`) with "The workspace still has an unfinished writer or save operation." The parent's remote workspace was held by a child's ended writer run whose save recovery was exhausted (`native_workspace_sync_out_retry_exhausted`, last failure `agent_stop_unconfirmed`). The child shared the parent's workspace because an older release created it before subtasks got their own workspace (#19860). **Expected behavior** A wake that cannot start because its workspace is held must not create a failed run every time. It should wait until the hold settles. Automatic retries must be bounded. **Root cause** - `server/src/services/heartbeat/workspace-preparation.ts` rethrows `RemoteWorkspaceWriterActiveError` as a setup failure when the ended writer is silent past `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` (documented, intended). - `server/src/services/recovery/blocker-stall-wakeups.ts` retried every non-delivered wake row (including `failed`) after `REFUSED_WAKE_RETRY_MS` (10 minutes) with no backoff and no limit, and it did not check whether the waiter could start at all. ## What Changed - `remote-workspace-writer-hold.ts`: add `remoteWorkspaceHoldRefusesStart`, the one rule for a hold that reservation refuses outright (unattributed writer, or a writer silent past the cutoff with no live owner fence). - `workspace-preparation.ts`: use that rule instead of an inline copy, so reservation and wake senders cannot drift. - `blocker-stall-wakeups.ts`: before waking a waiter, read its bound remote workspace hold. When the hold refuses a run, do not wake it; count it as deferred and re-check at most hourly. A workspace with an unchanged-release baseline is left to reservation, which can release it. - `blocker-stall-wakeups.ts`: a stall wake whose run failed is retried after 10, then 20 minutes, and not after the third failed run. Refusals (wake-on-demand off, agent paused) keep the 10-minute retry. A new stall (new recovery action) wakes again. - `doc/execution-semantics.md`: document both rules. ## Verification - Red before the fix (`server/src/__tests__/issue-blocker-stall-wakeups.test.ts`): 3 new tests failed. The held-workspace test got a delivered wake; the backoff test got a third wake 11 minutes after the second failure. - Green after the fix: `npx vitest run src/__tests__/issue-blocker-stall-wakeups.test.ts` passes 20/20. - The new tests use the staging shape: a legacy `shared_workspace` remote workspace owned by the parent, a child bound to it, the child's failed native writer run ended four days ago with `native_workspace_sync_out_retry_exhausted`, a dirty authority, and the child's board-owned exhausted recovery action. They also prove that a writer inside the cutoff still gets its wake (the run parks in a workspace wait), and that the wake is delivered once the hold settles. - Related suites pass: `heartbeat-remote-workspace-busy.test.ts`, `heartbeat-dependency-scheduling.test.ts`, `native-cancelled-workspace-preservation.test.ts`, `src/modules/run-dispatch`. - `npx tsc --noEmit -p server` passes. - `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global CI input; required CI supplies the broad coverage. ## Risks - Low. No schema or persisted-shape change. Saved state from the previous release (existing wake rows, existing workspace authority) is read as-is: earlier `failed` stall wake rows count toward the new three-run limit, which stops the current loop at once after deploy. - A parent whose workspace stays held is not woken for the stall until the hold settles. The stall is still listed on its page and the child's recovery action stays with the board. - Not fixed here: the exhausted save hold itself is never settled automatically (`agent_stop_unconfirmed` exhaustion in workspace recovery), and legacy children that still share a parent's workspace are not moved. Both touch workspace recovery code that is under concurrent change and are left for a follow-up. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser read-only inspection of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
344 lines
27 KiB
TypeScript
344 lines
27 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import { eq } from "drizzle-orm";
|
|
import { agents, agentWakeupRequests, applyPendingMigrations, companies, createDb, environmentLeases, environments, executionWorkspaces, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, plugins, workspaceOperations } from "@paperclipai/db";
|
|
import type { Environment } from "@paperclipai/shared";
|
|
import type { PluginWorkerManager } from "../plugin-worker-manager.js";
|
|
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
|
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
|
|
import { environmentRuntimeService } from "../environment-runtime.js";
|
|
import * as leases from "../environments.js";
|
|
import * as providers from "../sandbox-provider-runtime.js";
|
|
import { createSystemRecovery, failSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "../system-recovery-state.js";
|
|
import { recoverRetainedWorkspace } from "../workspace-recovery.js";
|
|
import { preserveLegacyWorkspaceRestoreSources } from "../legacy-workspace-restore-recovery.js";
|
|
import { settleStopOnlyCleanup } from "../sandbox-stop-and-retain.js";
|
|
import { settleRecoveredWorkspaceRestoreHolds } from "../workspace-restore-hold-settlement.js";
|
|
import { nativeWorkspaceExportPendingForLease, nativeWorkspaceExportPendingForResource } from "./native-workspace-finalization-state.js";
|
|
|
|
describe("native workspace export warm reuse fence", () => {
|
|
let db: ReturnType<typeof createDb>;
|
|
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
beforeAll(async () => {
|
|
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_RECOVERY_TEST_DATABASE_URL");
|
|
if (external) { await applyPendingMigrations(external); temporary = { connectionString: external, cleanup: async () => {} }; }
|
|
else temporary = await startEmbeddedPostgresTestDatabase("native-warm-reuse-");
|
|
db = createDb(temporary.connectionString);
|
|
}, 30_000);
|
|
afterEach(() => vi.restoreAllMocks());
|
|
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); });
|
|
|
|
async function seed(plugin: boolean, release = true) {
|
|
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), pluginId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Native warm fence", issuePrefix: `W${companyId.slice(0, 6)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "paperclip_runner" });
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Keep the source", assigneeAgentId: agentId });
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, nativeIssueId: issueId, runtimeMode: "native", status: "running" });
|
|
const provider = plugin ? "native-fence-plugin" : "fake";
|
|
if (plugin) await db.insert(plugins).values({ id: pluginId, pluginKey: `native-fence.${pluginId}`,
|
|
packageName: "native-fence-provider", version: "1.0.0", apiVersion: 1, categories: ["automation"], status: "ready",
|
|
manifestJson: { id: `native-fence.${pluginId}`, apiVersion: 1, version: "1.0.0", displayName: "Native fence provider",
|
|
description: "Controlled lifecycle boundary", author: "Paperclip", categories: ["automation"],
|
|
capabilities: ["environment.drivers.register"], entrypoints: { worker: "dist/worker.js" },
|
|
environmentDrivers: [{ driverKey: provider, kind: "sandbox_provider", displayName: "Native fence", supportsReusableLeases: true,
|
|
configSchema: { type: "object", properties: { image: { type: "string" }, reuseLease: { type: "boolean" } } } }] } });
|
|
const [environment] = await db.insert(environments).values({ name: runId, driver: "sandbox", status: "active",
|
|
config: { provider, image: "fake:test", reuseLease: true } }).returning();
|
|
const providerLeaseId: string = randomUUID();
|
|
let acquisitions = 0;
|
|
const call = vi.fn(async (_id: string, method: string, input?: Record<string, unknown>) => {
|
|
if (method === "environmentAcquireLease") return { providerLeaseId: acquisitions++ ? randomUUID() : providerLeaseId,
|
|
metadata: { remoteCwd: "/work" } };
|
|
if (method === "environmentResumeLease") return { providerLeaseId: String(input?.providerLeaseId ?? providerLeaseId), metadata: { remoteCwd: "/work" } };
|
|
if (["environmentReleaseLease", "environmentStopLease"].includes(method)) return { providerLeaseId, state: "stopped" };
|
|
if (method === "environmentDestroyLease") return { providerLeaseId, state: "destroyed" };
|
|
throw new Error(method);
|
|
});
|
|
const worker = { isRunning: () => true, call, getWorker: () => ({ supportedMethods: ["environmentResumeLease",
|
|
"environmentReleaseLease", "environmentStopLease", "environmentDestroyLease"] }) } as unknown as PluginWorkerManager;
|
|
if (!plugin) {
|
|
const original = providers.getSandboxProvider;
|
|
vi.spyOn(providers, "getSandboxProvider").mockImplementation(key => {
|
|
const value = original(key);
|
|
return value && key === "fake" ? Object.assign(Object.create(value), { supportsReusableLeases: true }) : value;
|
|
});
|
|
}
|
|
const runtime = environmentRuntimeService(db, { pluginWorkerManager: worker });
|
|
const input = { companyId, agentId, issueId, environment: environment as unknown as Environment,
|
|
heartbeatRunId: runId, persistedExecutionWorkspace: null };
|
|
const acquired = await runtime.acquireRunLease(input);
|
|
if (release) await runtime.releaseRunLeases(runId, "released", undefined, "keep_running");
|
|
const reference = { schema: "paperclip.native-workspace-sync/v3", state: "prepared", leaseId: acquired.lease.id,
|
|
providerLeaseId: acquired.lease.providerLeaseId, workspaceId: issueId, remoteCwd: "/work", resourceDisposition: "keep_running",
|
|
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null };
|
|
if (release) await db.update(heartbeatRuns).set({ status: "failed", runnerProfileJson: { nativeWorkspaceSync: reference } }).where(eq(heartbeatRuns.id, runId));
|
|
const nextRunId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id: nextRunId, companyId, agentId, issueId, nativeIssueId: issueId, runtimeMode: "native", status: "running" });
|
|
call.mockClear();
|
|
return { runtime, input, runId, nextRunId, reference, lease: acquired.lease, call };
|
|
}
|
|
|
|
async function expectFreshAllocation(f: Awaited<ReturnType<typeof seed>>, runtime = f.runtime) {
|
|
const [protectedLease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
|
const fresh = await runtime.acquireRunLease({ ...f.input, heartbeatRunId: f.nextRunId });
|
|
expect(fresh.lease.providerLeaseId).not.toBe(f.lease.providerLeaseId);
|
|
expect(fresh.lease.heartbeatRunId).toBe(f.nextRunId);
|
|
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0]).toEqual(protectedLease);
|
|
expect(f.call.mock.calls.map(call => call[1])).not.toContain("environmentResumeLease");
|
|
expect(f.call.mock.calls.map(call => call[1])).not.toContain("environmentDestroyLease");
|
|
return fresh;
|
|
}
|
|
|
|
it.each([false, true])("allocates a fresh sandbox while retaining its unfinished source for plugin=%s", async plugin => {
|
|
const f = await seed(plugin);
|
|
const acquire = vi.spyOn(providers, "acquireSandboxProviderLease");
|
|
const destroy = vi.spyOn(providers, "destroySandboxProviderLease");
|
|
await expectFreshAllocation(f);
|
|
expect(destroy).not.toHaveBeenCalled();
|
|
if (plugin) expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentAcquireLease"]);
|
|
else expect(acquire).toHaveBeenCalledWith(expect.objectContaining({ reusableProviderLeaseId: null }));
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].runnerProfileJson)
|
|
.toMatchObject({ nativeWorkspaceSync: { state: "prepared", leaseId: f.lease.id } });
|
|
});
|
|
|
|
it.each([{ plugin: false, obsolete: false }, { plugin: true, obsolete: false },
|
|
{ plugin: false, obsolete: true }, { plugin: true, obsolete: true }])(
|
|
"reuses a clean candidate without touching a pending allocation: plugin=$plugin obsolete=$obsolete", async ({ plugin, obsolete }) => {
|
|
const f = await seed(plugin), cleanRunId = randomUUID(), cleanLeaseId = randomUUID(), cleanProviderLeaseId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id: cleanRunId, companyId: f.input.companyId, agentId: f.input.agentId,
|
|
issueId: f.input.issueId, runtimeMode: "legacy", status: "succeeded", finishedAt: new Date() });
|
|
await db.insert(environmentLeases).values({ id: cleanLeaseId, companyId: f.input.companyId, environmentId: f.input.environment.id,
|
|
heartbeatRunId: cleanRunId, issueId: f.input.issueId, provider: f.lease.provider, providerLeaseId: cleanProviderLeaseId,
|
|
leasePolicy: "reuse_by_environment", status: "released", cleanupStatus: "success", metadata: f.lease.metadata });
|
|
if (obsolete) await db.update(environmentLeases).set({ metadata: { ...f.lease.metadata, reusableSandboxLease: null } })
|
|
.where(eq(environmentLeases.id, f.lease.id));
|
|
const [protectedLease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
|
// The provider boundary returns the exact clean allocation selected by
|
|
// the real policy. No source inspection or database reads are mocked.
|
|
const acquire = vi.spyOn(providers, "acquireSandboxProviderLease");
|
|
if (!plugin) acquire.mockImplementation(async input => ({ providerLeaseId: input.reusableProviderLeaseId!,
|
|
metadata: { provider: "fake", image: "fake:test", reuseLease: true } }));
|
|
const destroy = vi.spyOn(providers, "destroySandboxProviderLease");
|
|
const resumed = await f.runtime.acquireRunLease({ ...f.input, heartbeatRunId: f.nextRunId });
|
|
expect(resumed.lease).toMatchObject({ heartbeatRunId: f.nextRunId, providerLeaseId: cleanProviderLeaseId });
|
|
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0]).toEqual(protectedLease);
|
|
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, cleanLeaseId)))[0].status).toBe("expired");
|
|
expect(destroy).not.toHaveBeenCalled();
|
|
if (plugin) {
|
|
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentResumeLease"]);
|
|
expect(f.call.mock.calls[0]?.[2]).toMatchObject({ providerLeaseId: cleanProviderLeaseId });
|
|
} else expect(acquire).toHaveBeenCalledWith(expect.objectContaining({ reusableProviderLeaseId: cleanProviderLeaseId }));
|
|
});
|
|
|
|
it("checks the allocation even when the candidate snapshot contains only a rebound row", async () => {
|
|
const f = await seed(true);
|
|
const service = leases.environmentService(db);
|
|
const [rebound] = await db.insert(environmentLeases).values({ companyId: f.input.companyId, environmentId: f.input.environment.id,
|
|
issueId: f.input.issueId, heartbeatRunId: f.nextRunId, provider: f.lease.provider, providerLeaseId: f.lease.providerLeaseId,
|
|
leasePolicy: "reuse_by_environment", status: "retained", metadata: f.lease.metadata }).returning();
|
|
const list = service.listLeases;
|
|
service.listLeases = async (...args) => (await list(...args)).filter(lease => lease.id === rebound.id);
|
|
vi.spyOn(leases, "environmentService").mockReturnValue(service);
|
|
const runtime = environmentRuntimeService(db, { pluginWorkerManager: { isRunning: () => true, call: f.call,
|
|
getWorker: () => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] }) } as unknown as PluginWorkerManager });
|
|
await expectFreshAllocation(f, runtime);
|
|
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentAcquireLease"]);
|
|
});
|
|
|
|
it.each(["native", "legacy"])("fences a historical modern %s writer in one allocation read", async runtimeMode => {
|
|
const f = await seed(true), workspaceId = randomUUID();
|
|
const reference = { schema: "paperclip.remote-workspace-run.v1", workspaceId, leaseId: f.lease.id,
|
|
providerLeaseId: f.lease.providerLeaseId!, remoteCwd: "/work", epoch: 1, state: "prepared", snapshotId: null };
|
|
const authority = { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: null, source: { kind: "empty" },
|
|
writerRunId: f.runId, dirty: true, location: { environmentId: f.input.environment.id,
|
|
leaseId: f.lease.id, providerLeaseId: f.lease.providerLeaseId!, remoteCwd: "/work" } };
|
|
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId: f.input.companyId,
|
|
sourceIssueId: f.input.issueId, name: "Unsaved remote source", cwd: "/work", providerRef: "/work",
|
|
providerType: "cloud_sandbox", mode: "isolated_workspace", strategyType: "git_worktree",
|
|
status: "active", metadata: { remoteWorkspace: authority } });
|
|
await db.update(environmentLeases).set({ executionWorkspaceId: workspaceId }).where(eq(environmentLeases.id, f.lease.id));
|
|
await db.update(heartbeatRuns).set({ runtimeMode, nativeIssueId: runtimeMode === "native" ? f.input.issueId : null,
|
|
runnerProfileJson: { remoteWorkspace: reference } }).where(eq(heartbeatRuns.id, f.runId));
|
|
const [rebound] = await db.insert(environmentLeases).values({ companyId: f.input.companyId, environmentId: f.input.environment.id,
|
|
issueId: f.input.issueId, heartbeatRunId: f.nextRunId, provider: f.lease.provider, providerLeaseId: f.lease.providerLeaseId,
|
|
leasePolicy: "reuse_by_environment", status: "retained", metadata: f.lease.metadata }).returning();
|
|
const reads = vi.spyOn(db, "select");
|
|
expect(await nativeWorkspaceExportPendingForResource(db, rebound, f.nextRunId)).toBe(true);
|
|
expect(reads).toHaveBeenCalledTimes(1);
|
|
reads.mockRestore();
|
|
await expectFreshAllocation(f);
|
|
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentAcquireLease"]);
|
|
// A saved descriptor or a changed writer cannot keep fencing an allocation.
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: { remoteWorkspace: { ...reference, state: "saved" } } })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
expect(await nativeWorkspaceExportPendingForResource(db, rebound, f.nextRunId)).toBe(false);
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: { remoteWorkspace: reference } }).where(eq(heartbeatRuns.id, f.runId));
|
|
await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: { ...authority, writerRunId: f.nextRunId } } })
|
|
.where(eq(executionWorkspaces.id, workspaceId));
|
|
expect(await nativeWorkspaceExportPendingForResource(db, rebound, f.nextRunId)).toBe(false);
|
|
});
|
|
|
|
it("rechecks a prepared export after selecting a stale candidate and before resuming it", async () => {
|
|
const f = await seed(true);
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: { ...f.reference,
|
|
state: "finalized", finalHostSha256: "c".repeat(64) } } }).where(eq(heartbeatRuns.id, f.runId));
|
|
const service = leases.environmentService(db), list = service.listLeases;
|
|
service.listLeases = async (...args) => {
|
|
const snapshot = await list(...args);
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: f.reference } }).where(eq(heartbeatRuns.id, f.runId));
|
|
return snapshot;
|
|
};
|
|
vi.spyOn(leases, "environmentService").mockReturnValue(service);
|
|
const runtime = environmentRuntimeService(db, { pluginWorkerManager: { isRunning: () => true, call: f.call,
|
|
getWorker: () => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] }) } as unknown as PluginWorkerManager });
|
|
await expectFreshAllocation(f, runtime);
|
|
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentAcquireLease"]);
|
|
});
|
|
|
|
it("preserves exact same-run reattachment without allocating a replacement", async () => {
|
|
const f = await seed(true);
|
|
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, f.runId));
|
|
const resumed = await f.runtime.acquireRunLease(f.input);
|
|
expect(resumed.lease.id).toBe(f.lease.id);
|
|
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentResumeLease"]);
|
|
});
|
|
|
|
it("checks a long allocation history in one read and allows finalized workspace reuse", async () => {
|
|
const f = await seed(true);
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: { ...f.reference,
|
|
state: "finalized", finalHostSha256: "c".repeat(64) } } }).where(eq(heartbeatRuns.id, f.runId));
|
|
const history = Array.from({ length: 64 }, () => randomUUID());
|
|
await db.insert(heartbeatRuns).values(history.map(id => ({ id, companyId: f.input.companyId, agentId: f.input.agentId,
|
|
issueId: f.input.issueId, nativeIssueId: f.input.issueId, runtimeMode: "native", status: "succeeded",
|
|
runnerProfileJson: { nativeWorkspaceSync: { ...f.reference, state: "finalized", finalHostSha256: "c".repeat(64) } } })));
|
|
await db.insert(environmentLeases).values(history.map(heartbeatRunId => ({ companyId: f.input.companyId,
|
|
environmentId: f.input.environment.id, heartbeatRunId, provider: f.lease.provider,
|
|
providerLeaseId: f.lease.providerLeaseId, leasePolicy: "reuse_by_environment", status: "released" })));
|
|
const reads = vi.spyOn(db, "select");
|
|
expect(await nativeWorkspaceExportPendingForResource(db, f.lease, f.nextRunId)).toBe(false);
|
|
expect(reads).toHaveBeenCalledTimes(1);
|
|
reads.mockRestore();
|
|
const resumed = await f.runtime.acquireRunLease({ ...f.input, heartbeatRunId: f.nextRunId });
|
|
expect(resumed.lease.id).not.toBe(f.lease.id);
|
|
expect(resumed.lease).toMatchObject({ heartbeatRunId: f.nextRunId, providerLeaseId: f.lease.providerLeaseId });
|
|
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0])
|
|
.toMatchObject({ heartbeatRunId: f.runId, status: "expired" });
|
|
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentResumeLease"]);
|
|
});
|
|
|
|
it.each(["grok_bot", "muse"])("does not fence an exactly bound workspace-less %s run", async kind => {
|
|
const f = await seed(true);
|
|
const binding = { runId: f.runId, companyId: f.input.companyId, agentId: f.input.agentId, issueId: f.input.issueId };
|
|
const profile = { nativeWorkspaceSync: f.reference, nativeExecutionInput: {
|
|
schema: kind === "muse" ? "paperclip.native-execution-input.v8" : "paperclip.native-execution-input.v6",
|
|
binding, workspace: { access: "none", cwd: null }, provider: { kind, bridgeRevision: "muse-v1" } } };
|
|
await db.update(heartbeatRuns).set({ driverKind: kind === "muse" ? "muse_external" : "grok_bot_mcp",
|
|
runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId));
|
|
expect(await nativeWorkspaceExportPendingForLease(db, f.lease)).toBe(false);
|
|
expect(await nativeWorkspaceExportPendingForResource(db, f.lease, f.nextRunId)).toBe(false);
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...profile, nativeExecutionInput: {
|
|
...profile.nativeExecutionInput, binding: { ...binding, runId: randomUUID() } } } }).where(eq(heartbeatRuns.id, f.runId));
|
|
expect(await nativeWorkspaceExportPendingForResource(db, f.lease, f.nextRunId)).toBe(true);
|
|
});
|
|
|
|
it("does not take over a source while an ordinary provider resume is awaiting its database handoff", async () => {
|
|
const f = await seed(true), requestId = randomUUID(), actionId = randomUUID();
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: {} }).where(eq(heartbeatRuns.id, f.runId));
|
|
let entered!: () => void, release!: () => void;
|
|
const started = new Promise<void>(resolve => { entered = resolve; });
|
|
const pending = new Promise<void>(resolve => { release = resolve; });
|
|
f.call.mockImplementation(async (_id: string, method: string) => {
|
|
if (method !== "environmentResumeLease") throw new Error(method);
|
|
entered(); await pending;
|
|
return { providerLeaseId: f.lease.providerLeaseId!, metadata: { remoteCwd: "/work" } };
|
|
});
|
|
const ordinary = f.runtime.acquireRunLease({ ...f.input, heartbeatRunId: f.nextRunId });
|
|
try {
|
|
await started;
|
|
const failure = failSystemRecovery(reserveSystemRecoveryAttempt(createSystemRecovery(f.runId, requestId))!,
|
|
new SystemRecoveryFailure("source_binding_invalid", "service_required"));
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: f.reference },
|
|
resultJson: { workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: {
|
|
schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [f.lease.id] } } }).where(eq(heartbeatRuns.id, f.runId));
|
|
await db.insert(nativeRunFinalizations).values({ companyId: f.input.companyId, issueId: f.input.issueId,
|
|
runId: f.runId, phase: "terminal_failure" });
|
|
await db.insert(issueRecoveryActions).values({ id: actionId, companyId: f.input.companyId, sourceIssueId: f.input.issueId,
|
|
kind: "active_run_watchdog", ownerType: "system", cause: "native_workspace_sync_out_retry_exhausted", fingerprint: f.runId,
|
|
nextAction: "Preserve the source", evidence: { runId: f.runId, systemRecovery: failure, workspaceRecovery: {
|
|
schema: "paperclip.workspace-recovery.v1", runId: f.runId, status: "blocked", stage: "verifying_stop",
|
|
attempt: 1, requestIds: [requestId] } } });
|
|
await db.insert(agentWakeupRequests).values({ id: requestId, companyId: f.input.companyId, agentId: f.input.agentId,
|
|
source: "on_demand", reason: "task_retry", status: "deferred_issue_execution", requestedByActorType: "user", requestedByActorId: "board",
|
|
payload: { issueId: f.input.issueId, taskRetry: { schema: "paperclip.task-retry.v1", generationId: requestId,
|
|
sourceRunId: f.runId, actorId: "board", actorSource: "local_implicit", strategy: "evaluate" } } });
|
|
await recoverRetainedWorkspace({ db, companyId: f.input.companyId, issueId: f.input.issueId,
|
|
runId: f.runId, actionId, environmentRuntime: f.runtime });
|
|
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, actionId));
|
|
expect(action.evidence.systemRecovery).toEqual(failure);
|
|
expect(action.evidence.nativeSourceRetention).toBeUndefined();
|
|
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0].leasePolicy).toBe("reuse_by_environment");
|
|
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentResumeLease"]);
|
|
} finally { release(); await ordinary; }
|
|
});
|
|
|
|
// Regression: a run resumed a warm sandbox and was interrupted before provider
|
|
// work. The restore hold stopped the sandbox and changed its lease to
|
|
// retain_on_failure. Recovery saved the files, yet the next run got a created
|
|
// lease (and a full workspace realize) because nothing restored reuse.
|
|
async function settleRestoreHold(fault: "none" | "unsettled" | "dirty" | "unstopped") {
|
|
const f = await seed(true, false), actionId = randomUUID(), operationId = randomUUID();
|
|
// The interrupted run's real terminal hold over its active warm lease.
|
|
const [interrupted] = await db.update(heartbeatRuns).set({ status: "interrupted", finishedAt: new Date(),
|
|
resultJson: { workspaceRestoreFailure: "restore_failed" } }).where(eq(heartbeatRuns.id, f.runId)).returning();
|
|
const [source] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
|
expect(await db.transaction(tx => preserveLegacyWorkspaceRestoreSources(tx as unknown as typeof db, interrupted, source)))
|
|
.toEqual([f.lease.id]);
|
|
const [stopping] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
|
expect(stopping).toMatchObject({ leasePolicy: "retain_on_failure", status: "pending_cleanup" });
|
|
if (fault !== "unstopped") expect(await settleStopOnlyCleanup(db, stopping, { attemptId: String(stopping.metadata!.pendingCleanupAttemptId),
|
|
receipt: { providerLeaseId: f.lease.providerLeaseId, state: "stopped" } })).not.toBeNull();
|
|
// The recovery's own receipts: files saved from this allocation, and its finalize.
|
|
const system = { ...createSystemRecovery(f.runId), state: fault === "unsettled" ? "running" : "complete",
|
|
phase: "reconcile_effects", preservationOperationIds: [operationId] };
|
|
await db.insert(issueRecoveryActions).values({ id: actionId, companyId: f.input.companyId, sourceIssueId: f.input.issueId,
|
|
kind: "active_run_watchdog", ownerType: "system", cause: "native_continuation_requires_reconciliation", fingerprint: f.runId,
|
|
nextAction: "Recover saved work", evidence: { runId: f.runId, workspaceRestoreFailure: "restore_failed",
|
|
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [f.lease.id] }, systemRecovery: system,
|
|
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: f.runId,
|
|
status: fault === "unsettled" ? "recovering" : "recovered", stage: "saving_receipts", attempt: 1, requestIds: [] } } });
|
|
await db.insert(workspaceOperations).values([{ id: operationId, companyId: f.input.companyId, issueId: f.input.issueId,
|
|
heartbeatRunId: f.runId, phase: "workspace_recovery", status: "succeeded", finishedAt: new Date(),
|
|
metadata: { recoveryActionId: actionId, leaseId: f.lease.id, providerLeaseId: f.lease.providerLeaseId } },
|
|
{ companyId: f.input.companyId, issueId: f.input.issueId, heartbeatRunId: f.runId, phase: "workspace_finalize",
|
|
status: "succeeded", finishedAt: new Date(), metadata: { owningService: "workspace_recovery", preservationOperationIds: [operationId] } }]);
|
|
// An unsaved writer still bound to the stopped sandbox keeps it held.
|
|
if (fault === "dirty") await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: f.reference } })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
const restored = await settleRecoveredWorkspaceRestoreHolds(db, { companyId: f.input.companyId, issueId: f.input.issueId,
|
|
runId: f.runId, actionId });
|
|
f.call.mockClear();
|
|
return { ...f, restored };
|
|
}
|
|
|
|
it("resumes the warm sandbox on the same provider lease after a restore hold settles", async () => {
|
|
const f = await settleRestoreHold("none");
|
|
const next = await f.runtime.acquireRunLease({ ...f.input, heartbeatRunId: f.nextRunId });
|
|
expect(next.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "resumed" });
|
|
expect(next.lease).toMatchObject({ heartbeatRunId: f.nextRunId, providerLeaseId: f.lease.providerLeaseId,
|
|
leasePolicy: "reuse_by_environment" });
|
|
expect(f.restored).toEqual([f.lease.id]);
|
|
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentResumeLease"]);
|
|
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0])
|
|
.toMatchObject({ heartbeatRunId: f.runId, status: "expired", leasePolicy: "reuse_by_environment" });
|
|
});
|
|
|
|
it.each(["unsettled", "dirty", "unstopped"] as const)("keeps a %s restore hold out of warm reuse", async fault => {
|
|
const f = await settleRestoreHold(fault);
|
|
expect(f.restored).toEqual([]);
|
|
const [held] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
|
expect(held.leasePolicy).toBe("retain_on_failure");
|
|
expect(held.metadata?.workspaceRestoreRecovery).toMatchObject({ runId: f.runId, originalLeasePolicy: "reuse_by_environment" });
|
|
if (fault !== "unstopped") await expectFreshAllocation(f);
|
|
});
|
|
});
|