Files
PaperClipAI/server/src/services/native-runtime/native-workspace-warm-reuse.test.ts
T
DottaandClaude Opus 5.5 2dd7a98753 fix(recovery): return a settled restore hold's warm sandbox to reuse (#20278)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Recovery tells a waiting parent or dependent when a child or blocker
stalls on the board (`issue_blocker_stalled`)
> - A legacy child that an older release bound to its parent's remote
workspace failed its save, and its save recovery was exhausted. Its
ended run still holds the shared workspace, days past the one-hour
holder cutoff
> - Workspace reservation refuses a run on such a hold and fails setup
with "The workspace still has an unfinished writer or save operation."
Every stall wake of the parent created a run that could only fail this
way
> - The stall sweep treated each failed run as an undelivered wake and
retried it every ten minutes, with no backoff and no limit. On staging
this made 72 failed runs (58 of 60 failures since a deploy), in pairs
every ten minutes
> - This pull request makes the stall sweep hold a wake while the
waiter's workspace hold refuses a run, and bounds retries of a wake
whose run fails
> - The benefit is that a held workspace no longer produces a failed run
every ten minutes, and the wake still arrives once the hold settles

## Linked Issues or Issue Description

**What happened?**
A parent task was woken by `issue_blocker_stalled` twice every ten
minutes. Each run failed in setup (`setup_failed`,
`executionRecovery.kind=bootstrap`, `providerWorkStarted=false`) with
"The workspace still has an unfinished writer or save operation." The
parent's remote workspace was held by a child's ended writer run whose
save recovery was exhausted
(`native_workspace_sync_out_retry_exhausted`, last failure
`agent_stop_unconfirmed`). The child shared the parent's workspace
because an older release created it before subtasks got their own
workspace (#19860).

**Expected behavior**
A wake that cannot start because its workspace is held must not create a
failed run every time. It should wait until the hold settles. Automatic
retries must be bounded.

**Root cause**
- `server/src/services/heartbeat/workspace-preparation.ts` rethrows
`RemoteWorkspaceWriterActiveError` as a setup failure when the ended
writer is silent past `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS`
(documented, intended).
- `server/src/services/recovery/blocker-stall-wakeups.ts` retried every
non-delivered wake row (including `failed`) after
`REFUSED_WAKE_RETRY_MS` (10 minutes) with no backoff and no limit, and
it did not check whether the waiter could start at all.

## What Changed

- `remote-workspace-writer-hold.ts`: add
`remoteWorkspaceHoldRefusesStart`, the one rule for a hold that
reservation refuses outright (unattributed writer, or a writer silent
past the cutoff with no live owner fence).
- `workspace-preparation.ts`: use that rule instead of an inline copy,
so reservation and wake senders cannot drift.
- `blocker-stall-wakeups.ts`: before waking a waiter, read its bound
remote workspace hold. When the hold refuses a run, do not wake it;
count it as deferred and re-check at most hourly. A workspace with an
unchanged-release baseline is left to reservation, which can release it.
- `blocker-stall-wakeups.ts`: a stall wake whose run failed is retried
after 10, then 20 minutes, and not after the third failed run. Refusals
(wake-on-demand off, agent paused) keep the 10-minute retry. A new stall
(new recovery action) wakes again.
- `doc/execution-semantics.md`: document both rules.

## Verification

- Red before the fix
(`server/src/__tests__/issue-blocker-stall-wakeups.test.ts`): 3 new
tests failed. The held-workspace test got a delivered wake; the backoff
test got a third wake 11 minutes after the second failure.
- Green after the fix: `npx vitest run
src/__tests__/issue-blocker-stall-wakeups.test.ts` passes 20/20.
- The new tests use the staging shape: a legacy `shared_workspace`
remote workspace owned by the parent, a child bound to it, the child's
failed native writer run ended four days ago with
`native_workspace_sync_out_retry_exhausted`, a dirty authority, and the
child's board-owned exhausted recovery action. They also prove that a
writer inside the cutoff still gets its wake (the run parks in a
workspace wait), and that the wake is delivered once the hold settles.
- Related suites pass: `heartbeat-remote-workspace-busy.test.ts`,
`heartbeat-dependency-scheduling.test.ts`,
`native-cancelled-workspace-preservation.test.ts`,
`src/modules/run-dispatch`.
- `npx tsc --noEmit -p server` passes.
- `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global
CI input; required CI supplies the broad coverage.

## Risks

- Low. No schema or persisted-shape change. Saved state from the
previous release (existing wake rows, existing workspace authority) is
read as-is: earlier `failed` stall wake rows count toward the new
three-run limit, which stops the current loop at once after deploy.
- A parent whose workspace stays held is not woken for the stall until
the hold settles. The stall is still listed on its page and the child's
recovery action stays with the board.
- Not fixed here: the exhausted save hold itself is never settled
automatically (`agent_stop_unconfirmed` exhaustion in workspace
recovery), and legacy children that still share a parent's workspace are
not moved. Both touch workspace recovery code that is under concurrent
change and are left for a follow-up.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser read-only inspection of
staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 08:28:04 -06:00

344 lines
27 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { eq } from "drizzle-orm";
import { agents, agentWakeupRequests, applyPendingMigrations, companies, createDb, environmentLeases, environments, executionWorkspaces, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, plugins, workspaceOperations } from "@paperclipai/db";
import type { Environment } from "@paperclipai/shared";
import type { PluginWorkerManager } from "../plugin-worker-manager.js";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
import { environmentRuntimeService } from "../environment-runtime.js";
import * as leases from "../environments.js";
import * as providers from "../sandbox-provider-runtime.js";
import { createSystemRecovery, failSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "../system-recovery-state.js";
import { recoverRetainedWorkspace } from "../workspace-recovery.js";
import { preserveLegacyWorkspaceRestoreSources } from "../legacy-workspace-restore-recovery.js";
import { settleStopOnlyCleanup } from "../sandbox-stop-and-retain.js";
import { settleRecoveredWorkspaceRestoreHolds } from "../workspace-restore-hold-settlement.js";
import { nativeWorkspaceExportPendingForLease, nativeWorkspaceExportPendingForResource } from "./native-workspace-finalization-state.js";
describe("native workspace export warm reuse fence", () => {
let db: ReturnType<typeof createDb>;
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
beforeAll(async () => {
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_RECOVERY_TEST_DATABASE_URL");
if (external) { await applyPendingMigrations(external); temporary = { connectionString: external, cleanup: async () => {} }; }
else temporary = await startEmbeddedPostgresTestDatabase("native-warm-reuse-");
db = createDb(temporary.connectionString);
}, 30_000);
afterEach(() => vi.restoreAllMocks());
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); });
async function seed(plugin: boolean, release = true) {
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), pluginId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Native warm fence", issuePrefix: `W${companyId.slice(0, 6)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "paperclip_runner" });
await db.insert(issues).values({ id: issueId, companyId, title: "Keep the source", assigneeAgentId: agentId });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, nativeIssueId: issueId, runtimeMode: "native", status: "running" });
const provider = plugin ? "native-fence-plugin" : "fake";
if (plugin) await db.insert(plugins).values({ id: pluginId, pluginKey: `native-fence.${pluginId}`,
packageName: "native-fence-provider", version: "1.0.0", apiVersion: 1, categories: ["automation"], status: "ready",
manifestJson: { id: `native-fence.${pluginId}`, apiVersion: 1, version: "1.0.0", displayName: "Native fence provider",
description: "Controlled lifecycle boundary", author: "Paperclip", categories: ["automation"],
capabilities: ["environment.drivers.register"], entrypoints: { worker: "dist/worker.js" },
environmentDrivers: [{ driverKey: provider, kind: "sandbox_provider", displayName: "Native fence", supportsReusableLeases: true,
configSchema: { type: "object", properties: { image: { type: "string" }, reuseLease: { type: "boolean" } } } }] } });
const [environment] = await db.insert(environments).values({ name: runId, driver: "sandbox", status: "active",
config: { provider, image: "fake:test", reuseLease: true } }).returning();
const providerLeaseId: string = randomUUID();
let acquisitions = 0;
const call = vi.fn(async (_id: string, method: string, input?: Record<string, unknown>) => {
if (method === "environmentAcquireLease") return { providerLeaseId: acquisitions++ ? randomUUID() : providerLeaseId,
metadata: { remoteCwd: "/work" } };
if (method === "environmentResumeLease") return { providerLeaseId: String(input?.providerLeaseId ?? providerLeaseId), metadata: { remoteCwd: "/work" } };
if (["environmentReleaseLease", "environmentStopLease"].includes(method)) return { providerLeaseId, state: "stopped" };
if (method === "environmentDestroyLease") return { providerLeaseId, state: "destroyed" };
throw new Error(method);
});
const worker = { isRunning: () => true, call, getWorker: () => ({ supportedMethods: ["environmentResumeLease",
"environmentReleaseLease", "environmentStopLease", "environmentDestroyLease"] }) } as unknown as PluginWorkerManager;
if (!plugin) {
const original = providers.getSandboxProvider;
vi.spyOn(providers, "getSandboxProvider").mockImplementation(key => {
const value = original(key);
return value && key === "fake" ? Object.assign(Object.create(value), { supportsReusableLeases: true }) : value;
});
}
const runtime = environmentRuntimeService(db, { pluginWorkerManager: worker });
const input = { companyId, agentId, issueId, environment: environment as unknown as Environment,
heartbeatRunId: runId, persistedExecutionWorkspace: null };
const acquired = await runtime.acquireRunLease(input);
if (release) await runtime.releaseRunLeases(runId, "released", undefined, "keep_running");
const reference = { schema: "paperclip.native-workspace-sync/v3", state: "prepared", leaseId: acquired.lease.id,
providerLeaseId: acquired.lease.providerLeaseId, workspaceId: issueId, remoteCwd: "/work", resourceDisposition: "keep_running",
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null };
if (release) await db.update(heartbeatRuns).set({ status: "failed", runnerProfileJson: { nativeWorkspaceSync: reference } }).where(eq(heartbeatRuns.id, runId));
const nextRunId = randomUUID();
await db.insert(heartbeatRuns).values({ id: nextRunId, companyId, agentId, issueId, nativeIssueId: issueId, runtimeMode: "native", status: "running" });
call.mockClear();
return { runtime, input, runId, nextRunId, reference, lease: acquired.lease, call };
}
async function expectFreshAllocation(f: Awaited<ReturnType<typeof seed>>, runtime = f.runtime) {
const [protectedLease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
const fresh = await runtime.acquireRunLease({ ...f.input, heartbeatRunId: f.nextRunId });
expect(fresh.lease.providerLeaseId).not.toBe(f.lease.providerLeaseId);
expect(fresh.lease.heartbeatRunId).toBe(f.nextRunId);
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0]).toEqual(protectedLease);
expect(f.call.mock.calls.map(call => call[1])).not.toContain("environmentResumeLease");
expect(f.call.mock.calls.map(call => call[1])).not.toContain("environmentDestroyLease");
return fresh;
}
it.each([false, true])("allocates a fresh sandbox while retaining its unfinished source for plugin=%s", async plugin => {
const f = await seed(plugin);
const acquire = vi.spyOn(providers, "acquireSandboxProviderLease");
const destroy = vi.spyOn(providers, "destroySandboxProviderLease");
await expectFreshAllocation(f);
expect(destroy).not.toHaveBeenCalled();
if (plugin) expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentAcquireLease"]);
else expect(acquire).toHaveBeenCalledWith(expect.objectContaining({ reusableProviderLeaseId: null }));
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].runnerProfileJson)
.toMatchObject({ nativeWorkspaceSync: { state: "prepared", leaseId: f.lease.id } });
});
it.each([{ plugin: false, obsolete: false }, { plugin: true, obsolete: false },
{ plugin: false, obsolete: true }, { plugin: true, obsolete: true }])(
"reuses a clean candidate without touching a pending allocation: plugin=$plugin obsolete=$obsolete", async ({ plugin, obsolete }) => {
const f = await seed(plugin), cleanRunId = randomUUID(), cleanLeaseId = randomUUID(), cleanProviderLeaseId = randomUUID();
await db.insert(heartbeatRuns).values({ id: cleanRunId, companyId: f.input.companyId, agentId: f.input.agentId,
issueId: f.input.issueId, runtimeMode: "legacy", status: "succeeded", finishedAt: new Date() });
await db.insert(environmentLeases).values({ id: cleanLeaseId, companyId: f.input.companyId, environmentId: f.input.environment.id,
heartbeatRunId: cleanRunId, issueId: f.input.issueId, provider: f.lease.provider, providerLeaseId: cleanProviderLeaseId,
leasePolicy: "reuse_by_environment", status: "released", cleanupStatus: "success", metadata: f.lease.metadata });
if (obsolete) await db.update(environmentLeases).set({ metadata: { ...f.lease.metadata, reusableSandboxLease: null } })
.where(eq(environmentLeases.id, f.lease.id));
const [protectedLease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
// The provider boundary returns the exact clean allocation selected by
// the real policy. No source inspection or database reads are mocked.
const acquire = vi.spyOn(providers, "acquireSandboxProviderLease");
if (!plugin) acquire.mockImplementation(async input => ({ providerLeaseId: input.reusableProviderLeaseId!,
metadata: { provider: "fake", image: "fake:test", reuseLease: true } }));
const destroy = vi.spyOn(providers, "destroySandboxProviderLease");
const resumed = await f.runtime.acquireRunLease({ ...f.input, heartbeatRunId: f.nextRunId });
expect(resumed.lease).toMatchObject({ heartbeatRunId: f.nextRunId, providerLeaseId: cleanProviderLeaseId });
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0]).toEqual(protectedLease);
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, cleanLeaseId)))[0].status).toBe("expired");
expect(destroy).not.toHaveBeenCalled();
if (plugin) {
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentResumeLease"]);
expect(f.call.mock.calls[0]?.[2]).toMatchObject({ providerLeaseId: cleanProviderLeaseId });
} else expect(acquire).toHaveBeenCalledWith(expect.objectContaining({ reusableProviderLeaseId: cleanProviderLeaseId }));
});
it("checks the allocation even when the candidate snapshot contains only a rebound row", async () => {
const f = await seed(true);
const service = leases.environmentService(db);
const [rebound] = await db.insert(environmentLeases).values({ companyId: f.input.companyId, environmentId: f.input.environment.id,
issueId: f.input.issueId, heartbeatRunId: f.nextRunId, provider: f.lease.provider, providerLeaseId: f.lease.providerLeaseId,
leasePolicy: "reuse_by_environment", status: "retained", metadata: f.lease.metadata }).returning();
const list = service.listLeases;
service.listLeases = async (...args) => (await list(...args)).filter(lease => lease.id === rebound.id);
vi.spyOn(leases, "environmentService").mockReturnValue(service);
const runtime = environmentRuntimeService(db, { pluginWorkerManager: { isRunning: () => true, call: f.call,
getWorker: () => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] }) } as unknown as PluginWorkerManager });
await expectFreshAllocation(f, runtime);
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentAcquireLease"]);
});
it.each(["native", "legacy"])("fences a historical modern %s writer in one allocation read", async runtimeMode => {
const f = await seed(true), workspaceId = randomUUID();
const reference = { schema: "paperclip.remote-workspace-run.v1", workspaceId, leaseId: f.lease.id,
providerLeaseId: f.lease.providerLeaseId!, remoteCwd: "/work", epoch: 1, state: "prepared", snapshotId: null };
const authority = { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: null, source: { kind: "empty" },
writerRunId: f.runId, dirty: true, location: { environmentId: f.input.environment.id,
leaseId: f.lease.id, providerLeaseId: f.lease.providerLeaseId!, remoteCwd: "/work" } };
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId: f.input.companyId,
sourceIssueId: f.input.issueId, name: "Unsaved remote source", cwd: "/work", providerRef: "/work",
providerType: "cloud_sandbox", mode: "isolated_workspace", strategyType: "git_worktree",
status: "active", metadata: { remoteWorkspace: authority } });
await db.update(environmentLeases).set({ executionWorkspaceId: workspaceId }).where(eq(environmentLeases.id, f.lease.id));
await db.update(heartbeatRuns).set({ runtimeMode, nativeIssueId: runtimeMode === "native" ? f.input.issueId : null,
runnerProfileJson: { remoteWorkspace: reference } }).where(eq(heartbeatRuns.id, f.runId));
const [rebound] = await db.insert(environmentLeases).values({ companyId: f.input.companyId, environmentId: f.input.environment.id,
issueId: f.input.issueId, heartbeatRunId: f.nextRunId, provider: f.lease.provider, providerLeaseId: f.lease.providerLeaseId,
leasePolicy: "reuse_by_environment", status: "retained", metadata: f.lease.metadata }).returning();
const reads = vi.spyOn(db, "select");
expect(await nativeWorkspaceExportPendingForResource(db, rebound, f.nextRunId)).toBe(true);
expect(reads).toHaveBeenCalledTimes(1);
reads.mockRestore();
await expectFreshAllocation(f);
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentAcquireLease"]);
// A saved descriptor or a changed writer cannot keep fencing an allocation.
await db.update(heartbeatRuns).set({ runnerProfileJson: { remoteWorkspace: { ...reference, state: "saved" } } })
.where(eq(heartbeatRuns.id, f.runId));
expect(await nativeWorkspaceExportPendingForResource(db, rebound, f.nextRunId)).toBe(false);
await db.update(heartbeatRuns).set({ runnerProfileJson: { remoteWorkspace: reference } }).where(eq(heartbeatRuns.id, f.runId));
await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: { ...authority, writerRunId: f.nextRunId } } })
.where(eq(executionWorkspaces.id, workspaceId));
expect(await nativeWorkspaceExportPendingForResource(db, rebound, f.nextRunId)).toBe(false);
});
it("rechecks a prepared export after selecting a stale candidate and before resuming it", async () => {
const f = await seed(true);
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: { ...f.reference,
state: "finalized", finalHostSha256: "c".repeat(64) } } }).where(eq(heartbeatRuns.id, f.runId));
const service = leases.environmentService(db), list = service.listLeases;
service.listLeases = async (...args) => {
const snapshot = await list(...args);
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: f.reference } }).where(eq(heartbeatRuns.id, f.runId));
return snapshot;
};
vi.spyOn(leases, "environmentService").mockReturnValue(service);
const runtime = environmentRuntimeService(db, { pluginWorkerManager: { isRunning: () => true, call: f.call,
getWorker: () => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] }) } as unknown as PluginWorkerManager });
await expectFreshAllocation(f, runtime);
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentAcquireLease"]);
});
it("preserves exact same-run reattachment without allocating a replacement", async () => {
const f = await seed(true);
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, f.runId));
const resumed = await f.runtime.acquireRunLease(f.input);
expect(resumed.lease.id).toBe(f.lease.id);
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentResumeLease"]);
});
it("checks a long allocation history in one read and allows finalized workspace reuse", async () => {
const f = await seed(true);
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: { ...f.reference,
state: "finalized", finalHostSha256: "c".repeat(64) } } }).where(eq(heartbeatRuns.id, f.runId));
const history = Array.from({ length: 64 }, () => randomUUID());
await db.insert(heartbeatRuns).values(history.map(id => ({ id, companyId: f.input.companyId, agentId: f.input.agentId,
issueId: f.input.issueId, nativeIssueId: f.input.issueId, runtimeMode: "native", status: "succeeded",
runnerProfileJson: { nativeWorkspaceSync: { ...f.reference, state: "finalized", finalHostSha256: "c".repeat(64) } } })));
await db.insert(environmentLeases).values(history.map(heartbeatRunId => ({ companyId: f.input.companyId,
environmentId: f.input.environment.id, heartbeatRunId, provider: f.lease.provider,
providerLeaseId: f.lease.providerLeaseId, leasePolicy: "reuse_by_environment", status: "released" })));
const reads = vi.spyOn(db, "select");
expect(await nativeWorkspaceExportPendingForResource(db, f.lease, f.nextRunId)).toBe(false);
expect(reads).toHaveBeenCalledTimes(1);
reads.mockRestore();
const resumed = await f.runtime.acquireRunLease({ ...f.input, heartbeatRunId: f.nextRunId });
expect(resumed.lease.id).not.toBe(f.lease.id);
expect(resumed.lease).toMatchObject({ heartbeatRunId: f.nextRunId, providerLeaseId: f.lease.providerLeaseId });
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0])
.toMatchObject({ heartbeatRunId: f.runId, status: "expired" });
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentResumeLease"]);
});
it.each(["grok_bot", "muse"])("does not fence an exactly bound workspace-less %s run", async kind => {
const f = await seed(true);
const binding = { runId: f.runId, companyId: f.input.companyId, agentId: f.input.agentId, issueId: f.input.issueId };
const profile = { nativeWorkspaceSync: f.reference, nativeExecutionInput: {
schema: kind === "muse" ? "paperclip.native-execution-input.v8" : "paperclip.native-execution-input.v6",
binding, workspace: { access: "none", cwd: null }, provider: { kind, bridgeRevision: "muse-v1" } } };
await db.update(heartbeatRuns).set({ driverKind: kind === "muse" ? "muse_external" : "grok_bot_mcp",
runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId));
expect(await nativeWorkspaceExportPendingForLease(db, f.lease)).toBe(false);
expect(await nativeWorkspaceExportPendingForResource(db, f.lease, f.nextRunId)).toBe(false);
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...profile, nativeExecutionInput: {
...profile.nativeExecutionInput, binding: { ...binding, runId: randomUUID() } } } }).where(eq(heartbeatRuns.id, f.runId));
expect(await nativeWorkspaceExportPendingForResource(db, f.lease, f.nextRunId)).toBe(true);
});
it("does not take over a source while an ordinary provider resume is awaiting its database handoff", async () => {
const f = await seed(true), requestId = randomUUID(), actionId = randomUUID();
await db.update(heartbeatRuns).set({ runnerProfileJson: {} }).where(eq(heartbeatRuns.id, f.runId));
let entered!: () => void, release!: () => void;
const started = new Promise<void>(resolve => { entered = resolve; });
const pending = new Promise<void>(resolve => { release = resolve; });
f.call.mockImplementation(async (_id: string, method: string) => {
if (method !== "environmentResumeLease") throw new Error(method);
entered(); await pending;
return { providerLeaseId: f.lease.providerLeaseId!, metadata: { remoteCwd: "/work" } };
});
const ordinary = f.runtime.acquireRunLease({ ...f.input, heartbeatRunId: f.nextRunId });
try {
await started;
const failure = failSystemRecovery(reserveSystemRecoveryAttempt(createSystemRecovery(f.runId, requestId))!,
new SystemRecoveryFailure("source_binding_invalid", "service_required"));
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: f.reference },
resultJson: { workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: {
schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [f.lease.id] } } }).where(eq(heartbeatRuns.id, f.runId));
await db.insert(nativeRunFinalizations).values({ companyId: f.input.companyId, issueId: f.input.issueId,
runId: f.runId, phase: "terminal_failure" });
await db.insert(issueRecoveryActions).values({ id: actionId, companyId: f.input.companyId, sourceIssueId: f.input.issueId,
kind: "active_run_watchdog", ownerType: "system", cause: "native_workspace_sync_out_retry_exhausted", fingerprint: f.runId,
nextAction: "Preserve the source", evidence: { runId: f.runId, systemRecovery: failure, workspaceRecovery: {
schema: "paperclip.workspace-recovery.v1", runId: f.runId, status: "blocked", stage: "verifying_stop",
attempt: 1, requestIds: [requestId] } } });
await db.insert(agentWakeupRequests).values({ id: requestId, companyId: f.input.companyId, agentId: f.input.agentId,
source: "on_demand", reason: "task_retry", status: "deferred_issue_execution", requestedByActorType: "user", requestedByActorId: "board",
payload: { issueId: f.input.issueId, taskRetry: { schema: "paperclip.task-retry.v1", generationId: requestId,
sourceRunId: f.runId, actorId: "board", actorSource: "local_implicit", strategy: "evaluate" } } });
await recoverRetainedWorkspace({ db, companyId: f.input.companyId, issueId: f.input.issueId,
runId: f.runId, actionId, environmentRuntime: f.runtime });
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, actionId));
expect(action.evidence.systemRecovery).toEqual(failure);
expect(action.evidence.nativeSourceRetention).toBeUndefined();
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0].leasePolicy).toBe("reuse_by_environment");
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentResumeLease"]);
} finally { release(); await ordinary; }
});
// Regression: a run resumed a warm sandbox and was interrupted before provider
// work. The restore hold stopped the sandbox and changed its lease to
// retain_on_failure. Recovery saved the files, yet the next run got a created
// lease (and a full workspace realize) because nothing restored reuse.
async function settleRestoreHold(fault: "none" | "unsettled" | "dirty" | "unstopped") {
const f = await seed(true, false), actionId = randomUUID(), operationId = randomUUID();
// The interrupted run's real terminal hold over its active warm lease.
const [interrupted] = await db.update(heartbeatRuns).set({ status: "interrupted", finishedAt: new Date(),
resultJson: { workspaceRestoreFailure: "restore_failed" } }).where(eq(heartbeatRuns.id, f.runId)).returning();
const [source] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
expect(await db.transaction(tx => preserveLegacyWorkspaceRestoreSources(tx as unknown as typeof db, interrupted, source)))
.toEqual([f.lease.id]);
const [stopping] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
expect(stopping).toMatchObject({ leasePolicy: "retain_on_failure", status: "pending_cleanup" });
if (fault !== "unstopped") expect(await settleStopOnlyCleanup(db, stopping, { attemptId: String(stopping.metadata!.pendingCleanupAttemptId),
receipt: { providerLeaseId: f.lease.providerLeaseId, state: "stopped" } })).not.toBeNull();
// The recovery's own receipts: files saved from this allocation, and its finalize.
const system = { ...createSystemRecovery(f.runId), state: fault === "unsettled" ? "running" : "complete",
phase: "reconcile_effects", preservationOperationIds: [operationId] };
await db.insert(issueRecoveryActions).values({ id: actionId, companyId: f.input.companyId, sourceIssueId: f.input.issueId,
kind: "active_run_watchdog", ownerType: "system", cause: "native_continuation_requires_reconciliation", fingerprint: f.runId,
nextAction: "Recover saved work", evidence: { runId: f.runId, workspaceRestoreFailure: "restore_failed",
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [f.lease.id] }, systemRecovery: system,
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: f.runId,
status: fault === "unsettled" ? "recovering" : "recovered", stage: "saving_receipts", attempt: 1, requestIds: [] } } });
await db.insert(workspaceOperations).values([{ id: operationId, companyId: f.input.companyId, issueId: f.input.issueId,
heartbeatRunId: f.runId, phase: "workspace_recovery", status: "succeeded", finishedAt: new Date(),
metadata: { recoveryActionId: actionId, leaseId: f.lease.id, providerLeaseId: f.lease.providerLeaseId } },
{ companyId: f.input.companyId, issueId: f.input.issueId, heartbeatRunId: f.runId, phase: "workspace_finalize",
status: "succeeded", finishedAt: new Date(), metadata: { owningService: "workspace_recovery", preservationOperationIds: [operationId] } }]);
// An unsaved writer still bound to the stopped sandbox keeps it held.
if (fault === "dirty") await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: f.reference } })
.where(eq(heartbeatRuns.id, f.runId));
const restored = await settleRecoveredWorkspaceRestoreHolds(db, { companyId: f.input.companyId, issueId: f.input.issueId,
runId: f.runId, actionId });
f.call.mockClear();
return { ...f, restored };
}
it("resumes the warm sandbox on the same provider lease after a restore hold settles", async () => {
const f = await settleRestoreHold("none");
const next = await f.runtime.acquireRunLease({ ...f.input, heartbeatRunId: f.nextRunId });
expect(next.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "resumed" });
expect(next.lease).toMatchObject({ heartbeatRunId: f.nextRunId, providerLeaseId: f.lease.providerLeaseId,
leasePolicy: "reuse_by_environment" });
expect(f.restored).toEqual([f.lease.id]);
expect(f.call.mock.calls.map(call => call[1])).toEqual(["environmentResumeLease"]);
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0])
.toMatchObject({ heartbeatRunId: f.runId, status: "expired", leasePolicy: "reuse_by_environment" });
});
it.each(["unsettled", "dirty", "unstopped"] as const)("keeps a %s restore hold out of warm reuse", async fault => {
const f = await settleRestoreHold(fault);
expect(f.restored).toEqual([]);
const [held] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
expect(held.leasePolicy).toBe("retain_on_failure");
expect(held.metadata?.workspaceRestoreRecovery).toMatchObject({ runId: f.runId, originalLeasePolicy: "reuse_by_environment" });
if (fault !== "unstopped") await expectFreshAllocation(f);
});
});