Files
PaperClipAI/server/src/services/heartbeat/recovery.test.ts
T
DottaandClaude Opus 5.5 12ac7cfd9f perf(recovery): read only the run keys that active execution recovery checks use (#21602)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Live recovery checks each running run when recovery work wakes it.
The check is `recoverActiveExecutions` in
`server/src/services/heartbeat/recovery.ts`.
> - Each check read the full `heartbeat_runs` row and the agent's
`adapter_config`. That includes `context_snapshot` and
`runner_profile_json`, which are large JSON columns.
> - Most checks only defer the run to its lease deadline. They use a few
JSON keys and never use the rest of the row.
> - Recovery wakes come from runner checkpoints, run events and company
work signals. Each running run is checked about 4.5 times per minute
under that load, so this read grows with the number of running runs.
> - This pull request reads only the JSON keys that the check gates use.
A run that recovery settles as lost reads its full context, runner
profile and adapter config before the settle.
> - The benefit is about 97% fewer bytes per check, with the same
recovery decisions.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The read that live recovery does for each running run
(`recoverActiveExecutions`).

**Subsystem affected**
Server: heartbeat run recovery
(`server/src/services/heartbeat/recovery.ts`).

**Current behavior**
Each check selects the whole `heartbeat_runs` row plus
`agents.adapter_config`. With a realistic row (`context_snapshot` 115 KB
of text, 23 KB stored; `runner_profile_json` 48 KB of text, 10 KB
stored), one check receives 175 KB from Postgres. A server with 20
running runs did 90 checks per minute, which is 15.7 MB per minute from
this one statement. That was 99% of all bytes in the measured window.

**Proposed behavior**
The check selects all scalar run columns, a small JSON object with only
the keys its gates read, and no output excerpts or adapter config. The
process-lost settle path reads the full `context_snapshot`,
`runner_profile_json` and `adapter_config` for that one run before it
decides on a retry.

**Reason and benefit**
The staging server spends 21-24% of main-thread CPU on postgres.js
result parsing at about 26 KB per statement. Whole-row reads in
recurring loops cause that overfetch. This change removes about 171 KB
from each check.

**Breaking changes**
None. No schema, API or contract change.

**Additional context**
I examined the `+1 s` re-arm (when revocation finds a child process that
the inventory snapshot did not have). I kept it. It only fires when the
revocation sees a changed row. The revocation also writes a new
60-second lease for a random boot ID, so the next pass defers on that
live lease. It does not loop on unchanged state. In the 60-second
measurement it did not fire.

## What Changed

- `recoverActiveExecutions` selects `activeRecoveryRunColumns`. This is
all `heartbeat_runs` columns, except that `context_snapshot` holds only
`wakeReason`, `issueId` and `nativeQuestionCancellation`,
`runner_profile_json` holds only `nativeStartupContext`,
`nativeWorkspaceSync` and `remoteWorkspace`, and the stdout/stderr
excerpts are null. It does not select `agents.adapter_config`.
- New `readActiveRecoverySettlement` reads the full `context_snapshot`,
`runner_profile_json` and `adapter_config` of one run. The process-lost
settle uses that data for the conversation-adapter check, the retry
budget, the process-loss diagnostic, the stop metadata and the
first-heartbeat flag.
- New tests in `server/src/services/heartbeat/recovery.test.ts`:
- A regression test counts the bytes Postgres sends during one deferred
check through a TCP proxy. It must be below 32 KB.
- A settle test (two cases) gives a lost run its retry budget,
first-heartbeat flag and conversation adapter only in the JSON keys
outside the narrow read. The test checks that the settle still uses
them.
- A test checks that a run with `nativeQuestionCancellation` in its
context is still skipped.

## Verification

- Red/green: the byte test fails on `origin/master` (`expected 454426 to
be less than 32768`) and passes with this change. The other three new
tests pass both before and after.
- Focused local runs pass: `recovery.test.ts`,
`live-recovery-work.test.ts`, `native-startup-cleanup-recovery.test.ts`,
`ownership-reconciliation-report.test.ts` (4 files, 255 tests).
- `pnpm verify` selected 2,715 of 3,541 files (conservative retention).
On this machine 1,689 files passed and 32 failed. The failures were
`afterAll` hook timeouts (30 s) and unrelated tool-access, email and
runner artifact suites under local load. The recovery suites above pass
when run alone. Required CI gives the full coverage.
- Before/after measurement (local embedded Postgres, a wire proxy that
assigns each server response to its statement, the real live-recovery
scheduler, 60 s window, 20 running runs: 10 native runs held by a live
controller and 10 legacy runs with a live peer lease; a run recovery
signal every 5 s for each run, a company work signal every 10 s, legacy
lease renewal every 20 s):

| | Before | After |
| --- | --- | --- |
| Checks per minute | 90 | 90 |
| Bytes per check (inventory statement) | 174,954 | 4,470 |
| Inventory bytes per minute | 15,745,819 | 402,298 |
| All bytes per minute in the window | 15,848,137 | 504,640 |

## Risks

- Low risk. The narrow read keeps the keys that the gates read. The
settle path reads the full row data before it uses it.
- The settle path now reads `context_snapshot`, `runner_profile_json`
and `adapter_config` after the lease revocation, not at the start of the
check. The values can be some milliseconds newer. The revocation already
uses the committed row state, so this agrees with the rest of the
settle.
- The settle path has one more query. It only runs when a run is settled
as lost.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as an agent in
Claude Code with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:58:47 -06:00

1901 lines
125 KiB
TypeScript

import { retireInheritedRemoteRunnerIdentity } from "../native-runtime/native-inherited-runner-retry.js";
import { randomUUID } from "node:crypto";
import { spawn } from "node:child_process";
import { once } from "node:events";
import { connect, createServer, type AddressInfo, type Socket } from "node:net";
import type { Db } from "@paperclipai/db";
import { activityLog, agents, applyPendingMigrations, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environments, environmentLeases, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunnerCheckpoints } from "@paperclipai/db";
import { eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { classifyEnvironmentCapabilities, environmentRuntimeService } from "../environment-runtime.js";
import { environmentService } from "../environments.js";
import { createHeartbeatRecovery, type HeartbeatRecoveryDependencies } from "./recovery.js";
import { createHeartbeatRunState } from "./run-state.js";
import type { HotRestartIntent } from "../hot-restart.js";
import type { NativeRestartRecoveryClaim, NativeRestartRecoveryDisposition } from "../native-runtime/index.js";
import { NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE } from "../native-runtime/native-runner-ownership.js";
import { claimNativeRestartRecoveries, currentNativeControllerIdentity, NATIVE_RESTART_RECOVERY_FAILED } from "../native-runtime/native-restart-recovery.js";
import { renewNativeExecutionLeasesNow, startNativeSessionExecutionLeaseRenewal } from "../native-runtime/native-execution-lease.js";
import { createRunLogWriterAuthority } from "./run-log-authority.js";
import { allowLegacyShutdownWorkspaceCleanup, beginLegacyShutdownWorkspaceSettlement, finishLegacyShutdownWorkspaceSettlement, maintainLegacyShutdownWorkspaceSettlement, legacyShutdownWorkspaceResourceProtected, sweepLegacyShutdownWorkspaceSettlements } from "../legacy-shutdown-workspace-settlement.js";
import { adapterExecutionControls, createAdapterExecutionControl } from "../adapter-execution-control.js";
import { currentControllerProcessStamp, currentProcessSpaceHash, currentProcessStartedAtMs } from "../controller-process-identity.js";
import { createOwnershipReportState, runOwnershipReconciliationPass } from "../recovery/ownership-invariant-report.js";
import { OWNABLE_KINDS } from "../recovery/ownership-registry.js";
import { logger } from "../../middleware/logger.js";
import { listExpiredNativeRunnerAuthorities } from "../native-runtime/native-runner-checkpoint.js";
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../delivery-work-notifications.js";
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
import { hasRemoteTerminationReceipt, remoteTerminationReceipt } from "../remote-execution-termination.js";
import { persistHeartbeatRunProcessMetadata } from "./run-lifecycle.js";
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
import { hasNativeLocalProcessStop, PROCESS_START_REQUESTED, releaseSettledNativeRunProcess } from "../native-local-process-stop.js";
import { legacyControllerBootId } from "../legacy-controller-lease.js";
import { runBoundedShutdownStep } from "../../shutdown.js";
type Run = typeof heartbeatRuns.$inferSelect;
const native = vi.hoisted(() => ({
closeIdle: vi.fn(async () => ({ closed: 0, failed: 0 })),
detach: vi.fn(async () => 0),
finalizations: vi.fn(async (_db: Db, _ids?: string[], _options?: unknown) => undefined),
cleanups: vi.fn<() => Promise<void>>(async () => undefined),
drainMaintenance: vi.fn<() => Promise<void>>(async () => undefined),
claims: vi.fn<typeof claimNativeRestartRecoveries>(async () => []),
}));
const hotRestart = vi.hoisted(() => ({
read: vi.fn<() => Promise<HotRestartIntent | null>>(async () => null),
write: vi.fn(async () => undefined),
}));
vi.mock("../hot-restart.js", async (importOriginal) => ({
...await importOriginal<typeof import("../hot-restart.js")>(),
readHotRestartIntent: hotRestart.read,
writeHotRestartShutdownSnapshot: hotRestart.write,
}));
vi.mock("../native-runtime/index.js", async (importOriginal) => ({
...await importOriginal<typeof import("../native-runtime/index.js")>(),
closeIdleWarmNativeSessionsForRestart: native.closeIdle,
detachNativeSessionsForRestart: native.detach,
reconcileNativeFinalizations: native.finalizations,
claimNativeRestartRecoveries: native.claims,
}));
vi.mock("../native-runtime/retained-native-session-cleanup.js", async (importOriginal) => ({
...await importOriginal<typeof import("../native-runtime/retained-native-session-cleanup.js")>(),
reconcileRetainedNativeSessionCleanups: native.cleanups,
}));
function callbacks(db: Db) {
return {
enterShutdown: vi.fn(),
getRun: vi.fn<HeartbeatRecoveryDependencies["getRun"]>(async () => null),
getRunWorkspaceSettlement: vi.fn(createHeartbeatRunState(db).getRunWorkspaceSettlement),
getAgent: vi.fn<HeartbeatRecoveryDependencies["getAgent"]>(async () => null),
appendRunEvent: vi.fn<HeartbeatRecoveryDependencies["appendRunEvent"]>(async () => undefined),
setRunStatusIfRunning: vi.fn<HeartbeatRecoveryDependencies["setRunStatusIfRunning"]>(async () => ({ updated: false, run: null })),
setRunStatusFromLive: vi.fn<HeartbeatRecoveryDependencies["setRunStatusFromLive"]>(async () => ({ updated: false, run: null })),
setRunStatus: vi.fn<HeartbeatRecoveryDependencies["setRunStatus"]>(async () => null),
setWakeupStatus: vi.fn(async () => undefined),
releaseIssueExecutionAndPromote: vi.fn(async () => undefined),
finalizeAgentStatus: vi.fn(async () => undefined),
environmentRuntime: environmentRuntimeService(db),
environmentsSvc: environmentService(db),
instructionCopies: {
recoverStopped: vi.fn(async () => 0),
recoverCaptured: vi.fn(async () => 0),
},
runtimeEnv: {},
activeRunExecutions: new Set<string>(),
activeRunExecutionPromises: new Set<Promise<void>>(),
scheduleBoundedRetryForRun: vi.fn<HeartbeatRecoveryDependencies["scheduleBoundedRetryForRun"]>(async () => ({ outcome: "retry_exhausted", retryReason: "transient_failure", attempt: 1, maxAttempts: 0 })),
scheduleInteractionContinuationInfrastructureRetryIfEligible: vi.fn<HeartbeatRecoveryDependencies["scheduleInteractionContinuationInfrastructureRetryIfEligible"]>(async () => null),
timerClaimWasFirstHeartbeat: vi.fn(() => undefined),
executeRun: vi.fn<HeartbeatRecoveryDependencies["executeRun"]>(async () => undefined),
scheduleNativeSessionResumeDispatch: vi.fn(),
cancelHeartbeatNativeRun: vi.fn(async () => undefined),
terminateHeartbeatRunProcess: vi.fn(async () => undefined),
mergeRunStopMetadataForAgent: vi.fn<HeartbeatRecoveryDependencies["mergeRunStopMetadataForAgent"]>((_agent, _outcome, options) => options?.resultJson ?? null),
classifyAndPersistRunLiveness: vi.fn<HeartbeatRecoveryDependencies["classifyAndPersistRunLiveness"]>(async run => run),
releaseEnvironmentLeasesForRun: vi.fn<HeartbeatRecoveryDependencies["releaseEnvironmentLeasesForRun"]>(async () => undefined),
acknowledgeRemoteStop: vi.fn(async () => undefined),
resumeRemoteStopComments: vi.fn(async () => undefined),
dispatchPendingNativeStatusWakeups: vi.fn(async () => undefined),
cancelRunInternal: vi.fn(async () => undefined),
startNextQueuedRunForAgent: vi.fn(async () => undefined),
} satisfies HeartbeatRecoveryDependencies;
}
function guardedDatabase() {
const access = vi.fn(() => { throw new Error("Unexpected recovery database access"); });
return { db: new Proxy({}, { get: access }) as Db, access };
}
it("does not query checkpoint recovery without an expired active run", async () => {
const { db, access } = guardedDatabase();
expect(await listExpiredNativeRunnerAuthorities(db, new Date(), [])).toEqual([]);
expect(access).not.toHaveBeenCalled();
});
beforeEach(() => {
vi.clearAllMocks();
hotRestart.read.mockResolvedValue(null);
native.closeIdle.mockResolvedValue({ closed: 0, failed: 0 });
native.finalizations.mockResolvedValue(undefined);
native.cleanups.mockResolvedValue(undefined);
native.claims.mockResolvedValue([]);
});
describe("heartbeat recovery module callbacks", () => {
it("constructs without database access, shutdown, or background work", () => {
const database = guardedDatabase();
const deps = callbacks(database.db);
createHeartbeatRecovery(database.db, deps);
expect(database.access).not.toHaveBeenCalled();
for (const callback of Object.values(deps)) if (typeof callback === "function") expect(callback).not.toHaveBeenCalled();
for (const callback of Object.values(native)) expect(callback).not.toHaveBeenCalled();
});
it.each(["not_requested", "read_error"] as const)("enters shutdown before checkpointing for %s", async (mode) => {
const database = guardedDatabase();
const deps = callbacks(database.db);
if (mode === "read_error") hotRestart.read.mockRejectedValueOnce(new Error("unreadable intent"));
const recovery = createHeartbeatRecovery(database.db, deps);
expect(await recovery.prepareHotRestartShutdown("SIGTERM")).toEqual({ mode, skipDrain: false, activeRunIds: [] });
expect(native.closeIdle).toHaveBeenCalledAfter(deps.enterShutdown);
expect(hotRestart.read).toHaveBeenCalledAfter(native.closeIdle);
expect(database.access).not.toHaveBeenCalled();
});
it("does not adopt or query runs without a restart intent", async () => {
const database = guardedDatabase();
expect(await createHeartbeatRecovery(database.db, callbacks(database.db)).reconcileHotRestartAdoption()).toEqual({
mode: "not_requested", adoptedRunIds: [], finalizedWhileDownRunIds: [], lostRunIds: [], skippedRunIds: [],
});
expect(database.access).not.toHaveBeenCalled();
});
it("does not broaden an empty selective shutdown drain to every run", async () => {
const database = guardedDatabase();
const deps = callbacks(database.db);
expect(await createHeartbeatRecovery(database.db, deps).drainRunningRunsForShutdown("SIGINT", new Date(), [])).toEqual({
interrupted: 0, interruptedRunIds: [], retryRunIds: [], restartSuspendedRunIds: [],
});
expect(database.access).not.toHaveBeenCalled();
expect(deps.cancelHeartbeatNativeRun).not.toHaveBeenCalled();
expect(deps.terminateHeartbeatRunProcess).not.toHaveBeenCalled();
});
it.each(["running", "failed"] as const)("keeps a %s ownership hold when a status race wins", async (status) => {
const database = guardedDatabase();
const deps = callbacks(database.db);
const run = { id: "run", runtimeMode: "native", status } as Run;
const winner = { ...run, status: "cancelled" };
deps.setRunStatusFromLive.mockResolvedValue({ updated: false, run: winner });
expect(await createHeartbeatRecovery(database.db, deps).markNativeOwnershipUnverified(run, { reason: "live_process_identifier" })).toBe(winner);
expect(deps.setRunStatusFromLive).toHaveBeenCalledWith("run", status, [status], expect.objectContaining({ errorCode: NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE }));
expect(deps.appendRunEvent).not.toHaveBeenCalled();
expect(database.access).not.toHaveBeenCalled();
});
it.each(["adopted_runner_authentication_timeout"] as const)("persists a %s ownership hold before its event without terminating or retrying", async reason => {
const database = guardedDatabase();
const deps = callbacks(database.db);
const run = { id: "run", runtimeMode: "native", status: "running" } as Run;
const blocked = { ...run, nativePhase: "terminal_failure", errorCode: NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE };
deps.setRunStatusFromLive.mockResolvedValue({ updated: true, run: blocked });
expect(await createHeartbeatRecovery(database.db, deps).markNativeOwnershipUnverified(run, { reason })).toBe(blocked);
expect(deps.setRunStatusFromLive).toHaveBeenCalledWith("run", "running", ["running"], expect.objectContaining({ nativePhase: "terminal_failure" }));
expect(deps.appendRunEvent).toHaveBeenCalledAfter(deps.setRunStatusFromLive);
expect(deps.cancelHeartbeatNativeRun).not.toHaveBeenCalled();
expect(deps.terminateHeartbeatRunProcess).not.toHaveBeenCalled();
expect(deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
expect(database.access).not.toHaveBeenCalled();
});
});
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_RECOVERY_TEST_DATABASE_URL");
const support = externalUrl ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
const describePostgres = support.supported ? describe : describe.skip;
if (!support.supported) console.warn(`Skipping recovery module database tests: ${support.reason}`);
describePostgres("heartbeat recovery module ownership", () => {
let db: Db;
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
beforeAll(async () => {
if (externalUrl) await applyPendingMigrations(externalUrl);
else database = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-recovery-module-");
db = createDb(externalUrl ?? database.connectionString);
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
async function seedRun() {
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), issueId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Recovery", issuePrefix: `T${companyId.slice(0, 6)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Recovery", role: "engineer", adapterType: "process" });
await db.insert(issues).values({ id: issueId, companyId, title: "Recover this run", status: "in_progress", assigneeAgentId: agentId });
return (await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, invocationSource: "automation", status: "running", runtimeMode: "legacy" }).returning())[0];
}
it.each(["unlaunched", "dispatched", "reassigned", "ambiguous"] as const)("claims the same expired portable preparation only with its original task lock (%s)", async state => {
const run = await seedRun(), deps = callbacks(db);
const environmentId = randomUUID(), leaseId = randomUUID();
const oldBootId = randomUUID();
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
await db.update(heartbeatRuns).set({ scopeKind: "issue", controllerBootId: oldBootId, controllerLeaseExpiresAt: new Date(0),
executionStage: "preparing" }).where(eq(heartbeatRuns.id, run.id));
if (state !== "reassigned") await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, run.issueId!));
await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "plugin", config: {} });
await db.insert(environmentLeases).values({ id: leaseId, companyId: run.companyId, environmentId, heartbeatRunId: run.id,
issueId: run.issueId, provider: "plugin:fixture:tasks", providerLeaseId: "portable-task", status: "active",
metadata: { agentId: run.agentId, driver: "plugin", runnerTaskLease: true, portableTaskPreviousLeaseId: randomUUID(),
...(state === "dispatched" ? { runnerTaskDispatched: true } : {}) } });
if (state === "ambiguous") await db.insert(environmentLeases).values({ companyId: run.companyId, environmentId,
heartbeatRunId: run.id, issueId: run.issueId, provider: "plugin:fixture:tasks", providerLeaseId: "competing-task", status: "active" });
const scope = { companyId: run.companyId, runId: run.id };
const recovery = createHeartbeatRecovery(db, deps);
await Promise.all([recovery.recoverActiveExecutions(scope), recovery.recoverActiveExecutions(scope)]);
if (state === "unlaunched") {
expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(run.id, { portablePreparationLeaseId: leaseId });
expect(deps.setRunStatusFromLive).not.toHaveBeenCalled();
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
expect(current).toMatchObject({ status: "running", controllerBootId: legacyControllerBootId, executionStage: "preparing" });
expect(current.controllerLeaseExpiresAt!.getTime()).toBeGreaterThan(Date.now());
expect((await db.select().from(issues).where(eq(issues.id, run.issueId!)))[0].executionRunId).toBe(run.id);
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId)))[0]).toMatchObject({
status: "active", providerLeaseId: "portable-task", heartbeatRunId: run.id,
});
} else {
expect(deps.executeRun).not.toHaveBeenCalled();
expect(deps.setRunStatusFromLive).toHaveBeenCalledOnce();
}
});
it.each([false, true])("delivers scoped recovery to a preparation without a runner PID or checkpoint (locally tracked=%s)", async tracked => {
const run = await seedRun(), deps = callbacks(db);
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId,
nativePhase: "observed", runnerProfileJson: { nativePreparationFencingVersion: 1 },
}).where(eq(heartbeatRuns.id, run.id));
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!,
phase: "observed", leaseOwner: "old-preparation", leaseExpiresAt: new Date(0), controllerHostHash: "operator_attested" });
const claim: NativeRestartRecoveryClaim = { kind: "bootstrap_incomplete", runId: run.id, leaseOwner: "recovered-owner",
controllerGeneration: 2, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null };
native.claims.mockResolvedValue([claim]);
if (tracked) deps.activeRunExecutions.add(run.id);
try {
await createHeartbeatRecovery(db, deps).recoverActiveExecutions({ companyId: run.companyId, runId: run.id });
if (tracked) {
expect(native.claims).not.toHaveBeenCalled();
expect(deps.executeRun).not.toHaveBeenCalled();
} else {
expect(native.claims).toHaveBeenCalledWith(expect.objectContaining({ runIds: [run.id] }));
expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(run.id, { nativeLeaseOwner: claim.leaseOwner, nativeRestartRecovery: claim });
}
expect(deps.setRunStatusFromLive).not.toHaveBeenCalled();
expect(deps.terminateHeartbeatRunProcess).not.toHaveBeenCalled();
} finally {
await Promise.all([...deps.activeRunExecutionPromises]);
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
}
});
it("routes an expired pre-capability startup to cleanup before generic orphan ownership holds", async () => {
const run = await seedRun(), leaseId = randomUUID(), environmentId = randomUUID(), leaseOwner = randomUUID();
const deadlineAtMs = Date.now() - 1_000, contextId = randomUUID();
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, run.issueId!));
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId, nativePhase: "observed",
runnerInstanceId: randomUUID(), nativeSessionId: randomUUID(), runnerProfileJson: { nativeStartupContext: {
schema: "paperclip.native-startup-context.v1", companyId: run.companyId, agentId: run.agentId, runId: run.id,
contextId, deadlineAtMs, leaseId, leaseOwner, controllerGeneration: 1, attempt: 1 } } }).where(eq(heartbeatRuns.id, run.id));
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!,
phase: "observed", attempt: 1, leaseOwner, controllerGeneration: 1, leaseExpiresAt: new Date(Date.now() - 1_000),
controllerBootId: randomUUID(), controllerPid: process.pid, controllerProcessStartedAt: new Date() });
await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "plugin", config: {} });
await db.insert(environmentLeases).values({ id: leaseId, companyId: run.companyId, environmentId, heartbeatRunId: run.id,
issueId: run.issueId, provider: "plugin", providerLeaseId: "early-task", status: "active", metadata: { runnerTaskLease: true } });
const deps = callbacks(db);
deps.getRun.mockImplementation(async id => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0] ?? null);
native.claims.mockImplementationOnce(claimNativeRestartRecoveries);
try {
await createHeartbeatRecovery(db, deps).recoverActiveExecutions({ runId: run.id });
expect(native.claims).toHaveBeenCalledWith(expect.objectContaining({ runIds: [run.id] }));
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
expect(current).toMatchObject({ status: "running", errorCode: "native_startup_cleanup_pending",
resultJson: { nativeStartupCleanup: { contextId, state: "pending" } } });
expect(deps.executeRun).not.toHaveBeenCalled();
expect(deps.setRunStatusFromLive).not.toHaveBeenCalled();
} finally {
await db.delete(environmentLeases).where(eq(environmentLeases.id, leaseId));
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, run.id));
await db.delete(activityLog).where(eq(activityLog.runId, run.id));
await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, run.issueId!));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
}
});
async function byteCountingProxy(connectionString: string) {
const target = new URL(connectionString);
const sockets = new Set<Socket>();
let received = 0;
const server = createServer(client => {
const upstream = connect(Number(target.port || "5432"), target.hostname);
sockets.add(client).add(upstream);
upstream.on("data", chunk => { received += chunk.length; });
client.pipe(upstream);
upstream.pipe(client);
const close = () => { client.destroy(); upstream.destroy(); };
client.on("error", close);
upstream.on("error", close);
});
await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));
const proxied = new URL(connectionString);
proxied.hostname = "127.0.0.1";
proxied.port = String((server.address() as AddressInfo).port);
return {
url: proxied.toString(),
received: () => received,
async close() {
for (const socket of sockets) socket.destroy();
await new Promise(resolve => server.close(resolve));
},
};
}
async function seedLostLegacyRun(input: { contextSnapshot: Record<string, unknown>; runnerProfileJson: Record<string, unknown>;
adapterConfig: Record<string, unknown>; controllerLeaseExpiresAt: Date }) {
const run = await seedRun();
await db.update(agents).set({ adapterConfig: input.adapterConfig }).where(eq(agents.id, run.agentId));
const [updated] = await db.update(heartbeatRuns).set({ controllerBootId: randomUUID(),
controllerLeaseExpiresAt: input.controllerLeaseExpiresAt, executionStage: "dispatching",
contextSnapshot: input.contextSnapshot, runnerProfileJson: input.runnerProfileJson,
stdoutExcerpt: "stdout ".repeat(5_000), stderrExcerpt: "stderr ".repeat(5_000),
}).where(eq(heartbeatRuns.id, run.id)).returning();
return updated;
}
it("reads a bounded slice of a running run when active recovery only defers it", async () => {
const leaseExpiresAt = new Date(Date.now() + 60_000);
const run = await seedLostLegacyRun({
contextSnapshot: { issueId: randomUUID(), wakeReason: "issue_assigned", transcript: "context ".repeat(25_000) },
runnerProfileJson: { backend: "fixture", manifest: "profile ".repeat(12_000) },
adapterConfig: { instructions: "config ".repeat(12_000) },
controllerLeaseExpiresAt: leaseExpiresAt,
});
const proxy = await byteCountingProxy(externalUrl ?? database.connectionString);
const proxiedDb = createDb(proxy.url, { maxConnections: 1 });
try {
await proxiedDb.execute(sql`select 1`);
const deps = callbacks(proxiedDb);
const deadlines: number[] = [];
const before = proxy.received();
await createHeartbeatRecovery(proxiedDb, deps).recoverActiveExecutions({ companyId: run.companyId, runId: run.id,
onDeferredDeadline: at => deadlines.push(at) });
expect(proxy.received() - before).toBeLessThan(32_768);
expect(deadlines).toEqual([leaseExpiresAt.getTime()]);
expect(deps.setRunStatusFromLive).not.toHaveBeenCalled();
} finally {
await proxiedDb.$client.end();
await proxy.close();
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
}
});
it.each([
{ serverRestarts: 1, retries: true },
{ serverRestarts: 6, retries: false },
])("settles a lost run from its full context, runner profile and adapter config (server restarts $serverRestarts)", async ({ serverRestarts, retries }) => {
const adapterConfig = { model: "fixture", instructions: "config ".repeat(2_000) };
const run = await seedLostLegacyRun({
contextSnapshot: { issueId: randomUUID(), wakeReason: "issue_assigned", timerClaimWasFirstHeartbeat: true,
executionRetryAccounting: { version: 1, failureRetries: 0, maxTurnContinuations: 0, serverRestarts,
serverRestartWindowStartedAt: new Date().toISOString() },
transcript: "context ".repeat(2_000) },
runnerProfileJson: { adapterDispatch: { adapterType: "claude_local" }, manifest: "profile ".repeat(2_000) },
adapterConfig,
controllerLeaseExpiresAt: new Date(0),
});
const deps = {
...callbacks(db),
timerClaimWasFirstHeartbeat: vi.fn<HeartbeatRecoveryDependencies["timerClaimWasFirstHeartbeat"]>(source =>
source.contextSnapshot?.timerClaimWasFirstHeartbeat === true ? true : undefined),
};
deps.setRunStatusFromLive.mockImplementation(async (_id, status, _from, patch) =>
({ updated: true, run: { ...run, ...patch, status } as Run }));
try {
expect(await createHeartbeatRecovery(db, deps).recoverActiveExecutions({ companyId: run.companyId, runId: run.id },
{ staleThresholdMs: 0 })).toEqual([run.id]);
expect(deps.mergeRunStopMetadataForAgent).toHaveBeenCalledWith({ adapterType: "process", adapterConfig }, "failed",
expect.objectContaining({ conversationContinuationEligible: true }));
const [, , , patch] = deps.setRunStatusFromLive.mock.calls[0]!;
expect(patch?.error?.endsWith("; retrying once")).toBe(retries);
expect(deps.finalizeAgentStatus).toHaveBeenCalledWith(run.agentId, "failed", expect.any(String), { wasFirstHeartbeat: true });
} finally {
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, run.id));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
}
});
it("leaves a lost run that waits on a question cancellation to its own recovery", async () => {
const run = await seedLostLegacyRun({
contextSnapshot: { issueId: randomUUID(), nativeQuestionCancellation: { kind: "issue_terminal" }, transcript: "context ".repeat(2_000) },
runnerProfileJson: {},
adapterConfig: {},
controllerLeaseExpiresAt: new Date(0),
});
const deps = callbacks(db);
try {
expect(await createHeartbeatRecovery(db, deps).recoverActiveExecutions({ companyId: run.companyId, runId: run.id },
{ staleThresholdMs: 0 })).toEqual([]);
expect(deps.setRunStatusFromLive).not.toHaveBeenCalled();
} finally {
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
}
});
async function ownedProcess() {
const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)", `paperclip-recovery-${randomUUID()}`], {
stdio: "ignore", detached: true, env: { PATH: process.env.PATH },
});
await once(child, "spawn");
return {
pid: child.pid!,
async stop() {
if (child.exitCode !== null || child.signalCode !== null) return;
const exited = once(child, "exit");
child.kill("SIGTERM");
await exited;
},
alive() { return process.kill(child.pid!, 0); },
};
}
async function nativeProcessFixture(pid: number, targetKind: "local" | "remote", status: "running" | "failed") {
const run = await seedRun();
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId, status,
nativePhase: status === "failed" ? "retryable_failure" : "running",
errorCode: status === "failed" ? "native_session_interrupted" : null,
error: status === "failed" ? "native session timed out after 120000ms" : null,
finishedAt: status === "failed" ? new Date() : null,
}).where(eq(heartbeatRuns.id, run.id));
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!,
phase: status === "failed" ? "retryable_failure" : "running", nextAttemptAt: new Date(Date.now() - 1) });
await persistHeartbeatRunProcessMetadata(db, run.id, { pid, processGroupId: pid,
startedAt: new Date().toISOString(), targetKind });
const deps = callbacks(db);
deps.setRunStatusFromLive.mockImplementation(async (id, nextStatus, fromStatuses, patch) => {
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id));
if (!current || !fromStatuses.includes(current.status)) return { updated: false, run: current ?? null };
const [updated] = await db.update(heartbeatRuns).set({ ...patch, status: nextStatus })
.where(eq(heartbeatRuns.id, id)).returning();
return { updated: true, run: updated };
});
return {
run, deps,
current: async () => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0],
async cleanup() {
await Promise.all([...deps.activeRunExecutionPromises]);
await db.delete(environmentLeases).where(eq(environmentLeases.heartbeatRunId, run.id));
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, run.id));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
},
};
}
it.each(["failed", "running"] as const)("does not attribute a remote %s run's PID to a live host process", async status => {
const child = await ownedProcess();
const f = await nativeProcessFixture(child.pid, "remote", status);
try {
await db.update(agents).set({ adapterType: "process" }).where(eq(agents.id, f.run.agentId));
const before = await f.current();
expect(await createHeartbeatRecovery(db, f.deps).reapOrphanedRuns()).toEqual({ reaped: 0, runIds: [] });
expect(await f.current()).toMatchObject({ status, errorCode: before.errorCode, error: before.error,
processPid: child.pid, processGroupId: child.pid, processStartedAt: before.processStartedAt });
expect(f.deps.setRunStatusFromLive).not.toHaveBeenCalled();
expect(f.deps.executeRun).not.toHaveBeenCalled();
expect(f.deps.terminateHeartbeatRunProcess).not.toHaveBeenCalled();
expect(await hasNativeLocalProcessStop(db, f.run.companyId, f.run.id)).toBe(false);
expect(child.alive()).toBe(true);
} finally { try { await f.cleanup(); } finally { await child.stop(); } }
});
async function remoteRetryFixture(pid: number, target: "workspace" | "remote_workspace" | "plugin_task" = "workspace") {
const fixture = await nativeProcessFixture(pid, "remote", "failed");
const sessionId = randomUUID();
const [lease] = await db.insert(environmentLeases).values({ companyId: fixture.run.companyId,
heartbeatRunId: fixture.run.id, issueId: fixture.run.issueId, status: "active", provider: "daytona",
providerLeaseId: randomUUID(), metadata: target === "plugin_task" ? { runnerTaskLease: true } : {},
}).returning();
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, fixture.run.agentId));
await db.update(issues).set({ executionRunId: fixture.run.id }).where(eq(issues.id, fixture.run.issueId!));
await db.update(heartbeatRuns).set({ nativeSessionId: sessionId,
...(target === "remote_workspace" ? { processGroupId: null } : {}), runnerProfileJson: {
sessionCheckpoint: { identity: { runId: fixture.run.id, companyId: fixture.run.companyId,
issueId: fixture.run.issueId, agentId: fixture.run.agentId, sessionId }, providerSessionId: "existing-provider" },
...(target === "workspace" ? { nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v3", state: "prepared",
workspaceId: randomUUID(), leaseId: lease.id, providerLeaseId: lease.providerLeaseId, remoteCwd: "/work",
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null,
resourceDisposition: "keep_running" } } : {}),
...(target === "remote_workspace" ? { remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1",
workspaceId: randomUUID(), epoch: 3, leaseId: lease.id, providerLeaseId: lease.providerLeaseId,
remoteCwd: "/work", state: "prepared", snapshotId: randomUUID() } } : {}),
} }).where(eq(heartbeatRuns.id, fixture.run.id));
await db.insert(nativeRunnerCheckpoints).values({ runId: fixture.run.id, companyId: fixture.run.companyId,
issueId: fixture.run.issueId!, part: "opaque-checkpoint-record", material: { fixture: true } });
fixture.deps.getRun.mockImplementation(async id => (await db.select().from(heartbeatRuns)
.where(eq(heartbeatRuns.id, id)))[0] ?? null);
native.claims.mockImplementation(claimNativeRestartRecoveries);
return { ...fixture, lease };
}
it.each((["workspace", "remote_workspace"] as const).flatMap(target =>
(["failed", "running"] as const).map(status => ({ target, status })),
))("routes a due $target $status retry through verified recovery without local PID inspection", async ({ target, status }) => {
const child = await ownedProcess();
const f = await remoteRetryFixture(child.pid, target);
const processGroupId = target === "remote_workspace" ? null : child.pid;
const originalKill = process.kill.bind(process);
const kill = vi.spyOn(process, "kill").mockImplementation(originalKill);
try {
await db.update(heartbeatRuns).set({ status }).where(eq(heartbeatRuns.id, f.run.id));
await db.update(nativeRunFinalizations).set({ nextAttemptAt: null }).where(eq(nativeRunFinalizations.runId, f.run.id));
const recovery = createHeartbeatRecovery(db, f.deps);
await recovery.reapOrphanedRuns();
await Promise.all([...f.deps.activeRunExecutionPromises]);
expect(f.deps.executeRun).toHaveBeenCalledExactlyOnceWith(f.run.id, expect.objectContaining({
nativeRestartRecovery: expect.objectContaining({ kind: "reattach_remote_runner", runId: f.run.id,
remote: { providerLeaseId: f.lease.providerLeaseId, remoteCwd: "/work" } }),
}));
expect(native.claims).toHaveBeenCalledWith(expect.objectContaining({ runIds: [f.run.id] }));
expect(await f.current()).toMatchObject({ status: "running", processPid: child.pid, processGroupId });
expect(kill.mock.calls.filter(([pid]) => Math.abs(pid as number) === child.pid)).toEqual([]);
expect(await hasNativeLocalProcessStop(db, f.run.companyId, f.run.id)).toBe(false);
await recovery.reapOrphanedRuns();
expect(f.deps.executeRun).toHaveBeenCalledOnce();
expect(f.deps.terminateHeartbeatRunProcess).not.toHaveBeenCalled();
kill.mockRestore();
expect(child.alive()).toBe(true);
} finally { kill.mockRestore(); try { await f.cleanup(); } finally { await child.stop(); } }
});
// Staging shape: a chat continuation inherited its predecessor's saved
// provider session and warm runner identity, then failed while staging its
// provider pack, before it ever asked for a runner of its own. Its workspace
// reference is saved and its own sandbox lease is still active.
type InheritedCase = "staging" | "own_turn" | "own_launch" | "own_checkpoint" | "no_predecessor"
| "pending_side_effect" | "legacy_detail";
async function inheritedRunnerRetry(pid: number, condition: InheritedCase) {
const f = await remoteRetryFixture(pid, "remote_workspace");
const run = await f.current();
// The PID came from the predecessor, so this run recorded no process identity.
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.run.id));
if (condition !== "own_checkpoint") await db.delete(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.run.id));
const profile = run.runnerProfileJson ?? {};
const checkpoint = profile.sessionCheckpoint as Record<string, unknown>;
await db.update(agents).set({ status: "active" }).where(eq(agents.id, f.run.agentId));
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...profile, nativeExecutionInput: { fixture: true },
remoteWorkspace: { ...(profile.remoteWorkspace as Record<string, unknown>), state: "saved" },
sessionCheckpoint: { ...checkpoint, activeTurnId: null,
pendingRuntimeRequests: condition === "pending_side_effect" ? [{ requestId: "tool-1", kind: "tool_call" }] : [] },
} }).where(eq(heartbeatRuns.id, f.run.id));
await db.update(environmentLeases).set({ executionWorkspaceId: null }).where(eq(environmentLeases.id, f.lease.id));
await db.update(nativeRunFinalizations).set({ nextAttemptAt: null, failureDetail: {
message: "runner_remote_provider_archive_invalid", recoveryMode: "exact_checkpoint_resume", checkpointExists: true,
providerSessionEstablished: true, ...(condition === "legacy_detail" ? {} : { providerEventsExist: false }),
} }).where(eq(nativeRunFinalizations.runId, f.run.id));
const predecessorId = randomUUID();
if (condition !== "no_predecessor") await db.insert(heartbeatRuns).values({ id: predecessorId,
companyId: f.run.companyId, agentId: f.run.agentId, issueId: f.run.issueId, invocationSource: "automation",
status: "succeeded", runtimeMode: "native", nativeSessionId: run.nativeSessionId, createdAt: new Date(Date.now() - 600_000),
finishedAt: new Date(Date.now() - 300_000), processPid: run.processPid, processGroupId: run.processGroupId,
processStartedAt: run.processStartedAt });
if (condition === "own_turn") await appendHeartbeatRunEvent(db, { companyId: f.run.companyId, runId: f.run.id,
agentId: f.run.agentId, eventType: "turn.started", stream: "system", level: "info", message: "turn.started" });
if (condition === "own_launch") await appendHeartbeatRunEvent(db, { companyId: f.run.companyId, runId: f.run.id,
agentId: f.run.agentId, eventType: PROCESS_START_REQUESTED, stream: "system", level: "info" });
return { ...f, predecessorId, async cleanupAll() {
try { await f.cleanup(); } finally { await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, predecessorId)); }
} };
}
it("resumes a continuation that never launched its own runner in its retained sandbox, without reattaching", async () => {
const child = await ownedProcess();
const f = await inheritedRunnerRetry(child.pid, "staging");
try {
native.claims.mockClear();
await createHeartbeatRecovery(db, f.deps).resumeRetryableNativeProcessSessions({ runId: f.run.id });
await Promise.all([...f.deps.activeRunExecutionPromises]);
expect(native.claims.mock.calls.some(([input]) => input?.runIds?.includes(f.run.id))).toBe(false);
expect(f.deps.executeRun).toHaveBeenCalledExactlyOnceWith(f.run.id, { nativeLeaseOwner: expect.any(String) });
const current = await f.current();
expect(current).toMatchObject({ processPid: null, processGroupId: null, processStartedAt: null });
expect(current.runnerProfileJson?.nativeInheritedRunnerRetirement).toEqual({
schema: "paperclip.native-inherited-runner-retirement.v1", predecessorRunId: f.predecessorId,
retainedSandbox: { leaseId: f.lease.id, providerLeaseId: f.lease.providerLeaseId, remoteCwd: "/work" } });
const events = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.run.id));
expect(events.find(event => event.payload?.reason === "inherited_runner_never_launched")?.payload)
.toMatchObject({ predecessorRunId: f.predecessorId, retainedSandbox: true });
// The retained sandbox lease stays the run's only lease; nothing new was acquired.
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.heartbeatRunId, f.run.id)))
.toMatchObject([{ id: f.lease.id, status: "active" }]);
expect(f.deps.terminateHeartbeatRunProcess).not.toHaveBeenCalled();
expect(child.alive()).toBe(true);
} finally { try { await f.cleanupAll(); } finally { await child.stop(); } }
});
it("routes a run another sweep already retired to the same resume", async () => {
const child = await ownedProcess();
const f = await inheritedRunnerRetry(child.pid, "staging");
try {
expect(await retireInheritedRemoteRunnerIdentity(db, f.run.id)).toBe("retired");
expect(await retireInheritedRemoteRunnerIdentity(db, f.run.id)).toBe("retired");
const events = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.run.id));
expect(events.filter(event => event.payload?.reason === "inherited_runner_never_launched")).toHaveLength(1);
} finally { try { await f.cleanupAll(); } finally { await child.stop(); } }
});
it("defers instead of reattaching when the retirement cannot take its locks", async () => {
const child = await ownedProcess();
const f = await inheritedRunnerRetry(child.pid, "staging");
let release!: () => void;
const released = new Promise<void>(resolve => { release = resolve; });
let locked!: () => void;
const holding = new Promise<void>(resolve => { locked = resolve; });
const holder = db.transaction(async tx => {
await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, f.run.issueId!)).for("update");
locked();
await released;
});
try {
await holding;
native.claims.mockClear();
const deferred = vi.fn();
await createHeartbeatRecovery(db, f.deps).resumeRetryableNativeProcessSessions({ runId: f.run.id, onDeferredDeadline: deferred });
expect(deferred).toHaveBeenCalled();
expect(native.claims.mock.calls.some(([input]) => input?.runIds?.includes(f.run.id))).toBe(false);
expect(f.deps.executeRun).not.toHaveBeenCalled();
expect((await f.current()).processPid).toBe(child.pid);
} finally {
release(); await holder;
try { await f.cleanupAll(); } finally { await child.stop(); }
}
}, 30_000);
it("keeps the existing recovery path for a non-transient retirement error", async () => {
const failing = { transaction: async () => { throw new Error("unexpected retirement fault"); } } as unknown as Db;
expect(await retireInheritedRemoteRunnerIdentity(failing, randomUUID())).toBe("not_applicable");
const lockTimeout = Object.assign(new Error("Failed query"), { cause: Object.assign(new Error("lock timeout"), { code: "55P03" }) });
const contended = { transaction: async () => { throw lockTimeout; } } as unknown as Db;
expect(await retireInheritedRemoteRunnerIdentity(contended, randomUUID())).toBe("transient");
const statementTimeout = Object.assign(new Error("Failed query"), { cause: Object.assign(new Error("statement timeout"), { code: "57014" }) });
const slow = { transaction: async () => { throw statementTimeout; } } as unknown as Db;
expect(await retireInheritedRemoteRunnerIdentity(slow, randomUUID())).toBe("transient");
});
it.each(["own_turn", "own_launch", "own_checkpoint", "no_predecessor", "pending_side_effect", "legacy_detail"] as const)(
"keeps verified reattachment when the run's own runner may hold work or the proof is incomplete: %s", async condition => {
const child = await ownedProcess();
const f = await inheritedRunnerRetry(child.pid, condition);
try {
native.claims.mockClear();
await createHeartbeatRecovery(db, f.deps).resumeRetryableNativeProcessSessions({ runId: f.run.id });
await Promise.all([...f.deps.activeRunExecutionPromises]);
expect(native.claims.mock.calls.some(([input]) => input?.runIds?.includes(f.run.id))).toBe(true);
expect(f.deps.executeRun).not.toHaveBeenCalledWith(f.run.id, { nativeLeaseOwner: expect.any(String) });
const current = await f.current();
expect(current.processPid).toBe(child.pid);
expect(current.runnerProfileJson).not.toHaveProperty("nativeInheritedRunnerRetirement");
} finally { try { await f.cleanupAll(); } finally { await child.stop(); } }
},
);
it("routes a due plugin-task retry through its existing remote lease", async () => {
const child = await ownedProcess();
const f = await remoteRetryFixture(child.pid, "plugin_task");
try {
await createHeartbeatRecovery(db, f.deps).reapOrphanedRuns();
await Promise.all([...f.deps.activeRunExecutionPromises]);
expect(f.deps.executeRun).toHaveBeenCalledExactlyOnceWith(f.run.id, expect.objectContaining({
nativeRestartRecovery: expect.objectContaining({ kind: "reattach_remote_runner", runId: f.run.id,
remote: { kind: "plugin_task", leaseId: f.lease.id } }),
}));
expect(await f.current()).toMatchObject({ processPid: child.pid, processGroupId: child.pid });
expect(await hasNativeLocalProcessStop(db, f.run.companyId, f.run.id)).toBe(false);
expect(child.alive()).toBe(true);
} finally { try { await f.cleanup(); } finally { await child.stop(); } }
});
it.each((["workspace", "remote_workspace"] as const).flatMap(target =>
(["future_retry", "live_owner", "missing_checkpoint", "active_execution"] as const).map(condition => ({ target, condition })),
))(
"does not execute a $target retry without current authority: $condition", async ({ target, condition }) => {
const child = await ownedProcess();
const f = await remoteRetryFixture(child.pid, target);
const processGroupId = target === "remote_workspace" ? null : child.pid;
try {
if (condition === "future_retry") await db.update(nativeRunFinalizations)
.set({ nextAttemptAt: new Date(Date.now() + 60_000) }).where(eq(nativeRunFinalizations.runId, f.run.id));
if (condition === "live_owner") await db.update(nativeRunFinalizations)
.set({ leaseOwner: "other-controller", leaseExpiresAt: new Date(Date.now() + 60_000) })
.where(eq(nativeRunFinalizations.runId, f.run.id));
if (condition === "missing_checkpoint") {
const run = await f.current();
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run.runnerProfileJson, sessionCheckpoint: null } })
.where(eq(heartbeatRuns.id, f.run.id));
}
if (condition === "active_execution") f.deps.activeRunExecutions.add(f.run.id);
await createHeartbeatRecovery(db, f.deps).reapOrphanedRuns();
expect(f.deps.executeRun).not.toHaveBeenCalled();
expect(f.deps.terminateHeartbeatRunProcess).not.toHaveBeenCalled();
expect(await f.current()).toMatchObject({ processPid: child.pid, processGroupId });
if (condition === "future_retry") expect(native.claims).not.toHaveBeenCalled();
if (condition === "missing_checkpoint") {
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.run.id));
expect(coordinator).toMatchObject({ phase: "terminal_failure", recoveryState: "blocked",
failureCode: "native_restart_recovery_blocked" });
}
expect(child.alive()).toBe(true);
} finally { f.deps.activeRunExecutions.delete(f.run.id); try { await f.cleanup(); } finally { await child.stop(); } }
},
);
it.each(["unknown", "remote"] as const)("exposes expired observed %s ownership without attributing host liveness", async target => {
const child = await ownedProcess();
const f = await nativeProcessFixture(child.pid, "remote", "running");
const originalKill = process.kill.bind(process);
let restoreKill: (() => void) | undefined;
try {
if (target === "unknown") await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.run.id));
await db.update(heartbeatRuns).set({ nativePhase: null })
.where(eq(heartbeatRuns.id, f.run.id));
await db.update(nativeRunFinalizations).set({ phase: "observed", attempt: 2,
leaseOwner: "prior-native-owner", leaseExpiresAt: new Date(0), nextAttemptAt: null })
.where(eq(nativeRunFinalizations.runId, f.run.id));
const before = await f.current();
const coordinatorBefore = await db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, f.run.id));
const kill = vi.spyOn(process, "kill").mockImplementation(originalKill);
restoreKill = () => kill.mockRestore();
expect(await createHeartbeatRecovery(db, f.deps).reapOrphanedRuns()).toEqual({ reaped: 0, runIds: [] });
expect(kill.mock.calls.filter(([pid]) => Math.abs(pid as number) === child.pid)).toEqual([]);
expect(await f.current()).toMatchObject({ status: "running", nativePhase: null,
processPid: child.pid, processGroupId: child.pid, processStartedAt: before.processStartedAt,
errorCode: NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE });
expect(f.deps.appendRunEvent).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
payload: { reason: "observed_owner_unverified" },
}));
expect(await db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, f.run.id))).toEqual(coordinatorBefore);
expect(f.deps.executeRun).not.toHaveBeenCalled();
expect(f.deps.terminateHeartbeatRunProcess).not.toHaveBeenCalled();
kill.mockRestore(); restoreKill = undefined;
expect(child.alive()).toBe(true);
} finally { restoreKill?.(); try { await f.cleanup(); } finally { await child.stop(); } }
});
it.each(["failed", "running"] as const)("protects a verified local %s run's live process", async status => {
const child = await ownedProcess();
const f = await nativeProcessFixture(child.pid, "local", status);
try {
await db.update(agents).set({ adapterType: "paperclip_runner", adapterConfig: { driver: "daytona" } })
.where(eq(agents.id, f.run.agentId));
await createHeartbeatRecovery(db, f.deps).reapOrphanedRuns();
expect(await f.current()).toMatchObject({ status, processPid: child.pid, processGroupId: child.pid,
errorCode: NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE });
expect(f.deps.appendRunEvent).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
payload: { reason: "live_process_identifier", processPidAlive: true, processGroupAlive: true },
}));
expect(f.deps.executeRun).not.toHaveBeenCalled();
expect(f.deps.terminateHeartbeatRunProcess).not.toHaveBeenCalled();
expect(child.alive()).toBe(true);
} finally { try { await f.cleanup(); } finally { await child.stop(); } }
});
it.each(["missing", "new_launch", "wrong_birth", "foreign_host", "wrong_agent", "wrong_pid", "wrong_group", "native_source", "remote_workspace", "remote_lease"] as const)(
"holds a native PID without local authority: %s", async evidence => {
const child = await ownedProcess();
await child.stop();
const f = await nativeProcessFixture(child.pid, "local", "failed");
try {
if (evidence === "missing") await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.run.id));
if (evidence === "new_launch") await appendHeartbeatRunEvent(db, { companyId: f.run.companyId,
runId: f.run.id, agentId: f.run.agentId, eventType: PROCESS_START_REQUESTED });
if (["wrong_birth", "foreign_host", "wrong_agent", "wrong_pid", "wrong_group", "native_source"].includes(evidence)) {
const [event] = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.run.id));
const foreignAgentId = randomUUID();
if (evidence === "wrong_agent") await db.insert(agents).values({ id: foreignAgentId,
companyId: f.run.companyId, name: "Other process owner", role: "engineer", adapterType: "process" });
await db.update(heartbeatRunEvents).set({
...(evidence === "wrong_agent" ? { agentId: foreignAgentId } : {}),
...(evidence === "native_source" ? { sourceEventId: "provider-event" } : {}),
payload: { ...event.payload, ...(evidence === "wrong_birth"
? { processStartedAt: new Date(0).toISOString() } : evidence === "foreign_host" ? { hostname: "foreign-host" }
: evidence === "wrong_pid" ? { processPid: child.pid + 1 }
: evidence === "wrong_group" ? { processGroupId: child.pid + 1 } : {}) },
}).where(eq(heartbeatRunEvents.id, event.id));
}
if (evidence === "remote_workspace") await db.update(heartbeatRuns).set({
runnerProfileJson: { nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v3", state: "prepared",
workspaceId: randomUUID(), leaseId: randomUUID(), providerLeaseId: randomUUID(), remoteCwd: "/work",
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null, resourceDisposition: "destroy" } },
}).where(eq(heartbeatRuns.id, f.run.id));
if (evidence === "remote_lease") await db.insert(environmentLeases).values({ companyId: f.run.companyId,
heartbeatRunId: f.run.id, issueId: f.run.issueId, status: "active", provider: "daytona", providerLeaseId: randomUUID() });
const before = await f.current();
await createHeartbeatRecovery(db, f.deps).reapOrphanedRuns();
expect(await f.current()).toMatchObject({ processPid: child.pid, processGroupId: child.pid,
processStartedAt: before.processStartedAt, errorCode: "native_session_interrupted" });
expect(f.deps.executeRun).not.toHaveBeenCalled();
expect(f.deps.terminateHeartbeatRunProcess).not.toHaveBeenCalled();
expect(await hasNativeLocalProcessStop(db, f.run.companyId, f.run.id)).toBe(false);
} finally { await f.cleanup(); }
},
);
it("clears a verified stopped local retry identity and records its stop", async () => {
const child = await ownedProcess();
await child.stop();
const f = await nativeProcessFixture(child.pid, "local", "failed");
try {
await createHeartbeatRecovery(db, f.deps).reapOrphanedRuns();
expect(await f.current()).toMatchObject({ processPid: null, processGroupId: null, processStartedAt: null });
expect(await hasNativeLocalProcessStop(db, f.run.companyId, f.run.id)).toBe(true);
expect(f.deps.terminateHeartbeatRunProcess).not.toHaveBeenCalled();
} finally { await f.cleanup(); }
});
it("retires a remote identity only with an exact authenticated termination receipt", async () => {
const child = await ownedProcess();
const f = await nativeProcessFixture(child.pid, "remote", "failed");
try {
await db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: { fixture: true } } })
.where(eq(heartbeatRuns.id, f.run.id));
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure" })
.where(eq(nativeRunFinalizations.runId, f.run.id));
const [lease] = await db.insert(environmentLeases).values({ companyId: f.run.companyId,
heartbeatRunId: f.run.id, issueId: f.run.issueId, status: "active", provider: "daytona", providerLeaseId: randomUUID() }).returning();
expect(await releaseSettledNativeRunProcess(db, f.run.id)).toBe("pending");
const receipt = remoteTerminationReceipt(lease, { state: "destroyed", providerLeaseId: lease.providerLeaseId })!;
await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success", releasedAt: new Date(),
metadata: { remoteExecutionTermination: { ...receipt, runId: randomUUID() } } }).where(eq(environmentLeases.id, lease.id));
expect(await releaseSettledNativeRunProcess(db, f.run.id)).toBe("pending");
expect(await f.current()).toMatchObject({ processPid: child.pid, processGroupId: child.pid });
await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination: receipt } })
.where(eq(environmentLeases.id, lease.id));
expect(await releaseSettledNativeRunProcess(db, f.run.id)).toBe("released");
expect(await f.current()).toMatchObject({ processPid: null, processGroupId: null, processStartedAt: null });
expect(await hasNativeLocalProcessStop(db, f.run.companyId, f.run.id)).toBe(false);
expect(child.alive()).toBe(true);
} finally { try { await f.cleanup(); } finally { await child.stop(); } }
});
it.each(["ephemeral", "reuse_by_environment", "live_owner", "live_executor", "mismatched_source", "finalizing"] as const)(
"reconstructs missing native preservation during orphan cleanup without replay: %s", async state => {
const run = await seedRun();
const [environment] = await db.insert(environments).values({ name: run.id, driver: "sandbox" }).returning();
const [lease] = await db.insert(environmentLeases).values({ companyId: run.companyId, issueId: run.issueId,
heartbeatRunId: run.id, environmentId: environment.id, provider: "daytona", providerLeaseId: randomUUID(),
status: "active", leasePolicy: state === "reuse_by_environment" ? state : "ephemeral", updatedAt: new Date(0),
metadata: { driver: "sandbox", sandboxProviderPlugin: true, pluginId: randomUUID(), remoteCwd: "/work" },
}).returning();
await db.update(heartbeatRuns).set({ status: "failed", runtimeMode: "native", nativeIssueId: run.issueId,
finishedAt: new Date(), errorCode: "execution_finalization_deadline_exceeded", contextSnapshot: { issueId: run.issueId },
runnerProfileJson: { nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v3", state: "prepared",
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null,
workspaceId: run.id, leaseId: lease.id, providerLeaseId: lease.providerLeaseId,
remoteCwd: state === "mismatched_source" ? "/other" : "/work", resourceDisposition: "keep_running" } },
}).where(eq(heartbeatRuns.id, run.id));
await db.insert(nativeRunFinalizations).values({ companyId: run.companyId, issueId: run.issueId!, runId: run.id,
phase: state === "finalizing" ? "workspace_finalizing" : "terminal_failure",
leaseOwner: state === "live_owner" ? "another-controller" : null });
const deps = callbacks(db);
if (state === "live_executor") deps.activeRunExecutions.add(run.id);
const recovery = createHeartbeatRecovery(db, deps);
const wake = vi.fn(), unsubscribe = subscribeDeliveryWork(db, DELIVERY_QUEUES.workspaceRecovery, wake);
const shouldRecover = state === "ephemeral" || state === "reuse_by_environment";
try {
expect(await recovery.sweepOrphanedActiveLeases({ backoffMs: 0 })).toEqual({ recovered: shouldRecover ? 1 : 0 });
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
const [retained] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id));
const records = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, run.issueId!));
expect(source.status).toBe("failed");
if (shouldRecover) {
expect(source.resultJson).toMatchObject({ workspaceRestoreRecovery: { leaseIds: [lease.id] } });
expect(retained).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure",
metadata: { sandboxStopAndRetain: { leaseId: lease.id, runId: run.id } } });
expect(records).toHaveLength(1);
expect(records[0].evidence).toMatchObject({ automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } });
expect(wake).toHaveBeenCalledOnce();
const registered = source.resultJson;
expect(await recovery.sweepOrphanedActiveLeases({ backoffMs: 0 })).toEqual({ recovered: 0 });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].resultJson).toEqual(registered);
expect(wake).toHaveBeenCalledOnce();
} else {
expect(source.resultJson).toBeNull();
expect(retained.status).toBe("active");
expect(records).toHaveLength(0);
expect(wake).not.toHaveBeenCalled();
}
expect(hasRemoteTerminationReceipt(retained)).toBe(false);
expect(retained.metadata).not.toHaveProperty("remoteExecutionTermination");
expect(deps.executeRun).not.toHaveBeenCalled();
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, run.companyId))).toHaveLength(1);
} finally {
unsubscribe();
await db.delete(environmentLeases).where(eq(environmentLeases.id, lease.id));
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, run.issueId!));
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
await db.delete(activityLog).where(eq(activityLog.runId, run.id));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
}
},
);
it.each(["restored", "restore_failed"])("keeps the sandbox available until adapter workspace settlement: %s", async outcome => {
const run = await seedRun();
const deps = callbacks(db);
deps.setRunStatusIfRunning.mockImplementation(async (id, status, patch) => {
const [updated] = await db.update(heartbeatRuns).set({ ...patch, status }).where(eq(heartbeatRuns.id, id)).returning();
return { updated: true, run: updated };
});
deps.getRun.mockImplementation(async id => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0] ?? null);
const control = createAdapterExecutionControl();
adapterExecutionControls.set(run.id, control);
const drain = createHeartbeatRecovery(db, deps).drainRunningRunsForShutdown("SIGTERM", new Date(), [run.id]);
try {
await vi.waitFor(() => expect(control.controller.signal.aborted).toBe(true));
expect(deps.releaseEnvironmentLeasesForRun).not.toHaveBeenCalled();
expect(deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
expect((await deps.getRun(run.id))?.status).toBe("interrupted");
const resultJson = outcome === "restore_failed" ? { workspaceRestoreFailure: "restore_failed",
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [randomUUID()] } }
: { workspaceRestored: true };
await db.update(heartbeatRuns).set({ resultJson }).where(eq(heartbeatRuns.id, run.id));
control.finish();
await drain;
expect(deps.releaseEnvironmentLeasesForRun).toHaveBeenCalledOnce();
expect(deps.classifyAndPersistRunLiveness).toHaveBeenCalledWith(expect.objectContaining({
resultJson: expect.objectContaining(resultJson) }), expect.anything());
} finally {
control.finish();
adapterExecutionControls.delete(run.id);
await drain;
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
}
});
it("starts all owned shutdown cancellations within one settlement window", async () => {
const runs = await Promise.all([seedRun(), seedRun()]);
const controls = runs.map(run => {
const control = createAdapterExecutionControl(); adapterExecutionControls.set(run.id, control); return control;
});
const deps = callbacks(db);
deps.setRunStatusIfRunning.mockImplementation(async (id, status, patch) => {
const [updated] = await db.update(heartbeatRuns).set({ ...patch, status }).where(eq(heartbeatRuns.id, id)).returning();
return { updated: true, run: updated };
});
const drain = createHeartbeatRecovery(db, deps).drainRunningRunsForShutdown("SIGTERM", new Date(), runs.map(run => run.id));
try {
await vi.waitFor(() => expect(controls.every(control => control.controller.signal.aborted)).toBe(true));
expect(deps.releaseEnvironmentLeasesForRun).not.toHaveBeenCalled();
} finally {
controls.forEach(control => control.finish());
await drain;
for (const run of runs) { adapterExecutionControls.delete(run.id); await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); }
}
});
it.each(["ephemeral", "reuse_by_environment"] as const)("protects a %s export from another server and pins it after controller loss", async leasePolicy => {
const run = await seedRun();
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: run.issueId } }).where(eq(heartbeatRuns.id, run.id));
const [lease] = await db.insert(environmentLeases).values({ companyId: run.companyId, heartbeatRunId: run.id,
issueId: run.issueId, status: "active", provider: "daytona", providerLeaseId: randomUUID(), leasePolicy,
metadata: { driver: "sandbox", sandboxProviderPlugin: true, pluginId: "fixture-plugin", remoteCwd: "/home/daytona/workspace" } }).returning();
try {
const deadline = new Date(Date.now() + 30_000);
await beginLegacyShutdownWorkspaceSettlement(db, run, deadline);
await db.update(heartbeatRuns).set({ status: "interrupted", errorCode: "server_shutdown_interrupted", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run.id));
const foreignDeps = callbacks(db);
const foreign = createHeartbeatRecovery(db, foreignDeps);
expect(await foreign.sweepOrphanedActiveLeases({ backoffMs: 0 })).toMatchObject({ recovered: 0 });
expect(await environmentRuntimeService(db).releaseRunLeases(run.id)).toEqual([]);
await db.update(environmentLeases).set({ status: "pending_cleanup" }).where(eq(environmentLeases.id, lease.id));
expect(await foreign.sweepPendingCleanupLeases({ backoffMs: 0 })).toMatchObject({ destroyed: 0 });
await db.update(environmentLeases).set({ status: "active" }).where(eq(environmentLeases.id, lease.id));
let source = (await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id)))[0];
expect(source.status).toBe("active");
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, run.issueId!))).toHaveLength(0);
// No local adapter map or timeout handler survives a container SIGKILL.
expect(await allowLegacyShutdownWorkspaceCleanup(db, source, new Date(deadline.getTime() + 1))).toBe(true);
source = (await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id)))[0];
expect(source).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure",
metadata: { legacyShutdownWorkspaceSettlement: { state: "expired" }, sandboxStopAndRetain: { leaseId: lease.id } } });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].resultJson)
.toMatchObject({ workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { leaseIds: [lease.id] } });
expect(await foreign.sweepOrphanedActiveLeases({ backoffMs: 0 })).toMatchObject({ recovered: 0 });
} finally {
await db.delete(environmentLeases).where(eq(environmentLeases.id, lease.id));
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, run.issueId!));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
}
});
async function seedShutdownSource() {
const run = await seedRun();
const [lease] = await db.insert(environmentLeases).values({ companyId: run.companyId, heartbeatRunId: run.id,
issueId: run.issueId, status: "active", provider: "daytona", providerLeaseId: randomUUID(), leasePolicy: "ephemeral",
metadata: { driver: "sandbox", sandboxProviderPlugin: true, pluginId: "fixture-plugin", remoteCwd: "/home/daytona/workspace" } }).returning();
await beginLegacyShutdownWorkspaceSettlement(db, run, new Date(Date.now() + 30_000));
return { run, lease };
}
async function deleteShutdownSource(f: Awaited<ReturnType<typeof seedShutdownSource>>) {
await db.delete(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.run.issueId!));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
adapterExecutionControls.delete(f.run.id);
}
async function fenceFromPreviousServer(f: Awaited<ReturnType<typeof seedShutdownSource>>,
controller: Record<string, unknown> | null) {
const [row] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
const marker: Record<string, unknown> = { ...(row.metadata!.legacyShutdownWorkspaceSettlement as Record<string, unknown>),
controllerBootId: randomUUID() };
if (controller) marker.controller = { ...controller, bootId: marker.controllerBootId };
else delete marker.controller;
await db.update(environmentLeases).set({ metadata: { ...row.metadata, legacyShutdownWorkspaceSettlement: marker } })
.where(eq(environmentLeases.id, f.lease.id));
await db.update(heartbeatRuns).set({ status: "interrupted", errorCode: "server_shutdown_interrupted", finishedAt: new Date() })
.where(eq(heartbeatRuns.id, f.run.id));
}
const fenceState = async (leaseId: string) => ((await db.select().from(environmentLeases)
.where(eq(environmentLeases.id, leaseId)))[0].metadata!.legacyShutdownWorkspaceSettlement as { state: string });
const deadController = () => ({ pid: 4_194_303, hostHash: currentProcessSpaceHash(), processStartedAt: new Date(0).toISOString() });
it("stamps the shutdown fence with the committing server process", async () => {
await currentProcessStartedAtMs();
const f = await seedShutdownSource();
try {
const controller = currentControllerProcessStamp();
expect((await fenceState(f.lease.id) as unknown as { controller: unknown }).controller)
.toEqual({ bootId: controller.bootId, pid: process.pid, hostHash: controller.hostHash,
processStartedAt: controller.processStartedAt });
} finally { await deleteShutdownSource(f); }
});
it("settles the fence of a previous server proven gone without waiting out its deadline", async () => {
const f = await seedShutdownSource();
try {
await fenceFromPreviousServer(f, deadController());
await sweepLegacyShutdownWorkspaceSettlements(db);
expect(await fenceState(f.lease.id)).toMatchObject({ state: "expired" });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)))[0].resultJson)
.toMatchObject({ workspaceRestoreFailure: "restore_failed" });
} finally { await deleteShutdownSource(f); }
});
it("settles a proven-gone fence owner from the shared reconciler, and only that one", async () => {
const dead = await seedShutdownSource();
const foreign = await seedShutdownSource();
const legacy = await seedShutdownSource();
const kind = OWNABLE_KINDS.find(entry => entry.id === "legacy_shutdown_settlement_fence")!;
try {
await fenceFromPreviousServer(dead, deadController());
await fenceFromPreviousServer(foreign, { ...deadController(), hostHash: "another-host-pid-space" });
await fenceFromPreviousServer(legacy, null);
const readOnly = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), kinds: [kind], record: false });
expect(readOnly.kinds[0]).toMatchObject({ inProgress: 3, settled: 0 });
expect(await fenceState(dead.lease.id)).toMatchObject({ state: "pending" });
const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), kinds: [kind] });
expect(report.kinds[0]).toMatchObject({ inProgress: 3, settled: 1 });
expect(await fenceState(dead.lease.id)).toMatchObject({ state: "expired" });
await sweepLegacyShutdownWorkspaceSettlements(db);
expect(await fenceState(foreign.lease.id)).toMatchObject({ state: "pending" });
expect(await fenceState(legacy.lease.id)).toMatchObject({ state: "pending" });
await sweepLegacyShutdownWorkspaceSettlements(db, new Date(Date.now() + 60_000));
expect(await fenceState(foreign.lease.id)).toMatchObject({ state: "expired" });
expect(await fenceState(legacy.lease.id)).toMatchObject({ state: "expired" });
} finally { for (const f of [dead, foreign, legacy]) await deleteShutdownSource(f); }
});
it("does not let an elapsed shutdown wait stop a sandbox whose exporter still owns settlement", async () => {
const f = await seedShutdownSource();
const control = createAdapterExecutionControl();
control.controller.abort();
adapterExecutionControls.set(f.run.id, control);
const stop = vi.fn(async () => ({ providerLeaseId: f.lease.providerLeaseId, state: "stopped" }));
const runtime = environmentRuntimeService(db, { drivers: [{ driver: "sandbox",
resolveCapabilities: async () => classifyEnvironmentCapabilities({}),
acquireRunLease: async () => { throw new Error("Unexpected acquire"); },
releaseRunLease: vi.fn(async () => null), retryPendingSandboxTeardown: stop }] });
try {
await db.update(heartbeatRuns).set({ status: "interrupted", errorCode: "server_shutdown_interrupted", finishedAt: new Date() })
.where(eq(heartbeatRuns.id, f.run.id));
const [source] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
expect(await allowLegacyShutdownWorkspaceCleanup(db, source, new Date(Date.now() + 60_000))).toBe(false);
await runtime.releaseRunLeases(f.run.id);
expect(stop).not.toHaveBeenCalled();
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0].status).toBe("active");
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.run.issueId!))).toHaveLength(0);
} finally { control.finish(); await deleteShutdownSource(f); }
});
it("renews the durable export fence for other controllers and expires it after owner loss", async () => {
const f = await seedShutdownSource();
const [initial] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
const originalDeadline = (initial.metadata!.legacyShutdownWorkspaceSettlement as { deadlineAt: string }).deadlineAt;
vi.useFakeTimers({ toFake: ["Date", "setInterval", "clearInterval"] });
const stopRenewing = maintainLegacyShutdownWorkspaceSettlement(db, f.run);
try {
await vi.advanceTimersByTimeAsync(10_000);
await vi.waitFor(async () => {
const [current] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
expect(Date.parse((current.metadata!.legacyShutdownWorkspaceSettlement as { deadlineAt: string }).deadlineAt))
.toBeGreaterThan(Date.parse(originalDeadline));
});
expect(await allowLegacyShutdownWorkspaceCleanup(db, initial, new Date(Date.parse(originalDeadline) + 1))).toBe(false);
await stopRenewing();
await vi.advanceTimersByTimeAsync(60_000);
await db.update(heartbeatRuns).set({ status: "interrupted", finishedAt: new Date() }).where(eq(heartbeatRuns.id, f.run.id));
await sweepLegacyShutdownWorkspaceSettlements(db);
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0])
.toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure",
metadata: { legacyShutdownWorkspaceSettlement: { state: "expired" }, sandboxStopAndRetain: { leaseId: f.lease.id } } });
} finally { await stopRenewing(); vi.useRealTimers(); await deleteShutdownSource(f); }
});
it("bounds a contended settlement renewal without blocking unrelated database work", async () => {
const f = await seedShutdownSource();
const warning = vi.spyOn(logger, "warn");
let unlock!: () => void;
let locked!: () => void;
const lockReady = new Promise<void>(resolve => { locked = resolve; });
const releaseLock = new Promise<void>(resolve => { unlock = resolve; });
const lock = db.transaction(async tx => {
await tx.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)).for("update");
locked();
await releaseLock;
});
await lockReady;
vi.useFakeTimers({ toFake: ["Date", "setInterval", "clearInterval"] });
const stopRenewing = maintainLegacyShutdownWorkspaceSettlement(db, f.run);
try {
await vi.advanceTimersByTimeAsync(10_000);
expect(await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)))
.toEqual([{ id: f.run.id }]);
await stopRenewing();
expect(warning).toHaveBeenCalledWith({ runId: f.run.id, errorKind: "lock_timeout" },
"Shutdown workspace settlement lease renewal failed; source remains protected locally");
} finally {
unlock();
await lock;
await stopRenewing();
vi.useRealTimers();
warning.mockRestore();
await deleteShutdownSource(f);
}
});
it.each([["57014", "statement_cancelled"], ["08006", "connection_failed"], ["private-credential", "unknown"]])(
"reports renewal failure %s without raw database diagnostics", async (code, errorKind) => {
const run = { id: randomUUID(), companyId: randomUUID() };
const privateValue = "postgres://private-credential@example.invalid/database";
const failure = new Error(privateValue, { cause: { code, query: privateValue, params: [privateValue] } });
const dbBoundary = { transaction: vi.fn().mockRejectedValue(failure) } as unknown as Db;
const warning = vi.spyOn(logger, "warn").mockImplementation(() => undefined);
vi.useFakeTimers({ toFake: ["setInterval", "clearInterval"] });
const stopRenewing = maintainLegacyShutdownWorkspaceSettlement(dbBoundary, run);
try {
await vi.advanceTimersByTimeAsync(10_000);
await stopRenewing();
expect(warning).toHaveBeenCalledExactlyOnceWith({ runId: run.id, errorKind },
"Shutdown workspace settlement lease renewal failed; source remains protected locally");
expect(JSON.stringify(warning.mock.calls)).not.toContain("private-credential");
} finally { await stopRenewing(); vi.useRealTimers(); warning.mockRestore(); }
},
);
it("does not turn a stale pending snapshot into a repair hold after settlement", async () => {
const f = await seedShutdownSource();
try {
const old = (await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0];
await db.update(heartbeatRuns).set({ status: "interrupted", contextSnapshot: { issueId: f.run.issueId }, resultJson: { workspaceRestored: true } }).where(eq(heartbeatRuns.id, f.run.id));
await finishLegacyShutdownWorkspaceSettlement(db, f.run);
const single = createDb(externalUrl ?? database.connectionString, { maxConnections: 1 });
try { expect(await allowLegacyShutdownWorkspaceCleanup(single, old, new Date(Date.now() + 60_000))).toBe(true); }
finally { await single.$client.end({ timeout: 1 }); }
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0])
.toMatchObject({ status: "active", leasePolicy: "ephemeral", metadata: { legacyShutdownWorkspaceSettlement: { state: "settled" } } });
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.run.issueId!))).toHaveLength(0);
} finally { await deleteShutdownSource(f); }
});
it("allows the aborted owner to stop and retain its command while excluding outside cleanup", async () => {
const f = await seedShutdownSource();
const control = createAdapterExecutionControl(); control.controller.abort();
adapterExecutionControls.set(f.run.id, control);
const stop = vi.fn(async () => ({ providerLeaseId: f.lease.providerLeaseId, state: "stopped" }));
const release = vi.fn(async () => null);
const runtime = environmentRuntimeService(db, { drivers: [{ driver: "sandbox", resolveCapabilities: async () => classifyEnvironmentCapabilities({}),
acquireRunLease: async () => { throw new Error("Unexpected acquire"); }, releaseRunLease: release,
retryPendingSandboxTeardown: stop }] });
try {
await runtime.releaseRunLeases(f.run.id, "released", undefined, "stop_and_retain", false);
expect(stop).not.toHaveBeenCalled();
await runtime.releaseRunLeases(f.run.id, "released", undefined, "stop_and_retain", true);
expect(stop).toHaveBeenCalledOnce(); expect(release).not.toHaveBeenCalled();
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0])
.toMatchObject({ status: "released", cleanupStatus: "success", metadata: { remoteExecutionTermination: { state: "stopped" },
sandboxStopAndRetainReceipt: { leaseId: f.lease.id } } });
} finally { control.finish(); await deleteShutdownSource(f); }
});
it.each(["ephemeral", "reuse_by_environment"] as const)("recovers a stopped %s source after its shutdown controller disappears", async leasePolicy => {
const f = await seedShutdownSource();
await db.update(environmentLeases).set({ leasePolicy }).where(eq(environmentLeases.id, f.lease.id));
const control = createAdapterExecutionControl(); control.controller.abort(); adapterExecutionControls.set(f.run.id, control);
const [environmentRow] = await db.insert(environments).values({ name: "Guarded source", driver: "sandbox" }).returning();
const environment = (await environmentService(db).getById(environmentRow.id))!;
const leaseSnapshot = (await environmentService(db).getLeaseById(f.lease.id))!;
leaseSnapshot.metadata = f.lease.metadata;
const resume = vi.fn(async () => null); const destroy = vi.fn(async () => null);
const stop = vi.fn(async () => ({ providerLeaseId: f.lease.providerLeaseId, state: "stopped" }));
const runtime = environmentRuntimeService(db, { drivers: [{ driver: "sandbox", resolveCapabilities: async () => classifyEnvironmentCapabilities({}),
acquireRunLease: async () => { throw new Error("Unexpected acquire"); }, releaseRunLease: vi.fn(), retryPendingSandboxTeardown: stop, resumeRunLease: resume, destroyRunLease: destroy }] });
try {
await runtime.releaseRunLeases(f.run.id, "released", undefined, "stop_and_retain", true);
adapterExecutionControls.delete(f.run.id); // SIGKILL before export/finally settled.
await db.update(heartbeatRuns).set({ status: "interrupted", finishedAt: new Date() }).where(eq(heartbeatRuns.id, f.run.id));
const [source] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
expect(source.status).toBe("released");
// An old reuse receipt must not bypass the current allocation's fence.
expect(await legacyShutdownWorkspaceResourceProtected(db, { provider: source.provider, providerLeaseId: source.providerLeaseId })).toBe(true);
await sweepLegacyShutdownWorkspaceSettlements(db, new Date(Date.now() + 60_000));
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0])
.toMatchObject({ status: "released", leasePolicy: "retain_on_failure", metadata: {
legacyShutdownWorkspaceSettlement: { state: "expired" }, workspaceRestoreRecovery: { runId: f.run.id } } });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)))[0].resultJson)
.toMatchObject({ workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { leaseIds: [f.lease.id] } });
expect(await legacyShutdownWorkspaceResourceProtected(db, source)).toBe(true);
// Direct operations and pre-fence snapshots cannot bypass the source pin.
expect(await runtime.resumeRunLease({ environment, lease: leaseSnapshot })).toBeNull();
expect(await runtime.destroyRunLease({ environment, lease: leaseSnapshot })).toBeNull();
expect(resume).not.toHaveBeenCalled(); expect(destroy).not.toHaveBeenCalled();
expect(stop).toHaveBeenCalledOnce();
} finally { control.finish(); await deleteShutdownSource(f); await db.delete(environments).where(eq(environments.id, environment.id)); }
});
it.each(["ephemeral", "reuse_by_environment"] as const)("recovers failed ordinary cleanup of a fenced %s source", async leasePolicy => {
const f = await seedShutdownSource();
const stop = vi.fn(async () => ({ providerLeaseId: f.lease.providerLeaseId, state: "stopped" }));
const destroy = vi.fn(async () => null);
const runtime = environmentRuntimeService(db, { drivers: [{ driver: "sandbox", resolveCapabilities: async () => classifyEnvironmentCapabilities({}),
acquireRunLease: async () => { throw new Error("Unexpected acquire"); }, releaseRunLease: destroy, destroyRunLease: destroy,
retryPendingSandboxTeardown: stop }] });
try {
await db.update(heartbeatRuns).set({ status: "interrupted", finishedAt: new Date(), contextSnapshot: { issueId: f.run.issueId } }).where(eq(heartbeatRuns.id, f.run.id));
await db.update(environmentLeases).set({ status: "pending_cleanup", leasePolicy, cleanupStatus: "failed",
releasedAt: new Date(), failureReason: "release_cleanup_failed" }).where(eq(environmentLeases.id, f.lease.id));
const [source] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
expect(await allowLegacyShutdownWorkspaceCleanup(db, source, new Date(Date.now() + 60_000))).toBe(true);
const lease = (await environmentService(db).getLeaseById(f.lease.id))!;
expect(lease).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure", metadata: {
legacyShutdownWorkspaceSettlement: { state: "expired" }, sandboxStopAndRetain: { leaseId: f.lease.id } } });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)))[0].resultJson)
.toMatchObject({ workspaceRestoreRecovery: { leaseIds: [f.lease.id] } });
const deps = callbacks(db); deps.environmentRuntime = runtime;
expect(await createHeartbeatRecovery(db, deps).sweepPendingCleanupLeases({ backoffMs: 0 })).toMatchObject({ destroyed: 1 });
expect((await environmentService(db).getLeaseById(f.lease.id))!).toMatchObject({ status: "released", cleanupStatus: "success" });
expect(stop).toHaveBeenCalledOnce(); expect(destroy).not.toHaveBeenCalled();
} finally { await deleteShutdownSource(f); }
});
it("waits for a claimed ordinary cleanup attempt before converting its shutdown source", async () => {
const f = await seedShutdownSource(); const expiresAt = Date.now() + 120_000;
try {
await db.update(heartbeatRuns).set({ status: "interrupted", finishedAt: new Date(), contextSnapshot: { issueId: f.run.issueId } }).where(eq(heartbeatRuns.id, f.run.id));
const [marked] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
await db.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", releasedAt: new Date(),
metadata: { ...marked.metadata, pendingCleanupInFlight: true, pendingCleanupLeaseExpiresAtMs: expiresAt } }).where(eq(environmentLeases.id, f.lease.id));
const [source] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
expect(await allowLegacyShutdownWorkspaceCleanup(db, source, new Date(expiresAt - 1))).toBe(false);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)))[0].resultJson?.workspaceRestoreRecovery).toBeUndefined();
expect(await allowLegacyShutdownWorkspaceCleanup(db, source, new Date(expiresAt + 1))).toBe(true);
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0])
.toMatchObject({ status: "pending_cleanup", metadata: { sandboxStopAndRetain: { leaseId: f.lease.id } } });
} finally { await deleteShutdownSource(f); }
});
it("retries lease release when ordinary completion wins the shutdown status race", async () => {
const f = await seedShutdownSource(); const control = createAdapterExecutionControl();
adapterExecutionControls.set(f.run.id, control);
const deps = callbacks(db);
deps.setRunStatusIfRunning.mockImplementation(async () => {
await db.update(heartbeatRuns).set({ status: "succeeded", resultJson: { workspaceRestored: true } }).where(eq(heartbeatRuns.id, f.run.id));
// The adapter's first release was deferred by the fence; its finally ended.
control.finish(); return { updated: false, run: null };
});
deps.getRun.mockImplementation(async id => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0] ?? null);
deps.releaseEnvironmentLeasesForRun.mockImplementation(async input => {
expect(input.status).toBe("succeeded");
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0].metadata)
.toMatchObject({ legacyShutdownWorkspaceSettlement: { state: "settled" } });
await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success", releasedAt: new Date() }).where(eq(environmentLeases.id, f.lease.id));
});
try {
await createHeartbeatRecovery(db, deps).drainRunningRunsForShutdown("SIGTERM", new Date(), [f.run.id]);
expect(deps.releaseEnvironmentLeasesForRun).toHaveBeenCalledOnce();
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0].status).toBe("released");
expect(deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
} finally { control.finish(); await deleteShutdownSource(f); }
});
it("joins a late successful export before cleanup and continuation after the shutdown wait", async () => {
const run = await seedRun();
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: run.issueId } }).where(eq(heartbeatRuns.id, run.id));
const [lease] = await db.insert(environmentLeases).values({ companyId: run.companyId, heartbeatRunId: run.id,
issueId: run.issueId, status: "active", provider: "daytona", providerLeaseId: randomUUID(), leasePolicy: "ephemeral",
metadata: { driver: "sandbox", sandboxProviderPlugin: true, pluginId: "fixture-plugin", remoteCwd: "/home/daytona/workspace" } }).returning();
await db.update(agents).set({ adapterType: "codex_local" }).where(eq(agents.id, run.agentId));
const deps = callbacks(db);
deps.setRunStatusIfRunning.mockImplementation(async (id, status, patch) => {
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id)))[0].metadata)
.toMatchObject({ legacyShutdownWorkspaceSettlement: { state: "pending", runId: run.id } });
const [updated] = await db.update(heartbeatRuns).set({ ...patch, status }).where(eq(heartbeatRuns.id, id)).returning();
return { updated: true, run: updated };
});
deps.getRun.mockImplementation(async id => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0] ?? null);
deps.releaseEnvironmentLeasesForRun.mockImplementation(async () => {
expect((await deps.getRun(run.id))?.resultJson).toEqual({ workspaceRestored: true, conversationContinuation: "continue_conversation_v1" });
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id)))[0]).toMatchObject({
status: "active", leasePolicy: "ephemeral", metadata: { legacyShutdownWorkspaceSettlement: { state: "settled" } } });
await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success", releasedAt: new Date() }).where(eq(environmentLeases.id, lease.id));
});
const control = createAdapterExecutionControl();
adapterExecutionControls.set(run.id, control);
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
const drain = createHeartbeatRecovery(db, deps).drainRunningRunsForShutdown("SIGTERM", new Date(), [run.id]);
try {
await vi.waitFor(() => expect(control.controller.signal.aborted).toBe(true));
await vi.advanceTimersByTimeAsync(30_000);
expect(await drain).toMatchObject({ interruptedRunIds: [run.id] });
expect(deps.activeRunExecutionPromises.size).toBe(1);
expect(deps.releaseEnvironmentLeasesForRun).not.toHaveBeenCalled();
expect(deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, run.issueId!))).toHaveLength(0);
await db.update(heartbeatRuns).set({ resultJson: { workspaceRestored: true, conversationContinuation: "continue_conversation_v1" } }).where(eq(heartbeatRuns.id, run.id));
control.finish();
await Promise.all([...deps.activeRunExecutionPromises]);
expect(deps.releaseEnvironmentLeasesForRun).toHaveBeenCalledOnce();
expect(deps.scheduleBoundedRetryForRun).toHaveBeenCalledOnce();
expect(deps.scheduleBoundedRetryForRun).toHaveBeenCalledAfter(deps.releaseEnvironmentLeasesForRun);
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, run.issueId!))).toHaveLength(0);
} finally {
vi.useRealTimers();
control.finish();
adapterExecutionControls.delete(run.id);
await drain;
await Promise.all([...deps.activeRunExecutionPromises]);
await db.delete(environmentLeases).where(eq(environmentLeases.id, lease.id));
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, run.issueId!));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
}
});
it("uses the shared live-execution set after construction across two factories", async () => {
const run = await seedRun();
const deps = callbacks(db);
const first = createHeartbeatRecovery(db, deps);
const second = createHeartbeatRecovery(db, deps);
deps.activeRunExecutions.add(run.id);
expect(await first.reapOrphanedRuns()).toEqual({ reaped: 0, runIds: [] });
expect(await second.reapOrphanedRuns()).toEqual({ reaped: 0, runIds: [] });
expect(deps.setRunStatusFromLive).not.toHaveBeenCalled();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("running");
await Promise.all([...deps.activeRunExecutionPromises]);
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
});
it("settles persisted native results before claiming restart authority and tracks the exact execution", async () => {
const deps = callbacks(db);
const claim: NativeRestartRecoveryClaim = { kind: "bootstrap_incomplete", runId: randomUUID(), leaseOwner: "claimed-owner", controllerGeneration: 3, providerAttempt: 2, restartKind: "hard", recoveryRequestId: null };
native.claims.mockResolvedValue([claim]);
let finish!: () => void;
const execution = new Promise<void>(resolve => { finish = resolve; });
deps.executeRun.mockReturnValue(execution);
const result = await createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart();
expect(result.claims).toEqual([claim]);
expect(native.claims).toHaveBeenCalledAfter(native.finalizations);
expect(deps.executeRun).toHaveBeenCalledWith(claim.runId, { nativeLeaseOwner: claim.leaseOwner, nativeRestartRecovery: claim });
expect(deps.activeRunExecutionPromises.size).toBeGreaterThan(0);
finish();
await Promise.all([...deps.activeRunExecutionPromises]);
await vi.waitFor(() => expect(deps.activeRunExecutionPromises.size).toBe(0));
});
it.each(["completion", "failure_before_dispatch"] as const)("stops preparation renewal after recovered %s", async outcome => {
const run = await seedRun();
const deps = callbacks(db);
const controller = await currentNativeControllerIdentity();
const claim: NativeRestartRecoveryClaim = { kind: "reattach_remote_runner", runId: run.id,
leaseOwner: randomUUID(), controllerGeneration: 2, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null,
remote: { providerLeaseId: randomUUID(), remoteCwd: "/workspace" } };
const scope = { companyId: run.companyId, issueId: run.issueId!, controller, remote: true };
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId }).where(eq(heartbeatRuns.id, run.id));
const expiresAt = new Date(Date.now() + 60_000);
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!, phase: "observed", attempt: 1,
controllerBootId: controller.bootId, controllerPid: controller.pid,
controllerProcessStartedAt: controller.processStartedAt,
leaseOwner: claim.leaseOwner, leaseExpiresAt: expiresAt });
vi.useFakeTimers({ toFake: ["setInterval", "clearInterval"] });
native.claims.mockImplementationOnce(async input => {
await input.onClaim?.(claim, scope);
return [claim];
});
let released!: () => void, finish!: () => void;
const leaseReleased = new Promise<void>(resolve => { released = resolve; });
const finalization = new Promise<void>(resolve => { finish = resolve; });
deps.executeRun.mockImplementationOnce(async (_runId, options) => {
if (outcome === "failure_before_dispatch") {
released();
throw new Error("workspace preparation failed");
}
const execution = startNativeSessionExecutionLeaseRenewal({ db, ...scope, runId: run.id,
leaseOwner: claim.leaseOwner, attempt: claim.providerAttempt,
onLost: () => native.detach() });
try {
// The executor takes its reference before preparation releases its own.
await options?.onNativeExecutionLeaseAcquired?.();
await execution.stop();
await db.update(nativeRunFinalizations).set({ leaseOwner: null, leaseExpiresAt: null })
.where(eq(nativeRunFinalizations.runId, run.id));
released();
// Workspace/result settlement may still be pending after lease release.
await finalization;
} finally { await execution.stop(); }
});
try {
await createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart(new Date(), [run.id]);
await leaseReleased;
if (outcome === "failure_before_dispatch") await Promise.all([...deps.activeRunExecutionPromises]);
expect(vi.getTimerCount()).toBe(0);
await vi.advanceTimersByTimeAsync(21_000);
expect(deps.activeRunExecutionPromises.size).toBe(outcome === "completion" ? 1 : 0);
expect(native.detach).not.toHaveBeenCalled();
expect(deps.executeRun).toHaveBeenCalledOnce();
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
expect(coordinator.leaseExpiresAt).toEqual(outcome === "completion" ? null : expiresAt);
} finally {
finish();
await Promise.all([...deps.activeRunExecutionPromises]);
vi.useRealTimers();
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
}
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
it("keeps a remote bootstrap restart claim through preparation longer than its first lease, then hands it to the next attempt", async () => {
const run = await seedRun();
const deps = callbacks(db);
const controller = await currentNativeControllerIdentity();
const claim: NativeRestartRecoveryClaim = { kind: "bootstrap_incomplete", runId: run.id, leaseOwner: randomUUID(),
controllerGeneration: 3, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null };
const scope = { companyId: run.companyId, issueId: run.issueId!, controller, remote: true };
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId, status: "running" }).where(eq(heartbeatRuns.id, run.id));
const firstLeaseEndsAt = new Date(Date.now() + 4_000);
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!, phase: "observed", attempt: 1,
controllerBootId: controller.bootId, controllerPid: controller.pid, controllerProcessStartedAt: controller.processStartedAt,
leaseOwner: claim.leaseOwner, leaseExpiresAt: firstLeaseEndsAt });
native.claims.mockImplementationOnce(async input => {
await input.onClaim?.(claim, scope);
return [claim];
});
const [admitted] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
const authority = createRunLogWriterAuthority(db, admitted, claim.leaseOwner);
let finish!: () => void, report!: (outcome: unknown) => void;
const outcome = new Promise<unknown>(resolve => { report = resolve; });
const finished = new Promise<void>(resolve => { finish = resolve; });
deps.executeRun.mockImplementationOnce(async (_runId, options) => { try {
// The shared renewal timer renews the restart claim (preparation).
await renewNativeExecutionLeasesNow(db);
await vi.waitFor(async () => {
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
expect(coordinator.leaseExpiresAt!.getTime()).toBeGreaterThan(firstLeaseEndsAt.getTime() + 10_000);
});
await new Promise(resolve => setTimeout(resolve, Math.max(0, firstLeaseEndsAt.getTime() - Date.now() + 1_000)));
await authority();
await db.update(nativeRunFinalizations).set({ attempt: 2 }).where(eq(nativeRunFinalizations.runId, run.id));
const execution = startNativeSessionExecutionLeaseRenewal({ db, ...scope, runId: run.id,
leaseOwner: claim.leaseOwner, attempt: 2, onLost: () => native.detach() });
try {
await options?.onNativeExecutionLeaseAcquired?.();
const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
await renewNativeExecutionLeasesNow(db);
await vi.waitFor(async () => {
const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
expect(after.leaseExpiresAt!.getTime()).toBeGreaterThan(before.leaseExpiresAt!.getTime());
});
report("handed off");
await finished;
} finally { await execution.stop(); }
} catch (error) { report(error); } });
try {
native.detach.mockClear();
await createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart(new Date(), [run.id]);
expect(await outcome).toBe("handed off");
expect(deps.executeRun).toHaveBeenCalledWith(run.id, expect.objectContaining({ nativeLeaseOwner: claim.leaseOwner,
onNativeExecutionLeaseAcquired: expect.any(Function) }));
await vi.waitFor(() => expect(native.detach).not.toHaveBeenCalled());
} finally {
finish();
await Promise.all([...deps.activeRunExecutionPromises]);
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
}
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
it.each(["bootstrap_incomplete", "blocked"] as const)("retries durable remote takeover and preserves its %s disposition", async kind => {
const run = await seedRun();
const deps = callbacks(db);
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId }).where(eq(heartbeatRuns.id, run.id));
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!,
phase: "observed", recoveryState: "awaiting_runner_reattach", leaseOwner: "prior-server",
leaseExpiresAt: new Date(Date.now() + 60_000) });
await db.insert(nativeRunnerCheckpoints).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!,
part: "opaque-checkpoint-record", material: { fixture: true } });
const recovery = createHeartbeatRecovery(db, deps);
try {
await recovery.reapOrphanedRuns();
expect(native.claims).not.toHaveBeenCalled();
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1) }).where(eq(nativeRunFinalizations.runId, run.id));
const claim: NativeRestartRecoveryDisposition = kind === "blocked"
? { kind, runId: run.id, reason: "fixture_ambiguous_owner" }
: { kind, runId: run.id, leaseOwner: "new-server",
controllerGeneration: 2, providerAttempt: 0, restartKind: "hard", recoveryRequestId: null };
// Force the old inventory down the unverified-owner path unless recovery
// dispositions fence it. The runtime's decision must own this run.
await db.update(nativeRunFinalizations).set({ recoveryState: null }).where(eq(nativeRunFinalizations.runId, run.id));
native.claims.mockResolvedValue([claim]);
await recovery.reapOrphanedRuns();
expect(native.claims).toHaveBeenCalledWith(expect.objectContaining({ runIds: [run.id] }));
if (kind === "blocked") expect(deps.executeRun).not.toHaveBeenCalled();
else expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(run.id, { nativeLeaseOwner: "new-server", nativeRestartRecovery: claim });
expect(deps.setRunStatusFromLive).not.toHaveBeenCalled();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].errorCode)
.not.toBe(NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE);
await Promise.all([...deps.activeRunExecutionPromises]);
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, run.id));
native.claims.mockClear();
await recovery.reapOrphanedRuns();
expect(native.claims).not.toHaveBeenCalled();
} finally {
await Promise.all([...deps.activeRunExecutionPromises]);
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
}
});
it("notifies a replacement after committing the departing controller's released lease", async () => {
const run = await seedRun(), deps = callbacks(db);
const identity = await currentNativeControllerIdentity();
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId }).where(eq(heartbeatRuns.id, run.id));
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!,
phase: "observed", controllerBootId: identity.bootId, leaseOwner: "departing-server", leaseExpiresAt: new Date(Date.now() + 60_000) });
const wake = vi.fn(), unsubscribe = subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeProcessSessions, wake);
try {
const before = new Date();
await createHeartbeatRecovery(db, deps).drainRunningRunsForShutdown("SIGTERM", before, [run.id]);
expect(native.detach).toHaveBeenCalledWith([run.id]);
expect(wake).toHaveBeenCalled();
const [saved] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
expect(saved.leaseExpiresAt).toEqual(before);
expect(saved.recoveryState).toBe("awaiting_runner_reattach");
} finally {
unsubscribe();
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
}
});
it("does not detach a different server's native run during shutdown", async () => {
const run = await seedRun();
const deps = callbacks(db);
const bootId = randomUUID();
const expiresAt = new Date(Date.now() + 60_000);
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId }).where(eq(heartbeatRuns.id, run.id));
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!,
phase: "observed", controllerBootId: bootId, leaseOwner: "other-server", leaseExpiresAt: expiresAt });
try {
await createHeartbeatRecovery(db, deps).drainRunningRunsForShutdown("SIGTERM", new Date(), [run.id]);
expect(native.detach).not.toHaveBeenCalled();
expect(deps.releaseEnvironmentLeasesForRun).not.toHaveBeenCalled();
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id)))[0])
.toMatchObject({ controllerBootId: bootId, leaseOwner: "other-server", leaseExpiresAt: expiresAt, recoveryState: null });
} finally {
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
}
});
async function seedOwnedNativeRun() {
const run = await seedRun();
const identity = await currentNativeControllerIdentity();
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
const [nativeRun] = await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId, nativeSessionId: randomUUID() })
.where(eq(heartbeatRuns.id, run.id)).returning();
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!,
phase: "observed", controllerBootId: identity.bootId, leaseOwner: "departing-server", leaseExpiresAt: new Date(Date.now() + 60_000) });
return nativeRun;
}
async function removeNativeRun(runId: string) {
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
}
const finalization = async (runId: string) =>
(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0];
const drainHasStarted = () => new Promise(resolve => setTimeout(resolve, 250));
it("does not count an early hand-off as suspended while its detach is still pending at the shutdown drain", async () => {
const run = await seedOwnedNativeRun(), recovery = createHeartbeatRecovery(db, callbacks(db));
let finishDetach!: () => void;
native.detach.mockImplementationOnce(() => new Promise<number>(resolve => { finishDetach = () => resolve(0); }));
try {
const early = recovery.handOffNativeRunsForShutdown("SIGTERM");
await vi.waitFor(() => expect(native.detach).toHaveBeenCalledTimes(1));
let drained = false;
const drain = recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), [run.id]).finally(() => { drained = true; });
await drainHasStarted();
expect(drained).toBe(false);
expect((await finalization(run.id)).recoveryState).toBeNull();
finishDetach();
expect(await early).toEqual([run.id]);
expect((await drain).restartSuspendedRunIds).toEqual([run.id]);
expect(native.detach).toHaveBeenCalledTimes(1);
expect(await finalization(run.id)).toMatchObject({ recoveryState: "awaiting_runner_reattach" });
} finally {
finishDetach?.();
await removeNativeRun(run.id);
}
});
it("hands off a run at the shutdown drain when its pending early hand-off fails", async () => {
const run = await seedOwnedNativeRun(), recovery = createHeartbeatRecovery(db, callbacks(db));
let failDetach!: (error: Error) => void;
native.detach.mockImplementationOnce(() => new Promise<number>((_resolve, reject) => { failDetach = reject; }));
try {
const early = recovery.handOffNativeRunsForShutdown("SIGTERM");
await vi.waitFor(() => expect(native.detach).toHaveBeenCalledTimes(1));
const drain = recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), [run.id]);
await drainHasStarted();
failDetach(new Error("runner detach failed"));
expect(await early).toEqual([]);
expect((await drain).restartSuspendedRunIds).toEqual([run.id]);
expect(native.detach).toHaveBeenCalledTimes(2);
expect(await finalization(run.id)).toMatchObject({ recoveryState: "awaiting_runner_reattach" });
} finally {
await removeNativeRun(run.id);
}
});
it("writes no hot-restart snapshot and detaches nothing after a timed-out preparation hands over to the drain", async () => {
const run = await seedOwnedNativeRun(), recovery = createHeartbeatRecovery(db, callbacks(db));
hotRestart.read.mockResolvedValue({ version: 1, requestedAt: new Date().toISOString(), previousServerPid: process.pid,
previousServerVersion: null, drainRequired: false, requestedByRunId: null, preflightActiveRunIds: [] });
let finishIdleClose!: () => void;
native.closeIdle.mockImplementationOnce(() => new Promise(resolve => { finishIdleClose = () => resolve({ closed: 0, failed: 0 }); }));
let preparation: ReturnType<typeof recovery.prepareHotRestartShutdown> | undefined;
try {
const step = await runBoundedShutdownStep({ step: "hot_restart_prepare", signal: "SIGTERM", timeoutMs: 50,
log: { info: vi.fn(), error: vi.fn() },
run: abort => (preparation = recovery.prepareHotRestartShutdown("SIGTERM", new Date(), abort)) });
expect(step.outcome).toBe("timed_out");
expect((await recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), [run.id])).restartSuspendedRunIds).toEqual([run.id]);
expect(native.detach).toHaveBeenCalledTimes(1);
finishIdleClose();
await expect(preparation).rejects.toThrow("timed out");
expect(hotRestart.write).not.toHaveBeenCalled();
expect(native.detach).toHaveBeenCalledTimes(1);
} finally {
finishIdleClose?.();
await preparation?.catch(() => undefined);
await removeNativeRun(run.id);
}
});
it("records no more hot-restart run events after the preparation times out during its first event", async () => {
const runs = [await seedOwnedNativeRun(), await seedOwnedNativeRun(), await seedOwnedNativeRun()];
const deps = callbacks(db), recovery = createHeartbeatRecovery(db, deps);
hotRestart.read.mockResolvedValue({ version: 1, requestedAt: new Date().toISOString(), previousServerPid: process.pid,
previousServerVersion: null, drainRequired: false, requestedByRunId: null, preflightActiveRunIds: [] });
let finishFirstEvent!: () => void;
const firstEvent = new Promise<undefined>(resolve => { finishFirstEvent = () => resolve(undefined); });
deps.appendRunEvent.mockImplementationOnce(() => firstEvent);
const hotRestartEvents = () => deps.appendRunEvent.mock.calls
.filter(([, event]) => event.message?.startsWith("Hot restart requested") === true).length;
let preparation: ReturnType<typeof recovery.prepareHotRestartShutdown> | undefined;
try {
const step = await runBoundedShutdownStep({ step: "hot_restart_prepare", signal: "SIGTERM", timeoutMs: 2_000,
log: { info: vi.fn(), error: vi.fn() },
run: abort => (preparation = recovery.prepareHotRestartShutdown("SIGTERM", new Date(), abort)) });
expect(step.outcome).toBe("timed_out");
expect(hotRestartEvents()).toBe(1);
const runIds = runs.map(run => run.id);
expect((await recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), runIds)).restartSuspendedRunIds.sort())
.toEqual([...runIds].sort());
finishFirstEvent();
await expect(preparation).rejects.toThrow("timed out");
expect(hotRestartEvents()).toBe(1);
expect(native.detach).toHaveBeenCalledTimes(runs.length);
} finally {
finishFirstEvent?.();
await preparation?.catch(() => undefined);
for (const run of runs) await removeNativeRun(run.id);
}
});
it("does not dispatch recovery for a run still preparing in this heartbeat process", async () => {
const run = await seedRun();
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, run.issueId!));
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId }).where(eq(heartbeatRuns.id, run.id));
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!, phase: "observed" });
const deps = callbacks(db);
deps.activeRunExecutions.add(run.id);
deps.getRun.mockImplementation(async id => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0] ?? null);
native.claims.mockImplementationOnce(claimNativeRestartRecoveries);
const before = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
try {
const result = await createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart();
expect(result.awaitingEvidenceRunIds).toEqual([run.id]);
expect(result.claims).toEqual([]);
expect(deps.executeRun).not.toHaveBeenCalled();
expect(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id))).toEqual(before);
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, run.issueId!))).toHaveLength(0);
expect((await deps.getRun(run.id))?.status).toBe("running");
} finally {
deps.activeRunExecutions.delete(run.id);
await Promise.all([...deps.activeRunExecutionPromises]);
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
}
});
it.each(["getRun", "appendRunEvent"] as const)("dispatches committed claims when recording a transition fails in %s", async failing => {
const deps = callbacks(db);
const deferred = { kind: "awaiting_evidence" as const, runId: randomUUID(), reason: NATIVE_RESTART_RECOVERY_FAILED,
retryAt: new Date(Date.now() + 30_000) };
const claim: NativeRestartRecoveryClaim = { kind: "bootstrap_incomplete", runId: randomUUID(), leaseOwner: "claimed-owner",
controllerGeneration: 1, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null };
native.claims.mockResolvedValueOnce([deferred, claim]);
if (failing === "getRun") deps.getRun.mockRejectedValue(new Error("run read failed"));
else {
deps.getRun.mockImplementation(async id => ({ id }) as unknown as Awaited<ReturnType<HeartbeatRecoveryDependencies["getRun"]>>);
deps.appendRunEvent.mockRejectedValue(new Error("event write failed"));
}
const result = await createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart();
expect(result.claims).toEqual([claim]);
expect(result.awaitingEvidenceRunIds).toEqual([deferred.runId]);
expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(claim.runId, { nativeLeaseOwner: claim.leaseOwner, nativeRestartRecovery: claim });
if (failing === "appendRunEvent") {
expect(deps.appendRunEvent).toHaveBeenCalledWith(expect.objectContaining({ id: deferred.runId }), expect.objectContaining({
level: "warn", payload: expect.objectContaining({ reason: NATIVE_RESTART_RECOVERY_FAILED }) }));
}
await Promise.all([...deps.activeRunExecutionPromises]);
});
it("does not claim or execute recovery when persisted finalization fails", async () => {
const deps = callbacks(db);
native.finalizations.mockRejectedValueOnce(new Error("result settlement failed"));
await expect(createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart()).rejects.toThrow("result settlement failed");
expect(native.claims).not.toHaveBeenCalled();
expect(deps.executeRun).not.toHaveBeenCalled();
expect(native.cleanups).not.toHaveBeenCalled();
});
it("keeps blocked and ambiguous ownership out of execution", async () => {
const deps = callbacks(db);
const dispositions: NativeRestartRecoveryDisposition[] = [
{ kind: "blocked", runId: randomUUID(), reason: "conflicting_owner" },
{ kind: "awaiting_evidence", runId: randomUUID(), reason: "missing_stop_evidence" },
];
native.claims.mockResolvedValue(dispositions);
const result = await createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart();
expect(result.blockedRunIds).toEqual([dispositions[0].runId]);
expect(result.awaitingEvidenceRunIds).toEqual([dispositions[1].runId]);
expect(result.claims).toEqual([]);
expect(deps.executeRun).not.toHaveBeenCalled();
await Promise.all([...deps.activeRunExecutionPromises]);
});
it.each(["future", "overdue-owner"])("rearms the persisted native retry deadline through the service callback: %s", async scenario => {
const run = await seedRun();
const nextAttemptAt = new Date(Date.now() + (scenario === "future" ? 60_000 : -1_000));
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId, status: "failed" }).where(eq(heartbeatRuns.id, run.id));
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!, phase: "retryable_failure", nextAttemptAt,
failureCode: scenario === "overdue-owner" ? "native_controller_recovery_waiting_for_owner" : null });
const deps = callbacks(db);
const result = await createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart();
expect(result.scheduledRetryRunIds).toEqual([run.id]);
expect(deps.scheduleNativeSessionResumeDispatch).toHaveBeenCalledWith(run.id, nextAttemptAt);
expect(deps.executeRun).not.toHaveBeenCalled();
await Promise.all([...deps.activeRunExecutionPromises]);
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
});
it("schedules deferred remote retries without repeating unchanged transition events", async () => {
const run = await seedRun();
const deps = callbacks(db), retryAt = new Date(Date.now() + 30_000);
deps.getRun.mockResolvedValue(run);
const held = { kind: "blocked" as const, runId: run.id, reason: "native_retry_admission_ineligible", retryAt };
native.claims.mockResolvedValue([{ ...held, transitionUnchanged: false }]);
const recovery = createHeartbeatRecovery(db, deps);
await recovery.recoverNativeRunsAfterRestart(new Date(), [run.id], { skipDelegatedWork: true });
expect(deps.scheduleNativeSessionResumeDispatch).toHaveBeenCalledWith(run.id, retryAt, "restart");
expect(deps.appendRunEvent).toHaveBeenCalledTimes(1);
native.claims.mockResolvedValue([{ ...held, transitionUnchanged: true }]);
await recovery.recoverNativeRunsAfterRestart(new Date(), [run.id], { skipDelegatedWork: true });
expect(deps.appendRunEvent).toHaveBeenCalledTimes(1);
expect(deps.executeRun).not.toHaveBeenCalled();
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
});
it.each(["nativeWorkspaceSync", "remoteWorkspace"])("routes a due %s retry to remote ownership verification", async reference => {
const run = await seedRun();
const leaseId = randomUUID(), workspaceId = randomUUID();
await db.update(agents).set({ adapterType: "paperclip_runner", status: "active" }).where(eq(agents.id, run.agentId));
await db.update(issues).set({ status: "in_progress", assigneeAgentId: run.agentId,
executionRunId: null, checkoutRunId: null }).where(eq(issues.id, run.issueId!));
const location = { workspaceId, leaseId, providerLeaseId: "retained-sandbox", remoteCwd: "/workspace", state: "prepared" };
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId, status: "failed",
processPid: process.pid, processStartedAt: new Date(), runnerProfileJson: {
[reference]: reference === "remoteWorkspace"
? { ...location, schema: "paperclip.remote-workspace-run.v1", epoch: 1, snapshotId: null }
: { ...location, schema: "paperclip.native-workspace-sync/v3", descriptorSha256: "a".repeat(64),
baselineSha256: "b".repeat(64), finalHostSha256: null, resourceDisposition: "destroy" },
} }).where(eq(heartbeatRuns.id, run.id));
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!,
phase: "retryable_failure", nextAttemptAt: new Date(0),
failureDetail: { recoveryMode: "exact_checkpoint_resume" } });
const deps = callbacks(db);
deps.getRun.mockImplementation(async id => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0] ?? null);
native.claims.mockImplementation(claimNativeRestartRecoveries);
await createHeartbeatRecovery(db, deps).resumeRetryableNativeProcessSessions({ runId: run.id });
expect(native.claims).toHaveBeenCalledWith(expect.objectContaining({ runIds: [run.id] }));
expect(deps.executeRun).not.toHaveBeenCalled();
expect((await deps.getRun(run.id))?.processPid).toBe(process.pid);
expect((await db.select().from(issues).where(eq(issues.id, run.issueId!)))[0].executionRunId).toBeNull();
await Promise.all([...deps.activeRunExecutionPromises]);
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
});
it("retains a live remote owner's actual lease deadline without dispatching competing work", async () => {
const deps = callbacks(db);
const runId = randomUUID(), retryAt = new Date(Date.now() + 60_000);
native.claims.mockResolvedValue([{ kind: "awaiting_evidence", runId, reason: "remote_controller_lease_live", retryAt }]);
const result = await createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart();
expect(result.awaitingEvidenceRunIds).toEqual([runId]);
expect(deps.scheduleNativeSessionResumeDispatch).toHaveBeenCalledWith(runId, retryAt, "restart");
expect(deps.executeRun).not.toHaveBeenCalled();
});
it("retains shutdown ownership until background native cleanup physically settles", async () => {
const deps = callbacks(db);
let settle!: () => void;
native.cleanups.mockReturnValueOnce(new Promise<void>(resolve => { settle = resolve; }));
await createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart();
await vi.waitFor(() => expect(native.cleanups).toHaveBeenCalled());
expect(deps.activeRunExecutionPromises.size).toBe(1);
settle();
await Promise.all([...deps.activeRunExecutionPromises]);
await vi.waitFor(() => expect(deps.activeRunExecutionPromises.size).toBe(0));
});
});