mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime releases owners of failed runs. When the owner's controller is in a process space that Paperclip cannot verify, it shows controller namespace attention on the task instead. > - On a hosted instance with zero active runs, this writer made about 21% of all work signals. It inserted a new active attention row for the same task about every 26 seconds. > - The tasks were already done or cancelled. Recovery resolved each row as `source_terminal`. The active slot was then free, so the next owner-release pass inserted a new row and published a graph recovery signal again. The loop did not stop. > - A live task had the same defect: after the fresh-continuation fallback resolved its attention, the next pass made the same incident again. > - This pull request stops the writer from surfacing attention for finished tasks. It also stops the writer from making an incident again when that run authority was resolved after its lease expired. Those owners are no longer release candidates and do not set the release deadline. > - The benefit is that an idle instance does not write or publish anything for these runs. Attention still shows once for a live task. ## Linked Issues or Issue Description **What happened?** The failed-run owner release (`settleFailedRunOwner`, expired-owner path) called `surfaceNativeControllerNamespaceAttention` for every expired foreign owner that had no active attention. It did this even when the source task was `done` or `cancelled`. Recovery resolved that attention as `source_terminal`. The next pass found no active attention and inserted a new board row. Each insert published `notifyIssueGraphRecovery` and the other execution-control signals. The stranded-task check then ran again for the task graph. A hosted instance collected about 45,000 rows on 13 finished tasks in this way. **Expected behavior** A pass that changes nothing writes nothing and publishes nothing. A finished task never gets controller namespace attention. A live task gets one incident for each exact run authority. A resolved incident stays resolved until the authority changes. **Steps to reproduce** 1. Make a native run that failed, with an `observed` coordinator whose foreign lease has expired and whose controller host is not recorded. 2. Mark the task done or cancelled, or let the fresh-continuation fallback resolve the attention of a live task. 3. Run the execution-finalization worker with the issue recovery workers for 30 minutes. 4. On master, each task gets about 40 to 60 new attention rows. The writer publishes thousands of signals. ## What Changed - `surfaceNativeControllerNamespaceAttention` (shared helper): before it creates attention, it checks for a resolved incident for this run that was resolved at or after the owner's current lease expiry. If one exists, it writes nothing. - New `settledNativeControllerNamespaceAttention` SQL predicate. It is shared by the helper, the release candidate page, and the release deadline query, so all three use the same rule. - `settleFailedRunOwner` (expired-owner path) does not surface attention when the task is `done` or `cancelled`. The restart-recovery claim path has the same guard. - `releaseAbandonedNativeOwners` and `nextAbandonedNativeOwnerReleaseAt` skip foreign owners whose task is finished or whose incident is settled. A pass that has nothing to do selects nothing, and no immediate deadline is set again. Local owners keep the normal proof-based release. - The owner release proofs did not change. A foreign owner is still never released without proof. - Tests: `server/src/__tests__/native-failed-run-owner-release-idle.test.ts` (real PostgreSQL, real coordinator and workers, fake clock, statement and signal counters) and three new cases in `native-failed-run-owner-release.test.ts`. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/native-failed-run-owner-release-idle.test.ts src/services/native-runtime/native-failed-run-owner-release.test.ts src/services/native-runtime/native-restart-recovery.test.ts src/services/native-runtime/native-restart-recovery-isolation.test.ts`: 80 tests pass. - Red on master, green with this change. The test seeds one done task and one cancelled task (each with an earlier resolved `source_terminal` incident) and one live task, then runs 30 idle minutes: - master: 126 attention inserts, 249 attention writes, 7,024 signals, 8,025 statements. - this branch: 1 attention insert (the live task, surfaced once), 2 attention writes (that insert and the fallback's resolution), 36 signals, 350 statements. All of this is startup work. Nothing repeats. - After the fresh continuation settles a live task's attention, six unrelated finalization wakes over 30 minutes: - master: 3 attention inserts, 90 signals, 551 statements, and the release deadline stays due. - this branch: 0 attention writes, 0 run writes, 6 signals (the test's own wakes), 153 statements, and the release deadline is null. - New unit cases: a done or cancelled task never surfaces, and surfaces once when it is reopened. A settled incident stays settled over repeated passes. A renewed authority (a new lease expiry) surfaces one new incident. - The three new unit cases fail on master and pass on this branch. - `tsc --noEmit` for `server` passes. ## Risks - Low risk. The change only removes writes. It does not weaken a release proof, an authorization, a budget, a pause, or an approval check. - An incident that a person resolved or dismissed after the lease expired is not made again for the same authority. A renewed authority (a later lease expiry) still makes a new incident. - A superseded incident (status `cancelled`) still shows again when the superseding action clears. This keeps the current behavior. - The historical rows that the loop made are not removed. This change only stops new rows. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #21621, #21616, #21617 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
588 lines
37 KiB
TypeScript
588 lines
37 KiB
TypeScript
import { spawn } from "node:child_process";
|
|
import { randomUUID } from "node:crypto";
|
|
import { once } from "node:events";
|
|
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { and, eq } from "drizzle-orm";
|
|
import {
|
|
activityLog,
|
|
agents,
|
|
companies,
|
|
createDb,
|
|
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
|
|
environmentLeases,
|
|
heartbeatRuns,
|
|
issues,
|
|
issueRecoveryActions,
|
|
nativeRunFinalizations,
|
|
subscribeDatabaseWork,
|
|
} from "@paperclipai/db";
|
|
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
|
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
|
|
import { agentExecutionsHaveStopped } from "../agent-execution-stop.js";
|
|
import { readProcessStartedAt } from "../hot-restart.js";
|
|
import { currentProcessSpaceHash } from "../controller-process-identity.js";
|
|
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
|
|
import { environmentService } from "../environments.js";
|
|
import { executionControlSchedule } from "../execution-control-work.js";
|
|
import { reconcileAbandonedExecutionControl } from "../execution-control-reconciliation.js";
|
|
import { claimNativeSessionResumptions } from "./native-finalization-reconciler.js";
|
|
import {
|
|
nextAbandonedNativeOwnerReleaseAt,
|
|
releaseAbandonedNativeOwners,
|
|
resetAbandonedNativeOwnerReleaseForTests,
|
|
settleUnstartedNativeSessionResume,
|
|
} from "./native-failed-run-owner-release.js";
|
|
|
|
import * as controllerProcess from "../controller-process-identity.js";
|
|
|
|
const LEASE_FAILURE = 'Failed to acquire lease for environment "Managed computer" (sandbox): Sandbox creation has failed';
|
|
|
|
describe("failed native run owner release", () => {
|
|
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
|
let db: ReturnType<typeof createDb>;
|
|
let companyId: string;
|
|
let agentId: string;
|
|
|
|
beforeAll(async () => {
|
|
temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-owner-release-");
|
|
db = createDb(temporary.connectionString);
|
|
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
|
afterAll(async () => temporary?.cleanup());
|
|
// The release deadline is instance-wide; keep each case's rows to itself.
|
|
afterEach(async () => {
|
|
resetAbandonedNativeOwnerReleaseForTests();
|
|
await db.delete(environmentLeases);
|
|
await db.delete(nativeRunFinalizations);
|
|
});
|
|
|
|
beforeEach(async () => {
|
|
companyId = randomUUID();
|
|
agentId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Owner release", issuePrefix: `OR${companyId.slice(0, 6)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Resume agent", adapterType: "codex_local", status: "running" });
|
|
});
|
|
|
|
async function nativeRun(input: {
|
|
status: string;
|
|
errorCode?: string;
|
|
error?: string;
|
|
coordinator: Partial<typeof nativeRunFinalizations.$inferInsert>;
|
|
runId?: string;
|
|
}) {
|
|
const issueId = randomUUID();
|
|
const runId = input.runId ?? randomUUID();
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Resume the native run", status: "in_progress",
|
|
assigneeAgentId: agentId, workMode: "standard" });
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId, companyId, agentId, nativeIssueId: issueId, status: input.status, runtimeMode: "native",
|
|
runtimeModeResolvedAt: new Date(), contextSnapshot: { issueId },
|
|
startedAt: new Date(), finishedAt: input.status === "running" ? null : new Date(),
|
|
errorCode: input.errorCode ?? null, error: input.error ?? null,
|
|
runnerProfileJson: {
|
|
mode: "native",
|
|
nativeExecutionInput: { schema: "paperclip.native-execution-input.v1", binding: { runId } },
|
|
sessionCheckpoint: { backendKind: "codex_app_server", sessionId: "persisted-session", identity: { runId },
|
|
providerSessionId: "provider-session", activeTurnId: "active-turn" },
|
|
},
|
|
});
|
|
const current = await controllerProcess.currentControllerProcess();
|
|
expect(current.hostHash).not.toBe("unknown");
|
|
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", attempt: 1,
|
|
controllerHostHash: current.hostHash, ...input.coordinator } as typeof nativeRunFinalizations.$inferInsert);
|
|
return { runId, issueId };
|
|
}
|
|
|
|
const coordinator = async (runId: string) =>
|
|
(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0]!;
|
|
|
|
/** What the heartbeat setup-failure path writes; it never touches the coordinator. */
|
|
async function failRunDuringSetup(runId: string, errorCode = "setup_failed", error = LEASE_FAILURE) {
|
|
await db.update(heartbeatRuns).set({ status: "failed", errorCode, error, finishedAt: new Date() })
|
|
.where(eq(heartbeatRuns.id, runId));
|
|
}
|
|
|
|
it("releases a session resume that failed at environment lease acquisition", async () => {
|
|
const { runId } = await nativeRun({ status: "failed", errorCode: "native_session_interrupted",
|
|
coordinator: { phase: "retryable_failure", nextAttemptAt: new Date(0), recoveryState: "resuming_session" } });
|
|
const [claim] = await claimNativeSessionResumptions({ db, runnerInstanceId: "default", runIds: [runId] });
|
|
expect(claim).toEqual({ runId, leaseOwner: expect.stringContaining("default:resume:"), attempt: 1 });
|
|
expect(await coordinator(runId)).toMatchObject({ phase: "observed", leaseOwner: claim!.leaseOwner });
|
|
await failRunDuringSetup(runId);
|
|
// This is the leaked state that kept the agent in `pausing`.
|
|
expect(await agentExecutionsHaveStopped(db, [agentId])).toBe(false);
|
|
|
|
await expect(settleUnstartedNativeSessionResume(db, claim!)).resolves.toBe(true);
|
|
|
|
expect(await coordinator(runId)).toMatchObject({
|
|
phase: "terminal_failure", leaseOwner: null, leaseExpiresAt: null, recoveryState: "blocked",
|
|
failureCode: "setup_failed", resultId: null,
|
|
failureDetail: expect.objectContaining({ message: LEASE_FAILURE, originalFailureCode: "setup_failed",
|
|
releasedOwner: { reason: "unstarted_resume", leaseOwner: claim!.leaseOwner, attempt: 1 } }),
|
|
});
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0])
|
|
.toMatchObject({ status: "failed", nativePhase: "terminal_failure" });
|
|
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, companyId),
|
|
eq(activityLog.action, "native.failed_run_owner_released")))).toHaveLength(1);
|
|
expect(await agentExecutionsHaveStopped(db, [agentId])).toBe(true);
|
|
// Settlement is idempotent.
|
|
await expect(settleUnstartedNativeSessionResume(db, claim!)).resolves.toBe(false);
|
|
});
|
|
|
|
it("leaves an owner the native executor claimed, and a run that is still running", async () => {
|
|
const { runId } = await nativeRun({ status: "failed",
|
|
coordinator: { phase: "retryable_failure", nextAttemptAt: new Date(0) } });
|
|
const [claim] = await claimNativeSessionResumptions({ db, runnerInstanceId: "default", runIds: [runId] });
|
|
// Still preparing: the run has not ended.
|
|
await expect(settleUnstartedNativeSessionResume(db, claim!)).resolves.toBe(false);
|
|
// A failed run whose recorded process is still alive keeps its owner.
|
|
await failRunDuringSetup(runId);
|
|
const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1_000)"], { stdio: "ignore" });
|
|
await once(child, "spawn");
|
|
try {
|
|
await db.update(heartbeatRuns).set({ processPid: child.pid! }).where(eq(heartbeatRuns.id, runId));
|
|
await expect(settleUnstartedNativeSessionResume(db, claim!)).resolves.toBe(false);
|
|
} finally {
|
|
const exited = once(child, "exit");
|
|
child.kill("SIGKILL");
|
|
await exited;
|
|
}
|
|
await db.update(heartbeatRuns).set({ status: "running", processPid: null }).where(eq(heartbeatRuns.id, runId));
|
|
// The executor's claim bumps the attempt; its own failure path owns settlement.
|
|
await db.update(nativeRunFinalizations).set({ attempt: 2 }).where(eq(nativeRunFinalizations.runId, runId));
|
|
await failRunDuringSetup(runId, "adapter_failed", "provider turn failed");
|
|
await expect(settleUnstartedNativeSessionResume(db, claim!)).resolves.toBe(false);
|
|
expect(await coordinator(runId)).toMatchObject({ phase: "observed", leaseOwner: claim!.leaseOwner, attempt: 2 });
|
|
});
|
|
|
|
it("reaps an already leaked owner on a failed run so the agent can stop", async () => {
|
|
const leaseOwner = `default:resume:${randomUUID()}`;
|
|
const { runId } = await nativeRun({ status: "failed", errorCode: "adapter_failed",
|
|
error: "runner_state_identity_mismatch: session_scope_mismatch",
|
|
coordinator: { leaseOwner, leaseExpiresAt: new Date(Date.now() - 60_000), recoveryState: "resuming_session" } });
|
|
expect(await agentExecutionsHaveStopped(db, [agentId])).toBe(false);
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).not.toBeNull();
|
|
|
|
const reconciled = await reconcileAbandonedExecutionControl(db);
|
|
expect(reconciled.releasedOwners).toBe(1);
|
|
|
|
expect(await coordinator(runId)).toMatchObject({ phase: "terminal_failure", leaseOwner: null, leaseExpiresAt: null,
|
|
recoveryState: "blocked", failureCode: "adapter_failed",
|
|
failureDetail: expect.objectContaining({ releasedOwner: { reason: "expired_owner", leaseOwner, attempt: 1 } }) });
|
|
expect(await agentExecutionsHaveStopped(db, [agentId])).toBe(true);
|
|
// Nothing is left to schedule once the owner is settled.
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
|
|
});
|
|
|
|
it.each(["null", "foreign", "unknown-current"] as const)("keeps a failed namespace-held source out of automatic expired-owner release: %s", async mode => {
|
|
const actual = await controllerProcess.currentControllerProcess();
|
|
expect(actual.hostHash).not.toBe("unknown");
|
|
const recordedPid = 2_000_000_001;
|
|
const { runId } = await nativeRun({ status: "failed", errorCode: "setup_failed",
|
|
coordinator: { leaseOwner: `foreign-controller:${randomUUID()}`, leaseExpiresAt: new Date(0),
|
|
controllerBootId: randomUUID(), controllerPid: recordedPid,
|
|
controllerHostHash: mode === "null" ? null : mode === "foreign" ? `foreign:${actual.hostHash}` : actual.hostHash,
|
|
recoveryState: "awaiting_evidence" } });
|
|
await db.update(heartbeatRuns).set({ processPid: recordedPid, processGroupId: recordedPid })
|
|
.where(eq(heartbeatRuns.id, runId));
|
|
const before = await coordinator(runId);
|
|
const originalKill = process.kill.bind(process);
|
|
const probes: number[] = [];
|
|
const probe = vi.spyOn(process, "kill").mockImplementation((pid, signal) => {
|
|
if (Math.abs(pid) === recordedPid && signal === 0) {
|
|
probes.push(pid);
|
|
throw Object.assign(new Error("controlled local PID absent; not foreign stop proof"), { code: "ESRCH" });
|
|
}
|
|
return originalKill(pid, signal);
|
|
});
|
|
const identity = mode === "unknown-current" ? vi.spyOn(controllerProcess, "currentProcessSpaceHash")
|
|
.mockReturnValue("unknown") : null;
|
|
try {
|
|
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
|
|
expect(probes).toEqual([]);
|
|
expect(await coordinator(runId)).toEqual(before);
|
|
const [attention] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, before.issueId));
|
|
expect(attention).toMatchObject({ cause: "native_restart_recovery_blocked", attemptCount: 0, wakePolicy: null, monitorPolicy: null,
|
|
evidence: { runId, reason: "controller_process_space_unverified" } });
|
|
for (let inspection = 0; inspection < 3; inspection++) {
|
|
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
|
|
expect(await executionControlSchedule(db).finalization()).toBeNull();
|
|
}
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, before.issueId))).toEqual([attention]);
|
|
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, companyId),
|
|
eq(activityLog.action, "native.failed_run_owner_released")))).toHaveLength(0);
|
|
} finally { identity?.mockRestore(); probe.mockRestore(); }
|
|
});
|
|
|
|
it.each(["timed_out", "interrupted"] as const)("publishes quiet operator attention after an already %s owner expires", async status => {
|
|
const { runId, issueId } = await nativeRun({ status, coordinator: { leaseOwner: `foreign:${randomUUID()}`,
|
|
controllerBootId: randomUUID(), controllerHostHash: null, leaseExpiresAt: new Date(0) } });
|
|
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
|
|
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
|
|
expect(action).toMatchObject({ ownerType: "board", cause: "native_restart_recovery_blocked", attemptCount: 0,
|
|
evidence: { runId, reason: "controller_process_space_unverified", controllerAuthority: { digest: expect.any(String) } } });
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
|
|
expect(await coordinator(runId)).toMatchObject({ phase: "observed", leaseOwner: expect.any(String) });
|
|
});
|
|
|
|
it("keeps empty deadline and release scans from stamping controller ownership", async () => {
|
|
const stamped = vi.fn();
|
|
const unsubscribe = controllerProcess.onControllerStamp(stamped);
|
|
try {
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
|
|
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
|
|
expect(stamped).not.toHaveBeenCalled();
|
|
} finally { unsubscribe(); }
|
|
});
|
|
|
|
it("retains a foreign owner's original deadline until its returned unstarted claim settles", async () => {
|
|
const leaseExpiresAt = new Date(Date.now() + 60_000);
|
|
const leaseOwner = `default:resume:${randomUUID()}`;
|
|
// This dispatched claim returned before the executor took an attempt. Its
|
|
// exact returned token/attempt, not the PID or lease age, authorizes release.
|
|
const { runId, issueId } = await nativeRun({ status: "failed", errorCode: "setup_failed",
|
|
coordinator: { leaseOwner, leaseExpiresAt, controllerBootId: randomUUID(),
|
|
controllerHostHash: `foreign:${currentProcessSpaceHash()}` } });
|
|
const before = await coordinator(runId);
|
|
expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(leaseExpiresAt.getTime());
|
|
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
|
|
expect(await coordinator(runId)).toEqual(before);
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toEqual([]);
|
|
expect(await settleUnstartedNativeSessionResume(db, { runId, leaseOwner, attempt: before.attempt })).toBe(true);
|
|
expect(await coordinator(runId)).toMatchObject({ phase: "terminal_failure", leaseOwner: null });
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toEqual([]);
|
|
});
|
|
|
|
it("surfaces a foreign owner only at its original expiry without stamping or probing it", async () => {
|
|
const leaseExpiresAt = new Date(Date.now() + 60_000);
|
|
const recordedPid = 2_000_000_001;
|
|
const { runId, issueId } = await nativeRun({ status: "failed", coordinator: {
|
|
leaseOwner: `foreign:${randomUUID()}`, leaseExpiresAt, controllerBootId: randomUUID(),
|
|
controllerPid: recordedPid, controllerHostHash: null } });
|
|
const before = await coordinator(runId);
|
|
const stamped = vi.fn();
|
|
const unsubscribe = controllerProcess.onControllerStamp(stamped);
|
|
const kill = process.kill.bind(process);
|
|
const probes: number[] = [];
|
|
const probe = vi.spyOn(process, "kill").mockImplementation((pid, signal) => {
|
|
if (Math.abs(pid) === recordedPid && signal === 0) {
|
|
probes.push(pid);
|
|
throw Object.assign(new Error("foreign PID is not local stop evidence"), { code: "ESRCH" });
|
|
}
|
|
return kill(pid, signal);
|
|
});
|
|
try {
|
|
expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(leaseExpiresAt.getTime());
|
|
expect(await releaseAbandonedNativeOwners(db, new Date(leaseExpiresAt.getTime() - 1))).toBe(0);
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toEqual([]);
|
|
expect(await releaseAbandonedNativeOwners(db, leaseExpiresAt)).toBe(0);
|
|
const [attention] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
|
|
expect(attention).toMatchObject({ status: "active", ownerType: "board", attemptCount: 0,
|
|
wakePolicy: null, monitorPolicy: null, evidence: { runId, reason: "controller_process_space_unverified" } });
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
|
|
expect(await releaseAbandonedNativeOwners(db, leaseExpiresAt)).toBe(0);
|
|
expect(await coordinator(runId)).toEqual(before);
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toEqual([attention]);
|
|
expect(probes).toEqual([]);
|
|
expect(stamped).not.toHaveBeenCalled();
|
|
} finally { probe.mockRestore(); unsubscribe(); }
|
|
});
|
|
|
|
it("finishes each requested namespace inspection pass and preserves the original local deadline", async () => {
|
|
const foreign = [];
|
|
for (let index = 0; index < 5; index++) foreign.push(await nativeRun({ status: "failed", coordinator: {
|
|
leaseOwner: `foreign:${randomUUID()}`, controllerHostHash: null, leaseExpiresAt: new Date(0) } }));
|
|
const localDeadline = new Date(Date.now() + 600_000);
|
|
await nativeRun({ status: "failed", coordinator: { leaseOwner: `local:${randomUUID()}`, leaseExpiresAt: localDeadline } });
|
|
for (let page = 0; page < 2; page++) {
|
|
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
|
|
expect((await nextAbandonedNativeOwnerReleaseAt(db))!.getTime()).toBeLessThan(localDeadline.getTime());
|
|
}
|
|
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
|
|
expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(localDeadline.getTime());
|
|
const rows = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, companyId));
|
|
expect(rows).toHaveLength(foreign.length);
|
|
expect(rows.every(row => row.attemptCount === 0 && row.wakePolicy === null && row.monitorPolicy === null)).toBe(true);
|
|
// Deadline reads do not request another pass or turn surfaced holds into timers.
|
|
for (let read = 0; read < 2; read++) {
|
|
expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(localDeadline.getTime());
|
|
}
|
|
// A later explicit mutation inspection must recheck all retained authorities.
|
|
// Finish its known three-page pass before expecting the local timer again.
|
|
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
|
|
expect((await nextAbandonedNativeOwnerReleaseAt(db))!.getTime()).toBeLessThan(localDeadline.getTime());
|
|
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
|
|
expect((await nextAbandonedNativeOwnerReleaseAt(db))!.getTime()).toBeLessThan(localDeadline.getTime());
|
|
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
|
|
expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(localDeadline.getTime());
|
|
const reinspected = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, companyId));
|
|
const byId = (a: typeof rows[number], b: typeof rows[number]) => a.id.localeCompare(b.id);
|
|
expect(reinspected.sort(byId)).toEqual([...rows].sort(byId));
|
|
});
|
|
|
|
it("keeps a lower-sorting arrival durable when its notification coalesces between inspection pages", async () => {
|
|
for (let index = 1; index <= 4; index++) await nativeRun({ status: "failed",
|
|
runId: `ffffffff-ffff-4fff-8fff-${String(index).padStart(12, "0")}`, coordinator: {
|
|
leaseOwner: `foreign:${randomUUID()}`, controllerHostHash: null, leaseExpiresAt: new Date(0) } });
|
|
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
|
|
const lower = await nativeRun({ status: "failed", runId: "00000000-0000-4000-8000-000000000001", coordinator: {
|
|
leaseOwner: `foreign:${randomUUID()}`, controllerHostHash: null, leaseExpiresAt: new Date(0) } });
|
|
// Simulate one coalesced dirty signal by continuing the actual existing
|
|
// page, not resetting its cursor or creating a second artificial worker.
|
|
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).not.toBeNull();
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, lower.issueId))).toEqual([]);
|
|
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, lower.issueId))).toHaveLength(1);
|
|
expect(await releaseAbandonedNativeOwners(db, new Date(), 5)).toBe(0);
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
|
|
});
|
|
|
|
it("surfaces a preserved namespace waiter when an unrelated action is actually resolved", async () => {
|
|
const { runId, issueId } = await nativeRun({ status: "failed", coordinator: {
|
|
leaseOwner: `foreign:${randomUUID()}`, controllerHostHash: null, leaseExpiresAt: new Date(0) } });
|
|
const service = issueRecoveryActionService(db);
|
|
const unrelated = await service.upsertSourceScoped({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
|
|
cause: "independent-test-recovery", fingerprint: "independent-test-identity", ownerType: "board",
|
|
nextAction: "Resolve this independently owned action first.", wakePolicy: null, monitorPolicy: null, attemptCount: 0 });
|
|
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
|
|
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)))[0].id).toBe(unrelated.id);
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
|
|
expect(await committedWakes(DELIVERY_QUEUES.executionFinalization, () => service.resolveActiveForIssue({ companyId,
|
|
sourceIssueId: issueId, actionId: unrelated.id, status: "resolved", outcome: "restored" }))).toBeGreaterThan(0);
|
|
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
|
|
const rows = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
|
|
expect(rows).toHaveLength(2);
|
|
expect(rows.find(row => row.id === unrelated.id)).toMatchObject({ status: "resolved", fingerprint: "independent-test-identity" });
|
|
expect(rows.find(row => row.id !== unrelated.id)).toMatchObject({ attemptCount: 0,
|
|
evidence: { runId, reason: "controller_process_space_unverified" } });
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
|
|
});
|
|
|
|
it.each(["done", "cancelled"] as const)("never surfaces namespace attention for a %s task until it is reopened", async status => {
|
|
const { runId, issueId } = await nativeRun({ status: "failed", coordinator: {
|
|
leaseOwner: `foreign:${randomUUID()}`, controllerHostHash: null, leaseExpiresAt: new Date(0) } });
|
|
await db.update(issues).set({ status }).where(eq(issues.id, issueId));
|
|
const before = await coordinator(runId);
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
|
|
for (let pass = 0; pass < 3; pass++) {
|
|
expect(await committedWakes(DELIVERY_QUEUES.strandedAssignments, () => releaseAbandonedNativeOwners(db))).toBe(0);
|
|
}
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toEqual([]);
|
|
expect(await coordinator(runId)).toEqual(before);
|
|
expect(await executionControlSchedule(db).finalization()).toBeNull();
|
|
await db.update(issues).set({ status: "in_progress" }).where(eq(issues.id, issueId));
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).not.toBeNull();
|
|
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toEqual([
|
|
expect.objectContaining({ status: "active", evidence: expect.objectContaining({ runId }) })]);
|
|
});
|
|
|
|
it("keeps a settled namespace incident settled and surfaces a renewed authority once", async () => {
|
|
const { runId, issueId } = await nativeRun({ status: "failed", coordinator: {
|
|
leaseOwner: `foreign:${randomUUID()}`, controllerHostHash: null, leaseExpiresAt: new Date(Date.now() - 60_000) } });
|
|
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
|
|
const [surfaced] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
|
|
expect(surfaced).toMatchObject({ status: "active", evidence: { runId } });
|
|
await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: new Date(),
|
|
resolutionNote: "source_terminal" }).where(eq(issueRecoveryActions.id, surfaced.id));
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
|
|
for (let pass = 0; pass < 3; pass++) {
|
|
expect(await committedWakes(DELIVERY_QUEUES.strandedAssignments, () => releaseAbandonedNativeOwners(db))).toBe(0);
|
|
}
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(1);
|
|
const renewedExpiry = new Date(Date.now() + 60_000);
|
|
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: renewedExpiry }).where(eq(nativeRunFinalizations.runId, runId));
|
|
expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(renewedExpiry.getTime());
|
|
const afterRenewal = new Date(renewedExpiry.getTime() + 1);
|
|
expect(await releaseAbandonedNativeOwners(db, afterRenewal)).toBe(0);
|
|
expect(await releaseAbandonedNativeOwners(db, afterRenewal)).toBe(0);
|
|
const rows = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
|
|
expect(rows).toHaveLength(2);
|
|
expect(rows.find(row => row.id !== surfaced.id)).toMatchObject({ status: "active", evidence: { runId } });
|
|
expect(rows.find(row => row.id !== surfaced.id)?.fingerprint).not.toBe(surfaced.fingerprint);
|
|
});
|
|
|
|
it("keeps an unexpired owner blocking the stop check and schedules its expiry", async () => {
|
|
const leaseExpiresAt = new Date(Date.now() + 10 * 60_000);
|
|
const { runId } = await nativeRun({ status: "failed", errorCode: "setup_failed", error: LEASE_FAILURE,
|
|
coordinator: { leaseOwner: `default:resume:${randomUUID()}`, leaseExpiresAt } });
|
|
|
|
await expect(releaseAbandonedNativeOwners(db)).resolves.toBe(0);
|
|
expect(await coordinator(runId)).toMatchObject({ phase: "observed", leaseOwner: expect.any(String) });
|
|
expect(await agentExecutionsHaveStopped(db, [agentId])).toBe(false);
|
|
expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(leaseExpiresAt.getTime());
|
|
expect(await executionControlSchedule(db).finalization()).toBe(leaseExpiresAt.getTime());
|
|
});
|
|
|
|
it("does not reap while the run's environment lease or controller may still be live", async () => {
|
|
const leaseOwner = `default:resume:${randomUUID()}`;
|
|
const { runId, issueId } = await nativeRun({ status: "failed", errorCode: "adapter_failed",
|
|
coordinator: { leaseOwner, leaseExpiresAt: new Date(0) } });
|
|
const [lease] = await db.insert(environmentLeases).values({ companyId, issueId, heartbeatRunId: runId,
|
|
provider: "test", providerLeaseId: `lease-${runId}`, status: "active" }).returning();
|
|
await expect(releaseAbandonedNativeOwners(db)).resolves.toBe(0);
|
|
// An unreleased sandbox is outside this pass, so it is not polled for.
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
|
|
|
|
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success" })
|
|
.where(eq(environmentLeases.id, lease!.id));
|
|
const controller = spawn(process.execPath, ["-e", "setInterval(() => {}, 1_000)"], { stdio: "ignore" });
|
|
await once(controller, "spawn");
|
|
try {
|
|
await db.update(nativeRunFinalizations).set({ controllerPid: controller.pid!, controllerBootId: randomUUID() })
|
|
.where(eq(nativeRunFinalizations.runId, runId));
|
|
await expect(releaseAbandonedNativeOwners(db)).resolves.toBe(0);
|
|
expect(await coordinator(runId)).toMatchObject({ phase: "observed", leaseOwner });
|
|
} finally {
|
|
const exited = once(controller, "exit");
|
|
controller.kill("SIGKILL");
|
|
await exited;
|
|
}
|
|
await expect(releaseAbandonedNativeOwners(db)).resolves.toBe(1);
|
|
expect(await coordinator(runId)).toMatchObject({ phase: "terminal_failure", leaseOwner: null });
|
|
});
|
|
|
|
/** Records committed wake signals for one delivery queue while `work` runs. */
|
|
async function committedWakes(topic: string, work: () => Promise<unknown>) {
|
|
const events: string[] = [];
|
|
const unsubscribe = subscribeDatabaseWork(db, topic, event => { events.push(event); });
|
|
try { await work(); } finally { unsubscribe(); }
|
|
return events.filter(event => event === "settled").length;
|
|
}
|
|
|
|
it("wakes cost accounting when it settles an owner", async () => {
|
|
const { runId } = await nativeRun({ status: "failed", errorCode: "adapter_failed",
|
|
coordinator: { leaseOwner: `default:resume:${randomUUID()}`, leaseExpiresAt: new Date(0) } });
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: true }).where(eq(heartbeatRuns.id, runId));
|
|
expect(await committedWakes(DELIVERY_QUEUES.accounting, () => releaseAbandonedNativeOwners(db))).toBeGreaterThan(0);
|
|
expect(await coordinator(runId)).toMatchObject({ phase: "terminal_failure", leaseOwner: null });
|
|
});
|
|
|
|
it("wakes the finalization queue when the run's environment lease settles", async () => {
|
|
const leaseOwner = `default:resume:${randomUUID()}`;
|
|
const { runId, issueId } = await nativeRun({ status: "failed", errorCode: "adapter_failed",
|
|
coordinator: { leaseOwner, leaseExpiresAt: new Date(0) } });
|
|
const [lease] = await db.insert(environmentLeases).values({ companyId, issueId, heartbeatRunId: runId,
|
|
provider: "test", providerLeaseId: `lease-${runId}`, status: "active" }).returning();
|
|
// Blocked and unscheduled while the sandbox lease is unsettled.
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
|
|
expect(await committedWakes(DELIVERY_QUEUES.executionFinalization,
|
|
() => environmentService(db).releaseLease(lease!.id, "released", { cleanupStatus: "success" }))).toBeGreaterThan(0);
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).not.toBeNull();
|
|
await expect(releaseAbandonedNativeOwners(db)).resolves.toBe(1);
|
|
expect(await agentExecutionsHaveStopped(db, [agentId])).toBe(true);
|
|
});
|
|
|
|
it("ignores a recorded process and group whose PID now belongs to another process", async () => {
|
|
const { runId } = await nativeRun({ status: "failed", errorCode: "adapter_failed",
|
|
coordinator: { leaseOwner: `default:resume:${randomUUID()}`, leaseExpiresAt: new Date(0) } });
|
|
// A detached child leads its own group, as a local native runner does.
|
|
const unrelated = spawn(process.execPath, ["-e", "setInterval(() => {}, 1_000)"], { stdio: "ignore", detached: true });
|
|
await once(unrelated, "spawn");
|
|
try {
|
|
await db.update(heartbeatRuns).set({ processPid: unrelated.pid!, processGroupId: unrelated.pid!,
|
|
processStartedAt: new Date("2020-01-01T00:00:00.000Z") }).where(eq(heartbeatRuns.id, runId));
|
|
await expect(releaseAbandonedNativeOwners(db)).resolves.toBe(1);
|
|
expect(await coordinator(runId)).toMatchObject({ phase: "terminal_failure", leaseOwner: null });
|
|
} finally {
|
|
const exited = once(unrelated, "exit");
|
|
unrelated.kill("SIGKILL");
|
|
await exited;
|
|
}
|
|
});
|
|
|
|
// Phase 2: run processes and controllers are judged by the shared prover.
|
|
it("never releases on a dead PID recorded in another PID space", async () => {
|
|
const { runId } = await nativeRun({ status: "failed", errorCode: "adapter_failed",
|
|
coordinator: { leaseOwner: `default:resume:${randomUUID()}`, leaseExpiresAt: new Date(0) } });
|
|
await db.update(heartbeatRuns).set({ processPid: 2_000_000_001, processGroupId: null,
|
|
processStartedAt: new Date(Date.now() - 60_000), processHostHash: "another-pid-space" })
|
|
.where(eq(heartbeatRuns.id, runId));
|
|
await expect(releaseAbandonedNativeOwners(db)).resolves.toBe(0);
|
|
expect(await coordinator(runId)).toMatchObject({ phase: "observed" });
|
|
await db.update(heartbeatRuns).set({ processHostHash: currentProcessSpaceHash() }).where(eq(heartbeatRuns.id, runId));
|
|
await expect(releaseAbandonedNativeOwners(db)).resolves.toBe(1);
|
|
expect(await db.select().from(activityLog).where(and(eq(activityLog.runId, runId),
|
|
eq(activityLog.action, "legacy_host_unknown_dead_pid_proof")))).toEqual([]);
|
|
});
|
|
|
|
it("keeps the one-release fallback for a run process saved before its PID space, and records it", async () => {
|
|
const { runId } = await nativeRun({ status: "failed", errorCode: "adapter_failed",
|
|
coordinator: { leaseOwner: `default:resume:${randomUUID()}`, leaseExpiresAt: new Date(0) } });
|
|
await db.update(heartbeatRuns).set({ processPid: 2_000_000_001, processGroupId: null,
|
|
processStartedAt: new Date(Date.now() - 60_000), processHostHash: null }).where(eq(heartbeatRuns.id, runId));
|
|
await expect(releaseAbandonedNativeOwners(db)).resolves.toBe(1);
|
|
const proofs = await db.select().from(activityLog).where(and(eq(activityLog.runId, runId),
|
|
eq(activityLog.action, "legacy_host_unknown_dead_pid_proof")));
|
|
expect(proofs.map(proof => proof.details)).toEqual([{ reason: "pid_dead", source: "native_failed_run_process" }]);
|
|
});
|
|
|
|
// `ps -o lstart` on macOS reports whole seconds, while a recorded start can
|
|
// carry milliseconds. A sub-second gap must not be read as a recycled PID.
|
|
it.each(["controller", "run process"])("keeps the owner when the live %s start time differs by under a second", async (holder) => {
|
|
const { runId } = await nativeRun({ status: "failed", errorCode: "adapter_failed",
|
|
coordinator: { leaseOwner: `default:resume:${randomUUID()}`, leaseExpiresAt: new Date(0) } });
|
|
const live = spawn(process.execPath, ["-e", "setInterval(() => {}, 1_000)"], { stdio: "ignore", detached: true });
|
|
await once(live, "spawn");
|
|
try {
|
|
const observed = new Date((await readProcessStartedAt(live.pid!))!);
|
|
const recorded = new Date(observed.getTime() + 500);
|
|
if (holder === "controller") {
|
|
await db.update(nativeRunFinalizations).set({ controllerPid: live.pid!, controllerBootId: randomUUID(),
|
|
controllerProcessStartedAt: recorded }).where(eq(nativeRunFinalizations.runId, runId));
|
|
} else {
|
|
await db.update(heartbeatRuns).set({ processPid: live.pid!, processGroupId: live.pid!, processStartedAt: recorded })
|
|
.where(eq(heartbeatRuns.id, runId));
|
|
}
|
|
await expect(releaseAbandonedNativeOwners(db)).resolves.toBe(0);
|
|
expect(await coordinator(runId)).toMatchObject({ phase: "observed", leaseOwner: expect.any(String) });
|
|
} finally {
|
|
const exited = once(live, "exit");
|
|
live.kill("SIGKILL");
|
|
await exited;
|
|
}
|
|
});
|
|
|
|
it("rotates bounded passes past owners that a live controller still holds", async () => {
|
|
const runs = await Promise.all([0, 1, 2].map(() => nativeRun({ status: "failed", errorCode: "adapter_failed",
|
|
coordinator: { leaseOwner: `default:resume:${randomUUID()}`, leaseExpiresAt: new Date(0) } })));
|
|
const [held, ...free] = [...runs].sort((left, right) => left.runId.localeCompare(right.runId));
|
|
const controller = spawn(process.execPath, ["-e", "setInterval(() => {}, 1_000)"], { stdio: "ignore" });
|
|
await once(controller, "spawn");
|
|
try {
|
|
// The first page holds only the row a live controller still owns.
|
|
await db.update(nativeRunFinalizations).set({ controllerPid: controller.pid!, controllerBootId: randomUUID() })
|
|
.where(eq(nativeRunFinalizations.runId, held!.runId));
|
|
// Each pass handles one bounded page; the cursor moves past the held row.
|
|
const passes = [];
|
|
for (let pass = 0; pass < 3; pass += 1) passes.push(await releaseAbandonedNativeOwners(db, new Date(), 1));
|
|
expect(passes).toEqual([0, 1, 1]);
|
|
// After a full rotation the held row is inspected again, not skipped forever.
|
|
await expect(releaseAbandonedNativeOwners(db, new Date(), 1)).resolves.toBe(0);
|
|
expect(await coordinator(held!.runId)).toMatchObject({ phase: "observed", leaseOwner: expect.any(String) });
|
|
for (const { runId } of free) expect(await coordinator(runId)).toMatchObject({ phase: "terminal_failure", leaseOwner: null });
|
|
} finally {
|
|
const exited = once(controller, "exit");
|
|
controller.kill("SIGKILL");
|
|
await exited;
|
|
}
|
|
});
|
|
|
|
it("leaves cancelled runs and pending Stops to their own stop-proof paths", async () => {
|
|
const { runId: cancelledRunId } = await nativeRun({ status: "cancelled",
|
|
coordinator: { leaseOwner: `default:resume:${randomUUID()}`, leaseExpiresAt: new Date(0) } });
|
|
const { runId: stoppingRunId } = await nativeRun({ status: "interrupted",
|
|
coordinator: { leaseOwner: `default:resume:${randomUUID()}`, leaseExpiresAt: new Date(0) } });
|
|
await db.update(heartbeatRuns).set({ resultJson: { nativeCancellation: { dispatchState: "pending" } } })
|
|
.where(eq(heartbeatRuns.id, stoppingRunId));
|
|
await expect(releaseAbandonedNativeOwners(db)).resolves.toBe(0);
|
|
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
|
|
for (const runId of [cancelledRunId, stoppingRunId]) {
|
|
expect(await coordinator(runId)).toMatchObject({ phase: "observed", leaseOwner: expect.any(String) });
|
|
}
|
|
});
|
|
});
|