Files
PaperClipAI/server/src/__tests__/native-finalization-recovery.test.ts
T
DottaandClaude Opus 5.5 81d2e4d8d9 fix(native): end a run whose finalizer a newer run superseded, so it stops holding its task (#21106)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result, then a finalizer exports the
workspace and applies the result to the task.
> - When a newer run already committed the task's decision, the
finalizer retires the older result as `native_finalization_superseded`.
> - That retirement set the finalizer terminal, but it did not end the
run. The run stayed `running` and held the task and every run queued
behind it, with no way out except a person.
> - A stuck run must never hold a task and its queue forever.
> - This pull request ends the run when its finalizer is retired as
superseded, and it ends runs that an older release left in that state.
> - The benefit is that the task continues with its queued runs by
itself.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, a native run accepted its result at
11:31:08Z. The workspace export then failed
(`remote_workspace_archive_fs_eacces`). The task's last decision came
from another run that was created 42 seconds later but ran earlier. The
finalizer judged this run superseded: it set the finalizer to
`terminal_failure` with `native_finalization_superseded`. The run stayed
`running` with "Recovery needed". Over one hour later it still held the
task, and five queued runs waited behind it. A server upgrade at 12:02Z
did not change it.

**Expected behavior**
A run whose finalizer is retired ends. The task keeps the newer run's
decision, the retired result is never applied, and the queued runs
proceed.

**Steps to reproduce**
1. Accept a native result for a run.
2. Commit a status decision on the same task from a run that was created
later.
3. Let the older run's finalization fail once, so the finalizer records
the failure.
4. See that the finalizer is `terminal_failure` /
`native_finalization_superseded` and the run is still `running`.

**Root cause**
`recordRetryableFailure`
(`server/src/services/native-runtime/native-run-finalizer.ts`) projects
a terminal run status only when `exhausted && !supersededByNewerRun`. A
superseded retirement therefore never ended the run. No other path ends
a run whose finalizer is already `terminal_failure`: the reconciler
skips `terminal_failure` rows.

**Paperclip version or commit**
Staging Core `3eca7a6112`. The behavior is the same on master
`482063aeca`.

## What Changed

- `native-run-finalizer.ts`: a superseded retirement now ends the run
with the same status mapping as an exhausted finalization (workspace
failure: `failed`; otherwise the provider's terminal state). It does not
change the task, does not apply the result, and does not keep the
workspace export lease. The run's error says in plain words that a newer
run already decided the task.
- `native-run-finalizer.ts`: add `endSupersededNativeFinalizationRuns`.
It ends a native run that is still `running` while its finalizer is
`terminal_failure` / `native_finalization_superseded`. This covers rows
that an older release wrote. It locks in the finalizer's order
(coordinator, task, run), and re-checks the row under the lock.
- `native-run-projection-work.ts`: the registered native run projection
worker (it loads at startup, pages through all candidates, and wakes on
finalization changes) also selects such runs and ends them with
`endSupersededNativeFinalizationRuns`.
- `doc/execution-semantics.md`: document the rule.
- No schema change. Nothing changes on the path that starts a run or a
turn.

## Verification

- `server/src/__tests__/native-finalization-recovery.test.ts`, red on
master and green on this branch:
- "retires an older failed finalizer when a newer run already committed
the issue": the retired run now ends (`succeeded`, `finishedAt` set).
Master: the run stays `running`.
- "ends a run an earlier release left running after retiring its
finalizer as superseded": the shape of the staging row (workspace export
failed, superseded finalizer, run `running`). The repair ends it as
`failed` with the plain message. The task, its decision and the
finalizer stay unchanged, and no status decision is written for the run.
A second pass changes nothing.
- `server/src/__tests__/native-run-projection-work.test.ts`, "ends, at
startup, a run an earlier release left running after retiring its
finalizer as superseded": the registered worker ends the staging-shaped
row at startup and settles its claimed wake. Red without the worker
change.
- Related suites pass: `native-finalization-recovery`,
`native-run-projection-work`, `native-finalization-reconciler-timing`,
`native-reconciliation-timing`, `native-workspace-finalizer`,
`native-workspace-finalizer-failure`, `native-run-finalizer-telemetry`.
- `tsc --noEmit` for `server`.

## Risks

- Low risk. The change only ends runs whose finalizer is already
terminal and superseded. It never applies a result and never changes the
task.
- A superseded run whose workspace export failed now releases its
sandbox lease through the normal settlement instead of waiting forever.
Its result was not applied, because a newer run decided the task.
- Not changed here: the superseded check compares run creation times. A
run created earlier but executed later is still judged superseded. This
PR makes that verdict end the run; it does not change the verdict.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:07:18 -06:00

898 lines
36 KiB
TypeScript

import { findRetainedNativeSessionCleanupCandidates } from "../services/native-runtime/retained-native-session-cleanup.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { randomUUID } from "node:crypto";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { eq } from "drizzle-orm";
import {
agents,
companies,
completionContracts,
createDb,
heartbeatRuns,
issueRecoveryActions,
issues,
nativeRunFinalizations,
nativeRunResults,
statusDecisions,
workAssessments,
workspaceOperations,
} from "@paperclipai/db";
import {
CONTROL_PLANE_CONFORMANCE_OPEN,
CONTROL_PLANE_CONFORMANCE_RESULT,
CONTROL_PLANE_CONFORMANCE_TERMINAL,
} from "../vendor/paperclip-runner/testing.js";
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import {
reconcileNativeFinalizations,
reconcileRetainedNativeSessionCleanups,
} from "../services/native-runtime/native-finalization-reconciler.js";
import { PaperclipControlPlanePort } from "../services/native-runtime/paperclip-control-plane-port.js";
import { endSupersededNativeFinalizationRuns } from "../services/native-runtime/native-run-finalizer.js";
import { assertRetainedNativeSourceArchiveSettled, reconcileRetainedNativeSessionCleanup } from "../services/native-runtime/native-session-executor.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_FINALIZATION_TEST_DATABASE_URL");
describe("P6-16/P6-25/P6-28 native finalization recovery", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
const companyId = "72000000-0000-4000-8000-000000000001";
const agentId = "72000000-0000-4000-8000-000000000002";
const issueId = "72000000-0000-4000-8000-000000000003";
const contractId = "72000000-0000-4000-8000-000000000004";
const runId = "72000000-0000-4000-8000-000000000005";
const staleIssueId = "72000000-0000-4000-8000-000000000013";
const staleContractId = "72000000-0000-4000-8000-000000000014";
const staleRunId = "72000000-0000-4000-8000-000000000015";
beforeAll(async () => {
if (!external) temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-recovery-");
db = createDb(external ?? temporary.connectionString);
await db.insert(companies).values({ id: companyId, name: "Native recovery", issuePrefix: "NRC" });
await db.insert(agents).values({
id: agentId,
companyId,
name: "Recovery agent",
adapterType: "codex_local",
status: "running",
});
await db.insert(issues).values({
id: issueId,
companyId,
title: "Recover invalid native finalization",
status: "in_progress",
assigneeAgentId: agentId,
workMode: "standard",
});
await db.insert(completionContracts).values({
id: contractId,
companyId,
issueId,
revision: 1,
schemaVersion: "paperclip.completion-contract.v1",
policyVersion: "phase6-v1",
risk: "standard",
completionAuthority: "server_arbiter",
incompleteCriteriaPolicy: "preserve_non_terminal",
contractJson: {
revision: "phase6-v1",
objective: "Recover invalid native finalization",
criteria: [{ id: "objective", requirement: "Recovery remains live" }],
},
canonicalSha256: "native-recovery-contract",
createdByActorType: "system",
createdByActorId: "test",
});
await db.insert(heartbeatRuns).values({
id: runId,
companyId,
agentId,
status: "running",
runtimeMode: "native",
runtimeModeReason: "persisted_before_kill_switch",
nativeIssueId: issueId,
nativeSessionId: runId,
runnerInstanceId: contractId,
completionContractId: contractId,
completionContractSha256: "native-recovery-contract",
contextSnapshot: { issueId },
});
const port = new PaperclipControlPlanePort(db, {
companyId,
issueId,
runId,
agentId,
sessionId: runId,
completionContractId: contractId,
completionContractSha256: "native-recovery-contract",
sourceInstanceId: contractId,
controlPlaneSourceInstanceId: "recovery-control",
});
await port.openRun({
...CONTROL_PLANE_CONFORMANCE_OPEN,
identity: { companyId, issueId, runId, agentId, sessionId: runId },
sourceInstanceId: contractId,
});
await port.completeRun({
result: CONTROL_PLANE_CONFORMANCE_RESULT,
terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL,
callerResultId: "recovery-result",
});
const stored = await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, runId))
.limit(1).then((rows) => rows[0]!);
await db.update(nativeRunResults).set({
resultJson: {
...(stored.resultJson as Record<string, unknown>),
terminal: { ...CONTROL_PLANE_CONFORMANCE_TERMINAL, runTerminalState: "unknown" },
},
}).where(eq(nativeRunResults.id, stored.id));
await db.insert(workspaceOperations).values({
companyId,
heartbeatRunId: runId,
issueId,
phase: "workspace_finalize",
status: "succeeded",
});
await db.insert(issues).values({
id: staleIssueId,
companyId,
title: "Retire a stale invalid finalizer",
status: "in_progress",
assigneeAgentId: agentId,
workMode: "standard",
});
await db.insert(completionContracts).values({
id: staleContractId,
companyId,
issueId: staleIssueId,
revision: 1,
schemaVersion: "paperclip.completion-contract.v1",
policyVersion: "phase6-v1",
risk: "standard",
completionAuthority: "server_arbiter",
incompleteCriteriaPolicy: "preserve_non_terminal",
contractJson: {
revision: "phase6-v1",
objective: "Retire a stale invalid finalizer",
criteria: [{ id: "objective", requirement: "Keep the newer decision" }],
},
canonicalSha256: "stale-finalizer-contract",
createdByActorType: "system",
createdByActorId: "test",
});
await db.insert(heartbeatRuns).values({
id: staleRunId,
companyId,
agentId,
status: "running",
runtimeMode: "native",
runtimeModeReason: "persisted_before_kill_switch",
nativeIssueId: staleIssueId,
nativeSessionId: staleRunId,
runnerInstanceId: staleContractId,
completionContractId: staleContractId,
completionContractSha256: "stale-finalizer-contract",
contextSnapshot: { issueId: staleIssueId },
});
const stalePort = new PaperclipControlPlanePort(db, {
companyId,
issueId: staleIssueId,
runId: staleRunId,
agentId,
sessionId: staleRunId,
completionContractId: staleContractId,
completionContractSha256: "stale-finalizer-contract",
sourceInstanceId: staleContractId,
controlPlaneSourceInstanceId: "stale-control",
});
await stalePort.openRun({
...CONTROL_PLANE_CONFORMANCE_OPEN,
identity: { companyId, issueId: staleIssueId, runId: staleRunId, agentId, sessionId: staleRunId },
sourceInstanceId: staleContractId,
});
await stalePort.completeRun({
result: CONTROL_PLANE_CONFORMANCE_RESULT,
terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL,
callerResultId: "stale-result",
});
const staleStored = await db.select().from(nativeRunResults)
.where(eq(nativeRunResults.runId, staleRunId))
.limit(1).then((rows) => rows[0]!);
await db.update(nativeRunResults).set({
resultJson: {
...(staleStored.resultJson as Record<string, unknown>),
terminal: { ...CONTROL_PLANE_CONFORMANCE_TERMINAL, runTerminalState: "unknown" },
},
}).where(eq(nativeRunResults.id, staleStored.id));
await db.insert(workspaceOperations).values({
companyId,
heartbeatRunId: staleRunId,
issueId: staleIssueId,
phase: "workspace_finalize",
status: "succeeded",
});
}, 30_000);
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await temporary?.cleanup();
});
it("fails closed into bounded named recovery without consulting the live flag or falling back", async () => {
await expect(reconcileNativeFinalizations(db, [runId])).resolves.toEqual([
expect.objectContaining({ phase: "retryable_failure", failureCode: "native_finalization_invalid" }),
]);
await expect(db.select().from(issues).where(eq(issues.id, issueId))).resolves.toEqual([
expect.objectContaining({ status: "in_progress", statusVersion: 0, lastStatusDecisionId: null }),
]);
await expect(db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).resolves.toEqual([
expect.objectContaining({ status: "active", ownerAgentId: agentId, cause: "native_finalization_invalid" }),
]);
await expect(db.select().from(statusDecisions).where(eq(statusDecisions.issueId, issueId))).resolves.toHaveLength(0);
for (let retry = 0; retry < 2; retry += 1) {
await db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(0) })
.where(eq(nativeRunFinalizations.runId, runId));
await reconcileNativeFinalizations(db, [runId]);
}
await expect(db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId))).resolves.toEqual([
expect.objectContaining({
phase: "terminal_failure",
attempt: 3,
failureCode: "native_finalization_retry_exhausted",
nextAttemptAt: null,
}),
]);
await expect(db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId))).resolves.toEqual([
expect.objectContaining({
runtimeMode: "native",
status: "succeeded",
nativePhase: "terminal_failure",
resultJson: expect.objectContaining({ prpRunTerminalState: "succeeded" }),
}),
]);
await expect(reconcileNativeFinalizations(db, [runId])).resolves.toEqual([]);
});
it("retires an older failed finalizer when a newer run already committed the issue", async () => {
await expect(reconcileNativeFinalizations(db, [staleRunId])).resolves.toEqual([
expect.objectContaining({ phase: "retryable_failure", failureCode: "native_finalization_invalid" }),
]);
const newerRunId = "72000000-0000-4000-8000-000000000016";
await db.insert(heartbeatRuns).values({
id: newerRunId,
companyId,
agentId,
status: "succeeded",
runtimeMode: "native",
nativeIssueId: staleIssueId,
completionContractId: staleContractId,
completionContractSha256: "stale-finalizer-contract",
contextSnapshot: { issueId: staleIssueId },
});
const [newerResult] = await db.insert(nativeRunResults).values({
companyId,
issueId: staleIssueId,
runId: newerRunId,
completionContractId: staleContractId,
callerResultId: "newer-result",
serverFingerprint: "newer-result-fingerprint",
schemaStatus: "accepted",
resultJson: {},
canonicalSha256: "newer-result-sha",
}).returning();
const [newerAssessment] = await db.insert(workAssessments).values({
companyId,
issueId: staleIssueId,
runId: newerRunId,
contractId: staleContractId,
resultId: newerResult!.id,
triggerKind: "native_result",
triggerRef: newerResult!.id,
triggerCapability: "server_native_finalizer",
triggerActorCompanyId: companyId,
priorIssueStatus: "in_progress",
priorStatusVersion: 0,
policyVersion: "phase6-v3",
assessmentJson: {},
inputDigest: "newer-assessment-digest",
}).returning();
const [newerDecision] = await db.insert(statusDecisions).values({
companyId,
issueId: staleIssueId,
runId: newerRunId,
assessmentId: newerAssessment!.id,
decisionVersion: 1,
policyVersion: "phase6-v3",
fromStatus: "in_progress",
toStatus: "done",
reasonCode: "completion_claim_policy_accepted",
decisionJson: {},
decisionDigest: "newer-decision-digest",
applicationState: "applied",
appliedAt: new Date(),
}).returning();
await db.update(issues).set({
status: "done",
statusVersion: 1,
lastStatusDecisionId: newerDecision!.id,
}).where(eq(issues.id, staleIssueId));
await db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(0) })
.where(eq(nativeRunFinalizations.runId, staleRunId));
await expect(reconcileNativeFinalizations(db, [staleRunId])).resolves.toEqual([
expect.objectContaining({ phase: "terminal_failure", failureCode: "native_finalization_superseded" }),
]);
await expect(db.select().from(issues).where(eq(issues.id, staleIssueId))).resolves.toEqual([
expect.objectContaining({ status: "done", lastStatusDecisionId: newerDecision!.id }),
]);
await expect(db.select().from(issueRecoveryActions)
.where(eq(issueRecoveryActions.sourceIssueId, staleIssueId))).resolves.toEqual([
expect.objectContaining({ status: "resolved", outcome: "false_positive" }),
]);
// The retired run ends, so it no longer holds the task and its queue.
const [ended] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, staleRunId));
// Provider work succeeded; the result is kept for reference only.
expect(ended).toMatchObject({ status: "succeeded", nativePhase: "terminal_failure", finishedAt: expect.any(Date) });
});
it("ends a run an earlier release left running after retiring its finalizer as superseded", async () => {
// The row an earlier release wrote after a failed workspace export: a
// terminal superseded coordinator, and the run still running.
await db.update(heartbeatRuns).set({ status: "running", finishedAt: null, error: null, errorCode: null })
.where(eq(heartbeatRuns.id, staleRunId));
// Written here, so the case does not depend on the previous test's outcome.
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", leaseOwner: null, leaseExpiresAt: null,
failureCode: "native_finalization_superseded", nextAttemptAt: null, failureDetail: { workspaceFinalizeAttempt: 1,
originalFailureCode: "native_workspace_sync_out_failed", recoveryOwner: { kind: "none", reason: "newer_native_decision" } } })
.where(eq(nativeRunFinalizations.runId, staleRunId));
const [issueBefore] = await db.select().from(issues).where(eq(issues.id, staleIssueId));
const [coordinatorBefore] = await db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, staleRunId));
expect(coordinatorBefore).toMatchObject({ phase: "terminal_failure", failureCode: "native_finalization_superseded" });
await expect(endSupersededNativeFinalizationRuns(db, [staleRunId])).resolves.toEqual([staleRunId]);
const [ended] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, staleRunId));
expect(ended).toMatchObject({ status: "failed", errorCode: "native_finalization_superseded", finishedAt: expect.any(Date) });
expect(ended.error).toBe("A newer run already decided this task, so this run's result was kept for reference and not applied.");
// The durable verdict stands: no issue change, no re-applied result.
await expect(db.select().from(issues).where(eq(issues.id, staleIssueId))).resolves.toEqual([
expect.objectContaining({ status: issueBefore.status, statusVersion: issueBefore.statusVersion,
lastStatusDecisionId: issueBefore.lastStatusDecisionId }),
]);
await expect(db.select().from(statusDecisions).where(eq(statusDecisions.runId, staleRunId))).resolves.toHaveLength(0);
await expect(db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, staleRunId)))
.resolves.toEqual([expect.objectContaining({ phase: "terminal_failure", failureCode: "native_finalization_superseded",
resultId: coordinatorBefore.resultId })]);
// A second pass finds nothing left to end.
await expect(endSupersededNativeFinalizationRuns(db, [staleRunId])).resolves.toEqual([]);
await expect(db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, staleRunId))).resolves.toEqual([
expect.objectContaining({ status: "failed" }),
]);
});
});
describe("retained native cleanup discovery", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
const companyId = randomUUID();
const agentId = randomUUID();
const closeError =
"provider_transport_failed: runner did not durably suspend before checkpoint";
const legacyMaintenanceHistory = () => [
{
kind: "native_cleanup_maintenance",
version: 1,
phase: "started",
requestId: "native-cleanup:legacy-request",
sourceFingerprint: "a".repeat(64),
startedAt: "2026-09-08T12:00:00.000Z",
},
{
kind: "native_cleanup_maintenance",
version: 1,
phase: "operator_required",
requestId: "native-cleanup:legacy-request",
code: "native_cleanup_maintenance_unproven",
},
];
beforeAll(async () => {
if (!external) temporary = await startEmbeddedPostgresTestDatabase(
"paperclip-native-cleanup-sweep-",
);
db = createDb(external ?? temporary.connectionString);
await db
.insert(companies)
.values({ id: companyId, name: "Cleanup sweep", issuePrefix: "NCS" });
await db
.insert(agents)
.values({
id: agentId,
companyId,
name: "Cleanup",
adapterType: "codex_local",
});
});
afterEach(async () => {
// Remove only this suite's discovery rows; the throwaway database retains
// its accepted-result fixtures until teardown. No physical cleanup runs.
await db
.delete(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.companyId, companyId));
await reconcileRetainedNativeSessionCleanups(db, {
cleanup: async ({ runId }) => ({ runId, status: "not_eligible" }),
});
});
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await temporary?.cleanup();
});
async function candidate(
options: {
owner?: { companyId: string; agentId: string };
run?: Partial<typeof heartbeatRuns.$inferInsert>;
coordinator?: Partial<typeof nativeRunFinalizations.$inferInsert>;
schemaStatus?: string;
issueId?: string;
revision?: number;
} = {},
) {
const owner = options.owner ?? { companyId, agentId };
const runId = randomUUID();
const issueId = options.issueId ?? randomUUID();
const contractId = randomUUID();
if (!options.issueId) await db
.insert(issues)
.values({
id: issueId,
companyId: owner.companyId,
title: "Retained session",
assigneeAgentId: owner.agentId,
});
await db.insert(completionContracts).values({
id: contractId,
companyId: owner.companyId,
issueId,
revision: options.revision ?? 1,
schemaVersion: "paperclip.completion-contract.v1",
policyVersion: "phase6-v3",
risk: "standard",
completionAuthority: "server_arbiter",
incompleteCriteriaPolicy: "preserve_non_terminal",
contractJson: {},
canonicalSha256: runId,
createdByActorType: "system",
createdByActorId: "test",
});
await db.insert(heartbeatRuns).values({
id: runId,
companyId: owner.companyId,
agentId: owner.agentId,
runtimeMode: "native",
status: "succeeded",
nativeIssueId: issueId,
nativeSessionId: runId,
nativePhase: "terminal_failure",
completionContractId: contractId,
completionContractSha256: runId,
finishedAt: new Date(),
errorCode: "adapter_failed",
error: closeError,
...options.run,
});
const [result] = await db
.insert(nativeRunResults)
.values({
companyId: owner.companyId,
issueId,
runId,
completionContractId: contractId,
serverFingerprint: runId,
canonicalSha256: runId,
schemaStatus: options.schemaStatus ?? "accepted",
resultJson: {},
})
.returning();
const [assessment] = await db
.insert(workAssessments)
.values({
companyId: owner.companyId,
issueId,
runId,
contractId,
resultId: result!.id,
triggerKind: "native_result",
triggerActorCompanyId: owner.companyId,
priorIssueStatus: "in_progress",
priorStatusVersion: 0,
policyVersion: "phase6-v3",
assessmentJson: {},
inputDigest: runId,
})
.returning();
const [decision] = await db
.insert(statusDecisions)
.values({
companyId: owner.companyId,
issueId,
runId,
assessmentId: assessment!.id,
decisionVersion: options.revision ?? 1,
policyVersion: "phase6-v3",
fromStatus: "in_progress",
toStatus: "in_review",
reasonCode: "prior_status_terminal_preserved",
decisionJson: {},
decisionDigest: runId,
})
.returning();
await db.insert(nativeRunFinalizations).values({
companyId: owner.companyId,
issueId,
runId,
phase: "committed",
resultId: result!.id,
assessmentId: assessment!.id,
decisionId: decision!.id,
...options.coordinator,
});
return runId;
}
it("does not misclassify a real competing row lock as ineligible retained history", async () => {
const runId = await candidate();
await db.transaction(async tx => {
await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update");
await expect(reconcileRetainedNativeSessionCleanup(db, { companyId, runId }))
.rejects.toMatchObject({ cause: { code: "55P03" } });
});
});
it("reconstructs future lease deadlines without authorizing early legacy cleanup", async () => {
const leaseExpiresAt = new Date(Date.now() + 60_000);
const runId = await candidate({ coordinator: { leaseOwner: "other-owner", leaseExpiresAt } });
const scope = { companyId, runId };
expect(await findRetainedNativeSessionCleanupCandidates(db, { scope, limit: 100 })).toEqual([]);
expect(await findRetainedNativeSessionCleanupCandidates(db, { scope, limit: 100, includeLeased: true }))
.toEqual([expect.objectContaining({ runId, leaseOwner: "other-owner", leaseExpiresAt })]);
const cleanup = vi.fn();
await reconcileRetainedNativeSessionCleanups(db, { cleanup });
expect(cleanup).not.toHaveBeenCalled();
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1000) })
.where(eq(nativeRunFinalizations.runId, runId));
expect(await findRetainedNativeSessionCleanupCandidates(db, { scope, limit: 100 }))
.toEqual([expect.objectContaining({ runId })]);
});
it("scopes reconstruction to a company, run, task, or agent without leaking another tenant", async () => {
const other = { companyId: randomUUID(), agentId: randomUUID() };
await db.insert(companies).values({ id: other.companyId, name: "Other cleanup tenant", issuePrefix: "NCO" });
await db.insert(agents).values({ id: other.agentId, companyId: other.companyId, name: "Other cleanup", adapterType: "codex_local" });
try {
const first = await candidate();
const second = await candidate();
const foreign = await candidate({ owner: other });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, first));
for (const scope of [{ companyId, runId: first }, { companyId, issueId: run.nativeIssueId! }]) {
expect((await findRetainedNativeSessionCleanupCandidates(db, { scope, limit: 100, includeLeased: true }))
.map(row => row.runId)).toEqual([first]);
}
expect((await findRetainedNativeSessionCleanupCandidates(db, { scope: { companyId, agentId }, limit: 100 }))
.map(row => row.runId).sort()).toEqual([first, second].sort());
expect(await findRetainedNativeSessionCleanupCandidates(db, {
scope: { companyId, runId: foreign }, limit: 100,
})).toEqual([]);
expect((await findRetainedNativeSessionCleanupCandidates(db, { scope: other, limit: 100 }))
.map(row => row.runId)).toEqual([foreign]);
} finally {
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, other.companyId));
}
});
it("pages reconstruction past retained history without wrapping or admitting previously attempted cleanup", async () => {
const ids = [await candidate(), await candidate(), await candidate()].sort();
await candidate({ coordinator: { recoveryHistory: [{ kind: "native_cleanup_runner_epoch", phase: "spawned" }] } });
let after: string | undefined;
const found: string[] = [];
for (;;) {
const rows = await findRetainedNativeSessionCleanupCandidates(db, {
scope: { companyId }, after, limit: 1, includeLeased: true,
});
if (!rows.length) break;
found.push(rows[0].runId);
after = rows[0].runId;
expect(found.length).toBeLessThanOrEqual(ids.length);
}
expect(found).toEqual(ids);
});
it("discovers exact committed results even with stale nativePhase or privately recovered errors", async () => {
const stale = await candidate();
const recovered = await candidate({
run: {
error: null,
errorCode: null,
resultJson: {
recoveredExecutionFailure: {
error: closeError,
errorCode: "adapter_failed",
},
},
},
});
const before = await db
.select()
.from(heartbeatRuns)
.where(eq(heartbeatRuns.companyId, companyId));
const cleanup = vi.fn(
async ({ runId }: { runId: string; companyId: string }) => ({
runId,
status: "not_eligible" as const,
}),
);
await reconcileRetainedNativeSessionCleanups(db, { cleanup, limit: 5 });
expect(cleanup.mock.calls.map(([input]) => input.runId).sort()).toEqual(
[stale, recovered].sort(),
);
expect(
cleanup.mock.calls.every(([input]) => input.companyId === companyId),
).toBe(true);
expect(
await db
.select()
.from(heartbeatRuns)
.where(eq(heartbeatRuns.companyId, companyId)),
).toEqual(before);
});
it.each([
{ run: { runtimeMode: "legacy" } },
{ run: { status: "running" } },
{ run: { finishedAt: null } },
{ run: { error: "another failure" } },
{ run: { errorCode: "setup_failed" } },
{
run: {
error: "newer failure",
errorCode: "adapter_failed",
resultJson: {
recoveredExecutionFailure: {
error: closeError,
errorCode: "adapter_failed",
},
},
},
},
{ schemaStatus: "rejected" },
{ coordinator: { phase: "retryable_failure" } },
{ coordinator: { nextAttemptAt: new Date(0) } },
{
coordinator: {
leaseOwner: "another-controller",
leaseExpiresAt: new Date(Date.now() + 120_000),
},
},
...["started", "settled", "operator_required"].map((phase) => ({
coordinator: {
recoveryHistory: [{ kind: "native_cleanup_maintenance", phase }],
},
})),
{ coordinator: { recoveryHistory: [{ kind: "native_cleanup_source_archive", phase: "operator_required" }] } },
{ coordinator: { recoveryHistory: [
{ kind: "native_cleanup_source_archive", phase: "prepared" },
{ kind: "native_cleanup_runner_epoch", phase: "spawned", epoch: 1, pid: 88736 },
] } },
])(
"excludes ineligible or previously attempted cleanup: %j",
async (options) => {
await candidate(options);
const cleanup = vi.fn();
expect(
await reconcileRetainedNativeSessionCleanups(db, { cleanup, limit: 5 }),
).toEqual([]);
expect(cleanup).not.toHaveBeenCalled();
},
);
it("discovers a single legacy failed attempt only for independent physical proof", async () => {
const runId = await candidate({
coordinator: { recoveryHistory: legacyMaintenanceHistory() },
});
const before = await db
.select()
.from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, runId));
const cleanup = vi.fn(async (input: { runId: string }) => ({
runId: input.runId,
status: "not_eligible" as const,
}));
await reconcileRetainedNativeSessionCleanups(db, { cleanup });
expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId });
expect(
await db
.select()
.from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, runId)),
).toEqual(before);
});
it.each(["prepared", "archived"])("discovers prelaunch source archival at %s without changing authority", async (phase) => {
const runId = await candidate();
const prepared = {
kind: "native_cleanup_source_archive", version: 1, phase: "prepared",
requestId: "native-cleanup:source-archive", companyId, agentId, runId,
nativeSessionId: runId, runnerInstanceId: randomUUID(), stateKey: "a".repeat(64),
archiveName: `${"a".repeat(64)}.identity_indeterminate.cleanup.fixture`,
rootIdentity: { device: 1, inode: 2, mode: 0o40700 },
sourceFingerprint: "b".repeat(64), providerHomeFingerprint: "c".repeat(64),
};
const history = [prepared, ...(phase === "archived" ? [{ ...prepared, phase }] : [])];
await db.update(nativeRunFinalizations).set({ recoveryHistory: history })
.where(eq(nativeRunFinalizations.runId, runId));
const cleanup = vi.fn(async () => ({ runId, status: "not_eligible" as const }));
await reconcileRetainedNativeSessionCleanups(db, { cleanup });
expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId });
const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
expect(after?.recoveryHistory).toEqual(history);
expect(after?.phase).toBe("committed");
});
it("fences actual scoped admission until every archived owner has its matching latest settlement", async () => {
const first = await candidate();
const [owner] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, first));
const issueId = owner!.issueId;
const scope = { companyId, issueId, stateKey: "d".repeat(64) };
const prepared = { kind: "native_cleanup_source_archive", version: 1, phase: "prepared",
stateKey: scope.stateKey, requestId: "native-cleanup:archive", sourceFingerprint: "a".repeat(64) };
const settled = { kind: "native_cleanup_maintenance", phase: "settled",
sourceArchiveRequestId: prepared.requestId, sourceFingerprint: prepared.sourceFingerprint };
const setHistory = async (runId: string, history: Record<string, unknown>[]) =>
db.update(nativeRunFinalizations).set({ recoveryHistory: history }).where(eq(nativeRunFinalizations.runId, runId));
await assertRetainedNativeSourceArchiveSettled(db, scope);
await setHistory(first, [prepared]);
await expect(assertRetainedNativeSourceArchiveSettled(db, scope)).rejects.toMatchObject({ code: "native_session_cleanup_quarantined" });
await assertRetainedNativeSourceArchiveSettled(db, { ...scope, stateKey: "e".repeat(64) });
await assertRetainedNativeSourceArchiveSettled(db, { ...scope, companyId: randomUUID() });
await assertRetainedNativeSourceArchiveSettled(db, { ...scope, issueId: randomUUID() });
await setHistory(first, [prepared, { ...settled, sourceFingerprint: "b".repeat(64) }]);
await expect(assertRetainedNativeSourceArchiveSettled(db, scope)).rejects.toMatchObject({ code: "native_session_cleanup_quarantined" });
await setHistory(first, [prepared, settled]);
await assertRetainedNativeSourceArchiveSettled(db, scope);
await setHistory(first, [prepared, { ...prepared }, settled]);
await expect(assertRetainedNativeSourceArchiveSettled(db, scope)).rejects.toMatchObject({ code: "native_session_cleanup_quarantined" });
await setHistory(first, [{ ...prepared, version: null }, settled]);
await expect(assertRetainedNativeSourceArchiveSettled(db, scope)).rejects.toMatchObject({ code: "native_session_cleanup_quarantined" });
await setHistory(first, [{ ...prepared, requestId: null }, { kind: "native_cleanup_maintenance", phase: "settled", sourceFingerprint: prepared.sourceFingerprint }]);
await expect(assertRetainedNativeSourceArchiveSettled(db, scope)).rejects.toMatchObject({ code: "native_session_cleanup_quarantined" });
await setHistory(first, [prepared, settled]);
const second = await candidate({ issueId, revision: 2 });
await setHistory(second, [prepared, settled]);
await assertRetainedNativeSourceArchiveSettled(db, scope);
const third = await candidate({ issueId, revision: 3 });
await setHistory(third, [prepared]);
await expect(assertRetainedNativeSourceArchiveSettled(db, scope)).rejects.toMatchObject({ code: "native_session_cleanup_quarantined" });
await setHistory(third, [prepared, settled]);
await assertRetainedNativeSourceArchiveSettled(db, scope);
await setHistory(first, [prepared, settled, { ...settled, phase: "operator_required" }]);
await expect(assertRetainedNativeSourceArchiveSettled(db, scope)).rejects.toMatchObject({ code: "native_session_cleanup_quarantined" });
});
it.each([
[
"new epoch",
(entries: Record<string, unknown>[]) =>
entries.push({ kind: "native_cleanup_runner_epoch", phase: "intent" }),
],
[
"third attempt",
(entries: Record<string, unknown>[]) => entries.push({ ...entries[0] }),
],
[
"reordered phases",
(entries: Record<string, unknown>[]) => entries.reverse(),
],
[
"different request",
(entries: Record<string, unknown>[]) => {
entries[1]!.requestId = "native-cleanup:other";
},
],
[
"unversioned entry",
(entries: Record<string, unknown>[]) => {
delete entries[0]!.version;
},
],
[
"settled attempt",
(entries: Record<string, unknown>[]) => {
entries[1]!.phase = "settled";
},
],
[
"unknown failure",
(entries: Record<string, unknown>[]) => {
entries[1]!.code = "other_failure";
},
],
[
"missing source",
(entries: Record<string, unknown>[]) => {
delete entries[0]!.sourceFingerprint;
},
],
] as const)("excludes legacy discovery with %s", async (_name, mutate) => {
const history: Record<string, unknown>[] = legacyMaintenanceHistory();
mutate(history);
await candidate({ coordinator: { recoveryHistory: history } });
const cleanup = vi.fn();
expect(
await reconcileRetainedNativeSessionCleanups(db, { cleanup }),
).toEqual([]);
expect(cleanup).not.toHaveBeenCalled();
});
it("joins overlap and advances beyond a permanently ineligible first candidate", async () => {
const ids = [await candidate(), await candidate()].sort();
let release!: () => void;
const held = new Promise<void>((resolve) => {
release = resolve;
});
let started!: () => void;
const entered = new Promise<void>((resolve) => {
started = resolve;
});
const cleanup = vi.fn(
async ({ runId }: { runId: string; companyId: string }) => {
started();
await held;
return { runId, status: "not_eligible" as const };
},
);
const first = reconcileRetainedNativeSessionCleanups(db, { cleanup });
await entered;
const overlapping = reconcileRetainedNativeSessionCleanups(db, { cleanup });
expect(overlapping).toBe(first);
expect(cleanup).toHaveBeenCalledTimes(1);
release();
await first;
await reconcileRetainedNativeSessionCleanups(db, { cleanup });
expect(cleanup.mock.calls.map(([input]) => input.runId)).toEqual(ids);
await reconcileRetainedNativeSessionCleanups(db, { cleanup });
expect(cleanup.mock.calls.map(([input]) => input.runId)).toEqual([
...ids,
ids[0],
]);
});
it("isolates one failure and resets its joined owner for the next bounded sweep", async () => {
const ids = [await candidate(), await candidate()].sort();
const onError = vi.fn();
const cleanup = vi.fn(
async ({ runId }: { runId: string; companyId: string }) => {
if (runId === ids[0]) throw new Error("fixture cleanup refused");
return { runId, status: "operator_required" as const };
},
);
expect(
await reconcileRetainedNativeSessionCleanups(db, {
cleanup,
onError,
limit: 2,
}),
).toEqual([{ runId: ids[1], status: "operator_required" }]);
expect(onError).toHaveBeenCalledWith(expect.any(Error), ids[0]);
await reconcileRetainedNativeSessionCleanups(db, {
cleanup,
onError,
limit: 2,
});
expect(cleanup).toHaveBeenCalledTimes(4);
});
});