fix(native): finish a failed warm session's cleanup by itself and continue the task (#21977)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner runs keep a provider session per task. A follow-up run
can reuse the warm session of the run before it, in a remote sandbox.
> - When a warm follow-up fails, the previous session is closed. The
close saves a backup of the provider session first.
> - The close looked up the backup owner through the task's current tool
authority. The failed follow-up had already released that authority.
Every close attempt failed with `native_harness_storage_owner_missing`,
so the close could never finish.
> - After nine failed attempts, the task's admission gate became
`operator_required`. The gate retires only for a stopped local process
or for the exact run that owns the session. A remote sandbox session
owned by an earlier run never matched. A queued continuation and a Retry
stayed queued forever, and the task showed a Retry button that could not
work.
> - Separately, a remote transport failure could never produce the local
stop proof that the safe-replacement lane waits for. Nothing continued
the task until an unrelated wake arrived 15 minutes later.
> - This pull request makes the close save its backup under its own
owner, retires a cleanup gate when the task's sandbox is confirmed
terminated, and refuses an unprovable remote transport failure at once
so the task continues in a fresh isolated run.
> - The benefit is that recovery finishes by itself, and a Retry that is
shown starts a run.

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos wave, a warm follow-up run on a remote sandbox failed
with `provider_transport_failed` (`PRP command run.attach timed out`).
The previous run's session close then failed on every attempt with
`native_harness_storage_owner_missing`. The cleanup gate became
`operator_required`. The continuation run was created 15 minutes later
and then stayed queued until the wave ended ("queued run kept queued
while the task's previous session finishes cleanup"). The task showed a
Retry banner, but a Retry would wait behind the same gate.

**Expected behavior**
A failed session close settles or is retired by itself with evidence.
The next run of the task starts in seconds. A Retry that is shown starts
a run.

**Steps to reproduce**
1. Run a native task on a remote sandbox with a warm session lifecycle.
2. Make the warm follow-up run fail during startup, after it has reused
the previous session.
3. Watch the previous session's close: every attempt fails with
`native_harness_storage_owner_missing`, and the queued next run of the
task never starts.

**Paperclip version or commit**
master at b2b2c1e60b (wave) and 7685b7a4fc (this branch base).

**Deployment mode**
Hosted server with remote Daytona sandboxes (restart-chaos harness).

## What Changed

- `native-session-executor.ts`: the remote checkpoint resolves its
backup owner from the current scope authority, and falls back to the
backend's own authority epoch when the scope has none. The
controller-persistence `authorityRunId` closure uses the same fallback.
A newer run that took over the session still wins.
- `paperclip-runner` `native-session-runtime.ts`: new
`nativeSessionCleanupRemoteScopes` lists the remote resources whose
quarantined cleanup holds a task. New
`completeTerminatedRemoteNativeSessionCleanupForTask` retires settled
quarantines of the task in one terminated remote resource, whichever
run's session they hold. An in-flight close keeps its entry.
- `remote-execution-termination.ts`: new `terminatedRemoteCleanupScope`
confirms that the company's newest lease on the remote resource is
released with its termination receipt and that no lease on it is still
held.
- `native-cleanup-admission-hold.ts`: queue admission now retires a held
cleanup when its sandbox is confirmed terminated. It logs the retirement
and records a `native.session_cleanup_gate_retired` run event with the
termination receipt. The local stopped-process path is unchanged.
- `native-safe-replacement.ts`: a transport failure on a remote or
non-Codex session can never produce the local stop proof. It is now
refused at once with `provider_failure_meaning_unverified`. The existing
disposition pass then continues the task in a fresh isolated run from
saved work. A local Codex run keeps the existing wait for stop proof.

## Verification

- Red/green, each verified red without its fix:
- `server/src/services/native-runtime/native-session-executor.test.ts` —
"saves a closing session's backup under its own owner after a failed
follow-up released the scope authority". Red:
`native_harness_storage_owner_missing` (the incident error).
- `server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts`
(embedded Postgres) — "starts a queued run once the previous session's
sandbox is confirmed terminated, whichever run released it", plus "keeps
the run queued while a different sandbox of the task is terminated".
Red: the queued run stays queued.
- `server/src/services/native-runtime/native-safe-replacement.test.ts`
(embedded Postgres) — "continues a transport failure whose stop can
never be proven locally without waiting for a person" for a remote
sandbox (refused and continued fresh) and a local Codex run (unchanged).
Red: no denial and no hold.
- `pnpm verify` and server and runner typechecks pass locally.

## Risks

- The admission gate now trusts a provider termination receipt on the
newest lease of the same remote resource, not only one bound to the
session's own run. A reused sandbox with a newer held lease is not
retired.
- A remote transport failure no longer waits for a stop proof that
cannot arrive. The fresh continuation is isolated (write authority
revoked, new root) and stays under the existing per-task and per-minute
continuation brakes. Nothing is added before `turn.start`.
- The run.attach timeout itself is not fixed here. The controller held
the attach frame behind a checkpoint flush to the follow-up's writer.
The run's own diagnostics were lost because they were written to the
predecessor's run log. This PR makes the failure recover by itself; the
attach stall needs a separate fix with better diagnostics.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use and code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-17 02:25:02 -06:00
1 parent 7685b7a4fc
commit 22d3f9a03c
10 files changed
+341 -16

No files matched your search

@@ -275,6 +275,47 @@ export function completeTerminatedRemoteNativeSessionCleanup(binding: {
return true;
}
function quarantinedRemoteCleanupScope(
entry: QuarantinedSessionCleanup,
input: { companyId: string; issueId: string },
): string | null {
const [companyId, , , remoteCleanupScope] = JSON.parse(entry.domain.key) as string[];
return companyId === input.companyId && remoteCleanupScope &&
entry.domain.resources.includes(JSON.stringify(["issue", input.issueId]))
? remoteCleanupScope : null;
}
export function nativeSessionCleanupRemoteScopes(input: {
companyId: string;
issueId: string;
}): string[] {
const scopes = new Set<string>();
for (const entry of quarantinedSessionCleanups) {
const scope = quarantinedRemoteCleanupScope(entry, input);
if (scope) scopes.add(scope);
}
return [...scopes];
}
export function completeTerminatedRemoteNativeSessionCleanupForTask(input: {
companyId: string;
issueId: string;
remoteCleanupScope: string;
}): number {
if (!input.remoteCleanupScope) return 0;
let retired = 0;
for (const entry of [...quarantinedSessionCleanups]) {
if (quarantinedRemoteCleanupScope(entry, input) !== input.remoteCleanupScope) continue;
if (entry.attempt || entry.recovery) continue;
if (entry.timer) clearTimeout(entry.timer);
entry.timer = null;
quarantinedSessionCleanups.delete(entry);
announceCleanupSettled(entry.domain);
retired += 1;
}
return retired;
}
/** Host-only counterpart of remote resource termination. The caller has verified
* both exact local process identities and durably fenced their run. This removes
* only cleanup ownership; the retained checkpoint is never made resumable.
@@ -5,7 +5,7 @@ import { join } from "node:path";
import { eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { hostname } from "node:os";
import { agents, companies, createDb, heartbeatRuns, issues } from "@paperclipai/db";
import { agents, companies, createDb, environmentLeases, heartbeatRuns, issues } from "@paperclipai/db";
import {
executeNativeSession,
NativeSessionCloseUnrecoverableError,
@@ -55,7 +55,8 @@ describeEmbeddedPostgres("queued native run behind the task's previous session c
}
async function retainPreviousSessionClose(f: Awaited<ReturnType<typeof seed>>, close: () => Promise<void>,
owner: { runId: string; runnerInstanceId: string } = { runId: randomUUID(), runnerInstanceId: randomUUID() }) {
owner: { runId: string; runnerInstanceId: string } = { runId: randomUUID(), runnerInstanceId: randomUUID() },
remoteCleanupScope?: string) {
const runId = owner.runId;
const identity = { runId, sessionId: `closing-${runId}`, companyId: f.companyId, issueId: f.issueId, agentId: f.previousId };
const capabilities = { resume: true, typedEvents: true, steering: false, interruption: false, structuredResult: true };
@@ -83,6 +84,7 @@ describeEmbeddedPostgres("queued native run behind the task's previous session c
appendEvent: async () => ({ cursor: 0, highestContiguousSourceSeq: 0, disposition: "committed" }),
replayEvents: async () => ({ events: [], highestContiguousSourceSeq: 0 }), completeRun: async () => {} },
runnerInstanceId: owner.runnerInstanceId, controlPlaneInstanceId: "control",
...(remoteCleanupScope ? { remoteCleanupScope } : {}),
})).rejects.toThrow("stopped for reassignment");
}
@@ -167,6 +169,63 @@ describeEmbeddedPostgres("queued native run behind the task's previous session c
expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null, errorCode: null });
});
async function recordSandboxLease(f: Awaited<ReturnType<typeof seed>>, providerLeaseId: string, terminated: boolean) {
const [failed] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.nextId,
invocationSource: "automation", status: "failed", errorCode: "provider_transport_failed", runtimeMode: "native",
nativeIssueId: f.issueId, startedAt: new Date(Date.now() - 60_000), finishedAt: new Date(),
contextSnapshot: { issueId: f.issueId } }).returning();
const leaseId = randomUUID();
await db.insert(environmentLeases).values({ id: leaseId, companyId: f.companyId, issueId: f.issueId,
heartbeatRunId: failed!.id, provider: "fake-sandbox", providerLeaseId, leasePolicy: "ephemeral",
status: terminated ? "released" : "active", releasedAt: terminated ? new Date() : null,
cleanupStatus: terminated ? "success" : null,
metadata: terminated ? { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId: f.companyId,
runId: failed!.id, leaseId, provider: "fake-sandbox", providerLeaseId, state: "stopped",
confirmedAt: new Date().toISOString() } } : {} });
return { leaseId, runId: failed!.id };
}
it("starts a queued run once the previous session's sandbox is confirmed terminated, whichever run released it", async () => {
const worker = dispatchWorker();
try {
const f = await seed();
const providerLeaseId = `sandbox-${randomUUID()}`;
await retainPreviousSessionClose(f, async () => {
throw new NativeSessionCloseUnrecoverableError({ runnerSuspended: false, providerDrained: false });
}, undefined, JSON.stringify(["fake-sandbox", providerLeaseId]));
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("operator_required");
const held = await recordSandboxLease(f, providerLeaseId, false);
const run = await queueNextAssigneeRun(f);
await heartbeatService(db).resumeQueuedRuns();
expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null });
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("operator_required");
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success",
metadata: { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId: f.companyId,
runId: held.runId, leaseId: held.leaseId, provider: "fake-sandbox", providerLeaseId, state: "stopped",
confirmedAt: new Date().toISOString() } } }).where(eq(environmentLeases.id, held.leaseId));
await heartbeatService(db).resumeQueuedRuns();
await expect.poll(async () => (await readRun(run.id)).status, { timeout: 15_000 }).not.toBe("queued");
expect((await readRun(run.id)).errorCode).not.toBe("native_session_cleanup_quarantined");
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBeNull();
} finally {
worker.stop();
}
});
it("keeps the run queued while a different sandbox of the task is terminated", async () => {
const f = await seed();
const providerLeaseId = `sandbox-${randomUUID()}`;
await retainPreviousSessionClose(f, async () => {
throw new NativeSessionCloseUnrecoverableError({ runnerSuspended: false, providerDrained: false });
}, undefined, JSON.stringify(["fake-sandbox", providerLeaseId]));
await recordSandboxLease(f, `other-${providerLeaseId}`, true);
const run = await queueNextAssigneeRun(f);
await heartbeatService(db).resumeQueuedRuns();
expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null });
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("operator_required");
});
it("does not hold a run for a different task", async () => {
const f = await seed();
await retainPreviousSessionClose(f, () => new Promise<void>(() => {}));
@@ -593,7 +593,9 @@ export async function continueFreshInTransaction(tx: Db, input: Input): Promise<
const isolation: FreshContinuationIsolation = fenced || remoteSource ? "remote_epoch"
: writer === "ended" && !pastBound() && !refused ? "writer_ended" : "fresh_isolated_workspace";
if (isolation === "fresh_isolated_workspace" && delivery === "saved_input") delivery = "recorded";
const unverified = await unverifiedActions(tx, run);
const unconfirmed = object(coordinator?.failureDetail).unconfirmedActions;
const unverified = [...new Set([...await unverifiedActions(tx, run),
...(Array.isArray(unconfirmed) ? unconfirmed.filter((name): name is string => typeof name === "string") : [])])].slice(0, 20);
const created = await createdObjects(tx, run);
const note = freshContinuationNote({ runId: run.id, unverified, created,
workspace: isolation === "writer_ended" ? "kept" : "restored_last_save" });
@@ -1,11 +1,46 @@
import type { Db } from "@paperclipai/db";
import { nativeSessionCleanupBlock } from "../../vendor/paperclip-runner/index.js";
import { and, eq } from "drizzle-orm";
import { heartbeatRuns, type Db } from "@paperclipai/db";
import {
completeTerminatedRemoteNativeSessionCleanupForTask,
nativeSessionCleanupBlock,
nativeSessionCleanupRemoteScopes,
} from "../../vendor/paperclip-runner/index.js";
import { logger } from "../../middleware/logger.js";
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
import { terminatedRemoteCleanupScope } from "../remote-execution-termination.js";
async function retireTerminatedRemoteCleanup(db: Db, input: { companyId: string; issueId: string }) {
for (const remoteCleanupScope of nativeSessionCleanupRemoteScopes(input)) {
const lease = await terminatedRemoteCleanupScope(db, input.companyId, remoteCleanupScope).catch(() => null);
if (!lease) continue;
const retired = completeTerminatedRemoteNativeSessionCleanupForTask({ ...input, remoteCleanupScope });
if (retired === 0) continue;
logger.warn({ companyId: input.companyId, issueId: input.issueId, leaseId: lease.id, runId: lease.heartbeatRunId,
provider: lease.provider, providerLeaseId: lease.providerLeaseId, retired },
"native session cleanup retired because its sandbox is confirmed terminated");
if (!lease.heartbeatRunId) continue;
const [run] = await db.select({ agentId: heartbeatRuns.agentId }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, lease.heartbeatRunId)))
.limit(1).catch(() => []);
if (!run) continue;
await appendHeartbeatRunEvent(db, {
companyId: input.companyId, runId: lease.heartbeatRunId, agentId: run.agentId,
eventType: "native.session_cleanup_gate_retired", stream: "system", level: "info",
message: "The task's previous sandbox is confirmed terminated, so its session cleanup gate was retired.",
payload: { leaseId: lease.id, provider: lease.provider, providerLeaseId: lease.providerLeaseId,
termination: lease.metadata?.remoteExecutionTermination ?? null, retired },
}).catch(() => undefined);
}
}
export async function nativeCleanupHoldsAdmission(
db: Db,
input: { companyId: string; issueId: string },
): Promise<"pending" | "operator_required" | null> {
let block = nativeSessionCleanupBlock(input);
if (block === null) return null;
await retireTerminatedRemoteCleanup(db, input);
block = nativeSessionCleanupBlock(input);
if (block === "operator_required") {
const { retireStoppedSessionCleanupGates } = await import("./native-session-executor.js");
await retireStoppedSessionCleanupGates(db, input).catch(() => false);
@@ -226,6 +226,74 @@ const support = externalDatabaseUrl
}
},
);
it.each(["remote_sandbox", "remote_unconfirmed_action", "remote_completed_action", "remote_started_only_action",
"remote_failed_exit_action", "local_codex", "unknown_provider"] as const)(
"continues a transport failure whose stop can never be proven locally without waiting for a person (%s)",
async (mode) => {
const source = await seed();
const workspace = await mkdtemp(join(tmpdir(), "paperclip-remote-transport-"));
const remote = mode.startsWith("remote_");
try {
await db.update(heartbeatRuns).set({ finishedAt: new Date(), errorCode: "provider_transport_failed",
runnerProfileJson: mode === "unknown_provider" ? {} : { nativeExecutionInput: {
provider: { kind: remote ? "opencode" : "codex" }, workspace: { cwd: workspace } } } })
.where(eq(heartbeatRuns.id, source.runId));
await db.update(nativeRunFinalizations).set({ failureCode: "provider_transport_failed", failureDetail: {} })
.where(eq(nativeRunFinalizations.runId, source.runId));
if (remote) await db.insert(environmentLeases).values({ companyId: source.companyId,
issueId: source.issueId, heartbeatRunId: source.runId, provider: "fake-sandbox", providerLeaseId: "sandbox-1",
status: "released", releasedAt: new Date(), cleanupStatus: "success" });
if (mode === "remote_unconfirmed_action") await appendHeartbeatRunEvent(db, {
companyId: source.companyId, runId: source.runId, agentId: source.agentId,
eventType: "tool.execution.started", stream: "system",
payload: { name: "send_email", executionId: "unconfirmed-send", transport: "mcp" },
});
if (mode === "remote_completed_action" || mode === "remote_started_only_action" || mode === "remote_failed_exit_action") {
const execution = { schema: "paperclip.tool.execution.v1", executionId: "prt_1e7c7089f001EN3I04Bfmz",
transport: "mcp", name: "send_email" };
await appendHeartbeatRunEvent(db, { companyId: source.companyId, runId: source.runId, agentId: source.agentId,
eventType: "tool.execution.started", stream: "system",
payload: { prpEvent: { payload: { ...execution, status: "running" } } } });
if (mode !== "remote_started_only_action") await appendHeartbeatRunEvent(db, { companyId: source.companyId,
runId: source.runId, agentId: source.agentId, eventType: "tool.execution.completed", stream: "system",
payload: { prpEvent: { payload: { ...execution, status: "completed",
exitCode: mode === "remote_failed_exit_action" ? 1 : 0 } } } });
}
const verifyStoppedSession = vi.fn(async () => null);
invalidateNativeReplacementDiscovery(db);
await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession });
const [decision] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, source.runId));
const holds = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId));
if (!remote) {
expect(verifyStoppedSession).toHaveBeenCalled();
expect(decision!.failureDetail?.replacementDenied).toBeUndefined();
expect(holds).toHaveLength(0);
return;
}
if (mode === "remote_unconfirmed_action" || mode === "remote_started_only_action" || mode === "remote_failed_exit_action") {
expect(decision!.failureDetail?.replacementDenied).toBe("uncertain_external_action");
expect(holds).toMatchObject([{ status: "active", cause: "uncertain_external_action",
nextAction: expect.stringContaining("send_email") }]);
expect(decision!.failureDetail?.unconfirmedActions).toEqual(["send_email"]);
await settleUnrecoverableExecutions(db);
const [continued] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId));
expect(continued!.evidence).toMatchObject({ freshContinuation: { unverifiedActions: ["send_email"] },
executionReconciliation: { outcomeEvidence: expect.stringContaining("unverified outcomes: send_email") } });
return;
}
expect(decision!.failureDetail?.replacementDenied).toBe("provider_failure_meaning_unverified");
expect(decision!.failureDetail?.unconfirmedActions).toBeUndefined();
expect(holds).toMatchObject([{ status: "active", cause: "provider_failure_meaning_unverified" }]);
await settleUnrecoverableExecutions(db);
const [settled] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId));
expect(settled).toMatchObject({ status: "resolved", outcome: "restored",
evidence: { freshContinuation: { trigger: "recovery_refused", sourceRunId: source.runId, unverifiedActions: [] },
continuationDelivery: "pending" } });
} finally {
await rm(workspace, { recursive: true, force: true });
}
},
);
it("wakes a completed replacement scan after actual local stop proof commits", async () => {
const source = await seed(2);
const workspace = await mkdtemp(join(tmpdir(), "paperclip-late-stop-"));
@@ -21,7 +21,7 @@ import {
type Db,
} from "@paperclipai/db";
import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared";
import { decideNativeReplacement } from "./native-replacement-evidence.js";
import { decideNativeReplacement, type NativeReplacementDecision } from "./native-replacement-evidence.js";
import { issueService } from "../issues.js";
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
import { buildExecutionContinuation } from "../execution-continuation.js";
@@ -435,11 +435,15 @@ export async function reconcileSafeNativeReplacements(
if (historicalListener && !await historicalListenerGuardEvidence(db, run)) continue;
const preClaim = coordinator.failureCode === NATIVE_PRE_CLAIM_FAILURE_CODE;
if (preClaim && !await preClaimFailureEvidence(db, run)) continue;
const stoppedSession = coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener && !preClaim
? await options.verifyStoppedSession?.(run) ?? null : null;
const needsStopProof = coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener && !preClaim;
const stoppedSession = needsStopProof ? await options.verifyStoppedSession?.(run) ?? null : null;
// A transport label alone is not evidence. Keep inspecting these candidates
// as process cleanup and the final transcript become durable.
if (coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener && !preClaim && !stoppedSession) continue;
const providerKind = record(execution.provider).kind;
const stopProofUnavailable = needsStopProof && !stoppedSession
&& ((typeof providerKind === "string" && providerKind !== "codex")
|| leases.some(lease => lease.provider !== null && lease.provider !== "local"));
if (needsStopProof && !stoppedSession && !stopProofUnavailable) continue;
const invocations = await db
.select()
.from(toolInvocations)
@@ -540,7 +544,40 @@ export async function reconcileSafeNativeReplacements(
&& run.runnerProfileJson?.nativeWorkspaceSync == null && nativeRunHasNoWorkspaceAccess(run);
if (preClaim && (invocations.length || !noWorkspaceToSave
&& (!remoteWorkspace?.workspacePreserved || !remoteWorkspace.providerStopped))) continue;
const decision = decideNativeReplacement({
const completedInvocation = (row: typeof invocations[number]) =>
row.status === "succeeded" && Boolean(row.completedAt) && Boolean(row.resultHash);
const executions = stopProofUnavailable ? (await db
.select({ seq: heartbeatRunEvents.seq, eventType: heartbeatRunEvents.eventType, payload: heartbeatRunEvents.payload })
.from(heartbeatRunEvents)
.where(and(eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id),
inArray(heartbeatRunEvents.eventType, ["tool.execution.started", "tool.execution.completed"]))))
.map(event => {
const envelope = record(event.payload);
return { seq: event.seq, eventType: event.eventType,
payload: envelope.prpEvent ? record(record(envelope.prpEvent).payload) : envelope };
}) : [];
const unconfirmedActions = stopProofUnavailable ? [...new Set([
...invocations.filter(row => row.riskLevel !== "read" && !completedInvocation(row)).map(row => row.toolName),
...uncertainProviderActions.flatMap(action => {
const started = executions.find(event => event.eventType === "tool.execution.started"
&& String(event.payload.executionId ?? event.payload.name) === action);
const executionId = started?.payload.executionId;
if (started && executionId != null && executions.some(event => event.eventType === "tool.execution.completed"
&& event.seq > started.seq && event.payload.executionId === executionId && event.payload.status === "completed"
&& (event.payload.exitCode == null || event.payload.exitCode === 0) && event.payload.error == null))
return [];
if (started && started.payload.transport !== "process" && invocations.some(row => completedInvocation(row)
&& (row.id === executionId || row.correlationId === executionId))) return [];
return [typeof started?.payload.name === "string" ? started.payload.name : action];
}),
])] : [];
const unprovenAction = unconfirmedActions[0];
const decision: NativeReplacementDecision = stopProofUnavailable ? unprovenAction ? { allowed: false,
cause: "uncertain_external_action",
nextAction: `Reconcile ${unprovenAction} and preserve its result before continuing. Do not repeat it automatically.` }
: { allowed: false, cause: "provider_failure_meaning_unverified",
nextAction: "The previous session could not be confirmed stopped and recorded no unconfirmed action, so the task continues in a fresh run from its saved work." }
: decideNativeReplacement({
failedSession: true,
// A facade transport failure can conceal an authoritative protocol
// rejection. A stopped process and read receipts do not resolve that.
@@ -618,6 +655,7 @@ export async function reconcileSafeNativeReplacements(
...current.failureDetail,
replacementDenied: decision.cause,
nextAction,
...(unconfirmedActions.length ? { unconfirmedActions } : {}),
},
updatedAt: now,
})
@@ -22413,6 +22413,68 @@ describe("runnerd provider runtime wiring", () => {
leaseOwner: "provider-owner", bootId: "backup-boot", generation: 1 }));
});
it("saves a closing session's backup under its own owner after a failed follow-up released the scope authority", async () => {
const current = { ...execution, binding: { ...execution.binding, runId: randomUUID(), issueId: randomUUID(),
executionWorkspaceId: randomUUID() }, session: { ...execution.session, normalizedSessionId: randomUUID() } };
const coordinator = { phase: "observed", resultId: null as string | null, leaseOwner: "provider-owner" as string | null,
controllerGeneration: 1, controllerBootId: "backup-boot" };
const db = leaseDb(current), select = db.select.bind(db);
db.select = ((fields?: Record<string, unknown>) => ({ from: (table: unknown) => {
if (table !== nativeRunFinalizations) return select(fields as never).from(table as never);
const promise = Promise.resolve([coordinator]);
const query = { then: promise.then.bind(promise), where: () => query, limit: () => query, for: () => query };
return query;
} })) as unknown as Db["select"];
const storage = vi.spyOn(harnessStorage, "durableNativeHarnessStorage").mockReturnValue({ provider: "s3" } as never);
const validateOwner = vi.spyOn(harnessStorage, "assertNativeHarnessBackupOwner").mockResolvedValue(undefined);
const checkpointRegistration = vi.spyOn(runnerCheckpoint, "nativeRunnerCheckpointRegistration")
.mockReturnValue(async () => undefined);
onTestFinished(() => { storage.mockRestore(); validateOwner.mockRestore(); checkpointRegistration.mockRestore(); });
let savingWorkspace = false;
const remoteExecute = vi.fn(async (command: { args?: string[] }) => {
if (savingWorkspace && command.args?.[1]?.includes("base64 <")) throw new Error("stopped-provider-inspection-reached");
return { exitCode: command.args?.[1]?.includes("base64") ? 1 : 0, timedOut: false, stderr: "", stdout:
command.args?.[0] === "--build-metadata" ? JSON.stringify({
schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd", packageName: "@paperclipai/paperclip-runner",
binaryContractVersion: 2, durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"],
}) : command.args?.[0] === "--version" ? "codex-cli 0.160.0" : "" };
});
const controllerArtifact = join(isolatedStateDirectory, "backup-fixture-runnerd");
await writeFile(controllerArtifact, "fixture bootstrap artifact");
await createRunnerdBackend({ db, execution: current, runnerInstanceId: "released-scope-backup-runner",
runnerRemoteBinaryPath: controllerArtifact, runnerRemoteCodexPath: controllerArtifact,
controllerPersistence: { bootId: "backup-boot" }, runnerIngressAuthorized: true, runnerExecutionTarget: {
kind: "remote", transport: "sandbox", remoteCwd: "/workspace", environmentId: "environment", leaseId: "released-scope-backup-lease",
providerKey: "daytona", sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: "released-scope-backup-sandbox" },
effectiveCapabilities: { runnerWebSocketIngress: true }, runner: { execute: remoteExecute, syncIn: vi.fn(async () => undefined) },
getRunnerIngressEndpoint: async () => ({ kind: "authenticated_websocket", websocketUrl: "wss://sandbox.invalid/runner",
secretHeaders: [{ name: "Authorization", value: "fixture-ingress" }], generation: "1", refresh: vi.fn(), close: vi.fn(async () => undefined) }),
} as never });
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & {
controlPlaneRegistration: NonNullable<import("../../vendor/paperclip-runner/index.js").RunnerExecutionDriver["controlPlaneRegistration"]>;
};
const controller = new DurablePrpControlPlane({ stateDirectory: join(options.stateDirectory!, "control-plane"),
identity: options.prpIdentity! as ConstructorParameters<typeof DurablePrpControlPlane>[0]["identity"],
expectedRunnerVersion: "backup-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
controller.queueCommand("run.prepare", {});
controller.queueCommand("session.open", { reuse: "same_session" });
let registration: Awaited<ReturnType<typeof options.controlPlaneRegistration>> | undefined;
onTestFinished(async () => { await registration?.release(); await controller.stop(); });
registration = await options.controlPlaneRegistration(controller);
coordinator.phase = "workspace_finalizing";
coordinator.resultId = "accepted-result";
coordinator.leaseOwner = null;
savingWorkspace = true;
const authorities = sessionToolAuthorityCount();
await expect(createRunnerdBackend({ db, execution: current, runnerInstanceId: "released-scope-backup-runner",
executionContext: { signal: AbortSignal.abort() } as never })).rejects.toThrow();
expect(sessionToolAuthorityCount()).toBe(authorities - 1);
await expect(registration.checkpoint!("settled")).rejects.toThrow("stopped-provider-inspection-reached");
expect(validateOwner).toHaveBeenCalledWith(db, expect.objectContaining({ runId: current.binding.runId,
leaseOwner: "provider-owner", bootId: "backup-boot", generation: 1 }));
});
it.each([true, false])("restores a migrated warm backup into a created lease only for its exact session (matching: %s)", async matching => {
const remoteCwd = join(isolatedStateDirectory, "remote");
const current: NativeExecutionInputV1 = { ...execution,
@@ -18130,8 +18130,10 @@ async function createRunnerdBackendWithinSessionClaim(
)
return false;
const durableStorage = durableNativeHarnessStorage();
const currentAuthority = sessionToolAuthorityEpochs.get(sessionScopeId);
const backupOwner = durableStorage && currentAuthority ? sessionHarnessBackupOwners.get(currentAuthority) : null;
const currentAuthority = sessionToolAuthorityEpochs.get(sessionScopeId) ?? authorityEpoch;
const backupOwner = durableStorage
? sessionHarnessBackupOwners.get(currentAuthority) ?? sessionHarnessBackupOwners.get(authorityEpoch) ?? null
: null;
if (durableStorage && !backupOwner) throw new Error("native_harness_storage_owner_missing");
if (backupOwner) await assertNativeHarnessBackupOwner(input.db, backupOwner);
// Transport release also runs when provider bootstrap failed. In that case
@@ -18929,9 +18931,7 @@ async function createRunnerdBackendWithinSessionClaim(
target,
controllerPersistence: input.controllerPersistence ? nativeRunnerCheckpointPersistence(
input.db, input.controllerPersistence.bootId, () => {
const current = sessionToolAuthorityEpochs.get(sessionScopeId);
if (!current) throw new Error("native_session_tool_authority_unavailable");
return current.runId;
return (sessionToolAuthorityEpochs.get(sessionScopeId) ?? authorityEpoch).runId;
}, detachNativeSessionsForRestart) : undefined,
binding: input.remoteWarmRecovery
? { ...input.execution.binding, runId: input.remoteWarmRecovery.identity.runId }
@@ -1,4 +1,4 @@
import { and, eq } from "drizzle-orm";
import { and, desc, eq } from "drizzle-orm";
import { environmentLeases, type Db } from "@paperclipai/db";
import { retiredReusableLeasePredecessors } from "./environment-lease-handoff.js";
@@ -129,3 +129,21 @@ export async function remoteExecutionHasReleased(db: Db, companyId: string, runI
const leases = await runRemoteLeases(db, companyId, runId);
return leases.length > 0 && leases.every(hasRemoteTerminationReceipt);
}
export async function terminatedRemoteCleanupScope(db: Db, companyId: string, remoteCleanupScope: string) {
let parsed: unknown;
try {
parsed = JSON.parse(remoteCleanupScope);
} catch {
return null;
}
if (!Array.isArray(parsed) || parsed.length !== 2 || typeof parsed[0] !== "string" || typeof parsed[1] !== "string"
|| parsed[0] === "local") return null;
const [provider, providerLeaseId] = parsed as [string, string];
const leases = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId),
eq(environmentLeases.provider, provider), eq(environmentLeases.providerLeaseId, providerLeaseId)))
.orderBy(desc(environmentLeases.createdAt), desc(environmentLeases.id));
if (leases.length === 0 || leases.some(lease => !lease.releasedAt)) return null;
const newest = await resolveRemoteTerminationLease(db, leases[0]!);
return hasRemoteTerminationReceipt(newest) ? newest : null;
}
+2
View File
@@ -213,6 +213,8 @@ export const nativeRestartContinuationInFlight = runner.nativeRestartContinuatio
export const completeTerminatedRemoteNativeSessionCleanup = runner.completeTerminatedRemoteNativeSessionCleanup;
export const completeTerminatedLocalNativeSessionCleanup = runner.completeTerminatedLocalNativeSessionCleanup;
export const completeTerminatedRemoteNativeSessionCleanupForTask = runner.completeTerminatedRemoteNativeSessionCleanupForTask;
export const nativeSessionCleanupRemoteScopes = runner.nativeSessionCleanupRemoteScopes;
export const nativeSessionCleanupBlock = runner.nativeSessionCleanupBlock;
export const onNativeSessionCleanupSettled = runner.onNativeSessionCleanupSettled;
export const onNativeSessionCleanupQuarantined = runner.onNativeSessionCleanupQuarantined;