From 22d3f9a03c481e6f373bfa6221e7c3a77051d245 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 17 Nov 2026 02:25:02 -0600 Subject: [PATCH] fix(native): finish a failed warm session's cleanup by itself and continue the task (#21977) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runner runs keep a provider session per task. A follow-up run can reuse the warm session of the run before it, in a remote sandbox. > - When a warm follow-up fails, the previous session is closed. The close saves a backup of the provider session first. > - The close looked up the backup owner through the task's current tool authority. The failed follow-up had already released that authority. Every close attempt failed with `native_harness_storage_owner_missing`, so the close could never finish. > - After nine failed attempts, the task's admission gate became `operator_required`. The gate retires only for a stopped local process or for the exact run that owns the session. A remote sandbox session owned by an earlier run never matched. A queued continuation and a Retry stayed queued forever, and the task showed a Retry button that could not work. > - Separately, a remote transport failure could never produce the local stop proof that the safe-replacement lane waits for. Nothing continued the task until an unrelated wake arrived 15 minutes later. > - This pull request makes the close save its backup under its own owner, retires a cleanup gate when the task's sandbox is confirmed terminated, and refuses an unprovable remote transport failure at once so the task continues in a fresh isolated run. > - The benefit is that recovery finishes by itself, and a Retry that is shown starts a run. ## Linked Issues or Issue Description **What happened?** In a restart-chaos wave, a warm follow-up run on a remote sandbox failed with `provider_transport_failed` (`PRP command run.attach timed out`). The previous run's session close then failed on every attempt with `native_harness_storage_owner_missing`. The cleanup gate became `operator_required`. The continuation run was created 15 minutes later and then stayed queued until the wave ended ("queued run kept queued while the task's previous session finishes cleanup"). The task showed a Retry banner, but a Retry would wait behind the same gate. **Expected behavior** A failed session close settles or is retired by itself with evidence. The next run of the task starts in seconds. A Retry that is shown starts a run. **Steps to reproduce** 1. Run a native task on a remote sandbox with a warm session lifecycle. 2. Make the warm follow-up run fail during startup, after it has reused the previous session. 3. Watch the previous session's close: every attempt fails with `native_harness_storage_owner_missing`, and the queued next run of the task never starts. **Paperclip version or commit** master at b2b2c1e60b (wave) and 7685b7a4fc (this branch base). **Deployment mode** Hosted server with remote Daytona sandboxes (restart-chaos harness). ## What Changed - `native-session-executor.ts`: the remote checkpoint resolves its backup owner from the current scope authority, and falls back to the backend's own authority epoch when the scope has none. The controller-persistence `authorityRunId` closure uses the same fallback. A newer run that took over the session still wins. - `paperclip-runner` `native-session-runtime.ts`: new `nativeSessionCleanupRemoteScopes` lists the remote resources whose quarantined cleanup holds a task. New `completeTerminatedRemoteNativeSessionCleanupForTask` retires settled quarantines of the task in one terminated remote resource, whichever run's session they hold. An in-flight close keeps its entry. - `remote-execution-termination.ts`: new `terminatedRemoteCleanupScope` confirms that the company's newest lease on the remote resource is released with its termination receipt and that no lease on it is still held. - `native-cleanup-admission-hold.ts`: queue admission now retires a held cleanup when its sandbox is confirmed terminated. It logs the retirement and records a `native.session_cleanup_gate_retired` run event with the termination receipt. The local stopped-process path is unchanged. - `native-safe-replacement.ts`: a transport failure on a remote or non-Codex session can never produce the local stop proof. It is now refused at once with `provider_failure_meaning_unverified`. The existing disposition pass then continues the task in a fresh isolated run from saved work. A local Codex run keeps the existing wait for stop proof. ## Verification - Red/green, each verified red without its fix: - `server/src/services/native-runtime/native-session-executor.test.ts` — "saves a closing session's backup under its own owner after a failed follow-up released the scope authority". Red: `native_harness_storage_owner_missing` (the incident error). - `server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts` (embedded Postgres) — "starts a queued run once the previous session's sandbox is confirmed terminated, whichever run released it", plus "keeps the run queued while a different sandbox of the task is terminated". Red: the queued run stays queued. - `server/src/services/native-runtime/native-safe-replacement.test.ts` (embedded Postgres) — "continues a transport failure whose stop can never be proven locally without waiting for a person" for a remote sandbox (refused and continued fresh) and a local Codex run (unchanged). Red: no denial and no hold. - `pnpm verify` and server and runner typechecks pass locally. ## Risks - The admission gate now trusts a provider termination receipt on the newest lease of the same remote resource, not only one bound to the session's own run. A reused sandbox with a newer held lease is not retired. - A remote transport failure no longer waits for a stop proof that cannot arrive. The fresh continuation is isolated (write authority revoked, new root) and stays under the existing per-task and per-minute continuation brakes. Nothing is added before `turn.start`. - The run.attach timeout itself is not fixed here. The controller held the attach frame behind a checkpoint flush to the follow-up's writer. The run's own diagnostics were lost because they were written to the predecessor's run log. This PR makes the failure recover by itself; the attach stall needs a separate fix with better diagnostics. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../src/native-session-runtime.ts | 41 +++++++++++ ...beat-native-cleanup-admission-hold.test.ts | 63 ++++++++++++++++- .../services/fresh-continuation-fallback.ts | 4 +- .../native-cleanup-admission-hold.ts | 39 ++++++++++- .../native-safe-replacement.test.ts | 68 +++++++++++++++++++ .../native-runtime/native-safe-replacement.ts | 48 +++++++++++-- .../native-session-executor.test.ts | 62 +++++++++++++++++ .../native-runtime/native-session-executor.ts | 10 +-- .../services/remote-execution-termination.ts | 20 +++++- server/src/vendor/paperclip-runner/index.ts | 2 + 10 files changed, 341 insertions(+), 16 deletions(-) diff --git a/packages/paperclip-runner/src/native-session-runtime.ts b/packages/paperclip-runner/src/native-session-runtime.ts index e5aa6c1a5f0..ce262c411af 100644 --- a/packages/paperclip-runner/src/native-session-runtime.ts +++ b/packages/paperclip-runner/src/native-session-runtime.ts @@ -275,6 +275,47 @@ export function completeTerminatedRemoteNativeSessionCleanup(binding: { return true; } +function quarantinedRemoteCleanupScope( + entry: QuarantinedSessionCleanup, + input: { companyId: string; issueId: string }, +): string | null { + const [companyId, , , remoteCleanupScope] = JSON.parse(entry.domain.key) as string[]; + return companyId === input.companyId && remoteCleanupScope && + entry.domain.resources.includes(JSON.stringify(["issue", input.issueId])) + ? remoteCleanupScope : null; +} + +export function nativeSessionCleanupRemoteScopes(input: { + companyId: string; + issueId: string; +}): string[] { + const scopes = new Set(); + for (const entry of quarantinedSessionCleanups) { + const scope = quarantinedRemoteCleanupScope(entry, input); + if (scope) scopes.add(scope); + } + return [...scopes]; +} + +export function completeTerminatedRemoteNativeSessionCleanupForTask(input: { + companyId: string; + issueId: string; + remoteCleanupScope: string; +}): number { + if (!input.remoteCleanupScope) return 0; + let retired = 0; + for (const entry of [...quarantinedSessionCleanups]) { + if (quarantinedRemoteCleanupScope(entry, input) !== input.remoteCleanupScope) continue; + if (entry.attempt || entry.recovery) continue; + if (entry.timer) clearTimeout(entry.timer); + entry.timer = null; + quarantinedSessionCleanups.delete(entry); + announceCleanupSettled(entry.domain); + retired += 1; + } + return retired; +} + /** Host-only counterpart of remote resource termination. The caller has verified * both exact local process identities and durably fenced their run. This removes * only cleanup ownership; the retained checkpoint is never made resumable. diff --git a/server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts b/server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts index 3bcdfa8b9cb..60eace1b4c5 100644 --- a/server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts +++ b/server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts @@ -5,7 +5,7 @@ import { join } from "node:path"; import { eq } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { hostname } from "node:os"; -import { agents, companies, createDb, heartbeatRuns, issues } from "@paperclipai/db"; +import { agents, companies, createDb, environmentLeases, heartbeatRuns, issues } from "@paperclipai/db"; import { executeNativeSession, NativeSessionCloseUnrecoverableError, @@ -55,7 +55,8 @@ describeEmbeddedPostgres("queued native run behind the task's previous session c } async function retainPreviousSessionClose(f: Awaited>, close: () => Promise, - owner: { runId: string; runnerInstanceId: string } = { runId: randomUUID(), runnerInstanceId: randomUUID() }) { + owner: { runId: string; runnerInstanceId: string } = { runId: randomUUID(), runnerInstanceId: randomUUID() }, + remoteCleanupScope?: string) { const runId = owner.runId; const identity = { runId, sessionId: `closing-${runId}`, companyId: f.companyId, issueId: f.issueId, agentId: f.previousId }; const capabilities = { resume: true, typedEvents: true, steering: false, interruption: false, structuredResult: true }; @@ -83,6 +84,7 @@ describeEmbeddedPostgres("queued native run behind the task's previous session c appendEvent: async () => ({ cursor: 0, highestContiguousSourceSeq: 0, disposition: "committed" }), replayEvents: async () => ({ events: [], highestContiguousSourceSeq: 0 }), completeRun: async () => {} }, runnerInstanceId: owner.runnerInstanceId, controlPlaneInstanceId: "control", + ...(remoteCleanupScope ? { remoteCleanupScope } : {}), })).rejects.toThrow("stopped for reassignment"); } @@ -167,6 +169,63 @@ describeEmbeddedPostgres("queued native run behind the task's previous session c expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null, errorCode: null }); }); + async function recordSandboxLease(f: Awaited>, providerLeaseId: string, terminated: boolean) { + const [failed] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.nextId, + invocationSource: "automation", status: "failed", errorCode: "provider_transport_failed", runtimeMode: "native", + nativeIssueId: f.issueId, startedAt: new Date(Date.now() - 60_000), finishedAt: new Date(), + contextSnapshot: { issueId: f.issueId } }).returning(); + const leaseId = randomUUID(); + await db.insert(environmentLeases).values({ id: leaseId, companyId: f.companyId, issueId: f.issueId, + heartbeatRunId: failed!.id, provider: "fake-sandbox", providerLeaseId, leasePolicy: "ephemeral", + status: terminated ? "released" : "active", releasedAt: terminated ? new Date() : null, + cleanupStatus: terminated ? "success" : null, + metadata: terminated ? { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId: f.companyId, + runId: failed!.id, leaseId, provider: "fake-sandbox", providerLeaseId, state: "stopped", + confirmedAt: new Date().toISOString() } } : {} }); + return { leaseId, runId: failed!.id }; + } + + it("starts a queued run once the previous session's sandbox is confirmed terminated, whichever run released it", async () => { + const worker = dispatchWorker(); + try { + const f = await seed(); + const providerLeaseId = `sandbox-${randomUUID()}`; + await retainPreviousSessionClose(f, async () => { + throw new NativeSessionCloseUnrecoverableError({ runnerSuspended: false, providerDrained: false }); + }, undefined, JSON.stringify(["fake-sandbox", providerLeaseId])); + expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("operator_required"); + const held = await recordSandboxLease(f, providerLeaseId, false); + const run = await queueNextAssigneeRun(f); + await heartbeatService(db).resumeQueuedRuns(); + expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null }); + expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("operator_required"); + + await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success", + metadata: { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId: f.companyId, + runId: held.runId, leaseId: held.leaseId, provider: "fake-sandbox", providerLeaseId, state: "stopped", + confirmedAt: new Date().toISOString() } } }).where(eq(environmentLeases.id, held.leaseId)); + await heartbeatService(db).resumeQueuedRuns(); + await expect.poll(async () => (await readRun(run.id)).status, { timeout: 15_000 }).not.toBe("queued"); + expect((await readRun(run.id)).errorCode).not.toBe("native_session_cleanup_quarantined"); + expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBeNull(); + } finally { + worker.stop(); + } + }); + + it("keeps the run queued while a different sandbox of the task is terminated", async () => { + const f = await seed(); + const providerLeaseId = `sandbox-${randomUUID()}`; + await retainPreviousSessionClose(f, async () => { + throw new NativeSessionCloseUnrecoverableError({ runnerSuspended: false, providerDrained: false }); + }, undefined, JSON.stringify(["fake-sandbox", providerLeaseId])); + await recordSandboxLease(f, `other-${providerLeaseId}`, true); + const run = await queueNextAssigneeRun(f); + await heartbeatService(db).resumeQueuedRuns(); + expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null }); + expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("operator_required"); + }); + it("does not hold a run for a different task", async () => { const f = await seed(); await retainPreviousSessionClose(f, () => new Promise(() => {})); diff --git a/server/src/services/fresh-continuation-fallback.ts b/server/src/services/fresh-continuation-fallback.ts index 87b58d0f21e..a59dada1c1f 100644 --- a/server/src/services/fresh-continuation-fallback.ts +++ b/server/src/services/fresh-continuation-fallback.ts @@ -593,7 +593,9 @@ export async function continueFreshInTransaction(tx: Db, input: Input): Promise< const isolation: FreshContinuationIsolation = fenced || remoteSource ? "remote_epoch" : writer === "ended" && !pastBound() && !refused ? "writer_ended" : "fresh_isolated_workspace"; if (isolation === "fresh_isolated_workspace" && delivery === "saved_input") delivery = "recorded"; - const unverified = await unverifiedActions(tx, run); + const unconfirmed = object(coordinator?.failureDetail).unconfirmedActions; + const unverified = [...new Set([...await unverifiedActions(tx, run), + ...(Array.isArray(unconfirmed) ? unconfirmed.filter((name): name is string => typeof name === "string") : [])])].slice(0, 20); const created = await createdObjects(tx, run); const note = freshContinuationNote({ runId: run.id, unverified, created, workspace: isolation === "writer_ended" ? "kept" : "restored_last_save" }); diff --git a/server/src/services/native-runtime/native-cleanup-admission-hold.ts b/server/src/services/native-runtime/native-cleanup-admission-hold.ts index 56581df8453..7aedfcea6e3 100644 --- a/server/src/services/native-runtime/native-cleanup-admission-hold.ts +++ b/server/src/services/native-runtime/native-cleanup-admission-hold.ts @@ -1,11 +1,46 @@ -import type { Db } from "@paperclipai/db"; -import { nativeSessionCleanupBlock } from "../../vendor/paperclip-runner/index.js"; +import { and, eq } from "drizzle-orm"; +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { + completeTerminatedRemoteNativeSessionCleanupForTask, + nativeSessionCleanupBlock, + nativeSessionCleanupRemoteScopes, +} from "../../vendor/paperclip-runner/index.js"; +import { logger } from "../../middleware/logger.js"; +import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; +import { terminatedRemoteCleanupScope } from "../remote-execution-termination.js"; + +async function retireTerminatedRemoteCleanup(db: Db, input: { companyId: string; issueId: string }) { + for (const remoteCleanupScope of nativeSessionCleanupRemoteScopes(input)) { + const lease = await terminatedRemoteCleanupScope(db, input.companyId, remoteCleanupScope).catch(() => null); + if (!lease) continue; + const retired = completeTerminatedRemoteNativeSessionCleanupForTask({ ...input, remoteCleanupScope }); + if (retired === 0) continue; + logger.warn({ companyId: input.companyId, issueId: input.issueId, leaseId: lease.id, runId: lease.heartbeatRunId, + provider: lease.provider, providerLeaseId: lease.providerLeaseId, retired }, + "native session cleanup retired because its sandbox is confirmed terminated"); + if (!lease.heartbeatRunId) continue; + const [run] = await db.select({ agentId: heartbeatRuns.agentId }).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, lease.heartbeatRunId))) + .limit(1).catch(() => []); + if (!run) continue; + await appendHeartbeatRunEvent(db, { + companyId: input.companyId, runId: lease.heartbeatRunId, agentId: run.agentId, + eventType: "native.session_cleanup_gate_retired", stream: "system", level: "info", + message: "The task's previous sandbox is confirmed terminated, so its session cleanup gate was retired.", + payload: { leaseId: lease.id, provider: lease.provider, providerLeaseId: lease.providerLeaseId, + termination: lease.metadata?.remoteExecutionTermination ?? null, retired }, + }).catch(() => undefined); + } +} export async function nativeCleanupHoldsAdmission( db: Db, input: { companyId: string; issueId: string }, ): Promise<"pending" | "operator_required" | null> { let block = nativeSessionCleanupBlock(input); + if (block === null) return null; + await retireTerminatedRemoteCleanup(db, input); + block = nativeSessionCleanupBlock(input); if (block === "operator_required") { const { retireStoppedSessionCleanupGates } = await import("./native-session-executor.js"); await retireStoppedSessionCleanupGates(db, input).catch(() => false); diff --git a/server/src/services/native-runtime/native-safe-replacement.test.ts b/server/src/services/native-runtime/native-safe-replacement.test.ts index a0fdc9116c7..e7c4b033609 100644 --- a/server/src/services/native-runtime/native-safe-replacement.test.ts +++ b/server/src/services/native-runtime/native-safe-replacement.test.ts @@ -226,6 +226,74 @@ const support = externalDatabaseUrl } }, ); + it.each(["remote_sandbox", "remote_unconfirmed_action", "remote_completed_action", "remote_started_only_action", + "remote_failed_exit_action", "local_codex", "unknown_provider"] as const)( + "continues a transport failure whose stop can never be proven locally without waiting for a person (%s)", + async (mode) => { + const source = await seed(); + const workspace = await mkdtemp(join(tmpdir(), "paperclip-remote-transport-")); + const remote = mode.startsWith("remote_"); + try { + await db.update(heartbeatRuns).set({ finishedAt: new Date(), errorCode: "provider_transport_failed", + runnerProfileJson: mode === "unknown_provider" ? {} : { nativeExecutionInput: { + provider: { kind: remote ? "opencode" : "codex" }, workspace: { cwd: workspace } } } }) + .where(eq(heartbeatRuns.id, source.runId)); + await db.update(nativeRunFinalizations).set({ failureCode: "provider_transport_failed", failureDetail: {} }) + .where(eq(nativeRunFinalizations.runId, source.runId)); + if (remote) await db.insert(environmentLeases).values({ companyId: source.companyId, + issueId: source.issueId, heartbeatRunId: source.runId, provider: "fake-sandbox", providerLeaseId: "sandbox-1", + status: "released", releasedAt: new Date(), cleanupStatus: "success" }); + if (mode === "remote_unconfirmed_action") await appendHeartbeatRunEvent(db, { + companyId: source.companyId, runId: source.runId, agentId: source.agentId, + eventType: "tool.execution.started", stream: "system", + payload: { name: "send_email", executionId: "unconfirmed-send", transport: "mcp" }, + }); + if (mode === "remote_completed_action" || mode === "remote_started_only_action" || mode === "remote_failed_exit_action") { + const execution = { schema: "paperclip.tool.execution.v1", executionId: "prt_1e7c7089f001EN3I04Bfmz", + transport: "mcp", name: "send_email" }; + await appendHeartbeatRunEvent(db, { companyId: source.companyId, runId: source.runId, agentId: source.agentId, + eventType: "tool.execution.started", stream: "system", + payload: { prpEvent: { payload: { ...execution, status: "running" } } } }); + if (mode !== "remote_started_only_action") await appendHeartbeatRunEvent(db, { companyId: source.companyId, + runId: source.runId, agentId: source.agentId, eventType: "tool.execution.completed", stream: "system", + payload: { prpEvent: { payload: { ...execution, status: "completed", + exitCode: mode === "remote_failed_exit_action" ? 1 : 0 } } } }); + } + const verifyStoppedSession = vi.fn(async () => null); + invalidateNativeReplacementDiscovery(db); + await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession }); + const [decision] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, source.runId)); + const holds = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId)); + if (!remote) { + expect(verifyStoppedSession).toHaveBeenCalled(); + expect(decision!.failureDetail?.replacementDenied).toBeUndefined(); + expect(holds).toHaveLength(0); + return; + } + if (mode === "remote_unconfirmed_action" || mode === "remote_started_only_action" || mode === "remote_failed_exit_action") { + expect(decision!.failureDetail?.replacementDenied).toBe("uncertain_external_action"); + expect(holds).toMatchObject([{ status: "active", cause: "uncertain_external_action", + nextAction: expect.stringContaining("send_email") }]); + expect(decision!.failureDetail?.unconfirmedActions).toEqual(["send_email"]); + await settleUnrecoverableExecutions(db); + const [continued] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId)); + expect(continued!.evidence).toMatchObject({ freshContinuation: { unverifiedActions: ["send_email"] }, + executionReconciliation: { outcomeEvidence: expect.stringContaining("unverified outcomes: send_email") } }); + return; + } + expect(decision!.failureDetail?.replacementDenied).toBe("provider_failure_meaning_unverified"); + expect(decision!.failureDetail?.unconfirmedActions).toBeUndefined(); + expect(holds).toMatchObject([{ status: "active", cause: "provider_failure_meaning_unverified" }]); + await settleUnrecoverableExecutions(db); + const [settled] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId)); + expect(settled).toMatchObject({ status: "resolved", outcome: "restored", + evidence: { freshContinuation: { trigger: "recovery_refused", sourceRunId: source.runId, unverifiedActions: [] }, + continuationDelivery: "pending" } }); + } finally { + await rm(workspace, { recursive: true, force: true }); + } + }, + ); it("wakes a completed replacement scan after actual local stop proof commits", async () => { const source = await seed(2); const workspace = await mkdtemp(join(tmpdir(), "paperclip-late-stop-")); diff --git a/server/src/services/native-runtime/native-safe-replacement.ts b/server/src/services/native-runtime/native-safe-replacement.ts index 61d52c28aaf..0ca6f833b50 100644 --- a/server/src/services/native-runtime/native-safe-replacement.ts +++ b/server/src/services/native-runtime/native-safe-replacement.ts @@ -21,7 +21,7 @@ import { type Db, } from "@paperclipai/db"; import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared"; -import { decideNativeReplacement } from "./native-replacement-evidence.js"; +import { decideNativeReplacement, type NativeReplacementDecision } from "./native-replacement-evidence.js"; import { issueService } from "../issues.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { buildExecutionContinuation } from "../execution-continuation.js"; @@ -435,11 +435,15 @@ export async function reconcileSafeNativeReplacements( if (historicalListener && !await historicalListenerGuardEvidence(db, run)) continue; const preClaim = coordinator.failureCode === NATIVE_PRE_CLAIM_FAILURE_CODE; if (preClaim && !await preClaimFailureEvidence(db, run)) continue; - const stoppedSession = coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener && !preClaim - ? await options.verifyStoppedSession?.(run) ?? null : null; + const needsStopProof = coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener && !preClaim; + const stoppedSession = needsStopProof ? await options.verifyStoppedSession?.(run) ?? null : null; // A transport label alone is not evidence. Keep inspecting these candidates // as process cleanup and the final transcript become durable. - if (coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener && !preClaim && !stoppedSession) continue; + const providerKind = record(execution.provider).kind; + const stopProofUnavailable = needsStopProof && !stoppedSession + && ((typeof providerKind === "string" && providerKind !== "codex") + || leases.some(lease => lease.provider !== null && lease.provider !== "local")); + if (needsStopProof && !stoppedSession && !stopProofUnavailable) continue; const invocations = await db .select() .from(toolInvocations) @@ -540,7 +544,40 @@ export async function reconcileSafeNativeReplacements( && run.runnerProfileJson?.nativeWorkspaceSync == null && nativeRunHasNoWorkspaceAccess(run); if (preClaim && (invocations.length || !noWorkspaceToSave && (!remoteWorkspace?.workspacePreserved || !remoteWorkspace.providerStopped))) continue; - const decision = decideNativeReplacement({ + const completedInvocation = (row: typeof invocations[number]) => + row.status === "succeeded" && Boolean(row.completedAt) && Boolean(row.resultHash); + const executions = stopProofUnavailable ? (await db + .select({ seq: heartbeatRunEvents.seq, eventType: heartbeatRunEvents.eventType, payload: heartbeatRunEvents.payload }) + .from(heartbeatRunEvents) + .where(and(eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id), + inArray(heartbeatRunEvents.eventType, ["tool.execution.started", "tool.execution.completed"])))) + .map(event => { + const envelope = record(event.payload); + return { seq: event.seq, eventType: event.eventType, + payload: envelope.prpEvent ? record(record(envelope.prpEvent).payload) : envelope }; + }) : []; + const unconfirmedActions = stopProofUnavailable ? [...new Set([ + ...invocations.filter(row => row.riskLevel !== "read" && !completedInvocation(row)).map(row => row.toolName), + ...uncertainProviderActions.flatMap(action => { + const started = executions.find(event => event.eventType === "tool.execution.started" + && String(event.payload.executionId ?? event.payload.name) === action); + const executionId = started?.payload.executionId; + if (started && executionId != null && executions.some(event => event.eventType === "tool.execution.completed" + && event.seq > started.seq && event.payload.executionId === executionId && event.payload.status === "completed" + && (event.payload.exitCode == null || event.payload.exitCode === 0) && event.payload.error == null)) + return []; + if (started && started.payload.transport !== "process" && invocations.some(row => completedInvocation(row) + && (row.id === executionId || row.correlationId === executionId))) return []; + return [typeof started?.payload.name === "string" ? started.payload.name : action]; + }), + ])] : []; + const unprovenAction = unconfirmedActions[0]; + const decision: NativeReplacementDecision = stopProofUnavailable ? unprovenAction ? { allowed: false, + cause: "uncertain_external_action", + nextAction: `Reconcile ${unprovenAction} and preserve its result before continuing. Do not repeat it automatically.` } + : { allowed: false, cause: "provider_failure_meaning_unverified", + nextAction: "The previous session could not be confirmed stopped and recorded no unconfirmed action, so the task continues in a fresh run from its saved work." } + : decideNativeReplacement({ failedSession: true, // A facade transport failure can conceal an authoritative protocol // rejection. A stopped process and read receipts do not resolve that. @@ -618,6 +655,7 @@ export async function reconcileSafeNativeReplacements( ...current.failureDetail, replacementDenied: decision.cause, nextAction, + ...(unconfirmedActions.length ? { unconfirmedActions } : {}), }, updatedAt: now, }) diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 0dd70c67740..db296a1335e 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -22413,6 +22413,68 @@ describe("runnerd provider runtime wiring", () => { leaseOwner: "provider-owner", bootId: "backup-boot", generation: 1 })); }); + it("saves a closing session's backup under its own owner after a failed follow-up released the scope authority", async () => { + const current = { ...execution, binding: { ...execution.binding, runId: randomUUID(), issueId: randomUUID(), + executionWorkspaceId: randomUUID() }, session: { ...execution.session, normalizedSessionId: randomUUID() } }; + const coordinator = { phase: "observed", resultId: null as string | null, leaseOwner: "provider-owner" as string | null, + controllerGeneration: 1, controllerBootId: "backup-boot" }; + const db = leaseDb(current), select = db.select.bind(db); + db.select = ((fields?: Record) => ({ from: (table: unknown) => { + if (table !== nativeRunFinalizations) return select(fields as never).from(table as never); + const promise = Promise.resolve([coordinator]); + const query = { then: promise.then.bind(promise), where: () => query, limit: () => query, for: () => query }; + return query; + } })) as unknown as Db["select"]; + const storage = vi.spyOn(harnessStorage, "durableNativeHarnessStorage").mockReturnValue({ provider: "s3" } as never); + const validateOwner = vi.spyOn(harnessStorage, "assertNativeHarnessBackupOwner").mockResolvedValue(undefined); + const checkpointRegistration = vi.spyOn(runnerCheckpoint, "nativeRunnerCheckpointRegistration") + .mockReturnValue(async () => undefined); + onTestFinished(() => { storage.mockRestore(); validateOwner.mockRestore(); checkpointRegistration.mockRestore(); }); + let savingWorkspace = false; + const remoteExecute = vi.fn(async (command: { args?: string[] }) => { + if (savingWorkspace && command.args?.[1]?.includes("base64 <")) throw new Error("stopped-provider-inspection-reached"); + return { exitCode: command.args?.[1]?.includes("base64") ? 1 : 0, timedOut: false, stderr: "", stdout: + command.args?.[0] === "--build-metadata" ? JSON.stringify({ + schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd", packageName: "@paperclipai/paperclip-runner", + binaryContractVersion: 2, durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"], + }) : command.args?.[0] === "--version" ? "codex-cli 0.160.0" : "" }; + }); + const controllerArtifact = join(isolatedStateDirectory, "backup-fixture-runnerd"); + await writeFile(controllerArtifact, "fixture bootstrap artifact"); + await createRunnerdBackend({ db, execution: current, runnerInstanceId: "released-scope-backup-runner", + runnerRemoteBinaryPath: controllerArtifact, runnerRemoteCodexPath: controllerArtifact, + controllerPersistence: { bootId: "backup-boot" }, runnerIngressAuthorized: true, runnerExecutionTarget: { + kind: "remote", transport: "sandbox", remoteCwd: "/workspace", environmentId: "environment", leaseId: "released-scope-backup-lease", + providerKey: "daytona", sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: "released-scope-backup-sandbox" }, + effectiveCapabilities: { runnerWebSocketIngress: true }, runner: { execute: remoteExecute, syncIn: vi.fn(async () => undefined) }, + getRunnerIngressEndpoint: async () => ({ kind: "authenticated_websocket", websocketUrl: "wss://sandbox.invalid/runner", + secretHeaders: [{ name: "Authorization", value: "fixture-ingress" }], generation: "1", refresh: vi.fn(), close: vi.fn(async () => undefined) }), + } as never }); + state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); + const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { + controlPlaneRegistration: NonNullable; + }; + const controller = new DurablePrpControlPlane({ stateDirectory: join(options.stateDirectory!, "control-plane"), + identity: options.prpIdentity! as ConstructorParameters[0]["identity"], + expectedRunnerVersion: "backup-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` }); + controller.queueCommand("run.prepare", {}); + controller.queueCommand("session.open", { reuse: "same_session" }); + let registration: Awaited> | undefined; + onTestFinished(async () => { await registration?.release(); await controller.stop(); }); + registration = await options.controlPlaneRegistration(controller); + coordinator.phase = "workspace_finalizing"; + coordinator.resultId = "accepted-result"; + coordinator.leaseOwner = null; + savingWorkspace = true; + const authorities = sessionToolAuthorityCount(); + await expect(createRunnerdBackend({ db, execution: current, runnerInstanceId: "released-scope-backup-runner", + executionContext: { signal: AbortSignal.abort() } as never })).rejects.toThrow(); + expect(sessionToolAuthorityCount()).toBe(authorities - 1); + await expect(registration.checkpoint!("settled")).rejects.toThrow("stopped-provider-inspection-reached"); + expect(validateOwner).toHaveBeenCalledWith(db, expect.objectContaining({ runId: current.binding.runId, + leaseOwner: "provider-owner", bootId: "backup-boot", generation: 1 })); + }); + it.each([true, false])("restores a migrated warm backup into a created lease only for its exact session (matching: %s)", async matching => { const remoteCwd = join(isolatedStateDirectory, "remote"); const current: NativeExecutionInputV1 = { ...execution, diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index bee52eec15b..564a59f6e67 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -18130,8 +18130,10 @@ async function createRunnerdBackendWithinSessionClaim( ) return false; const durableStorage = durableNativeHarnessStorage(); - const currentAuthority = sessionToolAuthorityEpochs.get(sessionScopeId); - const backupOwner = durableStorage && currentAuthority ? sessionHarnessBackupOwners.get(currentAuthority) : null; + const currentAuthority = sessionToolAuthorityEpochs.get(sessionScopeId) ?? authorityEpoch; + const backupOwner = durableStorage + ? sessionHarnessBackupOwners.get(currentAuthority) ?? sessionHarnessBackupOwners.get(authorityEpoch) ?? null + : null; if (durableStorage && !backupOwner) throw new Error("native_harness_storage_owner_missing"); if (backupOwner) await assertNativeHarnessBackupOwner(input.db, backupOwner); // Transport release also runs when provider bootstrap failed. In that case @@ -18929,9 +18931,7 @@ async function createRunnerdBackendWithinSessionClaim( target, controllerPersistence: input.controllerPersistence ? nativeRunnerCheckpointPersistence( input.db, input.controllerPersistence.bootId, () => { - const current = sessionToolAuthorityEpochs.get(sessionScopeId); - if (!current) throw new Error("native_session_tool_authority_unavailable"); - return current.runId; + return (sessionToolAuthorityEpochs.get(sessionScopeId) ?? authorityEpoch).runId; }, detachNativeSessionsForRestart) : undefined, binding: input.remoteWarmRecovery ? { ...input.execution.binding, runId: input.remoteWarmRecovery.identity.runId } diff --git a/server/src/services/remote-execution-termination.ts b/server/src/services/remote-execution-termination.ts index 147982f4746..16fbba58a34 100644 --- a/server/src/services/remote-execution-termination.ts +++ b/server/src/services/remote-execution-termination.ts @@ -1,4 +1,4 @@ -import { and, eq } from "drizzle-orm"; +import { and, desc, eq } from "drizzle-orm"; import { environmentLeases, type Db } from "@paperclipai/db"; import { retiredReusableLeasePredecessors } from "./environment-lease-handoff.js"; @@ -129,3 +129,21 @@ export async function remoteExecutionHasReleased(db: Db, companyId: string, runI const leases = await runRemoteLeases(db, companyId, runId); return leases.length > 0 && leases.every(hasRemoteTerminationReceipt); } + +export async function terminatedRemoteCleanupScope(db: Db, companyId: string, remoteCleanupScope: string) { + let parsed: unknown; + try { + parsed = JSON.parse(remoteCleanupScope); + } catch { + return null; + } + if (!Array.isArray(parsed) || parsed.length !== 2 || typeof parsed[0] !== "string" || typeof parsed[1] !== "string" + || parsed[0] === "local") return null; + const [provider, providerLeaseId] = parsed as [string, string]; + const leases = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId), + eq(environmentLeases.provider, provider), eq(environmentLeases.providerLeaseId, providerLeaseId))) + .orderBy(desc(environmentLeases.createdAt), desc(environmentLeases.id)); + if (leases.length === 0 || leases.some(lease => !lease.releasedAt)) return null; + const newest = await resolveRemoteTerminationLease(db, leases[0]!); + return hasRemoteTerminationReceipt(newest) ? newest : null; +} diff --git a/server/src/vendor/paperclip-runner/index.ts b/server/src/vendor/paperclip-runner/index.ts index 5bf32f2c904..d7e0af910e6 100644 --- a/server/src/vendor/paperclip-runner/index.ts +++ b/server/src/vendor/paperclip-runner/index.ts @@ -213,6 +213,8 @@ export const nativeRestartContinuationInFlight = runner.nativeRestartContinuatio export const completeTerminatedRemoteNativeSessionCleanup = runner.completeTerminatedRemoteNativeSessionCleanup; export const completeTerminatedLocalNativeSessionCleanup = runner.completeTerminatedLocalNativeSessionCleanup; +export const completeTerminatedRemoteNativeSessionCleanupForTask = runner.completeTerminatedRemoteNativeSessionCleanupForTask; +export const nativeSessionCleanupRemoteScopes = runner.nativeSessionCleanupRemoteScopes; export const nativeSessionCleanupBlock = runner.nativeSessionCleanupBlock; export const onNativeSessionCleanupSettled = runner.onNativeSessionCleanupSettled; export const onNativeSessionCleanupQuarantined = runner.onNativeSessionCleanupQuarantined;