From 1b7254206f6e7b41b6600bd0db33abd3e1ece1c9 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 8 Nov 2026 14:21:20 -0600 Subject: [PATCH] fix(recovery): replay a preparation a restart interrupted once proof shows no provider work started (#20072) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path - Restarts are product failures: a task interrupted by a restart must resume on its own, never wait for a user Retry. - A run interrupted during preparation, before any provider work, is safe to replay; the agent never touched the workspace. Paperclip already has the proof (`isInterruptedNativePreparation`) and the settle (`settleInterruptedNativeBootstrap`) that replay such a run in the uncharged interrupted-preparation lane. - But a graceful shutdown terminalizes that run inside the draining process, where the proof cannot hold yet: the owning controller is still that process, and its sandbox lease may still be cleaning up. So recovery records `preserve_without_replay_v1` with replay blocked. - When the proof becomes available (the next process boots, and the lease is released with a confirmed stop), nothing re-judges the hold. The only re-evaluation point runs after a pending-cleanup sweep, so the task waits for a manual Retry. - The coordinator set the rule: replay automatically with durable proof of no provider work; anything uncertain stays held. This PR adds the re-evaluation point and no new proof logic. ## Linked Issues or Issue Description Refs #20048 **What happened?** In Daytona warm wave w-20261108T142906-s9501, task t05: scheduled retry c99b3a2e was 2 s into preparation (legacy, stage `preparing`, no runtime mode, no events, no finalization) when the K2 graceful shutdown interrupted it. Terminalization in the draining process recorded `legacy_execution_requires_reconciliation` with `preserve_without_replay_v1` and replay blocked, while its lease was still `pending_cleanup`. The lease was released with a stop receipt 2 s later, and the next process started, but the hold was never re-judged. The harness clicked Retry at 14:44:27. **Steps to reproduce** 1. Start a run that is still in preparation (legacy, `preparing`, adapter `paperclip_runner`). 2. Gracefully shut down the server; terminalization records a preserve-without-replay hold. 3. Let its lease release with a confirmed stop, and restart the server. 4. The hold stays `blocked`; the task needs a manual Retry. **Expected behavior** Once the owning controller is proven gone and every lease is released with a confirmed stop, the ownership reconciler re-judges the hold with the existing proof. With no provider work and every cleanup confirmed, the hold is settled and the preparation is replayed with no attempt charged. Possible provider work, a live controller, or an unfinished cleanup keeps the hold. ## What Changed - New ownership kind `legacy_interrupted_preparation_hold`, in `server/src/services/interrupted-preparation-hold-settlement.ts` and registered in `recovery/ownership-registry.ts`. - **Listing:** legacy runs that are `interrupted` or `failed`, in stage `preparing` with no `runtimeModeResolvedAt`. Each must have a system-recorded `legacy-execution:` hold: resolved/blocked, `preserve_without_replay_v1`, replay blocked, no owning user, and none of `ownerGoneAttestation`, `explicitUserContinuation` or `executionReconciliation`. Every lease must be released with no failed cleanup, so a lease that settles later is listed on the next pass. - **Owner:** the run's recorded controller (`legacyControllerOwnerStamp`). The shared reconciler settles only after `proveOwnerGone` proves it gone. A missing stamp, or one from another host or PID space, is unprovable and stays held. There is no age fallback. - **Settle:** re-checks the existing durable proof (`isInterruptedNativePreparation`: no coordinator, result or execution events; leases released with stop receipts; workspace repair when needed). It then resolves the hold through `settleInterruptedNativeBootstrap` as `false_positive` with replay allowed and continuation pending, delivered through the existing workspace-recovery queue in the uncharged interrupted-preparation lane. Its row locks make the transition happen once; the settle reports settled only when it resolved holds. - `server/src/services/legacy-execution-recovery.ts`: `settleInterruptedNativeBootstrap` takes an optional `proof`, recorded in the `issue.recovery_action_resolved` activity details as `ownerProof`, and an `onHoldsResolved` callback. - `server/src/__tests__/interrupted-preparation-hold-settle.test.ts`, K2-during-preparation shape, run through the reconciliation pass: - settled with a dead controller in this PID space, with the audit entry naming `pid_dead`; - kept on a `turn.started` event, a stamp from another process space, a missing stamp, the current process, a lease still cleaning up, an operator owner-gone attestation, and a user continuation; - two concurrent settles produce one resolution. - `ownership-reconciliation-report.test.ts`: adds the kind to the list of settling kinds. **Previous-release state:** holds written by the previous release (`preserve_without_replay_v1`, replay `blocked`) are what this re-judges, through the same proof. A run without a recorded controller stamp stays held. No saved state changes shape. ## Verification - `cd server && npx vitest run src/__tests__/interrupted-preparation-hold-settle.test.ts`: 9/9 pass. Removing the kind registration makes the settle case fail. - The ownership, retained-orphan and recovery suites (19 files) pass, after adding the kind to the report's settling-kinds list. - Not yet covered: the Daytona warm wave rerun, on measure/remote-warm with this change. ## Risks - **Replaying work that already ran.** Settlement requires both the shared owner proof and the existing durable no-provider-work proof. Anything unprovable stays held. - **Overriding an operator.** Only holds the system recorded are listed; attested, continued or owned holds are never touched. - **Load.** The listing is a filtered query on the 30 s reconciliation pass, gated so a run with an unsettled lease is not polled. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- ...nterrupted-preparation-hold-settle.test.ts | 187 ++++++++++++++++++ .../ownership-reconciliation-report.test.ts | 2 +- server/src/services/heartbeat.ts | 7 + ...interrupted-preparation-hold-settlement.ts | 86 ++++++++ .../src/services/legacy-execution-recovery.ts | 10 +- .../services/recovery/ownership-registry.ts | 17 +- 6 files changed, 305 insertions(+), 4 deletions(-) create mode 100644 server/src/__tests__/interrupted-preparation-hold-settle.test.ts create mode 100644 server/src/services/interrupted-preparation-hold-settlement.ts diff --git a/server/src/__tests__/interrupted-preparation-hold-settle.test.ts b/server/src/__tests__/interrupted-preparation-hold-settle.test.ts new file mode 100644 index 00000000000..4f4373898db --- /dev/null +++ b/server/src/__tests__/interrupted-preparation-hold-settle.test.ts @@ -0,0 +1,187 @@ +import { spawn } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { and, eq } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; +import { + activityLog, + agents, + authUsers, + companyMemberships, + companies, + createDb, + environmentLeases, + environments, + heartbeatRunEvents, + heartbeatRuns, + issueRecoveryActions, + issues, +} from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { currentProcessSpaceHash } from "../services/controller-process-identity.js"; +import { heartbeatService } from "../services/heartbeat.js"; +import { settleHeldInterruptedPreparation } from "../services/interrupted-preparation-hold-settlement.js"; +import { legacyControllerBootId } from "../services/legacy-controller-lease.js"; +import { createOwnershipReportState, runOwnershipReconciliationPass } from "../services/recovery/ownership-invariant-report.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const suite = support.supported ? describe : describe.skip; +const KIND = "legacy_interrupted_preparation_hold"; + +async function exitedPid(): Promise { + const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { stdio: "ignore" }); + const pid = child.pid!; + await new Promise(resolve => { child.once("exit", () => resolve()); child.kill("SIGKILL"); }); + return pid; +} + +/** + * The K2-during-preparation shape from Daytona warm wave w-20261108T142906-s9501 + * (t05, run c99b3a2e): a scheduled retry was 2 s into preparation when a + * graceful shutdown interrupted it. The draining process recorded a + * preserve-without-replay hold, because its own controller was still live and + * the sandbox lease was still cleaning up. The lease was released with a + * confirmed stop moments later and the next process booted, but nothing + * re-judged the hold, so the task waited for a manual Retry. + */ +suite("held interrupted preparations on the shared ownership path", () => { + let db!: ReturnType; + let temporary: Awaited> | null = null; + let companyId = "", agentId = "", environmentId = ""; + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("paperclip-interrupted-preparation-"); + db = createDb(temporary.connectionString); + companyId = randomUUID(); agentId = randomUUID(); environmentId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Paperclip", issuePrefix: "IPH", + requireBoardApprovalForNewAgents: false }); + await db.insert(agents).values({ id: agentId, companyId, name: "Coder", role: "engineer", status: "idle", + adapterType: "paperclip_runner", adapterConfig: {}, runtimeConfig: {}, permissions: {} }); + await db.insert(environments).values({ id: environmentId, name: "Sandbox", driver: "sandbox", status: "active", + config: { provider: "fake" } }); + await db.insert(authUsers).values({ id: "local-board", name: "Local operator", email: "local-board@example.invalid", + createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing(); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "local-board", membershipRole: "owner" }); + }, 30_000); + + afterEach(async () => { + await db.delete(activityLog); + await db.delete(issueRecoveryActions); + await db.delete(environmentLeases); + await db.delete(heartbeatRunEvents); + await db.delete(heartbeatRuns); + await db.delete(issues); + }); + + afterAll(async () => { await temporary?.cleanup(); }); + + /** The draining controller, now exited, in this PID space. */ + async function deadController(overrides: Record = {}) { + const bootId = randomUUID(); + return { bootId, process: { bootId, pid: await exitedPid(), hostHash: currentProcessSpaceHash(), + processStartedAt: new Date(Date.now() - 60_000).toISOString(), ...overrides } as Record | null }; + } + + async function interruptedPreparation(options: { controller?: { bootId: string; process: Record | null }; + leaseReleased?: boolean; holdEvidence?: Record } = {}) { + const issueId = randomUUID(), runId = randomUUID(), leaseId = randomUUID(), providerLeaseId = `sandbox-${leaseId}`; + const finishedAt = new Date(Date.now() - 10_000); + const controller = options.controller ?? await deadController(); + // Terminalization projected the task to blocked and kept a receipt so the + // settle can restore exactly that projection. + await db.insert(issues).values({ id: issueId, companyId, title: "Warm follow-up", status: "blocked", + assigneeAgentId: agentId, statusVersion: 2 }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "automation", + responsibleUserId: "local-board", issueId, + status: "interrupted", errorCode: "server_shutdown_interrupted", runtimeMode: "legacy", executionStage: "preparing", + startedAt: new Date(finishedAt.getTime() - 2_000), finishedAt, nextEventSeq: 1, + controllerBootId: controller.bootId, controllerProcess: controller.process, + scheduledRetryAttempt: 1, scheduledRetryReason: "transient_failure", + contextSnapshot: { issueId }, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, + resultJson: { stopReason: "interrupted", cancellation: { reason: "Server shutdown interrupted execution", + source: "shutdown", expected: true, initiator: { type: "system" }, recordedAt: finishedAt.toISOString() } } }); + const released = options.leaseReleased !== false; + await db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId, issueId, heartbeatRunId: runId, + status: released ? "released" : "pending_cleanup", cleanupStatus: released ? "success" : "failed", + leasePolicy: "reuse_by_environment", provider: "fake", providerLeaseId, + releasedAt: released ? new Date(finishedAt.getTime() + 2_000) : null, + metadata: released ? { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId, runId, + leaseId, provider: "fake", providerLeaseId, state: "stopped", confirmedAt: finishedAt.toISOString() } } : {} }); + const [hold] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, + kind: "execution_reconciliation", cause: "legacy_execution_requires_reconciliation", + fingerprint: `legacy-execution:${runId}`, status: "resolved", outcome: "blocked", ownerType: "board", + nextAction: "Inspect the stopped provider and recorded actions.", resolvedAt: finishedAt, + evidence: { runId, automaticRecovery: { policy: "preserve_without_replay_v1", runId, replay: "blocked", + actionOutcome: "unknown", recordedAt: finishedAt.toISOString() }, + nativeBootstrapFailureBlock: { runId, previousStatus: "in_progress", statusVersion: 2 }, + ...options.holdEvidence } }).returning(); + return { issueId, runId, holdId: hold!.id }; + } + + const hold = async (id: string) => (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)))[0]!; + const pass = async () => (await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: true })) + .kinds.find(entry => entry.kind === KIND); + + it("replays a preparation a restart interrupted once its controller is proven gone", async () => { + // The heartbeat service registers the resume path that schedules the replay. + heartbeatService(db); + const f = await interruptedPreparation(); + expect(await pass()).toMatchObject({ inProgress: 1, settled: 1 }); + // The task is unblocked and an uncharged interrupted-preparation successor exists. + expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ status: "in_progress" }); + const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.runId)); + expect(successors).toHaveLength(1); + expect(successors[0]).toMatchObject({ scheduledRetryReason: "interrupted_preparation" }); + const settled = await hold(f.holdId); + expect(settled).toMatchObject({ status: "resolved", outcome: "false_positive" }); + expect(settled.evidence).toMatchObject({ automaticRecovery: { replay: "allowed" }, + bootstrapPreparationContinuation: "delivered", bootstrapPreparationSettledAt: expect.any(String) }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]!.resultJson) + .toMatchObject({ executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }); + const [audit] = await db.select().from(activityLog).where(and(eq(activityLog.runId, f.runId), + eq(activityLog.action, "issue.recovery_action_resolved"))); + expect(audit?.details).toMatchObject({ reason: "native_bootstrap_stopped_before_provider", ownerProof: "pid_dead" }); + }); + + it.each([ + ["provider work may have started", async () => { + const f = await interruptedPreparation(); + await db.insert(heartbeatRunEvents).values({ companyId, runId: f.runId, agentId, seq: 1, eventType: "turn.started", + stream: "system", level: "info", message: "turn started" }); + return f; + }], + ["its controller stamp is from another process space", async () => + interruptedPreparation({ controller: await deadController({ hostHash: "other-container" }) })], + ["it has no controller stamp", async () => + interruptedPreparation({ controller: { bootId: randomUUID(), process: null } })], + ["its controller is this process", async () => interruptedPreparation({ controller: { bootId: legacyControllerBootId, + process: { bootId: legacyControllerBootId, pid: process.pid, hostHash: currentProcessSpaceHash(), processStartedAt: null } } })], + ["its sandbox lease is still cleaning up", async () => interruptedPreparation({ leaseReleased: false })], + ["an operator confirmed the owner gone on the hold", async () => + interruptedPreparation({ holdEvidence: { ownerGoneAttestation: { actorId: "board-user" } } })], + ["the user already continued", async () => + interruptedPreparation({ holdEvidence: { explicitUserContinuation: { actorId: "board-user" } } })], + ["it is a workspace-restore hold owned by restore recovery", async () => + interruptedPreparation({ holdEvidence: { workspaceRestoreFailure: "restore_failed", + workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [] } } })], + ] as const)("keeps the hold when %s", async (_label, arrange) => { + const f = await arrange(); + const before = await hold(f.holdId); + expect((await pass())?.settled ?? 0).toBe(0); + expect(await hold(f.holdId)).toEqual(before); + }); + + it("settles exactly once when two processes judge the same hold", async () => { + heartbeatService(db); + const f = await interruptedPreparation(); + const record = { recordId: f.runId, companyId, runId: f.runId, issueId: f.issueId, owner: { kind: "none" as const }, + updatedAt: null }; + const context = { now: new Date(), proof: { verdict: "gone" as const, reason: "pid_dead" as const } }; + const outcomes = await Promise.all([settleHeldInterruptedPreparation(db, record, context), + settleHeldInterruptedPreparation(db, record, context)]); + expect(outcomes.filter(outcome => outcome.settled)).toHaveLength(1); + const resolutions = await db.select().from(activityLog).where(and(eq(activityLog.runId, f.runId), + eq(activityLog.action, "issue.recovery_action_resolved"))); + expect(resolutions).toHaveLength(1); + }); +}); diff --git a/server/src/__tests__/ownership-reconciliation-report.test.ts b/server/src/__tests__/ownership-reconciliation-report.test.ts index f50c4d9dd04..309f0ab683f 100644 --- a/server/src/__tests__/ownership-reconciliation-report.test.ts +++ b/server/src/__tests__/ownership-reconciliation-report.test.ts @@ -144,7 +144,7 @@ describe("report-only ownership reconciliation", () => { expect(new Set(names).size).toBe(names.length); expect(names.length).toBeGreaterThanOrEqual(40); expect(OWNABLE_KINDS.filter(kind => kind.settle !== null).map(kind => kind.id)) - .toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "environment_login_probe_lease", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "execution_workspace_runtime_lease", "legacy_shutdown_settlement_fence", "copyback_owner_hold", "observed_workspace_preservation_fence", "wake_request_claimed", "legacy_controller_lease", "issue_execution_lock", "issue_checkout_lock", "agent_lifecycle_lease"]); + .toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "environment_login_probe_lease", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "execution_workspace_runtime_lease", "legacy_shutdown_settlement_fence", "copyback_owner_hold", "observed_workspace_preservation_fence", "wake_request_claimed", "legacy_controller_lease", "legacy_interrupted_preparation_hold", "issue_execution_lock", "issue_checkout_lock", "agent_lifecycle_lease"]); expect(OWNABLE_KINDS.filter(kind => !kind.listInProgress).every(kind => kind.coverageGap)).toBe(true); }); diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 19901756c0e..2997874085a 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,4 @@ +import { registerHeldPreparationResumer } from "./interrupted-preparation-hold-settlement.js"; import { resolveProviderApiPlacement } from "./native-runtime/provider-api-placement.js"; import type { StopInvocationOptions } from "./heartbeat/run-control.js"; import { isContextCompactionRun } from "./context-compaction.js"; @@ -1530,6 +1531,12 @@ export function heartbeatService( function resumeRemoteStopComments(...args: Parameters) { return runControl.resumeRemoteStopComments(...args); } + // The interrupted-preparation ownership kind settles a hold, then replays the + // preparation through this same resume path (bounded, uncharged retry). + registerHeldPreparationResumer(async ({ companyId, runId }) => { + const run = await getRun(runId); + if (run && run.companyId === companyId) await resumeRemoteStopComments(run); + }); function cancelRunInternal(...args: Parameters) { return runControl.cancelRunInternal(...args); diff --git a/server/src/services/interrupted-preparation-hold-settlement.ts b/server/src/services/interrupted-preparation-hold-settlement.ts new file mode 100644 index 00000000000..99de8f7ccc6 --- /dev/null +++ b/server/src/services/interrupted-preparation-hold-settlement.ts @@ -0,0 +1,86 @@ +import { and, eq, inArray, isNull, sql } from "drizzle-orm"; +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { isInterruptedNativePreparation } from "./cancelled-native-startup.js"; +import { LEGACY_RECOVERY_CAUSE, settleInterruptedNativeBootstrap } from "./legacy-execution-recovery.js"; +import { logger } from "../middleware/logger.js"; +import type { OwnerProof } from "./owner-proof.js"; +import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js"; +import { OWNER_GONE_ATTESTATION_KEY } from "./recovery/owner-gone-attestation.js"; + +/** + * `legacy_interrupted_preparation_hold`: a run a graceful shutdown interrupted + * while it was still preparing, held with `preserve_without_replay_v1`. + * + * The draining process terminalizes the run before the no-provider-work proof + * can hold: its own controller is still live and its sandbox lease may still + * be cleaning up. Once the controller is proven gone and every lease is + * released with a confirmed stop, the existing proof decides, and a run with + * no provider execution is settled and replayed in its own uncharged lane. + * + * Only holds the system recorded itself are listed: an operator decision (an + * owner-gone attestation, an explicit user continuation, an assigned owner) is + * never reopened, and a workspace-restore hold (restore failure or recovery + * evidence) belongs to restore recovery, not to this kind. A run with a lease still cleaning up is not listed yet; the + * next pass lists it once the lease settles. + */ +export function heldInterruptedPreparationWhere() { + return and(inArray(heartbeatRuns.status, ["interrupted", "failed"]), eq(heartbeatRuns.runtimeMode, "legacy"), + isNull(heartbeatRuns.runtimeModeResolvedAt), eq(heartbeatRuns.executionStage, "preparing"), + sql`exists (select 1 from issue_recovery_actions hold where hold.company_id = ${heartbeatRuns.companyId} + and hold.cause = ${LEGACY_RECOVERY_CAUSE} and hold.fingerprint = 'legacy-execution:' || ${heartbeatRuns.id}::text + and hold.evidence->>'runId' = ${heartbeatRuns.id}::text + and hold.status = 'resolved' and hold.outcome = 'blocked' and hold.owner_user_id is null + and hold.evidence->'automaticRecovery'->>'policy' = 'preserve_without_replay_v1' + and hold.evidence->'automaticRecovery'->>'replay' = 'blocked' + and not (hold.evidence ?| array['executionReconciliation', ${OWNER_GONE_ATTESTATION_KEY}, 'explicitUserContinuation', + 'workspaceRestoreFailure', 'workspaceRestoreRecovery']))`, + sql`not exists (select 1 from environment_leases lease where lease.company_id = ${heartbeatRuns.companyId} + and lease.heartbeat_run_id = ${heartbeatRuns.id} + and (lease.released_at is null or lease.status = 'pending_cleanup' or lease.cleanup_status = 'failed'))`); +} + +/** Continues a settled preparation through the ordinary resume path + * (`resumeRemoteStopComments`): it schedules the uncharged + * `interrupted_preparation` successor and records the continuation. The + * heartbeat service registers it; it owns the scheduling dependencies. */ +type HeldPreparationResumer = (input: { companyId: string; runId: string }) => Promise; +let heldPreparationResumer: HeldPreparationResumer | null = null; + +export function registerHeldPreparationResumer(resumer: HeldPreparationResumer): void { + heldPreparationResumer = resumer; +} + +/** + * Settle of the kind. The shared reconciler calls it only after + * `proveOwnerGone` proved the run's recorded controller gone; a run without a + * controller stamp, or one from another process space, is never settled. It + * re-checks the existing durable proof (no coordinator, result or execution + * events; leases released with stop receipts; workspace repair when needed) + * and resolves the hold through `settleInterruptedNativeBootstrap`, whose row + * locks make the transition, and the continuation it queues, happen once. + */ +export async function settleHeldInterruptedPreparation(db: Db, record: InProgressRecord, + context: { now: Date; proof: OwnerProof }): Promise { + if (!record.companyId) return { settled: false, reason: "run_has_no_company" }; + const [run] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, record.recordId), + eq(heartbeatRuns.companyId, record.companyId), heldInterruptedPreparationWhere())); + if (!run) return { settled: false, reason: "no_longer_a_held_preparation" }; + if (!await isInterruptedNativePreparation(db, run)) { + return { settled: false, reason: "provider_work_not_disproved", watch: false }; + } + // Settling without the resume path would leave the task with no successor; + // keep the hold for a process that runs the heartbeat service. + if (!heldPreparationResumer) return { settled: false, reason: "resume_path_unavailable" }; + let resolved = 0; + await settleInterruptedNativeBootstrap(db, { run, providerDispatchStarted: false, proof: context.proof.reason, + onHoldsResolved: count => { resolved = count; } }); + if (resolved === 0) return { settled: false, reason: "hold_changed_since_proof" }; + // Only the call that resolved the hold starts the replay, so it starts once. + await heldPreparationResumer({ companyId: record.companyId, runId: run.id }).catch(error => { + // The hold is settled; its pending continuation stays recorded, and the + // run's next resume (a wake or a cleanup completion) delivers it, as on + // the run-control path. Logged so a stuck replay is visible. + logger.warn({ err: error, runId: run.id }, "settled interrupted preparation could not schedule its replay"); + }); + return { settled: true, reason: context.proof.reason }; +} diff --git a/server/src/services/legacy-execution-recovery.ts b/server/src/services/legacy-execution-recovery.ts index 978528a0246..b202cafa4ec 100644 --- a/server/src/services/legacy-execution-recovery.ts +++ b/server/src/services/legacy-execution-recovery.ts @@ -328,7 +328,11 @@ export async function terminalizeLegacyExecution(input: { } export async function settleInterruptedNativeBootstrap( - db: Db, input: { run: Run; providerDispatchStarted: boolean }, + db: Db, input: { run: Run; providerDispatchStarted: boolean; + /** Owner proof that authorized this settle, recorded in the activity log. */ + proof?: string; + /** Called inside the transaction with the number of holds this call resolved. */ + onHoldsResolved?: (count: number) => void }, ): Promise { if (input.providerDispatchStarted) return null; const issueId = input.run.contextSnapshot?.issueId; @@ -386,8 +390,10 @@ export async function settleInterruptedNativeBootstrap( }).where(inArray(issueRecoveryActions.id, holds.map(hold => hold.id))).returning({ id: issueRecoveryActions.id }) : []; if (resolved.length) await logActivity(tx as unknown as Db, { companyId: run.companyId, actorType: "system", actorId: "system", action: "issue.recovery_action_resolved", entityType: "issue", entityId: task.id, agentId: run.agentId, runId: run.id, - details: { reason: "native_bootstrap_stopped_before_provider", recoveryActionIds: resolved.map(r => r.id) }, + details: { reason: "native_bootstrap_stopped_before_provider", recoveryActionIds: resolved.map(r => r.id), + ...(input.proof ? { ownerProof: input.proof } : {}) }, }); + input.onHoldsResolved?.(resolved.length); return settled; }); } diff --git a/server/src/services/recovery/ownership-registry.ts b/server/src/services/recovery/ownership-registry.ts index bbaca5f09ca..1a09345b7ea 100644 --- a/server/src/services/recovery/ownership-registry.ts +++ b/server/src/services/recovery/ownership-registry.ts @@ -46,6 +46,7 @@ import { nativeControllerOwnerStamp, runProcessOwnerStamp } from "../native-runt import { settleIssueLock } from "./issue-lock-settlement.js"; import { settleAgentLifecycleLease } from "./agent-lifecycle-lease-settlement.js"; import { legacyControllerOwnerStamp, settleLegacyControllerLease } from "../legacy-controller-lease.js"; +import { heldInterruptedPreparationWhere, settleHeldInterruptedPreparation } from "../interrupted-preparation-hold-settlement.js"; import { nativeLeaseHolderStamp, readNativeLeaseHolder, @@ -1026,8 +1027,22 @@ export const OWNABLE_KINDS: readonly OwnableKind[] = [ return records; }, settle: settleLegacyControllerLease, - }, versionLock(heartbeatRuns)), + { + // A preparation a shutdown interrupted, held before its no-provider-work + // proof could exist. Its owner is the recorded controller process: the + // shared prover must prove it gone (a missing stamp, or one from another + // host or PID space, is unprovable and stays held). The settle re-checks + // the durable proof and replays the preparation exactly once. + // Keep this entry after the environment teardown kinds: a lease they settle + // in a pass is released before this kind lists in that same pass. + ...runKind("legacy_interrupted_preparation_hold", "Interrupted preparation held for replay", "ownership", + heldInterruptedPreparationWhere(), row => { + const stamp = legacyControllerOwnerStamp(row); + return stamp ? { kind: "process", stamp } : { kind: "none" }; + }), + settle: settleHeldInterruptedPreparation, + }, runKind("detached_process_marker", "Detached-process marker", "ownership", and(eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.errorCode, "process_detached")), row => row.processPid ? { kind: "process", stamp: runProcessStamp(row, "detached_process_marker") } : { kind: "none" },