diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 238bf4b8b8a..daca73b1ba6 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1025,6 +1025,14 @@ In the normal non-terminal case, critical silence remains a UI signal and does n Source-resolved watchdog folding concerns stale active-run bookkeeping after a valid terminal disposition. It does not infer productivity from run counts, comment frequency, or elapsed task time. +Terminal task activity does not prove that the execution runtime has finished. +The scan leaves native runs with their native lifecycle owner. An active managed +environment lease, running or failed workspace finalization, or recorded +copy-back failure also prevents folding. These obligations are checked before +local process cleanup and again when the run is committed. The owning runtime +or its bounded recovery saves the files and settles the run; the watchdog does +not replace that work with a successful result or release its execution locks. + Detached process cleanup is operational hygiene, not source issue liveness. Cleanup should be best-effort and auditable. If cleanup fails but the source issue is already terminal with same-run durable evidence, Paperclip should preserve the cleanup failure on the run/watchdog audit trail and route only the cleanup concern to bounded recovery when a real owner/action remains. ## 13. Auto-Recover vs Explicit Recovery vs Human Escalation diff --git a/doc/workspace-restore-locks.md b/doc/workspace-restore-locks.md index d5e882f580f..7eea72328dc 100644 --- a/doc/workspace-restore-locks.md +++ b/doc/workspace-restore-locks.md @@ -90,10 +90,18 @@ Only a board operator can record this repair. It preserves the current task owner, conversation generation, status, and execution locks. It does not queue a continuation or replay the stopped run; after repair, any new work still follows the normal admission rules. -The board currently shows repair guidance and an Inspect run link, not a repair -form. Recording the decision is an operator API/runbook workflow; the recovery -API response contains the exact action and lease IDs. Provider access may require -an instance administrator or Cloud support. This change does not add a repair UI. +The original legacy run may be `succeeded` while required files still need +repair. Verified file-only repair accepts that terminal result and keeps it +unchanged. It still requires the retained-source record, exact remote stop +receipts, and recovered-file evidence. A successful result does not permit a +normal execution retry, and a queued or running source cannot be repaired this +way. Later saved messages can continue through ordinary admission after the +file hold clears. +The board shows repair guidance, an Inspect run link, and a Record file recovery +form for operators with runtime access. The form requires recovered-file evidence +and confirmation; the API verifies the original stop receipts. The operator API +also supports this workflow and returns the exact action and lease IDs. Provider +access may require an instance administrator or Cloud support. Then use the provider's operator console to remove that exact retained allocation when its files are no longer needed. Verify its immutable provider allocation ID against the recorded lease before removal. The ordinary reusable diff --git a/maps/agent-access-map/recovery.md b/maps/agent-access-map/recovery.md index 81b52a87279..2fa28a9abb3 100644 --- a/maps/agent-access-map/recovery.md +++ b/maps/agent-access-map/recovery.md @@ -141,12 +141,17 @@ API: `present` — representative operation IDs: - `POST /api/issues/{id}/recovery-actions/retry-workspace-export`. - `POST /api/issues/{id}/recovery-actions/resume-workspace-finalization`. +- `POST /api/issues/{id}/recovery-actions/resolve` for retained-source file repair. - `POST /api/execution-workspaces/{id}/reconcile-branch`. Agent access: `restricted` — Board/runtime management APIs; agent reads diagnosis and performs permitted workspace work. Policy: Export retry and branch reconciliation require board runtime management. +Retained legacy file-only repair requires the board, exact stopped-source receipts, +and recovered-file evidence. It can record repair for a successful legacy turn +without changing its result or authorizing replay. Queued and running sources +remain denied. Source-only reconciliation regressions cover this boundary. Physical-owner stop confirmation uses the separate `runtime:confirm_owner_stop` action. It requires active company owner/admin membership, trusted local board access, or instance administration. Agents and ordinary runtime managers are denied. diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index c21609715b9..038de76e049 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -137,6 +137,13 @@ cover evidence/confirmation, read-only access, stale or missing stop proof, source-only repair, and preserved task status. These tests use fixtures; separately verify original files and their attached artifact before recording a real repair. +Repeat source-only repair with a `succeeded` legacy run that still has a retained +file obligation. Its result must remain successful, its current task owner and +status must stay unchanged, and later saved input must become eligible for normal +admission. A running or queued source and a normal retry of a successful turn +remain denied. The [reconciliation regressions](../../server/src/services/workspace-restore-reconciliation.test.ts) +cover these cases and stop-receipt validation without claiming live provider proof. + For an abandoned native copyback owner, the harness verifies the exact stopped deployment and the reconciler restores saved work automatically. Task users see no process details or shutdown form. The documented operator API handles older diff --git a/server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts b/server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts index 197dc260ed4..42d8ad30150 100644 --- a/server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts +++ b/server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts @@ -7,6 +7,7 @@ import { agentWakeupRequests, companies, createDb, + environmentLeases, heartbeatRunEvents, heartbeatRunWatchdogDecisions, heartbeatRuns, @@ -14,6 +15,7 @@ import { issueRecoveryActions, issueRelations, issues, + workspaceOperations, } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, @@ -21,6 +23,7 @@ import { } from "./helpers/embedded-postgres.js"; import { truncateTablesWithDeadlockRetry } from "./helpers/truncate-with-deadlock-retry.js"; import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js"; +import { createPostgresWatchdogAdapter } from "../modules/active-run-watchdog/adapters/postgres.js"; import { ACTIVE_RUN_OUTPUT_CONTINUE_REARM_MS, ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS, @@ -446,6 +449,75 @@ describeEmbeddedPostgres("active-run output watchdog", () => { ))).toHaveLength(0); }); + it.each(["remote_lease", "workspace_finalize", "failed_finalize", "native_owner", "restore_failed"])( + "leaves a terminal task's run with its lifecycle owner while %s is unsettled", + async pending => { + const now = new Date("2026-04-22T20:00:00.000Z"); + const seeded = await seedRunningRun({ now, ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, + sourceStatus: "done", sameRunTerminalEvidence: true }); + if (pending === "remote_lease") { + await db.insert(environmentLeases).values({ companyId: seeded.companyId, heartbeatRunId: seeded.runId, + issueId: seeded.issueId, status: "active", provider: "daytona", providerLeaseId: "sandbox-with-unsaved-files" }); + } else if (pending === "workspace_finalize" || pending === "failed_finalize") { + await db.insert(workspaceOperations).values({ companyId: seeded.companyId, heartbeatRunId: seeded.runId, + issueId: seeded.issueId, phase: "workspace_finalize", status: pending === "failed_finalize" ? "failed" : "running" }); + } else if (pending === "native_owner") { + await db.update(heartbeatRuns).set({ runtimeMode: "native" }).where(eq(heartbeatRuns.id, seeded.runId)); + } else { + await db.update(heartbeatRuns).set({ resultJson: { workspaceRestoreFailure: "restore_failed" } }) + .where(eq(heartbeatRuns.id, seeded.runId)); + } + const { recovery, enqueueWakeup } = createRecovery(); + expect(await recovery.scanSilentActiveRuns({ now, companyId: seeded.companyId })) + .toMatchObject({ folded: 0, created: 0, skipped: 1 }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); + const [task] = await db.select().from(issues).where(eq(issues.id, seeded.issueId)); + const [agent] = await db.select().from(agents).where(eq(agents.id, seeded.coderId)); + expect(run.status).toBe("running"); + expect(task).toMatchObject({ status: "done", executionRunId: seeded.runId }); + expect(agent.status).toBe("running"); + expect(enqueueWakeup).not.toHaveBeenCalled(); + expect(await db.select().from(heartbeatRunWatchdogDecisions).where(eq(heartbeatRunWatchdogDecisions.runId, seeded.runId))) + .toHaveLength(0); + }, + ); + + it("folds stale bookkeeping after remote release and successful workspace finalization", async () => { + const now = new Date("2026-04-22T20:00:00.000Z"); + const seeded = await seedRunningRun({ now, ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, + sourceStatus: "done", sameRunTerminalEvidence: true }); + await db.insert(environmentLeases).values({ companyId: seeded.companyId, heartbeatRunId: seeded.runId, + issueId: seeded.issueId, status: "released", releasedAt: now, cleanupStatus: "success", + provider: "daytona", providerLeaseId: "finished-sandbox" }); + await db.insert(workspaceOperations).values({ companyId: seeded.companyId, heartbeatRunId: seeded.runId, + issueId: seeded.issueId, phase: "workspace_finalize", status: "succeeded", finishedAt: now }); + expect(await createRecovery().recovery.scanSilentActiveRuns({ now, companyId: seeded.companyId })) + .toMatchObject({ folded: 1, created: 0 }); + }); + + it("rechecks remote ownership when a lease appears after the initial fold check", async () => { + const now = new Date("2026-04-22T20:00:00.000Z"); + const seeded = await seedRunningRun({ now, ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, + sourceStatus: "done", sameRunTerminalEvidence: true }); + const adapter = createPostgresWatchdogAdapter(db); + const run = await adapter.findRunForCompany(seeded.companyId, seeded.runId); + expect(await adapter.isRuntimeSettlementPending(seeded.companyId, seeded.runId)).toBe(false); + await db.insert(environmentLeases).values({ companyId: seeded.companyId, heartbeatRunId: seeded.runId, + issueId: seeded.issueId, status: "active", provider: "daytona", providerLeaseId: "late-owned-sandbox" }); + expect(await adapter.foldSourceResolvedRun(seeded.companyId, { + run: run!, sourceIssue: { id: seeded.issueId, identifier: `${seeded.issuePrefix}-1`, status: "done", + originKind: "manual", isRecoveryOriginKind: false }, + evidence: { kind: "activity", id: randomUUID(), action: "issue.updated", createdAt: now }, + existingEvaluation: null, silenceStartedAt: seeded.startedAt, silenceAgeMs: 16 * 60 * 1000, + cleanup: { attempted: false, outcome: "no_process_metadata", adapterType: "grok_local" }, now, + })).toEqual({ kind: "stale" }); + const [persisted] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); + expect(persisted.status).toBe("running"); + expect(persisted.resultJson).toBeNull(); + expect(await db.select().from(heartbeatRunWatchdogDecisions).where(eq(heartbeatRunWatchdogDecisions.runId, seeded.runId))) + .toHaveLength(0); + }); + it("folds existing legacy evaluation and recovery rows idempotently", async () => { const now = new Date("2026-04-22T20:00:00.000Z"); const seeded = await seedRunningRun({ diff --git a/server/src/__tests__/workspace-restore-repair-route.test.ts b/server/src/__tests__/workspace-restore-repair-route.test.ts index db0b434e957..29e98239e43 100644 --- a/server/src/__tests__/workspace-restore-repair-route.test.ts +++ b/server/src/__tests__/workspace-restore-repair-route.test.ts @@ -117,8 +117,12 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu expect(await db.select().from(heartbeatRuns).where(inArray(heartbeatRuns.id, [f.sourceRunId, f.currentRunId]))).toHaveLength(2); } - it.each(["reassigned", "conversation_reset", "unassigned"])("records board repair after %s without restarting or changing the current task", async scenario => { + it.each(["reassigned", "conversation_reset", "unassigned", "successful_source"])("records board repair after %s without restarting or changing the current task", async scenario => { const f = await seed({ conversationReset: scenario === "conversation_reset", noCurrentOwner: scenario === "unassigned" }); + if (scenario === "successful_source") { + await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.sourceRunId)); + } + const [sourceBefore] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)); const { app, wake } = appFor(); expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull(); await request(app).post(`/api/issues/${f.issueId}/recovery-actions/resolve`).send(f.body).expect(200); @@ -137,6 +141,7 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu expect(await db.select().from(activityLog).where(eq(activityLog.companyId, f.companyId))).toEqual(auditBefore); await assertSourceUnchanged(f); expect(wake).not.toHaveBeenCalled(); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)))[0]).toEqual(sourceBefore); }); it.each(["blocked", "done", "cancelled", "backlog"])("records repair without changing the current %s task", async status => { diff --git a/server/src/modules/active-run-watchdog/adapters/postgres.ts b/server/src/modules/active-run-watchdog/adapters/postgres.ts index 356ef375c2f..c2e45643050 100644 --- a/server/src/modules/active-run-watchdog/adapters/postgres.ts +++ b/server/src/modules/active-run-watchdog/adapters/postgres.ts @@ -6,7 +6,9 @@ import { activityLog, heartbeatRunWatchdogDecisions, heartbeatRuns, + environmentLeases, issues, + workspaceOperations, } from "@paperclipai/db"; import { parseObject } from "../../../adapters/utils.js"; import { visibleIssueCondition } from "../../../services/issue-visibility.js"; @@ -65,6 +67,27 @@ function toRunSnapshot(row: typeof heartbeatRuns.$inferSelect): RunSnapshot { }; } +function runtimeSettlementPending(companyId: string, runId: string) { + return sql`( + ${heartbeatRuns.runtimeMode} = 'native' + or coalesce(${heartbeatRuns.resultJson}->>'workspaceRestoreFailure', '') <> '' + or exists ( + select 1 from ${environmentLeases} + where ${environmentLeases.companyId} = ${companyId} + and ${environmentLeases.heartbeatRunId} = ${runId} + and ${environmentLeases.providerLeaseId} is not null + and ${environmentLeases.status} in ('active', 'acquiring') + ) + or exists ( + select 1 from ${workspaceOperations} + where ${workspaceOperations.companyId} = ${companyId} + and ${workspaceOperations.heartbeatRunId} = ${runId} + and ${workspaceOperations.phase} = 'workspace_finalize' + and ${workspaceOperations.status} in ('running', 'failed') + ) + )`; +} + export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & WatchdogWriter { const issuesSvc = issueService(db); const recoveryActionsSvc = issueRecoveryActionService(db); @@ -119,6 +142,13 @@ export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & Watch return row ? toRunSnapshot(row) : null; } + async function isRuntimeSettlementPending(companyId: string, runId: string): Promise { + const [row] = await db.select({ + pending: sql`${heartbeatRuns.status} <> 'running' or ${runtimeSettlementPending(companyId, runId)}`, + }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId))); + return row?.pending ?? true; + } + async function findLatestDecision(companyId: string, runId: string, now: Date) { const [quietUntilRows, dismissedRows] = await Promise.all([ db @@ -321,6 +351,7 @@ export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & Watch eq(heartbeatRuns.id, input.run.id), eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.status, "running"), + sql`not ${runtimeSettlementPending(companyId, input.run.id)}`, )) .returning(); if (!updatedRun) return null; @@ -477,6 +508,7 @@ export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & Watch return { findCandidateSilentRuns, findRunForCompany, + isRuntimeSettlementPending, findLatestDecision, findOpenStaleRunEvaluation, findEvaluationIssueById, diff --git a/server/src/modules/active-run-watchdog/application/ports.ts b/server/src/modules/active-run-watchdog/application/ports.ts index d4f73b2a62b..594f0285a1c 100644 --- a/server/src/modules/active-run-watchdog/application/ports.ts +++ b/server/src/modules/active-run-watchdog/application/ports.ts @@ -53,6 +53,7 @@ export type RunningAgentSnapshot = { export interface WatchdogRunReader { findCandidateSilentRuns(input: FindCandidateSilentRunsInput): Promise; findRunForCompany(companyId: string, runId: string): Promise; + isRuntimeSettlementPending(companyId: string, runId: string): Promise; findLatestDecision(companyId: string, runId: string, now: Date): Promise; findOpenStaleRunEvaluation(companyId: string, runId: string): Promise; findLatestSameRunTerminalEvidence( diff --git a/server/src/modules/active-run-watchdog/application/use-cases.test.ts b/server/src/modules/active-run-watchdog/application/use-cases.test.ts index c8d1c1570fe..2b0f1f1e268 100644 --- a/server/src/modules/active-run-watchdog/application/use-cases.test.ts +++ b/server/src/modules/active-run-watchdog/application/use-cases.test.ts @@ -38,6 +38,7 @@ describe("application", () => { return { findCandidateSilentRuns: async () => [], findRunForCompany: async () => null, + isRuntimeSettlementPending: async () => false, findLatestDecision: async () => ({ quietUntilDecision: null, dismissedFalsePositive: false }), findOpenStaleRunEvaluation: async () => null, findLatestSameRunTerminalEvidence: async () => null, @@ -134,6 +135,27 @@ describe("application", () => { }; describe("createFoldSourceResolvedRun", () => { + it("does not terminate a process or settle the run while its runtime owns finalization", async () => { + const calls: string[] = []; + const fold = createFoldSourceResolvedRun({ + reader: makeReader({ isRuntimeSettlementPending: async (companyId, runId) => { + expect([companyId, runId]).toEqual(["company-1", "run-1"]); + return true; + } }), + processController: { cleanupRunProcess: async () => { + calls.push("cleanup"); + return { attempted: false, outcome: "no_process_metadata", adapterType: "grok_local" }; + } }, + writer: { recordDecision: async () => { throw new Error("unexpected decision"); }, + foldSourceResolvedRun: async () => { calls.push("fold"); return { kind: "folded", evaluationIssueId: null }; } }, + }); + expect(await fold({ run: makeRun({ processPid: 4242 }), runningAgentAdapterType: "grok_local", + sourceIssue: foldSourceIssue, evidence: foldEvidence, existingEvaluation: null, + silenceStartedAt: null, silenceAgeMs: null, now: new Date("2026-01-01T05:00:00Z") })) + .toEqual({ kind: "stale" }); + expect(calls).toEqual([]); + }); + it("cleans up the process before the folding persistence call", async () => { const calls: string[] = []; const processController: RunProcessController = { @@ -154,7 +176,7 @@ describe("application", () => { return { kind: "folded", evaluationIssueId: null }; }, }; - const foldSourceResolvedRun = createFoldSourceResolvedRun({ writer, processController }); + const foldSourceResolvedRun = createFoldSourceResolvedRun({ reader: makeReader(), writer, processController }); const outcome = await foldSourceResolvedRun({ run: makeRun({ processPid: 4242 }), @@ -187,7 +209,7 @@ describe("application", () => { }, foldSourceResolvedRun: async () => ({ kind: "stale" }), }; - const foldSourceResolvedRun = createFoldSourceResolvedRun({ writer, processController }); + const foldSourceResolvedRun = createFoldSourceResolvedRun({ reader: makeReader(), writer, processController }); const outcome = await foldSourceResolvedRun({ run: makeRun({ processPid: 4242 }), @@ -396,7 +418,7 @@ describe("application", () => { foldSourceResolvedRun: async () => ({ kind: "folded", evaluationIssueId: null }), ...writerOverrides, }; - return createFoldSourceResolvedRun({ writer, processController }); + return createFoldSourceResolvedRun({ reader: makeReader(), writer, processController }); } describe("createScanSilentActiveRuns", () => { diff --git a/server/src/modules/active-run-watchdog/application/use-cases.ts b/server/src/modules/active-run-watchdog/application/use-cases.ts index 478c10fefce..c8aa801ada3 100644 --- a/server/src/modules/active-run-watchdog/application/use-cases.ts +++ b/server/src/modules/active-run-watchdog/application/use-cases.ts @@ -72,6 +72,7 @@ export function createBuildRunOutputSilence(deps: BuildRunOutputSilenceDeps) { } export type FoldSourceResolvedRunDeps = { + reader: Pick; writer: WatchdogWriter; processController: RunProcessController; }; @@ -89,6 +90,9 @@ export type FoldSourceResolvedRunUseCaseInput = { export function createFoldSourceResolvedRun(deps: FoldSourceResolvedRunDeps) { return async function foldSourceResolvedRun(input: FoldSourceResolvedRunUseCaseInput): Promise { + if (await deps.reader.isRuntimeSettlementPending(input.run.companyId, input.run.id)) { + return { kind: "stale" }; + } const cleanup = await deps.processController.cleanupRunProcess({ runId: input.run.id, adapterType: input.runningAgentAdapterType, diff --git a/server/src/modules/active-run-watchdog/index.ts b/server/src/modules/active-run-watchdog/index.ts index 0de9687ee34..907420badf9 100644 --- a/server/src/modules/active-run-watchdog/index.ts +++ b/server/src/modules/active-run-watchdog/index.ts @@ -24,6 +24,7 @@ export function createActiveRunWatchdog(db: Db, config: ActiveRunWatchdogConfig) const processController = createProcessAdapter(); const foldSourceResolvedRun = createFoldSourceResolvedRun({ + reader: postgresAdapter, writer: postgresAdapter, processController, }); diff --git a/server/src/services/execution-recovery-resolution.ts b/server/src/services/execution-recovery-resolution.ts index 2bdf431e4a1..3edd84a3ad0 100644 --- a/server/src/services/execution-recovery-resolution.ts +++ b/server/src/services/execution-recovery-resolution.ts @@ -65,13 +65,15 @@ export async function validateExecutionReconciliation(input: { review.currentParticipant.agentId === run?.agentId; const retainedWorkspace = hasRequiredWorkspaceRecovery(run?.resultJson); const repairOnly = input.workspaceRepairOnly === true && retainedWorkspace; + const terminalRepairSource = ["failed", "interrupted", "timed_out", "cancelled"].includes(run?.status ?? "") || + (repairOnly && run?.runtimeMode === "legacy" && run.status === "succeeded"); if ( (input.workspaceRepairOnly && !retainedWorkspace) || !run || !task || (!repairOnly && (task.assigneeAgentId !== agentId || (run.agentId !== agentId && !isCurrentReviewer))) || (run.nativeIssueId ?? run.contextSnapshot?.issueId) !== issueId || - !["failed", "interrupted", "timed_out", "cancelled"].includes(run.status) + !terminalRepairSource ) { throw conflict( "The recovery source or task owner changed. Inspect the current execution before continuing.", diff --git a/server/src/services/workspace-restore-reconciliation.test.ts b/server/src/services/workspace-restore-reconciliation.test.ts index 873a06a3e9d..f205b3b6a14 100644 --- a/server/src/services/workspace-restore-reconciliation.test.ts +++ b/server/src/services/workspace-restore-reconciliation.test.ts @@ -80,6 +80,39 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu await expect(validateExecutionReconciliation({ db, ...f })).resolves.toMatchObject({ id: f.sourceRunId }); }); + it("repairs retained files from a succeeded run without reopening or replaying its completed work", async () => { + const f = await seed(); + await db.update(heartbeatRuns).set({ status: "succeeded", errorCode: null }).where(eq(heartbeatRuns.id, f.sourceRunId)); + await db.update(issues).set({ status: "todo" }).where(eq(issues.id, f.issueId)); + const repaired = await validateExecutionReconciliation({ db, ...f, workspaceRepairOnly: true }); + const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId)); + await markExecutionReconciliation(db, action, f.decision, "board", undefined, { workspaceRepairOnly: true }); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull(); + expect(repaired.status).toBe("succeeded"); + const [recorded] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId)); + expect(recorded.evidence.continuationDelivery).toBe("not_requested"); + const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId)); + expect(task).toMatchObject({ status: "todo", assigneeAgentId: f.agentId, executionRunId: null }); + }); + + it.each(["running", "queued", "succeeded"])("does not authorize a normal continuation of a %s retained run", async status => { + const f = await seed(); + await db.update(heartbeatRuns).set({ status }).where(eq(heartbeatRuns.id, f.sourceRunId)); + await expect(validateExecutionReconciliation({ db, ...f })).rejects.toThrow("source or task owner changed"); + }); + + it.each(["running", "queued"])("does not accept file repair while the source run is %s", async status => { + const f = await seed(); + await db.update(heartbeatRuns).set({ status }).where(eq(heartbeatRuns.id, f.sourceRunId)); + await expect(validateExecutionReconciliation({ db, ...f, workspaceRepairOnly: true })).rejects.toThrow("source or task owner changed"); + }); + + it("does not replace native finalization with legacy success repair", async () => { + const f = await seed(); + await db.update(heartbeatRuns).set({ status: "succeeded", runtimeMode: "native" }).where(eq(heartbeatRuns.id, f.sourceRunId)); + await expect(validateExecutionReconciliation({ db, ...f, workspaceRepairOnly: true })).rejects.toThrow("source or task owner changed"); + }); + it.each(["pending_stop", "wrong_receipt", "missing_source", "unrecorded_source", "empty_sources"])("rejects repair with %s even when releasedAt is populated", async scenario => { const f = await seed(); if (scenario === "pending_stop") {