From 795c3c4c1d509ad6dbd2591a797a31e91b265ea5 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 17:13:52 -0600 Subject: [PATCH] fix(recovery): continue saved input after historical retry pause (#19576) ## Thinking Path > - Paperclip manages work performed by AI agents. > - Its scheduler parks work when another run holds a shared workspace. > - A pause can refuse the scheduled retry before provider dispatch. > - Older schedulers recorded that refusal in the run log without a JSON marker. > - After the pause was released, a saved user message could remain held for execution recovery. > - This change verifies the original refusal and parent workspace-wait proof before admitting one continuation. > - Existing execution, Stop, budget, approval, and workspace guards remain in force. ## Linked Issues or Issue Description Refs #19466, #19552, #19580, #19602. **What happened?** A saved user message remained deferred after a pause was released. Its source was a never-claimed workspace retry. The historical source had an `issue_paused` error and no JSON cancellation marker. The continuation proof accepted only a different system-cancellation shape. **Expected behavior** The normal saved-message worker admits the existing input once when trusted scheduler evidence proves no provider dispatch occurred and current admission gates permit work. **Steps to reproduce** 1. Record a host-certified pre-provider workspace wait and its scheduled retry. 2. Let the real scheduler refuse promotion under an active task pause, including after the user presses Retry Now. 3. Keep the historical run-log-only cancellation shape, then release the pause. 4. Run concurrent normal saved-message recovery scans. **Paperclip version or commit** The regression fails on master `f0006c7783`. It leaves zero queued successors. The branch now normally integrates the fixture repairs in #19602 and #19628. ## What Changed - Accept the historical cancellation shape only when a trusted system lifecycle warning records the exact scheduler pause refusal. A prior Retry Now event does not invalidate that proof. - Bind the event to a released same-company pause and its task membership. - Reuse the existing parent workspace-wait certificate, no-provider-output checks, and retained-execution rejection. - Preserve historical cancellation records and consume the existing saved input through normal admission. - Document the contract and add real PostgreSQL coverage using actual scheduler suppression and pause release. ## Verification - RED: the real scheduler suppressed the retry under a pause. After release, two normal recovery scans left zero successors. - GREEN: the same path creates one successor and coalesces the original saved request. - All 389 tests in the complete native continuation suite pass before the final base update. The only changed continuation source from master is the separately merged one-line repair in #19552. - Fourteen focused cases passed at `2aaeb50f78ba0a22809ab96a00076ce50fcea559`, including a parent pause covering a child task. Thirteen cases also passed before the base update. They cover the normal path and missing event, wrong task, missing membership, active or mistimed pause, provider activity, retained lease, operator Stop, and invalid ancestry. - The earlier seven-layer server typecheck passed after building this worktree's Runner and SDK dependencies. The current review fix passes the exact CI test-layer typecheck. Fresh required CI will supply current-head source coverage. - The first sandboxed database run could not bind loopback. A later fresh dependency install omitted native PostgreSQL library symlinks and failed before executing tests. The package's own postinstall hydration repaired that setup. Previous-head verification passed all 14 focused cases at `2aaeb50f78ba0a22809ab96a00076ce50fcea559`. - Module-boundary and whitespace checks pass. - Commands: `cd server && pnpm exec vitest run src/services/explicit-native-continuation.test.ts`; `pnpm --filter @paperclipai/server typecheck`. - Environment: macOS arm64, Node 26.4.0, pnpm 9.15.4, real local PostgreSQL. - Previous head `2aaeb50f78ba0a22809ab96a00076ce50fcea559` had Greptile 5/5 and 56 passing checks, with three skipped. A later advisory review found the Retry Now event-order case. Those checks do not qualify the new head. Hosted recovery remains unverified. - Review RED: the real `retryScheduledRetryNow` producer wrote lifecycle/info at sequence 1 and the exact pause warning at sequence 2. After real hold release and two concurrent normal queued-message scans, both task-rooted and parent-rooted cases produced zero successors instead of one. Both failed before the production change (17.24s). - Review GREEN: all 17 focused cases pass at `f38e9934b8ce3f7ddfb23eaa0f979c3f44e7a990` (4.74s). The previous 14 cases remain covered. Two new cases use the real Retry Now producer, and one new case rejects contradictory provider execution evidence. No pause, membership, timing, ancestry, output, retained-execution, or Stop check was removed. - Current focused command: `pnpm exec vitest run server/src/services/explicit-native-continuation.test.ts -t 'historical pause|Retry Now pause history' --fsModuleCache`. Current test-layer typecheck: `cd server && node ../node_modules/typescript/bin/tsc --noEmit --project node_modules/.cache/paperclip-typecheck/tsconfig.tests.json --pretty false`. Both pass. - Final-head verification: all 17 focused cases passed again at `c88687d69f2c87fe1b81b5fffd670e3cf26a884d` (8.00s). The prior exact CI test-layer typecheck qualifies the unchanged typed logic; fresh CI supplies broader final-head coverage. - The follow-up commit removes only the new code comments to follow CONTRIBUTING.md. Recovery and test behavior are unchanged. That revision had Greptile 5/5 with its actionable review thread resolved. It is retained as historical verification; current-head qualification is below. - The first final-head CI attempt failed in the unchanged CLI company import/export fixture before assertions: its child server did not reach the runtime-info health check within 30 seconds. Other suites passed. The exact fixture passes locally with its original timeout. The single failed-shard retry passed at the unchanged head, together with its dependent CI checks. [Original failed job](https://github.com/paperclipai/paperclip/actions/runs/42279773036/job/128106195975). No assertion or timeout was changed. - The independent required Runner support failure reproduced five `IMAGE_ONLY: unbound variable` errors. #19602 repaired its fixture inputs and merged as `4d554f08218b04df49ec0f2a0d5c5dfdfaf65660`. This branch normally merges that master with no conflicts or force push. The current host-preparation log verifier from #19580 remains intact. - Final integrated head `ffc9e8d9516343a9823b4b80b5fcfccc548844ba`: 17/17 focused recovery cases pass (12.71s), all nine workflow-target cases pass through the official Runner Vitest config (1.10s), and all seven current server typecheck layers pass. Worktree preflight rebuilt current Runner dependencies and reused the Cargo seed; it completed in 61.3s with Rust debug binaries skipped. - Prior integrated head `ffc9e8d9516343a9823b4b80b5fcfccc548844ba` finished with 54 successful checks. Its only code-check failure was a global unhandled readiness rejection after 2,285 assertions passed in the server8 shard, plus dependent verify. The real-child race is fixed separately in #19624; it was not hidden by a rerun here. - Final normal integration head `550d1732b6960d6b349d03903f448fa1d71c17f6`: exact 17 paused-retry lifecycle regressions pass (71.06s), all nine official workflow target cases pass (2.43s), all seven server typecheck layers pass, and module/whitespace checks pass. Required preflight completed in 45.641s. Current preparation-log guards from #19580 remain intact. The diff against master contains only the original three owner/test/doc files. - Prior head 550d had two inherited model-default assertion failures plus a retained-cleanup backlog 15s timeout. The exact unchanged cleanup case passes locally (41.91s total including DB/hooks); no timer or assertion was weakened and no old CI job was rerun. The model fixture repair was independently merged in #19610 at 8621bcf53ee2e4bbefb7d193add42dd544ca06e8. - Current normal integration head `14103750d5f26d7f476f937e981e3e45c94054ab`: 17/17 exact lifecycle cases pass (75.33s), all 6 model normalization cases pass (6.58s), all seven server typecheck layers and source checks pass. Required preflight completed in 13.89s. No source guards or timeouts changed during this integration. - Current-head CI and Greptile are green. Earlier checks do not qualify this final integration head. - Current-head Greptile: 5/5 at `14103750d5f26d7f476f937e981e3e45c94054ab`, with all review threads resolved. Required CI passed: 56 successful checks, three intentional skips, one neutral review status. The separate optional Paperclip review job remains queued; no open findings. ## Risks This adds a bounded indexed evidence query for a narrow historical cancellation shape. Missing or contradictory evidence keeps work held. It does not replay a provider turn or change current pause, budget, approval, Stop, or workspace authority. ## Model Used OpenAI Codex, GPT-6 family, with repository analysis, code editing, and tool execution. The exact serving snapshot and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 7 ++ .../src/services/cancelled-native-startup.ts | 34 ++++-- .../explicit-native-continuation.test.ts | 105 ++++++++++++++++++ 3 files changed, 139 insertions(+), 7 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 672d8b479ee..36c2cc39842 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -2039,6 +2039,13 @@ task with a run refused for that hold. That covers a task created under the tree after the snapshot, a task moved out before release, and a task whose work an overlapping pause refused and released while this one still covered it. The scheduled-retry gate holds the pause row while it saves its refusal, so a concurrent release either sees the refusal or the retry sees no pause. +For an old workspace-wait retry without the JSON marker, a later saved user +message can continue only when an exact system lifecycle warning records that +pause refusal. An earlier Retry Now lifecycle record does not invalidate the +warning. The released pause and its task membership must match. +The parent must retain its host-issued pre-provider workspace-wait proof. +Provider activity, retained execution, an unmatched pause, or an operator Stop +keeps the old run held. The historical cancellation remains unchanged. The queued-run gate checks the hold again after saving its refusal and resumes the task itself if release already finished. A resume restores the assignee's execution only; like a wake that admission refuses during a hold, a non-assignee diff --git a/server/src/services/cancelled-native-startup.ts b/server/src/services/cancelled-native-startup.ts index 1a53d3971cf..1926dafef86 100644 --- a/server/src/services/cancelled-native-startup.ts +++ b/server/src/services/cancelled-native-startup.ts @@ -1,8 +1,8 @@ import { isRetainedWorkspaceWorkerStopProof } from "./native-runtime/native-workspace-platform-proof.js"; import { isPreDispatchReviewWait } from "./pre-dispatch-review-wait.js"; import { createHash } from "node:crypto"; -import { and, eq, inArray, isNotNull, isNull, lte, or, sql } from "drizzle-orm"; -import { environmentLeases, heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; +import { and, eq, gt, inArray, isNotNull, isNull, lte, or, sql } from "drizzle-orm"; +import { environmentLeases, heartbeatRunEvents, heartbeatRuns, issueTreeHolds, issueTreeHoldMembers, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; import { isUuidLike } from "@paperclipai/shared"; import { claimedAdapterType } from "./conversation-continuation.js"; import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-local-process-stop.js"; @@ -182,17 +182,37 @@ async function isCancelledUnclaimedWorkspaceWaitRetry(db: Db, run: Run): Promise const cancellation = readRunCancellation(run.resultJson); const issueId = run.issueId ?? run.contextSnapshot?.issueId; if (run.runtimeMode !== "legacy" || run.startedAt !== null || run.executionStage !== null - || run.runnerProfileJson !== null || typeof issueId !== "string" || !isUuidLike(issueId) - || run.errorCode !== "cancelled" || run.resultJson?.stopReason !== "cancelled" - || cancellation?.source !== "control_plane" || cancellation.expected !== true || cancellation.initiator.type !== "system") return false; + || run.runnerProfileJson !== null || typeof issueId !== "string" || !isUuidLike(issueId)) return false; + const systemCancellation = run.errorCode === "cancelled" && run.resultJson?.stopReason === "cancelled" + && cancellation?.source === "control_plane" && cancellation.expected === true && cancellation.initiator.type === "system"; + const pausedBeforePromotion = run.errorCode === "issue_paused" && run.resultJson === null && run.finishedAt + ? await db.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents) + .innerJoin(issueTreeHolds, and(eq(issueTreeHolds.companyId, run.companyId), + sql`${issueTreeHolds.id}::text = ${heartbeatRunEvents.payload}->>'holdId'`, + sql`${issueTreeHolds.rootIssueId}::text = ${heartbeatRunEvents.payload}->>'rootIssueId'`)) + .innerJoin(issueTreeHoldMembers, and(eq(issueTreeHoldMembers.companyId, run.companyId), + eq(issueTreeHoldMembers.holdId, issueTreeHolds.id), eq(issueTreeHoldMembers.issueId, issueId), + eq(issueTreeHoldMembers.skipped, false))) + .where(and(eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id), + eq(heartbeatRunEvents.eventType, "lifecycle"), + eq(heartbeatRunEvents.stream, "system"), eq(heartbeatRunEvents.level, "warn"), + eq(heartbeatRunEvents.message, "Scheduled retry suppressed because the issue is held by an active subtree pause hold"), + sql`${heartbeatRunEvents.payload}->>'issueId' = ${issueId}`, + sql`${heartbeatRunEvents.payload}->>'scheduledRetryReason' = 'workspace_busy'`, + eq(issueTreeHolds.mode, "pause"), eq(issueTreeHolds.status, "released"), + lte(issueTreeHolds.createdAt, run.finishedAt), isNotNull(issueTreeHolds.releasedAt), + gt(issueTreeHolds.releasedAt, run.finishedAt))).limit(1).then(rows => rows.length > 0) + : false; + if (!systemCancellation && !pausedBeforePromotion) return false; const [source] = await db.select({ agentId: heartbeatRuns.agentId, createdAt: heartbeatRuns.createdAt, retryOfRunId: heartbeatRuns.retryOfRunId, }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.scheduledRetryReason, "workspace_busy"), sql`${heartbeatRuns.contextSnapshot}->>'retryReason' = 'workspace_busy'`, sql`${heartbeatRuns.contextSnapshot}->>'retryOfRunId' = ${heartbeatRuns.retryOfRunId}::text`, - sql`(${heartbeatRuns.resultJson} - array['stopReason', 'cancellation', 'startupCancellation', - 'timeoutFired', 'timeoutSource', 'timeoutConfigured', 'effectiveTimeoutSec']) = '{}'::jsonb`, + pausedBeforePromotion ? isNull(heartbeatRuns.resultJson) + : sql`(${heartbeatRuns.resultJson} - array['stopReason', 'cancellation', 'startupCancellation', + 'timeoutFired', 'timeoutSource', 'timeoutConfigured', 'effectiveTimeoutSec']) = '{}'::jsonb`, noRetainedExecutionPredicate(), noProviderOutputPredicate(), isNull(heartbeatRuns.runnerProfileJson), isNull(heartbeatRuns.usageJson))); if (!source?.retryOfRunId) return false; diff --git a/server/src/services/explicit-native-continuation.test.ts b/server/src/services/explicit-native-continuation.test.ts index 756410efc6a..9da1c2a25de 100644 --- a/server/src/services/explicit-native-continuation.test.ts +++ b/server/src/services/explicit-native-continuation.test.ts @@ -4,6 +4,8 @@ import os from "node:os"; import { createSystemRecovery, failSystemRecovery, SystemRecoveryFailure } from "./system-recovery-state.js"; import { instanceSettingsService } from "./instance-settings.js"; import { createPostgresWakeQueueAdapter } from "../modules/wake-queue/adapters/postgres.js"; +import { createPostgresRunDispatchAdapter } from "../modules/run-dispatch/adapters/postgres.js"; +import { issueTreeControlService } from "./issue-tree-control.js"; import { createReleaseIssueExecution } from "../modules/wake-queue/application/use-cases.js"; import * as nativeExecutor from "./native-runtime/native-session-executor.js"; import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js"; @@ -1300,6 +1302,109 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes eq(heartbeatRuns.status, "queued")))).toHaveLength(1); }); + async function seedHistoricalPausedWorkspaceRetry(parentHold = false, retryNow = false) { + const f = await seedCancelledWorkspaceWaitRetry(); + const dueAt = new Date("2026-09-11T10:00:00Z"); + const pauseRootIssueId = parentHold ? randomUUID() : f.issueId; + if (parentHold) { + await db.insert(issues).values({ id: pauseRootIssueId, companyId: f.companyId, + title: "Shared workspace parent", status: "in_progress", assigneeAgentId: f.agentId }); + await db.update(issues).set({ parentId: pauseRootIssueId }).where(eq(issues.id, f.issueId)); + } + const [pause] = await db.insert(issueTreeHolds).values({ companyId: f.companyId, + rootIssueId: pauseRootIssueId, mode: "pause", status: "active", createdAt: new Date(dueAt.getTime() - 1_000), + releasePolicy: { strategy: "manual" } }).returning(); + await db.insert(issueTreeHoldMembers).values({ companyId: f.companyId, holdId: pause.id, + issueId: f.issueId, parentIssueId: parentHold ? pauseRootIssueId : null, depth: parentHold ? 1 : 0, + issueTitle: "Paused workspace retry", issueStatus: "blocked", assigneeAgentId: f.agentId }); + await db.update(heartbeatRuns).set({ status: "scheduled_retry", errorCode: null, + finishedAt: null, resultJson: null, scheduledRetryAt: dueAt, scheduledRetryAttempt: 1, + }).where(eq(heartbeatRuns.id, f.sourceRunId)); + if (retryNow) { + expect(await heartbeatService(db).retryScheduledRetryNow({ issueId: f.issueId, + actor: { actorType: "user", actorId: "board" }, now: dueAt })).toMatchObject({ outcome: "gate_suppressed" }); + const events = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.sourceRunId)) + .orderBy(heartbeatRunEvents.seq); + expect(events).toMatchObject([ + { seq: 1, eventType: "lifecycle", stream: "system", level: "info", message: "Scheduled retry was requested to run now" }, + { seq: 2, eventType: "lifecycle", stream: "system", level: "warn", + message: "Scheduled retry suppressed because the issue is held by an active subtree pause hold" }, + ]); + } else { + expect(await createPostgresRunDispatchAdapter(db).promoteOrCancelDueRetry({ + companyId: f.companyId, runId: f.sourceRunId, now: dueAt, + })).toMatchObject({ outcome: "gate_suppressed", errorCode: "issue_paused" }); + } + await db.update(heartbeatRuns).set({ resultJson: null }).where(eq(heartbeatRuns.id, f.sourceRunId)); + await issueTreeControlService(db).releaseHold(f.companyId, pauseRootIssueId, pause.id, + { actor: { actorType: "user", actorId: "board" } }); + return { ...f, pauseId: pause.id, dueAt }; + } + + it.each([ + { parentHold: false, retryNow: false }, { parentHold: true, retryNow: false }, + { parentHold: false, retryNow: true }, { parentHold: true, retryNow: true }, + ])("resumes saved input after a historical pause suppressed an unclaimed workspace retry (parent hold: $parentHold, retry now: $retryNow)", async ({ parentHold, retryNow }) => { + const f = await seedHistoricalPausedWorkspaceRetry(parentHold, retryNow); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); + const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, + source: "on_demand", reason: "issue_commented", requestedByActorType: "user", requestedByActorId: "board", + status: "deferred_issue_execution", updatedAt: new Date(0), payload: { issueId: f.issueId, commentId: f.commentId, + executionWait: { reason: "execution_recovery" }, + _paperclipWakeContext: { issueId: f.issueId, wakeCommentId: f.commentId, wakeCommentIds: [f.commentId] } }, + }).returning(); + const heartbeat = heartbeatService(db); + await Promise.all([heartbeat.resumeExecutionWaitComments(), heartbeat.resumeExecutionWaitComments()]); + const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), + eq(heartbeatRuns.status, "queued"))); + expect(successors).toHaveLength(1); + expect(successors[0].contextSnapshot).toMatchObject({ forceFreshSession: true, wakeCommentIds: [f.commentId] }); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)))[0]) + .toMatchObject({ status: "coalesced", runId: successors[0].id }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)))[0]) + .toMatchObject({ status: "cancelled", startedAt: null, errorCode: "issue_paused", resultJson: null }); + }); + + it("holds a Retry Now pause history that also contains provider execution evidence", async () => { + const f = await seedHistoricalPausedWorkspaceRetry(false, true); + await appendHeartbeatRunEvent(db, { companyId: f.companyId, agentId: f.agentId, runId: f.sourceRunId, + eventType: "turn.started", stream: "system", level: "info" }); + expect(await admit(f)).toBeNull(); + expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), + eq(heartbeatRuns.status, "queued")))).toHaveLength(0); + }); + + it.each(["missing_event", "wrong_event", "missing_member", "active_pause", "late_pause", "early_release", + "started", "provider_event", "lease", "operator_stop", "parent_executed", "retry_binding"])( + "holds a historical pause-suppressed workspace retry with %s evidence", async kind => { + const f = await seedHistoricalPausedWorkspaceRetry(); + if (kind === "missing_event") await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.sourceRunId)); + if (kind === "wrong_event") await db.update(heartbeatRunEvents).set({ payload: { holdId: f.pauseId, + rootIssueId: f.issueId, issueId: randomUUID(), scheduledRetryReason: "workspace_busy" } }) + .where(eq(heartbeatRunEvents.runId, f.sourceRunId)); + if (kind === "missing_member") await db.delete(issueTreeHoldMembers).where(eq(issueTreeHoldMembers.holdId, f.pauseId)); + if (kind === "active_pause") await db.update(issueTreeHolds).set({ status: "active", releasedAt: null }) + .where(eq(issueTreeHolds.id, f.pauseId)); + if (kind === "late_pause") await db.update(issueTreeHolds).set({ createdAt: new Date(f.dueAt.getTime() + 1) }) + .where(eq(issueTreeHolds.id, f.pauseId)); + if (kind === "early_release") await db.update(issueTreeHolds).set({ releasedAt: new Date(f.dueAt.getTime() - 1) }) + .where(eq(issueTreeHolds.id, f.pauseId)); + if (kind === "started") await db.update(heartbeatRuns).set({ startedAt: f.dueAt }).where(eq(heartbeatRuns.id, f.sourceRunId)); + if (kind === "provider_event") await appendHeartbeatRunEvent(db, { companyId: f.companyId, + agentId: f.agentId, runId: f.sourceRunId, eventType: "turn.started", stream: "system", level: "info" }); + if (kind === "lease") await db.insert(environmentLeases).values({ companyId: f.companyId, + heartbeatRunId: f.sourceRunId, provider: "local", leasePolicy: "ephemeral", status: "released", + cleanupStatus: "success", releasedAt: new Date() }); + if (kind === "operator_stop") await db.update(heartbeatRuns).set({ errorCode: "operator_interrupted", + resultJson: { stopReason: "cancelled", cancellation: { source: "operator", expected: true, + initiator: { type: "user", userId: "board" }, reason: "Stopped", recordedAt: f.dueAt.toISOString() } } }) + .where(eq(heartbeatRuns.id, f.sourceRunId)); + if (kind === "parent_executed") await db.update(heartbeatRuns).set({ resultJson: { + executionRecovery: { kind: "workspace_wait", providerWorkStarted: true } } }).where(eq(heartbeatRuns.id, f.parentRunId)); + if (kind === "retry_binding") await db.update(heartbeatRuns).set({ retryOfRunId: null }).where(eq(heartbeatRuns.id, f.sourceRunId)); + expect(await admit(f)).toBeNull(); + }); + it.each(["message", "retry"])("admits an unclaimed workspace retry through the heartbeat %s path", async kind => { const f = await seedCancelledWorkspaceWaitRetry(); await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });