From 6293e581f6c28db64eab32f0c85d8b6e27b3936e Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Thu, 5 Nov 2026 09:13:46 -0800 Subject: [PATCH] fix(recovery): resume cancelled workspace retries from verified waits (#19276) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A shared workspace can make a run wait before it calls a provider. > - The scheduler saves a retry for that wait. > - The control plane can cancel the retry while it is still queued. > - Recovery then asks for a process stop receipt for a run that never claimed execution. > - This pull request verifies the retry and its exact workspace-wait parent before it accepts a later user instruction. > - The benefit is that Retry and saved messages can continue through normal admission without replaying old work. ## Linked Issues or Issue Description Refs #13640 and #15483. Related fixes: #18838 covers dispatcher cancellation codes; this change covers a system-cancelled scheduled workspace retry. Related open PR #15383 covers claimed conversation-adapter preparation. This change requires an unclaimed retry with no run profile or owner. **What happened?** A scheduled workspace retry was cancelled before runtime selection. The run retained an expected system cancellation and a `startupCancellation.beforeNativeSelection: false` marker. It had no start timestamp, execution stage, run profile, controller, process, lease or provider events. Recovery recorded an unknown-action hold. A later user message and Retry could not pass that hold. **Expected behavior** A later authorized instruction can start a fresh turn when the saved retry and its exact parent prove that no provider work started. Missing or conflicting evidence must retain the hold. **Steps to reproduce** 1. Record a terminal workspace wait before provider dispatch, with `executionRecovery.kind: workspace_wait` and `providerWorkStarted: false`. 2. Save a scheduled retry for that same company, task and agent. 3. Cancel the retry through the control plane before it claims execution. 4. Send a new user message or request Retry after the cancellation. 5. On the base commit, admission returns no continuation. With this change, normal admission creates one fresh successor. **Paperclip version or commit** Reproduced in source based on master at `499712c68b3ca95801b6114d5f2f0f0ea6759318`. **Deployment mode** Source checkout with isolated embedded PostgreSQL. Hosted rollout and a real provider run remain to be verified after merge. ## What Changed - Extend the existing cancelled-startup proof with one source-bound workspace-retry case. Require the canonical retry parent and reason to match the saved context. - Verify the parent's company, task, agent, time order, immutable Runner claim and typed workspace-wait record. Reject runtime, process, lease, coordinator, result and output evidence on either run. - Keep validation of the full saved result in the database predicate. The bounded execution projection can add empty receipt fields without changing eligibility. - Reuse the existing admission, Retry and deferred-message paths. Preserve cancelled history and unknown action outcomes. Keep permission, Stop, budget and recovery-limit gates. - Add 23 regression cases, including heartbeat message and Retry admission, concurrent saved-message recovery, and 19 conflicting-evidence cases. Document the contract. The cancelled-startup service owns this proof. No provider protocol, schema, UI or shared contract changes. Reads use exact run IDs and return only narrow metadata. The new reads occur only for a cancelled, unstarted Runner retry candidate. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/explicit-native-continuation.test.ts src/services/legacy-execution-recovery.test.ts src/services/execution-blocker-read.test.ts src/services/task-retry.test.ts` — 452 passed. - `pnpm -r typecheck` — passed. - `pnpm build` — passed. - `node scripts/check-module-boundaries.mjs` and `git diff --check` — passed. - Local diff review and secret/private-context scan — passed. Test identities are synthetic. - Regression history: the original positive case failed on unchanged source while 11 negative guards passed. Initial test fixtures then exposed ownership/contract constraints and a missing saved-wake envelope; the fixtures now use valid persisted shapes. The heartbeat Retry test exposed extra keys in the bounded evidence projection. Full-record validation now runs in SQL, and both assembled heartbeat paths pass. The unknown action outcome stays unchanged. - Platform: macOS arm64, Node 25.6.1, pnpm 9.15.4. Tests use real isolated PostgreSQL. They stop at admission and do not call a live provider. - Exact-head GitHub CI: 54 successful checks, 3 skipped, no unfinished or failed checks. Greptile scored `cfbf55cea68e3c69f65f50afcba98f775fa6fa3d` 5/5, with no actionable findings and no review threads. - The full repository `pnpm test:run` was not run locally. A previous full run in this repair effort stalled and was stopped after 2h45. The four complete affected suites pass; full PR CI must pass before merge. ## Risks - False admission could allow work before a predecessor is safe. The proof fails closed on missing parent records, different ownership, a changed retry binding, later parent completion, restore obligations, any execution resource, or any provider/output evidence. - This deliberately covers only system-cancelled, unclaimed workspace retries. Other cancellations retain their current reconciliation path. It does not grant automatic replay, certify unknown external writes, or reset exhausted recovery limits. - No migration. Verify the serving version and an existing saved-message continuation after deployment. ## Model Used - OpenAI `gpt-6.1-sol`, xhigh reasoning, through the Codex harness in T3 Code. Used code editing, shell tools and test execution. The harness does not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/execution-semantics.md | 8 ++ .../src/services/cancelled-native-startup.ts | 65 +++++++++- .../explicit-native-continuation.test.ts | 112 ++++++++++++++++++ 3 files changed, 182 insertions(+), 3 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 44ff7b33b11..029a0a96de8 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -415,6 +415,14 @@ start timestamp, or process identity. A later eligible instruction can adopt the saved messages through normal admission. This rule does not prove a dispatched provider has stopped or change native cancellation acknowledgement. +A system-cancelled workspace retry can also accept a later eligible instruction +when it never acquired execution authority. Admission requires its exact parent +to record a same-company, same-task, same-agent workspace wait before provider +work. Both runs must have no conflicting provider, process, resource or output +evidence. The cancelled retry must have no owner, runtime selection or start +timestamp. Admission preserves the old run and any unknown action outcomes. It +does not replay the cancelled turn or reset recovery limits. + ### Known execution waits at admission A known execution hold is a waiting condition, not a new execution attempt. Every issue wake must read the current effective reconciliation hold under the issue admission lock before creating a run. Resolved recovery bookkeeping can still carry a no-replay hold; only clearing the effective hold makes admission eligible again. The final dispatch gate remains required for changes after admission. diff --git a/server/src/services/cancelled-native-startup.ts b/server/src/services/cancelled-native-startup.ts index f12926a96c3..9b147c04dc3 100644 --- a/server/src/services/cancelled-native-startup.ts +++ b/server/src/services/cancelled-native-startup.ts @@ -1,6 +1,6 @@ import { isPreDispatchReviewWait } from "./pre-dispatch-review-wait.js"; import { createHash } from "node:crypto"; -import { and, eq, inArray, isNotNull, or } from "drizzle-orm"; +import { and, eq, inArray, isNotNull, isNull, lte, or, sql } from "drizzle-orm"; import { environmentLeases, heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; import { isUuidLike } from "@paperclipai/shared"; import { claimedAdapterType } from "./conversation-continuation.js"; @@ -11,6 +11,7 @@ import { legacyControllerBootId } from "./legacy-controller-lease.js"; import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js"; import { hasVerifiedWorkspaceRepair } from "./workspace-recovery-receipt.js"; import { readFailedPreparationSettlement } from "./failed-preparation-settlement.js"; +import { executionStartedEventPredicate, noRetainedExecutionPredicate } from "./pre-dispatch-execution-evidence.js"; type Run = Pick { return isCancelledNativeStartup(db, run, coordinator); } +function noProviderOutputPredicate() { + return and(...[ + heartbeatRuns.runtimeModeResolverVersion, heartbeatRuns.runtimeModeReason, heartbeatRuns.runnerInstanceId, + heartbeatRuns.nativePhase, heartbeatRuns.driverKind, heartbeatRuns.driverVersion, + heartbeatRuns.completionContractId, heartbeatRuns.completionContractSha256, heartbeatRuns.externalRunId, + heartbeatRuns.logStore, heartbeatRuns.logRef, heartbeatRuns.logBytes, heartbeatRuns.logSha256, + heartbeatRuns.stdoutExcerpt, heartbeatRuns.stderrExcerpt, heartbeatRuns.lastOutputAt, + heartbeatRuns.lastOutputStream, heartbeatRuns.lastOutputBytes, heartbeatRuns.exitCode, heartbeatRuns.signal, + ].map(column => isNull(column)), eq(heartbeatRuns.lastOutputSeq, 0), eq(heartbeatRuns.logCompressed, false)); +} + +async function isCancelledUnclaimedWorkspaceWaitRetry(db: Db, run: Run): Promise { + const cancellation = readRunCancellation(run.resultJson); + const issueId = run.issueId ?? run.contextSnapshot?.issueId; + if (run.runtimeMode !== "legacy" || run.startedAt !== null || run.executionStage !== null + || run.runnerProfileJson !== null || typeof issueId !== "string" || !isUuidLike(issueId) + || run.errorCode !== "cancelled" || run.resultJson?.stopReason !== "cancelled" + || cancellation?.source !== "control_plane" || cancellation.expected !== true || cancellation.initiator.type !== "system") return false; + const [source] = await db.select({ agentId: heartbeatRuns.agentId, createdAt: heartbeatRuns.createdAt, + retryOfRunId: heartbeatRuns.retryOfRunId, + }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.id, run.id), + eq(heartbeatRuns.scheduledRetryReason, "workspace_busy"), + sql`${heartbeatRuns.contextSnapshot}->>'retryReason' = 'workspace_busy'`, + sql`${heartbeatRuns.contextSnapshot}->>'retryOfRunId' = ${heartbeatRuns.retryOfRunId}::text`, + sql`(${heartbeatRuns.resultJson} - array['stopReason', 'cancellation', 'startupCancellation', + 'timeoutFired', 'timeoutSource', 'timeoutConfigured', 'effectiveTimeoutSec']) = '{}'::jsonb`, + noRetainedExecutionPredicate(), noProviderOutputPredicate(), isNull(heartbeatRuns.runnerProfileJson), + isNull(heartbeatRuns.usageJson))); + if (!source?.retryOfRunId) return false; + const [parent] = await db.select({ id: heartbeatRuns.id, usageJson: heartbeatRuns.usageJson }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.agentId, source.agentId), eq(heartbeatRuns.id, source.retryOfRunId), + sql`coalesce(${heartbeatRuns.issueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') = ${issueId}`, + eq(heartbeatRuns.runtimeMode, "legacy"), eq(heartbeatRuns.status, "cancelled"), eq(heartbeatRuns.errorCode, "workspace_busy"), + eq(heartbeatRuns.executionStage, "settled"), isNotNull(heartbeatRuns.finishedAt), lte(heartbeatRuns.finishedAt, source.createdAt), + isNull(heartbeatRuns.runtimeModeResolvedAt), isNull(heartbeatRuns.controllerLeaseExpiresAt), + ...[heartbeatRuns.processPid, heartbeatRuns.processGroupId, heartbeatRuns.processStartedAt, + heartbeatRuns.nativeIssueId, heartbeatRuns.nativeSessionId, heartbeatRuns.sessionIdAfter].map(column => isNull(column)), + noProviderOutputPredicate(), + sql`${heartbeatRuns.runnerProfileJson} #>> '{adapterDispatch,adapterType}' = 'paperclip_runner'`, + sql`(${heartbeatRuns.runnerProfileJson} - 'adapterDispatch' - 'chatControlRecoveryAdmission') = '{}'::jsonb`, + sql`${heartbeatRuns.resultJson}->'executionRecovery'->>'kind' = 'workspace_wait'`, + sql`${heartbeatRuns.resultJson}->'executionRecovery'->'providerWorkStarted' = 'false'::jsonb`, + sql`(${heartbeatRuns.resultJson} - 'cancellation' - 'workspaceBusy' - 'executionRecovery') = '{}'::jsonb`, + sql`not exists (select 1 from ${environmentLeases} where ${environmentLeases.companyId} = ${heartbeatRuns.companyId} + and ${environmentLeases.heartbeatRunId} = ${heartbeatRuns.id})`, + sql`not exists (select 1 from ${nativeRunFinalizations} where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId} + and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id})`, + sql`not exists (select 1 from ${heartbeatRunEvents} where ${and(eq(heartbeatRunEvents.companyId, heartbeatRuns.companyId), + eq(heartbeatRunEvents.runId, heartbeatRuns.id), executionStartedEventPredicate())})`, + )); + if (!parent || !hasOnlyPreparationAccountingContext(parent.usageJson, issueId)) return false; + const [result] = await db.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and( + eq(nativeRunResults.companyId, run.companyId), inArray(nativeRunResults.runId, [run.id, parent.id]), + )).limit(1); + return !result; +} + /** Caller holds the coordinator and run locks when using this proof to admit * work. Attempt zero is a durable never-claimed receipt: every native executor * commits its first claim before it can start or attach a provider. */ @@ -170,7 +228,8 @@ export async function isCancelledNativeStartup(db: Db, run: Run, coordinator: Co Boolean(run.controllerBootId && run.controllerBootId !== legacyControllerBootId && run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt <= new Date()); const beforeReviewDispatch = isPreDispatchReviewWait(run) && !coordinator; - const beforeSelection = beforeReviewDispatch || historicalBeforeSelection || run.runtimeMode === "legacy" && !run.runtimeModeResolvedAt && + const unclaimedWorkspaceRetry = !coordinator && await isCancelledUnclaimedWorkspaceWaitRetry(db, run); + const beforeSelection = beforeReviewDispatch || unclaimedWorkspaceRetry || historicalBeforeSelection || run.runtimeMode === "legacy" && !run.runtimeModeResolvedAt && !run.nativeSessionId && !coordinator && claimedAdapterType(run) === "paperclip_runner" && cancellation?.beforeNativeSelection === true; const neverClaimed = run.runtimeMode === "native" && coordinator && @@ -179,7 +238,7 @@ export async function isCancelledNativeStartup(db: Db, run: Run, coordinator: Co !coordinator.controllerPid && !coordinator.leaseOwner && !coordinator.leaseExpiresAt && !coordinator.resultId && !coordinator.failureDetail?.successorRunId; if (!beforeSelection && !neverClaimed) return false; - const settled = beforeReviewDispatch || typeof run.resultJson?.startupPreparationSettledAt === "string"; + const settled = beforeReviewDispatch || unclaimedWorkspaceRetry || typeof run.resultJson?.startupPreparationSettledAt === "string"; // The old preparer can still be unwinding even though the run is terminal. if (!settled && run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > new Date()) return false; const leases = await db.select().from(environmentLeases).where(and( diff --git a/server/src/services/explicit-native-continuation.test.ts b/server/src/services/explicit-native-continuation.test.ts index ce0880a4175..95e7bc52985 100644 --- a/server/src/services/explicit-native-continuation.test.ts +++ b/server/src/services/explicit-native-continuation.test.ts @@ -1259,6 +1259,118 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes return f; } + async function seedCancelledWorkspaceWaitRetry() { + const f = await seed(), parentRunId = randomUUID(); + await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId)); + await db.insert(heartbeatRuns).values({ id: parentRunId, companyId: f.companyId, agentId: f.agentId, + scopeKind: "issue", issueId: f.issueId, + runtimeMode: "legacy", status: "cancelled", executionStage: "settled", errorCode: "workspace_busy", + createdAt: new Date("2026-09-11T09:57:00Z"), startedAt: new Date("2026-09-11T09:57:00Z"), + finishedAt: new Date("2026-09-11T09:58:00Z"), contextSnapshot: { issueId: f.issueId }, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" }, chatControlRecoveryAdmission: {} }, + usageJson: { ledgerScope: { issueId: f.issueId, projectId: null, billingCode: null } }, + resultJson: { executionRecovery: { kind: "workspace_wait", providerWorkStarted: false } } }); + await db.update(heartbeatRuns).set({ runtimeMode: "legacy", nativeIssueId: null, nativeSessionId: null, + scopeKind: "issue", issueId: f.issueId, retryOfRunId: parentRunId, scheduledRetryReason: "workspace_busy", + status: "cancelled", errorCode: "cancelled", startedAt: null, processPid: null, + createdAt: new Date("2026-09-11T09:59:00Z"), runnerProfileJson: null, + contextSnapshot: { issueId: f.issueId, source: "issue.comment", retryReason: "workspace_busy", retryOfRunId: parentRunId }, + resultJson: { stopReason: "cancelled", cancellation: { reason: "Cancelled by control plane", source: "control_plane", + expected: true, initiator: { type: "system" }, recordedAt: "2026-09-11T10:00:00Z" }, + startupCancellation: { errorCode: "cancelled", requestedAt: "2026-09-11T10:00:00Z", beforeNativeSelection: false } }, + }).where(eq(heartbeatRuns.id, f.sourceRunId)); + await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" }) + .where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + return { ...f, parentRunId }; + } + + it("admits a fresh direction after an unclaimed workspace retry was cancelled", async () => { + const f = await seedCancelledWorkspaceWaitRetry(); + expect(await admit(f)).toMatchObject({ previousRunId: f.sourceRunId, commentId: f.commentId }); + const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)); + expect(source).toMatchObject({ status: "cancelled", startedAt: null, processPid: null, runnerProfileJson: null }); + const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + expect(hold.evidence.automaticRecovery).toMatchObject({ replay: "explicit_user_continuation", actionOutcome: "unknown" }); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull(); + expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), + eq(heartbeatRuns.status, "queued")))).toHaveLength(1); + }); + + it.each(["message", "retry"])("admits an unclaimed workspace retry through the heartbeat %s path", async kind => { + const f = await seedCancelledWorkspaceWaitRetry(); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); + const successor = await heartbeatService(db).wakeup(f.agentId, { source: "on_demand", triggerDetail: "manual", + reason: kind === "retry" ? "retry_failed_run" : "issue_commented", + ...(kind === "retry" ? { failedRunId: f.sourceRunId } : {}), requestedByActorType: "user", requestedByActorId: "board", + payload: { issueId: f.issueId, ...(kind === "message" ? { commentId: f.commentId } : {}) }, + contextSnapshot: { issueId: f.issueId, ...(kind === "message" ? { wakeCommentId: f.commentId } : {}) }, + }); + expect(successor).toMatchObject({ status: "queued", contextSnapshot: { forceFreshSession: true, + previousRunId: f.sourceRunId, explicitUserContinuation: { commentId: kind === "message" ? f.commentId : null } } }); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull(); + }); + + it("resumes saved input after an unclaimed workspace retry exactly once", async () => { + const f = await seedCancelledWorkspaceWaitRetry(); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); + const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, + source: "on_demand", reason: "issue_commented", requestedByActorType: "user", requestedByActorId: "board", + status: "deferred_issue_execution", updatedAt: new Date(0), payload: { issueId: f.issueId, commentId: f.commentId, + executionWait: { reason: "execution_recovery" }, + _paperclipWakeContext: { issueId: f.issueId, wakeCommentId: f.commentId, wakeCommentIds: [f.commentId] } }, + }).returning(); + const heartbeat = heartbeatService(db); + await Promise.all([heartbeat.resumeExecutionWaitComments(), heartbeat.resumeExecutionWaitComments()]); + const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued"))); + expect(successors).toHaveLength(1); + expect(successors[0].contextSnapshot).toMatchObject({ forceFreshSession: true, wakeCommentIds: [f.commentId] }); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)))[0]) + .toMatchObject({ status: "coalesced", runId: successors[0].id }); + }); + + it.each(["started", "lease", "launch", "foreign_parent", "parent_executed", "parent_reason", "parent_receipt", "profile", "owner", "operator_stop", "output", + "parent_lease", "parent_launch", "parent_native", "retry_binding", "parent_output", "parent_profile", "parent_later", "restore"])( + "keeps cancelled workspace retries held with contradictory %s evidence", async kind => { + const f = await seedCancelledWorkspaceWaitRetry(); + if (kind === "started") await db.update(heartbeatRuns).set({ startedAt: new Date("2026-09-11T09:59:30Z") }).where(eq(heartbeatRuns.id, f.sourceRunId)); + if (kind === "lease") await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: f.sourceRunId, + provider: "local", leasePolicy: "ephemeral", status: "released", cleanupStatus: "success", releasedAt: new Date() }); + if (kind === "launch") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, + agentId: f.agentId, runId: f.sourceRunId, seq: 1, eventType: PROCESS_START_REQUESTED, payload: {} }); + if (kind === "foreign_parent") { + const foreign = await seed(); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, retryReason: "workspace_busy", + retryOfRunId: foreign.sourceRunId } }).where(eq(heartbeatRuns.id, f.sourceRunId)); + } + if (kind === "parent_executed" || kind === "parent_receipt") await db.update(heartbeatRuns).set({ resultJson: + kind === "parent_receipt" ? {} : { executionRecovery: { kind: "workspace_wait", providerWorkStarted: true } } }) + .where(eq(heartbeatRuns.id, f.parentRunId)); + if (kind === "parent_reason") await db.update(heartbeatRuns).set({ errorCode: "cancelled" }).where(eq(heartbeatRuns.id, f.parentRunId)); + if (kind === "profile") await db.update(heartbeatRuns).set({ runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } } }) + .where(eq(heartbeatRuns.id, f.sourceRunId)); + if (kind === "owner") await db.update(heartbeatRuns).set({ controllerBootId: randomUUID() }).where(eq(heartbeatRuns.id, f.sourceRunId)); + if (kind === "operator_stop") await db.update(heartbeatRuns).set({ resultJson: { stopReason: "cancelled", + cancellation: { reason: "Stopped", source: "operator", expected: true, initiator: { type: "user", userId: "board" }, + recordedAt: "2026-09-11T10:00:00Z" } } }).where(eq(heartbeatRuns.id, f.sourceRunId)); + if (kind === "output") await db.update(heartbeatRuns).set({ lastOutputSeq: 1 }).where(eq(heartbeatRuns.id, f.sourceRunId)); + if (kind === "parent_lease") await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: f.parentRunId, + provider: "local", leasePolicy: "ephemeral", status: "released", cleanupStatus: "success", releasedAt: new Date() }); + if (kind === "parent_launch") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, + agentId: f.agentId, runId: f.parentRunId, seq: 1, eventType: PROCESS_START_REQUESTED, payload: {} }); + if (kind === "parent_native") await db.update(heartbeatRuns).set({ nativeIssueId: f.issueId }).where(eq(heartbeatRuns.id, f.parentRunId)); + if (kind === "retry_binding") await db.update(heartbeatRuns).set({ retryOfRunId: null }).where(eq(heartbeatRuns.id, f.sourceRunId)); + if (kind === "parent_output") await db.update(heartbeatRuns).set({ lastOutputSeq: 1 }).where(eq(heartbeatRuns.id, f.parentRunId)); + if (kind === "parent_profile") await db.update(heartbeatRuns).set({ runnerProfileJson: { + adapterDispatch: { adapterType: "paperclip_runner" }, remoteWorkspace: {} } }).where(eq(heartbeatRuns.id, f.parentRunId)); + if (kind === "parent_later") await db.update(heartbeatRuns).set({ finishedAt: new Date("2026-09-11T10:01:00Z") }).where(eq(heartbeatRuns.id, f.parentRunId)); + if (kind === "restore") await db.update(heartbeatRuns).set({ resultJson: { stopReason: "cancelled", + workspaceRestoreFailure: "restore_unsafe_archive" } }).where(eq(heartbeatRuns.id, f.sourceRunId)); + expect(await admit(f)).toBeNull(); + if (kind === "lease" || kind === "parent_lease") await db.delete(environmentLeases) + .where(eq(environmentLeases.heartbeatRunId, kind === "lease" ? f.sourceRunId : f.parentRunId)); + }, + ); + it("keeps provider diagnostics out of admission evidence and preserves absent results", async () => { const f = await seedCancelledReviewWait(); const heartbeat = heartbeatService(db);