refactor(heartbeat): extract prepared runtime execution (#15803)

## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat prepares, dispatches, and completes each agent run. > - After the runtime-selection extraction, the main service still had 7,295 lines. > - Provider dispatch and workspace finalization formed one large block with shared cleanup rules. > - This pull request moves that block into a typed runtime-execution module. > - The service now has 6,417 lines. Readers can follow admission and outer cleanup without reading the provider execution details. ## Linked Issues or Issue Description Follow-up to #15746. I searched open heartbeat PRs and issues. I found no competing runtime-execution extraction. **What existing behavior does this improve?** The organization and test coverage of native and legacy heartbeat execution. **Current behavior** The heartbeat service contains a 912-line block for provider dispatch, usage capture, workspace finalization, and instruction cleanup. **Proposed behavior** Move that block to `server/src/services/heartbeat/runtime-execution.ts`. Keep the atomic dispatch gate, admission, terminal run status, and outer lease cleanup in the service. Preserve run behavior. **Reason and benefit** This gives runtime execution a named boundary and direct tests. It removes 878 lines from the main service in one bounded change. **Breaking changes** None intended. There are no schema, API, or policy changes. ## What Changed - Add `executeHeartbeatRuntime` for prepared native and legacy execution, usage capture, workspace finalization, and gateway/instruction cleanup. - Return an explicit dispatch result. Update outer ownership state through immediate callbacks, including before exceptions. Read instruction state through live getters. - Move the two internal recovery error classes with their existing behavior. The caller imports the same classes for `instanceof` checks. - Add 15 database-backed execution tests. Keep the existing heartbeat regression suites. - Repair the native Stop acceptance fixture: installed Codex now takes precedence over `PATH`. A test-only loader pins the fake provider. An isolated home and dummy key keep readiness offline. Add four loader tests. - Document the module boundary and fixture setup. ## Verification - Passed: eight focused heartbeat suites (483 distinct tests, including 15 new execution tests). The boundary suite passed again after adding the failed-restore retention case from review. - Passed: `node --test tests/e2e/fixtures/composer-stop-codex-loader.test.mjs` (four tests). - Passed: `pnpm -r typecheck` and `pnpm build`. - Passed: AST comparison of all 912 moved lines after the explicit boundary substitutions. SQL template text, recovery class bodies, public exports, and the rest of the executor match the baseline. - Passed: browser Stop journeys for legacy process and native Paperclip Runner (two tests). The native fixture log proves provider `turn/start` before Stop and `turn/interrupt` after Stop. - Passed: all three ACP continuation browser journeys (same-session queued input, no replay of an interrupted write, and a new direction after composer Stop). Five relevant browser journeys passed in total. - Passed: the full CI test groups, including all server, chat, workspace, serialized-server, runner, and browser shards. - Local full-suite limitation: `pnpm test:run` first hit the macOS shared-memory segment limit while starting embedded PostgreSQL. After releasing only this task’s unused acceptance-server segments, the retry progressed without reported failures. I stopped that duplicate run after the full CI test groups passed. - Passed: all 56 PR checks/statuses on `8694c116e8d43ed8ac11bc4c58623ea3af34a675`. Greptile is 5/5 with no unresolved review threads. ## Risks - Cleanup depends on state written before an error. The module reports dispatch, retention, and recovery ownership immediately. Tests cover early returns and exceptional cleanup. - Instruction callbacks can replace saved metadata during execution. Live getters preserve those changes. - Native acceptance uses a real server, database, browser, and Rust runner with a deterministic provider. It does not verify a live provider service. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and repository/tool access. The exact serving model ID and context-window limit were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 08:22:20 +02:00 · 2026-10-10 09:10:53 -05:00
1 parent 2a2071340d
commit 1fb3f9746e
@@ -1358,8 +1358,16 @@ Native selection retains the row lock that serializes it with cancellation and
 controller lease renewal. The executor receives an explicit selection result;
 lifecycle and ownership callbacks update its teardown state immediately, including
 when preparation later throws. Public sandbox lifecycle and model fallback helpers
-remain re-exported by `heartbeat.ts`. Provider dispatch and final cleanup stay in
-the executor.
+remain re-exported by `heartbeat.ts`.
+
+Runtime execution is in `server/src/services/heartbeat/runtime-execution.ts`.
+`executeHeartbeatRuntime` dispatches a prepared native session or legacy adapter,
+captures usage, finalizes workspace operations, and releases gateway tokens and
+instruction copies. The executor supplies the atomic dispatch gate and retains
+admission, terminal run status, retry handling, and outer lease cleanup. An explicit
+dispatch result preserves early cancellation returns; ownership callbacks update
+teardown state before errors propagate. Instruction getters retain updates made
+during provider execution. Keep policy changes separate from this extraction.

 ## Wake Context Delivery

@@ -159,7 +159,9 @@ pnpm exec playwright test --config tests/e2e/playwright-composer-stop.config.ts
 The suite boots a disposable local-trusted instance on port 3199 (override with
 `PAPERCLIP_E2E_PORT`). It never attaches to an existing server. Native coverage
 is explicitly skipped without the fixture; it must not use a logged-in provider
-as a fallback. Test companies are archived during cleanup.
+as a fallback. The acceptance server pins its Codex command resolver to the fixture
+and uses an empty credential home, because an installed Codex dependency takes
+precedence over `PATH`. Test companies are archived during cleanup.

 For each runner, the journey starts a parent, child, and unrelated task, plus a
 terminal child. It sends while running and verifies the durable queue, clicks
@@ -0,0 +1,252 @@
+import { randomUUID } from "node:crypto";
+import { mkdtemp, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { agents, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issues, nativeRunFinalizations, runUsageReceipts, workspaceOperations, type Db } from "@paperclipai/db";
+import { eq, sql } from "drizzle-orm";
+import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
+import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
+import * as legacyRecovery from "../legacy-execution-recovery.js";
+import { createAgentIdentityRedactor } from "../agent-identity-redaction.js";
+import { createAdapterExecutionControl } from "../adapter-execution-control.js";
+import { executionWorkspaceService } from "../execution-workspaces.js";
+import { workspaceOperationService } from "../workspace-operations.js";
+import { instanceSettingsService } from "../instance-settings.js";
+import { executePaperclipNativeSession, NativeCancellationPendingRecoveryError, NativeControllerDetachedForRestartError } from "../native-runtime/native-session-executor.js";
+import { NativeRunnerOwnershipUnverifiedError } from "../native-runtime/native-runner-ownership.js";
+import { NativeWorkspaceFinalizationBusyError, NativeWorkspaceFinalizationOwnershipLostError } from "../native-runtime/native-workspace-finalization-ownership.js";
+import { resolveHeartbeatNativeRuntimeMode } from "../native-runtime/runtime-mode.js";
+import { buildNativeExecutionInput } from "../native-runtime/native-execution-input.js";
+import { nativeRuntimeContextFixture } from "../native-runtime/runtime-context.test-fixture.js";
+import { createHeartbeatRunState } from "./run-state.js";
+import { buildEffectiveRunSessionConfigMetadata } from "./workspaces.js";
+import { revokeHeartbeatRunGatewayTokens } from "./run-preparation.js";
+import { executeHeartbeatRuntime, type HeartbeatRuntimeExecutionInput } from "./runtime-execution.js";
+
+// Keep real accounting, workspace barriers, and task-session persistence. Providers
+// and gateway delivery are external boundaries; browser acceptance covers their wiring.
+vi.mock("../native-runtime/native-session-executor.js", async original => ({
+  ...await original<typeof import("../native-runtime/native-session-executor.js")>(),
+  executePaperclipNativeSession: vi.fn(),
+}));
+vi.mock("./run-preparation.js", async original => ({
+  ...await original<typeof import("./run-preparation.js")>(),
+  buildPaperclipRuntimeMcpServers: vi.fn(async () => []),
+  createAdapterRuntimeToolAccess: vi.fn(() => null),
+  createManagedMcpRunConfig: vi.fn(async () => null),
+  revokeHeartbeatRunGatewayTokens: vi.fn(async () => {}),
+}));
+
+const support = await getEmbeddedPostgresTestSupport();
+describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () => {
+  let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
+  let db: Db;
+  let home: string;
+  beforeAll(async () => {
+    home = await mkdtemp(join(tmpdir(), "paperclip-runtime-execution-"));
+    vi.stubEnv("PAPERCLIP_HOME", home);
+    database = await startEmbeddedPostgresTestDatabase("paperclip-runtime-execution");
+    db = createDb(database.connectionString);
+    await db.execute(sql`set client_min_messages = warning`);
+  }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
+  afterAll(async () => { await db?.$client.end(); await database?.cleanup(); vi.unstubAllEnvs(); if (home) await rm(home, { recursive: true, force: true }); });
+  beforeEach(async () => {
+    await db.execute(sql`truncate table companies restart identity cascade`);
+    vi.clearAllMocks();
+    vi.mocked(executePaperclipNativeSession).mockResolvedValue({ exitCode: 0, signal: null, timedOut: false });
+  });
+
+  async function fixture(native = false) {
+    const [company] = await db.insert(companies).values({ name: "Runtime execution", issuePrefix: "EXE" }).returning();
+    const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Executing agent", status: "running", adapterType: native ? "paperclip_runner" : "process", adapterConfig: { provider: "codex" } }).returning();
+    const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Complete this task", assigneeAgentId: agent.id }).returning();
+    const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: native ? "native" : "legacy" }).returning();
+    const state = createHeartbeatRunState(db);
+    const executionControl = createAdapterExecutionControl();
+    const adapter = { type: "process", label: "Test process", testEnvironment: vi.fn(), execute: vi.fn<HeartbeatRuntimeExecutionInput["runtime"]["adapter"]["execute"]>(async () => ({ exitCode: 0, signal: null, timedOut: false })) };
+    let instructionSave: Record<string, unknown> | null = null;
+    const nativeExecution = native ? buildNativeExecutionInput({
+      companyId: company.id, agentId: agent.id, runId: run.id, issue,
+      taskPrompt: issue.title, workspace: { id: run.id, cwd: home, repoUrl: null, repoRef: null, branchName: null },
+      normalizedSessionId: null, provider: "codex", runtimeContext: nativeRuntimeContextFixture(),
+      completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", contract: { revision: "1", objective: issue.title, criteria: [{ id: "output", requirement: issue.title }] } },
+    }) : null;
+    const input: HeartbeatRuntimeExecutionInput = {
+      run, agent, options: {},
+      task: { issueRef: issue, issueContext: null, context: {}, issueId: issue.id, taskKey: issue.id, executionContinuation: null },
+      runtime: { nativeExecution, nativeRunnerInstanceId: native ? randomUUID() : null, getNativeFreshSessionHandoff: undefined,
+        nativeRuntimeResolution: resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: native, runtimeConfig: {}, adapterConfig: agent.adapterConfig, agent, issue, target: null, workspaceId: null }),
+        adapter, runtimeForAdapter: { taskKey: issue.id, sessionId: null, sessionParams: null, sessionDisplayId: null }, getFreshSessionHandoff: undefined, runOptions: {},
+      },
+      workspace: { persistedExecutionWorkspace: null, executionWorkspace: { baseCwd: home, cwd: home, source: "agent_home", projectId: null, workspaceId: null, repoUrl: null, repoRef: null, strategy: "project_primary", branchName: null, worktreePath: null, warnings: [], created: false, branchCreatedByRuntime: false },
+        executionTarget: null, nativeWorkspaceSync: null, workspaceRestoreSource: undefined, remoteExecution: null,
+        workspaceOperationRecorder: workspaceOperationService(db).createRecorder({ companyId: company.id, heartbeatRunId: run.id, issueId: issue.id }),
+      },
+      config: { runtimeConfig: {}, resolvedConfig: {}, managedAiRuntime: undefined,
+        resolvedInstanceSettings: { experimental: { enableWorkspaceBranchReconcileForward: false, enableWorkspaceDirtyQuarantineRepair: false } },
+        configuredTaskEnvironment: {}, adapterEnv: {}, agentIdentity: undefined, runtimeEnv: {}, useHostGitHub: false, githubSelection: { configured: false },
+      },
+      session: { runtimeSessionParamsForAdapter: null, configuredModel: null,
+        sessionConfigMetadata: await buildEffectiveRunSessionConfigMetadata({ adapterType: agent.adapterType, effectiveAdapterConfig: {}, agentRuntimeConfig: {}, issueOverrides: null, workspaceConfig: null, environment: null, environmentEnv: null, projectEnv: null, routineEnv: null, runtimeSkills: [] }), goalCheckpointSession: { current: null },
+      },
+      instructions: { instructionCopy: null, getInstructionSave: () => instructionSave, nativeInstructionWorkingCopy: () => undefined,
+        collectStoppedInstructions: async () => { instructionSave = { state: "saved", revisionId: "during-execution" }; },
+        nativeInstructionReservation: null, releaseInstructionCopy: vi.fn(async () => {}),
+      },
+      trace: { providerTraceCapture: null, nativeRunnerPreparationSpans: [], attestedQuestionResponseAtMs: null, attemptStartedAtMs: Date.now(), environmentAcquireStartedAtMs: Date.now(), environmentRealizeEndedAtMs: Date.now() },
+      control: { executionControl, executionPhaseContext: { onExecutionPhase: executionControl.phases.enter },
+        dispatchResolvedInteractionContinuationWithAtomicGate: async dispatch => ({ dispatched: true, resultPromise: dispatch(vi.fn()) }),
+      },
+      output: { identityRedactor: createAgentIdentityRedactor(), appendIdentityRedactedLog: vi.fn(async () => {}), onLog: vi.fn(async () => {}), onAdapterEvent: vi.fn(async () => {}), onAdapterMeta: vi.fn(async () => {}) },
+      services: { ...state, executionWorkspacesSvc: executionWorkspaceService(db), instanceSettings: instanceSettingsService(db),
+        envOrchestrator: { releaseForRun: vi.fn() }, cancelRunInternal: vi.fn(), enqueueWakeup: vi.fn(),
+        persistRunProcessMetadata: vi.fn(), recordCurrentHeartbeatRunRuntimeProgress: vi.fn(), dispatchPendingNativeStatusWakeups: vi.fn(),
+      },
+      effects: { onUsageCaptureReady: vi.fn(), onNativeDispatchStarted: vi.fn(), onLegacyAdapterEntered: vi.fn(), onWorkspaceRestoreFailure: vi.fn(), onProviderResourceDisposition: vi.fn(), onNativeWorkspaceFinalizeScheduled: vi.fn(), onNativeSessionResumeScheduled: vi.fn(), onNativeOwnershipHeld: vi.fn() },
+    };
+    return { input, adapter, run };
+  }
+  const read = (id: string) => db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)).then(rows => rows[0]!);
+  const finalizations = () => db.select().from(workspaceOperations).where(eq(workspaceOperations.phase, "workspace_finalize"));
+  function expectCleanup(input: HeartbeatRuntimeExecutionInput) {
+    expect(revokeHeartbeatRunGatewayTokens).toHaveBeenCalledWith({ db, companyId: input.agent.companyId, runId: input.run.id });
+    expect(input.instructions.releaseInstructionCopy).toHaveBeenCalledOnce();
+  }
+
+  it.each([false, true])("preserves the dispatch gate's early return and cleanup (native: %s)", async native => {
+    const { input, adapter } = await fixture(native);
+    input.control.dispatchResolvedInteractionContinuationWithAtomicGate = async () => ({ dispatched: false });
+    expect(await executeHeartbeatRuntime(db, input)).toEqual({ dispatched: false });
+    expect(adapter.execute).not.toHaveBeenCalled();
+    expect(executePaperclipNativeSession).not.toHaveBeenCalled();
+    expect(input.effects.onNativeDispatchStarted).not.toHaveBeenCalled();
+    expect(input.effects.onLegacyAdapterEntered).not.toHaveBeenCalled();
+    expect(await finalizations()).toHaveLength(0);
+    expectCleanup(input);
+  });
+
+  it("captures usage and live instruction saves before finalizing a legacy workspace", async () => {
+    const { input, adapter, run } = await fixture();
+    adapter.execute.mockImplementationOnce(async ctx => {
+      expect(input.effects.onLegacyAdapterEntered).toHaveBeenCalledOnce();
+      expect(input.effects.onUsageCaptureReady).toHaveBeenCalledOnce();
+      await ctx.onProviderStopped?.();
+      return { exitCode: 0, signal: null, timedOut: false, usage: { inputTokens: 10, outputTokens: 4 }, costUsd: 0.01, provider: "test" };
+    });
+    expect(await executeHeartbeatRuntime(db, input)).toMatchObject({ dispatched: true, adapterResult: { usageComplete: true, resultJson: { instructionSave: { state: "saved", revisionId: "during-execution" } } } });
+    expect(await read(run.id)).toMatchObject({ status: "running", costAccountingPending: true, usageJson: { inputTokens: 10, outputTokens: 4, accountingReceiptReady: true } });
+    expect(await db.select().from(runUsageReceipts)).toHaveLength(1);
+    expect(await finalizations()).toMatchObject([{ status: "succeeded" }]);
+    expectCleanup(input);
+  });
+
+  it("preserves Stop metadata and marks no-work accounting without manufacturing a receipt", async () => {
+    const { input, adapter, run } = await fixture();
+    await db.update(heartbeatRuns).set({ status: "cancelled", resultJson: { stoppedByUser: true } }).where(eq(heartbeatRuns.id, run.id));
+    adapter.execute.mockResolvedValueOnce({ exitCode: 0, signal: null, timedOut: false, executionRecovery: { kind: "bootstrap", providerWorkStarted: false } });
+    await executeHeartbeatRuntime(db, input);
+    expect(await read(run.id)).toMatchObject({ status: "cancelled", resultJson: { stoppedByUser: true }, usageJson: { accountingProviderWorkStarted: false } });
+    expect(await db.select().from(runUsageReceipts)).toHaveLength(0);
+    expectCleanup(input);
+  });
+
+  it("records the failed workspace and keeps the original provider error", async () => {
+    const { input, adapter } = await fixture();
+    const error = new Error("provider failed");
+    adapter.execute.mockRejectedValueOnce(error);
+    await expect(executeHeartbeatRuntime(db, input)).rejects.toBe(error);
+    expect(await finalizations()).toMatchObject([{ status: "failed", metadata: { errorMessage: "provider failed" } }]);
+    expect(input.effects.onLegacyAdapterEntered).toHaveBeenCalledOnce();
+    expectCleanup(input);
+  });
+
+  it.each([
+    [new NativeCancellationPendingRecoveryError(), null],
+    [new NativeControllerDetachedForRestartError(), "onNativeSessionResumeScheduled"],
+    [new NativeRunnerOwnershipUnverifiedError(), "onNativeOwnershipHeld"],
+    [new NativeWorkspaceFinalizationBusyError(), "onNativeWorkspaceFinalizeScheduled"],
+    [new NativeWorkspaceFinalizationOwnershipLostError(), "onNativeWorkspaceFinalizeScheduled"],
+  ] as const)("transfers recovery ownership for %s before cleanup", async (error, effect) => {
+    const { input } = await fixture(true);
+    vi.mocked(executePaperclipNativeSession).mockRejectedValueOnce(error);
+    input.instructions.releaseInstructionCopy = vi.fn(async () => {
+      if (effect) expect(input.effects[effect]).toHaveBeenCalledOnce();
+    });
+    await expect(executeHeartbeatRuntime(db, input)).rejects.toBe(error);
+    expect(input.effects.onNativeDispatchStarted).toHaveBeenCalledOnce();
+    expect(input.effects.onLegacyAdapterEntered).not.toHaveBeenCalled();
+    expect(await finalizations()).toHaveLength(0);
+    expectCleanup(input);
+  });
+
+  it("dispatches native execution with the selected session and persists its usage", async () => {
+    const { input, adapter, run } = await fixture(true);
+    vi.mocked(executePaperclipNativeSession).mockImplementationOnce(async native => {
+      expect(native.execution).toBe(input.runtime.nativeExecution);
+      expect(native.runnerInstanceId).toBe(input.runtime.nativeRunnerInstanceId);
+      await native.onUsage?.({ usage: { inputTokens: 7, outputTokens: 3 }, complete: true });
+      return { exitCode: 0, signal: null, timedOut: false, usageComplete: false };
+    });
+    expect(await executeHeartbeatRuntime(db, input)).toMatchObject({ dispatched: true, adapterResult: { usage: { inputTokens: 7, outputTokens: 3 }, usageComplete: true } });
+    expect(adapter.execute).not.toHaveBeenCalled();
+    expect(input.effects.onNativeDispatchStarted).toHaveBeenCalledOnce();
+    expect(await read(run.id)).toMatchObject({ status: "running", usageJson: { inputTokens: 7, outputTokens: 3, accountingReceiptReady: true } });
+    expect(await finalizations()).toMatchObject([{ status: "succeeded" }]);
+    expectCleanup(input);
+  });
+
+  it("retains the same-run retry signal without recording a failed workspace", async () => {
+    const { input, run } = await fixture(true);
+    await db.update(heartbeatRuns).set({ nativeIssueId: input.task.issueId }).where(eq(heartbeatRuns.id, run.id));
+    await db.insert(nativeRunFinalizations).values({ companyId: run.companyId, issueId: input.task.issueId!, runId: run.id, phase: "retryable_failure" });
+    const error = new Error("session connection lost");
+    vi.mocked(executePaperclipNativeSession).mockRejectedValueOnce(error);
+    await expect(executeHeartbeatRuntime(db, input)).rejects.toMatchObject({
+      name: "NativeSessionResumeScheduledError", original: error,
+    });
+    expect(input.effects.onNativeSessionResumeScheduled).toHaveBeenCalledOnce();
+    expect(await finalizations()).toHaveLength(0);
+    expectCleanup(input);
+  });
+
+  it("records a failed barrier if the successful finalization write fails", async () => {
+    const { input } = await fixture();
+    const error = new Error("finalization write failed");
+    vi.spyOn(input.workspace.workspaceOperationRecorder, "recordOperation").mockRejectedValueOnce(error);
+    await expect(executeHeartbeatRuntime(db, input)).rejects.toBe(error);
+    expect(await finalizations()).toMatchObject([{ status: "failed" }]);
+    expectCleanup(input);
+  });
+
+  it("retains a failed sandbox restore even when persisting its receipt throws", async () => {
+    const { input, adapter, run } = await fixture();
+    input.workspace.executionTarget = { kind: "remote", transport: "sandbox", remoteCwd: home };
+    const evidence = { workspaceRestoreFailure: "restore_failed", workspaceRestoreDiagnostic: { message: "copy-back failed" } };
+    adapter.execute.mockResolvedValueOnce({ exitCode: 0, signal: null, timedOut: false, resultJson: evidence });
+    const order: string[] = [];
+    input.effects.onWorkspaceRestoreFailure = vi.fn(() => { order.push("restore evidence"); });
+    input.effects.onProviderResourceDisposition = vi.fn(() => { order.push("retain sandbox"); });
+    input.instructions.releaseInstructionCopy = vi.fn(async () => { order.push("cleanup"); });
+    const error = new Error("restore receipt write failed");
+    const record = vi.spyOn(legacyRecovery, "recordLegacyWorkspaceRestoreFailure").mockImplementationOnce(async () => {
+      order.push("receipt write");
+      throw error;
+    });
+    try {
+      await expect(executeHeartbeatRuntime(db, input)).rejects.toBe(error);
+      expect(record).toHaveBeenCalledWith(db, run, evidence, input.workspace.workspaceRestoreSource);
+      expect(input.effects.onWorkspaceRestoreFailure).toHaveBeenCalledWith(evidence);
+      expect(input.effects.onProviderResourceDisposition).toHaveBeenCalledWith("stop_and_retain");
+      expect(order).toEqual(["restore evidence", "retain sandbox", "receipt write", "cleanup"]);
+      expect(await finalizations()).toMatchObject([{ status: "failed" }]);
+      expectCleanup(input);
+    } finally { record.mockRestore(); }
+  });
+
+  it("runs instruction cleanup even when gateway revocation fails", async () => {
+    const { input } = await fixture();
+    vi.mocked(revokeHeartbeatRunGatewayTokens).mockRejectedValueOnce(new Error("gateway unavailable"));
+    expect(await executeHeartbeatRuntime(db, input)).toMatchObject({ dispatched: true });
+    expectCleanup(input);
+  });
+});
@@ -0,0 +1,23 @@
+// The installed Codex dependency takes precedence over PATH. Pin only this
+// acceptance server's command resolver to its deterministic protocol fixture.
+import { accessSync, constants } from "node:fs";
+import { registerHooks } from "node:module";
+import { isAbsolute } from "node:path";
+
+const command = process.env.PAPERCLIP_STOP_CODEX_COMMAND;
+if (process.env.NODE_ENV !== "test" || !command || !isAbsolute(command)) {
+  throw new Error("Composer Stop requires an absolute fixture command in NODE_ENV=test");
+}
+accessSync(command, constants.X_OK);
+registerHooks({
+  load(url, context, nextLoad) {
+    if (/\/packages\/paperclip-runner\/(src|dist)\/drivers\/codex\/codex-command\.(js|ts)$/.test(url)) {
+      return {
+        format: "module",
+        shortCircuit: true,
+        source: `export const resolveCodexCommand = () => ${JSON.stringify(command)}; export const resolvePinnedCodexCommand = resolveCodexCommand;`,
+      };
+    }
+    return nextLoad(url, context);
+  },
+});
@@ -0,0 +1,41 @@
+import assert from "node:assert/strict";
+import { spawnSync } from "node:child_process";
+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { fileURLToPath } from "node:url";
+import { test } from "node:test";
+
+const loader = fileURLToPath(new URL("./composer-stop-codex-loader.mjs", import.meta.url));
+const resolver = new URL("../../../packages/paperclip-runner/src/drivers/codex/codex-command.ts", import.meta.url).href;
+function launch(env, source = "") {
+  return spawnSync(process.execPath, ["--import", loader, "--input-type=module", "-e", source], {
+    encoding: "utf8", env: { ...process.env, NODE_ENV: "test", PAPERCLIP_STOP_CODEX_COMMAND: "", ...env },
+  });
+}
+
+test("pins the fixture even when a bundled Codex dependency is installed", () => {
+  const dir = mkdtempSync(join(tmpdir(), "composer-stop-loader-"));
+  const command = join(dir, "codex");
+  try {
+    writeFileSync(command, "#!/bin/sh\nprintf 'composer-stop-fixture'\n", { mode: 0o755 });
+    const result = launch({ PAPERCLIP_STOP_CODEX_COMMAND: command }, `
+      import { resolveCodexCommand, resolvePinnedCodexCommand } from ${JSON.stringify(resolver)};
+      import { execFileSync } from "node:child_process";
+      import assert from "node:assert/strict";
+      assert.equal(resolveCodexCommand(), ${JSON.stringify(command)});
+      assert.equal(resolvePinnedCodexCommand(), ${JSON.stringify(command)});
+      process.stdout.write(execFileSync(resolveCodexCommand(), ["--version"]));
+    `);
+    assert.equal(result.status, 0, result.stderr);
+    assert.equal(result.stdout, "composer-stop-fixture");
+  } finally { rmSync(dir, { recursive: true, force: true }); }
+});
+
+for (const env of [{}, { PAPERCLIP_STOP_CODEX_COMMAND: "codex" }, { NODE_ENV: "production", PAPERCLIP_STOP_CODEX_COMMAND: process.execPath }]) {
+  test(`rejects a missing or unsafe fixture configuration: ${JSON.stringify(env)}`, () => {
+    const result = launch(env);
+    assert.notEqual(result.status, 0);
+    assert.match(result.stderr, /Composer Stop requires an absolute fixture command/);
+  });
+}
@@ -19,6 +19,7 @@ if (fixture) {
    throw new Error(
      "PAPERCLIP_STOP_FAKE_CODEX must name the built fake-codex-app-server binary",
    );
+  fs.mkdirSync(path.join(fixtureDir, "codex-home"));
  const quote = (value: string) => `'${value.replaceAll("'", "'\\''")}'`;
  fs.writeFileSync(
    path.join(fixtureDir, "codex"),
@@ -37,6 +38,14 @@ export default defineConfig({
      ...server?.env,
      HEARTBEAT_SCHEDULER_INTERVAL_MS: "10000",
      PATH: `${fixtureDir}${path.delimiter}${process.env.PATH ?? ""}`,
+      ...(fixture ? {
+        // Provider selection prefers the installed dependency over PATH.
+        // Scope this override and its empty credential home to the test server.
+        PAPERCLIP_STOP_CODEX_COMMAND: path.join(fixtureDir, "codex"),
+        CODEX_HOME: path.join(fixtureDir, "codex-home"),
+        OPENAI_API_KEY: "composer-stop-fixture-key",
+        NODE_OPTIONS: `${server?.env?.NODE_OPTIONS ?? ""} --import=${path.resolve(import.meta.dirname, "fixtures/composer-stop-codex-loader.mjs")}`,
+      } : {}),
    },
  },
 });