feat(recovery): instrument 20 durable write boundaries with crash points (#19938)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A control-plane restart between two durable writes leaves an
in-progress record owned by a dead process
> - #19937 added `crashPoint(name)` and a registry of 20 named
boundaries from the Phase 0 inventory
> - A name without a call site cannot fire, so the crash-point suite
cannot test it
> - This pull request puts one call at each registered boundary and adds
a test that keeps the registry and the call sites in step
> - The benefit is that the on-demand suite (next pull request) can kill
the control plane at every known write-ahead gap

## Linked Issues or Issue Description

Refs #19831 (plan, Phase 5). Refs #19846 (inventory, "Write-ahead
coverage"). Stacked on #19937 (helper and registry). Refs #19861
(ownable kind ids).

**Problem**
The registered crash points have no call sites.

**Expected behavior**
Each registered point is called at its gap: after the first durable
write or outside effect returns, before the next one starts. Disarmed
(always in production) each call is one comparison.

**Scope**
Call sites and a coverage test only. The suite runner and CI workflow
follow.

## What Changed

- Server call sites (`@paperclipai/shared/crash-points`):
- `sandbox-creation-operations.ts`: around the provider `create` call
(intent → provider → lease).
- `native-session-executor.ts`: after `native.process_start_requested`,
before spawn.
- `paperclip-control-plane-port.ts`: after the result-accept
transaction, before finalization starts (only when a result was
inserted).
- `native-run-finalizer.ts`: after `commitNativeStatusDecision`, before
the run projection.
- `workspace-operations.ts`: `workspace_finalize` only — after the
running row, and after `settle()` before the done row.
- `environment-runtime.ts`: provider release → lease row; teardown claim
→ destroy; reusable destroy → lease release; stop-and-retain claim →
stop.
- `heartbeat.ts`: lease release → runtime services; budget reservation
hold → dispatch.
- `heartbeat/queue.ts`: run claimed `running` → wake request `claimed`.
- `run-cost-accounting.ts`, `usage-receipts.ts`: accounting commit →
spool unlink; receipt spool → DB row → unlink.
- `packages/paperclip-runner/src/crash-point.ts`: a small copy of the
helper for controller code in this package, which does not depend on
`@paperclipai/shared` (adding that dependency would change the runner
content ID). It arms only `runner.*` names. runnerd never sees the
variable: its environment is an allowlist.
- Runner call sites: after the runnerd spawn, before its identity is
published (both spawn paths); `turn.start` result received → controller
journal saved → next send.
- `server/src/__tests__/crash-point-call-sites.test.ts`: every
registered point has a call site, every call names a registered point,
and every `kinds` entry is an ownable kind id of the Phase 1 registry.

## Verification

- `cd server && npx vitest run
src/__tests__/crash-point-call-sites.test.ts` → 3 passed.
- `pnpm --filter @paperclipai/server typecheck` → clean. `pnpm --filter
@paperclipai/paperclip-runner exec tsc --noEmit -p .` → clean.
- Behavior with nothing armed is unchanged: each call returns at its
first comparison (measured in #19937: about 3.7 ns).

## Risks

- Low risk. No call changes control flow unless `PAPERCLIP_CRASH_POINT`
names it and `NODE_ENV` is not `production`.
- Points on the path before `turn.start` (process start, runner spawn)
cost nanoseconds, within plan invariant 8.
- `usage-receipts.ts`: the persist-then-unlink one-liner became a block
with the same statements in the same order.
- The runner package change gives the runner a new content ID, as any
runner source change does. No persisted run state changes
(rules/upgrade-safe-run-state.mdc).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits) and a read-only exploration subagent that
mapped the boundaries.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts

@@ -45,6 +45,7 @@ import {
   type DurableRecoveryIdentity,
   type DurableWarmRunTransition,
 } from "./prp-transport-types.js";
+import { crashPoint } from "../crash-point.js";
 
 const protocol = "paperclip.runner";
 export const RUNNER_PRP_PROTOCOL_MIN = 1;
@@ -4030,9 +4031,12 @@ export class DurablePrpControlPlane {
       this.#ackWarmTransition(connection, receipt);
       return;
     }
+    const turnStart = command.type === "turn.start";
+    if (turnStart) crashPoint("runner.turn_start.after_runner_accept_before_controller_checkpoint");
     command.status = status;
     command.result = structuredClone(result);
     this.#store.save();
+    if (turnStart) crashPoint("runner.turn_start.after_controller_journal_before_db_checkpoint");
     this.#ackTerminalCommandResult(connection, command);
     this.#releaseServingLifecycle(connection, command);
     if (!this.#isTerminalLifecycleCommand(command) || connection.terminalLifecycleCommandId === null) {

packages/paperclip-runner/src/crash-point.ts

@@ -0,0 +1,50 @@
+/**
+ * Runner-side crash points: the same `PAPERCLIP_CRASH_POINT` contract as
+ * `@paperclipai/shared/crash-points`, for the controller code in this package
+ * (it runs inside the Paperclip server process). This package does not depend
+ * on `@paperclipai/shared`, so it keeps this small copy; the names are checked
+ * against the shared registry by server/src/__tests__/crash-point-call-sites.test.ts.
+ *
+ * Disarmed (always under NODE_ENV=production) a call is one comparison.
+ * runnerd never sees the variable: its environment is an allowlist.
+ */
+import { appendFileSync, existsSync } from "node:fs";
+
+export type RunnerCrashPointName =
+  | "runner.process.after_spawn_before_identity"
+  | "runner.turn_start.after_runner_accept_before_controller_checkpoint"
+  | "runner.turn_start.after_controller_journal_before_db_checkpoint";
+
+type Armed = { name: string; nth: number; hits: number; receiptPath: string | null; gatePath: string | null };
+
+function armFromEnvironment(): Armed | null {
+  const env = typeof process === "undefined" ? undefined : process.env;
+  const raw = env?.PAPERCLIP_CRASH_POINT?.trim();
+  if (!env || !raw || env.NODE_ENV === "production") return null;
+  const separator = raw.lastIndexOf(":");
+  const name = separator === -1 ? raw : raw.slice(0, separator);
+  // The shared helper validates the name; this copy only arms its own names.
+  if (!name.startsWith("runner.")) return null;
+  const nth = separator === -1 ? 1 : Number(raw.slice(separator + 1));
+  if (!Number.isInteger(nth) || nth < 1) return null;
+  return { name, nth, hits: 0, receiptPath: env.PAPERCLIP_CRASH_POINT_RECEIPT?.trim() || null,
+    gatePath: env.PAPERCLIP_CRASH_POINT_GATE?.trim() || null };
+}
+
+const armed: Armed | null = armFromEnvironment();
+
+export function crashPoint(name: RunnerCrashPointName): void {
+  if (armed === null || armed.name !== name) return;
+  if (armed.gatePath !== null) {
+    if (!existsSync(armed.gatePath)) return;
+    armed.gatePath = null;
+  }
+  armed.hits += 1;
+  if (armed.hits !== armed.nth) return;
+  const receipt = { point: name, hit: armed.hits, pid: process.pid, at: new Date().toISOString() };
+  if (armed.receiptPath) {
+    try { appendFileSync(armed.receiptPath, `${JSON.stringify(receipt)}\n`); } catch { /* the log line still records it */ }
+  }
+  try { process.stderr.write(`[crash-point] SIGKILL ${JSON.stringify(receipt)}\n`); } catch { /* ignore */ }
+  process.kill(process.pid, "SIGKILL");
+}

packages/paperclip-runner/src/live/runnerd-codex-transport.ts

@@ -104,6 +104,7 @@ import {
 } from "../drivers/runtime-context-materializer.js";
 import { resolvePackagedRunnerBinary } from "./runner-binary.js";
 import { RUNNERD_CANONICAL_ITEM, RUNNERD_RUNTIME_ACTIVITY } from "../drivers/codex/codex-driver-values.js";
+import { crashPoint } from "../crash-point.js";
 
 // URL directory conversion preserves a trailing separator while path-derived
 // build artifacts do not. Normalize once so a source build cannot be
@@ -5990,6 +5991,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
     });
     this.#handle = handle;
     this.#watchRunner(handle);
+    // A remote launcher returns before runnerd exists; only a local spawn reaches this gap.
+    if (!this.options.runnerProcessLauncher) crashPoint("runner.process.after_spawn_before_identity");
     await this.#publishSpawnedProcess(handle);
     this.#assertStartupOpen();
     await registration?.activate?.();
@@ -6827,6 +6830,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
     if (handle) {
       this.#handle = handle;
       this.#watchRunner(handle);
+      // A remote launcher returns before runnerd exists; only a local spawn reaches this gap.
+      if (!this.options.runnerProcessLauncher) crashPoint("runner.process.after_spawn_before_identity");
       await this.#publishSpawnedProcess(handle);
     }
     this.#assertStartupOpen();

packages/shared/src/crash-point-registry.ts

@@ -45,8 +45,8 @@ export const CRASH_POINTS = {
     invariants: ["I2", "I5", "I7"],
   },
   "runner.process.after_spawn_before_identity": {
-    description: "runnerd is spawned (own process group, survives the server); its pid and process identity are not yet recorded on the run. Recovery must find or stop the orphan by identity.",
-    environments: ["local", "daytona"],
+    description: "runnerd is spawned (own process group, survives the server); its pid and process identity are not yet recorded on the run. Recovery must find or stop the orphan by identity. Local spawns only: a remote launcher returns before runnerd exists.",
+    environments: ["local"],
     kinds: ["native_executor_claim", "detached_process_marker"],
     invariants: ["I1", "I2", "I5"],
   },

server/src/__tests__/crash-point-call-sites.test.ts

@@ -0,0 +1,52 @@
+import { readdirSync, readFileSync } from "node:fs";
+import path from "node:path";
+import { describe, expect, it } from "vitest";
+import { CRASH_POINTS } from "@paperclipai/shared/crash-points";
+import { OWNABLE_KINDS } from "../services/recovery/ownership-registry.js";
+
+const repositoryRoot = path.resolve(import.meta.dirname, "../../..");
+const SOURCE_ROOTS = ["server/src", "packages/paperclip-runner/src"];
+const CALL = /\bcrashPoint\(\s*"([^"]+)"\s*\)/g;
+
+function sourceFiles(directory: string): string[] {
+  return readdirSync(directory, { withFileTypes: true }).flatMap(entry => {
+    const full = path.join(directory, entry.name);
+    if (entry.isDirectory()) return entry.name === "node_modules" || entry.name === "__tests__" ? [] : sourceFiles(full);
+    return /\.tsx?$/.test(entry.name) && !/\.test\.tsx?$/.test(entry.name) ? [full] : [];
+  });
+}
+
+function callSites(): Map<string, string[]> {
+  const sites = new Map<string, string[]>();
+  for (const root of SOURCE_ROOTS) {
+    for (const file of sourceFiles(path.join(repositoryRoot, root))) {
+      for (const match of readFileSync(file, "utf8").matchAll(CALL)) {
+        const list = sites.get(match[1]!) ?? [];
+        list.push(path.relative(repositoryRoot, file));
+        sites.set(match[1]!, list);
+      }
+    }
+  }
+  return sites;
+}
+
+describe("crash-point call sites", () => {
+  const sites = callSites();
+
+  it("instruments every registered crash point", () => {
+    const missing = Object.keys(CRASH_POINTS).filter(name => !sites.has(name));
+    expect(missing).toEqual([]);
+  });
+
+  it("calls only registered crash points", () => {
+    const unknown = [...sites.keys()].filter(name => !(name in CRASH_POINTS));
+    expect(unknown).toEqual([]);
+  });
+
+  it("names only ownable kinds of the Phase 1 registry", () => {
+    const known = new Set(OWNABLE_KINDS.map(kind => kind.id));
+    const unknown = Object.entries(CRASH_POINTS).flatMap(([name, definition]) =>
+      definition.kinds.filter(kind => !known.has(kind)).map(kind => `${name}: ${kind}`));
+    expect(unknown).toEqual([]);
+  });
+});

server/src/services/environment-runtime.ts

@@ -110,6 +110,7 @@ import {
   type SandboxOrphanCleanupSpool,
 } from "./sandbox-orphan-cleanup-spool.js";
 import { logger } from "../middleware/logger.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 // The constant error kind for the durable orphan-cleanup-write-failed log. The
 // log never reads the caught exception, because the exception can carry a
@@ -2286,6 +2287,7 @@ function createSandboxEnvironmentDriver(
       } catch {
         cleanupStatus = "failed";
       }
+      if (cleanupStatus === "success") crashPoint("environment.lease_release.after_provider_before_lease");
       const releaseStatus = input.lease.leasePolicy === "retain_on_failure" && input.status === "failed"
         ? "retained" as const
         : input.status;
@@ -3025,6 +3027,8 @@ function createSandboxEnvironmentDriver(
     } else {
       cleanupStatus = "failed";
     }
+    // Daytona and other plugin providers release here, not in the built-in path.
+    if (cleanupStatus === "success") crashPoint("environment.lease_release.after_provider_before_lease");
 
     // A failed release verification leaves the provider resource active. The
     // cleanup reaper retries only `pending_cleanup` leases, so route a failed
@@ -3112,6 +3116,8 @@ function createSandboxEnvironmentDriver(
     } catch {
       cleanupStatus = "failed";
     }
+    // Only a destroy the provider confirmed reaches this gap.
+    if (cleanupStatus === "success") crashPoint("environment.teardown.after_destroy_before_lease");
 
     return await environmentsSvc.releaseLease(
       input.lease.id,
@@ -3736,6 +3742,7 @@ export function environmentRuntimeService(
       )`,
     )).returning(), { companyId: input.leaseRow.companyId, runId: input.leaseRow.heartbeatRunId ?? undefined });
     if (!claimed) return null;
+    crashPoint("environment.teardown.after_claim_before_destroy");
     const lease = toEnvironmentLeaseSnapshot(claimed);
     let renewalInFlight = false;
     const renewal = setInterval(() => {
@@ -3972,6 +3979,7 @@ export function environmentRuntimeService(
               sql`${environmentLeases.metadata}->${intentKey} = ${JSON.stringify(leaseSnapshot.metadata?.[intentKey])}::jsonb`,
             )).returning(), { companyId: leaseRow.companyId, runId: leaseRow.heartbeatRunId ?? undefined });
             if (!claimed) continue;
+            crashPoint("environment.stop_retain.after_claim_before_stop");
             const snapshot = toEnvironmentLeaseSnapshot(claimed);
             try {
               const driver = requireDriverKey(getLeaseDriverKey(snapshot, environment));

server/src/services/heartbeat.ts

@@ -52,6 +52,7 @@ import { preserveNativeFailedWorkspace } from "./native-runtime/native-failed-wo
 import { nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js";
 import { prepareAndSelectHeartbeatRuntime } from "./heartbeat/runtime-selection.js";
 import { createHeartbeatRunCompletion } from "./heartbeat/run-completion.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 export {
   MAX_TURN_CONTINUATION_WAKE_REASON,
   parseSandboxProviderPluginNotReadyFailureMessage,
@@ -2947,6 +2948,7 @@ export function heartbeatService(
           reserveRunBudget(db, run.companyId, run.id,
             readNonEmptyString(runLedgerScope.projectId), runLedgerScope, runOptions.nativeLeaseOwner))
           .catch(async (error: unknown) => { throw await relinquished() ? new NativeControllerDetachedForRestartError() : error; });
+        crashPoint("budget.reservation.after_hold_before_dispatch");
         let entered = false, handedOff = false;
         try {
           return await dispatchResolvedInteractionContinuationAfterAdmission((markDispatchStarted) => {
@@ -3934,6 +3936,7 @@ export function heartbeatService(
             providerResourceDisposition: providerResourceDispositionForRun,
             nativeLifecycleTelemetry: nativeLifecycleTelemetryForRun,
           }));
+          crashPoint("environment.lease.after_release_before_runtime_services");
           await releaseRuntimeServicesForRun(run.id).catch(() => undefined);
           caseHandoffCleanupSettled = true;
         }

server/src/services/heartbeat/queue.ts

@@ -191,6 +191,7 @@ import { isBoardOwnedBudgetBlock } from "../budget-block-ownership.js";
 import type { instanceSettingsService } from "../instance-settings.js";
 import type { createRunDispatch } from "../../modules/run-dispatch/index.js";
 import { hasRetainedExternalChatRetry, type PostCommitEffect as WakeQueuePostCommitEffect, type createWakeQueue } from "../../modules/wake-queue/index.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
 type HeartbeatRunState = ReturnType<typeof createHeartbeatRunState>;
@@ -1550,6 +1551,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
     publishRunLifecyclePluginEvent(claimed);
 
     if (!nativeReviewContext) {
+      crashPoint("wake.claim.after_run_running_before_wakeup_claimed");
       await setWakeupStatus(claimed.wakeupRequestId, "claimed", { claimedAt });
     }
 

server/src/services/native-runtime/native-run-finalizer.ts

@@ -107,6 +107,7 @@ import {
 import {
   isNativeRunnerOwnershipHeld,
 } from "./native-runner-ownership.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 function record(value: unknown): Record<string, unknown> {
   return typeof value === "object" && value !== null && !Array.isArray(value)
@@ -1697,6 +1698,7 @@ export async function finalizeNativeRun(input: {
             : undefined,
         failpoint: input.failpoint,
       });
+      crashPoint("native.finalize.after_decision_before_projection");
       const now = new Date();
       const finalizationFailed = decision.effects.some(
         (effect) => effect.kind === "record_finalization_error",

server/src/services/native-runtime/native-session-executor.ts

@@ -24,6 +24,7 @@ import { rehydratePinnedSkillAssets } from "./runtime-context.js";
 import { createNativeUsageDelivery } from "./native-usage-delivery.js";
 import { createNativeRecoveryUsageDelivery } from "./native-recovery-usage-delivery.js";
 import { createNativeTurnAccounting, NATIVE_LOST_TURN_USAGE_EVIDENCE, nativeUsageCostUsd, nativeUsageBiller, normalizeNativeUsage, usesNativeUsageDelivery, isNativeUsageAccountingEvent } from "./native-usage-accounting.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 export { nativeUsageCostUsd, nativeUsageBiller, normalizeNativeUsage } from "./native-usage-accounting.js";
 import { measureRunnerOperation, recordRunnerTimingReceipt } from "./runner-operation-timing.js";
 import { formatNativeRunLogEvent } from "./native-run-log-export.js";
@@ -10930,6 +10931,7 @@ async function executePaperclipNativeSessionWithinScope(
                 // Binds a later identity-less stop proof to the host that launched.
                 payload: { hostname: hostname() },
               });
+              crashPoint("native.process_start.after_requested_before_spawn");
             },
             input: runnerExecution,
             startup: executionContext && (!usesPluginTask || executionContext.pluginSupported) ? executionContext : undefined,

server/src/services/native-runtime/paperclip-control-plane-port.ts

@@ -35,6 +35,7 @@ import { createNativeControllerRecovery, NATIVE_CONTROLLER_RECOVERY_MAX_BYTES, t
 import { eligibleNativeTaskSessionPublication, publishNativeTaskSessionInTransaction, NATIVE_TASK_SESSION_PUBLICATION_KEY,
   type NativeTaskSessionPublicationContext, type NativeTaskSessionPublicationIssue } from "./native-task-session-publication.js";
 import { lockNativeRunnerCheckpointOwner, nativeRunnerCheckpointOwnerLock, validateNativeRunnerCheckpointOwner, inspectNativeRunnerUpgradeHeader, type NativeRunnerCheckpointOwner } from "./native-runner-checkpoint.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 export interface PaperclipControlPlaneBinding {
   companyId: string;
@@ -632,6 +633,7 @@ export class PaperclipControlPlanePort implements ControlPlanePort {
       this.#assertActive(options);
       return true;
     });
+    if (insertedResult) crashPoint("native.result.after_accept_before_finalize");
     if (insertedResult) { try { await this.#onResultCommitted?.(); } catch {} }
   }
 }

server/src/services/run-cost-accounting.ts

@@ -21,6 +21,7 @@ import { priceAnthropicReceipt } from "./anthropic-pricing.js";
 import { priceCodexReceipt } from "./codex-pricing.js";
 import type { AdapterUsageCheckpoint } from "@paperclipai/adapter-utils";
 import { replayUsageReceipts, indexPendingUsageReceipts, readPendingRunUsageReceipts, persistUsageReceiptInTransaction, reconcileLateRunUsageReceiptInTransaction, type UsageReceiptIndex } from "./usage-receipts.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 const terminalStatuses = ["succeeded", "failed", "timed_out", "cancelled", "interrupted"];
 const object = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
@@ -422,6 +423,8 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
         timeoutMinutes: ACCOUNTING_CLOSURE_TIMEOUT_MS / 60_000 } }, publications);
     return true;
   }), { "budget.accounting.owner": "run" });
+  // Only a closed account reaches this gap; an open one returned false above.
+  if (accounted) crashPoint("accounting.closure.after_commit_before_spool_unlink");
   for (const file of recoveredFiles) {
     try { await fs.rm(file, { force: true }); }
     catch (error) { logger.warn({ err: error, runId }, "Accounting committed; recovered receipt cleanup will retry"); }

server/src/services/sandbox-creation-operations.ts

@@ -8,6 +8,7 @@ import { assertEnvironmentCompanyBinding, environmentService } from "./environme
 import { ENVIRONMENT_CLEANUP_RETRY_MS, recoverableSandboxCreationCondition } from "./environment-cleanup-state.js";
 import { logger } from "../middleware/logger.js";
 import { notifyEnvironmentCleanup, writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 export type SandboxCreationOperation = typeof sandboxCreationOperations.$inferSelect;
 
@@ -122,7 +123,9 @@ export async function createOwnedSandbox(
       metadata: { ...existing.metadata, creationOperationId: operation.id } };
   }
   try {
+    crashPoint("sandbox.create.after_intent_before_provider");
     const lease = await create({ id, runId });
+    crashPoint("sandbox.create.after_provider_before_lease");
     return { ...lease, metadata: { ...lease.metadata, creationOperationId: id } };
   } catch (error) {
     if (recoverableTaskCreation(operation)) {

server/src/services/usage-receipts.ts

@@ -22,6 +22,7 @@ import { applyCostCorrectionInTransaction } from "./accounting-corrections.js";
 import { logActivity, type ActivityPublication } from "./activity-log.js";
 import { createCostEventInTransaction } from "./costs.js";
 import { queueBudgetEvaluation } from "./budget-evaluation-work.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 const token = z.number().int().nonnegative().max(2_147_483_647);
 const price = z.number().finite().nonnegative().nullable().optional();
@@ -694,7 +695,12 @@ export async function createRunUsageRecorder(db: Db, input: { companyId: string;
     }
     const envelope: UsageReceiptEnvelope = { schema: "paperclip/accounting-receipt/v1", id: randomUUID(), ...input, sourceId, sequence: ++sequence, receivedAt: new Date().toISOString(), receipt, liveCostCents };
     const file = await spoolUsageReceipt(envelope, directory);
-    try { await persistUsageReceipt(db, envelope); await fs.rm(file, { force: true }); }
+    crashPoint("accounting.usage_receipt.after_spool_before_db");
+    try {
+      await persistUsageReceipt(db, envelope);
+      crashPoint("accounting.usage_receipt.after_db_before_unlink");
+      await fs.rm(file, { force: true });
+    }
     catch (error) {
       // The fsynced receipt is durable even when the DB transaction failed.
       // Wake the owned spool recovery path; never certify that failed write as free.

server/src/services/workspace-operations.ts

@@ -13,6 +13,7 @@ import { redactCurrentUserText, redactCurrentUserValue } from "../log-redaction.
 import { instanceSettingsService } from "./instance-settings.js";
 import { currentProcessSpaceHash, processExists, sameProcessSpace } from "./controller-process-identity.js";
 import { getWorkspaceOperationLogStore } from "./workspace-operation-log-store.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 type WorkspaceOperationRow = typeof workspaceOperations.$inferSelect;
 
@@ -585,7 +586,10 @@ export function workspaceOperationService(db: Db) {
             throw insertError;
           }
           createdIds.push(id);
-          if (recordInput.phase === "workspace_finalize") hasFinalization = true;
+          if (recordInput.phase === "workspace_finalize") {
+            hasFinalization = true;
+            crashPoint("workspace.finalize.after_start_before_copyback");
+          }
 
           let heartbeatTimer: NodeJS.Timeout | null = null;
           let timeoutTimer: NodeJS.Timeout | null = null;
@@ -667,6 +671,7 @@ export function workspaceOperationService(db: Db) {
 
           try {
             const result = await settle();
+            if (recordInput.phase === "workspace_finalize") crashPoint("workspace.finalize.after_copyback_before_done");
             await append("system", result.system ?? null);
             await append("stdout", result.stdout ?? null);
             await append("stderr", result.stderr ?? null);