feat(recovery): instrument 20 durable write boundaries with crash points (#19938)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A control-plane restart between two durable writes leaves an
in-progress record owned by a dead process
> - #19937 added `crashPoint(name)` and a registry of 20 named
boundaries from the Phase 0 inventory
> - A name without a call site cannot fire, so the crash-point suite
cannot test it
> - This pull request puts one call at each registered boundary and adds
a test that keeps the registry and the call sites in step
> - The benefit is that the on-demand suite (next pull request) can kill
the control plane at every known write-ahead gap

## Linked Issues or Issue Description

Refs #19831 (plan, Phase 5). Refs #19846 (inventory, "Write-ahead
coverage"). Stacked on #19937 (helper and registry). Refs #19861
(ownable kind ids).

**Problem**
The registered crash points have no call sites.

**Expected behavior**
Each registered point is called at its gap: after the first durable
write or outside effect returns, before the next one starts. Disarmed
(always in production) each call is one comparison.

**Scope**
Call sites and a coverage test only. The suite runner and CI workflow
follow.

## What Changed

- Server call sites (`@paperclipai/shared/crash-points`):
- `sandbox-creation-operations.ts`: around the provider `create` call
(intent → provider → lease).
- `native-session-executor.ts`: after `native.process_start_requested`,
before spawn.
- `paperclip-control-plane-port.ts`: after the result-accept
transaction, before finalization starts (only when a result was
inserted).
- `native-run-finalizer.ts`: after `commitNativeStatusDecision`, before
the run projection.
- `workspace-operations.ts`: `workspace_finalize` only — after the
running row, and after `settle()` before the done row.
- `environment-runtime.ts`: provider release → lease row; teardown claim
→ destroy; reusable destroy → lease release; stop-and-retain claim →
stop.
- `heartbeat.ts`: lease release → runtime services; budget reservation
hold → dispatch.
- `heartbeat/queue.ts`: run claimed `running` → wake request `claimed`.
- `run-cost-accounting.ts`, `usage-receipts.ts`: accounting commit →
spool unlink; receipt spool → DB row → unlink.
- `packages/paperclip-runner/src/crash-point.ts`: a small copy of the
helper for controller code in this package, which does not depend on
`@paperclipai/shared` (adding that dependency would change the runner
content ID). It arms only `runner.*` names. runnerd never sees the
variable: its environment is an allowlist.
- Runner call sites: after the runnerd spawn, before its identity is
published (both spawn paths); `turn.start` result received → controller
journal saved → next send.
- `server/src/__tests__/crash-point-call-sites.test.ts`: every
registered point has a call site, every call names a registered point,
and every `kinds` entry is an ownable kind id of the Phase 1 registry.

## Verification

- `cd server && npx vitest run
src/__tests__/crash-point-call-sites.test.ts` → 3 passed.
- `pnpm --filter @paperclipai/server typecheck` → clean. `pnpm --filter
@paperclipai/paperclip-runner exec tsc --noEmit -p .` → clean.
- Behavior with nothing armed is unchanged: each call returns at its
first comparison (measured in #19937: about 3.7 ns).

## Risks

- Low risk. No call changes control flow unless `PAPERCLIP_CRASH_POINT`
names it and `NODE_ENV` is not `production`.
- Points on the path before `turn.start` (process start, runner spawn)
cost nanoseconds, within plan invariant 8.
- `usage-receipts.ts`: the persist-then-unlink one-liner became a block
with the same statements in the same order.
- The runner package change gives the runner a new content ID, as any
runner source change does. No persisted run state changes
(rules/upgrade-safe-run-state.mdc).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits) and a read-only exploration subagent that
mapped the boundaries.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/environment-runtime.ts

@@ -110,6 +110,7 @@ import {
   type SandboxOrphanCleanupSpool,
 } from "./sandbox-orphan-cleanup-spool.js";
 import { logger } from "../middleware/logger.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 // The constant error kind for the durable orphan-cleanup-write-failed log. The
 // log never reads the caught exception, because the exception can carry a
@@ -2286,6 +2287,7 @@ function createSandboxEnvironmentDriver(
       } catch {
         cleanupStatus = "failed";
       }
+      if (cleanupStatus === "success") crashPoint("environment.lease_release.after_provider_before_lease");
       const releaseStatus = input.lease.leasePolicy === "retain_on_failure" && input.status === "failed"
         ? "retained" as const
         : input.status;
@@ -3025,6 +3027,8 @@ function createSandboxEnvironmentDriver(
     } else {
       cleanupStatus = "failed";
     }
+    // Daytona and other plugin providers release here, not in the built-in path.
+    if (cleanupStatus === "success") crashPoint("environment.lease_release.after_provider_before_lease");
 
     // A failed release verification leaves the provider resource active. The
     // cleanup reaper retries only `pending_cleanup` leases, so route a failed
@@ -3112,6 +3116,8 @@ function createSandboxEnvironmentDriver(
     } catch {
       cleanupStatus = "failed";
     }
+    // Only a destroy the provider confirmed reaches this gap.
+    if (cleanupStatus === "success") crashPoint("environment.teardown.after_destroy_before_lease");
 
     return await environmentsSvc.releaseLease(
       input.lease.id,
@@ -3736,6 +3742,7 @@ export function environmentRuntimeService(
       )`,
     )).returning(), { companyId: input.leaseRow.companyId, runId: input.leaseRow.heartbeatRunId ?? undefined });
     if (!claimed) return null;
+    crashPoint("environment.teardown.after_claim_before_destroy");
     const lease = toEnvironmentLeaseSnapshot(claimed);
     let renewalInFlight = false;
     const renewal = setInterval(() => {
@@ -3972,6 +3979,7 @@ export function environmentRuntimeService(
               sql`${environmentLeases.metadata}->${intentKey} = ${JSON.stringify(leaseSnapshot.metadata?.[intentKey])}::jsonb`,
             )).returning(), { companyId: leaseRow.companyId, runId: leaseRow.heartbeatRunId ?? undefined });
             if (!claimed) continue;
+            crashPoint("environment.stop_retain.after_claim_before_stop");
             const snapshot = toEnvironmentLeaseSnapshot(claimed);
             try {
               const driver = requireDriverKey(getLeaseDriverKey(snapshot, environment));

server/src/services/heartbeat.ts

@@ -52,6 +52,7 @@ import { preserveNativeFailedWorkspace } from "./native-runtime/native-failed-wo
 import { nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js";
 import { prepareAndSelectHeartbeatRuntime } from "./heartbeat/runtime-selection.js";
 import { createHeartbeatRunCompletion } from "./heartbeat/run-completion.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 export {
   MAX_TURN_CONTINUATION_WAKE_REASON,
   parseSandboxProviderPluginNotReadyFailureMessage,
@@ -2947,6 +2948,7 @@ export function heartbeatService(
           reserveRunBudget(db, run.companyId, run.id,
             readNonEmptyString(runLedgerScope.projectId), runLedgerScope, runOptions.nativeLeaseOwner))
           .catch(async (error: unknown) => { throw await relinquished() ? new NativeControllerDetachedForRestartError() : error; });
+        crashPoint("budget.reservation.after_hold_before_dispatch");
         let entered = false, handedOff = false;
         try {
           return await dispatchResolvedInteractionContinuationAfterAdmission((markDispatchStarted) => {
@@ -3934,6 +3936,7 @@ export function heartbeatService(
             providerResourceDisposition: providerResourceDispositionForRun,
             nativeLifecycleTelemetry: nativeLifecycleTelemetryForRun,
           }));
+          crashPoint("environment.lease.after_release_before_runtime_services");
           await releaseRuntimeServicesForRun(run.id).catch(() => undefined);
           caseHandoffCleanupSettled = true;
         }

server/src/services/heartbeat/queue.ts

@@ -191,6 +191,7 @@ import { isBoardOwnedBudgetBlock } from "../budget-block-ownership.js";
 import type { instanceSettingsService } from "../instance-settings.js";
 import type { createRunDispatch } from "../../modules/run-dispatch/index.js";
 import { hasRetainedExternalChatRetry, type PostCommitEffect as WakeQueuePostCommitEffect, type createWakeQueue } from "../../modules/wake-queue/index.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
 type HeartbeatRunState = ReturnType<typeof createHeartbeatRunState>;
@@ -1550,6 +1551,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
     publishRunLifecyclePluginEvent(claimed);
 
     if (!nativeReviewContext) {
+      crashPoint("wake.claim.after_run_running_before_wakeup_claimed");
       await setWakeupStatus(claimed.wakeupRequestId, "claimed", { claimedAt });
     }
 

server/src/services/native-runtime/native-run-finalizer.ts

@@ -107,6 +107,7 @@ import {
 import {
   isNativeRunnerOwnershipHeld,
 } from "./native-runner-ownership.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 function record(value: unknown): Record<string, unknown> {
   return typeof value === "object" && value !== null && !Array.isArray(value)
@@ -1697,6 +1698,7 @@ export async function finalizeNativeRun(input: {
             : undefined,
         failpoint: input.failpoint,
       });
+      crashPoint("native.finalize.after_decision_before_projection");
       const now = new Date();
       const finalizationFailed = decision.effects.some(
         (effect) => effect.kind === "record_finalization_error",

server/src/services/native-runtime/native-session-executor.ts

@@ -24,6 +24,7 @@ import { rehydratePinnedSkillAssets } from "./runtime-context.js";
 import { createNativeUsageDelivery } from "./native-usage-delivery.js";
 import { createNativeRecoveryUsageDelivery } from "./native-recovery-usage-delivery.js";
 import { createNativeTurnAccounting, NATIVE_LOST_TURN_USAGE_EVIDENCE, nativeUsageCostUsd, nativeUsageBiller, normalizeNativeUsage, usesNativeUsageDelivery, isNativeUsageAccountingEvent } from "./native-usage-accounting.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 export { nativeUsageCostUsd, nativeUsageBiller, normalizeNativeUsage } from "./native-usage-accounting.js";
 import { measureRunnerOperation, recordRunnerTimingReceipt } from "./runner-operation-timing.js";
 import { formatNativeRunLogEvent } from "./native-run-log-export.js";
@@ -10930,6 +10931,7 @@ async function executePaperclipNativeSessionWithinScope(
                 // Binds a later identity-less stop proof to the host that launched.
                 payload: { hostname: hostname() },
               });
+              crashPoint("native.process_start.after_requested_before_spawn");
             },
             input: runnerExecution,
             startup: executionContext && (!usesPluginTask || executionContext.pluginSupported) ? executionContext : undefined,

server/src/services/native-runtime/paperclip-control-plane-port.ts

@@ -35,6 +35,7 @@ import { createNativeControllerRecovery, NATIVE_CONTROLLER_RECOVERY_MAX_BYTES, t
 import { eligibleNativeTaskSessionPublication, publishNativeTaskSessionInTransaction, NATIVE_TASK_SESSION_PUBLICATION_KEY,
   type NativeTaskSessionPublicationContext, type NativeTaskSessionPublicationIssue } from "./native-task-session-publication.js";
 import { lockNativeRunnerCheckpointOwner, nativeRunnerCheckpointOwnerLock, validateNativeRunnerCheckpointOwner, inspectNativeRunnerUpgradeHeader, type NativeRunnerCheckpointOwner } from "./native-runner-checkpoint.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 export interface PaperclipControlPlaneBinding {
   companyId: string;
@@ -632,6 +633,7 @@ export class PaperclipControlPlanePort implements ControlPlanePort {
       this.#assertActive(options);
       return true;
     });
+    if (insertedResult) crashPoint("native.result.after_accept_before_finalize");
     if (insertedResult) { try { await this.#onResultCommitted?.(); } catch {} }
   }
 }

server/src/services/run-cost-accounting.ts

@@ -21,6 +21,7 @@ import { priceAnthropicReceipt } from "./anthropic-pricing.js";
 import { priceCodexReceipt } from "./codex-pricing.js";
 import type { AdapterUsageCheckpoint } from "@paperclipai/adapter-utils";
 import { replayUsageReceipts, indexPendingUsageReceipts, readPendingRunUsageReceipts, persistUsageReceiptInTransaction, reconcileLateRunUsageReceiptInTransaction, type UsageReceiptIndex } from "./usage-receipts.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 const terminalStatuses = ["succeeded", "failed", "timed_out", "cancelled", "interrupted"];
 const object = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
@@ -422,6 +423,8 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
         timeoutMinutes: ACCOUNTING_CLOSURE_TIMEOUT_MS / 60_000 } }, publications);
     return true;
   }), { "budget.accounting.owner": "run" });
+  // Only a closed account reaches this gap; an open one returned false above.
+  if (accounted) crashPoint("accounting.closure.after_commit_before_spool_unlink");
   for (const file of recoveredFiles) {
     try { await fs.rm(file, { force: true }); }
     catch (error) { logger.warn({ err: error, runId }, "Accounting committed; recovered receipt cleanup will retry"); }

server/src/services/sandbox-creation-operations.ts

@@ -8,6 +8,7 @@ import { assertEnvironmentCompanyBinding, environmentService } from "./environme
 import { ENVIRONMENT_CLEANUP_RETRY_MS, recoverableSandboxCreationCondition } from "./environment-cleanup-state.js";
 import { logger } from "../middleware/logger.js";
 import { notifyEnvironmentCleanup, writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 export type SandboxCreationOperation = typeof sandboxCreationOperations.$inferSelect;
 
@@ -122,7 +123,9 @@ export async function createOwnedSandbox(
       metadata: { ...existing.metadata, creationOperationId: operation.id } };
   }
   try {
+    crashPoint("sandbox.create.after_intent_before_provider");
     const lease = await create({ id, runId });
+    crashPoint("sandbox.create.after_provider_before_lease");
     return { ...lease, metadata: { ...lease.metadata, creationOperationId: id } };
   } catch (error) {
     if (recoverableTaskCreation(operation)) {

server/src/services/usage-receipts.ts

@@ -22,6 +22,7 @@ import { applyCostCorrectionInTransaction } from "./accounting-corrections.js";
 import { logActivity, type ActivityPublication } from "./activity-log.js";
 import { createCostEventInTransaction } from "./costs.js";
 import { queueBudgetEvaluation } from "./budget-evaluation-work.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 const token = z.number().int().nonnegative().max(2_147_483_647);
 const price = z.number().finite().nonnegative().nullable().optional();
@@ -694,7 +695,12 @@ export async function createRunUsageRecorder(db: Db, input: { companyId: string;
     }
     const envelope: UsageReceiptEnvelope = { schema: "paperclip/accounting-receipt/v1", id: randomUUID(), ...input, sourceId, sequence: ++sequence, receivedAt: new Date().toISOString(), receipt, liveCostCents };
     const file = await spoolUsageReceipt(envelope, directory);
-    try { await persistUsageReceipt(db, envelope); await fs.rm(file, { force: true }); }
+    crashPoint("accounting.usage_receipt.after_spool_before_db");
+    try {
+      await persistUsageReceipt(db, envelope);
+      crashPoint("accounting.usage_receipt.after_db_before_unlink");
+      await fs.rm(file, { force: true });
+    }
     catch (error) {
       // The fsynced receipt is durable even when the DB transaction failed.
       // Wake the owned spool recovery path; never certify that failed write as free.

server/src/services/workspace-operations.ts

@@ -13,6 +13,7 @@ import { redactCurrentUserText, redactCurrentUserValue } from "../log-redaction.
 import { instanceSettingsService } from "./instance-settings.js";
 import { currentProcessSpaceHash, processExists, sameProcessSpace } from "./controller-process-identity.js";
 import { getWorkspaceOperationLogStore } from "./workspace-operation-log-store.js";
+import { crashPoint } from "@paperclipai/shared/crash-points";
 
 type WorkspaceOperationRow = typeof workspaceOperations.$inferSelect;
 
@@ -585,7 +586,10 @@ export function workspaceOperationService(db: Db) {
             throw insertError;
           }
           createdIds.push(id);
-          if (recordInput.phase === "workspace_finalize") hasFinalization = true;
+          if (recordInput.phase === "workspace_finalize") {
+            hasFinalization = true;
+            crashPoint("workspace.finalize.after_start_before_copyback");
+          }
 
           let heartbeatTimer: NodeJS.Timeout | null = null;
           let timeoutTimer: NodeJS.Timeout | null = null;
@@ -667,6 +671,7 @@ export function workspaceOperationService(db: Db) {
 
           try {
             const result = await settle();
+            if (recordInput.phase === "workspace_finalize") crashPoint("workspace.finalize.after_copyback_before_done");
             await append("system", result.system ?? null);
             await append("stdout", result.stdout ?? null);
             await append("stderr", result.stderr ?? null);