Files
PaperClipAI/server/src/services/system-recovery-state.ts
T
DottaandClaude 1ac8b6a711 fix(workspaces): stop re-saving a retained sandbox whose disk is still full (#20758)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a native run on a remote sandbox, Paperclip saves the sandbox
workspace back to durable storage. When that save fails, saved-work
recovery saves the files again from the retained sandbox, with no
provider turn.
> - A staging census of the last 48 hours
(`doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`, class F)
found saves that failed because the sandbox disk was full.
> - #20678 already ends finalization at the first full-disk copy-back
and hands the work to saved-work recovery. Recovery then deletes
inactive tool caches that it can prove safe, and saves again.
> - When that save still found the disk full, recovery recorded a
transient `export_interrupted` failure with cause `provider_error`. It
then saved again on its automatic schedule (up to 5 retries in 30
minutes). Nothing frees space in a stopped, retained sandbox between
those attempts, so each attempt failed the same way. The task showed
"Recovering saved work…" and no Retry while this went on.
> - This pull request ends the recovery generation at the first such
failure with a named cause, `sandbox_disk_full`, and hands it to the
board with a Retry.
> - The benefit is that the task does not spend 30 minutes on saves that
cannot succeed, the user sees why, and one Retry saves the files and
starts one run.

## Linked Issues or Issue Description

Refs #20678, #20695 (named sandbox disk-full failures for copy-back and
restore)

**What happened?**
A native run on a remote sandbox finished, and its workspace copy-back
hit a full sandbox disk. Finalization handed the task to saved-work
recovery (#20678). Recovery's own save also hit the full disk. Recovery
classified this as `export_interrupted` / `retry_transient` with cause
`provider_error`, so it scheduled more automatic saves of the same
sandbox. Each one failed the same way. The task showed "Recovering saved
work…" with no Retry until the 30-minute window ran out.

**Expected behavior**
A save that still finds the disk full after the safe cache cleanup is
not repeated automatically. The recovery ends at once with the cause
`sandbox_disk_full`, and the board gets a Retry. The workspace keeps its
previous saved version, and the run's unsaved files stay in the retained
sandbox. One Retry saves the files once there is room and starts exactly
one run.

**Steps to reproduce**
1. Give a native run an accepted result whose copy-back failed
(`native_workspace_sync_out_retry_exhausted`).
2. Make every save from its retained sandbox fail with
`remote_workspace_sandbox_disk_full`, with no inactive tool cache to
delete.
3. Run saved-work recovery. Before this change, it records
`export_interrupted` with cause `provider_error` and schedules another
save.

## What Changed

- `workspace-recovery.ts`: a save that fails with
`remote_workspace_sandbox_disk_full` after a confirmed cache cleanup (or
with nothing to clean up) now fails the generation with
`export_interrupted`, retry class `user_action_required`, and cause `{
label: "sandbox_disk_full", code: "ENOSPC" }`. That class is not retried
automatically, and it is not a permanent-source class, so recovery never
falls back to the last save and never drops the run's unsaved files. The
next sweep hands the hold to the board.
- A cleanup whose outcome is unknown (lost acknowledgement, interrupted
journal) keeps the existing scheduled retry, because its journal resumes
on the next attempt and can still free space.
- `retained-provider-cache.ts`: `reclaimInactiveNativeProviderCache`
returns `{ changed, confirmed }` instead of a boolean, so the caller can
tell "nothing to delete" from "outcome unknown". It has one caller.
- `system-recovery-state.ts`: the board handoff text for this cause says
that the sandbox disk is full, that Paperclip deletes only inactive tool
caches it can prove safe, and that Retry saves the files again.
- New `workspace-recovery-disk-full.test.ts` covers the whole path: one
failed attempt, no further automatic saves, the board handoff text, the
previous saved version unchanged, the agent's files still in the
sandbox, no unfinished in-sandbox capture attempt, a Retry offered in
the task presentation and blocker, `POST /issues/:id/retry` accepted
(202), the files saved after Retry, and exactly one run started (also
with a replayed dispatch).
- `doc/execution-semantics.md`: documents the behavior.

## Verification

- Red on master: `cd server && npx vitest run
src/services/workspace-recovery-disk-full.test.ts` fails. The recovery
state is `scheduled` with cause `provider_error` instead of `exhausted`
with cause `sandbox_disk_full`.
- Green on this branch: the same command passes.
- Related suites pass: `workspace-recovery-retry-copyback.test.ts` (79
tests, including the 8 cache-cleanup resume tests that need the
scheduled retry after an unconfirmed cleanup),
`system-recovery-state.test.ts`,
`workspace-recovery-capture-policy.test.ts`,
`workspace-recovery-fallback.test.ts`,
`workspace-recovery-stop-proof.test.ts`,
`native-run-finalizer-telemetry.test.ts` (167 tests in all).
- `npx tsc --noEmit -p server/tsconfig.json` is clean.
- `pnpm verify`: exit 2 (broader verification required), because
`doc/execution-semantics.md` is a global CI input. The focused checks
above pass. Required CI supplies the broad coverage.

## Risks

- Low. The change touches only the saved-work recovery path when its
save fails with the exact full-disk message.
- Upgrade-safe: the failure code (`export_interrupted`), the retry class
(`user_action_required`) and the cause label all pass the earlier
release's recovery schema. An earlier release that reads this state
shows its generic "Retry to try again" text.
- A full-disk failure no longer gets automatic retries in its 30-minute
window. If space became free by itself in that window, the user must now
press Retry. Nothing in a stopped, retained sandbox frees space by
itself, so this is expected to be rare.
- No Runner (Rust) change, so no new sandbox image is needed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 01:31:50 -06:00

226 lines
14 KiB
TypeScript

import { createHash, randomUUID } from "node:crypto";
import { z } from "zod";
import { ENVIRONMENT_LEASE_POLICIES, WORKSPACE_SOURCE_VERIFICATION_REASONS,
type WorkspaceSourceVerificationFailure } from "@paperclipai/shared";
import { REMOTE_WORKSPACE_CAPTURE_POLICY_VERSION } from "./remote-workspace-snapshot-archive.js";
export const SYSTEM_RECOVERY_POLICY_VERSION = 1;
export const SYSTEM_RECOVERY_MAX_RETRIES = 5;
export const SYSTEM_RECOVERY_WINDOW_MS = 30 * 60_000;
export function infrastructureRetryDelayMs(retryNumber: number, sample: number) {
return Math.round(30_000 * 2 ** (retryNumber - 1) * (0.8 + Math.min(1, Math.max(0, sample)) * 0.4));
}
/** Assumed provider limit cooldown when the provider reports no reset time. */
export const PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS = 60 * 60_000;
export const recoveryFailureCodes = [
"historical_source_missing", "source_binding_invalid", "allocation_ownership_ambiguous",
"agent_stop_unconfirmed", "copyback_stop_unconfirmed", "workspace_writer_active",
"provider_unavailable", "export_interrupted", "archive_unsafe", "snapshot_verification_failed",
"source_unavailable", "effect_unresolved",
] as const;
export type RecoveryFailureCode = typeof recoveryFailureCodes[number];
export const recoveryRetryClasses = [
"retry_transient", "recheck_evidence", "use_alternative_strategy", "service_required", "user_action_required",
] as const;
export type RecoveryRetryClass = typeof recoveryRetryClasses[number];
export class SystemRecoveryFailure extends Error {
constructor(readonly code: RecoveryFailureCode, readonly retryClass: RecoveryRetryClass, message = code,
readonly sourceVerification?: WorkspaceSourceVerificationFailure) {
super(message);
}
}
export const workspaceSourceVerificationFailureSchema = z.object({
reason: z.enum(WORKSPACE_SOURCE_VERIFICATION_REASONS),
observedLeasePolicy: z.enum([...ENVIRONMENT_LEASE_POLICIES, "unrecognized"]).optional(),
});
/** Why an attempt failed, in closed labels only. Provider text never appears. */
export const recoveryFailureCauseSchema = z.object({
label: z.string().regex(/^[a-z][a-z0-9_]{0,47}$/),
code: z.string().regex(/^[A-Z][A-Z0-9_]{1,31}$/).nullable(),
status: z.number().int().min(100).max(599).nullable(),
});
export type RecoveryFailureCause = z.infer<typeof recoveryFailureCauseSchema>;
export const systemRecoverySchema = z.object({
schema: z.literal("paperclip.system-recovery.v1"),
policyVersion: z.literal(SYSTEM_RECOVERY_POLICY_VERSION),
sourceRunId: z.string().uuid(),
revision: z.number().int().nonnegative(),
generationId: z.string().uuid(),
intentId: z.string().uuid().nullable(),
phase: z.enum(["verify_execution", "preserve_files", "reconcile_effects", "settle_result_or_dispatch"]),
state: z.enum(["scheduled", "running", "awaiting_evidence", "exhausted", "complete"]),
attemptId: z.string().uuid().nullable(),
attempts: z.number().int().nonnegative(),
retriesUsed: z.number().int().min(0).max(SYSTEM_RECOVERY_MAX_RETRIES),
firstFailureAt: z.number().nullable(),
nextAttemptAt: z.number().nullable(),
deadlineAt: z.number().nullable(),
providerCooldownAt: z.number().nullable().default(null),
failure: z.object({
code: z.enum(recoveryFailureCodes),
retryClass: z.enum(recoveryRetryClasses),
evidenceFingerprint: z.string(),
sourceVerification: workspaceSourceVerificationFailureSchema.optional(),
cause: recoveryFailureCauseSchema.optional(),
}).nullable(),
preservationOperationIds: z.array(z.string().uuid()),
freshAttemptRunId: z.string().uuid().nullable(),
/** The workspace capture policy this generation saves with. Absent on
* generations written before the field existed; those used version 1. */
capturePolicyVersion: z.number().int().positive().optional(),
});
export type SystemRecoveryState = z.infer<typeof systemRecoverySchema>;
export function readSystemRecovery(evidence: Record<string, unknown> | null | undefined): SystemRecoveryState | null {
const parsed = systemRecoverySchema.safeParse(evidence?.systemRecovery);
return parsed.success ? parsed.data : null;
}
/** New generations require new intent or one-time policy adoption, never a sweep. */
export function createSystemRecovery(sourceRunId: string, intentId: string | null = null): SystemRecoveryState {
return {
schema: "paperclip.system-recovery.v1", policyVersion: SYSTEM_RECOVERY_POLICY_VERSION,
sourceRunId, revision: 0, generationId: intentId ?? randomUUID(), intentId,
phase: "verify_execution", state: "scheduled", attemptId: null, attempts: 0, retriesUsed: 0,
firstFailureAt: null, nextAttemptAt: null, deadlineAt: null, providerCooldownAt: null, failure: null,
preservationOperationIds: [], freshAttemptRunId: null,
capturePolicyVersion: REMOTE_WORKSPACE_CAPTURE_POLICY_VERSION,
};
}
export function reserveSystemRecoveryAttempt(state: SystemRecoveryState, now = Date.now()): SystemRecoveryState | null {
if (state.state === "complete" || state.state === "exhausted" ||
(state.nextAttemptAt !== null && state.nextAttemptAt > now)) return null;
// A reclaimed physical owner first reconciles the same interrupted attempt.
// Starting another call is not evidence that the prior call failed.
if (state.state === "running" && state.attemptId) return { ...state, revision: state.revision + 1 };
if ((state.deadlineAt !== null && now >= state.deadlineAt) ||
(state.attempts > 0 && state.retriesUsed >= SYSTEM_RECOVERY_MAX_RETRIES)) return null;
return { ...state, revision: state.revision + 1, state: "running", attemptId: randomUUID(),
attempts: state.attempts + 1, retriesUsed: state.retriesUsed + (state.attempts > 0 ? 1 : 0), nextAttemptAt: null };
}
/** The caller persists this timestamp. A restart must not recalculate jitter. */
export function failSystemRecovery(
state: SystemRecoveryState, failure: SystemRecoveryFailure,
options: { now?: number; retryAfterAt?: number; evidenceFingerprint?: string; cause?: RecoveryFailureCause } = {},
): SystemRecoveryState {
const now = options.now ?? Date.now();
const firstFailureAt = state.firstFailureAt ?? now;
const deadlineAt = state.deadlineAt ?? firstFailureAt + SYSTEM_RECOVERY_WINDOW_MS;
const retryNumber = state.retriesUsed + 1;
const entropy = createHash("sha256").update(`${state.generationId}:${retryNumber}`).digest().readUInt32BE(0) / 0xffffffff;
const delay = infrastructureRetryDelayMs(retryNumber, entropy);
const nextAttemptAt = Math.max(now + delay, options.retryAfterAt ?? 0);
const retryable = ["retry_transient", "recheck_evidence"].includes(failure.retryClass);
const retry = retryable && retryNumber <= SYSTEM_RECOVERY_MAX_RETRIES && nextAttemptAt < deadlineAt;
const sourceVerification = workspaceSourceVerificationFailureSchema.safeParse(failure.sourceVerification);
const cause = recoveryFailureCauseSchema.safeParse(options.cause);
return { ...state, revision: state.revision + 1, firstFailureAt, deadlineAt,
providerCooldownAt: options.retryAfterAt ?? null,
state: retry ? failure.retryClass === "recheck_evidence" ? "awaiting_evidence" : "scheduled" : "exhausted",
nextAttemptAt: retry ? nextAttemptAt : null,
failure: { code: failure.code, retryClass: failure.retryClass,
...(sourceVerification.success ? { sourceVerification: sourceVerification.data } : {}),
...(cause.success ? { cause: cause.data } : {}),
evidenceFingerprint: options.evidenceFingerprint ?? createHash("sha256").update(`${state.sourceRunId}:${failure.code}`).digest("hex") },
};
}
/**
* A save that failed while preserving files under an older workspace capture
* policy. The current policy may accept what the older one refused (for
* example a workspace with a symlink out of it), so recovery retries it once
* under the current policy (workspace-recovery-capture-policy). The new
* generation carries the current version, so the retry is bounded per policy.
*/
export const CAPTURE_POLICY_RECOVERY_KEY = "capturePolicyRecovery";
export const CAPTURE_POLICY_RECOVERY_SCHEMA = "paperclip.capture-policy-recovery.v1";
// Capture refusals surface as export_interrupted. archive_unsafe names a restore
// refusal, which the capture policy does not change, so it is not re-armed.
export const CAPTURE_POLICY_REARM_CODES = ["export_interrupted"] as const;
export function isCapturePolicyRearmable(state: Pick<SystemRecoveryState, "state" | "phase" | "failure" | "capturePolicyVersion"> | null | undefined) {
return state?.state === "exhausted" && state.phase === "preserve_files"
&& (CAPTURE_POLICY_REARM_CODES as readonly string[]).includes(state.failure?.code ?? "")
&& (state.capturePolicyVersion ?? 1) < REMOTE_WORKSPACE_CAPTURE_POLICY_VERSION;
}
/**
* An exhausted save whose last failure was an unproven agent stop is not a
* permanent refusal: once the source run has ended, a later attempt can prove
* the stop. Recovery re-arms it a bounded number of times on a backoff
* (workspace-recovery-stop-proof), then an export hold continues from the last
* save with the unsaved files attached, like a refused source.
*/
export const STOP_PROOF_RECOVERY_KEY = "stopProofRecovery";
export const STOP_PROOF_RECOVERY_SCHEMA = "paperclip.stop-proof-recovery.v1";
export const STOP_PROOF_MAX_REARMS = 4;
export const STOP_PROOF_WAIT_CODE = "agent_stop_unconfirmed";
/** Coordinator phases after which no finalizer or provider work runs under the coordinator. */
export const FINISHED_COORDINATOR_PHASES = ["committed", "terminal_failure", "applied"] as const;
export function readStopProofRecovery(evidence: Record<string, unknown> | null | undefined): { rearms: number; nextRearmAt: number | null } | null {
const marker = evidence?.[STOP_PROOF_RECOVERY_KEY] as Record<string, unknown> | undefined;
if (!marker || marker.schema !== STOP_PROOF_RECOVERY_SCHEMA || typeof marker.rearms !== "number") return null;
return { rearms: marker.rearms, nextRearmAt: typeof marker.nextRearmAt === "number" ? marker.nextRearmAt : null };
}
export function isExhaustedStopProofWait(state: Pick<SystemRecoveryState, "state" | "failure"> | null | undefined) {
return state?.state === "exhausted" && state.failure?.code === STOP_PROOF_WAIT_CODE
&& state.failure.retryClass === "recheck_evidence";
}
/** Every bounded re-arm of an unproven stop is spent. */
export function stopProofRearmsSpent(evidence: Record<string, unknown> | null | undefined) {
return (readStopProofRecovery(evidence)?.rearms ?? 0) >= STOP_PROOF_MAX_REARMS;
}
/** User-facing text for a spent recovery budget. It names what happens next and
* never asks a person to inspect a sandbox or environment. The code stays for support. */
export function exhaustedSystemRecoveryAction(state: SystemRecoveryState, evidence?: Record<string, unknown> | null): string {
// Another Retry cannot release a copyback owner that stopped without reporting.
if (state.failure?.code === "copyback_stop_unconfirmed")
return "Recovery of saved work is waiting for the previous server's file copy to stop. "
+ "Recovery resumes once Paperclip can confirm that it stopped. No provider turn is submitted.";
// An unproven agent stop is re-proven automatically once the run is over
// (workspace-recovery-stop-proof), while a bounded re-arm remains.
if (evidence !== undefined && isExhaustedStopProofWait(state) && !stopProofRearmsSpent(evidence))
return "Recovery of saved work is waiting to confirm that the previous run stopped. "
+ "Paperclip tries again automatically, and the agent will not repeat its work.";
const cause = state.failure?.cause ? `, cause: ${state.failure.cause.label}` : "";
if (state.failure?.cause?.label === "sandbox_disk_full")
return "Saving the agent's files stopped because the sandbox disk is full. Paperclip deletes only inactive tool "
+ "caches it can prove safe, and that did not free enough space. Automatic recovery does not repeat a save that fails the same way. "
+ "Retry saves the files again. The agent's result is kept, and the agent will not repeat its work.";
if (isDeterministicRecoveryRefusal(state)) {
const code = state.failure!.code;
// Never promise a release. A retained-file hold the fallback settles
// (workspace-recovery-fallback) replaces this text with its own outcome.
return `Paperclip refused to recover the saved files: ${recoveryRefusalReasons[code] ?? "a safety check on the saved files did not pass"} `
+ `(${code}${cause}). Automatic recovery does not repeat this check, because the same saved state gives the same result. `
+ "Retry runs the check again. The agent's result is saved, and the agent will not repeat its work.";
}
const reason = state.failure?.code ? ` (last failure: ${state.failure.code}${cause})` : "";
const outcome = state.attempts > 1 ? `stopped after ${state.attempts} attempts` : "failed";
return `Automatic recovery of saved work ${outcome}${reason}. `
+ "Retry to try again; the agent will not repeat its work.";
}
export function isDeterministicRecoveryRefusal(state: Pick<SystemRecoveryState, "state" | "failure"> | null | undefined) {
return state?.state === "exhausted" && state.failure?.retryClass === "service_required";
}
const recoveryRefusalReasons: Partial<Record<RecoveryFailureCode, string>> = {
allocation_ownership_ambiguous: "Paperclip cannot prove which runs used the sandbox that holds the saved files",
source_binding_invalid: "the recorded source of the saved files does not match the run that saved them",
snapshot_verification_failed: "the saved snapshot of the files did not pass verification",
archive_unsafe: "the saved file archive contains entries that are not safe to apply",
source_unavailable: "the saved files are no longer available where the run recorded them",
};