mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a native run on a remote sandbox, Paperclip saves the sandbox workspace back to durable storage. When that save fails, saved-work recovery saves the files again from the retained sandbox, with no provider turn. > - A staging census of the last 48 hours (`doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`, class F) found saves that failed because the sandbox disk was full. > - #20678 already ends finalization at the first full-disk copy-back and hands the work to saved-work recovery. Recovery then deletes inactive tool caches that it can prove safe, and saves again. > - When that save still found the disk full, recovery recorded a transient `export_interrupted` failure with cause `provider_error`. It then saved again on its automatic schedule (up to 5 retries in 30 minutes). Nothing frees space in a stopped, retained sandbox between those attempts, so each attempt failed the same way. The task showed "Recovering saved work…" and no Retry while this went on. > - This pull request ends the recovery generation at the first such failure with a named cause, `sandbox_disk_full`, and hands it to the board with a Retry. > - The benefit is that the task does not spend 30 minutes on saves that cannot succeed, the user sees why, and one Retry saves the files and starts one run. ## Linked Issues or Issue Description Refs #20678, #20695 (named sandbox disk-full failures for copy-back and restore) **What happened?** A native run on a remote sandbox finished, and its workspace copy-back hit a full sandbox disk. Finalization handed the task to saved-work recovery (#20678). Recovery's own save also hit the full disk. Recovery classified this as `export_interrupted` / `retry_transient` with cause `provider_error`, so it scheduled more automatic saves of the same sandbox. Each one failed the same way. The task showed "Recovering saved work…" with no Retry until the 30-minute window ran out. **Expected behavior** A save that still finds the disk full after the safe cache cleanup is not repeated automatically. The recovery ends at once with the cause `sandbox_disk_full`, and the board gets a Retry. The workspace keeps its previous saved version, and the run's unsaved files stay in the retained sandbox. One Retry saves the files once there is room and starts exactly one run. **Steps to reproduce** 1. Give a native run an accepted result whose copy-back failed (`native_workspace_sync_out_retry_exhausted`). 2. Make every save from its retained sandbox fail with `remote_workspace_sandbox_disk_full`, with no inactive tool cache to delete. 3. Run saved-work recovery. Before this change, it records `export_interrupted` with cause `provider_error` and schedules another save. ## What Changed - `workspace-recovery.ts`: a save that fails with `remote_workspace_sandbox_disk_full` after a confirmed cache cleanup (or with nothing to clean up) now fails the generation with `export_interrupted`, retry class `user_action_required`, and cause `{ label: "sandbox_disk_full", code: "ENOSPC" }`. That class is not retried automatically, and it is not a permanent-source class, so recovery never falls back to the last save and never drops the run's unsaved files. The next sweep hands the hold to the board. - A cleanup whose outcome is unknown (lost acknowledgement, interrupted journal) keeps the existing scheduled retry, because its journal resumes on the next attempt and can still free space. - `retained-provider-cache.ts`: `reclaimInactiveNativeProviderCache` returns `{ changed, confirmed }` instead of a boolean, so the caller can tell "nothing to delete" from "outcome unknown". It has one caller. - `system-recovery-state.ts`: the board handoff text for this cause says that the sandbox disk is full, that Paperclip deletes only inactive tool caches it can prove safe, and that Retry saves the files again. - New `workspace-recovery-disk-full.test.ts` covers the whole path: one failed attempt, no further automatic saves, the board handoff text, the previous saved version unchanged, the agent's files still in the sandbox, no unfinished in-sandbox capture attempt, a Retry offered in the task presentation and blocker, `POST /issues/:id/retry` accepted (202), the files saved after Retry, and exactly one run started (also with a replayed dispatch). - `doc/execution-semantics.md`: documents the behavior. ## Verification - Red on master: `cd server && npx vitest run src/services/workspace-recovery-disk-full.test.ts` fails. The recovery state is `scheduled` with cause `provider_error` instead of `exhausted` with cause `sandbox_disk_full`. - Green on this branch: the same command passes. - Related suites pass: `workspace-recovery-retry-copyback.test.ts` (79 tests, including the 8 cache-cleanup resume tests that need the scheduled retry after an unconfirmed cleanup), `system-recovery-state.test.ts`, `workspace-recovery-capture-policy.test.ts`, `workspace-recovery-fallback.test.ts`, `workspace-recovery-stop-proof.test.ts`, `native-run-finalizer-telemetry.test.ts` (167 tests in all). - `npx tsc --noEmit -p server/tsconfig.json` is clean. - `pnpm verify`: exit 2 (broader verification required), because `doc/execution-semantics.md` is a global CI input. The focused checks above pass. Required CI supplies the broad coverage. ## Risks - Low. The change touches only the saved-work recovery path when its save fails with the exact full-disk message. - Upgrade-safe: the failure code (`export_interrupted`), the retry class (`user_action_required`) and the cause label all pass the earlier release's recovery schema. An earlier release that reads this state shows its generic "Retry to try again" text. - A full-disk failure no longer gets automatic retries in its 30-minute window. If space became free by itself in that window, the user must now press Retry. Nothing in a stopped, retained sandbox frees space by itself, so this is expected to be rare. - No Runner (Rust) change, so no new sandbox image is needed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing, test execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
226 lines
14 KiB
TypeScript
226 lines
14 KiB
TypeScript
import { createHash, randomUUID } from "node:crypto";
|
|
import { z } from "zod";
|
|
import { ENVIRONMENT_LEASE_POLICIES, WORKSPACE_SOURCE_VERIFICATION_REASONS,
|
|
type WorkspaceSourceVerificationFailure } from "@paperclipai/shared";
|
|
import { REMOTE_WORKSPACE_CAPTURE_POLICY_VERSION } from "./remote-workspace-snapshot-archive.js";
|
|
|
|
export const SYSTEM_RECOVERY_POLICY_VERSION = 1;
|
|
export const SYSTEM_RECOVERY_MAX_RETRIES = 5;
|
|
export const SYSTEM_RECOVERY_WINDOW_MS = 30 * 60_000;
|
|
export function infrastructureRetryDelayMs(retryNumber: number, sample: number) {
|
|
return Math.round(30_000 * 2 ** (retryNumber - 1) * (0.8 + Math.min(1, Math.max(0, sample)) * 0.4));
|
|
}
|
|
/** Assumed provider limit cooldown when the provider reports no reset time. */
|
|
export const PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS = 60 * 60_000;
|
|
|
|
export const recoveryFailureCodes = [
|
|
"historical_source_missing", "source_binding_invalid", "allocation_ownership_ambiguous",
|
|
"agent_stop_unconfirmed", "copyback_stop_unconfirmed", "workspace_writer_active",
|
|
"provider_unavailable", "export_interrupted", "archive_unsafe", "snapshot_verification_failed",
|
|
"source_unavailable", "effect_unresolved",
|
|
] as const;
|
|
export type RecoveryFailureCode = typeof recoveryFailureCodes[number];
|
|
export const recoveryRetryClasses = [
|
|
"retry_transient", "recheck_evidence", "use_alternative_strategy", "service_required", "user_action_required",
|
|
] as const;
|
|
export type RecoveryRetryClass = typeof recoveryRetryClasses[number];
|
|
|
|
export class SystemRecoveryFailure extends Error {
|
|
constructor(readonly code: RecoveryFailureCode, readonly retryClass: RecoveryRetryClass, message = code,
|
|
readonly sourceVerification?: WorkspaceSourceVerificationFailure) {
|
|
super(message);
|
|
}
|
|
}
|
|
|
|
export const workspaceSourceVerificationFailureSchema = z.object({
|
|
reason: z.enum(WORKSPACE_SOURCE_VERIFICATION_REASONS),
|
|
observedLeasePolicy: z.enum([...ENVIRONMENT_LEASE_POLICIES, "unrecognized"]).optional(),
|
|
});
|
|
|
|
/** Why an attempt failed, in closed labels only. Provider text never appears. */
|
|
export const recoveryFailureCauseSchema = z.object({
|
|
label: z.string().regex(/^[a-z][a-z0-9_]{0,47}$/),
|
|
code: z.string().regex(/^[A-Z][A-Z0-9_]{1,31}$/).nullable(),
|
|
status: z.number().int().min(100).max(599).nullable(),
|
|
});
|
|
export type RecoveryFailureCause = z.infer<typeof recoveryFailureCauseSchema>;
|
|
|
|
export const systemRecoverySchema = z.object({
|
|
schema: z.literal("paperclip.system-recovery.v1"),
|
|
policyVersion: z.literal(SYSTEM_RECOVERY_POLICY_VERSION),
|
|
sourceRunId: z.string().uuid(),
|
|
revision: z.number().int().nonnegative(),
|
|
generationId: z.string().uuid(),
|
|
intentId: z.string().uuid().nullable(),
|
|
phase: z.enum(["verify_execution", "preserve_files", "reconcile_effects", "settle_result_or_dispatch"]),
|
|
state: z.enum(["scheduled", "running", "awaiting_evidence", "exhausted", "complete"]),
|
|
attemptId: z.string().uuid().nullable(),
|
|
attempts: z.number().int().nonnegative(),
|
|
retriesUsed: z.number().int().min(0).max(SYSTEM_RECOVERY_MAX_RETRIES),
|
|
firstFailureAt: z.number().nullable(),
|
|
nextAttemptAt: z.number().nullable(),
|
|
deadlineAt: z.number().nullable(),
|
|
providerCooldownAt: z.number().nullable().default(null),
|
|
failure: z.object({
|
|
code: z.enum(recoveryFailureCodes),
|
|
retryClass: z.enum(recoveryRetryClasses),
|
|
evidenceFingerprint: z.string(),
|
|
sourceVerification: workspaceSourceVerificationFailureSchema.optional(),
|
|
cause: recoveryFailureCauseSchema.optional(),
|
|
}).nullable(),
|
|
preservationOperationIds: z.array(z.string().uuid()),
|
|
freshAttemptRunId: z.string().uuid().nullable(),
|
|
/** The workspace capture policy this generation saves with. Absent on
|
|
* generations written before the field existed; those used version 1. */
|
|
capturePolicyVersion: z.number().int().positive().optional(),
|
|
});
|
|
export type SystemRecoveryState = z.infer<typeof systemRecoverySchema>;
|
|
|
|
export function readSystemRecovery(evidence: Record<string, unknown> | null | undefined): SystemRecoveryState | null {
|
|
const parsed = systemRecoverySchema.safeParse(evidence?.systemRecovery);
|
|
return parsed.success ? parsed.data : null;
|
|
}
|
|
|
|
/** New generations require new intent or one-time policy adoption, never a sweep. */
|
|
export function createSystemRecovery(sourceRunId: string, intentId: string | null = null): SystemRecoveryState {
|
|
return {
|
|
schema: "paperclip.system-recovery.v1", policyVersion: SYSTEM_RECOVERY_POLICY_VERSION,
|
|
sourceRunId, revision: 0, generationId: intentId ?? randomUUID(), intentId,
|
|
phase: "verify_execution", state: "scheduled", attemptId: null, attempts: 0, retriesUsed: 0,
|
|
firstFailureAt: null, nextAttemptAt: null, deadlineAt: null, providerCooldownAt: null, failure: null,
|
|
preservationOperationIds: [], freshAttemptRunId: null,
|
|
capturePolicyVersion: REMOTE_WORKSPACE_CAPTURE_POLICY_VERSION,
|
|
};
|
|
}
|
|
|
|
export function reserveSystemRecoveryAttempt(state: SystemRecoveryState, now = Date.now()): SystemRecoveryState | null {
|
|
if (state.state === "complete" || state.state === "exhausted" ||
|
|
(state.nextAttemptAt !== null && state.nextAttemptAt > now)) return null;
|
|
// A reclaimed physical owner first reconciles the same interrupted attempt.
|
|
// Starting another call is not evidence that the prior call failed.
|
|
if (state.state === "running" && state.attemptId) return { ...state, revision: state.revision + 1 };
|
|
if ((state.deadlineAt !== null && now >= state.deadlineAt) ||
|
|
(state.attempts > 0 && state.retriesUsed >= SYSTEM_RECOVERY_MAX_RETRIES)) return null;
|
|
return { ...state, revision: state.revision + 1, state: "running", attemptId: randomUUID(),
|
|
attempts: state.attempts + 1, retriesUsed: state.retriesUsed + (state.attempts > 0 ? 1 : 0), nextAttemptAt: null };
|
|
}
|
|
|
|
/** The caller persists this timestamp. A restart must not recalculate jitter. */
|
|
export function failSystemRecovery(
|
|
state: SystemRecoveryState, failure: SystemRecoveryFailure,
|
|
options: { now?: number; retryAfterAt?: number; evidenceFingerprint?: string; cause?: RecoveryFailureCause } = {},
|
|
): SystemRecoveryState {
|
|
const now = options.now ?? Date.now();
|
|
const firstFailureAt = state.firstFailureAt ?? now;
|
|
const deadlineAt = state.deadlineAt ?? firstFailureAt + SYSTEM_RECOVERY_WINDOW_MS;
|
|
const retryNumber = state.retriesUsed + 1;
|
|
const entropy = createHash("sha256").update(`${state.generationId}:${retryNumber}`).digest().readUInt32BE(0) / 0xffffffff;
|
|
const delay = infrastructureRetryDelayMs(retryNumber, entropy);
|
|
const nextAttemptAt = Math.max(now + delay, options.retryAfterAt ?? 0);
|
|
const retryable = ["retry_transient", "recheck_evidence"].includes(failure.retryClass);
|
|
const retry = retryable && retryNumber <= SYSTEM_RECOVERY_MAX_RETRIES && nextAttemptAt < deadlineAt;
|
|
const sourceVerification = workspaceSourceVerificationFailureSchema.safeParse(failure.sourceVerification);
|
|
const cause = recoveryFailureCauseSchema.safeParse(options.cause);
|
|
return { ...state, revision: state.revision + 1, firstFailureAt, deadlineAt,
|
|
providerCooldownAt: options.retryAfterAt ?? null,
|
|
state: retry ? failure.retryClass === "recheck_evidence" ? "awaiting_evidence" : "scheduled" : "exhausted",
|
|
nextAttemptAt: retry ? nextAttemptAt : null,
|
|
failure: { code: failure.code, retryClass: failure.retryClass,
|
|
...(sourceVerification.success ? { sourceVerification: sourceVerification.data } : {}),
|
|
...(cause.success ? { cause: cause.data } : {}),
|
|
evidenceFingerprint: options.evidenceFingerprint ?? createHash("sha256").update(`${state.sourceRunId}:${failure.code}`).digest("hex") },
|
|
};
|
|
}
|
|
|
|
/**
|
|
* A save that failed while preserving files under an older workspace capture
|
|
* policy. The current policy may accept what the older one refused (for
|
|
* example a workspace with a symlink out of it), so recovery retries it once
|
|
* under the current policy (workspace-recovery-capture-policy). The new
|
|
* generation carries the current version, so the retry is bounded per policy.
|
|
*/
|
|
export const CAPTURE_POLICY_RECOVERY_KEY = "capturePolicyRecovery";
|
|
export const CAPTURE_POLICY_RECOVERY_SCHEMA = "paperclip.capture-policy-recovery.v1";
|
|
// Capture refusals surface as export_interrupted. archive_unsafe names a restore
|
|
// refusal, which the capture policy does not change, so it is not re-armed.
|
|
export const CAPTURE_POLICY_REARM_CODES = ["export_interrupted"] as const;
|
|
|
|
export function isCapturePolicyRearmable(state: Pick<SystemRecoveryState, "state" | "phase" | "failure" | "capturePolicyVersion"> | null | undefined) {
|
|
return state?.state === "exhausted" && state.phase === "preserve_files"
|
|
&& (CAPTURE_POLICY_REARM_CODES as readonly string[]).includes(state.failure?.code ?? "")
|
|
&& (state.capturePolicyVersion ?? 1) < REMOTE_WORKSPACE_CAPTURE_POLICY_VERSION;
|
|
}
|
|
|
|
/**
|
|
* An exhausted save whose last failure was an unproven agent stop is not a
|
|
* permanent refusal: once the source run has ended, a later attempt can prove
|
|
* the stop. Recovery re-arms it a bounded number of times on a backoff
|
|
* (workspace-recovery-stop-proof), then an export hold continues from the last
|
|
* save with the unsaved files attached, like a refused source.
|
|
*/
|
|
export const STOP_PROOF_RECOVERY_KEY = "stopProofRecovery";
|
|
export const STOP_PROOF_RECOVERY_SCHEMA = "paperclip.stop-proof-recovery.v1";
|
|
export const STOP_PROOF_MAX_REARMS = 4;
|
|
export const STOP_PROOF_WAIT_CODE = "agent_stop_unconfirmed";
|
|
/** Coordinator phases after which no finalizer or provider work runs under the coordinator. */
|
|
export const FINISHED_COORDINATOR_PHASES = ["committed", "terminal_failure", "applied"] as const;
|
|
|
|
export function readStopProofRecovery(evidence: Record<string, unknown> | null | undefined): { rearms: number; nextRearmAt: number | null } | null {
|
|
const marker = evidence?.[STOP_PROOF_RECOVERY_KEY] as Record<string, unknown> | undefined;
|
|
if (!marker || marker.schema !== STOP_PROOF_RECOVERY_SCHEMA || typeof marker.rearms !== "number") return null;
|
|
return { rearms: marker.rearms, nextRearmAt: typeof marker.nextRearmAt === "number" ? marker.nextRearmAt : null };
|
|
}
|
|
|
|
export function isExhaustedStopProofWait(state: Pick<SystemRecoveryState, "state" | "failure"> | null | undefined) {
|
|
return state?.state === "exhausted" && state.failure?.code === STOP_PROOF_WAIT_CODE
|
|
&& state.failure.retryClass === "recheck_evidence";
|
|
}
|
|
|
|
/** Every bounded re-arm of an unproven stop is spent. */
|
|
export function stopProofRearmsSpent(evidence: Record<string, unknown> | null | undefined) {
|
|
return (readStopProofRecovery(evidence)?.rearms ?? 0) >= STOP_PROOF_MAX_REARMS;
|
|
}
|
|
|
|
/** User-facing text for a spent recovery budget. It names what happens next and
|
|
* never asks a person to inspect a sandbox or environment. The code stays for support. */
|
|
export function exhaustedSystemRecoveryAction(state: SystemRecoveryState, evidence?: Record<string, unknown> | null): string {
|
|
// Another Retry cannot release a copyback owner that stopped without reporting.
|
|
if (state.failure?.code === "copyback_stop_unconfirmed")
|
|
return "Recovery of saved work is waiting for the previous server's file copy to stop. "
|
|
+ "Recovery resumes once Paperclip can confirm that it stopped. No provider turn is submitted.";
|
|
// An unproven agent stop is re-proven automatically once the run is over
|
|
// (workspace-recovery-stop-proof), while a bounded re-arm remains.
|
|
if (evidence !== undefined && isExhaustedStopProofWait(state) && !stopProofRearmsSpent(evidence))
|
|
return "Recovery of saved work is waiting to confirm that the previous run stopped. "
|
|
+ "Paperclip tries again automatically, and the agent will not repeat its work.";
|
|
const cause = state.failure?.cause ? `, cause: ${state.failure.cause.label}` : "";
|
|
if (state.failure?.cause?.label === "sandbox_disk_full")
|
|
return "Saving the agent's files stopped because the sandbox disk is full. Paperclip deletes only inactive tool "
|
|
+ "caches it can prove safe, and that did not free enough space. Automatic recovery does not repeat a save that fails the same way. "
|
|
+ "Retry saves the files again. The agent's result is kept, and the agent will not repeat its work.";
|
|
if (isDeterministicRecoveryRefusal(state)) {
|
|
const code = state.failure!.code;
|
|
// Never promise a release. A retained-file hold the fallback settles
|
|
// (workspace-recovery-fallback) replaces this text with its own outcome.
|
|
return `Paperclip refused to recover the saved files: ${recoveryRefusalReasons[code] ?? "a safety check on the saved files did not pass"} `
|
|
+ `(${code}${cause}). Automatic recovery does not repeat this check, because the same saved state gives the same result. `
|
|
+ "Retry runs the check again. The agent's result is saved, and the agent will not repeat its work.";
|
|
}
|
|
const reason = state.failure?.code ? ` (last failure: ${state.failure.code}${cause})` : "";
|
|
const outcome = state.attempts > 1 ? `stopped after ${state.attempts} attempts` : "failed";
|
|
return `Automatic recovery of saved work ${outcome}${reason}. `
|
|
+ "Retry to try again; the agent will not repeat its work.";
|
|
}
|
|
|
|
export function isDeterministicRecoveryRefusal(state: Pick<SystemRecoveryState, "state" | "failure"> | null | undefined) {
|
|
return state?.state === "exhausted" && state.failure?.retryClass === "service_required";
|
|
}
|
|
|
|
const recoveryRefusalReasons: Partial<Record<RecoveryFailureCode, string>> = {
|
|
allocation_ownership_ambiguous: "Paperclip cannot prove which runs used the sandbox that holds the saved files",
|
|
source_binding_invalid: "the recorded source of the saved files does not match the run that saved them",
|
|
snapshot_verification_failed: "the saved snapshot of the files did not pass verification",
|
|
archive_unsafe: "the saved file archive contains entries that are not safe to apply",
|
|
source_unavailable: "the saved files are no longer available where the run recorded them",
|
|
};
|