Files
PaperClipAI/packages/adapter-utils/src/workspace-restore-diagnostics.ts
T
Devin FoleyandClaude Opus 5.5 7f5d0793bd fix(daytona): record a bounded errno when a sandbox command fails (#19297)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Daytona sandbox provider copies the agent's workspace back to
the host after each turn (syncOut)
> - When the sandbox `tar` fails in the `archive_create` step, the
restore diagnostic records only `exitCode: 2` and `errorCode: "unknown"`
> - The command output is not sent across the plugin RPC boundary, so an
operator cannot tell a full disk from a permission error or a path that
is too long
> - This pull request maps the system error text at the end of a failed
sandbox command's output line to the existing allowlisted `errorCode`
> - The benefit is that the next failure records the real errno, and the
output text stays private

## Linked Issues or Issue Description

Refs #18172
Refs #18212

**What happened?**
A hosted Daytona run failed its workspace restore. The `archive_create`
step exited 2 on all three attempts. The run log recorded
`{"step":"workspace_transfer","transferStep":"archive_create","transferFailureKind":"command_failed","exitCode":2,"errorCode":"unknown"}`.
Two causes are possible. The sandbox disk was full, or a path was longer
than PATH_MAX, for example from deeply nested GitHub helper config
directories before #18212. The diagnostic cannot tell these causes
apart.

**Expected behavior**
A failed sandbox command records a bounded errno such as `ENOSPC`,
`EACCES` or `ENAMETOOLONG` when its output shows one. The output text
itself must not be recorded.

**Steps to reproduce**
1. Run `performSyncOut` against a sandbox whose `tar -c` exits 2 with
`tar: ./x: Cannot open: Permission denied`.
2. Read the environment sync diagnostic from the thrown error.
3. See that `errorCode` is `unknown`.

**Paperclip version or commit**
`master` at 3fca8a1.

## What Changed

- `packages/plugins/sandbox-providers/daytona/src/file-sync.ts`:
`assertSandboxCommandOkWithOutput` maps the strerror suffix of a failed
command's output to an allowlisted errno. It reads only the last 16 KiB
in memory. A line must end with `: <message>`. Fatal causes are checked
first (ENOSPC, EIO, ENAMETOOLONG, EACCES, EPERM, ENOTDIR, EISDIR,
ENOENT), so a warning about a vanished file does not hide the error that
stopped `tar`. The error message, capacity fallback (#18172), restore
and retain-on-failure behavior do not change.
- `packages/plugins/sdk/src/environment-sync-error.ts` and
`packages/adapter-utils/src/workspace-restore-diagnostics.ts`: add
`ENAMETOOLONG` to both allowlists. Both sides still validate
independently.
- Tests: 7 new Daytona cases, plus one allowlist case each in the SDK
and adapter-utils.

## Verification

- `vitest run --config
packages/plugins/sandbox-providers/daytona/vitest.config.ts
packages/plugins/sandbox-providers/daytona/src/file-sync-diagnostics.test.ts`:
18 passed. Without the `file-sync.ts` change, the 7 new cases fail with
`errorCode: "unknown"`.
- `packages/plugins/sdk`: `vitest run
tests/environment-sync-error.test.ts`: 20 passed.
- `packages/adapter-utils`: `vitest run
src/workspace-restore-diagnostics.test.ts`: 50 passed.
- `tsc --noEmit` passes for the SDK and adapter-utils. `git diff
--check` is clean.
- I did not run the Daytona package typecheck or its full suite locally
because the sandbox disk was full. CI covers them.

## Risks

- Low risk. The change adds a diagnostic field only. Control flow, error
messages and the capacity fallback regex do not change.
- Localized `tar` output does not match, so it stays `unknown`. That is
the current behavior.
- `ENAMETOOLONG` is a new allowlisted code on both sides. It is a fixed
string and carries no path or output text.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 11:34:41 -08:00

288 lines
14 KiB
TypeScript

import { AsyncLocalStorage } from "node:async_hooks";
import type { RuntimeProgressSink } from "./runtime-progress.js";
type RestorePhase = "workspace" | "asset";
const RESTORE_STEPS = new Set([
"git_export", "git_import", "workspace_transfer", "workspace_extract",
"directory_merge", "git_integration", "index_reset", "git_ref_cleanup", "asset_restore",
] as const);
export type WorkspaceRestoreStep = typeof RESTORE_STEPS extends Set<infer T> ? T : never;
const GIT_COMMANDS = new Set([
"rev_parse", "symbolic_ref", "merge_base", "merge_tree", "commit_tree", "update_ref", "log",
] as const);
export type WorkspaceRestoreGitCommand = typeof GIT_COMMANDS extends Set<infer T> ? T : never;
const GIT_FAILURE_KINDS = new Set([
"merge_conflict", "invalid_object", "ref_conflict", "permission_denied", "unknown",
] as const);
type GitFailureKind = typeof GIT_FAILURE_KINDS extends Set<infer T> ? T : never;
const TRANSFER_STEPS = new Set([
"sandbox_access", "sandbox_guard", "file_download", "file_finalize",
"archive_create", "archive_download", "archive_validate", "archive_extract",
] as const);
type TransferStep = typeof TRANSFER_STEPS extends Set<infer T> ? T : never;
const TRANSFER_FAILURE_KINDS = new Set([
"command_failed", "download_failed", "download_missing", "unsafe_archive",
"listing_timeout", "listing_entry_limit", "listing_byte_limit", "listing_line_limit", "listing_stderr_limit",
] as const);
type TransferFailureKind = typeof TRANSFER_FAILURE_KINDS extends Set<infer T> ? T : never;
// Deliberately revalidate the known plugin RPC constants at the persistence boundary.
const RPC_CODES = new Set([-32000, -32001, -32002, -32003, -32004, -32005, -32006, -32099]);
export interface WorkspaceRestoreDiagnostic {
phase: RestorePhase;
step?: WorkspaceRestoreStep;
errorCode: string;
httpStatus?: number;
exitCode?: number;
gitCommand?: WorkspaceRestoreGitCommand;
gitFailureKind?: GitFailureKind;
transferStep?: TransferStep;
transferFailureKind?: TransferFailureKind;
rpcCode?: number;
}
const ERROR_CODES = new Set([
"ENOENT", "EACCES", "EPERM", "ENOSPC", "EIO", "EXDEV", "ENOTDIR", "EISDIR", "ENAMETOOLONG",
"ECONNRESET", "ECONNREFUSED", "ETIMEDOUT", "EPIPE", "ENOTFOUND", "EAI_AGAIN",
"ABORT_ERR", "UND_ERR_CONNECT_TIMEOUT", "UND_ERR_SOCKET",
]);
type ErrorDiagnostic = Pick<WorkspaceRestoreDiagnostic, "errorCode" | "httpStatus" | "exitCode" | "transferStep" | "transferFailureKind" | "rpcCode">;
interface DiagnosticScope {
active: boolean;
sequence: number;
failures: Map<unknown, {
sequence: number;
step?: WorkspaceRestoreStep;
gitCommand?: WorkspaceRestoreGitCommand;
gitFailureKind?: GitFailureKind;
}>;
}
const activeDiagnostic = new AsyncLocalStorage<DiagnosticScope>();
// Never attach a raw cause to an error just to retain a numeric Git exit code.
const wrappedDiagnostics = new WeakMap<object, ErrorDiagnostic>();
const restoreDiagnostics = new WeakMap<object, WorkspaceRestoreDiagnostic>();
const diagnosticCapture = new AsyncLocalStorage<{
restores: WeakMap<object, WorkspaceRestoreDiagnostic>;
wrapped: WeakMap<object, ErrorDiagnostic>;
}>();
/** Keep each settlement's diagnostic receipt isolated across asynchronous work. */
export function withWorkspaceRestoreDiagnosticCapture<T>(operation: () => Promise<T>): Promise<T> {
return diagnosticCapture.run({ restores: new WeakMap(), wrapped: new WeakMap() }, operation);
}
function readField(value: Record<string, unknown>, key: string): unknown {
try { return value[key]; } catch { return undefined; }
}
function boundedInteger(value: unknown, minimum: number, maximum: number): number | undefined {
return typeof value === "number" && Number.isInteger(value) && value >= minimum && value <= maximum
? value : undefined;
}
/** Inspect bounded stderr only in memory; never persist Git text or arguments. */
function gitFailureKind(command: WorkspaceRestoreGitCommand, error: unknown): GitFailureKind {
if (!error || typeof error !== "object") return "unknown";
const value = error as Record<string, unknown>;
const code = readField(value, "code");
if (code === "EACCES" || code === "EPERM") return "permission_denied";
if (readField(value, "killed") || readField(value, "signal")) return "unknown";
const stderr = readField(value, "stderr");
const text = typeof stderr === "string" ? stderr.slice(0, 16 * 1024) : "";
if (command === "update_ref" && /cannot lock ref '[^'\r\n]+': is at [a-f0-9]{40,64} but expected [a-f0-9]{40,64}(?:\s|$)/m.test(text)) {
return "ref_conflict";
}
if (/^fatal: (?:bad object |Not a valid object name |not a valid object name |Not a valid commit name )/m.test(text)
|| (command === "merge_tree" && /^merge-tree: [a-f0-9]{40,64} - not something we can merge\s*$/m.test(text))) {
return "invalid_object";
}
// merge-tree --write-tree prints the merged tree before reporting conflicts.
// Exit 1 alone is insufficient: some Git versions also use it for bad objects.
const stdout = readField(value, "stdout");
if (command === "merge_tree" && code === 1 && typeof stdout === "string"
&& /^(?:[a-f0-9]{40}|[a-f0-9]{64})\r?\n/.test(stdout.slice(0, 66))) return "merge_conflict";
return "unknown";
}
/** Annotate the original thrown error in its restore scope; never change it. */
export async function withWorkspaceRestoreGitCommand<T>(command: WorkspaceRestoreGitCommand, operation: () => Promise<T>): Promise<T> {
const scope = activeDiagnostic.getStore();
if (!scope?.active || !GIT_COMMANDS.has(command)) return await operation();
const started = scope.sequence;
try {
return await operation();
} catch (error) {
// A ref transaction can fail in its nested branch-identity probe. Retain
// that more specific command, while later retries get fresh attribution.
if (scope.active && (scope.failures.get(error)?.sequence ?? -1) <= started) scope.failures.set(error, {
sequence: ++scope.sequence, gitCommand: command, gitFailureKind: gitFailureKind(command, error),
});
throw error;
}
}
/** Only fixed codes and bounded numbers may enter the company-readable run log. */
function diagnostic(error: unknown): ErrorDiagnostic {
const result: ErrorDiagnostic = { errorCode: "unknown" };
let current = error;
// SDKs wrap transport errors in a cause. Bound traversal, including cycles.
for (let depth = 0; depth < 4 && current && typeof current === "object"; depth++) {
const value = current as Record<string, unknown>;
const saved = (diagnosticCapture.getStore()?.wrapped ?? wrappedDiagnostics).get(value);
result.transferStep ??= saved?.transferStep;
result.transferFailureKind ??= saved?.transferFailureKind;
result.rpcCode ??= saved?.rpcCode;
const code = saved?.errorCode !== "unknown" && saved?.errorCode !== undefined
? saved.errorCode : readField(value, "code");
if (result.errorCode === "unknown" && typeof code === "string" && ERROR_CODES.has(code)) {
result.errorCode = code;
}
const status = saved?.httpStatus ?? boundedInteger(readField(value, "status"), 400, 599)
?? boundedInteger(readField(value, "statusCode"), 400, 599);
if (result.httpStatus === undefined && status !== undefined) {
result.httpStatus = status;
}
const exitCode = saved?.exitCode ?? boundedInteger(readField(value, "exitCode"), 1, 255) ?? boundedInteger(code, 1, 255);
if (result.exitCode === undefined && exitCode !== undefined) {
result.exitCode = exitCode;
}
current = readField(value, "cause");
}
return result;
}
/** Retain bounded fields across an existing error wrapper, without a raw cause. */
export function preserveWorkspaceRestoreErrorDiagnostic<T extends object>(wrapper: T, source: unknown, transferEvidence?: unknown): T {
(diagnosticCapture.getStore()?.wrapped ?? wrappedDiagnostics).set(wrapper, { ...diagnostic(source), ...sanitizeTransferEvidence(transferEvidence) });
const scope = activeDiagnostic.getStore();
const failure = scope?.active ? scope.failures.get(source) : undefined;
if (failure) scope!.failures.set(wrapper, failure);
return wrapper;
}
function sanitizeTransferEvidence(value: unknown): Pick<WorkspaceRestoreDiagnostic, "transferStep" | "transferFailureKind" | "rpcCode"> {
if (!value || typeof value !== "object") return {};
const record = value as Record<string, unknown>;
const step = readField(record, "transferStep");
const kind = readField(record, "transferFailureKind");
const rpcCode = readField(record, "rpcCode");
return {
...(typeof step === "string" && TRANSFER_STEPS.has(step as TransferStep) ? { transferStep: step as TransferStep } : {}),
...(typeof kind === "string" && TRANSFER_FAILURE_KINDS.has(kind as TransferFailureKind) ? { transferFailureKind: kind as TransferFailureKind } : {}),
...(typeof rpcCode === "number" && RPC_CODES.has(rpcCode) ? { rpcCode } : {}),
};
}
/** Decode persisted adapter metadata; never copy arbitrary properties or text. */
export function sanitizeWorkspaceRestoreDiagnostic(value: unknown): WorkspaceRestoreDiagnostic | undefined {
if (!value || typeof value !== "object") return undefined;
const record = value as Record<string, unknown>;
const phase = readField(record, "phase");
if (phase !== "workspace" && phase !== "asset") return undefined;
const step = readField(record, "step");
const code = readField(record, "errorCode");
const httpStatus = boundedInteger(readField(record, "httpStatus"), 400, 599);
const exitCode = boundedInteger(readField(record, "exitCode"), 1, 255);
const gitCommand = readField(record, "gitCommand");
const gitFailureKind = readField(record, "gitFailureKind");
const hasGitCommand = phase === "workspace" && step === "git_integration"
&& typeof gitCommand === "string" && GIT_COMMANDS.has(gitCommand as WorkspaceRestoreGitCommand);
return {
phase,
...(step === "workspace_transfer" || step === "asset_restore" ? sanitizeTransferEvidence(record) : {}),
...(typeof step === "string" && RESTORE_STEPS.has(step as WorkspaceRestoreStep)
? { step: step as WorkspaceRestoreStep } : {}),
errorCode: typeof code === "string" && ERROR_CODES.has(code) ? code : "unknown",
...(httpStatus !== undefined ? { httpStatus } : {}),
...(exitCode !== undefined ? { exitCode } : {}),
...(hasGitCommand ? {
gitCommand: gitCommand as WorkspaceRestoreGitCommand,
gitFailureKind: typeof gitFailureKind === "string" && GIT_FAILURE_KINDS.has(gitFailureKind as GitFailureKind)
? gitFailureKind as GitFailureKind : "unknown",
} : {}),
};
}
export function getWorkspaceRestoreDiagnostic(error: unknown): WorkspaceRestoreDiagnostic | undefined {
return error && typeof error === "object"
? sanitizeWorkspaceRestoreDiagnostic((diagnosticCapture.getStore()?.restores ?? restoreDiagnostics).get(error)) : undefined;
}
/** Preserve the scheduler-selected task's snapshot when errors share identity. */
export function recordWorkspaceRestoreDiagnostic(error: unknown, diagnostic: WorkspaceRestoreDiagnostic | undefined): void {
const safe = sanitizeWorkspaceRestoreDiagnostic(diagnostic);
if (error && typeof error === "object") {
const receipts = diagnosticCapture.getStore()?.restores ?? restoreDiagnostics;
if (safe) receipts.set(error, safe);
else receipts.delete(error);
}
const scope = activeDiagnostic.getStore();
if (scope?.active) {
if (safe?.step) scope.failures.set(error, {
sequence: ++scope.sequence, step: safe.step,
...(safe.gitCommand ? { gitCommand: safe.gitCommand, gitFailureKind: safe.gitFailureKind } : {}),
});
else scope.failures.delete(error);
}
}
/** Record only the failing step. Successful cleanup must not replace evidence. */
export async function withWorkspaceRestoreStep<T>(step: WorkspaceRestoreStep, operation: () => Promise<T>): Promise<T> {
const scope = activeDiagnostic.getStore();
if (!scope?.active) return await operation();
const started = scope.sequence;
try {
return await operation();
} catch (error) {
// A nested step owns its failure. A caught/retried error may be thrown again
// by a later step, so identity alone cannot identify the current failure.
if (scope.active) {
const failure = scope.failures.get(error);
if (failure && failure.sequence > started) {
failure.step ??= step;
} else {
scope.failures.set(error, { sequence: ++scope.sequence, step });
}
}
throw error;
}
}
/** Add evidence without changing the thrown error, restore policy, or task ordering. */
export async function withWorkspaceRestoreDiagnostics<T>(
phase: RestorePhase,
operation: () => Promise<T>,
onProgress?: RuntimeProgressSink,
onDiagnostic?: (diagnostic: WorkspaceRestoreDiagnostic) => void,
): Promise<T> {
// A nested repository restore propagates to its enclosing workspace task.
// That task owns the diagnostic. Independent parallel tasks retain their own
// async scopes, so two failed tasks still produce two diagnostic lines.
const parent = activeDiagnostic.getStore();
const scope: DiagnosticScope = { active: true, sequence: 0, failures: new Map() };
return await activeDiagnostic.run(scope, async () => {
try {
return await operation();
} catch (error) {
const failure = scope.failures.get(error);
const step = failure?.step;
const fields = sanitizeWorkspaceRestoreDiagnostic({ phase, ...(step ? { step } : {}), ...diagnostic(error),
...(phase === "workspace" && step === "git_integration" && failure?.gitCommand
? { gitCommand: failure.gitCommand, gitFailureKind: failure.gitFailureKind } : {}),
})!;
recordWorkspaceRestoreDiagnostic(error, fields);
if (parent?.active && step) parent.failures.set(error, { ...failure, sequence: ++parent.sequence, step });
try {
onDiagnostic?.({ ...fields });
} catch { /* Diagnostic consumers cannot replace a restore failure. */ }
try {
if (!parent?.active) await onProgress?.(`[paperclip] Workspace restore diagnostic: ${JSON.stringify(fields)}\n`);
} catch {
// A broken log sink must not replace a restore failure or relax its safety classification.
}
throw error;
} finally {
scope.active = false;
scope.failures.clear();
}
});
}