mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 15:35:12 +01:00
## Thinking Path > - Paperclip manages AI agents and the work they produce. > - Sandbox work must return to the host when an agent run ends. > - A failed transfer must keep its source available for recovery. > - Some transfer errors lose their useful fields before restore diagnostics reach Sentry. > - This change records fixed transfer stages, failure kinds, and numeric RPC codes. > - The next failure can identify the operation that failed without exposing workspace contents. ## Linked Issues or Issue Description **What happened?** A workspace transfer can fail after model execution succeeds. Several command, archive validation, and download errors become plain errors. The existing RPC envelope then reports `unknown`, with no transfer stage or command exit code. A host RPC timeout also loses its code from restore diagnostics. **Expected behavior** Keep bounded evidence through the provider, worker RPC, restore result, and Sentry context. Keep the same exception, error message, retry rules, and source retention policy. A transfer timeout must remain separate from the model execution timeout flag. **Steps to reproduce** Return a nonzero exit code from the sandbox archive command, return a per-file download error, exceed a tar listing limit, or time out the sync-out RPC. Observe the missing fields in the saved restore diagnostic. The added tests use local fixtures for these cases. Related public work: #15479 preserves the source after restore failure. #15481 carries bounded sync-out diagnostics through RPC. This change supplies missing producer evidence and extends that same envelope. I checked the roadmap and searched open PRs for duplicate transfer diagnostic work. ## What Changed - Add optional transfer stage and failure kind fields to the existing diagnostic envelope. Capture command exits and listing deadlines at their producers. - Record only known codes from typed RPC errors at the sync-out boundary. Revalidate every field before persistence and Sentry projection. - Keep annotations private to each outbound request and restore settlement. Preserve frozen error identity and prevent evidence from leaking across concurrent or later calls. - Document the fields. Cover producer failures, quota limits, old workers, concurrent error reuse, and redaction through the real Sentry SDK. ## Verification - Focused SDK, provider, host, persistence, and real Sentry tests: 449 passed, no skips. Independent review also ran the focused contracts and all provider tests. The optional Sentry SDK is required for this check, so the contract tests cannot skip. - A compiled Daytona transfer and compiled SDK worker pass a local RPC round trip. This uses the development TypeScript loader for workspace dependency exports. The transfer and capture modules are compiled JavaScript. - `pnpm -r typecheck` and `pnpm build` passed. The excluded Daytona package also passed `tsc --noEmit`. - The initial local `pnpm test:run` stopped in the general-server group: 60 suites could not initialize embedded PostgreSQL because the isolated install skipped its Darwin library-link setup. The existing package postinstall repairs this; `initdb --version` now passes. All 60 affected suites then passed: 1,342 tests, no skips. No local full aggregate pass is claimed. - Independent source review covers privacy, concurrency, packaging, and recovery behavior. ## Risks - These diagnostics help identify future failures. They do not establish or repair the cause of a past transfer failure. - New fields are optional. Older workers remain compatible. Unknown values are omitted. - The change adds a small request-local diagnostic scope. It keeps the original thrown errors, archive confinement, cleanup order, timeout values, retry count, and source retention policy. - No schema or deployment changes are required. No live provider actions are part of validation. ## Model Used OpenAI Codex, based on GPT-6, with repository inspection, code execution, and independent agent review. The exact deployed model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
288 lines
14 KiB
TypeScript
288 lines
14 KiB
TypeScript
import { AsyncLocalStorage } from "node:async_hooks";
|
|
import type { RuntimeProgressSink } from "./runtime-progress.js";
|
|
|
|
type RestorePhase = "workspace" | "asset";
|
|
const RESTORE_STEPS = new Set([
|
|
"git_export", "git_import", "workspace_transfer", "workspace_extract",
|
|
"directory_merge", "git_integration", "index_reset", "git_ref_cleanup", "asset_restore",
|
|
] as const);
|
|
export type WorkspaceRestoreStep = typeof RESTORE_STEPS extends Set<infer T> ? T : never;
|
|
const GIT_COMMANDS = new Set([
|
|
"rev_parse", "symbolic_ref", "merge_base", "merge_tree", "commit_tree", "update_ref", "log",
|
|
] as const);
|
|
export type WorkspaceRestoreGitCommand = typeof GIT_COMMANDS extends Set<infer T> ? T : never;
|
|
const GIT_FAILURE_KINDS = new Set([
|
|
"merge_conflict", "invalid_object", "ref_conflict", "permission_denied", "unknown",
|
|
] as const);
|
|
type GitFailureKind = typeof GIT_FAILURE_KINDS extends Set<infer T> ? T : never;
|
|
const TRANSFER_STEPS = new Set([
|
|
"sandbox_access", "sandbox_guard", "file_download", "file_finalize",
|
|
"archive_create", "archive_download", "archive_validate", "archive_extract",
|
|
] as const);
|
|
type TransferStep = typeof TRANSFER_STEPS extends Set<infer T> ? T : never;
|
|
const TRANSFER_FAILURE_KINDS = new Set([
|
|
"command_failed", "download_failed", "download_missing", "unsafe_archive",
|
|
"listing_timeout", "listing_entry_limit", "listing_byte_limit", "listing_line_limit", "listing_stderr_limit",
|
|
] as const);
|
|
type TransferFailureKind = typeof TRANSFER_FAILURE_KINDS extends Set<infer T> ? T : never;
|
|
// Deliberately revalidate the known plugin RPC constants at the persistence boundary.
|
|
const RPC_CODES = new Set([-32000, -32001, -32002, -32003, -32004, -32005, -32006, -32099]);
|
|
export interface WorkspaceRestoreDiagnostic {
|
|
phase: RestorePhase;
|
|
step?: WorkspaceRestoreStep;
|
|
errorCode: string;
|
|
httpStatus?: number;
|
|
exitCode?: number;
|
|
gitCommand?: WorkspaceRestoreGitCommand;
|
|
gitFailureKind?: GitFailureKind;
|
|
transferStep?: TransferStep;
|
|
transferFailureKind?: TransferFailureKind;
|
|
rpcCode?: number;
|
|
}
|
|
const ERROR_CODES = new Set([
|
|
"ENOENT", "EACCES", "EPERM", "ENOSPC", "EIO", "EXDEV", "ENOTDIR", "EISDIR",
|
|
"ECONNRESET", "ECONNREFUSED", "ETIMEDOUT", "EPIPE", "ENOTFOUND", "EAI_AGAIN",
|
|
"ABORT_ERR", "UND_ERR_CONNECT_TIMEOUT", "UND_ERR_SOCKET",
|
|
]);
|
|
type ErrorDiagnostic = Pick<WorkspaceRestoreDiagnostic, "errorCode" | "httpStatus" | "exitCode" | "transferStep" | "transferFailureKind" | "rpcCode">;
|
|
interface DiagnosticScope {
|
|
active: boolean;
|
|
sequence: number;
|
|
failures: Map<unknown, {
|
|
sequence: number;
|
|
step?: WorkspaceRestoreStep;
|
|
gitCommand?: WorkspaceRestoreGitCommand;
|
|
gitFailureKind?: GitFailureKind;
|
|
}>;
|
|
}
|
|
const activeDiagnostic = new AsyncLocalStorage<DiagnosticScope>();
|
|
// Never attach a raw cause to an error just to retain a numeric Git exit code.
|
|
const wrappedDiagnostics = new WeakMap<object, ErrorDiagnostic>();
|
|
const restoreDiagnostics = new WeakMap<object, WorkspaceRestoreDiagnostic>();
|
|
const diagnosticCapture = new AsyncLocalStorage<{
|
|
restores: WeakMap<object, WorkspaceRestoreDiagnostic>;
|
|
wrapped: WeakMap<object, ErrorDiagnostic>;
|
|
}>();
|
|
|
|
/** Keep each settlement's diagnostic receipt isolated across asynchronous work. */
|
|
export function withWorkspaceRestoreDiagnosticCapture<T>(operation: () => Promise<T>): Promise<T> {
|
|
return diagnosticCapture.run({ restores: new WeakMap(), wrapped: new WeakMap() }, operation);
|
|
}
|
|
|
|
function readField(value: Record<string, unknown>, key: string): unknown {
|
|
try { return value[key]; } catch { return undefined; }
|
|
}
|
|
|
|
function boundedInteger(value: unknown, minimum: number, maximum: number): number | undefined {
|
|
return typeof value === "number" && Number.isInteger(value) && value >= minimum && value <= maximum
|
|
? value : undefined;
|
|
}
|
|
|
|
/** Inspect bounded stderr only in memory; never persist Git text or arguments. */
|
|
function gitFailureKind(command: WorkspaceRestoreGitCommand, error: unknown): GitFailureKind {
|
|
if (!error || typeof error !== "object") return "unknown";
|
|
const value = error as Record<string, unknown>;
|
|
const code = readField(value, "code");
|
|
if (code === "EACCES" || code === "EPERM") return "permission_denied";
|
|
if (readField(value, "killed") || readField(value, "signal")) return "unknown";
|
|
const stderr = readField(value, "stderr");
|
|
const text = typeof stderr === "string" ? stderr.slice(0, 16 * 1024) : "";
|
|
if (command === "update_ref" && /cannot lock ref '[^'\r\n]+': is at [a-f0-9]{40,64} but expected [a-f0-9]{40,64}(?:\s|$)/m.test(text)) {
|
|
return "ref_conflict";
|
|
}
|
|
if (/^fatal: (?:bad object |Not a valid object name |not a valid object name |Not a valid commit name )/m.test(text)
|
|
|| (command === "merge_tree" && /^merge-tree: [a-f0-9]{40,64} - not something we can merge\s*$/m.test(text))) {
|
|
return "invalid_object";
|
|
}
|
|
// merge-tree --write-tree prints the merged tree before reporting conflicts.
|
|
// Exit 1 alone is insufficient: some Git versions also use it for bad objects.
|
|
const stdout = readField(value, "stdout");
|
|
if (command === "merge_tree" && code === 1 && typeof stdout === "string"
|
|
&& /^(?:[a-f0-9]{40}|[a-f0-9]{64})\r?\n/.test(stdout.slice(0, 66))) return "merge_conflict";
|
|
return "unknown";
|
|
}
|
|
|
|
/** Annotate the original thrown error in its restore scope; never change it. */
|
|
export async function withWorkspaceRestoreGitCommand<T>(command: WorkspaceRestoreGitCommand, operation: () => Promise<T>): Promise<T> {
|
|
const scope = activeDiagnostic.getStore();
|
|
if (!scope?.active || !GIT_COMMANDS.has(command)) return await operation();
|
|
const started = scope.sequence;
|
|
try {
|
|
return await operation();
|
|
} catch (error) {
|
|
// A ref transaction can fail in its nested branch-identity probe. Retain
|
|
// that more specific command, while later retries get fresh attribution.
|
|
if (scope.active && (scope.failures.get(error)?.sequence ?? -1) <= started) scope.failures.set(error, {
|
|
sequence: ++scope.sequence, gitCommand: command, gitFailureKind: gitFailureKind(command, error),
|
|
});
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
/** Only fixed codes and bounded numbers may enter the company-readable run log. */
|
|
function diagnostic(error: unknown): ErrorDiagnostic {
|
|
const result: ErrorDiagnostic = { errorCode: "unknown" };
|
|
let current = error;
|
|
// SDKs wrap transport errors in a cause. Bound traversal, including cycles.
|
|
for (let depth = 0; depth < 4 && current && typeof current === "object"; depth++) {
|
|
const value = current as Record<string, unknown>;
|
|
const saved = (diagnosticCapture.getStore()?.wrapped ?? wrappedDiagnostics).get(value);
|
|
result.transferStep ??= saved?.transferStep;
|
|
result.transferFailureKind ??= saved?.transferFailureKind;
|
|
result.rpcCode ??= saved?.rpcCode;
|
|
const code = saved?.errorCode !== "unknown" && saved?.errorCode !== undefined
|
|
? saved.errorCode : readField(value, "code");
|
|
if (result.errorCode === "unknown" && typeof code === "string" && ERROR_CODES.has(code)) {
|
|
result.errorCode = code;
|
|
}
|
|
const status = saved?.httpStatus ?? boundedInteger(readField(value, "status"), 400, 599)
|
|
?? boundedInteger(readField(value, "statusCode"), 400, 599);
|
|
if (result.httpStatus === undefined && status !== undefined) {
|
|
result.httpStatus = status;
|
|
}
|
|
const exitCode = saved?.exitCode ?? boundedInteger(readField(value, "exitCode"), 1, 255) ?? boundedInteger(code, 1, 255);
|
|
if (result.exitCode === undefined && exitCode !== undefined) {
|
|
result.exitCode = exitCode;
|
|
}
|
|
current = readField(value, "cause");
|
|
}
|
|
return result;
|
|
}
|
|
|
|
/** Retain bounded fields across an existing error wrapper, without a raw cause. */
|
|
export function preserveWorkspaceRestoreErrorDiagnostic<T extends object>(wrapper: T, source: unknown, transferEvidence?: unknown): T {
|
|
(diagnosticCapture.getStore()?.wrapped ?? wrappedDiagnostics).set(wrapper, { ...diagnostic(source), ...sanitizeTransferEvidence(transferEvidence) });
|
|
const scope = activeDiagnostic.getStore();
|
|
const failure = scope?.active ? scope.failures.get(source) : undefined;
|
|
if (failure) scope!.failures.set(wrapper, failure);
|
|
return wrapper;
|
|
}
|
|
|
|
function sanitizeTransferEvidence(value: unknown): Pick<WorkspaceRestoreDiagnostic, "transferStep" | "transferFailureKind" | "rpcCode"> {
|
|
if (!value || typeof value !== "object") return {};
|
|
const record = value as Record<string, unknown>;
|
|
const step = readField(record, "transferStep");
|
|
const kind = readField(record, "transferFailureKind");
|
|
const rpcCode = readField(record, "rpcCode");
|
|
return {
|
|
...(typeof step === "string" && TRANSFER_STEPS.has(step as TransferStep) ? { transferStep: step as TransferStep } : {}),
|
|
...(typeof kind === "string" && TRANSFER_FAILURE_KINDS.has(kind as TransferFailureKind) ? { transferFailureKind: kind as TransferFailureKind } : {}),
|
|
...(typeof rpcCode === "number" && RPC_CODES.has(rpcCode) ? { rpcCode } : {}),
|
|
};
|
|
}
|
|
|
|
/** Decode persisted adapter metadata; never copy arbitrary properties or text. */
|
|
export function sanitizeWorkspaceRestoreDiagnostic(value: unknown): WorkspaceRestoreDiagnostic | undefined {
|
|
if (!value || typeof value !== "object") return undefined;
|
|
const record = value as Record<string, unknown>;
|
|
const phase = readField(record, "phase");
|
|
if (phase !== "workspace" && phase !== "asset") return undefined;
|
|
const step = readField(record, "step");
|
|
const code = readField(record, "errorCode");
|
|
const httpStatus = boundedInteger(readField(record, "httpStatus"), 400, 599);
|
|
const exitCode = boundedInteger(readField(record, "exitCode"), 1, 255);
|
|
const gitCommand = readField(record, "gitCommand");
|
|
const gitFailureKind = readField(record, "gitFailureKind");
|
|
const hasGitCommand = phase === "workspace" && step === "git_integration"
|
|
&& typeof gitCommand === "string" && GIT_COMMANDS.has(gitCommand as WorkspaceRestoreGitCommand);
|
|
return {
|
|
phase,
|
|
...(step === "workspace_transfer" || step === "asset_restore" ? sanitizeTransferEvidence(record) : {}),
|
|
...(typeof step === "string" && RESTORE_STEPS.has(step as WorkspaceRestoreStep)
|
|
? { step: step as WorkspaceRestoreStep } : {}),
|
|
errorCode: typeof code === "string" && ERROR_CODES.has(code) ? code : "unknown",
|
|
...(httpStatus !== undefined ? { httpStatus } : {}),
|
|
...(exitCode !== undefined ? { exitCode } : {}),
|
|
...(hasGitCommand ? {
|
|
gitCommand: gitCommand as WorkspaceRestoreGitCommand,
|
|
gitFailureKind: typeof gitFailureKind === "string" && GIT_FAILURE_KINDS.has(gitFailureKind as GitFailureKind)
|
|
? gitFailureKind as GitFailureKind : "unknown",
|
|
} : {}),
|
|
};
|
|
}
|
|
|
|
export function getWorkspaceRestoreDiagnostic(error: unknown): WorkspaceRestoreDiagnostic | undefined {
|
|
return error && typeof error === "object"
|
|
? sanitizeWorkspaceRestoreDiagnostic((diagnosticCapture.getStore()?.restores ?? restoreDiagnostics).get(error)) : undefined;
|
|
}
|
|
|
|
/** Preserve the scheduler-selected task's snapshot when errors share identity. */
|
|
export function recordWorkspaceRestoreDiagnostic(error: unknown, diagnostic: WorkspaceRestoreDiagnostic | undefined): void {
|
|
const safe = sanitizeWorkspaceRestoreDiagnostic(diagnostic);
|
|
if (error && typeof error === "object") {
|
|
const receipts = diagnosticCapture.getStore()?.restores ?? restoreDiagnostics;
|
|
if (safe) receipts.set(error, safe);
|
|
else receipts.delete(error);
|
|
}
|
|
const scope = activeDiagnostic.getStore();
|
|
if (scope?.active) {
|
|
if (safe?.step) scope.failures.set(error, {
|
|
sequence: ++scope.sequence, step: safe.step,
|
|
...(safe.gitCommand ? { gitCommand: safe.gitCommand, gitFailureKind: safe.gitFailureKind } : {}),
|
|
});
|
|
else scope.failures.delete(error);
|
|
}
|
|
}
|
|
|
|
/** Record only the failing step. Successful cleanup must not replace evidence. */
|
|
export async function withWorkspaceRestoreStep<T>(step: WorkspaceRestoreStep, operation: () => Promise<T>): Promise<T> {
|
|
const scope = activeDiagnostic.getStore();
|
|
if (!scope?.active) return await operation();
|
|
const started = scope.sequence;
|
|
try {
|
|
return await operation();
|
|
} catch (error) {
|
|
// A nested step owns its failure. A caught/retried error may be thrown again
|
|
// by a later step, so identity alone cannot identify the current failure.
|
|
if (scope.active) {
|
|
const failure = scope.failures.get(error);
|
|
if (failure && failure.sequence > started) {
|
|
failure.step ??= step;
|
|
} else {
|
|
scope.failures.set(error, { sequence: ++scope.sequence, step });
|
|
}
|
|
}
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
/** Add evidence without changing the thrown error, restore policy, or task ordering. */
|
|
export async function withWorkspaceRestoreDiagnostics<T>(
|
|
phase: RestorePhase,
|
|
operation: () => Promise<T>,
|
|
onProgress?: RuntimeProgressSink,
|
|
onDiagnostic?: (diagnostic: WorkspaceRestoreDiagnostic) => void,
|
|
): Promise<T> {
|
|
// A nested repository restore propagates to its enclosing workspace task.
|
|
// That task owns the diagnostic. Independent parallel tasks retain their own
|
|
// async scopes, so two failed tasks still produce two diagnostic lines.
|
|
const parent = activeDiagnostic.getStore();
|
|
const scope: DiagnosticScope = { active: true, sequence: 0, failures: new Map() };
|
|
return await activeDiagnostic.run(scope, async () => {
|
|
try {
|
|
return await operation();
|
|
} catch (error) {
|
|
const failure = scope.failures.get(error);
|
|
const step = failure?.step;
|
|
const fields = sanitizeWorkspaceRestoreDiagnostic({ phase, ...(step ? { step } : {}), ...diagnostic(error),
|
|
...(phase === "workspace" && step === "git_integration" && failure?.gitCommand
|
|
? { gitCommand: failure.gitCommand, gitFailureKind: failure.gitFailureKind } : {}),
|
|
})!;
|
|
recordWorkspaceRestoreDiagnostic(error, fields);
|
|
if (parent?.active && step) parent.failures.set(error, { ...failure, sequence: ++parent.sequence, step });
|
|
try {
|
|
onDiagnostic?.({ ...fields });
|
|
} catch { /* Diagnostic consumers cannot replace a restore failure. */ }
|
|
try {
|
|
if (!parent?.active) await onProgress?.(`[paperclip] Workspace restore diagnostic: ${JSON.stringify(fields)}\n`);
|
|
} catch {
|
|
// A broken log sink must not replace a restore failure or relax its safety classification.
|
|
}
|
|
throw error;
|
|
} finally {
|
|
scope.active = false;
|
|
scope.failures.clear();
|
|
}
|
|
});
|
|
}
|