Files
PaperClipAI/packages/adapter-utils/src/workspace-restore-diagnostics.ts
T
Devin FoleyandPaperclip efac8ff2f4 Add bounded Git integration restore diagnostics (#15291)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Sandbox runs must restore their workspace before finalization can
succeed.
> - Restore diagnostics identify the failed phase and operation.
> - A Git integration exit code can still describe several different
failures.
> - This pull request adds fixed command labels and supported failure
classes.
> - Operators can distinguish these failures without collecting private
Git output.

## Linked Issues or Issue Description

Refs #15005. This branch includes merged #15268. It labels that change's
locked ref transaction and nested branch probe without changing their
behavior.

**What happened?**

A failed Git integration can report only `git_integration`, `unknown`,
and an exit code. That evidence does not identify the failed command.
Some Git versions also return exit 1 for both a merge conflict and an
invalid object.

**Expected behavior**

Record a fixed command family and a supported failure class. Keep
unknown cases as `unknown`. Exclude command arguments, process output,
paths, repository URLs, filenames, and ref names.

**Steps to reproduce**

The tests create local repositories with conflicting commits, a missing
object, and an expected-old ref mismatch. They call the real Git
operations and inspect the resulting diagnostic. No hosted workspace or
external provider is used.

**Paperclip version or commit**

Base: `e99854249c`.

**Deployment mode**

Built from source. The diagnostic applies to sandbox workspace restore.

## What Changed

- Label Git integration calls with a closed command enum. Preserve
arguments, options, errors, and retry behavior.
- Recognize supported object and ref errors and OS permission codes.
Require both exit 1 and completed tree output for a merge conflict.
- Carry the closed fields through the existing restore receipt, saved
adapter result, and Sentry projection. Revalidate saved metadata before
projection.
- Test nested wrappers, parallel failures, reused errors, handled
probes, result settlement, and privacy with the real Sentry SDK.
- Document the field contract and its limits.

## Verification

- Eight focused suites pass: 295 tests. These cover Git sync, restore
diagnostics, result settlement, teardown, sandbox runtime, failure
projection, the real Sentry SDK, and native warm-workspace Git history.
- `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1 pnpm exec vitest run
packages/adapter-utils/src/workspace-restore-diagnostics.test.ts
packages/adapter-utils/src/workspace-restore-result.test.ts
packages/adapter-utils/src/git-workspace-sync.test.ts
packages/adapter-utils/src/workspace-restore-teardown.test.ts
packages/adapter-utils/src/sandbox-managed-runtime.test.ts
server/src/services/__tests__/run-failure-diagnostics.test.ts
server/src/__tests__/run-failure-sentry-real-sdk.test.ts
server/src/__tests__/native-workspace-sync-history.test.ts`
- Local checks use Node 24.21.0 and the pinned pnpm 9.15.4. The optional
Sentry SDK is pinned to the declared 10.71.0 and uses an in-memory
transport.
- `pnpm -r typecheck` and `pnpm build`: pass on the rebased head.
- The full local `pnpm test:run` was stopped before source changes when
#15268 merged and required a rebase. It reported three pre-existing
company-skill cache test failures on macOS. The same three cases fail on
clean bases `2c43b39167` and `e99854249c` and the earlier head with
`EACCES` when publishing a read-only cache directory. The relevant test,
service, and cache source blobs are identical. No cache changes are
included. Later local test groups were not reached.
- All 54 checks pass on exact head `8448896bc6`, including the
post-ready security scan, with two intentional Storybook skips. Greptile
scores this head 5/5 with no review threads. Full Linux CI covers the
local groups that were not reached.
- Independent review found no blocker. Its diagnostics, Git workspace,
and native history suites pass 115/115 on this head.
- A source comparison confirms that removing only diagnostic wrappers
yields the merged upstream Git integration code exactly, including ref
locks, transaction protocol, arguments, options, and retry decisions.
- The clean base has stale dependency overrides in its lockfile. Local
installation resolved them as the existing PR CI fallback does. No
manifest or lockfile change is included.
- `git diff --check` and a local secrets/PII review pass.

## Risks

A Git version or localized message may not match a known failure form.
Such cases remain `unknown`. Up to 16 KiB of stderr and the bounded
tree-ID prefix of stdout are inspected only in memory. The saved fields
contain enum values only. These diagnostics do not establish workspace
recovery or authorize retries. Restore decisions and Git mutations are
unchanged. No schema migration is required.

## Model Used

OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The
service does not expose the exact model deployment ID or context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-05 17:10:05 -07:00

253 lines
12 KiB
TypeScript

import { AsyncLocalStorage } from "node:async_hooks";
import type { RuntimeProgressSink } from "./runtime-progress.js";
type RestorePhase = "workspace" | "asset";
const RESTORE_STEPS = new Set([
"git_export", "git_import", "workspace_transfer", "workspace_extract",
"directory_merge", "git_integration", "index_reset", "git_ref_cleanup", "asset_restore",
] as const);
export type WorkspaceRestoreStep = typeof RESTORE_STEPS extends Set<infer T> ? T : never;
const GIT_COMMANDS = new Set([
"rev_parse", "symbolic_ref", "merge_base", "merge_tree", "commit_tree", "update_ref", "log",
] as const);
export type WorkspaceRestoreGitCommand = typeof GIT_COMMANDS extends Set<infer T> ? T : never;
const GIT_FAILURE_KINDS = new Set([
"merge_conflict", "invalid_object", "ref_conflict", "permission_denied", "unknown",
] as const);
type GitFailureKind = typeof GIT_FAILURE_KINDS extends Set<infer T> ? T : never;
export interface WorkspaceRestoreDiagnostic {
phase: RestorePhase;
step?: WorkspaceRestoreStep;
errorCode: string;
httpStatus?: number;
exitCode?: number;
gitCommand?: WorkspaceRestoreGitCommand;
gitFailureKind?: GitFailureKind;
}
const ERROR_CODES = new Set([
"ENOENT", "EACCES", "EPERM", "ENOSPC", "EIO", "EXDEV", "ENOTDIR", "EISDIR",
"ECONNRESET", "ECONNREFUSED", "ETIMEDOUT", "EPIPE", "ENOTFOUND", "EAI_AGAIN",
"ABORT_ERR", "UND_ERR_CONNECT_TIMEOUT", "UND_ERR_SOCKET",
]);
type ErrorDiagnostic = Pick<WorkspaceRestoreDiagnostic, "errorCode" | "httpStatus" | "exitCode">;
interface DiagnosticScope {
active: boolean;
sequence: number;
failures: Map<unknown, {
sequence: number;
step?: WorkspaceRestoreStep;
gitCommand?: WorkspaceRestoreGitCommand;
gitFailureKind?: GitFailureKind;
}>;
}
const activeDiagnostic = new AsyncLocalStorage<DiagnosticScope>();
// Never attach a raw cause to an error just to retain a numeric Git exit code.
const wrappedDiagnostics = new WeakMap<object, ErrorDiagnostic>();
const restoreDiagnostics = new WeakMap<object, WorkspaceRestoreDiagnostic>();
const diagnosticCapture = new AsyncLocalStorage<WeakMap<object, WorkspaceRestoreDiagnostic>>();
/** Keep each settlement's diagnostic receipt isolated across asynchronous work. */
export function withWorkspaceRestoreDiagnosticCapture<T>(operation: () => Promise<T>): Promise<T> {
return diagnosticCapture.run(new WeakMap(), operation);
}
function readField(value: Record<string, unknown>, key: string): unknown {
try { return value[key]; } catch { return undefined; }
}
function boundedInteger(value: unknown, minimum: number, maximum: number): number | undefined {
return typeof value === "number" && Number.isInteger(value) && value >= minimum && value <= maximum
? value : undefined;
}
/** Inspect bounded stderr only in memory; never persist Git text or arguments. */
function gitFailureKind(command: WorkspaceRestoreGitCommand, error: unknown): GitFailureKind {
if (!error || typeof error !== "object") return "unknown";
const value = error as Record<string, unknown>;
const code = readField(value, "code");
if (code === "EACCES" || code === "EPERM") return "permission_denied";
if (readField(value, "killed") || readField(value, "signal")) return "unknown";
const stderr = readField(value, "stderr");
const text = typeof stderr === "string" ? stderr.slice(0, 16 * 1024) : "";
if (command === "update_ref" && /cannot lock ref '[^'\r\n]+': is at [a-f0-9]{40,64} but expected [a-f0-9]{40,64}(?:\s|$)/m.test(text)) {
return "ref_conflict";
}
if (/^fatal: (?:bad object |Not a valid object name |not a valid object name |Not a valid commit name )/m.test(text)
|| (command === "merge_tree" && /^merge-tree: [a-f0-9]{40,64} - not something we can merge\s*$/m.test(text))) {
return "invalid_object";
}
// merge-tree --write-tree prints the merged tree before reporting conflicts.
// Exit 1 alone is insufficient: some Git versions also use it for bad objects.
const stdout = readField(value, "stdout");
if (command === "merge_tree" && code === 1 && typeof stdout === "string"
&& /^(?:[a-f0-9]{40}|[a-f0-9]{64})\r?\n/.test(stdout.slice(0, 66))) return "merge_conflict";
return "unknown";
}
/** Annotate the original thrown error in its restore scope; never change it. */
export async function withWorkspaceRestoreGitCommand<T>(command: WorkspaceRestoreGitCommand, operation: () => Promise<T>): Promise<T> {
const scope = activeDiagnostic.getStore();
if (!scope?.active || !GIT_COMMANDS.has(command)) return await operation();
const started = scope.sequence;
try {
return await operation();
} catch (error) {
// A ref transaction can fail in its nested branch-identity probe. Retain
// that more specific command, while later retries get fresh attribution.
if (scope.active && (scope.failures.get(error)?.sequence ?? -1) <= started) scope.failures.set(error, {
sequence: ++scope.sequence, gitCommand: command, gitFailureKind: gitFailureKind(command, error),
});
throw error;
}
}
/** Only fixed codes and bounded numbers may enter the company-readable run log. */
function diagnostic(error: unknown): ErrorDiagnostic {
const result: ErrorDiagnostic = { errorCode: "unknown" };
let current = error;
// SDKs wrap transport errors in a cause. Bound traversal, including cycles.
for (let depth = 0; depth < 4 && current && typeof current === "object"; depth++) {
const value = current as Record<string, unknown>;
const saved = wrappedDiagnostics.get(value);
const code = saved?.errorCode !== "unknown" && saved?.errorCode !== undefined
? saved.errorCode : readField(value, "code");
if (result.errorCode === "unknown" && typeof code === "string" && ERROR_CODES.has(code)) {
result.errorCode = code;
}
const status = saved?.httpStatus ?? boundedInteger(readField(value, "status"), 400, 599)
?? boundedInteger(readField(value, "statusCode"), 400, 599);
if (result.httpStatus === undefined && status !== undefined) {
result.httpStatus = status;
}
const exitCode = saved?.exitCode ?? boundedInteger(readField(value, "exitCode"), 1, 255) ?? boundedInteger(code, 1, 255);
if (result.exitCode === undefined && exitCode !== undefined) {
result.exitCode = exitCode;
}
current = readField(value, "cause");
}
return result;
}
/** Retain bounded fields across an existing error wrapper, without a raw cause. */
export function preserveWorkspaceRestoreErrorDiagnostic<T extends object>(wrapper: T, source: unknown): T {
wrappedDiagnostics.set(wrapper, diagnostic(source));
const scope = activeDiagnostic.getStore();
const failure = scope?.active ? scope.failures.get(source) : undefined;
if (failure) scope!.failures.set(wrapper, failure);
return wrapper;
}
/** Decode persisted adapter metadata; never copy arbitrary properties or text. */
export function sanitizeWorkspaceRestoreDiagnostic(value: unknown): WorkspaceRestoreDiagnostic | undefined {
if (!value || typeof value !== "object") return undefined;
const record = value as Record<string, unknown>;
const phase = readField(record, "phase");
if (phase !== "workspace" && phase !== "asset") return undefined;
const step = readField(record, "step");
const code = readField(record, "errorCode");
const httpStatus = boundedInteger(readField(record, "httpStatus"), 400, 599);
const exitCode = boundedInteger(readField(record, "exitCode"), 1, 255);
const gitCommand = readField(record, "gitCommand");
const gitFailureKind = readField(record, "gitFailureKind");
const hasGitCommand = phase === "workspace" && step === "git_integration"
&& typeof gitCommand === "string" && GIT_COMMANDS.has(gitCommand as WorkspaceRestoreGitCommand);
return {
phase,
...(typeof step === "string" && RESTORE_STEPS.has(step as WorkspaceRestoreStep)
? { step: step as WorkspaceRestoreStep } : {}),
errorCode: typeof code === "string" && ERROR_CODES.has(code) ? code : "unknown",
...(httpStatus !== undefined ? { httpStatus } : {}),
...(exitCode !== undefined ? { exitCode } : {}),
...(hasGitCommand ? {
gitCommand: gitCommand as WorkspaceRestoreGitCommand,
gitFailureKind: typeof gitFailureKind === "string" && GIT_FAILURE_KINDS.has(gitFailureKind as GitFailureKind)
? gitFailureKind as GitFailureKind : "unknown",
} : {}),
};
}
export function getWorkspaceRestoreDiagnostic(error: unknown): WorkspaceRestoreDiagnostic | undefined {
return error && typeof error === "object"
? sanitizeWorkspaceRestoreDiagnostic((diagnosticCapture.getStore() ?? restoreDiagnostics).get(error)) : undefined;
}
/** Preserve the scheduler-selected task's snapshot when errors share identity. */
export function recordWorkspaceRestoreDiagnostic(error: unknown, diagnostic: WorkspaceRestoreDiagnostic | undefined): void {
const safe = sanitizeWorkspaceRestoreDiagnostic(diagnostic);
if (error && typeof error === "object") {
const receipts = diagnosticCapture.getStore() ?? restoreDiagnostics;
if (safe) receipts.set(error, safe);
else receipts.delete(error);
}
const scope = activeDiagnostic.getStore();
if (scope?.active) {
if (safe?.step) scope.failures.set(error, {
sequence: ++scope.sequence, step: safe.step,
...(safe.gitCommand ? { gitCommand: safe.gitCommand, gitFailureKind: safe.gitFailureKind } : {}),
});
else scope.failures.delete(error);
}
}
/** Record only the failing step. Successful cleanup must not replace evidence. */
export async function withWorkspaceRestoreStep<T>(step: WorkspaceRestoreStep, operation: () => Promise<T>): Promise<T> {
const scope = activeDiagnostic.getStore();
if (!scope?.active) return await operation();
const started = scope.sequence;
try {
return await operation();
} catch (error) {
// A nested step owns its failure. A caught/retried error may be thrown again
// by a later step, so identity alone cannot identify the current failure.
if (scope.active) {
const failure = scope.failures.get(error);
if (failure && failure.sequence > started) {
failure.step ??= step;
} else {
scope.failures.set(error, { sequence: ++scope.sequence, step });
}
}
throw error;
}
}
/** Add evidence without changing the thrown error, restore policy, or task ordering. */
export async function withWorkspaceRestoreDiagnostics<T>(
phase: RestorePhase,
operation: () => Promise<T>,
onProgress?: RuntimeProgressSink,
onDiagnostic?: (diagnostic: WorkspaceRestoreDiagnostic) => void,
): Promise<T> {
// A nested repository restore propagates to its enclosing workspace task.
// That task owns the diagnostic. Independent parallel tasks retain their own
// async scopes, so two failed tasks still produce two diagnostic lines.
const parent = activeDiagnostic.getStore();
const scope: DiagnosticScope = { active: true, sequence: 0, failures: new Map() };
return await activeDiagnostic.run(scope, async () => {
try {
return await operation();
} catch (error) {
const failure = scope.failures.get(error);
const step = failure?.step;
const fields = { phase, ...(step ? { step } : {}), ...diagnostic(error),
...(phase === "workspace" && step === "git_integration" && failure?.gitCommand
? { gitCommand: failure.gitCommand, gitFailureKind: failure.gitFailureKind } : {}),
};
recordWorkspaceRestoreDiagnostic(error, fields);
if (parent?.active && step) parent.failures.set(error, { ...failure, sequence: ++parent.sequence, step });
try {
onDiagnostic?.({ ...fields });
} catch { /* Diagnostic consumers cannot replace a restore failure. */ }
try {
if (!parent?.active) await onProgress?.(`[paperclip] Workspace restore diagnostic: ${JSON.stringify(fields)}\n`);
} catch {
// A broken log sink must not replace a restore failure or relax its safety classification.
}
throw error;
} finally {
scope.active = false;
scope.failures.clear();
}
});
}