mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-11 05:31:46 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Managed runs copy a selected workspace to an execution environment and restore its changes. > - Git snapshots select the files for that copy and for later recovery. > - A fixed output limit stops large generated trees before the run can start. > - Increasing the limit still keeps the complete filename lists in memory. > - This pull request stores those lists and merge baselines in disk manifests. > - Large snapshots can now complete with bounded filename buffers and explicit failure handling. ## Linked Issues or Issue Description Refs #14194. This is the streaming follow-up to the merged 32 MiB limit fix. Related: #13619 and #11621 cover workspace scan admission and demand. This change keeps the shared scheduler and changes the snapshot data path. ## What Changed - Stream changed, untracked, deleted, and ignored paths through the shared scheduler and the standalone adapter path. - Use SQLite manifests for file selection, duplicate removal, ignored-path lookup, baseline capture, and merge lookup. - Set a configurable 30-minute snapshot deadline. Keep the existing interactive scan deadlines. - Wait for each child process and pending sink write before removing temporary storage after failure or cancellation. - Use NUL archive lists and bounded deletion batches. Preserve unusual names, explicit selection, nested repositories, and source root checks. - Store manifest references in native recovery format v2. Check their location and digest before recovery reads. Keep v1 descriptors readable. - Remove temporary manifests at lifecycle completion. Use fixed-size temporary copy names for long basenames. - Admit each manifest with a SQLite page allowance based on current disk capacity. Keep a configurable free-space reserve and fail explicitly when either limit is reached. - Preserve a host file that replaces a directory deleted by the sandbox, and continue the rest of the restore. ## Verification - Current head `5ab622ec43cd16d35429d79dedee6a5d8e3d2df2` has 54 successful checks/statuses and two skipped Storybook jobs. No checks failed or remain pending. - [CI passed](https://github.com/paperclipai/paperclip/actions/runs/36318966368): typecheck, build, all test shards, E2E, Rust checks, and the aggregate verify job. - [Greptile is 5/5](https://github.com/paperclipai/paperclip/pull/14253#issuecomment-5855670338) on the current head. All four review threads are resolved. Security checks passed. - 229 focused tests passed across Git sync, runtime staging, merge, manifest integrity, native recovery, and the scheduler (214 adapter/runtime tests and 15 scheduler tests). - A real 40,000-file fixture produces 43,428,890 filename bytes. The original standalone and scheduled scans fail. The new test passes all four filename paths, complete staging, exclusion of late files, unusual names, and deletion replay. - Recovery tests reject changed bytes, symlinks, and paths outside the controller state directory. Adapter-utils typecheck passed. - A test executor returned buffered output and caused two retry integration failures. The fixture now uses the shared streaming scheduler. All 13 tests passed with `corepack pnpm exec vitest run server/src/__tests__/heartbeat-project-repositories.test.ts`. The same CI shard now passes. - Ran `pnpm -r typecheck`, `pnpm test:run`, and `pnpm build` locally. Each full local command hit SIGKILL/exit 137 in the 4 GiB container. These local commands did not pass. The current-head CI gates above provide the full verification. - A real-Git disk-capacity regression confirms a typed failure and removal of the incomplete manifest. Repeated writer attempts cannot exceed the permitted page count. - Follow-up real Daytona and separate staging qualification passed with the related archive validator (#14315) and exact-owner finalization fix (#14314). Three successive turns copied back all 60,000 files with 39,828,890 filename bytes and five unusual names. Independent host inventories verified every file and the pinned Git HEAD. Native, provider, session, and process identities stayed fixed; no retry remained. The task reached Done, and its browser-downloaded final proof matched exactly. The reusable regression is #14316, including an assertion of the effective environment idle policy. ## Risks - SQLite manifests use disk space. Each receives one quarter of the available capacity above the host reserve at creation. The reserve defaults to 256 MiB and has a 64 MiB configuration minimum. Disk capacity, filesystem quotas, per-path limits, Git resource use, and execution deadlines remain limits. - Each path and sink chunk has a 64 KiB limit. SQLite connections use a 1 MiB page cache. Invalid or incomplete records fail explicitly. - Restore transport keeps fixed and configured archive exclusions. A remotely created Git-ignored file can be transferred, but the host merge excludes it through the manifest. - Provider archive buffers, Git and tar memory, repository metadata, legacy v1 arrays, and the separate referenced-source resolver retain their own limits. Existing provider safety validators still buffer textual tar listings: Daytona allows 32 MiB and Kubernetes allows 64 MiB. These separate transport limits can stop a sufficiently large restore before merge. This change does not claim bounded total process memory or unlimited transport size. - New descriptors use v2. Existing v1 recovery remains supported; a downgrade cannot read v2 descriptors. ## Model Used OpenAI GPT-6 through Codex. The exact deployment ID and context limit are not exposed in this run. The agent used code editing, terminal execution, tests, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
94 lines
4.3 KiB
TypeScript
94 lines
4.3 KiB
TypeScript
import { spawn, type ChildProcess } from "node:child_process";
|
|
import { WORKSPACE_STREAM_CHUNK_BYTES } from "./workspace-manifest.js";
|
|
|
|
export interface WorkspaceGitProcessInput {
|
|
cwd: string;
|
|
args: readonly string[];
|
|
env?: NodeJS.ProcessEnv;
|
|
signal?: AbortSignal;
|
|
timeoutMs: number;
|
|
killGraceMs?: number;
|
|
maxStdoutBytes: number;
|
|
maxStderrBytes: number;
|
|
onStdout?: (chunk: Buffer) => Promise<void> | void;
|
|
gitBinary?: string;
|
|
gitArgsPrefix?: readonly string[];
|
|
}
|
|
|
|
function failure(code: string, message: string, details: Record<string, unknown> = {}): Error {
|
|
return Object.assign(new Error(message), { code, details });
|
|
}
|
|
|
|
function signalProcess(child: ChildProcess, signal: NodeJS.Signals): void {
|
|
// The leader may have exited while a descendant still holds its pipes open.
|
|
if (process.platform !== "win32" && child.pid) {
|
|
try { process.kill(-child.pid, signal); return; } catch { /* direct child fallback */ }
|
|
}
|
|
try { child.kill(signal); } catch { /* close owns settlement */ }
|
|
}
|
|
|
|
/** Completion is a barrier for the process, its pipes, and the awaited sink. */
|
|
export async function runWorkspaceGitProcess(input: WorkspaceGitProcessInput): Promise<{ stdout: string; stderr: string }> {
|
|
if (input.signal?.aborted) throw failure("workspace_git_scan_cancelled", "Workspace Git scan was cancelled");
|
|
const child = spawn(input.gitBinary ?? "git", [...(input.gitArgsPrefix ?? []), "-C", input.cwd, ...input.args], {
|
|
cwd: input.cwd, env: input.env ?? process.env,
|
|
stdio: ["ignore", "pipe", "pipe"], detached: process.platform !== "win32", windowsHide: true,
|
|
});
|
|
let error: Error | null = null;
|
|
let killTimer: NodeJS.Timeout | undefined;
|
|
const terminate = (reason: Error) => {
|
|
if (error) return;
|
|
error = reason;
|
|
signalProcess(child, "SIGTERM");
|
|
killTimer = setTimeout(() => signalProcess(child, "SIGKILL"), input.killGraceMs ?? 250);
|
|
killTimer.unref();
|
|
};
|
|
const onAbort = () => terminate(failure("workspace_git_scan_cancelled", "Workspace Git scan was cancelled"));
|
|
input.signal?.addEventListener("abort", onAbort, { once: true });
|
|
if (input.signal?.aborted) onAbort();
|
|
const timeout = setTimeout(() => terminate(failure("workspace_git_scan_timeout", `Workspace Git scan timed out after ${input.timeoutMs}ms`, { timeoutMs: input.timeoutMs })), input.timeoutMs);
|
|
timeout.unref();
|
|
const closed = new Promise<{ code: number | null; signal: NodeJS.Signals | null }>((resolve) => {
|
|
child.once("error", (cause) => terminate(failure("workspace_git_scan_failed", "Workspace Git scan could not start", { cause: cause.message })));
|
|
child.once("close", (code, signal) => resolve({ code, signal }));
|
|
});
|
|
const collect = async (stream: NonNullable<typeof child.stdout>, limit: number, sink?: WorkspaceGitProcessInput["onStdout"]) => {
|
|
const chunks: Buffer[] = [];
|
|
let bytes = 0;
|
|
try {
|
|
for await (const raw of stream) {
|
|
const buffer = Buffer.isBuffer(raw) ? raw : Buffer.from(raw);
|
|
if (error) continue;
|
|
if (sink) {
|
|
for (let offset = 0; offset < buffer.length; offset += WORKSPACE_STREAM_CHUNK_BYTES) {
|
|
if (error) break;
|
|
await sink(buffer.subarray(offset, offset + WORKSPACE_STREAM_CHUNK_BYTES));
|
|
}
|
|
} else {
|
|
bytes += buffer.length;
|
|
if (bytes > limit) {
|
|
terminate(failure("workspace_git_scan_output_limit", "Workspace Git scan exceeded its output limit"));
|
|
} else chunks.push(buffer);
|
|
}
|
|
}
|
|
} catch (cause) {
|
|
terminate(failure("workspace_git_scan_failed", "Workspace Git scan stream failed", { cause: cause instanceof Error ? cause.message : String(cause) }));
|
|
}
|
|
return Buffer.concat(chunks).toString("utf8");
|
|
};
|
|
try {
|
|
const [outcome, stdout, stderr] = await Promise.all([
|
|
closed, collect(child.stdout!, input.maxStdoutBytes, input.onStdout), collect(child.stderr!, input.maxStderrBytes),
|
|
]);
|
|
if (error) throw error;
|
|
if (outcome.code !== 0) throw failure("workspace_git_scan_failed", "Workspace Git scan failed", {
|
|
exitCode: outcome.code, signal: outcome.signal, stderr: stderr.trim().slice(0, 1000),
|
|
});
|
|
return { stdout, stderr };
|
|
} finally {
|
|
clearTimeout(timeout);
|
|
if (killTimer) clearTimeout(killTimer);
|
|
input.signal?.removeEventListener("abort", onAbort);
|
|
}
|
|
}
|