mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 11:46:36 +01:00
fix(workspaces): recover full sandboxes through read-only export (#19465)
## Thinking Path > - Paperclip preserves agent work across failed runs and server restarts. > - A remote workspace must have a durable save before its source can be released. > - Portable capture normally writes a Git mirror and a full archive inside the sandbox. > - A full sandbox can prevent capture before the provider's streaming download starts. > - Retrying that write cannot create the required free space. > - This change transfers a checked standalone source and seals its archive on the controller. > - The source remains retained unless the existing durable save completes. ## Linked Issues or Issue Description Related: #18644 reduced temporary capture space. #18172 added the provider's disk-free directory transfer. This repair connects portable capture to that transfer when the archive itself cannot be created. **What happened?** Workspace capture returns `remote_workspace_sandbox_disk_full` after provider work has finished. File recovery repeats the same sandbox write. **Expected behavior** Recover a stable standalone workspace without requiring another full archive on the source disk. Preserve staged changes, untracked work, confined links and the original source. **Steps to reproduce** 1. Create a standalone Git repository with a commit, staged changes and an untracked result. 2. Use the existing real-command transport fixture. Inject ENOSPC into the sandbox helper write or archive write. 3. Run portable capture. Before this change it rejects the helper failure. 4. With the change, restore the saved archive after removing the fixture source. Check the commit, staged content, untracked result and Git connectivity. ## What Changed - Attempt one controller-side capture after an exact sandbox disk-full failure. - Read a source inventory without installing a helper. Use the existing provider bulk transfer with confined links and runtime exclusions. - Compare the transferred inventory with source fingerprints before and after sealing. - Reject copied Git pointers, alternates and administrative symlinks. Parse config as data without includes; allow only portable keys before controller Git normalization. - Preserve socket exclusions. Retain the source and remove incomplete controller staging after failure. - Update execution semantics and focused regression coverage. No database migration or new execution authority. ## Verification - TDD red on base `2f35eb1080`: the new full-sandbox test failed with `remote_workspace_sandbox_disk_full` before production edits. - Green: 202 transport and archive tests passed before the additional Git configuration guards. At head `8541086cb5`, all 116 transport tests and the PostgreSQL durability acceptance pass. The final `e24e173ec9` exclusion guard passes those 116 cases plus its new Git-history regression across the full run and focused rerun. Environment: macOS arm64, Node 26.4.0; real Node/Git commands and filesystem transfer. - Review regressions were also red before their fixes: an internal Git config symlink bypassed the check, and `core.attributesFile` could select a controller path. Git administrative symlinks and nonportable config now fail before normalization. The focused safety selection passes 11 cases. - Cases cover helper and archive ENOSPC, commit/staged/untracked continuity after source removal, confined links, sockets, changed source, corrupt or extra copied files, unsafe Git metadata, retained diagnostic text and failed download. - Real PostgreSQL acceptance passes: normal workspace save recovers one sandbox ENOSPC, commits the encrypted snapshot, clears its writer, loses the original allocation, then independently reloads the saved result twice. The first acceptance attempt used an incorrect expected status (`available`); the documented state is `committed`. No production change was needed for that correction. - Review regressions for untracked Python/Node environments and excessive Git roots failed before their fixes. Dependency exclusions now match normal capture. Controller parsing starts only after whole-tree/repository budgets and layout checks; refspec/settings budgets and a shared two-minute verification/normalization deadline bound further work. The excessive-root test verifies zero Git subprocesses. - Real provider-boundary red/green: Daytona direct tar and capacity-fallback PTY transfers first dropped `._result.txt`. The new optional mapping contract preserves all user names and excludes only current transfer scratch. Kubernetes implements the same contract. Explicit caller exclusions and archive validation remain in force. - Standalone Linux qualification on Node 24.22.2/Debian bookworm: fresh shared/SDK build; Daytona and Kubernetes typecheck and build; Daytona file-sync/capacity/diagnostics 68/68 pass; Kubernetes file-sync 31/31 pass, including the actual `/proc` case. Qualification was refreshed again on the final `be6f7feacc` source tree: current shared/SDK rebuilt; all 815 staged source/config/manifest hashes match the delivered tree before and after testing. An initial container run as root bypassed a read-only file test; the same source passes as uid 1000. Installing zstd enabled eight initially skipped cases. - A further review regression proved `.git/pyvenv.cfg` could exclude Git history as a dependency. The inventory now handles `.git` before dependency filtering, matching normal capture. The full capture/restore test verifies the original commit and saved work. The first green attempt incorrectly compared command-result object identity; correcting the test to compare stdout passed with no production change. - Further red/green regressions cover read-only private-copy cleanup, a regular file replaced by a FIFO before open, case aliases of Git administrative metadata, and permission preservation under restrictive umask. Nonblocking no-follow file handles verify identity before and after reading. Cleanup changes only private controller directories. Provider extraction preserves ordinary permissions, refuses special mode bits before writing, and does not adopt source ownership, ACLs, attributes or flags. Final macOS Daytona tests pass 65/65; the final Linux qualification above was repeated on these exact provider sources. - Focused strict TypeScript compilation of the inventory and archive modules passes. - The first sandboxed archive run failed three Unix socket fixtures with EPERM. The permitted run outside that filesystem sandbox passed all tests. - Initial full local server typecheck could not resolve the unbuilt Runner package and produced cascading errors. Exact-head `e24e173ec9` CI typecheck, build, Runner and browser checks pass. That CI attempt failed two server shards: the unchanged retained-cleanup backlog test timed out at 15 seconds and passes locally unchanged; ten cache-reclaim fixture oracles failed because directory fallback consumed their sealed-archive counters/faults. Final commit `caed4e4f83` isolates the intended unavailable-directory boundary while preserving archive counts, recovery status, integrity and no-second-reclamation assertions. Those ten failures reproduced locally before the fixture edit. The full affected file passes 394/395; its sole other HTTP socket-hang-up case passes unchanged in an isolated rerun. This is combined coverage of all 395 cases, not a single complete green run. The final branch integrates master `53645a0104` at `98ea07225592930cbb0d18bcf250bc64f6bb13b4`, preserving upstream workspace phase timings. After integration: 117 transport cases, the PostgreSQL durability acceptance, the existing object-storage/timing regression, and 12 affected cache cases pass. The expanded recovery file now has 421 cases. Final-head `be6f7feacc45c1e32a26a2417e973f1e3752fd8d` CI passes all 55 applicable checks (three skipped, one neutral review), including the expanded recovery suite, typecheck, builds and Runner coverage. [CI result](https://github.com/paperclipai/paperclip/actions/runs/42251345554). - Provider exclusion regressions were red through four real Linux transfer paths: Daytona direct archive, direct safe-link repair, capacity PTY repair, and Kubernetes streaming. A root socket and nested regular file share a basename; literal glob-character files/subtrees also survive. Preserve mode now anchors tar exclusions as literal root-relative paths and escapes `find -path` pruning. Default callers keep glob semantics. Receipt-count regressions were also red before correcting preserved-output counting. All 31 Kubernetes and 68 Daytona tests, builds and typechecks pass on the final source with 815 matching inputs. - The final semantics documentation explicitly records special mode bits, the stricter config allowlist (including innocuous unlisted keys), and provider qualification limited to Daytona/Kubernetes; these address the optional current-head review observations without broadening controller Git authority. - No Daytona credential is available in this session. Provider transfer is a local filesystem boundary in these tests. Hosted task recovery and rollout remain unverified. ## Risks - Recovery reads the source repeatedly and transfers the standalone tree. It can take longer than normal capture. - Linked worktrees, initialized submodules, Git administrative symlinks, nonportable Git configuration, special mode bits and alternate object stores remain retained if the normal capture cannot finish. This fallback does not follow their pointers onto the controller. - A provider without usable directory export, a changed source, or insufficient controller space cannot produce a receipt. The original source stays retained. - The server, SDK and affected provider plugins must be built from this revision together. An older provider can fail inventory verification on names it omits; it cannot publish an incomplete save. No merge, publication or deployment is included. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, code editing, shell tools and automated tests. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
55ea9cfa1b
commit
c6573962f8
13 files changed
+888
-44
No files matched your search
@@ -1015,8 +1015,45 @@ does not repack an already dissociated mirror. Each streamed file must match its
|
||||
recorded hash, size and mode and retain its filesystem identity while read.
|
||||
Capture rechecks the complete source inventory and Git state before returning a
|
||||
receipt. A changed source or cancelled capture removes its incomplete archive;
|
||||
an existing archive remains untouched. Temporary space must still hold the
|
||||
normalized Git metadata and the complete archive together.
|
||||
an existing archive remains untouched. Temporary sandbox space normally holds
|
||||
the normalized Git metadata and the complete archive together. An exact disk-full
|
||||
failure invokes one read-only export fallback. It fingerprints a standalone
|
||||
source, transfers its files through the provider's existing bulk channel, and
|
||||
checks the received inventory and the source again before sealing a portable
|
||||
archive on the controller. It rechecks the source after sealing. No source files
|
||||
or provider caches are removed. Linked Git worktrees, initialized submodules, alternate object stores,
|
||||
Git administrative symlinks, setuid/setgid/sticky entries, and configuration outside the portable key contract
|
||||
cannot use this fallback. Pure file-based config reads do not follow includes;
|
||||
controller Git normalization starts only after every config passes that contract.
|
||||
They remain retained if their normal archive cannot be saved. Transfer failures
|
||||
or changed files retain the original disk-full error and do not produce a
|
||||
preservation receipt.
|
||||
|
||||
The fallback's `portableKey` allowlist in
|
||||
`server/src/services/remote-workspace-readonly-inventory.ts` is authoritative and
|
||||
intentionally stricter than normal capture's `portableGitSettings` sanitizer.
|
||||
Normal capture drops unrelated settings; the fallback rejects every unlisted key
|
||||
before controller Git can read the copied source configuration. This includes
|
||||
otherwise ordinary settings such as `pull.rebase`, `commit.gpgsign`, `fetch.prune`,
|
||||
`extensions.worktreeConfig`, and `remote.origin.mirror`.
|
||||
|
||||
The directory transfer requests `preserveSourceEntries` to preserve ordinary
|
||||
filenames and permissions. Its exclusions are literal root-relative paths,
|
||||
including directory descendants; ordinary transfers keep glob exclusions. Daytona and
|
||||
Kubernetes implement and qualify this mapping contract. Other providers have not
|
||||
been qualified for this fallback; inventory verification still prevents a
|
||||
provider that omits or alters entries from publishing an incomplete save.
|
||||
|
||||
The fallback applies the normal untracked dependency exclusions before inspecting
|
||||
links inside reinstallable environments. Tracked environment files remain part of
|
||||
the save. Its outbound mapping requests `preserveSourceEntries`; providers retain
|
||||
ordinary `._*` and scratch-like user filenames while excluding only the current
|
||||
transfer's own scratch paths and the explicit source exclusions. Older providers
|
||||
that omit files fail inventory verification rather than publish an incomplete save.
|
||||
Fallback inventory is limited to 100,000 entries, 64 standalone repositories,
|
||||
1 MiB of configuration, 1,024 settings and 256 refspec/merge entries. Controller
|
||||
verification and archive normalization share a two-minute cancellation deadline.
|
||||
These limits reject excessive work before publication and keep the source retained.
|
||||
|
||||
Shallow and partial Git checkouts retain their available object files instead
|
||||
of asking a mirror clone to transfer promised or index-only objects. Capture
|
||||
|
||||
@@ -470,8 +470,17 @@ export interface SandboxSyncFileMapping {
|
||||
targetPath: string;
|
||||
kind: "file" | "directory";
|
||||
mode?: number;
|
||||
/** Directory glob patterns; literal root-relative paths when preserving source entries. */
|
||||
exclude?: string[];
|
||||
followSymlinks?: boolean;
|
||||
/**
|
||||
* Outbound directory transfers preserve source entry names, including `._*`
|
||||
* and provider scratch-like names. In this mode `exclude` contains literal
|
||||
* root-relative paths; providers omit only those paths and scratch paths
|
||||
* created by this exact transfer. Archive member and symlink validation stay
|
||||
* unchanged. Ignored for syncIn/files.
|
||||
*/
|
||||
preserveSourceEntries?: boolean;
|
||||
/**
|
||||
* Advisory read-write intent for the sandbox target. `"rw"` marks a target the
|
||||
* agent may change and keep; `"ro"` marks a read-only tree. An absent value
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { createServer } from "node:net";
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import { promises as fs } from "node:fs";
|
||||
import os from "node:os";
|
||||
@@ -12,7 +13,8 @@ vi.mock("./plugin.js", async () => {
|
||||
import { performSyncOut } from "./file-sync.js";
|
||||
|
||||
const roots: string[] = [];
|
||||
afterEach(async () => { await Promise.all(roots.splice(0).map(root => fs.rm(root, { recursive: true, force: true }))); });
|
||||
const literalSocketCleanup: Array<() => Promise<void>> = [];
|
||||
afterEach(async () => { await Promise.all(literalSocketCleanup.splice(0).map(close => close())); vi.unstubAllEnvs(); await Promise.all(roots.splice(0).map(root => fs.rm(root, { recursive: true, force: true }))); });
|
||||
const executable = (names: string[], version: string) => names.map(name => {
|
||||
const resolved = spawnSync("/bin/sh", ["-c", 'command -v "$1"', "sh", name], { encoding: "utf8" }).stdout?.trim();
|
||||
return resolved && spawnSync(resolved, ["--version"], { encoding: "utf8" }).stdout?.includes(version) ? resolved : undefined;
|
||||
@@ -49,7 +51,11 @@ while True:
|
||||
data = data[os.write(dst, data):]
|
||||
`;
|
||||
|
||||
it.skipIf(Object.values(requiredTools).some(tool => !tool)).each(["success", "producer failure", "sparse archive over budget"])("preserves source bytes under archive capacity failure: %s", async scenario => {
|
||||
it.skipIf(Object.values(requiredTools).some(tool => !tool)).each(["success", "producer failure", "sparse archive over budget", "preserve source entries", "preserve entries direct archive", "preserve entries GNU host archive", "preserve modes native host", "preserve modes GNU host", "preserve special file native host", "preserve special directory GNU host", "preserve literal direct archive", "preserve literal direct repair", "preserve literal capacity repair"])("preserves source bytes under archive capacity failure: %s", async scenario => {
|
||||
const preserveSourceEntries = scenario.startsWith("preserve");
|
||||
const literal = scenario.startsWith("preserve literal");
|
||||
const directArchive = scenario === "preserve entries direct archive" || scenario.startsWith("preserve literal direct");
|
||||
const unsafeLink = scenario !== "preserve literal direct archive";
|
||||
const producerFails = scenario === "producer failure";
|
||||
const sparseOverBudget = scenario === "sparse archive over budget";
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "daytona-capacity-"));
|
||||
@@ -63,28 +69,73 @@ it.skipIf(Object.values(requiredTools).some(tool => !tool)).each(["success", "pr
|
||||
await fs.symlink(requiredTools[name]!, path.join(bin, name));
|
||||
}
|
||||
const tar = requiredTools.tar!;
|
||||
const specialMode = scenario.startsWith("preserve special");
|
||||
const modeRegression = scenario.startsWith("preserve modes") || specialMode;
|
||||
if (producerFails) {
|
||||
await fs.writeFile(path.join(bin, "tar"), `#!/bin/sh\n'${tar}' "$@"\nstatus=$?\ncase " $* " in *" -f - "*) exit 23 ;; esac\nexit "$status"\n`, { mode: 0o755 });
|
||||
} else if (modeRegression) {
|
||||
await fs.writeFile(path.join(bin, "tar"), `#!/bin/sh\nexec '${tar}' --owner=12345 --group=12345 "$@"\n`, { mode: 0o755 });
|
||||
} else {
|
||||
await fs.symlink(tar, path.join(bin, "tar"));
|
||||
}
|
||||
if (specialMode) {
|
||||
await fs.mkdir(output);
|
||||
await fs.writeFile(path.join(output, "retained.txt"), "existing host work");
|
||||
}
|
||||
if (producerFails || sparseOverBudget) {
|
||||
await fs.mkdir(output);
|
||||
await fs.writeFile(path.join(output, "result.bin"), "newer host work");
|
||||
}
|
||||
const env = { ...process.env, PATH: `${bin}:${process.env.PATH}` };
|
||||
if (scenario === "preserve entries GNU host archive") vi.stubEnv("PATH", env.PATH);
|
||||
if (modeRegression) {
|
||||
const hostBin = path.join(root, "host-bin");
|
||||
await fs.mkdir(hostBin);
|
||||
const hostTar = scenario.endsWith("GNU host") ? tar : executable(["tar"], "bsdtar") ?? tar;
|
||||
await fs.writeFile(path.join(hostBin, "tar"), `#!/bin/sh\numask 077\nexec '${hostTar}' "$@"\n`, { mode: 0o755 });
|
||||
vi.stubEnv("PATH", `${hostBin}:${process.env.PATH}`);
|
||||
}
|
||||
const bytes = Buffer.alloc(5 * 1024 * 1024, sparseOverBudget ? 0 : "saved work\n");
|
||||
await fs.writeFile(path.join(remote, "result.bin"), sparseOverBudget ? "" : bytes, { mode: 0o600 });
|
||||
if (sparseOverBudget) await fs.truncate(path.join(remote, "result.bin"), bytes.length);
|
||||
await fs.symlink("/etc/passwd", path.join(remote, "unsafe-link"));
|
||||
if (unsafeLink) await fs.symlink("/etc/passwd", path.join(remote, "unsafe-link"));
|
||||
await fs.symlink("result.bin", path.join(remote, "safe-link"));
|
||||
if (preserveSourceEntries) {
|
||||
await fs.writeFile(path.join(remote, "excluded.txt"), "excluded by caller");
|
||||
await fs.mkdir(path.join(remote, "nested"));
|
||||
if (modeRegression) await fs.chmod(path.join(remote, "nested"), 0o775);
|
||||
for (const name of ["._result.txt", ".paperclip-upload-result", "nested/._result.txt", "nested/.paperclip-upload-result"]) {
|
||||
await fs.writeFile(path.join(remote, name), `user work: ${name}`);
|
||||
if (modeRegression) await fs.chmod(path.join(remote, name), 0o664);
|
||||
}
|
||||
}
|
||||
if (literal) {
|
||||
const socket = createServer();
|
||||
await new Promise<void>((resolve, reject) => { socket.once("error", reject); socket.listen(path.join(remote, "ipc"), resolve); });
|
||||
// Closing unlinks the socket, so retain it throughout the actual export.
|
||||
const close = () => new Promise<void>(resolve => socket.close(() => resolve()));
|
||||
literalSocketCleanup.push(close);
|
||||
for (const name of ["nested/ipc", "nested/omit[ab]*?", "omitaZZx", "nested/skip[ab]*?/saved.txt"]) {
|
||||
await fs.mkdir(path.dirname(path.join(remote, name)), { recursive: true });
|
||||
await fs.writeFile(path.join(remote, name), `literal work: ${name}`);
|
||||
}
|
||||
await fs.writeFile(path.join(remote, "omit[ab]*?"), "excluded literal file");
|
||||
await fs.mkdir(path.join(remote, "skip[ab]*?"));
|
||||
await fs.writeFile(path.join(remote, "skip[ab]*?/excluded.txt"), "excluded subtree");
|
||||
}
|
||||
if (specialMode) {
|
||||
const specialPath = path.join(remote, scenario.includes("directory") ? "nested" : "._result.txt");
|
||||
const specialBits = scenario.includes("directory") ? 0o1775 : 0o1664;
|
||||
await fs.chmod(specialPath, specialBits);
|
||||
expect((await fs.stat(specialPath)).mode & 0o7777).toBe(specialBits);
|
||||
}
|
||||
const failures: string[] = [];
|
||||
const sandbox = {
|
||||
// A miniature provider-reported quota makes the sparse-file case bounded.
|
||||
disk: sparseOverBudget ? 1 / 4096 : 1,
|
||||
process: {
|
||||
executeCommand: async (command: string) => {
|
||||
const result = spawnSync("bash", ["-c", 'ulimit -f 128; exec sh -c "$1"', "bash", command], { env, encoding: "utf8" });
|
||||
const result = spawnSync("bash", ["-c", directArchive ? 'exec sh -c "$1"' : 'ulimit -f 128; exec sh -c "$1"', "bash", command], { env, encoding: "utf8" });
|
||||
if (result.status !== 0) failures.push(result.stderr);
|
||||
return { exitCode: result.status ?? 1, result: result.stdout + result.stderr };
|
||||
},
|
||||
@@ -111,9 +162,18 @@ it.skipIf(Object.values(requiredTools).some(tool => !tool)).each(["success", "pr
|
||||
deleteFile: (file: string) => fs.rm(file, { force: true }),
|
||||
},
|
||||
};
|
||||
const sync = performSyncOut({ sandbox: sandbox as never, remoteDir: remote, timeoutSeconds: 30,
|
||||
operations: [{ operationId: "export", files: [{ sourcePath: remote, targetPath: output, kind: "directory" }] }],
|
||||
const sync = performSyncOut({ sandbox: sandbox as never, remoteDir: directArchive ? root : remote, timeoutSeconds: 30,
|
||||
operations: [{ operationId: "export", files: [{ sourcePath: remote, targetPath: output, kind: "directory", preserveSourceEntries, exclude: literal ? ["ipc", "omit[ab]*?", "skip[ab]*?"] : preserveSourceEntries && !directArchive ? ["excluded.txt"] : undefined }] }],
|
||||
});
|
||||
if (specialMode) {
|
||||
await expect(sync).rejects.toThrow("special mode bits");
|
||||
expect(await fs.readdir(output)).toEqual(["retained.txt"]);
|
||||
expect(await fs.readFile(path.join(output, "retained.txt"), "utf8")).toBe("existing host work");
|
||||
expect((await fs.stat(path.join(remote, scenario.includes("directory") ? "nested" : "._result.txt"))).mode & 0o7777)
|
||||
.toBe(scenario.includes("directory") ? 0o1775 : 0o1664);
|
||||
expect((await fs.readFile(path.join(remote, "result.bin"))).equals(bytes)).toBe(true);
|
||||
return;
|
||||
}
|
||||
if (producerFails || sparseOverBudget) {
|
||||
await expect(sync).rejects.toThrow(sparseOverBudget ? "sandbox disk budget" : "without a successful archive receipt");
|
||||
expect(await fs.readFile(path.join(output, "result.bin"), "utf8")).toBe("newer host work");
|
||||
@@ -121,15 +181,39 @@ it.skipIf(Object.values(requiredTools).some(tool => !tool)).each(["success", "pr
|
||||
expect((await fs.readFile(path.join(remote, "result.bin"))).equals(bytes)).toBe(true);
|
||||
return;
|
||||
}
|
||||
await sync;
|
||||
expect(failures.join("\n")).toMatch(/File\s?size limit exceeded/i);
|
||||
const result = await sync;
|
||||
if (literal) expect(result.operations[0].filesTransferred).toBe(11);
|
||||
if (!directArchive) expect(failures.join("\n")).toMatch(/File\s?size limit exceeded/i);
|
||||
expect((await fs.readFile(path.join(output, "result.bin"))).equals(bytes)).toBe(true);
|
||||
expect((await fs.readFile(path.join(remote, "result.bin"))).equals(bytes)).toBe(true);
|
||||
expect((await fs.stat(path.join(output, "result.bin"))).mode & 0o777).toBe(0o600);
|
||||
expect(await fs.readlink(path.join(output, "safe-link"))).toBe("result.bin");
|
||||
expect(await fs.readdir(output)).not.toContain("unsafe-link");
|
||||
expect(await fs.readlink(path.join(remote, "unsafe-link"))).toBe("/etc/passwd");
|
||||
expect((await fs.readdir(remote)).filter(name => name.startsWith(".paperclip-upload"))).toEqual([]);
|
||||
if (unsafeLink) expect(await fs.readlink(path.join(remote, "unsafe-link"))).toBe("/etc/passwd");
|
||||
if (preserveSourceEntries) {
|
||||
if (directArchive || literal) expect(await fs.readFile(path.join(output, "excluded.txt"), "utf8")).toBe("excluded by caller");
|
||||
else await expect(fs.stat(path.join(output, "excluded.txt"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(await fs.readFile(path.join(remote, "excluded.txt"), "utf8")).toBe("excluded by caller");
|
||||
for (const name of ["._result.txt", ".paperclip-upload-result", "nested/._result.txt", "nested/.paperclip-upload-result"]) {
|
||||
expect(await fs.readFile(path.join(output, name), "utf8")).toBe(`user work: ${name}`);
|
||||
expect(await fs.readFile(path.join(remote, name), "utf8")).toBe(`user work: ${name}`);
|
||||
if (modeRegression) {
|
||||
const restored = await fs.stat(path.join(output, name));
|
||||
expect(restored.mode & 0o777).toBe(0o664);
|
||||
expect(restored.uid).toBe(process.getuid!());
|
||||
}
|
||||
}
|
||||
}
|
||||
if (literal) {
|
||||
for (const name of ["nested/ipc", "nested/omit[ab]*?", "omitaZZx", "nested/skip[ab]*?/saved.txt"]) {
|
||||
expect(await fs.readFile(path.join(output, name), "utf8")).toBe(`literal work: ${name}`);
|
||||
expect(await fs.readFile(path.join(remote, name), "utf8")).toBe(`literal work: ${name}`);
|
||||
}
|
||||
for (const name of ["ipc", "omit[ab]*?", "skip[ab]*?"]) await expect(fs.lstat(path.join(output, name))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect((await fs.lstat(path.join(remote, "ipc"))).isSocket()).toBe(true);
|
||||
}
|
||||
if (modeRegression) expect((await fs.stat(path.join(output, "nested"))).mode & 0o777).toBe(0o775);
|
||||
expect((await fs.readdir(remote)).filter(name => name.startsWith(".paperclip-upload"))).toEqual(preserveSourceEntries ? [".paperclip-upload-result"] : []);
|
||||
}, 40_000);
|
||||
|
||||
it.skipIf(Object.values(requiredTools).some(tool => !tool)).each([
|
||||
|
||||
@@ -254,12 +254,17 @@ export function splitLinkEntryOnce(field: string, delimiter: string): { name: st
|
||||
*/
|
||||
class UnsafeOutboundArchiveError extends Error {}
|
||||
|
||||
function assertTarListingLineConfined(line: string): void {
|
||||
function assertTarListingLineConfined(line: string, rejectSpecialModes = false): void {
|
||||
if (line.trim().length === 0) return;
|
||||
const parsed = parseTarVerboseListingLine(line);
|
||||
if (!parsed) {
|
||||
throw new UnsafeOutboundArchiveError(`Daytona syncOut refusing tarball with an unparseable entry listing: ${line}`);
|
||||
}
|
||||
// Source preservation promises ordinary 0777 permissions, not host special
|
||||
// execution bits. Refuse before extraction; this must not trigger link repair.
|
||||
if (rejectSpecialModes && /[sStT]/.test(line.trimStart().slice(0, 10))) {
|
||||
throw new Error("Daytona syncOut source preservation does not support special mode bits");
|
||||
}
|
||||
const typeFlag = parsed.typeFlag;
|
||||
let name = parsed.rest;
|
||||
let linkTarget: string | null = null;
|
||||
@@ -302,12 +307,17 @@ const TAR_LISTING_TIMEOUT_MS = 120_000;
|
||||
export async function assertTarballEntriesConfined(
|
||||
archivePath: string,
|
||||
timeoutMs = TAR_LISTING_TIMEOUT_MS,
|
||||
rejectSpecialModes = false,
|
||||
): Promise<void> {
|
||||
// A valid large workspace can exceed execFile's buffer. Stream within both
|
||||
// aggregate admission quotas and per-entry/diagnostic memory bounds, checking
|
||||
// every entry before extraction. Keep bytes until a full line to preserve
|
||||
// UTF-8 characters split across pipe chunks.
|
||||
const child = spawn("tar", ["-tvf", archivePath], {
|
||||
// AppleDouble interpretation must also be disabled while validating, or
|
||||
// libarchive hides `._*` members that the preserving extractor will write.
|
||||
const rawMacEntries = rejectSpecialModes && process.platform === "darwin"
|
||||
&& (await execFileAsync("tar", ["--version"])).stdout.includes("bsdtar");
|
||||
const child = spawn("tar", ["-tvf", archivePath, ...(rawMacEntries ? ["--options=!mac-ext"] : [])], {
|
||||
env: { ...process.env, COPYFILE_DISABLE: "1" },
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
@@ -322,7 +332,7 @@ export async function assertTarballEntriesConfined(
|
||||
if (++entries > TAR_LISTING_MAX_ENTRIES) {
|
||||
throw recordEnvironmentSyncError(new Error("Daytona syncOut tar listing entry limit exceeded (250000)"), { transferFailureKind: "listing_entry_limit" });
|
||||
}
|
||||
try { assertTarListingLineConfined(line.toString("utf8")); }
|
||||
try { assertTarListingLineConfined(line.toString("utf8"), rejectSpecialModes); }
|
||||
catch (error) {
|
||||
if (error instanceof UnsafeOutboundArchiveError) recordEnvironmentSyncError(error, { transferFailureKind: "unsafe_archive" });
|
||||
throw error;
|
||||
@@ -386,13 +396,22 @@ export async function assertTarballEntriesConfined(
|
||||
}
|
||||
}
|
||||
|
||||
async function extractHostTarball(input: { archivePath: string; localDir: string }): Promise<void> {
|
||||
async function extractHostTarball(input: { archivePath: string; localDir: string; preserveSourceEntries?: boolean }): Promise<void> {
|
||||
// The archive is sandbox-authored and untrusted: validate every member (and
|
||||
// link target) is confined before letting host-side tar write a single byte.
|
||||
await withEnvironmentSyncTransferStep("archive_validate", () => assertTarballEntriesConfined(input.archivePath));
|
||||
await withEnvironmentSyncTransferStep("archive_validate", () => assertTarballEntriesConfined(input.archivePath, TAR_LISTING_TIMEOUT_MS, input.preserveSourceEntries));
|
||||
await withEnvironmentSyncTransferStep("archive_extract", async () => {
|
||||
await fs.mkdir(input.localDir, { recursive: true });
|
||||
await execFileAsync("tar", ["-xf", input.archivePath, "-C", input.localDir], {
|
||||
// macOS tar otherwise consumes source files named `._*` as AppleDouble
|
||||
// metadata instead of writing their bytes. These are ordinary user entries.
|
||||
// Preserve mode also ignores the host umask without adopting archive owners,
|
||||
// ACLs, attributes or file flags from the sandbox.
|
||||
const hostTarIsBsd = input.preserveSourceEntries
|
||||
&& (await execFileAsync("tar", ["--version"])).stdout.includes("bsdtar");
|
||||
await execFileAsync("tar", ["-xf", input.archivePath, "-C", input.localDir,
|
||||
...(input.preserveSourceEntries ? ["-p", "--no-same-owner", "--no-acls", "--no-xattrs"] : []),
|
||||
...(hostTarIsBsd ? ["--no-fflags"] : []),
|
||||
...(hostTarIsBsd && process.platform === "darwin" ? ["--no-mac-metadata", "--options=!mac-ext"] : [])], {
|
||||
env: { ...process.env, COPYFILE_DISABLE: "1" },
|
||||
maxBuffer: 32 * 1024 * 1024,
|
||||
});
|
||||
@@ -1346,11 +1365,20 @@ async function syncOutDirectoryMapping(input: {
|
||||
return withHostTempDir(async (tmp) => {
|
||||
const remoteTar = path.posix.join(remoteDir, scratchName(".tar"));
|
||||
const remoteList = path.posix.join(remoteDir, scratchName(".list"));
|
||||
const excludes = ["._*", `${SCRATCH_PREFIX}*`, ...(mapping.exclude ?? [])];
|
||||
const excludeFlags = excludes
|
||||
.flatMap((entry) => [entry, `${entry.replace(/\/$/, "")}/*`])
|
||||
.map((entry) => `--exclude ${shellQuote(entry)}`)
|
||||
.join(" ");
|
||||
const scratchPaths = [remoteTar, remoteList].flatMap(file => {
|
||||
const relative = path.posix.relative(mapping.sourcePath, file);
|
||||
return relative && !relative.startsWith("../") && !path.posix.isAbsolute(relative)
|
||||
? [relative, `./${relative}`] : [];
|
||||
});
|
||||
const excludes = [...(mapping.preserveSourceEntries ? scratchPaths : ["._*", `${SCRATCH_PREFIX}*`]), ...(mapping.exclude ?? [])];
|
||||
// Preservation exclusions are literal paths relative to this source, not
|
||||
// basename patterns. Both archive naming forms occur in the repair path.
|
||||
const literalExcludes = [...new Set(excludes.map(entry => entry.replace(/^(?:\.\/)+/, "").replace(/\/+$/, "")))];
|
||||
const excludeFlags = (mapping.preserveSourceEntries ? "--anchored --no-wildcards " : "") +
|
||||
(mapping.preserveSourceEntries
|
||||
? literalExcludes.flatMap(entry => [entry, `./${entry}`])
|
||||
: excludes.flatMap(entry => [entry, `${entry.replace(/\/$/, "")}/*`]))
|
||||
.map(entry => `--exclude ${shellQuote(entry)}`).join(" ");
|
||||
// Tar the source in-sandbox (naming top-level entries so no "." self-entry is
|
||||
// embedded), reproducing the `followSymlinks` → `-h` mapping, then stream the
|
||||
// single archive back over the native bulk channel.
|
||||
@@ -1366,9 +1394,13 @@ async function syncOutDirectoryMapping(input: {
|
||||
// resolved targets remain inside this mapping. Nothing is deleted or
|
||||
// dereferenced. Host validation still checks the rebuilt archive, including
|
||||
// links changed by the sandbox between enumeration and tar creation.
|
||||
const prunePaths = excludes.flatMap((entry) => [
|
||||
`-path ${shellQuote(`./${entry}`)}`, `-path ${shellQuote(`*/${entry}`)}`,
|
||||
]).join(" -o ");
|
||||
// find -path also interprets glob characters, even inside shell quotes.
|
||||
// Pruning a literal directory already removes its complete subtree.
|
||||
const prunePaths = (mapping.preserveSourceEntries
|
||||
? literalExcludes.map(entry => `-path ${shellQuote(`./${entry.replace(/[\\*?[\]]/g, "\\$&")}`)}`)
|
||||
: excludes.flatMap(entry => [
|
||||
`-path ${shellQuote(`./${entry}`)}`, `-path ${shellQuote(`*/${entry}`)}`,
|
||||
])).join(" -o ");
|
||||
const filterLinks = [
|
||||
...canonicalizerPreamble(shellQuote(mapping.sourcePath)),
|
||||
'for _pc_link do',
|
||||
@@ -1381,7 +1413,7 @@ async function syncOutDirectoryMapping(input: {
|
||||
].join("\n");
|
||||
const confinedEntriesScript = (archive: string | null) => [
|
||||
`cd ${shellQuote(mapping.sourcePath)}`,
|
||||
`find . -mindepth 1 \\( ${prunePaths} \\) -prune -o -type l -exec sh -c ${shellQuote(filterLinks)} sh {} + -o \\( -type f -o -type d \\) -print0 ${archive ? `> ${shellQuote(remoteList)} && ` : "| "}` +
|
||||
`find . -mindepth 1 ${prunePaths ? `\\( ${prunePaths} \\) -prune -o ` : ""}-type l -exec sh -c ${shellQuote(filterLinks)} sh {} + -o \\( -type f -o -type d \\) -print0 ${archive ? `> ${shellQuote(remoteList)} && ` : "| "}` +
|
||||
`tar -c --no-xattrs --hard-dereference --no-recursion --null ${excludeFlags} -f ${shellQuote(archive ?? "-")} -T ${shellQuote(archive ? remoteList : "-")}`,
|
||||
].join(" && ");
|
||||
|
||||
@@ -1431,7 +1463,7 @@ async function syncOutDirectoryMapping(input: {
|
||||
}
|
||||
bytesTransferred += await withEnvironmentSyncTransferStep("archive_download", async () => (await fs.stat(localTar)).size);
|
||||
try {
|
||||
await extractHostTarball({ archivePath: localTar, localDir: mapping.targetPath });
|
||||
await extractHostTarball({ archivePath: localTar, localDir: mapping.targetPath, preserveSourceEntries: mapping.preserveSourceEntries });
|
||||
break;
|
||||
} catch (error) {
|
||||
if (!(error instanceof UnsafeOutboundArchiveError) || attempt > 0) throw error;
|
||||
@@ -1444,7 +1476,7 @@ async function syncOutDirectoryMapping(input: {
|
||||
// cleaned by withHostTempDir.
|
||||
await Promise.all([remoteTar, remoteList].map((file) => sandbox.fs.deleteFile(file).catch(() => undefined)));
|
||||
}
|
||||
const filesTransferred = await countHostFiles(mapping.targetPath, mapping.exclude);
|
||||
const filesTransferred = await countHostFiles(mapping.targetPath, mapping.preserveSourceEntries ? undefined : mapping.exclude);
|
||||
return { filesTransferred, bytesTransferred };
|
||||
});
|
||||
}
|
||||
|
||||
@@ -369,8 +369,11 @@ export function splitLinkEntryOnce(field: string, delimiter: string): { name: st
|
||||
* escapes the tree. Legitimate in-tree relative links are preserved. Parses the
|
||||
* `-tvf` verbose listing; any unparseable line fails closed.
|
||||
*/
|
||||
async function assertTarballEntriesConfined(archivePath: string): Promise<void> {
|
||||
const { stdout } = await execFileAsync("tar", ["-tvf", archivePath], {
|
||||
async function assertTarballEntriesConfined(archivePath: string, rejectSpecialModes = false): Promise<void> {
|
||||
// Validate the same raw `._*` members the preserving extractor will write.
|
||||
const rawMacEntries = rejectSpecialModes && process.platform === "darwin"
|
||||
&& (await execFileAsync("tar", ["--version"])).stdout.includes("bsdtar");
|
||||
const { stdout } = await execFileAsync("tar", ["-tvf", archivePath, ...(rawMacEntries ? ["--options=!mac-ext"] : [])], {
|
||||
env: { ...process.env, COPYFILE_DISABLE: "1" },
|
||||
maxBuffer: 64 * 1024 * 1024,
|
||||
});
|
||||
@@ -380,6 +383,11 @@ async function assertTarballEntriesConfined(archivePath: string): Promise<void>
|
||||
if (!parsed) {
|
||||
throw new Error(`Kubernetes syncOut refusing tarball with an unparseable entry listing: ${line}`);
|
||||
}
|
||||
// Source preservation promises 0777 modes only. Refuse special execution
|
||||
// bits before the host extractor can apply them, retaining every source file.
|
||||
if (rejectSpecialModes && /[sStT]/.test(line.trimStart().slice(0, 10))) {
|
||||
throw new Error("Kubernetes syncOut source preservation does not support special mode bits");
|
||||
}
|
||||
const typeFlag = parsed.typeFlag;
|
||||
let name = parsed.rest;
|
||||
let linkTarget: string | null = null;
|
||||
@@ -409,12 +417,21 @@ async function assertTarballEntriesConfined(archivePath: string): Promise<void>
|
||||
}
|
||||
}
|
||||
|
||||
async function extractHostTarball(input: { archivePath: string; localDir: string }): Promise<void> {
|
||||
async function extractHostTarball(input: { archivePath: string; localDir: string; preserveSourceEntries?: boolean }): Promise<void> {
|
||||
// The archive is sandbox-authored and untrusted: validate every member (and
|
||||
// link target) is confined before letting host-side tar write a single byte.
|
||||
await assertTarballEntriesConfined(input.archivePath);
|
||||
await assertTarballEntriesConfined(input.archivePath, input.preserveSourceEntries);
|
||||
await fs.mkdir(input.localDir, { recursive: true });
|
||||
await execFileAsync("tar", ["-xf", input.archivePath, "-C", input.localDir], {
|
||||
// macOS tar otherwise consumes source files named `._*` as AppleDouble
|
||||
// metadata instead of writing their bytes. These are ordinary user entries.
|
||||
// Preserve mode also ignores the host umask without adopting archive owners,
|
||||
// ACLs, attributes or file flags from the sandbox.
|
||||
const hostTarIsBsd = input.preserveSourceEntries
|
||||
&& (await execFileAsync("tar", ["--version"])).stdout.includes("bsdtar");
|
||||
await execFileAsync("tar", ["-xf", input.archivePath, "-C", input.localDir,
|
||||
...(input.preserveSourceEntries ? ["-p", "--no-same-owner", "--no-acls", "--no-xattrs"] : []),
|
||||
...(hostTarIsBsd ? ["--no-fflags"] : []),
|
||||
...(hostTarIsBsd && process.platform === "darwin" ? ["--no-mac-metadata", "--options=!mac-ext"] : [])], {
|
||||
env: { ...process.env, COPYFILE_DISABLE: "1" },
|
||||
maxBuffer: 64 * 1024 * 1024,
|
||||
});
|
||||
@@ -835,9 +852,13 @@ async function syncOutDirectoryMapping(input: {
|
||||
const { exec, mapping, remoteDir, timeoutMs, maxOutputBytes } = input;
|
||||
assertConfinedSandboxPath(remoteDir, mapping.sourcePath, "source");
|
||||
return withHostTempDir(async (tmp) => {
|
||||
const excludeFlags = ["._*", ...(mapping.exclude ?? [])]
|
||||
.map((entry) => `--exclude ${shQuote(entry)}`)
|
||||
.join(" ");
|
||||
// Preservation exclusions name literal source-relative paths. GNU tar's
|
||||
// default basename/glob matching could silently omit unrelated user files.
|
||||
const excludes = [...(mapping.preserveSourceEntries ? [] : ["._*"]), ...(mapping.exclude ?? [])];
|
||||
const excludeFlags = (mapping.preserveSourceEntries ? "--anchored --no-wildcards " : "") +
|
||||
(mapping.preserveSourceEntries
|
||||
? [...new Set(excludes.map(entry => entry.replace(/^(?:\.\/)+/, "").replace(/\/+$/, "")))].flatMap(entry => [entry, `./${entry}`])
|
||||
: excludes).map(entry => `--exclude ${shQuote(entry)}`).join(" ");
|
||||
const script = [
|
||||
...canonicalizerPreamble(shQuote(remoteDir)),
|
||||
// Confine the source dir through an open-then-verify pinned fd before taring.
|
||||
@@ -867,8 +888,8 @@ async function syncOutDirectoryMapping(input: {
|
||||
label: "syncOut directory transfer",
|
||||
});
|
||||
const bytesTransferred = (await fs.stat(localTar)).size;
|
||||
await extractHostTarball({ archivePath: localTar, localDir: mapping.targetPath });
|
||||
const filesTransferred = await countHostFiles(mapping.targetPath, mapping.exclude);
|
||||
await extractHostTarball({ archivePath: localTar, localDir: mapping.targetPath, preserveSourceEntries: mapping.preserveSourceEntries });
|
||||
const filesTransferred = await countHostFiles(mapping.targetPath, mapping.preserveSourceEntries ? undefined : mapping.exclude);
|
||||
return { filesTransferred, bytesTransferred };
|
||||
});
|
||||
}
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { createServer } from "node:net";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { promises as fs } from "node:fs";
|
||||
import os from "node:os";
|
||||
@@ -379,6 +380,77 @@ describe("kubernetes onEnvironmentSyncIn (native single-exec transfer)", () => {
|
||||
});
|
||||
|
||||
describe("kubernetes onEnvironmentSyncOut (native single-exec transfer)", () => {
|
||||
// The pod's pinned descriptor protocol depends on Linux /proc/self/fd.
|
||||
it.skipIf(process.platform !== "linux")("preserves source entry names when requested while retaining explicit exclusions", async () => {
|
||||
const remoteDir = await makeTmp("k8s-source-entry-"), hostOut = await makeTmp("k8s-source-output-");
|
||||
await fs.mkdir(path.join(remoteDir, "nested"));
|
||||
const names = ["._result.txt", ".paperclip-upload-result", "nested/._result.txt", "nested/.paperclip-upload-result"];
|
||||
await fs.chmod(path.join(remoteDir, "nested"), 0o775);
|
||||
for (const name of names) {
|
||||
await fs.writeFile(path.join(remoteDir, name), `user work: ${name}`);
|
||||
await fs.chmod(path.join(remoteDir, name), 0o664);
|
||||
}
|
||||
await fs.writeFile(path.join(remoteDir, "excluded.txt"), "excluded by caller");
|
||||
const { exec } = makeRealExec();
|
||||
await performSyncOut({ exec, remoteDir, timeoutMs: 30_000, operations: [{ operationId: "preserve", files: [{
|
||||
kind: "directory", sourcePath: remoteDir, targetPath: hostOut, preserveSourceEntries: true, exclude: ["excluded.txt"],
|
||||
}] }] });
|
||||
for (const name of names) {
|
||||
expect(await fs.readFile(path.join(hostOut, name), "utf8")).toBe(`user work: ${name}`);
|
||||
expect(await fs.readFile(path.join(remoteDir, name), "utf8")).toBe(`user work: ${name}`);
|
||||
expect((await fs.stat(path.join(hostOut, name))).mode & 0o777).toBe(0o664);
|
||||
}
|
||||
expect((await fs.stat(path.join(hostOut, "nested"))).mode & 0o777).toBe(0o775);
|
||||
await expect(fs.stat(path.join(hostOut, "excluded.txt"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
});
|
||||
|
||||
it.skipIf(process.platform !== "linux")("preserves literal excluded paths without excluding same basenames or glob matches", async () => {
|
||||
const remoteDir = await makeTmp("k8s-literal-"), hostOut = await makeTmp("k8s-literal-out-");
|
||||
await fs.mkdir(path.join(remoteDir, "nested"));
|
||||
const socket = createServer();
|
||||
await new Promise<void>((resolve, reject) => { socket.once("error", reject); socket.listen(path.join(remoteDir, "ipc"), resolve); });
|
||||
try {
|
||||
const names = ["nested/ipc", "nested/omit[ab]*?", "omitaZZx", "nested/skip[ab]*?/saved.txt"];
|
||||
for (const name of names) {
|
||||
await fs.mkdir(path.dirname(path.join(remoteDir, name)), { recursive: true });
|
||||
await fs.writeFile(path.join(remoteDir, name), `literal work: ${name}`);
|
||||
}
|
||||
await fs.writeFile(path.join(remoteDir, "omit[ab]*?"), "excluded literal file");
|
||||
await fs.mkdir(path.join(remoteDir, "skip[ab]*?"));
|
||||
await fs.writeFile(path.join(remoteDir, "skip[ab]*?/excluded.txt"), "excluded subtree");
|
||||
const { exec } = makeRealExec();
|
||||
const result = await performSyncOut({ exec, remoteDir, timeoutMs: 30_000, operations: [{ operationId: "literal", files: [{
|
||||
kind: "directory", sourcePath: remoteDir, targetPath: hostOut, preserveSourceEntries: true, exclude: ["ipc", "omit[ab]*?", "skip[ab]*?"],
|
||||
}] }] });
|
||||
expect(result.operations[0].filesTransferred).toBe(4);
|
||||
for (const name of names) {
|
||||
expect(await fs.readFile(path.join(hostOut, name), "utf8")).toBe(`literal work: ${name}`);
|
||||
expect(await fs.readFile(path.join(remoteDir, name), "utf8")).toBe(`literal work: ${name}`);
|
||||
}
|
||||
for (const name of ["ipc", "omit[ab]*?", "skip[ab]*?"]) await expect(fs.lstat(path.join(hostOut, name))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect((await fs.lstat(path.join(remoteDir, "ipc"))).isSocket()).toBe(true);
|
||||
} finally { await new Promise<void>(resolve => socket.close(() => resolve())); }
|
||||
});
|
||||
|
||||
it.skipIf(process.platform !== "linux").each(["file", "directory"])("refuses preserved %s special modes before host writes", async kind => {
|
||||
const remoteDir = await makeTmp("k8s-special-source-"), hostOut = await makeTmp("k8s-special-output-");
|
||||
await fs.mkdir(path.join(remoteDir, "nested"));
|
||||
await fs.writeFile(path.join(remoteDir, "result.txt"), "source work");
|
||||
await fs.writeFile(path.join(hostOut, "retained.txt"), "existing host work");
|
||||
const source = path.join(remoteDir, kind === "directory" ? "nested" : "result.txt");
|
||||
const mode = kind === "directory" ? 0o1775 : 0o1664;
|
||||
await fs.chmod(source, mode);
|
||||
expect((await fs.stat(source)).mode & 0o7777).toBe(mode);
|
||||
const { exec } = makeRealExec();
|
||||
await expect(performSyncOut({ exec, remoteDir, timeoutMs: 30_000, operations: [{ operationId: "preserve", files: [{
|
||||
kind: "directory", sourcePath: remoteDir, targetPath: hostOut, preserveSourceEntries: true,
|
||||
}] }] })).rejects.toThrow("special mode bits");
|
||||
expect(await fs.readdir(hostOut)).toEqual(["retained.txt"]);
|
||||
expect(await fs.readFile(path.join(hostOut, "retained.txt"), "utf8")).toBe("existing host work");
|
||||
expect((await fs.stat(source)).mode & 0o7777).toBe(mode);
|
||||
expect(await fs.readFile(path.join(remoteDir, "result.txt"), "utf8")).toBe("source work");
|
||||
});
|
||||
|
||||
it("streams all file mappings back over a SINGLE exec and reassembles them at host targets, preserving mode and per-operation counts", async () => {
|
||||
const remoteDir = await makeTmp("k8s-sandbox-");
|
||||
const hostOut = await makeTmp("k8s-hostout-");
|
||||
|
||||
@@ -810,13 +810,21 @@ export interface PluginSyncFileMapping {
|
||||
* shows the bytes only to code that already runs in that sandbox.
|
||||
*/
|
||||
mode?: number;
|
||||
/** Glob patterns to exclude when `kind` is `"directory"`. */
|
||||
/** Directory glob patterns; literal root-relative paths when preserving source entries. */
|
||||
exclude?: string[];
|
||||
/**
|
||||
* Symlink handling for `kind: "directory"` transfers. Falsy preserves symlinks
|
||||
* as links; `true` dereferences them to their target bytes. Mirrors tar's `-h`.
|
||||
*/
|
||||
followSymlinks?: boolean;
|
||||
/**
|
||||
* Outbound directory transfers preserve source entry names, including `._*`
|
||||
* and provider scratch-like names. In this mode `exclude` contains literal
|
||||
* root-relative paths; providers omit only those paths and scratch paths
|
||||
* created by this exact transfer. Archive member and symlink validation stay
|
||||
* unchanged. Ignored for syncIn/files.
|
||||
*/
|
||||
preserveSourceEntries?: boolean;
|
||||
/**
|
||||
* Advisory read-write intent for the sandbox target. `"rw"` means the author
|
||||
* expects the agent to change the bytes at the target and keep the change.
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
/** Runs unchanged in the sandbox and on its transferred, private controller copy. */
|
||||
export async function readWorkspaceInventory(root: string, controllerCopy = false, signal?: AbortSignal) {
|
||||
const fs = await import("node:fs/promises"), path = await import("node:path"), crypto = await import("node:crypto");
|
||||
const fail = (): never => { throw new Error("remote_workspace_readonly_capture_unsafe"); };
|
||||
const budget = (): never => { throw new Error("remote_workspace_readonly_capture_budget"); };
|
||||
const check = () => signal?.throwIfAborted();
|
||||
check();
|
||||
if (!path.isAbsolute(root) || await fs.realpath(root) !== root || !(await fs.lstat(root)).isDirectory()) fail();
|
||||
const digest = crypto.createHash("sha256"), exclude = [".paperclip-runtime"];
|
||||
const repositories: string[] = [], configs: string[] = [];
|
||||
let entries = 0, scannedEntries = 0, byteSize = 0, configBytes = 0;
|
||||
const gitRead = async (cwd: string, args: string[]) => {
|
||||
check();
|
||||
const { execFile } = await import("node:child_process");
|
||||
return new Promise<string>((resolve, reject) => execFile("git", ["-c", "core.fsmonitor=false", "-c", "core.hooksPath=/dev/null", "-C", cwd, ...args], {
|
||||
signal, timeout: 10_000, maxBuffer: 64 * 1024 * 1024,
|
||||
env: { PATH: process.env.PATH, GIT_CONFIG_NOSYSTEM: "1", GIT_CONFIG_GLOBAL: "/dev/null", GIT_CONFIG_COUNT: "0",
|
||||
GIT_OPTIONAL_LOCKS: "0", GIT_NO_LAZY_FETCH: "1" },
|
||||
}, (error, result) => error ? reject(new Error("remote_workspace_readonly_capture_unsafe")) : resolve(result)));
|
||||
};
|
||||
const visit = async (relative: string, parentRepository: string | null = null) => {
|
||||
check();
|
||||
const directory = path.join(root, relative), before = await fs.lstat(directory);
|
||||
if (!before.isDirectory() || before.isSymbolicLink() || await fs.realpath(directory) !== directory) fail();
|
||||
const names: string[] = [];
|
||||
for await (const entry of await fs.opendir(directory)) {
|
||||
check(); if (++scannedEntries > 100_000) budget(); names.push(entry.name);
|
||||
}
|
||||
const inGit = relative.split("/").includes(".git");
|
||||
const repository = !inGit && names.includes(".git") ? directory : parentRepository;
|
||||
if (!inGit && names.includes(".git")) {
|
||||
const entry = await fs.lstat(path.join(directory, ".git"));
|
||||
if (!entry.isDirectory() || entry.isSymbolicLink()) fail();
|
||||
repositories.push(directory); if (repositories.length > 64) budget();
|
||||
// The sandbox uses the same tracked-dependency rule as normal capture.
|
||||
// Controller verification never invokes repository-aware Git here.
|
||||
if (!controllerCopy && await fs.realpath((await gitRead(directory, ["rev-parse", "--show-toplevel"])).trim()) !== directory) fail();
|
||||
}
|
||||
for (const name of names.sort()) {
|
||||
check();
|
||||
const next = relative ? `${relative}/${name}` : name;
|
||||
if (next === ".paperclip-runtime") continue;
|
||||
if (!name || name === "." || name === ".." || /[\\\u0000-\u001f\u007f]/u.test(name)) fail();
|
||||
const file = path.join(root, next), stat = await fs.lstat(file);
|
||||
// A copied worktree pointer or alternate could resolve against the
|
||||
// controller's filesystem. Only standalone, self-contained Git metadata
|
||||
// can use this fallback; the sandbox remains retained otherwise.
|
||||
const parts = next.split("/"), git = parts.indexOf(".git");
|
||||
if (name === ".git" && !stat.isDirectory()) fail();
|
||||
// Git follows configuration and administrative links itself. A link
|
||||
// inside this tree must not bypass the controller config-file check.
|
||||
if (git >= 0 && stat.isSymbolicLink()) fail();
|
||||
const administrationPath = parts.slice(git + 1).join("/");
|
||||
const foldedAdministrationPath = administrationPath.toLowerCase();
|
||||
// A case-sensitive sandbox may be copied onto a case-insensitive host.
|
||||
// Reject aliases before Git can interpret them as controller metadata.
|
||||
if (git >= 0 && ((foldedAdministrationPath === "config" && administrationPath !== "config")
|
||||
|| ["commondir", "config.worktree", "objects/info/alternates", "objects/info/http-alternates"]
|
||||
.includes(foldedAdministrationPath))) fail();
|
||||
if (stat.isSocket()) { exclude.push(next); continue; }
|
||||
if (stat.isDirectory()) {
|
||||
if (!controllerCopy && !inGit && name !== ".git") {
|
||||
const marker = await fs.lstat(name === "node_modules" ? path.join(directory, "package.json")
|
||||
: path.join(file, "pyvenv.cfg")).catch(() => null);
|
||||
if (marker?.isFile() && !marker.isSymbolicLink()
|
||||
&& (!repository || !(await gitRead(directory, ["ls-files", "-z", "--", name])))) {
|
||||
exclude.push(next); continue;
|
||||
}
|
||||
}
|
||||
digest.update(JSON.stringify([next, "directory", stat.mode & 0o777])); entries++;
|
||||
await visit(next, repository);
|
||||
} else if (stat.isSymbolicLink()) {
|
||||
const target = await fs.readlink(file), resolved = path.posix.normalize(path.posix.join(path.posix.dirname(next), target));
|
||||
if (!target || path.posix.isAbsolute(target) || /[\\\u0000-\u001f\u007f]/u.test(target)
|
||||
|| resolved === ".." || resolved.startsWith("../")) fail();
|
||||
digest.update(JSON.stringify([next, "symlink", target])); entries++;
|
||||
} else if (stat.isFile()) {
|
||||
if (git >= 0 && parts.slice(git + 1).join("/") === "config") {
|
||||
configs.push(file); configBytes += stat.size;
|
||||
if (stat.size > 65_536 || configBytes > 1024 * 1024) budget();
|
||||
}
|
||||
const original = await fs.lstat(file, { bigint: true });
|
||||
const handle = await fs.open(file, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | fs.constants.O_NONBLOCK);
|
||||
const hash = crypto.createHash("sha256"); let size = 0;
|
||||
try {
|
||||
const before = await handle.stat({ bigint: true });
|
||||
const fields = ["dev", "ino", "size", "mode", "mtimeNs", "ctimeNs"] as const;
|
||||
if (!before.isFile() || !original.isFile() || Number(before.dev) !== stat.dev || Number(before.ino) !== stat.ino
|
||||
|| Number(before.size) !== stat.size || Number(before.mode) !== stat.mode
|
||||
|| fields.some(key => original[key] !== before[key]) || await fs.realpath(file) !== file) fail();
|
||||
for await (const bytes of handle.createReadStream({ autoClose: false, signal })) { check(); size += bytes.length; hash.update(bytes); }
|
||||
const after = await handle.stat({ bigint: true }), current = await fs.lstat(file, { bigint: true });
|
||||
if (!current.isFile() || size !== Number(before.size) || await fs.realpath(file) !== file
|
||||
|| fields.some(key => before[key] !== after[key] || after[key] !== current[key])) fail();
|
||||
} finally { await handle.close(); }
|
||||
byteSize += size;
|
||||
if (!Number.isSafeInteger(byteSize)) fail();
|
||||
digest.update(JSON.stringify([next, "file", stat.mode & 0o777, size, hash.digest("hex")])); entries++;
|
||||
} else fail();
|
||||
}
|
||||
const after = await fs.lstat(directory);
|
||||
if (before.dev !== after.dev || before.ino !== after.ino || before.mtimeMs !== after.mtimeMs || before.ctimeMs !== after.ctimeMs) fail();
|
||||
};
|
||||
await visit("");
|
||||
if (Buffer.byteLength(JSON.stringify(exclude)) > 65_536) fail();
|
||||
if (controllerCopy) {
|
||||
// Bound the complete tree and verify every repository layout before the
|
||||
// first controller subprocess. Files named config inside Git metadata are
|
||||
// ordinary data unless they are the exact repository-root configuration.
|
||||
for (const repository of repositories) {
|
||||
check();
|
||||
const directory = path.join(repository, ".git"), head = path.join(directory, "HEAD");
|
||||
if ((await fs.stat(head)).size > 4096 || !/^(ref: refs\/\S+|[0-9a-f]{40}|[0-9a-f]{64})\s*$/u.test(await fs.readFile(head, "utf8"))) fail();
|
||||
if (!(await fs.lstat(path.join(directory, "objects"))).isDirectory()
|
||||
|| !(await fs.lstat(path.join(directory, "refs")).catch(() => fs.lstat(path.join(directory, "reftable")))).isDirectory()) fail();
|
||||
}
|
||||
let settings = 0, refspecs = 0;
|
||||
for (const config of configs) {
|
||||
check();
|
||||
// Parse this file as data. Never discover a repository or follow includes.
|
||||
const keys = (await gitRead(path.dirname(root), ["config", "--file", config, "--no-includes", "--null", "--name-only", "--list"]))
|
||||
.split("\0").filter(Boolean);
|
||||
settings += keys.length; refspecs += keys.filter(key => /^remote\..+\.fetch$|^branch\..+\.merge$/u.test(key)).length;
|
||||
if (settings > 1024 || refspecs > 256) budget();
|
||||
const portableKey = /^(?:core\.(?:repositoryformatversion|filemode|bare|logallrefupdates|ignorecase|precomposeunicode|symlinks|sparsecheckout|sparsecheckoutcone)|index\.sparse|extensions\.(?:objectformat|refstorage|partialclone)|remote\..+\.(?:url|pushurl|fetch|promisor|partialclonefilter)|branch\..+\.(?:remote|merge|rebase)|user\.(?:name|email))$/u;
|
||||
if (!keys.every(key => portableKey.test(key))) fail();
|
||||
}
|
||||
}
|
||||
return { sha256: digest.digest("hex"), byteSize, entries, exclude };
|
||||
}
|
||||
|
||||
export function readonlyWorkspaceInventoryScript() {
|
||||
const source = readWorkspaceInventory.toString().replaceAll("__vite_ssr_dynamic_import__", "import");
|
||||
return `const __name=value=>value;const inspect=${source};inspect(process.argv[1]).then(value=>process.stdout.write(JSON.stringify(value))).catch(()=>{process.stderr.write('remote_workspace_readonly_capture_unsafe');process.exitCode=1});`;
|
||||
}
|
||||
@@ -306,6 +306,10 @@ async function workspaceArchive(input: {
|
||||
};
|
||||
const directories: string[] = [];
|
||||
const excludedPaths = new Set<string>();
|
||||
for (const relative of input.exclude ?? []) {
|
||||
if (!safeRelative(relative)) fail("unsupported_filename");
|
||||
excludedPaths.add(relative);
|
||||
}
|
||||
const excluded = (next: string) => input.exclude?.some(item => next === item || next.startsWith(`${item}/`))
|
||||
|| next === ".paperclip-runtime" || next.startsWith(".paperclip-runtime/");
|
||||
const walk = async (root: string, skipGit = false) => {
|
||||
|
||||
@@ -6,6 +6,7 @@ import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { promisify } from "node:util";
|
||||
import { deflateSync } from "node:zlib";
|
||||
import net from "node:net";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
|
||||
import {
|
||||
@@ -18,6 +19,8 @@ import {
|
||||
type RemoteWorkspaceTarget,
|
||||
} from "./remote-workspace-transport.js";
|
||||
import { classifyNativeWorkspaceFailure } from "./native-runtime/native-workspace-failure.js";
|
||||
import { restoreRemoteWorkspaceArchive } from "./remote-workspace-snapshot-archive.js";
|
||||
import { readWorkspaceInventory, readonlyWorkspaceInventoryScript } from "./remote-workspace-readonly-inventory.js";
|
||||
|
||||
const exec = promisify(execFile);
|
||||
|
||||
@@ -468,6 +471,58 @@ describe("remote workspace transport safety with real sandbox commands", () => {
|
||||
.rejects.toThrow(/^remote_workspace_operation_timed_out: archive_helper \(configured timeout 300000ms\)$/);
|
||||
});
|
||||
|
||||
it.each(["helper", "archive"])("saves a full sandbox through read-only transfer when the %s cannot write", async failureStage => {
|
||||
await git(workspace, "init", "--initial-branch=main");
|
||||
await fs.writeFile(path.join(workspace, "result.txt"), "committed result\n");
|
||||
await git(workspace, "add", "."); await git(workspace, "commit", "-m", "Saved work");
|
||||
const head = (await git(workspace, "rev-parse", "HEAD")).stdout;
|
||||
await fs.writeFile(path.join(workspace, "result.txt"), "staged result\n");
|
||||
await git(workspace, "add", "result.txt");
|
||||
await fs.writeFile(path.join(workspace, "notes.txt"), "untracked result\n");
|
||||
await fs.symlink("notes.txt", path.join(workspace, "notes-link"));
|
||||
const preload = path.join(scratch, "full-sandbox.cjs");
|
||||
const failing = failureStage === "helper" ? String.raw`/archive\.cjs\.[^/]+\.tmp$/` : String.raw`/workspace\.pcws$/`;
|
||||
await fs.writeFile(preload, String.raw`const fs=require('node:fs/promises'),failing=${failing};
|
||||
const fail=()=>{throw Object.assign(new Error('No space left on device'),{code:'ENOSPC'});};
|
||||
const writeFile=fs.writeFile,open=fs.open;fs.writeFile=async(file,...args)=>{
|
||||
if(failing.test(String(file)))fail();return writeFile(file,...args);};
|
||||
fs.open=async(file,...args)=>{const handle=await open(file,...args);if(failing.test(String(file)))handle.writeFile=async()=>fail();return handle;};
|
||||
require('node:module').syncBuiltinESMExports();`);
|
||||
const { remote, execute } = target();
|
||||
const run = execute.getMockImplementation()!;
|
||||
execute.mockImplementation(input => run({ ...input, env: { ...input.env, NODE_OPTIONS: `--require ${preload}` } }));
|
||||
const transfer = remote.runner!.syncOut!;
|
||||
remote.runner!.syncOut = async (operations, options) => {
|
||||
for (const operation of operations) for (const file of operation.files) {
|
||||
if (file.kind === "directory") {
|
||||
expect(file.followSymlinks).toBe(false);
|
||||
expect(file.exclude).toContain(".paperclip-runtime");
|
||||
await fs.cp(file.sourcePath, file.targetPath, { recursive: true, verbatimSymlinks: true,
|
||||
filter: source => !file.exclude!.some(excluded => {
|
||||
const relative = path.relative(file.sourcePath, source).split(path.sep).join("/");
|
||||
return relative === excluded || relative.startsWith(`${excluded}/`);
|
||||
}) });
|
||||
} else return transfer(operations, options);
|
||||
}
|
||||
return { operations: operations.map(operation => ({ operationId: operation.operationId,
|
||||
filesTransferred: operation.files.length, bytesTransferred: 0 })) };
|
||||
};
|
||||
const saved = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() });
|
||||
try {
|
||||
expect(await fs.readFile(path.join(workspace, "result.txt"), "utf8")).toBe("staged result\n");
|
||||
expect(await fs.readFile(path.join(workspace, "notes.txt"), "utf8")).toBe("untracked result\n");
|
||||
await fs.rm(workspace, { recursive: true });
|
||||
const restored = path.join(scratch, "restored");
|
||||
await restoreRemoteWorkspaceArchive({ archivePath: saved.archivePath, targetDir: restored,
|
||||
expectedManifestDigest: saved.manifestDigest });
|
||||
expect((await git(restored, "rev-parse", "HEAD")).stdout).toBe(head);
|
||||
expect((await git(restored, "show", ":result.txt")).stdout).toBe("staged result\n");
|
||||
expect(await fs.readFile(path.join(restored, "notes.txt"), "utf8")).toBe("untracked result\n");
|
||||
expect(await fs.readlink(path.join(restored, "notes-link"))).toBe("notes.txt");
|
||||
await git(restored, "fsck", "--full");
|
||||
} finally { await saved.cleanup(); }
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ step: "helper install", failing: String.raw`/archive\.cjs\.[^/]+\.tmp$/`, outputMode: "split" as const },
|
||||
{ step: "archive write", failing: String.raw`/workspace\.pcws$/`, outputMode: "split" as const },
|
||||
@@ -489,6 +544,7 @@ describe("remote workspace transport safety with real sandbox commands", () => {
|
||||
const { remote, execute } = target({}, outputMode);
|
||||
const run = execute.getMockImplementation()!;
|
||||
execute.mockImplementation(input => run({ ...input, env: { ...input.env, NODE_OPTIONS: `--require ${preload}` } }));
|
||||
remote.runner!.syncOut = async () => { throw new Error("synthetic transfer unavailable"); };
|
||||
const operationId = randomUUID();
|
||||
await expect(captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId }))
|
||||
.rejects.toThrow(/^remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace$/);
|
||||
@@ -498,6 +554,260 @@ describe("remote workspace transport safety with real sandbox commands", () => {
|
||||
expect(await fs.readFile(path.join(workspace, "result.txt"), "utf8")).toBe("synthetic saved result\n");
|
||||
});
|
||||
|
||||
it("preserves work beside a socket and records its exclusion during full-disk recovery", async () => {
|
||||
await fs.writeFile(path.join(workspace, "saved.txt"), "retained work\n");
|
||||
const socket = path.join(workspace, "ipc"), server = net.createServer();
|
||||
await new Promise<void>((resolve, reject) => { server.once("error", reject); server.listen(socket, resolve); });
|
||||
let saved: Awaited<ReturnType<typeof captureRemoteWorkspaceArchive>> | undefined;
|
||||
try {
|
||||
const { remote, execute } = target();
|
||||
execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace",
|
||||
timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() });
|
||||
remote.runner!.syncOut = async operations => {
|
||||
const mapping = operations[0]!.files[0]!;
|
||||
await fs.cp(mapping.sourcePath, mapping.targetPath, { recursive: true, verbatimSymlinks: true,
|
||||
filter: file => !mapping.exclude!.includes(path.relative(mapping.sourcePath, file)) });
|
||||
return { operations: [] };
|
||||
};
|
||||
saved = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() });
|
||||
const bytes = await fs.readFile(saved.archivePath), length = bytes.readUInt32BE(6);
|
||||
expect(JSON.parse(bytes.subarray(10, 10 + length).toString()).excluded).toContain("ipc");
|
||||
const restored = path.join(scratch, "restored");
|
||||
await restoreRemoteWorkspaceArchive({ archivePath: saved.archivePath, targetDir: restored, expectedManifestDigest: saved.manifestDigest });
|
||||
expect(await fs.readdir(restored)).toEqual(["saved.txt"]);
|
||||
expect((await fs.lstat(socket)).isSocket()).toBe(true);
|
||||
} finally {
|
||||
await saved?.cleanup();
|
||||
await new Promise<void>((resolve, reject) => server.close(error => error ? reject(error) : resolve()));
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["source changed", "copy changed", "extra file"])("rejects a read-only fallback when %s", async fault => {
|
||||
await fs.writeFile(path.join(workspace, "saved.txt"), "retained work\n");
|
||||
const { remote, execute } = target();
|
||||
execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace",
|
||||
timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() });
|
||||
let copiedRoot: string | undefined;
|
||||
remote.runner!.syncOut = async operations => {
|
||||
const file = operations[0]!.files[0]!; copiedRoot = path.dirname(file.targetPath);
|
||||
await fs.cp(file.sourcePath, file.targetPath, { recursive: true, verbatimSymlinks: true });
|
||||
if (fault === "source changed") await fs.writeFile(path.join(workspace, "saved.txt"), "newer work\n");
|
||||
else await fs.writeFile(path.join(file.targetPath, fault === "extra file" ? "extra.txt" : "saved.txt"), "wrong copy\n");
|
||||
return { operations: [] };
|
||||
};
|
||||
await expect(captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() }))
|
||||
.rejects.toMatchObject({ message: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace",
|
||||
cause: { message: "remote_workspace_readonly_capture_failed: stage=verify reason=source_changed transport=provider_error" } });
|
||||
expect(await fs.readFile(path.join(workspace, "saved.txt"), "utf8")).toBe(fault === "source changed" ? "newer work\n" : "retained work\n");
|
||||
expect(copiedRoot).toBeDefined();
|
||||
await expect(fs.access(copiedRoot!)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
});
|
||||
|
||||
it.each(["python", "node"])("recovers a full sandbox without copying an untracked %s environment", async kind => {
|
||||
await git(workspace, "init", "--initial-branch=main");
|
||||
await fs.writeFile(path.join(workspace, "saved.txt"), "retained work\n");
|
||||
await git(workspace, "add", "saved.txt"); await git(workspace, "commit", "-m", "Saved work");
|
||||
const dependency = kind === "python" ? ".venv" : "node_modules";
|
||||
await fs.mkdir(path.join(workspace, dependency, "bin"), { recursive: true });
|
||||
await fs.writeFile(path.join(workspace, kind === "python" ? ".venv/pyvenv.cfg" : "package.json"), kind === "python" ? "home = /usr/bin\n" : "{}\n");
|
||||
await fs.symlink(process.execPath, path.join(workspace, dependency, "bin", "runtime"));
|
||||
const { remote, execute } = target();
|
||||
execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace",
|
||||
timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() });
|
||||
remote.runner!.syncOut = async operations => {
|
||||
const mapping = operations[0]!.files[0]!;
|
||||
expect(mapping.exclude).toContain(dependency);
|
||||
await fs.cp(mapping.sourcePath, mapping.targetPath, { recursive: true, verbatimSymlinks: true,
|
||||
filter: file => !(mapping.exclude ?? []).some(excluded => {
|
||||
const relative = path.relative(mapping.sourcePath, file).split(path.sep).join("/");
|
||||
return relative === excluded || relative.startsWith(`${excluded}/`);
|
||||
}) });
|
||||
return { operations: [] };
|
||||
};
|
||||
const saved = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() });
|
||||
try {
|
||||
const restored = path.join(scratch, "restored");
|
||||
await restoreRemoteWorkspaceArchive({ archivePath: saved.archivePath, targetDir: restored, expectedManifestDigest: saved.manifestDigest });
|
||||
expect(await fs.readFile(path.join(restored, "saved.txt"), "utf8")).toBe("retained work\n");
|
||||
await expect(fs.access(path.join(restored, dependency))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(await fs.readlink(path.join(workspace, dependency, "bin", "runtime"))).toBe(process.execPath);
|
||||
} finally { await saved.cleanup(); }
|
||||
});
|
||||
|
||||
it("never excludes Git history because its metadata contains a virtualenv marker", async () => {
|
||||
await git(workspace, "init", "--initial-branch=main");
|
||||
await fs.writeFile(path.join(workspace, "saved.txt"), "retained work\n");
|
||||
await git(workspace, "add", "saved.txt"); await git(workspace, "commit", "-m", "Saved work");
|
||||
const commit = (await git(workspace, "rev-parse", "HEAD")).stdout;
|
||||
await fs.writeFile(path.join(workspace, ".git", "pyvenv.cfg"), "home = /usr/bin\n");
|
||||
const { remote, execute } = target();
|
||||
execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace",
|
||||
timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() });
|
||||
remote.runner!.syncOut = async operations => {
|
||||
const mapping = operations[0]!.files[0]!;
|
||||
expect(mapping.exclude).not.toContain(".git");
|
||||
await fs.cp(mapping.sourcePath, mapping.targetPath, { recursive: true, verbatimSymlinks: true });
|
||||
return { operations: [] };
|
||||
};
|
||||
const saved = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() });
|
||||
try {
|
||||
const restored = path.join(scratch, "restored");
|
||||
await restoreRemoteWorkspaceArchive({ archivePath: saved.archivePath, targetDir: restored, expectedManifestDigest: saved.manifestDigest });
|
||||
expect((await git(restored, "rev-parse", "HEAD")).stdout).toBe(commit);
|
||||
expect(await fs.readFile(path.join(restored, "saved.txt"), "utf8")).toBe("retained work\n");
|
||||
} finally { await saved.cleanup(); }
|
||||
});
|
||||
|
||||
it("rejects excessive Git roots before parsing controller configuration", async () => {
|
||||
for (let index = 0; index < 65; index++) {
|
||||
const directory = path.join(workspace, `repo-${index}`, ".git"); await fs.mkdir(directory, { recursive: true });
|
||||
await fs.writeFile(path.join(directory, "config"), "");
|
||||
}
|
||||
const bin = path.join(scratch, "bin"), invoked = path.join(scratch, "git-invoked"); await fs.mkdir(bin);
|
||||
await fs.writeFile(path.join(bin, "git"), `#!/bin/sh\nprintf invoked > '${invoked}'\nexit 1\n`, { mode: 0o700 });
|
||||
vi.stubEnv("PATH", `${bin}${path.delimiter}${process.env.PATH}`);
|
||||
try {
|
||||
await expect(readWorkspaceInventory(workspace, true)).rejects.toThrow("remote_workspace_readonly_capture_budget");
|
||||
await expect(fs.access(invoked)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
} finally { vi.unstubAllEnvs(); }
|
||||
});
|
||||
|
||||
it("bounds refspec work before controller normalization", async () => {
|
||||
await git(workspace, "init", "--initial-branch=main");
|
||||
await fs.appendFile(path.join(workspace, ".git", "config"), `[remote "origin"]\n${"fetch = +refs/heads/*:refs/remotes/origin/*\n".repeat(257)}`);
|
||||
await expect(readWorkspaceInventory(workspace, true)).rejects.toThrow("remote_workspace_readonly_capture_budget");
|
||||
});
|
||||
|
||||
it("cancels controller inventory before reading its source", async () => {
|
||||
const reason = new Error("verification deadline");
|
||||
await expect(readWorkspaceInventory(path.join(scratch, "missing"), true, AbortSignal.abort(reason))).rejects.toBe(reason);
|
||||
});
|
||||
|
||||
it("retains tracked environment files during full-disk recovery", async () => {
|
||||
await git(workspace, "init", "--initial-branch=main");
|
||||
await fs.mkdir(path.join(workspace, ".venv"));
|
||||
await fs.writeFile(path.join(workspace, ".venv", "pyvenv.cfg"), "tracked user data\n");
|
||||
await fs.writeFile(path.join(workspace, ".venv", "result.txt"), "must survive\n");
|
||||
await git(workspace, "add", ".venv"); await git(workspace, "commit", "-m", "Tracked environment data");
|
||||
const { remote, execute } = target();
|
||||
execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace",
|
||||
timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() });
|
||||
const saved = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() });
|
||||
try {
|
||||
const restored = path.join(scratch, "restored");
|
||||
await restoreRemoteWorkspaceArchive({ archivePath: saved.archivePath, targetDir: restored, expectedManifestDigest: saved.manifestDigest });
|
||||
expect(await fs.readFile(path.join(restored, ".venv", "result.txt"), "utf8")).toBe("must survive\n");
|
||||
} finally { await saved.cleanup(); }
|
||||
});
|
||||
|
||||
it.skipIf(process.platform === "win32")("rejects a file replaced by a FIFO before inventory open without blocking", async () => {
|
||||
const file = path.join(workspace, "saved.txt"), fifo = path.join(scratch, "fifo"), preload = path.join(scratch, "replace-with-fifo.cjs");
|
||||
await fs.writeFile(file, "retained work\n"); await exec("mkfifo", [fifo]);
|
||||
await fs.writeFile(preload, `const fs=require('node:fs/promises');const lstat=fs.lstat;let replaced=false;
|
||||
fs.lstat=async(file,...rest)=>{const stat=await lstat(file,...rest);
|
||||
if(!replaced&&String(file)===${JSON.stringify(file)}&&stat.isFile()){replaced=true;
|
||||
await fs.rename(file,file+'.retained');await fs.rename(${JSON.stringify(fifo)},file);}return stat;};
|
||||
require('node:module').syncBuiltinESMExports();`);
|
||||
await expect(exec(process.execPath, ["--require", preload, "-e", readonlyWorkspaceInventoryScript(), workspace], { timeout: 3000 }))
|
||||
.rejects.toMatchObject({ code: 1, stderr: "remote_workspace_readonly_capture_unsafe" });
|
||||
expect(await fs.readFile(`${file}.retained`, "utf8")).toBe("retained work\n");
|
||||
});
|
||||
|
||||
it("rejects internal Git config symlinks before controller normalization can follow an include", async () => {
|
||||
await fs.mkdir(path.join(workspace, ".git"));
|
||||
const outside = path.join(scratch, "outside-config");
|
||||
await fs.writeFile(outside, "[user]\nname = outside\n");
|
||||
await fs.writeFile(path.join(workspace, "payload-config"), `[include]\npath = ${outside}\n`);
|
||||
await fs.symlink("../payload-config", path.join(workspace, ".git", "config"));
|
||||
await expect(readWorkspaceInventory(workspace, true)).rejects.toThrow("remote_workspace_readonly_capture_unsafe");
|
||||
expect(await fs.readFile(outside, "utf8")).toBe("[user]\nname = outside\n");
|
||||
});
|
||||
|
||||
it("saves read-only directories and removes only the private controller copy", async () => {
|
||||
const source = path.join(workspace, "readonly"), restored = path.join(scratch, "restored");
|
||||
await fs.mkdir(source); await fs.writeFile(path.join(source, "saved.txt"), "read-only work\n"); await fs.chmod(source, 0o555);
|
||||
const { remote, execute } = target();
|
||||
execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace",
|
||||
timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() });
|
||||
const transfer = remote.runner!.syncOut!; let copy: string | undefined;
|
||||
remote.runner!.syncOut = (operations, options) => { copy = operations[0]!.files[0]!.targetPath; return transfer(operations, options); };
|
||||
let saved: Awaited<ReturnType<typeof captureRemoteWorkspaceArchive>> | undefined;
|
||||
try {
|
||||
saved = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() });
|
||||
expect((await fs.stat(source)).mode & 0o777).toBe(0o555);
|
||||
expect(await fs.readFile(path.join(source, "saved.txt"), "utf8")).toBe("read-only work\n");
|
||||
await expect(fs.access(copy!)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await restoreRemoteWorkspaceArchive({ archivePath: saved.archivePath, targetDir: restored, expectedManifestDigest: saved.manifestDigest });
|
||||
expect((await fs.stat(path.join(restored, "readonly"))).mode & 0o777).toBe(0o555);
|
||||
expect(await fs.readFile(path.join(restored, "readonly", "saved.txt"), "utf8")).toBe("read-only work\n");
|
||||
} finally {
|
||||
for (const directory of [source, path.join(restored, "readonly"), ...(copy ? [path.join(copy, "readonly")] : [])]) {
|
||||
await fs.chmod(directory, 0o700).catch(() => undefined);
|
||||
}
|
||||
await saved?.cleanup();
|
||||
if (copy) await fs.rm(path.dirname(copy), { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("retains initialized submodule workspaces when a full sandbox cannot normalize their Git pointers", async () => {
|
||||
const origin = await repository();
|
||||
await git(workspace, "init", "--initial-branch=main");
|
||||
await fs.writeFile(path.join(workspace, "result.txt"), "retain parent work\n");
|
||||
await git(workspace, "add", "result.txt"); await git(workspace, "commit", "-m", "Parent");
|
||||
await git(workspace, "-c", "protocol.file.allow=always", "submodule", "add", "--", origin, "sub");
|
||||
const before = (await git(workspace, "status", "--porcelain")).stdout;
|
||||
const { remote, execute } = target();
|
||||
execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace",
|
||||
timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() });
|
||||
const syncOut = vi.fn(remote.runner!.syncOut!); remote.runner!.syncOut = syncOut;
|
||||
await expect(captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() }))
|
||||
.rejects.toThrow(/^remote_workspace_sandbox_disk_full:/);
|
||||
expect(syncOut).not.toHaveBeenCalled();
|
||||
expect((await git(workspace, "status", "--porcelain")).stdout).toBe(before);
|
||||
expect(await fs.readFile(path.join(workspace, "sub", "source.txt"), "utf8")).toBe("initial synthetic source\n");
|
||||
});
|
||||
|
||||
it.each(["Config", "objects/info/Alternates"])("rejects case aliases of Git administration paths before controller Git reads: %s", async relative => {
|
||||
await git(workspace, "init", "--initial-branch=main");
|
||||
const outside = path.join(scratch, "outside"); await fs.mkdir(outside);
|
||||
if (relative === "Config") {
|
||||
await fs.rename(path.join(workspace, ".git", "config"), path.join(workspace, ".git", "Config"));
|
||||
await fs.writeFile(path.join(workspace, ".git", "Config"), `[include]\npath = ${path.join(outside, "config")}\n`);
|
||||
} else await fs.writeFile(path.join(workspace, ".git", relative), `${outside}\n`);
|
||||
await expect(readWorkspaceInventory(workspace, true)).rejects.toThrow("remote_workspace_readonly_capture_unsafe");
|
||||
expect(await fs.readdir(outside)).toEqual([]);
|
||||
});
|
||||
|
||||
it("rejects nonportable Git config before Git can read a controller attributes file", async () => {
|
||||
await git(workspace, "init", "--initial-branch=main");
|
||||
await git(workspace, "config", "core.attributesFile", path.join(scratch, "outside-attributes"));
|
||||
await expect(readWorkspaceInventory(workspace, true)).rejects.toThrow("remote_workspace_readonly_capture_unsafe");
|
||||
});
|
||||
|
||||
it.each(["worktree pointer", "alternate store", "config include", "core worktree"])("retains a full sandbox with a %s instead of resolving controller paths", async fault => {
|
||||
const outside = path.join(scratch, "outside"); await fs.mkdir(outside);
|
||||
await fs.writeFile(path.join(outside, "sentinel"), "must stay outside\n");
|
||||
await fs.writeFile(path.join(workspace, "saved.txt"), "retained work\n");
|
||||
if (fault === "worktree pointer") await fs.writeFile(path.join(workspace, ".git"), `gitdir: ${outside}\n`);
|
||||
else {
|
||||
await git(workspace, "init", "--initial-branch=main");
|
||||
if (fault === "alternate store") await fs.writeFile(path.join(workspace, ".git", "objects", "info", "alternates"), `${outside}\n`);
|
||||
if (fault === "config include") await git(workspace, "config", "include.path", path.join(outside, "sentinel"));
|
||||
if (fault === "core worktree") await git(workspace, "config", "core.worktree", outside);
|
||||
}
|
||||
const { remote, execute } = target();
|
||||
execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace",
|
||||
timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() });
|
||||
const transfer = remote.runner!.syncOut!;
|
||||
const syncOut = vi.fn(transfer); remote.runner!.syncOut = syncOut;
|
||||
await expect(captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() }))
|
||||
.rejects.toThrow(/^remote_workspace_sandbox_disk_full:/);
|
||||
if (["worktree pointer", "alternate store"].includes(fault)) expect(syncOut).not.toHaveBeenCalled();
|
||||
expect(await fs.readFile(path.join(workspace, "saved.txt"), "utf8")).toBe("retained work\n");
|
||||
expect(await fs.readdir(outside)).toEqual(["sentinel"]);
|
||||
expect(await fs.readFile(path.join(outside, "sentinel"), "utf8")).toBe("must stay outside\n");
|
||||
});
|
||||
|
||||
it("keeps the full-disk diagnostic exact", async () => {
|
||||
const { remote, execute } = target();
|
||||
execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", timedOut: false, signal: null, pid: process.pid,
|
||||
@@ -507,13 +817,14 @@ describe("remote workspace transport safety with real sandbox commands", () => {
|
||||
.rejects.toThrow(/^remote_workspace_operation_failed: operation=[a-z_]+ exit=1$/);
|
||||
});
|
||||
|
||||
it.each(["split", "one-shot"] as const)("preserves a Git child disk failure with %s output without its private text", async outputMode => {
|
||||
it.each(["split", "one-shot"] as const)("preserves a Git child disk failure with %s output when read-only download is unavailable", async outputMode => {
|
||||
await git(workspace, "init", "--initial-branch=main");
|
||||
await fs.writeFile(path.join(workspace, "saved.txt"), "retained source bytes\n");
|
||||
const bin = path.join(scratch, "controlled-bin"); await fs.mkdir(bin);
|
||||
const gitWrapper = path.join(bin, "git");
|
||||
await fs.writeFile(gitWrapper, "#!/bin/sh\nprintf '%s' 'No space left on device: SYNTHETIC_PRIVATE_GIT_PATH' >&2\nexit 2\n", { mode: 0o700 });
|
||||
const { remote } = target({ PATH: `${bin}${path.delimiter}${process.env.PATH}` }, outputMode);
|
||||
remote.runner!.syncOut = async () => { throw new Error("synthetic transfer unavailable"); };
|
||||
await expect(captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() }))
|
||||
.rejects.toThrow(/^remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace$/);
|
||||
expect(await fs.readFile(path.join(workspace, "saved.txt"), "utf8")).toBe("retained source bytes\n");
|
||||
@@ -567,6 +878,7 @@ describe("remote workspace transport safety with real sandbox commands", () => {
|
||||
const diskFull = "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace";
|
||||
execute.mockResolvedValueOnce({ exitCode: 1, stdout: diskFull, stderr: "remote_workspace_archive_fs_enoent",
|
||||
timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() });
|
||||
remote.runner!.syncOut = async () => { throw new Error("synthetic transfer unavailable"); };
|
||||
await expect(captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() }))
|
||||
.rejects.toThrow(diskFull);
|
||||
});
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { createReadStream } from "node:fs";
|
||||
import { mkdtemp, rm, stat } from "node:fs/promises";
|
||||
import { mkdtemp, realpath, rm, stat } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
||||
import type { RemoteWorkspaceSource } from "@paperclipai/shared";
|
||||
import { remoteWorkspaceArchiveScript, remoteWorkspaceCaptureFailureMessage, remoteWorkspaceFailureDetail } from "./remote-workspace-snapshot-archive.js";
|
||||
import { captureRemoteWorkspaceDirectory, remoteWorkspaceArchiveScript, remoteWorkspaceCaptureFailureMessage, remoteWorkspaceFailureDetail } from "./remote-workspace-snapshot-archive.js";
|
||||
import { readWorkspaceInventory, readonlyWorkspaceInventoryScript } from "./remote-workspace-readonly-inventory.js";
|
||||
import { removeTaskDirectoryTree } from "./isolated-task-directory-snapshot.js";
|
||||
import type { GitRemoteAuthProvider } from "./git-credentials.js";
|
||||
import { classifyWorkspaceRestoreFailure } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
||||
import { MAX_SANDBOX_SYNC_TIMEOUT_MS, type SandboxSyncOperation } from "@paperclipai/adapter-utils/sandbox-managed-runtime";
|
||||
@@ -418,6 +420,73 @@ const elapsedSince = (started: number) => Math.max(0, Math.round(performance.now
|
||||
|
||||
export async function captureRemoteWorkspaceArchive(input: {
|
||||
target: RemoteWorkspaceTarget; remoteCwd: string; operationId: string; timing?: RemoteWorkspaceSavePhaseTiming;
|
||||
}) {
|
||||
try { return await captureSandboxWorkspaceArchive(input); }
|
||||
catch (error) {
|
||||
if (!(error instanceof Error) || error.message !== REMOTE_WORKSPACE_SANDBOX_DISK_FULL) throw error;
|
||||
// Disk exhaustion cannot be repaired by writing another archive to the
|
||||
// same sandbox. Transfer a verified standalone source without writing to
|
||||
// it, then normalize and seal it on the controller. Failed verification
|
||||
// retains the original disk-full failure and never permits source cleanup.
|
||||
try { return await captureReadOnlyWorkspaceArchive(input); }
|
||||
catch (cause) { throw new Error(error.message, { cause }); }
|
||||
}
|
||||
}
|
||||
|
||||
async function captureReadOnlyWorkspaceArchive(input: {
|
||||
target: RemoteWorkspaceTarget; remoteCwd: string; operationId: string;
|
||||
}) {
|
||||
if (!/^[a-f0-9-]{36}$/.test(input.operationId)) throw new Error("remote_workspace_operation_identity_invalid");
|
||||
const inspect = async (timeoutMs = 300_000) => {
|
||||
const value = JSON.parse(await command(input.target, "capture", ["-e", readonlyWorkspaceInventoryScript(), input.remoteCwd], undefined, timeoutMs)) as Awaited<ReturnType<typeof readWorkspaceInventory>>;
|
||||
if (!/^[a-f0-9]{64}$/.test(value.sha256) || !Number.isSafeInteger(value.byteSize) || value.byteSize < 0
|
||||
|| !Number.isSafeInteger(value.entries) || value.entries < 0 || !Array.isArray(value.exclude)
|
||||
|| value.exclude.length > 4096 || !value.exclude.every(item => typeof item === "string" && item.length > 0
|
||||
&& !/[\\\u0000-\u001f\u007f]/u.test(item) && !path.posix.isAbsolute(item)
|
||||
&& item.split("/").every(part => part !== "" && part !== "." && part !== ".."))
|
||||
|| !value.exclude.includes(".paperclip-runtime")) throw new Error("remote_workspace_readonly_capture_unsafe");
|
||||
return value;
|
||||
};
|
||||
let root: string | undefined;
|
||||
let stage: "inspect" | "transfer" | "verify" | "archive" = "inspect";
|
||||
try {
|
||||
const before = await inspect();
|
||||
root = await mkdtemp(path.join(os.tmpdir(), "paperclip-workspace-readonly-"));
|
||||
root = await realpath(root);
|
||||
const copy = path.join(root, "source"), archivePath = path.join(root, "workspace.pcws");
|
||||
stage = "transfer";
|
||||
await transferArchive(input.target, "syncOut", [{ operationId: input.operationId,
|
||||
files: [{ kind: "directory", sourcePath: input.remoteCwd, targetPath: copy,
|
||||
followSymlinks: false, preserveSourceEntries: true, exclude: before.exclude }] }], Math.max(1, before.byteSize));
|
||||
stage = "verify";
|
||||
const verificationStarted = performance.now();
|
||||
const verificationSignal = AbortSignal.timeout(120_000);
|
||||
const remaining = () => { verificationSignal.throwIfAborted(); return Math.max(1, Math.ceil(120_000 - (performance.now() - verificationStarted))); };
|
||||
const received = await readWorkspaceInventory(copy, true, verificationSignal);
|
||||
const same = (observed: typeof before) => observed.sha256 === before.sha256
|
||||
&& observed.byteSize === before.byteSize && observed.entries === before.entries;
|
||||
if (!same(received) || !same(await inspect(remaining()))) throw new Error("remote_workspace_readonly_capture_changed");
|
||||
stage = "archive";
|
||||
const receipt = await captureRemoteWorkspaceDirectory({ cwd: copy, archivePath, exclude: before.exclude, signal: verificationSignal });
|
||||
stage = "verify";
|
||||
if (!same(await inspect(remaining()))) throw new Error("remote_workspace_readonly_capture_changed");
|
||||
verificationSignal.throwIfAborted();
|
||||
// Only the sealed portable archive remains a candidate for durable upload.
|
||||
await removeTaskDirectoryTree(copy);
|
||||
const savedRoot = root;
|
||||
return { ...receipt, archivePath, cleanup: () => removeTaskDirectoryTree(savedRoot) };
|
||||
} catch (error) {
|
||||
if (root) await removeTaskDirectoryTree(root);
|
||||
const reason = error instanceof Error && error.message === "remote_workspace_readonly_capture_unsafe" ? "unsafe_source"
|
||||
: error instanceof Error && error.message === "remote_workspace_readonly_capture_changed" ? "source_changed"
|
||||
: error instanceof Error && error.message === "remote_workspace_readonly_capture_budget" ? "budget_exceeded" : "operation_failed";
|
||||
const { label, code } = sandboxProviderFailureLabel(error);
|
||||
throw new Error(`remote_workspace_readonly_capture_failed: stage=${stage} reason=${reason} transport=${label}${code ? ` code=${code}` : ""}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function captureSandboxWorkspaceArchive(input: {
|
||||
target: RemoteWorkspaceTarget; remoteCwd: string; operationId: string; timing?: RemoteWorkspaceSavePhaseTiming;
|
||||
}) {
|
||||
const timing = input.timing ?? {};
|
||||
const captureStarted = performance.now();
|
||||
|
||||
@@ -1105,6 +1105,52 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
|
||||
expect(operation?.issueId).toBe(secondIssue);
|
||||
}, 60_000);
|
||||
|
||||
it("durably saves a full sandbox and restores its work after losing the original allocation", async () => {
|
||||
const reserved = await reserve({ kind: "empty" }), remote = await target("full-sandbox");
|
||||
const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: remote });
|
||||
await fs.writeFile(path.join(prepared.remoteRoot, "saved.txt"), "completed work before restart\n");
|
||||
const execute = remote.runner!.execute.bind(remote.runner);
|
||||
let fullDiskFailures = 0;
|
||||
vi.spyOn(remote.runner!, "execute").mockImplementation(input => {
|
||||
if (input.args?.some(arg => arg.endsWith("/archive.cjs"))) {
|
||||
fullDiskFailures++;
|
||||
return Promise.resolve({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace",
|
||||
timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() });
|
||||
}
|
||||
return execute(input);
|
||||
});
|
||||
const syncOut = remote.runner!.syncOut!;
|
||||
remote.runner!.syncOut = async (operations, options) => {
|
||||
if (!operations.some(operation => operation.files.some(file => file.kind === "directory"))) return syncOut(operations, options);
|
||||
for (const operation of operations) for (const mapping of operation.files) {
|
||||
await fs.cp(mapping.sourcePath, mapping.targetPath, { recursive: true, verbatimSymlinks: true,
|
||||
filter: source => !(mapping.exclude ?? []).some(excluded => {
|
||||
const relative = path.relative(mapping.sourcePath, source).split(path.sep).join("/");
|
||||
return relative === excluded || relative.startsWith(`${excluded}/`);
|
||||
}) });
|
||||
}
|
||||
return { operations: [] };
|
||||
};
|
||||
const saved = await saveRemoteWorkspace({ db, storage, workspace: prepared.workspace, runId: ids.run,
|
||||
lease: prepared.lease, target: remote, boundary: "final", assertOwned: async () => undefined });
|
||||
expect(fullDiskFailures).toBe(1);
|
||||
expect(readRemoteWorkspaceAuthority((await workspace(reserved.id)).metadata)).toMatchObject({
|
||||
snapshotId: saved.snapshotId, dirty: false, writerRunId: null,
|
||||
});
|
||||
const [snapshot] = await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, saved.snapshotId));
|
||||
expect(snapshot).toMatchObject({ status: "committed" });
|
||||
await fs.rm(remote.remoteCwd, { recursive: true, force: true });
|
||||
const load = createSavedWorkspaceLoader(db, storage);
|
||||
for (let reconnect = 0; reconnect < 2; reconnect++) {
|
||||
const restored = await load({ companyId: ids.company, workspaceId: reserved.id, snapshotId: saved.snapshotId });
|
||||
try { expect(await fs.readFile(path.join(restored.root, "saved.txt"), "utf8")).toBe("completed work before restart\n"); }
|
||||
finally { await restored.cleanup(); }
|
||||
}
|
||||
const operations = await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.heartbeatRunId, ids.run),
|
||||
eq(workspaceOperations.phase, "remote_workspace_snapshot")));
|
||||
expect(operations.at(-1)).toMatchObject({ status: "succeeded" });
|
||||
}, 60_000);
|
||||
|
||||
it("retains the writer and previous snapshot when object storage fails", async () => {
|
||||
const reserved = await reserve({ kind: "empty" }), first = await target("outage-sandbox");
|
||||
const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: first });
|
||||
|
||||
@@ -1118,6 +1118,7 @@ describe("automatic retained workspace recovery", () => {
|
||||
} }).where(eq(heartbeatRuns.id, f.runId));
|
||||
const target = targets.get(f.lease.id)!, syncOut = target.runner!.syncOut!, execute = target.runner!.execute;
|
||||
let transfers = 0, sandboxWriteFailures = 0;
|
||||
const readonlyExports: Array<Parameters<typeof syncOut>[0][number]["files"][number]> = [];
|
||||
if (pressure === "sandbox-write") target.runner!.execute = async input => {
|
||||
const fault = `(()=>{const fs=require('node:fs/promises'),sync=require('node:fs');
|
||||
const full=()=>${JSON.stringify([path.join(pack, "node_modules"), path.join(pack, "provider-assets")])}.some(p=>sync.existsSync(p));
|
||||
@@ -1135,13 +1136,21 @@ describe("automatic retained workspace recovery", () => {
|
||||
return response;
|
||||
};
|
||||
target.runner!.syncOut = async operations => {
|
||||
const mappings = operations.flatMap(operation => operation.files);
|
||||
if (mappings.some(mapping => mapping.kind === "directory")) {
|
||||
// These regressions isolate journaled cache reclamation after BOTH
|
||||
// export paths fail. Directory-export recovery has its own real-provider
|
||||
// coverage; this boundary explicitly cannot perform that alternative.
|
||||
readonlyExports.push(...mappings);
|
||||
throw new Error("cache-reclaim fixture directory export unavailable");
|
||||
}
|
||||
transfers++;
|
||||
if (pressure === "transfer" && (await Promise.all(["node_modules", "provider-assets"].map(name =>
|
||||
lstat(path.join(pack, name)).then(() => true, () => false)))).some(Boolean))
|
||||
throw new Error(REMOTE_WORKSPACE_SANDBOX_DISK_FULL);
|
||||
return syncOut(operations);
|
||||
};
|
||||
return { ...f, runner, pack, sessions, digest, transfers: () => transfers,
|
||||
return { ...f, runner, pack, sessions, digest, transfers: () => transfers, readonlyExports,
|
||||
sandboxWriteFailures: () => sandboxWriteFailures };
|
||||
}
|
||||
|
||||
@@ -1189,6 +1198,9 @@ describe("automatic retained workspace recovery", () => {
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
expect(readWorkspaceRecovery((await action(f.actionId)).evidence)?.status).toBe("recovered");
|
||||
expect(f.transfers()).toBe(2);
|
||||
expect(f.readonlyExports).toEqual([{ kind: "directory", sourcePath: await realpath(f.remote),
|
||||
targetPath: expect.stringMatching(/[/]source$/), followSymlinks: false, preserveSourceEntries: true,
|
||||
exclude: [".paperclip-runtime"] }]);
|
||||
await expect(lstat(path.join(f.pack, "node_modules"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(lstat(path.join(f.pack, "provider-assets"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(JSON.parse(await readFile(path.join(f.pack, "provider-pack.json"), "utf8")).digest).toBe(f.digest);
|
||||
@@ -1264,6 +1276,9 @@ describe("automatic retained workspace recovery", () => {
|
||||
const f = await pressuredToolCache(), target = targets.get(f.lease.id)!, syncOut = target.runner!.syncOut!;
|
||||
let transfers = 0;
|
||||
target.runner!.syncOut = async operations => {
|
||||
// Fail the second sealed-archive transfer, after cache cleanup, rather
|
||||
// than consuming this fault on the earlier read-only directory fallback.
|
||||
if (operations.some(operation => operation.files.some(mapping => mapping.kind === "directory"))) return syncOut(operations);
|
||||
if (++transfers === 2) throw new Error("controlled later transfer failure");
|
||||
return syncOut(operations);
|
||||
};
|
||||
|
||||
Reference in new issue
Block a user