diff --git a/doc/execution-github-identity.md b/doc/execution-github-identity.md index ce3eaf8358d..6a7ce2dbe1a 100644 --- a/doc/execution-github-identity.md +++ b/doc/execution-github-identity.md @@ -14,7 +14,7 @@ Delegated work and interactions persist their originating context. Retries retai Executions with managed GitHub configured receive token-free `git` and `gh` launchers. Each launcher invocation resolves one eligible credential from the current run's accepted identity at operation start. A `gh` command's child Git processes inherit that command's captured identity; later steering does not change already-started operations. -A `git` command that never uses the network does not ask the broker. The launcher reads past the global options, such as `-C` and `-c`, and checks the subcommand against a short list (for example `status`, `diff`, `log`, `rev-parse`, `add`, `config`, `branch`, `checkout`). An unknown subcommand or option still asks. `commit` and `var` still ask, because the managed author comes from the broker answer. The launcher keeps no broker answer between invocations and never writes a credential to a file. +A `git` command that never uses the network does not ask the broker. The launcher reads past the global options, such as `-C` and `-c`, and checks the subcommand against a short list (for example `status`, `diff`, `diff-files`, `write-tree`, `log`, `rev-parse`, `add`, `config`, `branch`, `checkout`). This includes the index comparison and tree writing used by workspace snapshots. An unknown subcommand or option still asks. `commit` and `var` still ask, because the managed author comes from the broker answer. The launcher keeps no broker answer between invocations and never writes a credential to a file. GitHub credential capture reads the current task, project, and retained trust policy bindings without transferring saved prompts, runner profiles, results, diff --git a/doc/plans/2026-10-09-task-workspaces.md b/doc/plans/2026-10-09-task-workspaces.md index 04a4420f1be..ad18204ea45 100644 --- a/doc/plans/2026-10-09-task-workspaces.md +++ b/doc/plans/2026-10-09-task-workspaces.md @@ -74,11 +74,13 @@ Preserve simultaneous local/SSH tasks in an explicitly configured identical phys Concurrent remote sharing requires proven common physical placement, separate session authority and coherent snapshot capability. Unsupported new concurrent sharing returns a typed unavailable result; do not instantiate separate clones and label them shared. Already-admitted descriptors remain recoverable unchanged. Local authoritative roots need no replica copyback; SSH retains its actual existing persistence contract. -Sparse filesystem checkpoints use disk-backed full manifests, changed bytes/deletions, safe-link and existing ignore semantics, and separate nested Git history. Capture still enumerates and hashes files with stable-read validation; a metadata hash cache is deferred. No-change sends zero file-content payload where capability is supported; manifest metadata still transfers; unsupported transport uses explicit confined full fallback. Sparse capture capacity failures before outbound data is read also use the existing confined full transfer, so a tiny output can recover when the sandbox cannot preserve the sparse manifest reserve. Returned manifest, path, digest and payload validation failures remain errors and never trigger fallback. Full checkpoint exports validate the completed staging root before merge, reject unsupported entries, and omit absolute or escaping links without following them; safe relative links remain supported. Reuse immutable durable seed generations, not only warm sandbox contents; reuse still verifies archive digests. If warm identity validation requires a host-current transfer, that fallback retains the same company/workspace seed-cache scope, so a verified generation can supply the transfer without a second Git clone and archive rebuild. Missing or invalid generations still rebuild from the host. When the generation cache declines a new snapshot, warm fallback may replay the current preparation's private durable seed after its existing file-type and digest checks. It does not raise the cache quotas or reuse a prior run's seed. Missing or changed current-run archives retain the host rebuild path. Existing run references pin seeds/deltas until all admitted runs/recovery obligations release them. Retain the final host rescan because the run ownership fence does not exclude external writers to shared folders; do not reuse a stale post-merge manifest. +Sparse filesystem checkpoints use disk-backed full manifests, changed bytes/deletions, safe-link and existing ignore semantics, and separate nested Git history. Capture still enumerates and hashes files with stable-read validation; a metadata hash cache is deferred. No-change sends zero file-content payload where capability is supported; manifest metadata still transfers; unsupported transport uses explicit confined full fallback. Sparse capture capacity failures before outbound data is read also use the existing confined full transfer, so a tiny output can recover when the sandbox cannot preserve the sparse manifest reserve. Returned manifest, path, digest and payload validation failures remain errors and never trigger fallback. Full checkpoint exports validate the completed staging root before merge, reject unsupported entries, and omit absolute or escaping links without following them; safe relative links remain supported. Reuse immutable durable seed generations, not only warm sandbox contents; reuse still verifies archive digests. If warm identity validation requires a host-current transfer, that fallback retains the same company seed-cache scope, so a verified generation can supply the transfer without a second Git clone and archive rebuild. Missing or invalid generations still rebuild from the host. When the generation cache declines a new snapshot, warm fallback may replay the current preparation's private durable seed after its existing file-type and digest checks. It does not raise the cache quotas or reuse a prior run's seed. Missing or changed current-run archives retain the host rebuild path. Existing run references pin seeds/deltas until all admitted runs/recovery obligations release them. Retain the final host rescan because the run ownership fence does not exclude external writers to shared folders; do not reuse a stale post-merge manifest. A returned sparse snapshot must describe a coherent tree before any host mutation. Every nested entry requires an immediate parent marked as a directory in that same returned snapshot, including baseline-unchanged files. Checking every directory also verifies recursive ancestry. Top-level entries use the implicit root. Invalid trees fail without overwriting retained host files and keep recovery evidence. The check uses the existing indexed manifest interface; it adds no host-path scans or unbounded in-memory inventory. -Seed recovery deliberately uses **no delta chain** in this implementation. An unchanged generation reuses its verified content-addressed immutable full seed; a changed generation creates a fresh full seed. Sparse output transfer is separate from seed recovery. This revises the sketch's 16-delta compaction proposal: composing recovery generations would add a new failure surface before the simpler cache has been qualified. Existing references retain old seeds; no automatic eviction/deletion policy is introduced. Optional cache publication is capped at 16 generations/1 GiB per workspace and 256 generations/256 workspace directories/8 GiB per company. A company-scoped filesystem lock serializes final quota checks and publication across workspace roots. Optional seed reconstruction and baseline verification run outside that lock, alongside inbound upload. Preparation settles both operations before returning or removing temporary archives. Publication rechecks capacity and directory identity under the lock after verification; a competing publication can consume the remaining capacity and cause this optional cache write to be skipped. Archive and receipt bytes, corrupt entries and abandoned pending generations count against capacity; publication also preserves the manifest storage free-space reserve (256 MiB by default), accounting for a full-copy fallback. Saturation or cache failure skips publication and keeps the already-written per-run recovery seed. Existing over-capacity caches stop growing without deleting admitted evidence. This deliberately trades cache hit rate after saturation for bounded additional storage and unchanged recovery semantics; no global cross-company quota is added. Changed input generations still pay full snapshot creation. New cache entries are reconstructed and checked against the admitted baseline before publication, so concurrent writes cannot label an archive as the wrong generation; this adds cold verification work. A corrupted cache entry remains a miss and falls back to a fresh seed; automatic cache repair and garbage collection are deferred. The existing Git seed contract captures the selected branch/HEAD, not unrelated refs. +Seed recovery deliberately uses **no delta chain** in this implementation. An unchanged generation reuses its verified content-addressed immutable full seed; a changed generation creates a fresh full seed. Sparse output transfer is separate from seed recovery. This revises the sketch's 16-delta compaction proposal: composing recovery generations would add a new failure surface before the simpler cache has been qualified. Existing references retain their private recovery seeds. Optional cache publication is capped at 16 generations/1 GiB per bucket and 256 generations/256 cache directories/8 GiB per company. Existing idle and capacity eviction rules apply only to the optional cache. A company-scoped filesystem lock serializes final quota checks and publication across workspace roots. Optional seed reconstruction and baseline verification run outside that lock, alongside inbound upload. Preparation settles both operations before returning or removing temporary archives. Publication rechecks capacity and directory identity under the lock after verification; a competing publication can consume the remaining capacity and cause this optional cache write to be skipped. Archive and receipt bytes, corrupt entries and abandoned pending generations count against capacity; publication also preserves the manifest storage free-space reserve (256 MiB by default), accounting for a full-copy fallback. Saturation or cache failure skips publication and keeps the already-written per-run recovery seed. Existing over-capacity caches stop growing without deleting admitted evidence. This deliberately trades cache hit rate after saturation for bounded additional storage and unchanged recovery semantics; no global cross-company quota is added. Changed input generations still pay full snapshot creation. New cache entries are reconstructed and checked against the admitted baseline before publication, so concurrent writes cannot label an archive as the wrong generation; this adds cold verification work. A corrupted cache entry remains a miss and falls back to a fresh seed; automatic cache repair and garbage collection are deferred. The private per-run Git seed captures the selected branch/HEAD, not unrelated refs. + +Native host-current staging uses a company-shared `shared-v1` cache bucket. Its generation binds filesystem contents, Git HEAD, sanitized origin, and nested repository layout; branch names are applied separately. The shared Git archive has detached HEADs and no task branches. Staging copies that immutable archive and appends each selected branch/HEAD before transferring it or saving the private per-run recovery archive. Private archives therefore remain usable by the existing durable-seed replay path. Changing source content, Git HEAD, origin, or repository layout misses the cache; another company cannot read this bucket. The shared bucket keeps the existing 16-generation/1 GiB bucket limit and the existing aggregate company limits. Legacy workspace cache directories remain subject to those aggregate limits and eviction rules. Ordinary settlement remains host-durable before successful completion. Preserve export-only recovery and exact-source lease retention. Preserve the specific unsafe-export omission exception and its run-log-only diagnostic. Do not change global merge conflict semantics as a side effect. diff --git a/packages/adapter-utils/src/command-managed-runtime.ts b/packages/adapter-utils/src/command-managed-runtime.ts index 50265a7bfd0..68321689715 100644 --- a/packages/adapter-utils/src/command-managed-runtime.ts +++ b/packages/adapter-utils/src/command-managed-runtime.ts @@ -551,6 +551,7 @@ export async function prepareCommandManagedRuntime(input: { onWorkspaceCheckpoint?: (metrics: WorkspaceCheckpointMetrics) => Promise; workspaceSeedCacheDirectory?: string; workspaceSeedCacheCompanyDirectory?: string; + workspaceSeedCacheScope?: "workspace" | "company"; workspaceBaseline?: DirectorySnapshot; workspaceGitSnapshot?: GitWorkspaceSnapshot | null; workspaceRepositories?: NonNullable; @@ -626,6 +627,7 @@ export async function prepareCommandManagedRuntime(input: { onWorkspaceCheckpoint: input.onWorkspaceCheckpoint, workspaceSeedCacheDirectory: input.workspaceSeedCacheDirectory, workspaceSeedCacheCompanyDirectory: input.workspaceSeedCacheCompanyDirectory, + workspaceSeedCacheScope: input.workspaceSeedCacheScope, workspaceBaseline: input.workspaceBaseline, workspaceGitSnapshot: input.workspaceGitSnapshot, workspaceRepositories: input.workspaceRepositories, @@ -678,6 +680,7 @@ export async function prepareCommandManagedRuntime(input: { onWorkspaceCheckpoint: input.onWorkspaceCheckpoint, workspaceSeedCacheDirectory: input.workspaceSeedCacheDirectory, workspaceSeedCacheCompanyDirectory: input.workspaceSeedCacheCompanyDirectory, + workspaceSeedCacheScope: input.workspaceSeedCacheScope, workspaceBaseline: input.workspaceBaseline, workspaceGitSnapshot: input.workspaceGitSnapshot, workspaceRepositories: input.workspaceRepositories, diff --git a/packages/adapter-utils/src/execution-target.ts b/packages/adapter-utils/src/execution-target.ts index cd8f00574f3..d166ed95ce7 100644 --- a/packages/adapter-utils/src/execution-target.ts +++ b/packages/adapter-utils/src/execution-target.ts @@ -1451,6 +1451,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: { onWorkspaceCheckpoint?: (metrics: WorkspaceCheckpointMetrics) => Promise; workspaceSeedCacheDirectory?: string; workspaceSeedCacheCompanyDirectory?: string; + workspaceSeedCacheScope?: "workspace" | "company"; workspaceBaseline?: DirectorySnapshot; workspaceGitSnapshot?: GitWorkspaceSnapshot | null; workspaceRepositories?: NonNullable; @@ -1546,6 +1547,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: { onWorkspaceCheckpoint: input.onWorkspaceCheckpoint, workspaceSeedCacheDirectory: input.workspaceSeedCacheDirectory, workspaceSeedCacheCompanyDirectory: input.workspaceSeedCacheCompanyDirectory, + workspaceSeedCacheScope: input.workspaceSeedCacheScope, workspaceBaseline: input.workspaceBaseline, workspaceGitSnapshot: input.workspaceGitSnapshot, workspaceRepositories: input.workspaceRepositories, diff --git a/packages/adapter-utils/src/github-launcher.test.ts b/packages/adapter-utils/src/github-launcher.test.ts index 7101b86ed43..5dc86798bd8 100644 --- a/packages/adapter-utils/src/github-launcher.test.ts +++ b/packages/adapter-utils/src/github-launcher.test.ts @@ -11,6 +11,35 @@ const cleanups: Array<() => Promise> = []; afterEach(async () => { for (const cleanup of cleanups.splice(0).reverse()) await cleanup(); }); describe("managed GitHub launchers", () => { + it("snapshots a staged tree and reports unstaged files without capturing GitHub credentials", async () => { + const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-github-snapshot-")); + cleanups.push(() => rm(root, { recursive: true, force: true })); + const bin = path.join(root, "managed"), repo = path.join(root, "repo"); + await mkdir(bin); await mkdir(repo); + await exec("git", ["init", "-q", repo]); + await writeFile(path.join(repo, "tracked.txt"), "staged contents\n"); + await exec("git", ["-C", repo, "add", "tracked.txt"]); + const expectedTree = (await exec("git", ["-C", repo, "write-tree"])).stdout; + await writeFile(path.join(repo, "tracked.txt"), "unstaged contents\n"); + await writeFile(path.join(bin, "git"), githubLauncherSource(), { mode: 0o700 }); + let captures = 0; + const server = createServer((_req, res) => { + captures++; + res.setHeader("content-type", "application/json"); + res.end(JSON.stringify({ status: "unavailable", reason: "No GitHub connection" })); + }); + await new Promise(resolve => server.listen(0, "127.0.0.1", resolve)); + cleanups.push(() => new Promise(resolve => server.close(() => resolve()))); + const { port } = server.address() as { port: number }; + const env = { ...process.env, ...githubBrokerEnvironment({}, { url: `http://127.0.0.1:${port}`, token: "run-capability" }), PATH: `${bin}:${process.env.PATH}` }; + const git = (...args: string[]) => exec(path.join(bin, "git"), args, { cwd: root, env }); + const tree = await git("--git-dir", path.join(repo, ".git"), "--work-tree", repo, "write-tree"); + const diff = await git("-C", repo, "-c", "core.quotepath=false", "diff-files", "--name-only", "-z", "--", "."); + expect(tree.stdout).toBe(expectedTree); + expect(diff.stdout).toBe("tracked.txt\0"); + expect(captures).toBe(0); + }); + it.each(["missing", "absent", "unavailable", "rejected"])("commits local work without an identity when GitHub access is %s", async (access) => { const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-github-default-author-")); cleanups.push(() => rm(root, { recursive: true, force: true })); diff --git a/packages/adapter-utils/src/github-launcher.ts b/packages/adapter-utils/src/github-launcher.ts index 2b16211745c..6c750c30447 100644 --- a/packages/adapter-utils/src/github-launcher.ts +++ b/packages/adapter-utils/src/github-launcher.ts @@ -17,7 +17,7 @@ if (!['git', 'gh'].includes(program) || !executable) { process.stderr.write('Paperclip: requested GitHub command is not installed.\n'); process.exit(127); } -const localGitCommands = new Set(['init', 'status', 'add', 'config', 'rev-parse', 'log', 'diff', 'show', 'branch', 'checkout', 'switch', 'restore', 'reset', 'rm', 'mv', 'worktree', 'grep', 'blame', 'describe', 'ls-files', 'ls-tree', 'cat-file', 'rev-list', 'merge-base', 'show-ref', 'for-each-ref', 'symbolic-ref', 'check-ignore', 'version', 'help']); +const localGitCommands = new Set(['init', 'status', 'add', 'config', 'rev-parse', 'log', 'diff', 'diff-files', 'write-tree', 'show', 'branch', 'checkout', 'switch', 'restore', 'reset', 'rm', 'mv', 'worktree', 'grep', 'blame', 'describe', 'ls-files', 'ls-tree', 'cat-file', 'rev-list', 'merge-base', 'show-ref', 'for-each-ref', 'symbolic-ref', 'check-ignore', 'version', 'help']); const gitValueOptions = ['-C', '-c', '--git-dir', '--work-tree', '--namespace', '--config-env', '--super-prefix', '--attr-source']; const gitFlagOptions = ['-p', '-P', '--paginate', '--no-pager', '--bare', '--no-replace-objects', '--no-lazy-fetch', '--no-optional-locks', '--no-advice', '--literal-pathspecs', '--glob-pathspecs', '--noglob-pathspecs', '--icase-pathspecs']; function isLocalGit(args) { diff --git a/packages/adapter-utils/src/sandbox-managed-runtime.ts b/packages/adapter-utils/src/sandbox-managed-runtime.ts index 0a034cf931f..4114d9cd3ee 100644 --- a/packages/adapter-utils/src/sandbox-managed-runtime.ts +++ b/packages/adapter-utils/src/sandbox-managed-runtime.ts @@ -1,5 +1,5 @@ import type { WorkspaceCheckpointMetrics } from "./workspace-checkpoint.js"; -import { workspaceSeedGeneration, readWorkspaceSeedGeneration, publishWorkspaceSeedGeneration } from "./workspace-seed-cache.js"; +import { workspaceSeedGeneration, workspaceSeedCloneSnapshot, readWorkspaceSeedGeneration, publishWorkspaceSeedGeneration } from "./workspace-seed-cache.js"; import { writeCheckpointBaseline, workspaceCheckpointScript, readCheckpointSnapshot, validateCheckpointPayload } from "./workspace-checkpoint.js"; import { execFile as execFileCallback } from "node:child_process"; import { createHash, randomUUID } from "node:crypto"; @@ -724,6 +724,40 @@ function buildRemoveDeletedPathsCommand(input: { remoteDir: string; manifestPath return `cd ${shellQuote(input.remoteDir)} && xargs -0 -r -n 64 sh -c ${shellQuote(remove)} sh < ${shellQuote(input.manifestPath)} && rm -f -- ${shellQuote(input.manifestPath)}`; } +async function bindWorkspaceSeedGitArchive(input: { + sourcePath: string; targetPath: string; directory: string; + git: GitWorkspaceSnapshot | null; repositories: NonNullable; +}): Promise { + await fs.copyFile(input.sourcePath, input.targetPath, fsConstants.COPYFILE_FICLONE); + const entries: string[] = []; + const bind = async (relative: string, snapshot: GitWorkspaceSnapshot) => { + if (!/^(?:[a-f0-9]{40}|[a-f0-9]{64})$/.test(snapshot.headCommit)) throw new Error("workspace_seed_git_head_invalid"); + const headPath = path.posix.join(relative, ".git", "HEAD"); + await fs.mkdir(path.dirname(path.join(input.directory, headPath)), { recursive: true }); + if (snapshot.branchName === null) await fs.writeFile(path.join(input.directory, headPath), `${snapshot.headCommit}\n`); + else { + await execFile("git", ["check-ref-format", "--branch", snapshot.branchName]); + const ref = `refs/heads/${snapshot.branchName}`; + const refPath = path.posix.join(relative, ".git", ref); + await fs.mkdir(path.dirname(path.join(input.directory, refPath)), { recursive: true }); + await fs.writeFile(path.join(input.directory, refPath), `${snapshot.headCommit}\n`); + await fs.writeFile(path.join(input.directory, headPath), `ref: ${ref}\n`); + entries.push(refPath); + } + entries.push(headPath); + for (const repository of snapshot.repositories ?? []) await bindChild(relative, repository); + }; + const bindChild = async (relative: string, repository: (typeof input.repositories)[number]) => { + if (path.posix.isAbsolute(repository.path) || repository.path.split("/").includes("..")) throw new Error("workspace_seed_git_path_invalid"); + await bind(path.posix.join(relative, repository.path), repository.snapshot); + }; + if (input.git) await bind("", input.git); + else for (const repository of input.repositories) await bindChild("", repository); + const list = path.join(input.directory, "entries.nul"); + await fs.writeFile(list, entries.join("\0") + "\0"); + await execTar(["-rf", input.targetPath, "-C", input.directory, "--null", "-T", list]); +} + function buildUniqueStagingPath(input: { targetPath: string; suffix: string }): string { return path.join(path.dirname(input.targetPath), `${input.suffix}.${randomUUID()}`); } @@ -1227,6 +1261,7 @@ export async function prepareSandboxManagedRuntime(input: { onWorkspaceCheckpoint?: (metrics: WorkspaceCheckpointMetrics) => Promise; workspaceSeedCacheDirectory?: string; workspaceSeedCacheCompanyDirectory?: string; + workspaceSeedCacheScope?: "workspace" | "company"; /** Durable snapshots supplied when reconstructing an interrupted restore. */ workspaceBaseline?: DirectorySnapshot; workspaceGitSnapshot?: GitWorkspaceSnapshot | null; @@ -1417,11 +1452,12 @@ export async function prepareSandboxManagedRuntime(input: { } const seedGeneration = input.workspaceSeedCacheDirectory && baselineSnapshot && workspaceInboundMode !== "durable_seed" - ? await runStepSpan("workspace.seed_generation", async () => workspaceSeedGeneration(baselineSnapshot!, gitSnapshot, repositories)) : null; + ? await runStepSpan("workspace.seed_generation", async () => workspaceSeedGeneration(baselineSnapshot!, gitSnapshot, repositories, input.workspaceSeedCacheScope)) : null; const cachedSeed = seedGeneration ? await runStepSpan("workspace.seed_lookup", () => readWorkspaceSeedGeneration(input.workspaceSeedCacheDirectory!, seedGeneration, { companyDirectory: input.workspaceSeedCacheCompanyDirectory })) : null; const replaySeed = workspaceInboundMode === "durable_seed" ? input.workspaceDurableSeed : cachedSeed; + const sharedSeed = input.workspaceSeedCacheScope === "company" && seedGeneration !== null; // Every inbound staging step delegates to the provider through `client.syncIn`: // the orchestrator no longer inlines `writeFile`+`run` or chooses a transport, @@ -1558,6 +1594,7 @@ export async function prepareSandboxManagedRuntime(input: { const workspaceFiles: SandboxSyncFileMapping[] = []; const workspacePostUploadCommands: SandboxPostUploadCommand[] = []; let workspaceUploadBytes = 0; + const gitSeedTarPath = path.join(tempDir, sharedSeed ? "git-seed.tar" : "git-workspace.tar"); // Build both host tarballs (the git-history tar and the workspace-overlay // tar) inside one host span named `pack`. This span makes the host pack @@ -1585,24 +1622,27 @@ export async function prepareSandboxManagedRuntime(input: { } await runStepSpan("workspace.seed_replay", () => copyDurableSeedArchive({ sourcePath: durableGitArchive, - targetPath: gitTarPath, + targetPath: gitSeedTarPath, expectedSha256: replaySeed.gitArchiveSha256, })); } else { - const archive = async (cloneDir: string) => runStepSpan("workspace.git_archive", () => createTarballFromDirectory({ localDir: cloneDir, archivePath: gitTarPath, exclude: [".paperclip-runtime"] })); - if (gitSnapshot) await runStepSpan("workspace.git_clone", () => withShallowGitWorkspaceClone({ localDir: input.workspaceLocalDir, snapshot: gitSnapshot }, archive)); + const archive = async (cloneDir: string) => runStepSpan("workspace.git_archive", () => createTarballFromDirectory({ localDir: cloneDir, archivePath: gitSeedTarPath, exclude: [".paperclip-runtime"] })); + const snapshot = (git: GitWorkspaceSnapshot) => sharedSeed ? workspaceSeedCloneSnapshot(git) : git; + if (gitSnapshot) await runStepSpan("workspace.git_clone", () => withShallowGitWorkspaceClone({ localDir: input.workspaceLocalDir, snapshot: snapshot(gitSnapshot) }, archive)); else await runStepSpan("workspace.git_clone", () => withTempDir("paperclip-managed-repositories-", async (cloneDir) => { - for (const repo of repositories) await withShallowGitWorkspaceClone({ localDir: path.join(input.workspaceLocalDir, repo.path), snapshot: repo.snapshot }, async (child) => copyCloneTree(child, path.join(cloneDir, repo.path))); + for (const repo of repositories) await withShallowGitWorkspaceClone({ localDir: path.join(input.workspaceLocalDir, repo.path), snapshot: snapshot(repo.snapshot) }, async (child) => copyCloneTree(child, path.join(cloneDir, repo.path))); await archive(cloneDir); })); - if (input.workspaceDurableSeed?.gitArchivePath) { - await runStepSpan("workspace.seed_persist", () => persistDurableSeedArchive({ - sourcePath: gitTarPath, - targetPath: input.workspaceDurableSeed!.gitArchivePath!, - })); - } } - if (cachedSeed && input.workspaceDurableSeed?.gitArchivePath) await runStepSpan("workspace.seed_persist", () => persistDurableSeedArchive({ sourcePath: gitTarPath, targetPath: input.workspaceDurableSeed!.gitArchivePath! })); + if (sharedSeed) await bindWorkspaceSeedGitArchive({ + sourcePath: gitSeedTarPath, targetPath: gitTarPath, directory: path.join(tempDir, "git-binding"), git: gitSnapshot, repositories, + }); + if ((!replaySeed || cachedSeed) && input.workspaceDurableSeed?.gitArchivePath) { + await runStepSpan("workspace.seed_persist", () => persistDurableSeedArchive({ + sourcePath: gitTarPath, + targetPath: input.workspaceDurableSeed!.gitArchivePath!, + })); + } workspaceFiles.push({ sourcePath: gitTarPath, targetPath: remoteGitTar, @@ -1691,13 +1731,15 @@ export async function prepareSandboxManagedRuntime(input: { const publishSeed = async () => { if (seedGeneration && !cachedSeed && input.workspaceDurableSeed) { const seed = input.workspaceDurableSeed; + const gitArchivePath = gitSnapshot || repositories.length + ? (sharedSeed ? gitSeedTarPath : seed.gitArchivePath) : null; await runStepSpan("workspace.seed_publish", () => publishWorkspaceSeedGeneration(input.workspaceSeedCacheDirectory!, seedGeneration, - { ...seed, gitArchivePath: gitSnapshot || repositories.length ? seed.gitArchivePath : null }, { + { ...seed, gitArchivePath }, { companyDirectory: input.workspaceSeedCacheCompanyDirectory, verify: () => runStepSpan("seed.verify", async () => { const verifiedRoot = path.join(tempDir, "verify-seed"); try { - if (gitSnapshot || repositories.length) await runStepSpan("workspace.seed_extract", () => extractTarballToDirectory({ archivePath: path.join(tempDir, "git-workspace.tar"), localDir: verifiedRoot })); + if (gitSnapshot || repositories.length) await runStepSpan("workspace.seed_extract", () => extractTarballToDirectory({ archivePath: gitSeedTarPath, localDir: verifiedRoot })); await runStepSpan("workspace.seed_extract", () => extractTarballToDirectory({ archivePath: path.join(tempDir, "workspace.tar"), localDir: verifiedRoot, exclude: repositoryMetadataExcludes })); for (const relative of workspacePaths(deletedPaths)) { let parent = verifiedRoot; diff --git a/packages/adapter-utils/src/workspace-seed-cache.ts b/packages/adapter-utils/src/workspace-seed-cache.ts index 003f1640c6c..a48a8392d20 100644 --- a/packages/adapter-utils/src/workspace-seed-cache.ts +++ b/packages/adapter-utils/src/workspace-seed-cache.ts @@ -7,12 +7,18 @@ import { assertWorkspaceManifestDiskSpace } from "./workspace-manifest.js"; import type { GitWorkspaceSnapshot } from "./git-workspace-sync.js"; import type { WorkspaceDurableSeedPaths } from "./sandbox-managed-runtime.js"; -export function workspaceSeedGeneration(baseline: DirectorySnapshot, git: GitWorkspaceSnapshot | null, repositories: NonNullable = []): string { +export function workspaceSeedGeneration(baseline: DirectorySnapshot, git: GitWorkspaceSnapshot | null, repositories: NonNullable = [], scope: "workspace" | "company" = "workspace"): string { const identity = (snapshot: GitWorkspaceSnapshot | null): unknown => snapshot && ({ - head: snapshot.headCommit, branch: snapshot.branchName, origin: snapshot.originUrl, + head: snapshot.headCommit, ...(scope === "workspace" ? { branch: snapshot.branchName } : {}), origin: snapshot.originUrl, repositories: snapshot.repositories?.map((repo) => ({ path: repo.path, git: identity(repo.snapshot) })), }); - return createHash("sha256").update("verified-seed-v1\0").update(JSON.stringify([directorySnapshotSha256(baseline), identity(git), repositories.map((repo) => ({ path: repo.path, git: identity(repo.snapshot) }))])).digest("hex"); + return createHash("sha256").update(scope === "company" ? "verified-company-seed-v1\0" : "verified-seed-v1\0").update(JSON.stringify([directorySnapshotSha256(baseline), identity(git), repositories.map((repo) => ({ path: repo.path, git: identity(repo.snapshot) }))])).digest("hex"); +} + +export function workspaceSeedCloneSnapshot(snapshot: GitWorkspaceSnapshot): GitWorkspaceSnapshot { + return { ...snapshot, branchName: null, + ...(snapshot.repositories ? { repositories: snapshot.repositories.map(repository => ({ ...repository, snapshot: workspaceSeedCloneSnapshot(repository.snapshot) })) } : {}), + }; } async function digest(file: string): Promise { const hash = createHash("sha256"); diff --git a/packages/adapter-utils/src/workspace-seed-reuse.test.ts b/packages/adapter-utils/src/workspace-seed-reuse.test.ts new file mode 100644 index 00000000000..1e0837c4f51 --- /dev/null +++ b/packages/adapter-utils/src/workspace-seed-reuse.test.ts @@ -0,0 +1,178 @@ +import { execFile } from "node:child_process"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { promisify } from "node:util"; +import { afterEach, expect, it, vi } from "vitest"; +import { prepareSandboxManagedRuntime, type SandboxManagedRuntimeClient, type SandboxSyncOperation } from "./sandbox-managed-runtime.js"; +import { readWorkspaceSeedGeneration, workspaceSeedGeneration } from "./workspace-seed-cache.js"; +import { captureDirectorySnapshot, disposeDirectorySnapshot } from "./workspace-restore-merge.js"; +import type { GitWorkspaceSnapshot } from "./git-workspace-sync.js"; +import { prepareAdapterExecutionTargetRuntime } from "./execution-target.js"; +import { runChildProcess } from "./server-utils.js"; + +const exec = promisify(execFile); +const roots: string[] = []; +afterEach(async () => { + vi.unstubAllEnvs(); + await Promise.all(roots.splice(0).map(root => fs.rm(root, { recursive: true, force: true }))); +}); +const git = async (cwd: string, ...args: string[]) => (await exec("git", args, { cwd })).stdout.trim(); + +function client(): SandboxManagedRuntimeClient { + const sync = async (operations: SandboxSyncOperation[]) => ({ operations: await Promise.all(operations.map(async operation => { + let bytesTransferred = 0; + for (const file of operation.files) { + await fs.mkdir(path.dirname(file.targetPath), { recursive: true }); + if (file.kind === "directory") await fs.cp(file.sourcePath, file.targetPath, { recursive: true, verbatimSymlinks: true }); + else { + const bytes = await fs.readFile(file.sourcePath); + await fs.writeFile(file.targetPath, bytes); + bytesTransferred += bytes.length; + } + } + for (const command of operation.postUploadCommands ?? []) await exec("sh", ["-c", command.command]); + return { operationId: operation.operationId, filesTransferred: operation.files.length, bytesTransferred }; + })) }); + return { + makeDir: directory => fs.mkdir(directory, { recursive: true }).then(() => {}), + writeFile: (file, bytes) => fs.writeFile(file, Buffer.from(bytes)), + readFile: file => fs.readFile(file), + listFiles: async directory => (await fs.readdir(directory, { withFileTypes: true })).filter(entry => entry.isFile()).map(entry => entry.name), + remove: file => fs.rm(file, { recursive: true, force: true }), + run: command => exec("sh", ["-c", command]).then(() => {}), + syncIn: sync, + syncOut: sync, + }; +} + +it.each([ + { nested: false, secondBranch: "issue/second" }, + { nested: true, secondBranch: "issue/second" }, + { nested: false, secondBranch: null }, + { nested: true, secondBranch: null }, +])("reuses company seed content and restores branch identity ($nested, $secondBranch)", async ({ nested, secondBranch }) => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-shared-seed-")); + roots.push(root); + vi.stubEnv("PAPERCLIP_HOME", path.join(root, "home")); + const source = path.join(root, "source"); + const repositories = nested ? ["", ".paperclip-repositories/backend"] : [""]; + for (const relative of repositories) { + const cwd = path.join(source, relative); + await fs.mkdir(cwd, { recursive: true }); + await git(cwd, "init", "-q", "-b", "main"); + await fs.writeFile(path.join(cwd, "tracked.txt"), "committed bytes\n"); + await fs.writeFile(path.join(cwd, "deleted.txt"), "delete in host\n"); + await fs.writeFile(path.join(cwd, ".gitignore"), "secret.txt\n.paperclip-repositories/\n"); + await fs.symlink("tracked.txt", path.join(cwd, "link.txt")); + await git(cwd, "add", "."); + await git(cwd, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "Seed"); + } + const company = path.join(root, "company"); + const prepare = async (name: string, branch: string | null, companyDirectory = company) => { + const host = path.join(root, name, "host"), remote = path.join(root, name, "remote"); + for (const relative of repositories) { + const cwd = path.join(host, relative); + await fs.mkdir(path.dirname(cwd), { recursive: true }); + await exec("git", ["clone", "--shared", "-q", path.join(source, relative), cwd]); + await git(cwd, "checkout", "-q", ...(branch ? ["-b", branch] : ["--detach", "HEAD"])); + await fs.writeFile(path.join(cwd, "tracked.txt"), "working-tree bytes\n"); + await fs.rm(path.join(cwd, "deleted.txt")); + await fs.writeFile(path.join(cwd, "new.txt"), "untracked overlay\n"); + await fs.writeFile(path.join(cwd, "secret.txt"), "must stay in host\n"); + } + const phases: string[] = []; + const durable = path.join(root, name, "durable"); + const input: Parameters[0] = { + spec: { transport: "sandbox", provider: "filesystem", sandboxId: name, remoteCwd: remote, timeoutMs: 30_000, apiKey: null }, + adapterKey: "test", client: client(), workspaceLocalDir: host, workspaceCheckpoint: true, + workspaceSeedCacheScope: "company", + workspaceSeedCacheDirectory: path.join(companyDirectory, "shared-v1"), workspaceSeedCacheCompanyDirectory: companyDirectory, + workspaceDurableSeed: { workspaceArchivePath: path.join(durable, "workspace.tar"), gitArchivePath: path.join(durable, "git.tar") }, + runtimeSpan: async (name, work) => { phases.push(name); return work(); }, + }; + const transport = client(); + const prepared = await prepareAdapterExecutionTargetRuntime({ + ...input, runId: name, + target: { + kind: "remote", transport: "sandbox", providerKey: "filesystem", remoteCwd: remote, + runner: { + syncIn: operations => transport.syncIn!(operations), syncOut: operations => transport.syncOut!(operations), + execute: command => runChildProcess(name, command.command, command.args ?? [], { + cwd: command.cwd ?? root, env: command.env ?? {}, stdin: command.stdin, + timeoutSec: 30, graceSec: 1, onLog: command.onLog ?? (async () => {}), + }), + }, + }, + ...(nested ? { installCommand: "exit 1", detectCommand: "git" } : {}), + }); + return { prepared, host, remote, phases, input }; + }; + const first = await prepare("first", "issue/first"); + const second = await prepare("second", secondBranch); + for (const relative of repositories) { + const cwd = path.join(second.remote, relative); + expect(await git(cwd, "rev-parse", "--abbrev-ref", "HEAD")).toBe(secondBranch ?? "HEAD"); + expect(await git(cwd, "branch", "--list", "issue/first")).toBe(""); + expect(await fs.readFile(path.join(cwd, "tracked.txt"), "utf8")).toBe("working-tree bytes\n"); + expect(await fs.readFile(path.join(cwd, "new.txt"), "utf8")).toBe("untracked overlay\n"); + expect(await fs.readlink(path.join(cwd, "link.txt"))).toBe("tracked.txt"); + await expect(fs.stat(path.join(cwd, "deleted.txt"))).rejects.toMatchObject({ code: "ENOENT" }); + await expect(fs.stat(path.join(cwd, "secret.txt"))).rejects.toMatchObject({ code: "ENOENT" }); + } + expect(first.phases).toContain("workspace.git_clone"); + expect(second.phases).not.toContain("workspace.git_clone"); + expect(second.phases).not.toContain("workspace.seed_publish"); + const generations = await fs.readdir(path.join(company, "shared-v1")); + expect(generations).toHaveLength(1); + expect(await readWorkspaceSeedGeneration(path.join(company, "shared-v1"), generations[0]!, { companyDirectory: company })).not.toBeNull(); + const recovery = path.join(root, "replacement"); + const recovered = await prepareSandboxManagedRuntime({ + ...second.input, + spec: { ...second.input.spec, remoteCwd: recovery }, + workspaceInboundMode: "durable_seed", + workspaceSeedCacheScope: undefined, workspaceSeedCacheDirectory: undefined, workspaceSeedCacheCompanyDirectory: undefined, + workspaceBaseline: second.prepared.workspaceSyncSnapshot!.baseline, + workspaceGitSnapshot: second.prepared.workspaceSyncSnapshot!.gitSnapshot, + }); + second.remote = recovery; + for (const relative of repositories) { + const cwd = path.join(second.remote, relative); + expect(await git(cwd, "rev-parse", "--abbrev-ref", "HEAD")).toBe(secondBranch ?? "HEAD"); + expect(await fs.readFile(path.join(cwd, "new.txt"), "utf8")).toBe("untracked overlay\n"); + await fs.writeFile(path.join(cwd, "tracked.txt"), "agent edit\n"); + await git(cwd, "add", "tracked.txt"); + await git(cwd, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "Agent change"); + } + await recovered.restoreWorkspace(); + for (const relative of repositories) { + expect(await git(path.join(second.host, relative), "rev-parse", "--abbrev-ref", "HEAD")).toBe(secondBranch ?? "HEAD"); + expect(await fs.readFile(path.join(second.host, relative, "tracked.txt"), "utf8")).toBe("agent edit\n"); + expect(await fs.readFile(path.join(second.host, relative, "secret.txt"), "utf8")).toBe("must stay in host\n"); + expect(await fs.readFile(path.join(first.host, relative, "tracked.txt"), "utf8")).toBe("working-tree bytes\n"); + } + await first.prepared.cleanupWorkspaceSnapshot?.(); + const isolated = await prepare("other-company", secondBranch, path.join(root, "other-company-cache")); + expect(isolated.phases).toContain("workspace.git_clone"); + await isolated.prepared.cleanupWorkspaceSnapshot?.(); +}, 30_000); + +it("separates shared seeds when file contents, Git head, origin or repository layout differ", async () => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-seed-identity-")); + roots.push(root); + await fs.writeFile(path.join(root, "file.txt"), "first bytes"); + const first = await captureDirectorySnapshot(root); + const git: GitWorkspaceSnapshot = { headCommit: "a".repeat(40), branchName: "first", originUrl: "https://example.test/one.git", overlayPaths: [], deletedPaths: [], ignoredPaths: [] }; + const generation = workspaceSeedGeneration(first, git, [], "company"); + expect(workspaceSeedGeneration(first, { ...git, branchName: "second" }, [], "company")).toBe(generation); + expect(workspaceSeedGeneration(first, git)).not.toBe(workspaceSeedGeneration(first, { ...git, branchName: "second" })); + expect(workspaceSeedGeneration(first, { ...git, headCommit: "b".repeat(40) }, [], "company")).not.toBe(generation); + expect(workspaceSeedGeneration(first, { ...git, originUrl: "https://example.test/two.git" }, [], "company")).not.toBe(generation); + expect(workspaceSeedGeneration(first, git, [{ path: "nested", snapshot: git }], "company")).not.toBe(generation); + expect(workspaceSeedGeneration(first, null, [{ path: "first", snapshot: git }], "company")) + .not.toBe(workspaceSeedGeneration(first, null, [{ path: "second", snapshot: git }], "company")); + await fs.writeFile(path.join(root, "file.txt"), "second bytes"); + const second = await captureDirectorySnapshot(root); + expect(workspaceSeedGeneration(second, git, [], "company")).not.toBe(generation); + await disposeDirectorySnapshot(first); await disposeDirectorySnapshot(second); +}); diff --git a/server/src/__tests__/native-workspace-sync-history.test.ts b/server/src/__tests__/native-workspace-sync-history.test.ts index ef1c83ba0f4..87585c21b9b 100644 --- a/server/src/__tests__/native-workspace-sync-history.test.ts +++ b/server/src/__tests__/native-workspace-sync-history.test.ts @@ -49,7 +49,7 @@ describe("native warm workspace Git history", () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); }); - it.each(["unchanged", "host_commit", "host_branch", "host_origin", "remote_commit", "remote_commit_corrupt_seed", "remote_commit_missing_seed", "remote_commit_cache_refused", "host_bytes_cache_refused", "remote_commit_missing_run_seed", "remote_commit_corrupt_run_seed", "nested_commit", "plain_nested_commit"] as const)( + it.each(["unchanged", "host_commit", "host_branch", "new_workspace_branch", "host_origin", "remote_commit", "remote_commit_corrupt_seed", "remote_commit_missing_seed", "remote_commit_cache_refused", "host_bytes_cache_refused", "remote_commit_missing_run_seed", "remote_commit_corrupt_run_seed", "nested_commit", "plain_nested_commit"] as const)( "checks Git identity before reusing a warm sandbox: %s", async (change) => { const invalidRunSeed = change === "remote_commit_missing_run_seed" || change === "remote_commit_corrupt_run_seed"; const cacheRefused = change === "remote_commit_cache_refused" || change === "host_bytes_cache_refused" || invalidRunSeed; @@ -60,7 +60,7 @@ describe("native warm workspace Git history", () => { process.env.PAPERCLIP_HOME = path.join(root, "home"); process.env.PAPERCLIP_INSTANCE_ID = "test"; const host = path.join(root, "host"); - const remote = path.join(root, "remote"); + let remote = path.join(root, "remote"); async function initRepo(dir: string) { await mkdir(dir, { recursive: true }); await git(dir, "init", "-b", "work"); @@ -116,8 +116,12 @@ describe("native warm workspace Git history", () => { if (change === "host_commit" || change === "nested_commit" || change === "plain_nested_commit") { await git(change === "nested_commit" || change === "plain_nested_commit" ? path.join(host, nestedPath) : host, "commit", "--allow-empty", "-m", "host only"); - } else if (change === "host_branch") { + } else if (change === "host_branch" || change === "new_workspace_branch") { await git(host, "checkout", "-b", "other"); + if (change === "new_workspace_branch") { + remote = path.join(root, "another-remote"); + lease.metadata = {}; + } } else if (change === "host_origin") { await git(host, "remote", "add", "origin", "https://github.com/paperclipai/new-source.git"); } else if (change === "host_bytes_cache_refused") { @@ -127,7 +131,7 @@ describe("native warm workspace Git history", () => { await git(remote, "-c", "user.name=Test", "-c", "user.email=test@paperclip.dev", "commit", "--allow-empty", "-m", "remote only"); } if (change === "remote_commit_corrupt_seed" || change === "remote_commit_missing_seed") { - const cache = path.join(process.env.PAPERCLIP_HOME!, "instances", "test", "native-workspace-sync", "generations", "company", "workspace"); + const cache = path.join(process.env.PAPERCLIP_HOME!, "instances", "test", "native-workspace-sync", "generations", "company", "shared-v1"); const [generation] = (await readdir(cache)).filter(name => /^[a-f0-9]{64}$/.test(name)); expect(generation).toBeDefined(); const archive = path.join(cache, generation!, "workspace.tar"); @@ -135,7 +139,7 @@ describe("native warm workspace Git history", () => { if (change === "remote_commit_corrupt_seed") await writeFile(archive, "invalid cached archive"); } if (cacheRefused) { - await rm(path.join(process.env.PAPERCLIP_HOME!, "instances", "test", "native-workspace-sync", "generations", "company", "workspace"), { recursive: true, force: true }); + await rm(path.join(process.env.PAPERCLIP_HOME!, "instances", "test", "native-workspace-sync", "generations", "company", "shared-v1"), { recursive: true, force: true }); vi.spyOn(workspaceSeedCache, "publishWorkspaceSeedGeneration").mockResolvedValue(false); } const hostRepo = change === "plain_nested_commit" ? path.join(host, nestedPath) : host; @@ -160,17 +164,19 @@ describe("native warm workspace Git history", () => { }, }; const reuseEvents: import("@paperclipai/adapter-utils").AdapterRuntimeEvent[] = []; - const second = (await prepareNativeWorkspaceSync({ ...input, runId: "second", onEvent: async event => { reuseEvents.push(event); }, target: { - ...target, runner: secondRunner, sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" }, + const second = (await prepareNativeWorkspaceSync({ ...input, runId: "second", workspaceId: change === "new_workspace_branch" ? "another-workspace" : input.workspaceId, onEvent: async event => { reuseEvents.push(event); }, target: { + ...target, remoteCwd: remote, runner: secondRunner, sandboxLeaseAcquisition: { outcome: change === "new_workspace_branch" ? "created" : "resumed", providerLeaseId: "sandbox" }, } }))!; try { if (change === "remote_commit") expect(packClone).not.toHaveBeenCalled(); + if (change === "new_workspace_branch") expect(packClone).not.toHaveBeenCalled(); if (cacheRefused) expect(packClone).toHaveBeenCalledTimes(invalidRunSeed ? 2 : 1); if (invalidRunSeed) expect(invalidatedRunSeed).toBe(true); if (change === "remote_commit_corrupt_seed" || change === "remote_commit_missing_seed") expect(packClone).toHaveBeenCalled(); expect(second.mode).toBe(change === "unchanged" ? "adopt_remote" : "host_current"); expect(summarizeStartupReuse(reuseEvents).layers.workspace).toEqual(change === "unchanged" ? { state: "reused", reason: "verified_reuse" } + : change === "new_workspace_branch" ? { state: "fresh", reason: "created" } : { state: "fresh", reason: "workspace_changed" }); expect(await readFile(path.join(remoteRepo, "file.txt"), "utf8")).toBe(expectedContents); if (change === "host_bytes_cache_refused") expect(await readFile(path.join(remote, "extra.txt"), "utf8")).toBe("new host file\n"); diff --git a/server/src/services/native-runtime/native-workspace-sync.ts b/server/src/services/native-runtime/native-workspace-sync.ts index 7e56e3500e3..4b5080b51cf 100644 --- a/server/src/services/native-runtime/native-workspace-sync.ts +++ b/server/src/services/native-runtime/native-workspace-sync.ts @@ -837,6 +837,7 @@ async function prepareRuntime(input: { onWorkspaceCheckpoint: input.onCheckpoint, workspaceSeedCacheDirectory: input.seedCacheDirectory, workspaceSeedCacheCompanyDirectory: input.seedCacheCompanyDirectory, + workspaceSeedCacheScope: "company", runtimeSpan: input.onPhase, })); } @@ -1006,7 +1007,7 @@ export async function prepareNativeWorkspaceSync(input: { hasPriorStamp: priorStamp !== null, }); const seedCacheCompanyDirectory = path.join(resolvePaperclipInstanceRoot(), STATE_ROOT_NAME, "generations", requireSafeSegment(input.companyId, "company_id")); - const seedCacheDirectory = path.join(seedCacheCompanyDirectory, requireSafeSegment(input.workspaceId, "workspace_id")); + const seedCacheDirectory = path.join(seedCacheCompanyDirectory, "shared-v1"); runtime = await prepareRuntime({ runId: input.runId, target,