mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
fix(workspaces): preserve ignored files in archived Git imports (#19402)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspace migration must preserve archived files and saved recovery data. > - An archived Git workspace can contain ignored recovery receipts and bundles in its encrypted restore point. > - The importer reconstructs a private repository from a verified source before it applies that restore point. > - Reconstruction omitted the source repository's local ignore rules. The restored files then changed classification, and the exact restore-point check failed. > - This pull request copies the verified local ignore rules into the private repository before restoration. > - The source stays unchanged, and the original restore-point integrity checks still apply. ## Linked Issues or Issue Description **What happened?** An archived workspace with saved files under an ignored recovery directory failed migration with `restored_contents_mismatch`. The task commit and staged index matched. The ignored files were restored, but the private repository treated them as untracked work because its `info/exclude` rules were missing. **Expected behavior** Reconstruction from an existing verified source must preserve its local ignore policy. Saved ignored files must survive migration and a cold object-store restore with their original classification. Import must not follow a symlink to an unrelated ignore file or edit the source repository. **Steps to reproduce** 1. Exclude a recovery directory in the managed source repository's `info/exclude` file. 2. Save staged edits, unstaged edits, an ignored recovery receipt, and an ignored binary bundle in an encrypted restore point. 3. Archive and remove the task worktree. 4. Import the archived workspace. Before this change, restore-point validation fails because the reconstructed repository lacks the local exclusion rule. **Paperclip version or commit** Reproduced on master at `879140cbbd`. **Deployment mode** Cloud workspace migration with a retained verified Git source, PostgreSQL, and encrypted durable object storage. The tests use real disposable PostgreSQL, Git repositories, and encrypted filesystem-backed object storage. Related migration work: #18826, #19138, and #19396. No duplicate open fix was found. ## What Changed - Copy the verified source's `info/exclude` file into the private reconstructed repository before applying its restore point. - Bound the read to one MiB. Reject symlinked, non-regular, hard-linked, or changing source files. - Keep the original source bytes unchanged. - Add regressions that check staged and unstaged edits, ignored recovery files, cold restore, and rejection of a symlinked ignore file. - Document the reconstruction behavior. ## Verification - Before the repair, the saved-ignored-files regression fails with `restored_contents_mismatch`. - After the repair, both focused cases pass. - At rebased head `81a0b1c041`, all 206 tests across migration, snapshot storage, archive, and migration concurrency suites pass. The merged interrupted-upload and nested-source approval regressions remain intact. - Server package typecheck passes. - Full repository typecheck and build pass at the rebased head. The full local test suite is running. The prior local full-test run was interrupted for rebase, and is not a passing result. - Initial CI hit a 15-second timeout in unchanged `activity-routes.test.ts`; the exact case passed in a focused local run. The browser shard lost its self-hosted runner, as confirmed by its check annotation; GitHub did not retain its full log. Those failures remain recorded. An unchanged retry was superseded by the required conflict rebase. At exact head `81a0b1c041`, all 53 CI checks and both required aggregate gates pass; Greptile is 5/5. The full local suite remains in progress, so no local full-suite PASS is claimed. A second automated review noted the existing nested-checkout counterpart. This PR fixes root reconstruction. The nested linked-checkout repair and its saved-ignored-files/cold-restore regression are implemented in #19409, which is required before the migration rollout uses nested imports. This PR leaves that sibling recovery path unchanged. The two identical scope warnings are addressed through that linked repair. ## Risks The copy applies only when an archived Git restore-point source is still present and verified. It changes only the private reconstruction directory. Missing sources retain the existing restoration path. Unsafe or unstable local ignore files block import and retain the original migration obligation. No database migration or saved format changes are included. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
612da1e8e0
commit
66d3fa97a9
3 files changed
+66
-1
No files matched your search
@@ -1939,6 +1939,12 @@ project with the same repository URL; publication locks and rechecks those sourc
|
||||
rows. Imports from mutable local folders still recapture and compare the
|
||||
current files; a saved upload does not permit later local edits to be discarded.
|
||||
|
||||
When the verified Git source still exists, reconstruction also copies its local
|
||||
`info/exclude` rules into the private repository before applying a saved restore
|
||||
point. This keeps saved ignored files, including recovery receipts and bundles,
|
||||
in their original classification. The source ignore file stays unchanged.
|
||||
Symlinked, oversized, or changing source ignore files block reconstruction.
|
||||
|
||||
Shared physical folders require Cloud's offline enrollment maintenance process.
|
||||
A task-level hold cannot fence another workspace record pointing at the same
|
||||
folder. Maintenance verifies that legacy deployments have stopped and that no
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import { constants } from "node:fs";
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
@@ -180,6 +181,38 @@ export async function lockArchivedGitMigrationSources(db: Pick<Db, "select">, wo
|
||||
}
|
||||
}
|
||||
|
||||
async function copyArchivedGitExcludes(source: string, cwd: string) {
|
||||
const common = await fs.realpath(path.resolve(source, await git(source, ["rev-parse", "--path-format=absolute", "--git-common-dir"])));
|
||||
const info = path.join(common, "info"), file = path.join(info, "exclude");
|
||||
const directory = await fs.lstat(info).catch((error: NodeJS.ErrnoException) => {
|
||||
if (error.code === "ENOENT") return null;
|
||||
throw error;
|
||||
});
|
||||
if (!directory) return;
|
||||
if (!directory.isDirectory() || directory.isSymbolicLink()) throw unavailable();
|
||||
const handle = await fs.open(file, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK)
|
||||
.catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return null; throw unavailable(); });
|
||||
if (!handle) return;
|
||||
try {
|
||||
const before = await handle.stat({ bigint: true });
|
||||
if (!before.isFile() || before.nlink !== 1n || before.size > 1024n * 1024n) throw unavailable();
|
||||
const bytes = Buffer.alloc(Number(before.size) + 1); let length = 0;
|
||||
while (length < bytes.length) {
|
||||
const read = await handle.read(bytes, length, bytes.length - length, length);
|
||||
if (!read.bytesRead) break;
|
||||
length += read.bytesRead;
|
||||
}
|
||||
const after = await fs.lstat(file, { bigint: true });
|
||||
if (!after.isFile() || after.isSymbolicLink() || BigInt(length) !== before.size
|
||||
|| before.dev !== after.dev || before.ino !== after.ino || before.size !== after.size || before.mtimeNs !== after.mtimeNs
|
||||
|| before.mode !== after.mode || before.ctimeNs !== after.ctimeNs || before.nlink !== after.nlink
|
||||
|| await fs.realpath(info) !== info) throw unavailable();
|
||||
const target = path.join(cwd, ".git", "info", "exclude");
|
||||
await fs.writeFile(target, bytes.subarray(0, length));
|
||||
await fs.chmod(target, Number(before.mode & 0o777n));
|
||||
} finally { await handle.close(); }
|
||||
}
|
||||
|
||||
export async function materializeArchivedGitMigrationSource(input: {
|
||||
db: Db; storage: StorageService; workspace: Workspace; source: ArchivedGitMigrationSource; scratch: string;
|
||||
}) {
|
||||
@@ -199,6 +232,7 @@ export async function materializeArchivedGitMigrationSource(input: {
|
||||
: await verifiedReceiptObjectSource(workspace, source.sourcePath);
|
||||
if (source.retainedBranch && (await git(verified, ["rev-parse", "--verify", `refs/heads/${source.retainedBranch}^{commit}`])) !== source.commit) throw unavailable();
|
||||
await fs.mkdir(cwd, { mode: 0o700 }); await git(cwd, ["init", "--quiet"]);
|
||||
if (source.recipe?.restorePoint) await copyArchivedGitExcludes(verified, cwd);
|
||||
// Fetch objects, never the current working tree of a shared project root.
|
||||
const presentCommit = await git(verified, ["cat-file", "-e", `${source.commit}^{commit}`]).then(() => true, () => false);
|
||||
if (presentCommit) {
|
||||
|
||||
@@ -660,12 +660,20 @@ describe.skipIf(!support.supported)("remote workspace migration", () => {
|
||||
const address = server.address(); if (!address || typeof address === "string") throw new Error("test server not bound");
|
||||
return { bare, repoUrl: `http://127.0.0.1:${address.port}/repository.git` };
|
||||
}
|
||||
it.each(["present", "pruned", "retry", "missing-base", "missing-base-retry", "missing-base-uploaded-retry", "missing-base-changed-retry", "nested", "nested-retry", "nested-source-changed-retry", "unapproved-nested", "missing", "expired", "corrupt"])("preserves archived Git edits or refuses an unavailable restore point (%s)", async state => {
|
||||
it.each(["present", "pruned", "retry", "source-excludes", "source-excludes-symlink", "missing-base", "missing-base-retry", "missing-base-uploaded-retry", "missing-base-changed-retry", "nested", "nested-retry", "nested-source-changed-retry", "unapproved-nested", "missing", "expired", "corrupt"])("preserves archived Git edits or refuses an unavailable restore point (%s)", async state => {
|
||||
const remote = state.startsWith("missing-base") ? await serveRepository() : null;
|
||||
const f = await gitArchiveFixture(remote?.repoUrl);
|
||||
if (remote) { await git(remote.bare, "fetch", f.base, "main:refs/heads/main"); await git(remote.bare, "update-server-info"); }
|
||||
await fs.writeFile(path.join(f.cwd, "saved.txt"), "staged edit\n"); await git(f.cwd, "add", "saved.txt");
|
||||
await fs.writeFile(path.join(f.cwd, "saved.txt"), "unstaged edit\n");
|
||||
if (state.startsWith("source-excludes")) {
|
||||
await fs.appendFile(path.join(f.base, ".git", "info", "exclude"), "\n/.paperclip/recovered-runs/\n");
|
||||
const recovered = path.join(f.cwd, ".paperclip", "recovered-runs", "saved");
|
||||
await fs.mkdir(recovered, { recursive: true });
|
||||
await fs.writeFile(path.join(recovered, "repair-receipt.json"), '{"preserved":true}\n');
|
||||
await fs.writeFile(path.join(recovered, "source-delta.bundle"), Buffer.from([0, 255, 17, 128]));
|
||||
}
|
||||
const sourceExcludes = state.startsWith("source-excludes") ? await fs.readFile(path.join(f.base, ".git", "info", "exclude")) : null;
|
||||
const captured = (await captureRestorePoint({ cwd: f.cwd, upstreamCommits: [f.upstream] }))!;
|
||||
const id = randomUUID(), generation = 1, sealed = await sealRestorePoint(captured.payload,
|
||||
{ companyId: f.companyId, workspaceId: f.workspaceId, restorePointId: id, generation });
|
||||
@@ -697,6 +705,15 @@ describe.skipIf(!support.supported)("remote workspace migration", () => {
|
||||
gitBranchOwnershipVersion: 1, verifiedAt: new Date().toISOString(), restorePoint, contents },
|
||||
} }).where(eq(executionWorkspaces.id, f.workspaceId));
|
||||
await git(f.base, "worktree", "remove", "--force", f.cwd); await git(f.base, "branch", "-D", f.branchName);
|
||||
if (state === "source-excludes-symlink") {
|
||||
const outside = path.join(root, "private-ignore.txt"); await fs.writeFile(outside, "/outside-private-pattern/\n");
|
||||
await fs.rm(path.join(f.base, ".git", "info", "exclude"));
|
||||
await fs.symlink(outside, path.join(f.base, ".git", "info", "exclude"));
|
||||
await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("remote_workspace_migration_archived_git_source_unavailable");
|
||||
expect(readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)).toBeNull();
|
||||
expect(await fs.readFile(outside, "utf8")).toBe("/outside-private-pattern/\n");
|
||||
return;
|
||||
}
|
||||
if (state.startsWith("missing-base")) await fs.rm(f.base, { recursive: true });
|
||||
if (state === "unapproved-nested") {
|
||||
await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow();
|
||||
@@ -784,6 +801,7 @@ describe.skipIf(!support.supported)("remote workspace migration", () => {
|
||||
expect(operations[0]).toMatchObject({ id: interrupted.id, status: "succeeded" });
|
||||
}
|
||||
const authority = readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)!;
|
||||
if (sourceExcludes) expect(await fs.readFile(path.join(f.base, ".git", "info", "exclude"))).toEqual(sourceExcludes);
|
||||
await fs.rm(f.base, { recursive: true, force: true }); await fs.rm(path.join(root, "nested-source"), { recursive: true, force: true }); await storage.deleteObject(f.companyId, objectKey);
|
||||
const ref = await getWorkspaceSnapshot(db, { ...f, snapshotId: authority.snapshotId! });
|
||||
const archivePath = path.join(root, "retained-git.pcws"), targetDir = path.join(root, "restored-retained-git");
|
||||
@@ -797,6 +815,13 @@ describe.skipIf(!support.supported)("remote workspace migration", () => {
|
||||
}
|
||||
expect(await git(targetDir, "show", ":saved.txt")).toBe("staged edit");
|
||||
expect(await fs.readFile(path.join(targetDir, "saved.txt"), "utf8")).toBe("unstaged edit\n");
|
||||
if (state === "source-excludes") {
|
||||
const recovered = path.join(targetDir, ".paperclip", "recovered-runs", "saved");
|
||||
expect(await fs.readFile(path.join(recovered, "repair-receipt.json"), "utf8")).toBe('{"preserved":true}\n');
|
||||
expect(await fs.readFile(path.join(recovered, "source-delta.bundle"))).toEqual(Buffer.from([0, 255, 17, 128]));
|
||||
expect(await git(targetDir, "check-ignore", ".paperclip/recovered-runs/saved/repair-receipt.json"))
|
||||
.toBe(".paperclip/recovered-runs/saved/repair-receipt.json");
|
||||
}
|
||||
});
|
||||
async function retainDirectory(f: Awaited<ReturnType<typeof seed>>) {
|
||||
const captured = await captureTaskDirectorySnapshot(f.cwd), id = randomUUID(), generation = 1;
|
||||
|
||||
Reference in new issue
Block a user