Preserve fully covered historical workspace sources (#19872)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Workspace migration must preserve local files while retaining
unfinished recovery obligations.
> - A shared Git source needs proof for its companion workspaces before
import can publish.
> - Recovery backfill can add a historical-source marker to an exhausted
prepared-export hold.
> - The hold owner rejected that marker even when every recorded
allocation already belonged to its proven prepared source.
> - This pull request accepts only that fully validated form through the
existing source owners and publication fences.
> - Unrelated work can continue while import saves the local files and
keeps the remote recovery hold intact.

## Linked Issues or Issue Description

Refs #19830. No separate public issue was found. A search for the marker
found no open duplicate PR.

**What happened?**

A companion workspace with an exhausted prepared-export hold blocked the
whole shared Git cohort after canonical recovery backfill added its
historical-source marker. The prepared and historical source readers
both accepted the same allocation.

**Expected behavior**

Local file preservation can proceed when all historical references are
already covered by the proven prepared source. The recovery hold and
remote obligations remain unchanged. Unknown or uncovered sources stay
blocked.

**Steps to reproduce**

1. Create a shared Git repository with a selected leaf and a companion
workspace.
2. Give the companion a settled failed export, accepted result,
exhausted recovery and blocked task.
3. Run canonical historical recovery backfill for its retained
allocation.
4. Import the selected leaf through the normal online migration owner.

**Paperclip version or commit**

Reproduced on `4185e308659abf0887930e20049ca6e7cbfc0e81`.

**Deployment mode**

A source installation with legacy local Git workspaces and object
storage.

## What Changed

- Validate the exact historical marker schema, run and unique lease
references against the already proven prepared source and declared
allocations.
- Reuse the canonical historical-source inspector and compare the same
provider allocation and remote working directory.
- Reject unknown fields, malformed markers and foreign, duplicate or
uncovered references.
- Keep the full marker in the existing hold fingerprint and retain the
existing source, task, lifecycle, finalizer and publication checks.
- Add real PostgreSQL/Git tests through shared-cohort inspection and the
encrypted import/restore owners.

## Verification

- The unchanged owner failed the actual selected-leaf importer
regression with `online_workspace_source_scope_unproven`. The canonical
prepared/historical reader controls passed before that refusal.
- Production head `3e678e10fa` passed 41 focused PostgreSQL/Git cases
across three files in 18.93 seconds after rebasing onto the new
finalizer schema migration. This includes 14 new cases; 80 unrelated
cases were filtered out. The final test-only review update passed both
additional multi-allocation controls plus the original selected-leaf
cold restore (3/3, 11.29 seconds); TypeScript passed again.
- The positive uses the actual historical backfill producer. It
cold-restores an encrypted selected-leaf snapshot with the original Git
tree unavailable and keeps the peer's task, finalizer, result,
allocation and recovery action unchanged.
- The tests retain an unrelated active object-backed run, prove the peer
finalizer lock is held, refuse bad bindings and reject late marker
changes without discarding the uploaded snapshot.
- Server TypeScript, module-boundary checks and `git diff --check`
passed. The canonical PostgreSQL wrapper exited and cleaned up.
- Final-head CI passed: 56 successful checks and four intentional skips,
with no failures or pending jobs
([run](https://github.com/paperclipai/paperclip/actions/runs/42397090541)).
Greptile reviewed `0cbf0a0255` at 5/5 with no open findings; GitHub
reports CLEAN/MERGEABLE.

## Risks

- The allowance applies only to the existing exhausted prepared-export
hold. Other recovery forms and unknown or uncovered references remain
blockers.
- A marked hold adds database reads proportional to its already declared
allocations during maintenance inspection. It adds no provider calls,
payload download or recurring background work.
- Import saves the current local files. It does not claim that unsaved
remote files were recovered, that a provider stopped or that execution
may resume. Existing holds and warnings remain authoritative.
- Local tests use the real encryption and restore owners with a local
object backend. Hosted backfill and normal-delivery qualification remain
separate.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, repository tools and code
execution. The exact serving snapshot and context-window limit are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-07 11:53:00 -06:00
1 parent 51c352ec2e
commit 4d5265fe7e
3 files changed
+354 -4

No files matched your search

+14
View File
@@ -979,6 +979,20 @@ recovery hold. The existing migration recipe routes later file recovery away
from the retired local destination. This neither declares preexecution nor
proves remote export, provider termination or permission to resume execution.
An exhausted prepared-export hold can also retain the canonical historical-source
marker added by recovery backfill. Its schema and run must match, and every unique
recorded lease must already belong to the proven prepared source or its declared
recovery allocations. The existing historical-source inspector verifies those
same allocation bindings. Unknown fields, malformed markers and foreign, duplicate
or uncovered lease references still block import.
The full historical marker stays in the action fingerprint. Existing source,
task, lifecycle and finalizer locks and publication reinspection remain required.
This permits preservation of the current local files only. It neither clears the
hold nor claims that remote files were copied or a provider stopped. Marked holds
add bounded database reads during maintenance inspection; they add no provider
calls or recurring background work.
The workspace fingerprint pins the recorded source. An arriving reservation can
reach a legacy workspace before refusing it, so publication must follow this
order too. The lifecycle advisory lock alone does not fence runtime operations.
@@ -0,0 +1,310 @@
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { createHash, randomUUID } from "node:crypto";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { sql, eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { activityLog, agents, companies, completionContracts, createDb, environmentLeases, environments, executionWorkspaceSnapshots, executionWorkspaces, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, projects, projectWorkspaces, workspaceOperations, workspaceRuntimeServices, type Db } from "@paperclipai/db";
import { readRemoteWorkspaceAuthority, readRemoteWorkspaceRunReference } from "@paperclipai/shared";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { resolveManagedProjectWorkspaceDir } from "../home-paths.js";
import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js";
import { createStorageService } from "../storage/service.js";
import type { StorageService } from "../storage/types.js";
import { migrateRemoteWorkspace } from "./remote-workspace-migration.js";
import { getWorkspaceSnapshot, loadWorkspaceSnapshotArchive } from "./remote-workspace-snapshot-storage.js";
import { restoreRemoteWorkspaceArchive } from "./remote-workspace-snapshot-archive.js";
import { tryLockNativeWorkspaceFinalization } from "./native-runtime/native-workspace-finalization-ownership.js";
import { recordNativeFinalizationFailure } from "./native-runtime/native-run-finalizer.js";
import { inspectOnlineWorkspaceSource } from "./online-workspace-source.js";
import { inspectOnlineLegacyWorkspaceRecoverySource } from "./online-legacy-workspace-recovery.js";
import { inspectWorkspaceRecoverySource } from "./workspace-recovery-source.js";
import { createSystemRecovery } from "./system-recovery-state.js";
import { nativeSha256 } from "./native-runtime/canonical.js";
import { adoptHistoricalWorkspaceRecoveries } from "./workspace-recovery-backfill.js";
const support = await getEmbeddedPostgresTestSupport();
describe.skipIf(!support.supported)("covered historical prepared-source preservation", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>, home: string, storage: StorageService;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("covered-historical-source-");
db = createDb(database.connectionString, { maxConnections: 1 });
await db.execute(sql`set client_min_messages = warning`);
}, 90_000);
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await database?.cleanup(); });
beforeEach(async () => {
home = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "covered-historical-source-")));
vi.stubEnv("PAPERCLIP_HOME", home);
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "61".repeat(32));
storage = createStorageService(createLocalDiskStorageProvider(path.join(home, "objects")));
});
afterEach(async () => {
vi.restoreAllMocks();
await db.delete(activityLog);
await db.delete(workspaceOperations);
await db.delete(executionWorkspaceSnapshots);
await db.delete(workspaceRuntimeServices);
await db.delete(nativeRunFinalizations);
await db.delete(nativeRunResults);
await db.delete(environmentLeases);
await db.delete(environments);
await db.delete(heartbeatRunEvents);
await db.delete(heartbeatRuns);
await db.delete(completionContracts);
await db.delete(issues);
await db.delete(agents);
await db.delete(executionWorkspaces);
await db.delete(projectWorkspaces);
await db.delete(projects);
await db.delete(companies);
vi.unstubAllEnvs();
await fs.rm(home, { recursive: true, force: true });
});
async function fixture() {
const companyId = randomUUID(), projectId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Online source", issuePrefix: `O${companyId.slice(0, 6)}` });
await db.insert(projects).values({ id: projectId, companyId, name: "Shared legacy project" });
const cwd = resolveManagedProjectWorkspaceDir({ companyId, projectId });
await fs.mkdir(cwd, { recursive: true });
await fs.writeFile(path.join(cwd, "saved.txt"), "saved legacy edits\n");
const aliases = [];
for (let n = 0; n < 3; n++) {
const [issue] = await db.insert(issues).values({ companyId, projectId, title: `Legacy task ${n}` }).returning();
const [workspace] = await db.insert(executionWorkspaces).values({ companyId, projectId, sourceIssueId: issue.id,
name: `Alias ${n}`, mode: "shared_workspace", strategyType: "project_primary", providerType: "local_fs", cwd, providerRef: cwd }).returning();
await db.update(issues).set({ executionWorkspaceId: workspace.id }).where(eq(issues.id, issue.id));
aliases.push({ issue, workspace });
}
return { companyId, projectId, cwd, aliases, workspaceId: aliases[0].workspace.id };
}
const execute = promisify(execFile);
const git = async (cwd: string, ...args: string[]) => (await execute("git", ["-c", `core.hooksPath=${os.devNull}`, "-C", cwd, ...args], {
env: { ...process.env, GIT_CONFIG_NOSYSTEM: "1", GIT_CONFIG_GLOBAL: os.devNull, GIT_TERMINAL_PROMPT: "0",
GIT_AUTHOR_NAME: "Source fixture", GIT_AUTHOR_EMAIL: "source@example.test", GIT_COMMITTER_NAME: "Source fixture", GIT_COMMITTER_EMAIL: "source@example.test" },
})).stdout.trim();
async function gitFixture(f = undefined as Awaited<ReturnType<typeof fixture>> | undefined) {
const base = f ?? await fixture();
await git(base.cwd, "init", "-q", "-b", "main");
await git(base.cwd, "add", "."); await git(base.cwd, "commit", "-qm", "Saved legacy work");
const repoUrl = "https://example.invalid/saved-work.git";
await git(base.cwd, "remote", "add", "origin", repoUrl);
for (const alias of base.aliases) await db.update(executionWorkspaces).set({ repoUrl, branchName: "main" })
.where(eq(executionWorkspaces.id, alias.workspace.id));
const [issue] = await db.insert(issues).values({ companyId: base.companyId, projectId: base.projectId, title: "Shared Git child" }).returning();
const cwd = path.join(base.cwd, ".paperclip", "worktrees", "child");
await git(base.cwd, "worktree", "add", "-qb", "child", cwd);
const [workspace] = await db.insert(executionWorkspaces).values({ companyId: base.companyId, projectId: base.projectId,
sourceIssueId: issue.id, name: "Shared Git child", mode: "isolated_workspace", strategyType: "git_worktree",
providerType: "git_worktree", cwd, providerRef: cwd, repoUrl, branchName: "child", baseRef: "main" }).returning();
await db.update(issues).set({ executionWorkspaceId: workspace.id }).where(eq(issues.id, issue.id));
return { ...base, child: workspace, repoUrl };
}
async function stored(workspaceId: string) {
return (await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)))[0];
}
async function recoveryFixture(descendant = false) {
const f = await fixture(), runId = randomUUID();
const issue = descendant ? (await db.insert(issues).values({ companyId: f.companyId, projectId: f.projectId,
parentId: f.aliases[0].issue.id, title: "Descendant prepared recovery", executionWorkspaceId: f.workspaceId }).returning())[0]
: f.aliases[0].issue;
const [agent] = await db.insert(agents).values({ companyId: f.companyId, name: "Terminal native worker", adapterType: "paperclip_runner" }).returning();
const reference = { schema: "paperclip.native-workspace-sync/v3", state: "prepared", workspaceId: f.workspaceId,
leaseId: randomUUID(), providerLeaseId: randomUUID(), remoteCwd: "/workspaces/retained",
descriptorSha256: "12".repeat(32), baselineSha256: "34".repeat(32), resourceDisposition: "destroy" };
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: agent.id,
nativeIssueId: issue.id, status: "failed", runtimeMode: "native", runnerProfileJson: { nativeWorkspaceSync: reference } }).returning();
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: issue.id,
runId, phase: "observed" });
const [environment] = await db.insert(environments).values({ name: runId, driver: "daytona" }).returning();
const [lease] = await db.insert(environmentLeases).values({ id: reference.leaseId, companyId: f.companyId,
issueId: issue.id, heartbeatRunId: runId, executionWorkspaceId: f.workspaceId, environmentId: environment.id,
provider: "daytona", providerLeaseId: reference.providerLeaseId, status: "retained", leasePolicy: "retain_on_failure",
metadata: { remoteCwd: reference.remoteCwd } }).returning();
await recordNativeFinalizationFailure({ db, runId, error: new Error("native_workspace_sync_out_unrecoverable"),
failureScope: "workspace", permanent: true });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
const [owner] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
const [warning] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issue.id));
expect(warning).toMatchObject({ kind: "active_run_watchdog", cause: "native_workspace_sync_out_unrecoverable",
status: "active", evidence: { runId } });
return { ...f, run, owner, lease, warning };
}
async function remoteWriter(companyId: string, agentId: string) {
const workspaceId = randomUUID(), runId = randomUUID(), leaseId = randomUUID(), providerLeaseId = randomUUID();
const [issue] = await db.insert(issues).values({ companyId, title: "Unrelated object-backed task" }).returning();
const [environment] = await db.insert(environments).values({ name: runId, driver: "daytona" }).returning();
const remoteCwd = `/workspaces/${workspaceId}`;
const authority = { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: null, source: { kind: "empty" },
location: { environmentId: environment.id, leaseId, providerLeaseId, remoteCwd }, writerRunId: runId, dirty: true };
const [workspace] = await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issue.id,
name: "Unrelated remote writer", mode: "isolated_workspace", strategyType: "project_primary", providerType: "cloud_sandbox",
cwd: remoteCwd, metadata: { remoteWorkspace: authority } }).returning();
const reference = { schema: "paperclip.remote-workspace-run.v1", workspaceId, leaseId, providerLeaseId,
remoteCwd, epoch: 1, state: "prepared", snapshotId: null };
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId: issue.id,
nativeIssueId: issue.id, status: "running", runtimeMode: "native", contextSnapshot: { executionWorkspaceId: workspaceId },
runnerProfileJson: { remoteWorkspace: reference } }).returning();
await db.insert(environmentLeases).values({ id: leaseId, companyId, heartbeatRunId: runId, issueId: issue.id,
executionWorkspaceId: workspaceId, environmentId: environment.id, provider: "daytona", providerLeaseId,
status: "active", metadata: { remoteCwd } });
expect(readRemoteWorkspaceAuthority(workspace.metadata)).toEqual(authority);
expect(readRemoteWorkspaceRunReference(run.runnerProfileJson?.remoteWorkspace)).toEqual(reference);
return run;
}
async function coveredFixture(additionalSource = false) {
const f = await recoveryFixture(), cause = "native_workspace_sync_out_retry_exhausted", contractId = randomUUID(), resultId = randomUUID();
const reference = { ...f.run.runnerProfileJson!.nativeWorkspaceSync as Record<string, unknown>, resourceDisposition: "keep_running" };
await db.insert(completionContracts).values({ id: contractId, companyId: f.companyId, issueId: f.run.nativeIssueId!, revision: 1,
schemaVersion: "v1", policyVersion: "v1", risk: "low", completionAuthority: "agent", incompleteCriteriaPolicy: "block",
contractJson: {}, canonicalSha256: "45".repeat(32), createdByActorType: "system", createdByActorId: "fixture" });
await db.update(heartbeatRuns).set({ completionContractId: contractId, completionContractSha256: "45".repeat(32),
runnerProfileJson: { ...f.run.runnerProfileJson, nativeWorkspaceSync: reference },
resultJson: { retained: "original failed export" } }).where(eq(heartbeatRuns.id, f.run.id));
await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.run.nativeIssueId!, runId: f.run.id,
completionContractId: contractId, serverFingerprint: "retained accepted result", schemaStatus: "accepted",
resultJson: { terminal: { runTerminalState: "succeeded" }, retained: "original result" }, canonicalSha256: "67".repeat(32) });
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", resultId, failureCode: cause, nextAttemptAt: null,
recoveryHistory: [{ retained: "remote export remains unresolved" }] }).where(eq(nativeRunFinalizations.runId, f.run.id));
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date("2026-01-01T00:00:00Z") })
.where(eq(environmentLeases.id, f.lease.id));
await db.update(issues).set({ status: "blocked", statusVersion: 6, executionRunId: null, checkoutRunId: null })
.where(eq(issues.id, f.run.nativeIssueId!));
await db.update(issueRecoveryActions).set({ cause, ownerType: "board", ownerAgentId: null, returnOwnerAgentId: f.run.agentId,
wakePolicy: null, fingerprint: nativeSha256({ runId: f.run.id, failureCode: cause }), evidence: {
runId: f.run.id, coordinatorAttempt: 3, workspaceFinalizeAttempt: 3,
systemRecovery: { ...createSystemRecovery(f.run.id), state: "exhausted" },
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: f.run.id, status: "blocked", stage: "verifying_stop", attempt: 4, requestIds: [] },
previousWorkspaceRecovery: { retained: "original recovery history" },
} }).where(eq(issueRecoveryActions.id, f.warning.id));
const [additionalLease] = additionalSource ? await db.insert(environmentLeases).values({ companyId: f.companyId,
issueId: f.run.nativeIssueId!, heartbeatRunId: f.run.id, executionWorkspaceId: f.workspaceId,
environmentId: f.lease.environmentId, provider: "daytona", providerLeaseId: randomUUID(),
status: "released", releasedAt: new Date("2026-01-01T00:00:00Z"), leasePolicy: "retain_on_failure",
metadata: { remoteCwd: "/workspaces/additional-retained" } }).returning() : [];
if (additionalLease) await db.update(heartbeatRuns).set({ resultJson: { retained: "original failed export",
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1",
leaseIds: [f.lease.id, additionalLease.id] } } }).where(eq(heartbeatRuns.id, f.run.id));
await adoptHistoricalWorkspaceRecoveries(db);
const [warning] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.warning.id));
expect(warning.evidence.historicalWorkspaceSource).toEqual({ schema: "paperclip.historical-workspace-source.v1",
runId: f.run.id, leaseIds: [f.lease.id, ...(additionalLease ? [additionalLease.id] : [])].sort() });
return { ...f, ...await gitFixture(f), warning, additionalLease };
}
async function retained(f: Awaited<ReturnType<typeof coveredFixture>>) {
return { run: (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)))[0],
owner: (await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.run.id)))[0],
result: await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.run.id)),
lease: (await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0],
task: (await db.select().from(issues).where(eq(issues.id, f.run.nativeIssueId!)))[0],
warning: (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.warning.id)))[0] };
}
it("imports and cold-restores a selected Git leaf while preserving its peer's canonical historical recovery", async () => {
const f = await coveredFixture(), selected = { companyId: f.companyId, workspaceId: f.child.id };
const before = await retained(f), unrelated = await remoteWriter(f.companyId, f.run.agentId);
expect(await inspectOnlineLegacyWorkspaceRecoverySource(db, { workspace: (await stored(f.workspaceId))!, runId: f.run.id })).not.toBeNull();
expect((await inspectWorkspaceRecoverySource(db, { companyId: f.companyId, issueId: f.run.nativeIssueId!, runId: f.run.id, actionId: f.warning.id })).sources.map(s=>s.lease.id)).toEqual([f.lease.id]);
const scope = await inspectOnlineWorkspaceSource(db, f.child), put = storage.putFile.bind(storage);
vi.spyOn(storage, "putFile").mockImplementationOnce(async input => {
await db.transaction(async tx => expect(await tryLockNativeWorkspaceFinalization(tx as unknown as Db, { companyId: f.companyId, runId: f.run.id })).toBe(false));
return put(input);
});
await expect(migrateRemoteWorkspace({ db, storage, ...selected, apply: true })).resolves.toMatchObject({ applied: true });
expect(scope?.workspaceIds).toContain(f.workspaceId);
expect(scope?.companions.find(c=>c.workspaceId===f.workspaceId)?.holds).toHaveLength(1);
expect(await retained(f)).toEqual(before);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, unrelated.id)))[0]).toEqual(unrelated);
const authority = readRemoteWorkspaceAuthority((await stored(f.child.id))!.metadata)!;
const ref = (await getWorkspaceSnapshot(db, { ...selected, snapshotId: authority.snapshotId! }))!;
const cipher = await storage.getObject(f.companyId, ref.objectKey), chunks: Buffer[] = [];
for await (const part of cipher.stream) chunks.push(Buffer.from(part));
expect(Buffer.concat(chunks).subarray(0, 7).toString()).toBe("PCWSE1\n");
const head = await git(f.child.cwd!, "rev-parse", "HEAD"), parked = path.join(home, "parked-original");
await fs.rename(f.cwd, parked);
try {
const archivePath = path.join(home, "selected.pcws"), targetDir = path.join(home, "restored");
await loadWorkspaceSnapshotArchive({ storage, reference: ref, targetPath: archivePath });
await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: ref.manifestDigest });
expect(await fs.readFile(path.join(targetDir, "saved.txt"), "utf8")).toBe("saved legacy edits\n");
expect(await git(targetDir, "rev-parse", "HEAD")).toBe(head);
} finally { await fs.rename(parked, f.cwd); }
expect(await fs.readFile(path.join(f.cwd, "saved.txt"), "utf8")).toBe("saved legacy edits\n");
});
it.each([false, true])("preserves both declared historical allocations and refuses an uncovered third (%s)", async uncovered => {
const f = await coveredFixture(true), additional = f.additionalLease!;
const [additionalBefore] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, additional.id));
const source = await inspectOnlineLegacyWorkspaceRecoverySource(db, { workspace: (await stored(f.workspaceId))!, runId: f.run.id });
expect(source?.recoveryAllocations?.map(item => item.leaseId).sort()).toEqual([f.lease.id, additional.id].sort());
expect((await inspectWorkspaceRecoverySource(db, { companyId: f.companyId, issueId: f.run.nativeIssueId!,
runId: f.run.id, actionId: f.warning.id })).sources.map(item => item.lease.id).sort()).toEqual([f.lease.id, additional.id].sort());
if (uncovered) await db.update(issueRecoveryActions).set({ evidence: { ...f.warning.evidence,
historicalWorkspaceSource: { ...(f.warning.evidence.historicalWorkspaceSource as Record<string, unknown>),
leaseIds: [f.lease.id, additional.id, randomUUID()] } } }).where(eq(issueRecoveryActions.id, f.warning.id));
const before = await retained(f), input = { db, storage, companyId: f.companyId, workspaceId: f.child.id, apply: true };
if (uncovered) {
expect(await inspectOnlineWorkspaceSource(db, f.child)).toBeNull();
await expect(migrateRemoteWorkspace(input)).rejects.toThrow();
expect(await db.select().from(workspaceOperations)).toEqual([]);
expect(await db.select().from(executionWorkspaceSnapshots)).toEqual([]);
} else {
const scope = await inspectOnlineWorkspaceSource(db, f.child);
expect(scope?.companions.find(item => item.workspaceId === f.workspaceId)?.holds).toHaveLength(1);
await expect(migrateRemoteWorkspace(input)).resolves.toMatchObject({ applied: true });
expect(readRemoteWorkspaceAuthority((await stored(f.child.id))!.metadata)).not.toBeNull();
}
expect(await retained(f)).toEqual(before);
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, additional.id)))[0]).toEqual(additionalBefore);
expect(await fs.readFile(path.join(f.child.cwd!, "saved.txt"), "utf8")).toBe("saved legacy edits\n");
});
it.each(["schema", "run", "duplicate", "extra", "foreign", "missing", "malformed", "unknown-field", "provider", "cwd", "owner", "task"])("refuses a historical source with invalid %s binding without changing the hold", async mode => {
const f = await coveredFixture(), marker = f.warning.evidence.historicalWorkspaceSource as Record<string, unknown>;
if (mode === "schema") marker.schema = "unrecognized";
if (mode === "run") marker.runId = randomUUID();
if (mode === "duplicate") marker.leaseIds = [f.lease.id, f.lease.id];
if (mode === "extra") marker.leaseIds = [f.lease.id, randomUUID()];
if (mode === "missing") marker.leaseIds = [randomUUID()];
if (mode === "foreign") {
const companyId = randomUUID(); await db.insert(companies).values({ id: companyId, name: "Foreign retained source", issuePrefix: "FOREIGN" });
const [lease] = await db.insert(environmentLeases).values({ companyId, environmentId: f.lease.environmentId,
provider: "daytona", providerLeaseId: randomUUID(), leasePolicy: "retain_on_failure", status: "retained" }).returning();
marker.leaseIds = [lease.id];
}
if (mode === "unknown-field") marker.additionalSource = randomUUID();
await db.update(issueRecoveryActions).set({ evidence: { ...f.warning.evidence,
historicalWorkspaceSource: mode === "malformed" ? [marker] : marker } }).where(eq(issueRecoveryActions.id, f.warning.id));
if (mode === "provider") await db.update(environmentLeases).set({ providerLeaseId: randomUUID() }).where(eq(environmentLeases.id, f.lease.id));
if (mode === "cwd") { const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
await db.update(environmentLeases).set({ metadata: { ...lease.metadata, remoteCwd: "/different" } }).where(eq(environmentLeases.id, f.lease.id)); }
if (mode === "owner") await db.update(nativeRunFinalizations).set({ leaseOwner: "unreleased", leaseExpiresAt: new Date(0) }).where(eq(nativeRunFinalizations.runId, f.run.id));
if (mode === "task") { const run = await remoteWriter(f.companyId, f.run.agentId);
await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, f.run.nativeIssueId!)); }
const before = await retained(f);
expect(await inspectOnlineWorkspaceSource(db, f.child)).toBeNull();
await expect(migrateRemoteWorkspace({ db, storage, companyId: f.companyId, workspaceId: f.child.id, apply: true })).rejects.toThrow();
expect(await db.select().from(workspaceOperations)).toEqual([]); expect(await db.select().from(executionWorkspaceSnapshots)).toEqual([]);
expect(await retained(f)).toEqual(before);
expect(await fs.readFile(path.join(f.child.cwd!, "saved.txt"), "utf8")).toBe("saved legacy edits\n");
});
it("retains the uploaded snapshot but refuses publication if the peer historical marker changes", async () => {
const f = await coveredFixture(), before = await retained(f), put = storage.putFile.bind(storage);
vi.spyOn(storage, "putFile").mockImplementationOnce(async input => {
const result = await put(input);
const marker = f.warning.evidence.historicalWorkspaceSource as Record<string, unknown>;
await db.update(issueRecoveryActions).set({ evidence: { ...f.warning.evidence,
historicalWorkspaceSource: { ...marker, leaseIds: [randomUUID()] } } }).where(eq(issueRecoveryActions.id, f.warning.id));
return result;
});
await expect(migrateRemoteWorkspace({ db, storage, companyId: f.companyId, workspaceId: f.child.id, apply: true })).rejects.toThrow();
expect(readRemoteWorkspaceAuthority((await stored(f.child.id))!.metadata)).toBeNull();
const [snapshot] = await db.select().from(executionWorkspaceSnapshots); expect(snapshot.status).toBe("uploaded");
expect((await storage.headObject(f.companyId, snapshot.objectKey!)).exists).toBe(true);
const after = await retained(f);
expect({ ...after, warning: undefined }).toEqual({ ...before, warning: undefined });
expect(after.warning.evidence.historicalWorkspaceSource).not.toEqual(before.warning.evidence.historicalWorkspaceSource);
expect(await fs.readFile(path.join(f.child.cwd!, "saved.txt"), "utf8")).toBe("saved legacy edits\n");
});
});
@@ -6,9 +6,9 @@ import { inspectFinalizedWorkspaceRecoveryHold, type FinalizedWorkspaceRecoveryH
import { isLegacyWorkspaceTaskRelated, readLegacyWorkspaceRecoveryAllocations, type LegacyWorkspaceRecoverySource } from "./legacy-workspace-recovery-migration.js";
import { tryLockNativeWorkspaceFinalization } from "./native-runtime/native-workspace-finalization-ownership.js";
import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js";
import { workspaceExportCauses } from "./workspace-recovery-source.js";
import { inspectWorkspaceRecoverySource, WorkspaceSourceInspectionLimit, workspaceExportCauses } from "./workspace-recovery-source.js";
import { nativeSha256 } from "./native-runtime/canonical.js";
import { readSystemRecovery } from "./system-recovery-state.js";
import { readSystemRecovery, SystemRecoveryFailure } from "./system-recovery-state.js";
import { readWorkspaceRecovery } from "./workspace-recovery-receipt.js";
type ReadDb = Pick<Db, "select">;
@@ -116,6 +116,31 @@ export type OnlineLegacyWorkspaceRecoveryHold = {
preparedSource?: LegacyWorkspaceRecoverySource;
};
async function coveredHistoricalSource(db: ReadDb, workspace: Workspace, recovery: Recovery, source: LegacyWorkspaceRecoverySource) {
const raw = recovery.evidence.historicalWorkspaceSource;
if (raw == null) return true;
if (typeof raw !== "object" || Array.isArray(raw)) return false;
const historical = raw as Record<string, unknown>, ids = historical.leaseIds;
const covered = new Map([...(source.reference ? [source.reference] : []), ...(source.recoveryAllocations ?? [])]
.map(binding => [binding.leaseId, binding]));
if (historical.schema !== "paperclip.historical-workspace-source.v1" || historical.runId !== source.runId
|| Object.keys(historical).some(key => !["schema", "runId", "leaseIds"].includes(key))
|| !Array.isArray(ids) || !ids.length || ids.length > covered.size || new Set(ids).size !== ids.length
|| ids.some(id => typeof id !== "string" || !covered.has(id))) return false;
try {
const inspected = await inspectWorkspaceRecoverySource(db as Db, { companyId: workspace.companyId,
issueId: source.issueId, runId: source.runId, actionId: recovery.id }, { maximumSources: covered.size });
const verifiedIds = new Set(inspected.sources.map(item => item.lease.id));
return ids.every(id => verifiedIds.has(id)) && inspected.sources.every(item => {
const binding = covered.get(item.lease.id);
return binding && binding.providerLeaseId === item.lease.providerLeaseId && binding.remoteCwd === item.remoteCwd;
});
} catch (error) {
if (error instanceof SystemRecoveryFailure || error instanceof WorkspaceSourceInspectionLimit) return false;
throw error;
}
}
export async function inspectOnlineLegacyWorkspaceRecoveryHold(db: ReadDb, input: {
workspace: Workspace; recovery: Recovery;
}): Promise<OnlineLegacyWorkspaceRecoveryHold | null> {
@@ -175,7 +200,7 @@ export async function inspectOnlineLegacyWorkspaceRecoveryHold(db: ReadDb, input
&& recovery.fingerprint === nativeSha256({ runId: settled.run.id, failureCode: recovery.cause })
&& system?.sourceRunId === settled.run.id && system.state === "exhausted" && system.nextAttemptAt === null
&& workspaceRecovery?.runId === settled.run.id && workspaceRecovery.status === "blocked"
&& !["workspaceRestoreRecovery", "historicalWorkspaceSource", "nativeFailureBlock"].some(key => recovery.evidence?.[key] != null)) {
&& !["workspaceRestoreRecovery", "nativeFailureBlock"].some(key => recovery.evidence?.[key] != null)) {
const [task] = await db.select({ id: issues.id, status: issues.status, statusVersion: issues.statusVersion,
executionRunId: issues.executionRunId, checkoutRunId: issues.checkoutRunId }).from(issues).where(and(
eq(issues.companyId, input.workspace.companyId), eq(issues.id, recovery.sourceIssueId)));
@@ -185,7 +210,8 @@ export async function inspectOnlineLegacyWorkspaceRecoveryHold(db: ReadDb, input
eq(nativeRunResults.completionContractId, settled.run.completionContractId), eq(nativeRunResults.schemaStatus, "accepted")));
if (!task || task.status !== "blocked" || task.executionRunId !== null || task.checkoutRunId !== null || !result) return null;
const source = await inspectOnlineLegacyWorkspaceRecoverySource(db, { workspace: input.workspace, runId: settled.run.id });
if (source?.reference?.state === "prepared" && source.issueId === recovery.sourceIssueId)
if (source?.reference?.state === "prepared" && source.issueId === recovery.sourceIssueId
&& await coveredHistoricalSource(db, input.workspace, recovery, source))
return { fingerprint: fingerprint({ recovery, system, workspaceRecovery, source, task, result }), finalizedHold: null, preparedSource: source };
}
const held = await inspectFinalizedWorkspaceRecoveryHold(db, {