fix(server): recover authenticated empty retention checkpoints (#19417)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Ephemeral server enrollment must preserve existing task files.
> - Archived task directories can depend on encrypted retention
checkpoints.
> - A missing object blocks migration even when its authenticated
contents were exactly empty.
> - Metadata counters alone cannot prove that no saved files existed.
> - This change verifies the exact candidate against the original
encrypted key envelope before normal migration publication.
> - The benefit is lossless migration of this bounded case without
resetting task history or weakening ordinary restore.

## Linked Issues or Issue Description

**What happened?**
Migration stops at an archived empty directory when its retained
encrypted object is missing. The original encrypted key envelope still
authenticates the payload hash, byte length, company, workspace,
checkpoint ID, and generation.

**Expected behavior**
Administrative migration can reconstruct those exact original bytes when
storage confirms absence and every authenticated check passes. It must
keep missing nonempty, expired, corrupt, or ambiguous sources blocked.

**Steps to reproduce**
1. Capture and encrypt an empty task directory.
2. Retain its reference and encrypted key envelope.
3. Remove the old directory and storage object.
4. Run the normal remote workspace migration.

**Paperclip version or commit**
The regression fails before this change at `205ffcb606`. Related
preservation work is in #19396, #19402, and #19409. No duplicate open
fix was found.

**Deployment mode**
Administrative migration to external workspace storage. Tests use real
encryption, real filesystem storage, and disposable PostgreSQL.

## What Changed

- Let the encryption owner verify a candidate against the existing
scoped encrypted key envelope. Normal ciphertext validation keeps its
previous order and checks.
- Share the task-directory encoder with an exact empty payload
constructor.
- Add a default-off loader option for confirmed missing empty
checkpoints. Only the administrative migration caller enables it.
- Keep the original checkpoint reference and object unchanged. Publish
the recovered bytes through the existing snapshot and source-fence
owners.
- Reject forged counters, wrong identities, missing nonempty data,
present corruption, unreadable present objects, unknown absence, and
expiry.

## Verification

- Final affected run: all 238 tests pass across five crypto, snapshot,
retention, and migration suites. Full repository typecheck and build
pass; server typecheck and full build also pass after the cancellation
repair.
- RED then GREEN: exact missing-empty recovery and cancellation of a
stalled HEAD through the real storage-service boundary. Cancellation
retains the caller's original abort reason.
- Disposable PostgreSQL migration publishes a cold-restorable empty
archive. Archived status, task status, task binding, original reference,
and the absent original source stay unchanged. Replay keeps one
canonical snapshot.
- Negative cases reject forged empty counters, an authenticated
directory or file, every foreign identity field, expiry, present
corruption, unreadable present objects, unknown absence, and missing
nonempty data. Ordinary restore keeps its previous failure behavior.
- Initial verification retained one new fixture-path failure after 145
passes, and a typecheck rejected the fixture's widened schema literal.
Correct the fixture to use the returned staging path and its explicit
reference type. The final complete affected run passes.
- The new HEAD request runs only after a failed read with the
administrative option enabled. It uses the existing cancellation
contract and adds no polling or background work.
- Final-head full CI passes; review is 5/5 with no open findings. This
PR does not claim deployed enrollment or a live migration result.

## Risks

Recovery requires the original valid encrypted key envelope and an exact
format match. A format change or unavailable key continues to block
migration. Existing ciphertext hash, size, authentication, and ordinary
restore behavior remain enforced. This change has no schema,
provider-session, API, or UI contract changes.

## Model Used

OpenAI GPT-6 in Codex, with code editing, tool use, and test execution.
The exact runtime model identifier and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
Dotta authored and GitHub committed 2026-11-06 00:50:28 -06:00
1 parent a4bb059f59
commit 92452524f4
7 files changed
+198 -17

No files matched your search

@@ -24,6 +24,16 @@ branch-deletion receipts. PostgreSQL and real-file tests prove cold restoration
of saved bytes, index state, and unpushed commits, including after pruning the
old Git object store. Invalid or unavailable retained data still blocks import.
Administrative migration can recover a missing empty task-directory payload only
when storage confirms that its object is absent. The existing encrypted key
envelope must authenticate the exact empty payload hash and length, company,
workspace, restore-point ID, and generation. Metadata counters alone never
establish emptiness. This path leaves the original reference and missing object
unchanged and publishes the recovered bytes through the normal verified remote
snapshot owner. Ordinary restoration remains unchanged. Expired checkpoints,
nonempty or forged payloads, unreadable objects, and unknown object absence still
block migration.
An archived isolated checkout can be gone while its recorded deleted commit
survives in the company/project Git cache. After validating the exact successful
branch-deletion receipt, the importer can read immutable objects from that
@@ -0,0 +1,117 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { randomUUID } from "node:crypto";
import { afterEach, beforeEach, expect, it, vi } from "vitest";
import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js";
import { createStorageService } from "../storage/service.js";
import { sealRestorePoint, type RestorePointIdentity } from "../services/git-restore-point.js";
import { captureTaskDirectorySnapshot, materializeTaskDirectorySnapshot } from "../services/isolated-task-directory-snapshot.js";
import { loadTaskDirectoryRestorePoint, TASK_DIRECTORY_RESTORE_POINT_SCHEMA, type TaskDirectoryRestorePointRef } from "../services/isolated-task-directory-retention.js";
let root: string;
beforeEach(async () => {
root = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "pc-empty-retention-")));
vi.stubEnv("PAPERCLIP_HOME", path.join(root, "home"));
vi.stubEnv("PAPERCLIP_INSTANCE_ID", "empty-retention-test");
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
});
afterEach(async () => {
vi.restoreAllMocks(); vi.unstubAllEnvs();
await fs.rm(root, { recursive: true, force: true });
});
async function fixture(kind = "empty", scope?: keyof RestorePointIdentity) {
const directory = path.join(root, "source"); await fs.mkdir(directory);
const empty = await captureTaskDirectorySnapshot(directory);
if (kind === "file") await fs.writeFile(path.join(directory, "saved.txt"), "must never disappear\n");
if (kind === "directory") await fs.mkdir(path.join(directory, "saved-directory"));
const snapshot = await captureTaskDirectorySnapshot(directory);
const identity = { companyId: randomUUID(), workspaceId: randomUUID(), restorePointId: randomUUID(), generation: 1 };
const foreign = { ...identity };
if (scope === "generation") foreign.generation = 2;
else if (scope) foreign[scope] = randomUUID();
const sealed = await sealRestorePoint(snapshot.payload, foreign);
const provider = createLocalDiskStorageProvider(path.join(root, "objects"));
const storage = createStorageService(provider);
const objectKey = `${identity.companyId}/git-restore-points/${identity.workspaceId}/1/task-directory/${identity.restorePointId}.pcrp`;
const put = (body: Buffer) => storage.putFile({ companyId: identity.companyId, namespace: "git-restore-points", objectKey,
originalFilename: null, contentType: "application/octet-stream", body });
await put(sealed.ciphertext);
const ref: TaskDirectoryRestorePointRef = { schema: TASK_DIRECTORY_RESTORE_POINT_SCHEMA, id: identity.restorePointId, objectKey, generation: 1,
sha256: sealed.sha256, byteSize: sealed.ciphertext.length, key: sealed.key, digest: snapshot.digest, files: snapshot.files, excluded: snapshot.excluded };
const load = (recoverMissingEmpty = true) => loadTaskDirectoryRestorePoint({ storage, workspace: { companyId: identity.companyId, id: identity.workspaceId }, ref, recoverMissingEmpty });
return { ref, storage, provider, put, load, empty, snapshot, directory, remove: () => storage.deleteObject(identity.companyId, objectKey) };
}
it("recovers only the exact empty payload authenticated by the original scoped key envelope", async () => {
const f = await fixture(); await f.remove(); await fs.rm(f.directory, { recursive: true });
await expect(f.load(false)).rejects.toMatchObject({ reason: "restore_point_unavailable" });
const payload = await f.load(); expect(payload).toEqual(f.snapshot.payload);
const restored = await materializeTaskDirectorySnapshot({ payload, target: path.join(root, "restored"), expectedDigest: f.ref.digest });
expect(await fs.readdir(restored)).toEqual([]);
expect(await f.storage.headObject(f.ref.objectKey.split("/")[0]!, f.ref.objectKey)).toMatchObject({ exists: false });
});
it.each(["file", "directory"])("refuses forged empty counters and digest for authenticated %s contents", async kind => {
const f = await fixture(kind); await f.remove();
f.ref.files = 0; f.ref.excluded = 0; f.ref.digest = f.empty.digest;
await expect(f.load()).rejects.toMatchObject({ reason: "plaintext_hash_mismatch" });
});
it.each(["companyId", "workspaceId", "restorePointId", "generation"] as const)("rejects a candidate from another %s", async scope => {
const f = await fixture("empty", scope); await f.remove();
await expect(f.load()).rejects.toMatchObject({ reason: "key_identity_mismatch" });
});
it("does not replace a present corrupt encrypted object with a reconstructed payload", async () => {
const f = await fixture(); await f.put(Buffer.from("corrupted"));
await expect(f.load()).rejects.toThrow();
});
it("retains a missing nonempty checkpoint as an unresolved obligation", async () => {
const f = await fixture("file"); await f.remove();
await expect(f.load()).rejects.toMatchObject({ reason: "restore_point_unavailable" });
expect(await fs.readFile(path.join(f.directory, "saved.txt"), "utf8")).toBe("must never disappear\n");
});
it("does not treat an unreadable present object as missing", async () => {
const f = await fixture();
vi.spyOn(f.storage, "getObject").mockRejectedValue(new Error("read unavailable"));
await expect(f.load()).rejects.toMatchObject({ reason: "restore_point_unavailable" });
});
it("does not reconstruct when object absence cannot be established", async () => {
const f = await fixture(); await f.remove();
vi.spyOn(f.storage, "headObject").mockRejectedValue(new Error("head unavailable"));
await expect(f.load()).rejects.toThrow();
});
it("refuses an expired checkpoint even when its exact contents are reconstructible", async () => {
const f = await fixture(); await f.remove();
await expect(loadTaskDirectoryRestorePoint({ storage: f.storage, workspace: { companyId: f.ref.objectKey.split("/")[0]!, id: f.ref.objectKey.split("/")[2]! },
ref: { ...f.ref, expiredAt: new Date().toISOString() }, recoverMissingEmpty: true })).rejects.toMatchObject({ reason: "restore_point_expired" });
});
it("cancels a stalled absence check through the real storage service boundary", async () => {
const f = await fixture(); await f.remove();
const controller = new AbortController(), reason = new Error("migration deadline");
let entered!: () => void, release!: () => void;
const started = new Promise<void>(resolve => { entered = resolve; });
vi.spyOn(f.provider, "headObject").mockImplementation(() => new Promise(resolve => {
release = () => resolve({ exists: false }); entered();
}));
const result = loadTaskDirectoryRestorePoint({ storage: f.storage,
workspace: { companyId: f.ref.objectKey.split("/")[0]!, id: f.ref.objectKey.split("/")[2]! },
ref: f.ref, recoverMissingEmpty: true, signal: controller.signal }).then(payload => ({ payload }), error => ({ error }));
await started; controller.abort(reason);
let timeout: ReturnType<typeof setTimeout> | undefined;
try {
const observed = await Promise.race([result, new Promise(resolve => { timeout = setTimeout(() => resolve("still waiting"), 1000); })]);
expect(observed).toEqual({ error: reason });
} finally {
clearTimeout(timeout); release(); await result;
}
});
+24 -8
View File
@@ -554,6 +554,28 @@ export async function sealRestorePoint(payload: Buffer, identity: RestorePointId
return { ciphertext, sha256: createHash("sha256").update(ciphertext).digest("hex"), key: material };
}
async function restorePointKey(input: { key: StoredSecretVersionMaterial; identity: RestorePointIdentity }) {
const unwrapped = JSON.parse(await localEncryptedProvider.resolveVersion({ material: input.key, externalRef: null } as Parameters<typeof localEncryptedProvider.resolveVersion>[0]));
const { identity } = input;
if (unwrapped.schema !== KEY_SCHEMA || unwrapped.companyId !== identity.companyId || unwrapped.workspaceId !== identity.workspaceId
|| unwrapped.restorePointId !== identity.restorePointId || unwrapped.generation !== identity.generation) {
throw new RestorePointRestoreFailure("key_identity_mismatch");
}
return unwrapped as RestorePointIdentity & { dataKey: string; plaintextBytes: number; plaintextSha256: string };
}
function verifyRestorePointPayload(payload: Buffer, key: { plaintextBytes: number; plaintextSha256: string }) {
if (payload.length !== key.plaintextBytes || createHash("sha256").update(payload).digest("hex") !== key.plaintextSha256) {
throw new RestorePointRestoreFailure("plaintext_hash_mismatch");
}
}
export async function verifyRestorePointPayloadCandidate(input: {
payload: Buffer; key: StoredSecretVersionMaterial; identity: RestorePointIdentity;
}) {
verifyRestorePointPayload(input.payload, await restorePointKey(input));
}
export async function openRestorePoint(input: {
ciphertext: Buffer;
sha256: string;
@@ -567,12 +589,8 @@ export async function openRestorePoint(input: {
if (input.ciphertext.length < minimum || !input.ciphertext.subarray(0, SEALED_MAGIC.length).equals(SEALED_MAGIC)) {
throw new RestorePointRestoreFailure("invalid_ciphertext");
}
const unwrapped = JSON.parse(await localEncryptedProvider.resolveVersion({ material: input.key, externalRef: null } as Parameters<typeof localEncryptedProvider.resolveVersion>[0]));
const unwrapped = await restorePointKey(input);
const { identity } = input;
if (unwrapped.schema !== KEY_SCHEMA || unwrapped.companyId !== identity.companyId || unwrapped.workspaceId !== identity.workspaceId
|| unwrapped.restorePointId !== identity.restorePointId || unwrapped.generation !== identity.generation) {
throw new RestorePointRestoreFailure("key_identity_mismatch");
}
const iv = input.ciphertext.subarray(SEALED_MAGIC.length, SEALED_MAGIC.length + 12);
const tag = input.ciphertext.subarray(input.ciphertext.length - 16);
const body = input.ciphertext.subarray(SEALED_MAGIC.length + 12, input.ciphertext.length - 16);
@@ -585,9 +603,7 @@ export async function openRestorePoint(input: {
} catch {
throw new RestorePointRestoreFailure("ciphertext_authentication_failed");
}
if (payload.length !== unwrapped.plaintextBytes || createHash("sha256").update(payload).digest("hex") !== unwrapped.plaintextSha256) {
throw new RestorePointRestoreFailure("plaintext_hash_mismatch");
}
verifyRestorePointPayload(payload, unwrapped);
return payload;
}
@@ -7,7 +7,7 @@ import { environmentLeases, executionWorkspaceRestorePoints, executionWorkspaces
import type { StoredSecretVersionMaterial } from "../secrets/types.js";
import type { StorageService } from "../storage/types.js";
import { resolvePaperclipInstanceRoot } from "../home-paths.js";
import { openRestorePoint, sealRestorePoint } from "./git-restore-point.js";
import { openRestorePoint, sealRestorePoint, verifyRestorePointPayloadCandidate } from "./git-restore-point.js";
import { notifyGitRestorePointWork } from "./git-restore-point-ledger.js";
import { abandonGitRestorePoint, activateGitRestorePoints, readRestorePointObject, storeSealedRestorePoint } from "./git-restore-point-storage.js";
import { allocatedBytes, describeGitExpirationFailure, expirationScope, isGitWorkspaceExpirationEligible, type GitExpirationFailure } from "./git-workspace-expiration.js";
@@ -15,6 +15,7 @@ import { IsolatedRepositorySourceBusy, withIsolatedRepositorySourceLock } from "
import { logActivity } from "./activity-log.js";
import {
captureTaskDirectorySnapshot,
createEmptyTaskDirectorySnapshot,
inspectTaskDirectory,
materializeTaskDirectorySnapshot,
newestTaskDirectoryMtime,
@@ -143,16 +144,26 @@ export async function loadTaskDirectoryRestorePoint(input: {
storage: StorageService;
workspace: { companyId: string; id: string };
ref: TaskDirectoryRestorePointRef;
recoverMissingEmpty?: boolean;
signal?: AbortSignal;
}) {
const ref = validateTaskDirectoryRestorePointRef(input.ref, input.workspace);
if (ref.expiredAt) throw new TaskDirectoryRestoreFailure("restore_point_expired");
const ciphertext = await readRestorePointObject(input.storage, input.workspace.companyId, ref.objectKey, input.signal).catch(() => {
throw new TaskDirectoryRestoreFailure("restore_point_unavailable");
const identity = { companyId: input.workspace.companyId, workspaceId: input.workspace.id, restorePointId: ref.id, generation: ref.generation };
const ciphertext = await readRestorePointObject(input.storage, input.workspace.companyId, ref.objectKey, input.signal).catch(async () => {
input.signal?.throwIfAborted();
if (!input.recoverMissingEmpty || ref.files !== 0 || ref.excluded !== 0) throw new TaskDirectoryRestoreFailure("restore_point_unavailable");
const head = await input.storage.headObject(input.workspace.companyId, ref.objectKey, { signal: input.signal }).catch(() => null);
input.signal?.throwIfAborted();
if (head?.exists !== false) throw new TaskDirectoryRestoreFailure("restore_point_unavailable");
const candidate = createEmptyTaskDirectorySnapshot();
if (candidate.digest !== ref.digest) throw new TaskDirectoryRestoreFailure("restored_contents_mismatch");
await verifyRestorePointPayloadCandidate({ payload: candidate.payload, key: ref.key, identity });
return null;
});
if (ciphertext === null) return createEmptyTaskDirectorySnapshot().payload;
if (ciphertext.length !== ref.byteSize) throw new TaskDirectoryRestoreFailure("ciphertext_size_mismatch");
return openRestorePoint({ ciphertext, sha256: ref.sha256, key: ref.key,
identity: { companyId: input.workspace.companyId, workspaceId: input.workspace.id, restorePointId: ref.id, generation: ref.generation } });
return openRestorePoint({ ciphertext, sha256: ref.sha256, key: ref.key, identity });
}
function skipReason(error: unknown) {
@@ -148,7 +148,14 @@ export async function captureTaskDirectorySnapshot(root: string, options: { limi
await handle.close();
}
} });
const manifest: TaskDirectorySnapshotManifest = { schema: TASK_DIRECTORY_SNAPSHOT_SCHEMA, entries: walked.entries, excluded: walked.excluded };
return encodeTaskDirectorySnapshot({ schema: TASK_DIRECTORY_SNAPSHOT_SCHEMA, entries: walked.entries, excluded: walked.excluded }, bodies);
}
export function createEmptyTaskDirectorySnapshot() {
return encodeTaskDirectorySnapshot({ schema: TASK_DIRECTORY_SNAPSHOT_SCHEMA, entries: [], excluded: [] }, []);
}
function encodeTaskDirectorySnapshot(manifest: TaskDirectorySnapshotManifest, bodies: Buffer[]) {
const encoded = Buffer.from(JSON.stringify(manifest));
const length = Buffer.alloc(4);
length.writeUInt32BE(encoded.length);
@@ -158,8 +165,8 @@ export async function captureTaskDirectorySnapshot(root: string, options: { limi
return {
payload,
digest: taskDirectoryManifestDigest(manifest),
files: walked.entries.filter(entry => entry.type === "file").length,
excluded: walked.excluded.length,
files: manifest.entries.filter(entry => entry.type === "file").length,
excluded: manifest.excluded.length,
};
}
@@ -853,6 +853,26 @@ describe.skipIf(!support.supported)("remote workspace migration", () => {
await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: ref!.manifestDigest });
expect(await fs.readFile(path.join(targetDir, "saved.txt"), "utf8")).toBe("original saved work\n");
});
it("migrates an exactly authenticated empty retained payload without rewriting its missing source", async () => {
const f = await seed(); await fs.rm(path.join(f.cwd, "saved.txt"));
const retained = await retainDirectory(f); await storage.deleteObject(f.companyId, retained.objectKey);
await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId));
const before = await workspace(f.workspaceId);
expect(await migrateRemoteWorkspace({ db, storage, ...f, apply: true })).toMatchObject({ applied: true });
const saved = await workspace(f.workspaceId), authority = readRemoteWorkspaceAuthority(saved.metadata)!;
expect(saved).toMatchObject({ status: "archived", closedAt: before.closedAt });
expect(saved.metadata?.isolatedTaskDirectoryRetention).toEqual(before.metadata?.isolatedTaskDirectoryRetention);
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ status: "done", executionWorkspaceId: f.workspaceId });
expect(await fs.stat(f.cwd).catch(() => null)).toBeNull();
expect(await storage.headObject(f.companyId, retained.objectKey)).toMatchObject({ exists: false });
const ref = await getWorkspaceSnapshot(db, { ...f, snapshotId: authority.snapshotId! });
const archivePath = path.join(root, "recovered-empty.pcws"), targetDir = path.join(root, "recovered-empty");
await loadWorkspaceSnapshotArchive({ storage, reference: ref!, targetPath: archivePath });
await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: ref!.manifestDigest });
expect(await fs.readdir(targetDir)).toEqual([]);
expect(await migrateRemoteWorkspace({ db, storage, ...f, apply: true })).toMatchObject({ applied: true, replayed: true });
expect(readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)?.snapshotId).toBe(authority.snapshotId);
});
it.each(["missing", "expired", "corrupt"])("does not replace an unavailable retained directory with an empty workspace (%s)", async failure => {
const f = await seed(), retained = await retainDirectory(f);
if (failure === "missing") await storage.deleteObject(f.companyId, retained.objectKey);
@@ -321,7 +321,7 @@ export async function migrateRemoteWorkspace(input: {
snapshotId: snapshot.id, targetPath: archivePath, signal: input.signal,
}) : null;
const sourceRoot = recovered ? null : retained ? await materializeTaskDirectorySnapshot({
payload: await loadTaskDirectoryRestorePoint({ storage: input.storage, workspace: reserved.workspace, ref: retained }),
payload: await loadTaskDirectoryRestorePoint({ storage: input.storage, workspace: reserved.workspace, ref: retained, recoverMissingEmpty: true, signal: input.signal }),
target: path.join(scratch, "retained-source"), expectedDigest: retained.digest,
}) : reserved.archivedGitSource ? await materializeArchivedGitMigrationSource({ db: input.db, storage: input.storage,
workspace: reserved.workspace, source: reserved.archivedGitSource, scratch }) : reserved.workspace.cwd ?? reserved.workspace.providerRef!;