Files
PaperClipAI/server/src/services/native-runtime/native-harness-backup-stamp.ts
T
DottaandPaperclip d1306f338b fix(runtime): preserve settled failed harness backups online (#19715)
## Thinking Path

> - Paperclip coordinates agents, tasks, and their saved execution
state.
> - A replacement server needs native provider backups in shared
storage.
> - The online pre-copy owner accepts successful runs with a committed
result.
> - A sealed backup from a failed or cancelled run can still contain
user history.
> - The existing owner defers those files even after their source
ownership is settled.
> - This change adds a separate preservation owner with exact source and
publication checks.
> - The copy retains the failed outcome and original files while making
the backup readable from a fresh home.

## Linked Issues or Issue Description

Refs #19137 and #19108 for the existing pre-copy and encrypted backup
owners.

**What happened?**

Online pre-copy reports `source_not_successful` for a sealed legacy
backup from a settled failed or cancelled run. The provider files remain
tied to the old disk.

**Expected behavior**

Copy the exact recorded backup when the source has no remaining writer
ownership. Preserve the run status, warnings, result, allocation,
selected session, and local files.

**Steps to reproduce**

1. Create a lease-stamped suspended native backup.
2. Settle its source as a failed run with a terminal-failure coordinator
and no owner.
3. Keep an unrelated execution active.
4. Run canonical online pre-copy. The previous owner defers the source.

## What Changed

- Add a narrow preservation owner for terminal source files. It checks
the physical finalization lock, task, coordinator, run, selected
session, source lease, and source fingerprints.
- Recheck that owner before the encrypted upload receipt and
current-backup publication.
- Reuse the finite workspace keepalive during long source checks and
defer source-local invalid archives.
- Add optional exact-root stamp verification. A valid previous backup
cannot authorize a changed current payload. Existing callers keep their
fallback behavior.
- Bind the archive input to the checked owner and record an activity
event after publication.
- Keep pending object receipts after an uncertain or failed publication.
- Add focused PostgreSQL and filesystem controls. Update the existing
migration runbook.

## Verification

- Real PostgreSQL baseline on unchanged production: the new positive
case fails with `source_not_successful` instead of publishing the
backup.
- Same case passes after the change and decrypts/restores the exact
provider history.
- Final head 4b6cf093be: three focused suites pass, 81 tests in 62.26
seconds, including existing storage and stamp consumers. The canonical
PostgreSQL wrapper exited zero and completed cleanup.
- Controls cover cancelled and failed sources, real committed results,
expired nonnull owners, physical owner/lock contention, late
source/owner changes, previous-tree fallback, and archive root/scope
binding.
- Both review regressions fail with the committed pre-fix owners: a real
30-second database idle timeout expires during a 32-second exact-source
check, and a stamp-valid invalid archive stops the pass. The fixed owner
keeps its transaction live and copies the next valid source. Source
hashes and automatic file restoration were recorded for each leg.
- Final-head strict server TypeScript and module-boundary checks pass
after the master rebase and required Runner/shared declaration rebuild.
- No source migration, provider mutation, or archive apply was executed.

## Risks

This path does not reconstruct missing backups or prove that missing
provider state is recoverable. Such sources remain blockers. It does not
take expired owners or report failed work as successful. Source hashing
adds bounded disk work while source row locks are held; unrelated
executions remain active in the PostgreSQL fixture. Existing archive
size limits, authenticated encryption, object readback, and publication
fencing remain in place. No schema, API route, scheduler, or
rollout-policy change is included.

## Model Used

OpenAI Codex (GPT-6 family; exact runtime model ID and context capacity
are not exposed in this session), with reasoning, repository editing,
code execution, and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 21:08:27 -06:00

263 lines
11 KiB
TypeScript

import { createHash } from "node:crypto";
import { constants, lstatSync, readFileSync, readdirSync, readlinkSync } from "node:fs";
import { lstat, open, readdir, readlink } from "node:fs/promises";
import { MAX_NATIVE_HARNESS_BYTES, MAX_NATIVE_HARNESS_ENTRIES } from "./native-harness-backup-archive.js";
import { dirname, resolve } from "node:path";
import { resolvePaperclipInstanceRoot } from "../../home-paths.js";
export interface NativeHarnessBackupStamp {
schema: "paperclip.native-harness-backup-stamp.v2";
normalizedSessionId: string;
runnerInstanceId: string;
sessionScopeSha256: string;
sourceProviderLeaseId: string;
authorizedProviderLeaseId: string;
manifestSha256: string;
completedAt: string;
}
export function nativeHarnessStateBase(): string {
return resolve(
process.env.PAPERCLIP_RUNNER_STATE_DIR ??
resolve(
resolvePaperclipInstanceRoot(),
"runtime",
"paperclip-runner",
"durable-sessions",
),
);
}
function stateRootFromDigest(digest: string): string | null {
if (!/^[0-9a-f]{64}$/.test(digest)) return null;
const base = nativeHarnessStateBase();
const root = resolve(base, digest);
return dirname(root) === base ? root : null;
}
function isRealDirectory(path: string): boolean {
try {
const metadata = lstatSync(path);
return metadata.isDirectory() && !metadata.isSymbolicLink();
} catch {
return false;
}
}
function isRealFile(path: string): boolean {
try {
const metadata = lstatSync(path);
return metadata.isFile() && !metadata.isSymbolicLink();
} catch {
return false;
}
}
function digestDirectory(directory: string): { sha256: string; bytes: number } {
const hash = createHash("sha256");
let bytes = 0;
const visit = (current: string, relative: string) => {
const entries = readdirSync(current, { withFileTypes: true }).sort(
(left, right) => left.name.localeCompare(right.name),
);
if (entries.length === 0) hash.update(`directory:${relative}\0`);
for (const entry of entries) {
const path = resolve(current, entry.name);
const relativePath = relative ? `${relative}/${entry.name}` : entry.name;
const stats = lstatSync(path);
if (entry.isDirectory()) {
hash.update(`directory:${relativePath}:${stats.mode & 0o777}\0`);
visit(path, relativePath);
} else if (entry.isSymbolicLink()) {
hash.update(`symlink:${relativePath}:${readlinkSync(path)}\0`);
} else if (entry.isFile()) {
const contents = readFileSync(path);
bytes += contents.byteLength;
hash.update(
`file:${relativePath}:${stats.mode & 0o777}:${contents.byteLength}\0`,
);
hash.update(contents);
} else {
throw new Error(
`runner_harness_backup_unsupported_entry:${relativePath}`,
);
}
}
};
visit(directory, "");
return { sha256: `sha256:${hash.digest("hex")}`, bytes };
}
export function createNativeHarnessBackupStamp(input: {
manifestPath: string;
sessionScopeId: string;
authorizedProviderLeaseId: string;
normalizedSessionId: string;
runnerInstanceId: string;
completedAt: string;
}): NativeHarnessBackupStamp {
if (!input.authorizedProviderLeaseId) {
throw new Error("runner_harness_backup_lease_missing");
}
const sessionScopeSha256 = createHash("sha256")
.update(input.sessionScopeId)
.digest("hex");
const root = stateRootFromDigest(sessionScopeSha256);
if (!root || !isRealDirectory(root)) {
throw new Error("runner_harness_backup_scope_invalid");
}
const resolvedManifestPath = resolve(input.manifestPath);
const allowedManifestPaths = ["current", "previous"].map((candidate) =>
resolve(root, "failover-backups", candidate, "manifest.json"),
);
if (
!allowedManifestPaths.includes(resolvedManifestPath) ||
!isRealDirectory(dirname(resolvedManifestPath)) ||
!isRealFile(resolvedManifestPath)
) {
throw new Error("runner_harness_backup_scope_invalid");
}
const manifestBytes = readFileSync(input.manifestPath);
const manifest = JSON.parse(manifestBytes.toString("utf8")) as Record<
string,
unknown
>;
if (
manifest.schema !== "paperclip.native-harness-backup.v1" ||
manifest.normalizedSessionId !== input.normalizedSessionId ||
manifest.runnerInstanceId !== input.runnerInstanceId ||
manifest.completedAt !== input.completedAt ||
typeof manifest.sourceProviderLeaseId !== "string" ||
!manifest.sourceProviderLeaseId
) {
throw new Error("runner_harness_backup_scope_invalid");
}
return {
schema: "paperclip.native-harness-backup-stamp.v2",
normalizedSessionId: input.normalizedSessionId,
runnerInstanceId: input.runnerInstanceId,
sessionScopeSha256,
sourceProviderLeaseId: manifest.sourceProviderLeaseId,
authorizedProviderLeaseId: input.authorizedProviderLeaseId,
manifestSha256: `sha256:${createHash("sha256").update(manifestBytes).digest("hex")}`,
completedAt: input.completedAt,
};
}
export function readNativeHarnessBackupStamp(
value: unknown,
expectedProviderLeaseId: string,
): NativeHarnessBackupStamp | null {
if (!expectedProviderLeaseId) return null;
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
const stamp = value as Record<string, unknown>;
if (
stamp.schema !== "paperclip.native-harness-backup-stamp.v2" ||
typeof stamp.normalizedSessionId !== "string" ||
!stamp.normalizedSessionId ||
typeof stamp.runnerInstanceId !== "string" ||
!stamp.runnerInstanceId ||
typeof stamp.sessionScopeSha256 !== "string" ||
typeof stamp.sourceProviderLeaseId !== "string" ||
!stamp.sourceProviderLeaseId ||
stamp.authorizedProviderLeaseId !== expectedProviderLeaseId ||
typeof stamp.manifestSha256 !== "string" ||
!/^sha256:[a-f0-9]{64}$/.test(stamp.manifestSha256) ||
!/^[a-f0-9]{64}$/.test(stamp.sessionScopeSha256) ||
typeof stamp.completedAt !== "string" || !Number.isFinite(Date.parse(stamp.completedAt))
)
return null;
return stamp as unknown as NativeHarnessBackupStamp;
}
function matchingBackupDirectories(candidate: string, stamp: NativeHarnessBackupStamp): Array<{ path: string; sha256: string; bytes: number }> | null {
const manifestPath = resolve(candidate, "manifest.json");
if (!isRealDirectory(candidate) || !isRealFile(manifestPath) || lstatSync(manifestPath).size > 4 * 1024 * 1024) return null;
const bytes = readFileSync(manifestPath);
if (`sha256:${createHash("sha256").update(bytes).digest("hex")}` !== stamp.manifestSha256) return null;
const manifest = JSON.parse(bytes.toString("utf8"));
if (manifest.schema !== "paperclip.native-harness-backup.v1" || manifest.normalizedSessionId !== stamp.normalizedSessionId
|| manifest.runnerInstanceId !== stamp.runnerInstanceId || manifest.sourceProviderLeaseId !== stamp.sourceProviderLeaseId
|| !Array.isArray(manifest.directories) || manifest.directories.length === 0) return null;
const directories: Array<{ path: string; sha256: string; bytes: number }> = [];
for (const entry of manifest.directories) {
if (!entry || typeof entry !== "object" || Array.isArray(entry) || typeof entry.name !== "string"
|| !/^[A-Za-z0-9._-]+$/.test(entry.name) || [".", ".."].includes(entry.name)
|| typeof entry.sha256 !== "string" || !Number.isSafeInteger(entry.bytes) || entry.bytes < 0) return null;
const path = resolve(candidate, entry.name);
if (!isRealDirectory(path)) return null;
directories.push({ path, sha256: entry.sha256, bytes: entry.bytes });
}
return directories;
}
function backupCandidates(stamp: NativeHarnessBackupStamp) {
const root = stateRootFromDigest(stamp.sessionScopeSha256);
return !root || !isRealDirectory(root) ? [] : ["current", "previous"].map(name => resolve(root, "failover-backups", name));
}
export function verifyNativeHarnessBackupStamp(value: unknown, expectedProviderLeaseId: string): boolean {
const stamp = readNativeHarnessBackupStamp(value, expectedProviderLeaseId);
if (!stamp) return false;
for (const candidate of backupCandidates(stamp)) {
try {
const directories = matchingBackupDirectories(candidate, stamp);
if (directories?.every(directory => {
const actual = digestDirectory(directory.path);
return actual.sha256 === directory.sha256 && actual.bytes === directory.bytes;
})) return true;
} catch {}
}
return false;
}
export async function verifyNativeHarnessBackupStampAsync(value: unknown, expectedProviderLeaseId: string, signal: AbortSignal,
exactRoot?: string): Promise<boolean> {
const stamp = readNativeHarnessBackupStamp(value, expectedProviderLeaseId);
if (!stamp) return false;
for (const candidate of backupCandidates(stamp)) {
// A preservation publisher must authenticate the tree it actually copied;
// an intact previous backup cannot authorize a changed current payload.
if (exactRoot !== undefined && resolve(candidate) !== resolve(exactRoot)) continue;
signal.throwIfAborted();
try {
const directories = matchingBackupDirectories(candidate, stamp);
if (!directories || directories.reduce((total, entry) => total + entry.bytes, 0) > MAX_NATIVE_HARNESS_BYTES) continue;
let valid = true, count = 0;
for (const directory of directories) {
const hash = createHash("sha256"); let bytes = 0;
const visit = async (current: string, relative: string): Promise<void> => {
signal.throwIfAborted();
const entries = (await readdir(current, { withFileTypes: true })).sort((a, b) => a.name.localeCompare(b.name));
if (!entries.length) hash.update(`directory:${relative}\0`);
for (const entry of entries) {
if (++count > MAX_NATIVE_HARNESS_ENTRIES) throw new Error("native_harness_backup_entry_limit");
const path = resolve(current, entry.name), name = relative ? `${relative}/${entry.name}` : entry.name;
const stats = await lstat(path);
if (stats.isDirectory()) { hash.update(`directory:${name}:${stats.mode & 0o777}\0`); await visit(path, name); }
else if (stats.isSymbolicLink()) hash.update(`symlink:${name}:${await readlink(path)}\0`);
else if (stats.isFile()) {
hash.update(`file:${name}:${stats.mode & 0o777}:${stats.size}\0`);
const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
try {
const before = await handle.stat();
if (!before.isFile() || stats.ino !== before.ino || stats.dev !== before.dev || stats.size !== before.size) throw new Error("native_harness_backup_source_changed");
signal.throwIfAborted();
for await (const chunk of handle.createReadStream({ signal, autoClose: false })) {
bytes += chunk.length; if (bytes > directory.bytes) throw new Error("native_harness_backup_size_mismatch"); hash.update(chunk);
}
const after = await handle.stat();
if (before.size !== after.size || before.mtimeMs !== after.mtimeMs || before.ctimeMs !== after.ctimeMs) throw new Error("native_harness_backup_source_changed");
} finally { await handle.close(); }
} else throw new Error("native_harness_backup_unsupported_entry");
}
};
await visit(directory.path, "");
if (`sha256:${hash.digest("hex")}` !== directory.sha256 || bytes !== directory.bytes) { valid = false; break; }
}
if (valid) return true;
} catch { signal.throwIfAborted(); }
}
return false;
}