Files
PaperClipAI/server/src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts
T
DottaandPaperclip de7ceac0de fix: preserve compatible historical Codex WEB evidence (#20665)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native session migration preserves local evidence in encrypted
archives.
> - Each WEB receipt binds the exact provider directory and exclusion
policy.
> - The Codex credential refresh change added two excluded files and
changed that policy hash.
> - A new reader then rejected old receipts even when their facts,
files, skills, and authority still matched.
> - This pull request accepts the exact prior Codex policy while
checking archive paths against the current credential policy.
> - Operators can keep valid saved evidence across the upgrade without
copying it again.

## Linked Issues or Issue Description

Refs: #19814, #20646.

**What happened?**

A reader upgrade rejected saved Codex WEB evidence because the profile
hash changed when `auth-refresh.json` and `auth-refresh.json.tmp` became
excluded credentials. The other receipt bindings still matched. This
blocked the native harness migration gate.

**Expected behavior**

The exact known prior profile remains readable when the current archive
path checks accept every saved entry. An archive that includes either
newly excluded credential file must still fail. Unknown profiles and
changed authority must fail.

**Steps to reproduce**

1. Preserve WEB evidence with the frozen prior Codex profile and no
credential refresh files.
2. Read its unchanged receipt with the current profile.
3. Observe that the old reader returns false solely because the profile
hash changed.

**Paperclip version or commit**

Reproduced against `a3b34a9d88`; final branch is based on `e9343d8501`.

## What Changed

- Accept the exact prior Codex profile hash only when the complete
current profile has the reviewed successor hash and the provider is
Codex.
- Keep the current entry and excluded-path checks. Keep facts,
controller, archive, skill, and ownership checks.
- Add a frozen prior policy fixture with its literal hash. Test old
receipt reads, both credential files, unknown policies, changed
authority, and current writes.

## Verification

- Real PostgreSQL baseline: the frozen old-policy receipt test failed
because the reader returned false. The unchanged fixture passes with
this fix.
- All 18 selected PostgreSQL cases pass on final head after the master
rebase: six new cases and twelve existing WEB archive and authority
guards (14.93 seconds).
- Full layered server typecheck passes on final head after the rebase.
The module boundary check passes.
- A single read-only native-only candidate check on the deployed
dependency graph validated all 54 retained scopes. The 29 prior failures
passed, and the other 25 stayed valid. The check made no storage calls
or data writes. It did not replace final deployed enrollment acceptance.
- Final head `14112d0e3abb6e318a86fe9799d95bddd2dbf76b` passes all
[required CI
checks](https://github.com/paperclipai/paperclip/actions/runs/42919867108):
55 successful checks, four expected skips, and one neutral review. The
legacy security status also passes. Exact-head Greptile review is 5/5
with no open review threads.

## Risks

The compatibility branch accepts only the two reviewed policy hashes. A
future policy change can require another explicit compatibility
decision. The reader still rejects saved credential refresh files. It
does not rewrite receipts or change execution authority. No stored
schema or operator command changes.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window capacity are not exposed in this session. Used reasoning,
source editing, shell execution, and tool calls.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 17:22:08 -06:00

1415 lines
116 KiB
TypeScript

import { createHash, randomUUID } from "node:crypto";
import { chmod, copyFile, cp, lstat, mkdir, mkdtemp, readFile, realpath, rename, rm, symlink, truncate, utimes, writeFile } from "node:fs/promises";
import { tmpdir, userInfo } from "node:os";
import { basename, join } from "node:path";
import { and, eq } from "drizzle-orm";
import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared";
import { afterAll, afterEach, beforeAll, expect, it, vi } from "vitest";
import { activityLog, agentTaskSessions, agents, companies, companySkills, companySkillVersions, companySkillRepositorySnapshots, createDb, environmentLeases,
heartbeatRuns, instanceSettings, issues, nativeHarnessBackups, nativeRunnerCheckpoints, nativeRunFinalizations, nativeRunResults } from "@paperclipai/db";
import { describeEmbeddedPostgres } from "../../__tests__/helpers/native-restart.js";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js";
import { createStorageService } from "../../storage/service.js";
import { canonicalNativeRuntimeContextDigest, captureLegacyDurablePrpCheckpoint, DurablePrpControlPlane, parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
import { materializeAsset } from "./runtime-context.js";
import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js";
import { preserveRuntimeSkillAssets } from "./runtime-skill-preservation.js";
import { nativeControllerMigrationDescriptor, scopedRunnerdStateRoot } from "./native-session-executor.js";
import * as nativeSessionExecutor from "./native-session-executor.js";
import { hasLocalNativeHarnessObligations } from "./native-harness-backup-migration.js";
import { preserveNativeHarnessFilesystemIndependence, hasValidNativeHarnessFilesystemIndependence } from "./native-harness-filesystem-independence.js";
import * as filesystemIndependence from "./native-harness-filesystem-independence.js";
import { originalHarnessVolumeAnchorPath, verifyOriginalHarnessVolumeSource, type OriginalHarnessVolumeAnchor,
type ReviewedOriginalHarnessVolumeWitness } from "./native-harness-original-volume.js";
import { removeRuntimeAssetTree } from "./runtime-asset-retention.js";
import { skillPackageStore } from "../skill-package-store.js";
import { repositorySkillProjection } from "../skill-repository-package.js";
import { skillFileBytes } from "../skill-snapshot.js";
import { tryLockNativeWorkspaceFinalization } from "./native-workspace-finalization-ownership.js";
import { loadNativeRunnerCheckpoint, nativeRunnerCheckpointWriter } from "./native-runner-checkpoint.js";
import { createNativeControllerRecovery } from "./native-controller-recovery.js";
import { ensureNativeCompletionContract } from "./completion-contracts.js";
import { restoreDurableNativeHarnessBackup } from "./native-harness-backup-storage.js";
import { nativeToolContractFingerprintForTarget } from "./native-session-resume.js";
import { nativeCheckpointRetirement } from "./native-runner-checkpoint-retirement.js";
import { nativeSha256 } from "./canonical.js";
import { nativeControllerEvidenceDescriptor, NATIVE_CONTROLLER_WEB_LIMITS, type NativeControllerHistoryManifestEntry } from "./native-controller-archive.js";
import { logActivity } from "../activity-log.js";
import { LEGACY_WORKSPACE_RECOVERY_MIGRATION } from "../legacy-workspace-recovery-migration.js";
vi.mock("../../build-commit.js", () => ({ readBuildCommit: () => "a".repeat(40) }));
const CODEX_WEB_PROFILE_8E4 = {
providerKind: "codex", driverKind: "codex_app_server",
directories: [
{ name: "runner", location: "runner", excludeEntries: [] },
{ name: "codex-home", location: "filesystem", excludeEntries: [
"tmp", ".tmp", "auth.json", "config.toml", ".remote-plugin-install-staging", "cache/remote_plugin_catalog",
"plugins/.remote-plugin-install-staging", "plugins/cache/openai-curated-remote", "plugins/cache/created-by-me-remote",
"plugins/cache/workspace-directory", "plugins/cache/workspace-shared-with-me",
"plugins/cache/workspace-shared-with-me-private", "plugins/cache/workspace-shared-with-me-unlisted",
] },
],
} as const;
const CODEX_WEB_PROFILE_8E4_DIGEST = "6f10c4b4424e84defcdb4c7e932921d09ea73dc0f39c96e61777c8b20c5f637b";
describeEmbeddedPostgres("owner-verified native harness filesystem independence", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>, db: ReturnType<typeof createDb>;
const roots: string[] = [];
beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("native-harness-filesystem-"); db = createDb(database.connectionString); }, 90_000);
afterEach(async () => { vi.restoreAllMocks(); vi.unstubAllEnvs(); await db.delete(agentTaskSessions); for (const root of roots.splice(0)) await removeRuntimeAssetTree(root); });
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await database?.cleanup(); });
async function fixture(kind: "unversioned" | "repository" | "package" | "deleted" | "none" | "many-web-files" = "unversioned", managedWorkspace = false) {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "61".repeat(32));
const mount = await realpath(await mkdtemp(join(tmpdir(), "native-harness-original-"))); roots.push(mount);
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), sessionId = randomUUID(), runnerId = randomUUID();
const projectId = randomUUID(), serviceId = randomUUID(), environmentId = randomUUID(), volumeId = randomUUID(), deploymentId = randomUUID();
for (const [key, value] of Object.entries({ PAPERCLIP_HOME: mount, PAPERCLIP_INSTANCE_ID: "default", RAILWAY_PROJECT_ID: projectId,
RAILWAY_SERVICE_ID: serviceId, RAILWAY_ENVIRONMENT_ID: environmentId, RAILWAY_VOLUME_ID: volumeId,
RAILWAY_DEPLOYMENT_ID: deploymentId, RAILWAY_VOLUME_MOUNT_PATH: mount,
PAPERCLIP_RUNNER_STATE_DIR: join(mount, "instances/default/runtime/paperclip-runner/durable-sessions") })) vi.stubEnv(key, value);
await mkdir(process.env.PAPERCLIP_RUNNER_STATE_DIR!, { recursive: true, mode: 0o700 });
await mkdir(join(mount, "instances/default/recovery"), { recursive: true, mode: 0o700 });
const m = await lstat(mount), instance = await lstat(join(mount, "instances/default"));
const anchor: OriginalHarnessVolumeAnchor = { schema: "paperclip.native-harness-original-volume-anchor.v1", nonce: "b".repeat(64),
projectId, serviceId, environmentId, volumeId, volumeInstanceId: randomUUID(), mountPath: mount,
mountIdentity: { dev: m.dev, ino: m.ino }, instanceIdentity: { dev: instance.dev, ino: instance.ino },
createdFrom: { deploymentId: randomUUID(), commit: "c".repeat(40), providerWitnessSha256: "d".repeat(64) }, createdAt: new Date().toISOString() };
const bytes = Buffer.from(JSON.stringify(anchor)); await writeFile(originalHarnessVolumeAnchorPath(), bytes, { mode: 0o600 });
const observed = Date.now();
const witness: ReviewedOriginalHarnessVolumeWitness = { projectId, serviceId, environmentId, volumeId, volumeInstanceId: anchor.volumeInstanceId,
mountPath: mount, currentDeploymentId: deploymentId, currentCommit: "a".repeat(40), currentMountIdentity: anchor.mountIdentity,
currentInstanceIdentity: anchor.instanceIdentity, observedAt: new Date(observed).toISOString(), expiresAt: new Date(observed + 120_000).toISOString(),
anchorSha256: createHash("sha256").update(bytes).digest("hex"), providerWitnessSha256: "e".repeat(64) };
const source = await verifyOriginalHarnessVolumeSource(witness);
const storage = createStorageService({ ...createLocalDiskStorageProvider(join(mount, "objects")), id: "s3" });
await db.insert(companies).values({ id: companyId, name: "Original-volume preservation", issuePrefix: `F${companyId.slice(0, 6)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Retained source" });
await db.insert(issues).values({ id: issueId, companyId, title: "Retain prior failure", status: "blocked" });
const runtimeContext = nativeRuntimeContextFixture();
if (kind === "many-web-files") {
runtimeContext.skills = [];
for (const [name, count] of [["first", 462], ["second", 462], ["third", 3]] as const) {
const bundle = await materializeAsset(Array.from({ length: count }, (_, index) => ({
path: `file-${index}.md`, content: Buffer.from(`${name}-${index}\n`.padEnd(20_000, "x")), mode: 0o444,
})));
runtimeContext.skills.push({ key: name, runtimeName: name, versionId: null, bundle });
}
runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(runtimeContext);
} else if (kind !== "none") {
let versionId: string | null = null;
const files: CompanySkillVersionFileInventoryEntry[] = kind === "deleted"
? [{ path: "SKILL.md", kind: "skill", content: "# Historical acceptance skill\n\n".padEnd(281, "x") }]
: [{ path: "SKILL.md", kind: "skill", content: "# original immutable skill\n" },
{ path: "tool.sh", kind: "other" as const, content: "#!/bin/sh\nexit 0\n", executable: true }];
let projected = files;
if (kind === "repository") {
files[0]!.content = "---\nname: retained\ndescription: retained\n---\n# original immutable skill\n";
projected = repositorySkillProjection(files, "SKILL.md");
}
if (kind === "repository" || kind === "package" || kind === "deleted") {
const packageId = await skillPackageStore(db, storage).prepare(companyId, files, true);
const [skill] = await db.insert(companySkills).values({ companyId, key: "retained", slug: "retained", name: "Retained", markdown: projected[0]!.content }).returning();
const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skill!.id, revisionNumber: 1,
...(kind === "repository" ? { repositorySnapshotId: packageId, repositorySkillPath: "SKILL.md" } : { packageSnapshotId: packageId }), fileInventory: [] }).returning();
versionId = version!.id;
}
const bundle = await materializeAsset(projected.map(file => ({ path: file.path, content: skillFileBytes(file), mode: file.executable ? 0o555 : 0o444 })));
runtimeContext.skills = [{ key: "retained", runtimeName: "retained", versionId, bundle }];
runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(runtimeContext);
}
const execution = parseNativeExecutionInput({ schema: "paperclip.native-execution-input.v3", executionMode: "default", planningContext: null,
binding: { companyId, agentId, issueId, runId, executionWorkspaceId: managedWorkspace ? randomUUID() : runId }, provider: { kind: "codex", model: null },
task: { identifier: "FS-1", title: "Retain failure", description: null, prompt: "Retain failure", workMode: "standard" },
workspace: { cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: sessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
completionContract: { id: "contract", sha256: "saved", schemaVersion: "paperclip.completion-contract.v1",
contract: { revision: "1", objective: "Retain failure", criteria: [{ id: "retain", requirement: "Retain failure" }] } },
runtimeContext, interactionResponses: [], credentialBindings: [] });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issueId, nativeSessionId: sessionId, runnerInstanceId: runnerId,
runtimeMode: "native", status: "failed", finishedAt: new Date(), processPid: 1234, processStartedAt: new Date(0),
runnerProfileJson: { nativeExecutionInput: execution, nativeToolContractFingerprint: nativeToolContractFingerprintForTarget("remote", "codex"), retainedWarning: "provider resume remains failed", sessionCheckpoint: {
backendKind: "runner", driverKind: "codex_app_server", sessionId: "saved-provider-driver", providerSessionId: "saved-provider-session",
identity: { companyId, agentId, issueId, runId, sessionId }, activeTurnId: null, cursor: "2", goal: null,
} } });
await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId, phase: "terminal_failure", recoveryState: "blocked" });
const [selection] = await db.insert(agentTaskSessions).values({ companyId, agentId, adapterType: "paperclip_runner", taskKey: issueId,
lastRunId: runId, lastError: "existing provider failure" }).returning();
const scopeDigest = basename(scopedRunnerdStateRoot(execution)), leaseId = randomUUID(), providerLeaseId = `destroyed-${randomUUID()}`;
await db.insert(environmentLeases).values({ id: leaseId, companyId, issueId, heartbeatRunId: runId, provider: "daytona", providerLeaseId,
status: "released", cleanupStatus: "success", releasedAt: new Date(), metadata: {
nativeHarnessBackup: { schema: "paperclip.native-harness-backup-stamp.v2", normalizedSessionId: sessionId, runnerInstanceId: runnerId,
sessionScopeSha256: scopeDigest, sourceProviderLeaseId: providerLeaseId, authorizedProviderLeaseId: providerLeaseId,
manifestSha256: `sha256:${"f".repeat(64)}`, completedAt: new Date().toISOString() },
remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId, runId, leaseId, provider: "daytona", providerLeaseId,
state: "destroyed", confirmedAt: new Date().toISOString() },
} });
if (kind !== "none") {
if (kind === "unversioned") await preserveRuntimeSkillAssets(db, { companyId, runId }, storage);
await mkdir(join(scopedRunnerdStateRoot(execution), "codex-home/skills"), { recursive: true, mode: 0o700 });
for (const skill of runtimeContext.skills) {
await cp(skill.bundle.rootPath, join(scopedRunnerdStateRoot(execution), "codex-home/skills", skill.runtimeName), { recursive: true });
}
}
const input = { companyId, selectedSessionId: selection!.id, scopeDigest, source, storage, signal: AbortSignal.timeout(60_000) };
return { input, runId, issueId, agentId, leaseId, execution, witness, storage, mount, sourceRoot: scopedRunnerdStateRoot(execution) };
}
async function controllerFixture(managedWorkspace = false, kind: Parameters<typeof fixture>[0] = "unversioned", checkpointParts?: number) {
const f = await fixture(kind, managedWorkspace);
const completion = await ensureNativeCompletionContract({ db, companyId: f.input.companyId,
issue: { id: f.issueId, title: "Retain controller evidence", description: null }, actorId: f.agentId });
const execution = parseNativeExecutionInput({ ...f.execution, session: { ...f.execution.session, lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 } },
completionContract: { id: completion.row.id, sha256: completion.row.canonicalSha256,
schemaVersion: completion.row.schemaVersion, contract: completion.contract } });
const [prior] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const snapshot = prior!.runnerProfileJson!.sessionCheckpoint;
const descriptor = nativeControllerMigrationDescriptor(execution, nativeToolContractFingerprintForTarget("remote", "codex"));
const bootId = randomUUID(), leaseOwner = randomUUID();
await db.update(heartbeatRuns).set({ status: "running", finishedAt: null, completionContractId: completion.row.id,
completionContractSha256: completion.row.canonicalSha256 }).where(eq(heartbeatRuns.id, f.runId));
await db.update(nativeRunFinalizations).set({ phase: "observed", controllerBootId: bootId, controllerGeneration: 1,
leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000) }).where(eq(nativeRunFinalizations.runId, f.runId));
const identity = { runId: f.runId, normalizedSessionId: execution.session.normalizedSessionId!, runnerInstanceId: prior!.runnerInstanceId!,
environmentLeaseId: f.leaseId, turnId: randomUUID(), itemId: randomUUID() };
const control = new DurablePrpControlPlane({ stateDirectory: join(f.sourceRoot, "control-plane"), identity,
expectedRunnerVersion: "raw-evidence-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
for (const type of ["run.prepare", "session.open", "turn.start", "runner.suspend"]) {
Object.assign(control.queueCommand(type), { status: "completed", result: { status: "completed" } });
}
control.store.state.committedEvents.push({ sourceSeq: 1, sourceEventId: "raw-stream", eventType: "item.delta", priority: 1,
envelope: { ...identity, text: "Raw stream bytes absent from control-v1 records" }, deliveryCount: 1, logicalEffectCount: 1 },
{ sourceSeq: 2, sourceEventId: "raw-terminal", eventType: "run.terminal", priority: 0,
envelope: { ...identity }, deliveryCount: 1, logicalEffectCount: 1 });
control.store.state.ackedSourceSeq = 2;
if (checkpointParts) {
const artifact = { version: "raw-evidence-fixture", digest: `sha256:${"a".repeat(64)}` };
for (let count = 0; count < 512 && captureLegacyDurablePrpCheckpoint(JSON.stringify(control.store.state), identity, artifact).records.size < checkpointParts; count++) {
Object.assign(control.queueCommand("runner.suspend"), { status: "completed", result: { status: "completed" } });
}
expect(captureLegacyDurablePrpCheckpoint(JSON.stringify(control.store.state), identity, artifact).records.size).toBe(checkpointParts);
}
const raw = JSON.stringify(control.store.state);
control.setCheckpointPersistence(nativeRunnerCheckpointWriter(db, { companyId: f.input.companyId, issueId: f.issueId,
runId: f.runId, authorityRunId: f.runId, bootId, generation: 1, leaseOwner }));
await control.flushCheckpoint();
await control.stop();
const controllerPath = join(f.sourceRoot, "control-plane/control-plane-state.json");
await writeFile(controllerPath, raw, { mode: 0o600 });
const supervisor = JSON.stringify({ schema: "paperclip.native-session-supervisor.v1", configDigest: descriptor.configDigest,
processConfigDigest: null, snapshot });
await mkdir(join(descriptor.supervisorPath, ".."), { recursive: true, mode: 0o700 });
await writeFile(descriptor.supervisorPath, supervisor, { mode: 0o600 });
const [result] = await db.insert(nativeRunResults).values({ companyId: f.input.companyId, issueId: f.issueId, runId: f.runId,
completionContractId: completion.row.id, serverFingerprint: "raw-evidence-fixture", schemaStatus: "accepted", resultJson: {}, canonicalSha256: "a".repeat(64) }).returning();
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date(), runnerProfileJson: { ...prior!.runnerProfileJson,
nativeExecutionInput: execution, nativeControllerRecovery: createNativeControllerRecovery(snapshot as never,
{ configDigest: descriptor.configDigest, processConfigDigest: null },
{ runId: f.runId, runnerInstanceId: prior!.runnerInstanceId!, normalizedSessionId: execution.session.normalizedSessionId! }) } }).where(eq(heartbeatRuns.id, f.runId));
await db.update(nativeRunFinalizations).set({ phase: "committed", resultId: result!.id, recoveryState: null,
leaseOwner: null, leaseExpiresAt: null }).where(eq(nativeRunFinalizations.runId, f.runId));
return { ...f, execution, controllerPath, raw, supervisor, supervisorPath: descriptor.supervisorPath, identity };
}
it("reads the frozen pre-auth-refresh Codex WEB proof without changing authority", async () => {
const f = await controllerFixture(true, "none");
expect(nativeSha256(CODEX_WEB_PROFILE_8E4.directories)).toBe(CODEX_WEB_PROFILE_8E4_DIGEST);
const oldPolicy = vi.spyOn(nativeSessionExecutor, "resolveNativeHarnessPersistenceProfile").mockReturnValue(structuredClone(CODEX_WEB_PROFILE_8E4));
try { await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, mode: "web_evidence" }); }
finally { oldPolicy.mockRestore(); }
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
expect((before!.runnerProfileJson!.nativeHarnessFilesystemIndependence as { web: { profileDigest: string } }).web.profileDigest)
.toBe(CODEX_WEB_PROFILE_8E4_DIGEST);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(before);
await db.update(nativeRunFinalizations).set({ leaseOwner: randomUUID(), leaseExpiresAt: new Date(Date.now() + 60_000) })
.where(eq(nativeRunFinalizations.runId, f.runId));
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
});
it.each(["auth-refresh.json", "auth-refresh.json.tmp"])("refuses a legacy WEB archive that contains %s", async name => {
const f = await controllerFixture(true, "none");
await mkdir(join(f.sourceRoot, "codex-home"), { mode: 0o700 });
await writeFile(join(f.sourceRoot, "codex-home", name), "private credential fixture", { mode: 0o600 });
const oldPolicy = vi.spyOn(nativeSessionExecutor, "resolveNativeHarnessPersistenceProfile").mockReturnValue(structuredClone(CODEX_WEB_PROFILE_8E4));
try { await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, mode: "web_evidence" }); }
finally { oldPolicy.mockRestore(); }
const [row] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const web = (row!.runnerProfileJson!.nativeHarnessFilesystemIndependence as { web: { entries: Array<{ path: string }> } }).web;
expect(web.entries.some(entry => entry.path === `web-home/codex-home/${name}`)).toBe(true);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
expect(await readFile(join(f.sourceRoot, "codex-home", name), "utf8")).toBe("private credential fixture");
});
it.each(["extra-exclusion", "missing-existing-exclusion"])("refuses an unknown historical Codex WEB policy: %s", async change => {
const f = await controllerFixture(true, "none");
const exclusions = change === "extra-exclusion" ? [...CODEX_WEB_PROFILE_8E4.directories[1].excludeEntries, "unrecognized-cache"]
: CODEX_WEB_PROFILE_8E4.directories[1].excludeEntries.filter(entry => entry !== "auth.json");
const unknownPolicy = vi.spyOn(nativeSessionExecutor, "resolveNativeHarnessPersistenceProfile").mockReturnValue({
...CODEX_WEB_PROFILE_8E4, directories: [CODEX_WEB_PROFILE_8E4.directories[0], { ...CODEX_WEB_PROFILE_8E4.directories[1], excludeEntries: exclusions }],
});
try { await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, mode: "web_evidence" }); }
finally { unknownPolicy.mockRestore(); }
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
});
it("keeps current Codex WEB writers on the current credential exclusion policy", async () => {
const f = await controllerFixture(true, "none");
await mkdir(join(f.sourceRoot, "codex-home"), { mode: 0o700 });
await writeFile(join(f.sourceRoot, "codex-home/auth-refresh.json"), "private credential fixture", { mode: 0o600 });
await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, mode: "web_evidence" });
const [row] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const web = (row!.runnerProfileJson!.nativeHarnessFilesystemIndependence as { web: { profileDigest: string; excluded: Array<{ path: string }> } }).web;
expect(web.profileDigest).toBe("e5248e68685fb06ae32336d9517d7054e8aff785bdfb220b4481823a82ff8e78");
expect(web.excluded.some(entry => entry.path === "codex-home/auth-refresh.json")).toBe(true);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
});
it("validates retained WEB receipts with bounded parallel database proofs", async () => {
const fixtures = [];
for (let index = 0; index < 5; index++) {
const f = await controllerFixture(true, "none");
await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, mode: "web_evidence" });
fixtures.push(f);
}
const validate = filesystemIndependence.hasValidNativeHarnessFilesystemIndependence;
let active = 0, maximum = 0, completed = 0, release!: () => void;
const blocked = new Promise<void>(resolve => { release = resolve; });
const reader = vi.spyOn(filesystemIndependence, "hasValidNativeHarnessFilesystemIndependence").mockImplementation(async (...args) => {
active++; maximum = Math.max(maximum, active);
try { await blocked; const valid = await validate(...args); completed++; return valid; }
finally { active--; }
});
const scan = hasLocalNativeHarnessObligations(db);
try {
await vi.waitFor(() => expect(active).toBe(4));
expect(reader).toHaveBeenCalledTimes(4);
release();
expect(await scan).toBe(false);
expect(maximum).toBe(4); expect(completed).toBe(fixtures.length); expect(active).toBe(0);
expect(reader).toHaveBeenCalledTimes(fixtures.length);
} finally { release(); await scan; }
});
it.each(["invalid", "error"] as const)("joins started WEB proofs after an %s result without admitting more", async outcome => {
for (let index = 0; index < 5; index++) {
const f = await controllerFixture(true, "none");
await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, mode: "web_evidence" });
}
const validate = filesystemIndependence.hasValidNativeHarnessFilesystemIndependence;
let started = 0, active = 0, settled = false, releaseFirst!: () => void, releasePeers!: () => void, failed!: () => void;
const first = new Promise<void>(resolve => { releaseFirst = resolve; });
const peers = new Promise<void>(resolve => { releasePeers = resolve; });
const failure = new Promise<void>(resolve => { failed = resolve; });
const expected = new Error("fixture_independence_reader_failed");
vi.spyOn(filesystemIndependence, "hasValidNativeHarnessFilesystemIndependence").mockImplementation(async (...args) => {
const index = started++; active++;
try {
const valid = await validate(...args);
if (index === 0) { await first; failed(); if (outcome === "error") throw expected; return false; }
await peers; return valid;
} finally { active--; }
});
const scan = hasLocalNativeHarnessObligations(db);
void scan.then(() => { settled = true; }, () => { settled = true; });
try {
await vi.waitFor(() => expect(started).toBe(4));
releaseFirst(); await failure; await new Promise<void>(resolve => setImmediate(resolve));
expect(settled).toBe(false); expect(started).toBe(4); expect(active).toBe(3);
releasePeers();
if (outcome === "error") await expect(scan).rejects.toBe(expected);
else expect(await scan).toBe(true);
expect(active).toBe(0); expect(started).toBe(4);
} finally { releaseFirst(); releasePeers(); await scan.catch(() => undefined); }
});
it("preserves WEB-only evidence with a succeeded source and unresolved external lease", async () => {
const f = await controllerFixture(true, "deleted");
if (!("runtimeContext" in f.execution)) throw new Error("fixture runtime context missing");
const versionId = f.execution.runtimeContext.skills[0]!.versionId!;
await db.delete(companySkillVersions).where(eq(companySkillVersions.id, versionId));
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
const metadata = { ...lease!.metadata }; delete metadata.remoteExecutionTermination;
await db.update(environmentLeases).set({ metadata }).where(eq(environmentLeases.id, f.leaseId));
const beforeLeases = await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId));
const beforeRuns = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId));
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow("unverified");
const result = await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, mode: "web_evidence" });
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
expect(await hasLocalNativeHarnessObligations(db)).toBe(false);
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const receipt = run!.runnerProfileJson!.nativeHarnessFilesystemIndependence as { schema: string; controller: { archive: { id: string } } };
expect(receipt.schema).toBe("paperclip.native-harness-web-evidence.v1");
const [archive] = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, receipt.controller.archive.id));
const target = join(f.mount, "isolated-web-evidence");
expect(await restoreDurableNativeHarnessBackup({ db, storage: f.storage, companyId: f.input.companyId,
scopeDigest: archive!.scopeDigest, backupRoot: target })).toBe(true);
expect(await readFile(join(target, "current/control-plane-state.json"), "utf8")).toBe(f.raw);
expect(await readFile(join(target, "current/supervisor.json"), "utf8")).toBe(f.supervisor);
const original = join(f.sourceRoot, "codex-home/skills/retained/SKILL.md"), restored = join(target, "current/web-home/codex-home/skills/retained/SKILL.md");
expect(await readFile(restored)).toEqual(await readFile(original));
expect((await lstat(restored)).mode & 0o777).toBe(0o444);
expect((await readFile(restored)).length).toBe(281);
expect(await db.select().from(companySkillVersions).where(eq(companySkillVersions.id, versionId))).toEqual([]);
expect(await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.input.scopeDigest))).toEqual([]);
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId))).toEqual(beforeLeases);
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId))).toEqual(beforeRuns.map(r => ({ ...r,
runnerProfileJson: { ...r.runnerProfileJson, nativeHarnessFilesystemIndependence: run!.runnerProfileJson!.nativeHarnessFilesystemIndependence } })));
vi.stubEnv("PAPERCLIP_HOME", join(f.mount, "absent-web")); vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", join(f.mount, "absent-web/state"));
vi.stubEnv("RAILWAY_VOLUME_ID", ""); vi.spyOn(f.storage, "getObject").mockRejectedValue(new Error("replacement must not read storage"));
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
expect(result.receiptDigest).toMatch(/^[a-f0-9]{64}$/);
});
async function webFixture(checkpointParts?: number) {
const f = await controllerFixture(true, "unversioned", checkpointParts);
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
const metadata = { ...lease!.metadata }; delete metadata.remoteExecutionTermination;
await db.update(environmentLeases).set({ status: "retained", cleanupStatus: "failed", releasedAt: null, metadata })
.where(eq(environmentLeases.id, f.leaseId));
return { ...f, input: { ...f.input, mode: "web_evidence" as const } };
}
it("preserves all 927 local WEB skill files and validates their archive on a cold replacement", async () => {
const f = await controllerFixture(true, "many-web-files");
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
const metadata = { ...lease!.metadata }; delete metadata.remoteExecutionTermination;
await db.update(environmentLeases).set({ metadata }).where(eq(environmentLeases.id, f.leaseId));
const originalLeases = await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId));
const result = await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, mode: "web_evidence" });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const receipt = run!.runnerProfileJson!.nativeHarnessFilesystemIndependence as {
web: { entries: Array<{ path: string; type: string }> }; controller: { archive: { id: string } };
};
expect(receipt.web.entries.length).toBeGreaterThan(927);
const [archive] = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, receipt.controller.archive.id));
const target = join(f.mount, "cold-many-web-files");
expect(await restoreDurableNativeHarnessBackup({ db, storage: f.storage, companyId: f.input.companyId,
scopeDigest: archive!.scopeDigest, backupRoot: target })).toBe(true);
if (!("runtimeContext" in f.execution)) throw new Error("fixture runtime context missing");
const files: Array<{ relative: string; sha256: string }> = [];
for (const skill of f.execution.runtimeContext.skills) {
const manifest = JSON.parse(await readFile(join(f.mount, "instances/default/runtime-context-assets/manifests", `${skill.bundle.digest}.json`), "utf8"));
for (const file of manifest.files as Array<{ path: string; sha256: string }>) {
files.push({ relative: join("codex-home/skills", skill.runtimeName, file.path), sha256: file.sha256 });
}
}
expect(files).toHaveLength(927);
for (let offset = 0; offset < files.length; offset += 16) {
await Promise.all(files.slice(offset, offset + 16).map(async file => {
const restored = join(target, "current/web-home", file.relative);
const [originalBytes, restoredBytes, stat] = await Promise.all([
readFile(join(f.sourceRoot, file.relative)), readFile(restored), lstat(restored),
]);
expect(restoredBytes).toEqual(originalBytes);
expect(createHash("sha256").update(originalBytes).digest("hex")).toBe(file.sha256);
expect(stat.mode & 0o777).toBe(0o444);
}));
}
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId))).toEqual(originalLeases);
vi.stubEnv("PAPERCLIP_HOME", join(f.mount, "absent-many-web-files"));
vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", join(f.mount, "absent-many-web-files/state")); vi.stubEnv("RAILWAY_VOLUME_ID", "");
vi.spyOn(f.storage, "getObject").mockRejectedValue(new Error("database validation must not read storage"));
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
expect(result.receiptDigest).toMatch(/^[a-f0-9]{64}$/);
}, 120_000);
function webDescriptor(entries: NativeControllerHistoryManifestEntry[]) {
return nativeControllerEvidenceDescriptor({ companyId: "company", issueId: "issue", runId: "run", controllerDigest: null,
supervisorDigest: null, purpose: "filesystem_independence", web: { bindingDigest: "a".repeat(64), entries } });
}
it("keeps the existing 128-entry WEB manifest bytes unchanged", () => {
const entries: NativeControllerHistoryManifestEntry[] = [{ path: "web-home", mode: 0o700, type: "directory" },
...Array.from({ length: 127 }, (_, index) => ({ path: `web-home/entry-${index}`, mode: 0o700, type: "directory" as const }))];
const descriptor = webDescriptor(entries);
const manifest = JSON.stringify({ schema: "paperclip.legacy-controller-archive.v1", companyId: "company", issueId: "issue",
runId: "run", reason: "original_volume_source_evidence", files: {}, web: { entries, bindingDigest: "a".repeat(64), controllerAbsent: true } });
expect(descriptor.manifest).toBe(manifest);
expect(descriptor.manifestSha256).toBe(`sha256:${createHash("sha256").update(manifest).digest("hex")}`);
});
it.each(["entry-count", "metadata-bytes", "manifest-bytes", "path-bytes", "file-bytes", "total-bytes"])("refuses a WEB manifest beyond its %s bound", control => {
const entries: NativeControllerHistoryManifestEntry[] = [{ path: "web-home", mode: 0o700, type: "directory" }];
if (control === "entry-count") entries.push(...Array.from({ length: NATIVE_CONTROLLER_WEB_LIMITS.entries }, (_, index) => ({
path: `web-home/entry-${index}`, mode: 0o700, type: "directory" as const,
})));
if (control === "metadata-bytes") entries.push(...Array.from({ length: 1_050 }, (_, index) => ({
path: `web-home/${index}-${"x".repeat(3_990)}`, mode: 0o700, type: "directory" as const,
})));
if (control === "manifest-bytes") {
entries.push(...Array.from({ length: NATIVE_CONTROLLER_WEB_LIMITS.entries - 1 }, (_, index) => ({
path: `web-home/${index}-${"x".repeat(240)}`, mode: 0o444, type: "file" as const, size: 0, sha256: "b".repeat(64),
})));
expect(Buffer.byteLength(JSON.stringify(entries))).toBeLessThan(NATIVE_CONTROLLER_WEB_LIMITS.metadataBytes);
}
if (control === "path-bytes") entries.push({ path: `web-home/${"x".repeat(NATIVE_CONTROLLER_WEB_LIMITS.pathBytes)}`, mode: 0o700, type: "directory" });
if (control === "file-bytes") entries.push({ path: "web-home/oversized", mode: 0o600, type: "file", size: 16 * 1024 * 1024 + 1, sha256: "b".repeat(64) });
if (control === "total-bytes") entries.push(...Array.from({ length: 5 }, (_, index) => ({ path: `web-home/file-${index}`,
mode: 0o600, type: "file" as const, size: 16 * 1024 * 1024, sha256: "b".repeat(64) })));
expect(() => webDescriptor(entries)).toThrow("native_controller_archive_authority_invalid");
});
it.each(["entry-count", "file-bytes"])("retains oversized WEB filesystem evidence without publication for %s", async control => {
const f = await webFixture(), directory = join(f.sourceRoot, "codex-home/oversized");
await mkdir(directory, { mode: 0o700 });
if (control === "entry-count") {
for (let offset = 0; offset <= NATIVE_CONTROLLER_WEB_LIMITS.entries; offset += 32) {
await Promise.all(Array.from({ length: Math.min(32, NATIVE_CONTROLLER_WEB_LIMITS.entries + 1 - offset) }, (_, index) =>
writeFile(join(directory, `entry-${offset + index}`), "", { mode: 0o600 })));
}
} else {
const file = join(directory, "oversized"); await writeFile(file, "", { mode: 0o600 });
await truncate(file, 16 * 1024 * 1024 + 1);
}
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow("unverified");
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
expect(run!.runnerProfileJson!.nativeHarnessFilesystemIndependence).toBeUndefined();
expect(await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.runId, f.runId))).toEqual([]);
expect(await readFile(f.controllerPath, "utf8")).toBe(f.raw);
});
it("preserves a genuinely absent controller with its supervisor and 267 durable checkpoint parts", async () => {
const f = await webFixture(267);
await rm(f.controllerPath);
const before = await db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId));
expect(before).toHaveLength(267);
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const lastRunId = randomUUID(), execution = parseNativeExecutionInput({ ...f.execution, binding: { ...f.execution.binding, runId: lastRunId } });
await db.insert(heartbeatRuns).values({ ...source!, id: lastRunId, status: "failed", processPid: null, processStartedAt: null,
createdAt: new Date(source!.createdAt.getTime() + 1_000), runnerProfileJson: { nativeExecutionInput: execution, retainedWarning: "existing failed admission" } });
await db.insert(nativeRunFinalizations).values({ companyId: f.input.companyId, issueId: f.issueId, runId: lastRunId,
phase: "terminal_failure", recoveryState: "blocked" });
await db.update(agentTaskSessions).set({ lastRunId }).where(eq(agentTaskSessions.id, f.input.selectedSessionId));
await preserveNativeHarnessFilesystemIndependence(db, f.input);
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, lastRunId));
const receipt = run!.runnerProfileJson!.nativeHarnessFilesystemIndependence as { web: { controllerAbsent: boolean };
controller: { archive: { id: string; controllerDigest: null } } };
expect(receipt.web.controllerAbsent).toBe(true);
expect(receipt.controller.archive.controllerDigest).toBeNull();
const [archive] = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, receipt.controller.archive.id));
const target = join(f.mount, "isolated-supervisor-only");
expect(await restoreDurableNativeHarnessBackup({ db, storage: f.storage, companyId: f.input.companyId,
scopeDigest: archive!.scopeDigest, backupRoot: target })).toBe(true);
await expect(lstat(join(target, "current/control-plane-state.json"))).rejects.toMatchObject({ code: "ENOENT" });
expect(await readFile(join(target, "current/supervisor.json"), "utf8")).toBe(f.supervisor);
expect(await db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId))).toEqual(before);
vi.stubEnv("PAPERCLIP_HOME", join(f.mount, "absent-supervisor-host"));
vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", join(f.mount, "absent-supervisor-host/state")); vi.stubEnv("RAILWAY_VOLUME_ID", "");
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
});
it("keeps WEB coverage valid through external cleanup observations without changing recovery authority", async () => {
const f = await webFixture();
const result = await preserveNativeHarnessFilesystemIndependence(db, f.input);
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
await db.update(environmentLeases).set({ status: "expired", cleanupStatus: "pending", expiresAt: new Date(),
lastUsedAt: new Date(), releasedAt: new Date(), metadata: { ...lease!.metadata, sandboxState: "archived",
pendingCleanupRetryAfterMs: Date.now(), pendingCleanupManualAttemptId: randomUUID(),
pendingCleanupController: { attemptId: randomUUID(), countedAttempt: true }, remoteExecutionTermination: { observation: "not a stop receipt" } } })
.where(eq(environmentLeases.id, f.leaseId));
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
expect((await preserveNativeHarnessFilesystemIndependence(db, f.input)).receiptDigest).toBe(result.receiptDigest);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(1);
await db.update(environmentLeases).set({ providerLeaseId: "different-allocation" }).where(eq(environmentLeases.id, f.leaseId));
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
});
async function staleNativeFixture(checkpointParts?: number) {
const f = await controllerFixture(true, "unversioned", checkpointParts);
const first = await preserveNativeHarnessFilesystemIndependence(db, f.input);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
const [saved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const previous = saved!.runnerProfileJson!.nativeHarnessFilesystemIndependence as Record<string, unknown>;
const marker = { schema: LEGACY_WORKSPACE_RECOVERY_MIGRATION, operationId: randomUUID(),
workspaceId: f.execution.binding.executionWorkspaceId };
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...saved!.runnerProfileJson,
legacyWorkspaceRecoveryMigration: marker }, updatedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId));
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
return { ...f, first, previous };
}
it("reproves stale native facts after a legacy migration marker without replacing preserved evidence", async () => {
const f = await staleNativeFixture(453), { first, previous } = f;
const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId));
const leases = await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId));
const finalizers = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, f.input.companyId));
const results = await db.select().from(nativeRunResults).where(eq(nativeRunResults.companyId, f.input.companyId));
const parts = await db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId));
const backups = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId));
const packages = await db.select().from(companySkillRepositorySnapshots).where(eq(companySkillRepositorySnapshots.companyId, f.input.companyId));
const versions = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, f.input.companyId));
const put = vi.spyOn(f.storage, "putFile"), get = vi.spyOn(f.storage, "getObject");
const result = await preserveNativeHarnessFilesystemIndependence(db, f.input);
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const receipt = current!.runnerProfileJson!.nativeHarnessFilesystemIndependence as Record<string, unknown>;
expect(receipt).toEqual({ ...previous, factsDigest: expect.any(String), observedAt: expect.any(String) });
expect(receipt.factsDigest).not.toBe(previous.factsDigest);
expect(result.receiptDigest).not.toBe(first.receiptDigest);
expect(put).not.toHaveBeenCalled(); expect(get).toHaveBeenCalled();
expect(await readFile(f.controllerPath, "utf8")).toBe(f.raw);
expect(await readFile(f.supervisorPath, "utf8")).toBe(f.supervisor);
expect(parts).toHaveLength(453);
expect(await db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId))).toEqual(parts);
expect(await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId))).toEqual(backups);
expect(await db.select().from(companySkillRepositorySnapshots).where(eq(companySkillRepositorySnapshots.companyId, f.input.companyId))).toEqual(packages);
expect(await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, f.input.companyId))).toEqual(versions);
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId))).toEqual(leases);
expect(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, f.input.companyId))).toEqual(finalizers);
expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.companyId, f.input.companyId))).toEqual(results);
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId)))
.toEqual(runs.map(run => ({ ...run, runnerProfileJson: { ...run.runnerProfileJson, nativeHarnessFilesystemIndependence: receipt } })));
const audits = await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")));
expect(audits).toHaveLength(2);
const audit = audits.find(row => row.details?.previousReceiptDigest === first.receiptDigest)!.details!;
expect(nativeSha256({ ...receipt, ...audit.previousReceiptState as Record<string, unknown> })).toBe(first.receiptDigest);
expect((await preserveNativeHarnessFilesystemIndependence(db, f.input)).receiptDigest).toBe(result.receiptDigest);
expect(put).not.toHaveBeenCalled();
vi.stubEnv("PAPERCLIP_HOME", join(f.mount, "absent-native-reproof"));
vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", join(f.mount, "absent-native-reproof/state")); vi.stubEnv("RAILWAY_VOLUME_ID", "");
get.mockRejectedValue(new Error("DB-only validation must not read storage"));
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
}, 30_000);
it.each(["assets-binding", "raw-controller", "termination", "unknown-field"])(
"retains stale native proof when reproof rejects %s", async control => {
const f = await staleNativeFixture();
if (control === "raw-controller") {
const changed = f.raw.replace("Raw stream bytes absent from control-v1 records", "Changed raw streaming evidence");
expect(changed).not.toBe(f.raw); await writeFile(f.controllerPath, changed);
} else if (control === "termination") {
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
const metadata = { ...lease!.metadata }; delete metadata.remoteExecutionTermination;
await db.update(environmentLeases).set({ metadata }).where(eq(environmentLeases.id, f.leaseId));
} else {
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const receipt = { ...f.previous, ...(control === "assets-binding" ? { assetsDigest: "0".repeat(64) } : { unknownEvidence: "retain" }) };
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run!.runnerProfileJson,
nativeHarnessFilesystemIndependence: receipt } }).where(eq(heartbeatRuns.id, f.runId));
}
const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId));
const leases = await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId));
const backups = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId));
const put = vi.spyOn(f.storage, "putFile");
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow("unverified");
expect(put).not.toHaveBeenCalled();
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId))).toEqual(runs);
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId))).toEqual(leases);
expect(await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId))).toEqual(backups);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(1);
});
it.each(["matching-winner", "changed-receipt", "active-owner"])(
"guards stale native reproof publication after %s", async control => {
const f = await staleNativeFixture();
let release!: () => void, entered!: () => void, delay = true;
const blocked = new Promise<void>(resolve => { release = resolve; });
const ready = new Promise<void>(resolve => { entered = resolve; });
const get = f.storage.getObject.bind(f.storage), put = vi.spyOn(f.storage, "putFile");
vi.spyOn(f.storage, "getObject").mockImplementation(async (...args) => {
if (delay) { delay = false; entered(); await blocked; }
return get(...args);
});
const late = preserveNativeHarnessFilesystemIndependence(db, f.input);
try {
await ready;
const winner = control === "matching-winner" ? await preserveNativeHarnessFilesystemIndependence(db, f.input) : null;
if (control === "changed-receipt") {
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run!.runnerProfileJson,
nativeHarnessFilesystemIndependence: { ...f.previous, observedAt: new Date(0).toISOString() } } })
.where(eq(heartbeatRuns.id, f.runId));
}
if (control === "active-owner") await db.update(nativeRunFinalizations).set({ leaseOwner: "new-owner",
leaseExpiresAt: new Date(Date.now() + 60_000) }).where(eq(nativeRunFinalizations.runId, f.runId));
const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId));
const finalizers = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, f.input.companyId));
const backups = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId));
release();
if (winner) expect((await late).receiptDigest).toBe(winner.receiptDigest);
else await expect(late).rejects.toThrow("unverified");
expect(put).not.toHaveBeenCalled();
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId))).toEqual(runs);
expect(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, f.input.companyId))).toEqual(finalizers);
expect(await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId))).toEqual(backups);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(winner ? 2 : 1);
} finally { release(); await late.catch(() => undefined); }
});
async function staleWebFixture() {
const f = await webFixture(), first = await preserveNativeHarnessFilesystemIndependence(db, f.input);
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const previous = run!.runnerProfileJson!.nativeHarnessFilesystemIndependence as Record<string, unknown>;
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run!.runnerProfileJson, retainedWarning: "later preservation metadata" } })
.where(eq(heartbeatRuns.id, f.runId));
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
return { ...f, first, previous };
}
it.each([false, true])("refreshes stale WEB facts with the same verified source and archive while auditing the prior receipt (touched: %s)", async touched => {
const f = await staleWebFixture();
if (touched) { const stat = await lstat(f.supervisorPath); await utimes(f.supervisorPath, stat.atime, new Date(stat.mtimeMs + 5_000)); }
const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId));
const leases = await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId));
const finalizers = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, f.input.companyId));
const backups = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId));
const put = vi.spyOn(f.storage, "putFile");
const get = vi.spyOn(f.storage, "getObject");
const now = Date.now(), source = await verifyOriginalHarnessVolumeSource({ ...f.witness,
observedAt: new Date(now).toISOString(), expiresAt: new Date(now + 120_000).toISOString() });
expect(source.fingerprint).not.toBe(f.input.source.fingerprint);
const result = await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, source });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const receipt = run!.runnerProfileJson!.nativeHarnessFilesystemIndependence as Record<string, unknown>;
expect(receipt).toEqual({ ...f.previous, sourceCapabilityFingerprint: source.fingerprint,
factsDigest: expect.any(String), localFactsDigest: expect.any(String), observedAt: expect.any(String) });
if (touched) expect(receipt.localFactsDigest).not.toBe(f.previous.localFactsDigest);
expect(receipt.factsDigest).not.toBe(f.previous.factsDigest);
expect(result.receiptDigest).not.toBe(f.first.receiptDigest);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
expect(put).not.toHaveBeenCalled(); expect(get).toHaveBeenCalled();
expect(await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId))).toEqual(backups);
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId))).toEqual(leases);
expect(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, f.input.companyId))).toEqual(finalizers);
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId)))
.toEqual(runs.map(row => ({ ...row, runnerProfileJson: { ...row.runnerProfileJson, nativeHarnessFilesystemIndependence: receipt } })));
const audits = await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")));
expect(audits).toHaveLength(2);
expect(audits.find(row => row.details?.previousReceiptDigest === f.first.receiptDigest)?.details)
.toMatchObject({ previousReceiptDigest: f.first.receiptDigest, previousReceiptState: { factsDigest: f.previous.factsDigest,
localFactsDigest: f.previous.localFactsDigest, sourceCapabilityFingerprint: f.previous.sourceCapabilityFingerprint,
observedAt: f.previous.observedAt }, factsDigest: receipt.factsDigest });
const audit = audits.find(row => row.details?.previousReceiptDigest === f.first.receiptDigest)!.details!;
expect(nativeSha256({ ...receipt, ...audit.previousReceiptState as Record<string, unknown> })).toBe(f.first.receiptDigest);
expect((await preserveNativeHarnessFilesystemIndependence(db, f.input)).receiptDigest).toBe(result.receiptDigest);
expect(put).not.toHaveBeenCalled();
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(2);
});
it.each(["touch", "new-inode"])("preserves a valid WEB receipt after metadata-only %s", async control => {
const f = await webFixture(), first = await preserveNativeHarnessFilesystemIndependence(db, f.input);
const before = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const backups = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId));
const stat = await lstat(f.supervisorPath);
if (control === "touch") await utimes(f.supervisorPath, stat.atime, new Date(stat.mtimeMs + 5_000));
else { await copyFile(f.supervisorPath, `${f.supervisorPath}.copy`); await rename(`${f.supervisorPath}.copy`, f.supervisorPath);
expect((await lstat(f.supervisorPath)).ino).not.toBe(stat.ino); }
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
const put = vi.spyOn(f.storage, "putFile"), get = vi.spyOn(f.storage, "getObject");
expect((await preserveNativeHarnessFilesystemIndependence(db, f.input)).receiptDigest).toBe(first.receiptDigest);
expect(get).toHaveBeenCalled(); expect(put).not.toHaveBeenCalled();
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId))).toEqual(before);
expect(await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId))).toEqual(backups);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(1);
});
it("keeps prior WEB proof reconstructable when activity redaction masks a source path", async () => {
const f = await webFixture(), path = `codex-home/${userInfo().username}/notes.txt`;
await mkdir(join(f.sourceRoot, "codex-home", userInfo().username));
await writeFile(join(f.sourceRoot, path), "preserved local notes");
const first = await preserveNativeHarnessFilesystemIndependence(db, f.input);
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const previous = run!.runnerProfileJson!.nativeHarnessFilesystemIndependence as Record<string, unknown>;
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run!.runnerProfileJson, retainedWarning: "later metadata" } })
.where(eq(heartbeatRuns.id, f.runId));
const [settings] = await db.select().from(instanceSettings);
await db.update(instanceSettings).set({ general: { ...settings!.general, censorUsernameInLogs: true } }).where(eq(instanceSettings.id, settings!.id));
try {
await logActivity(db, { companyId: f.input.companyId, actorType: "system", actorId: "redaction-fixture",
action: "redaction_fixture", entityType: "heartbeat_run", entityId: f.runId, details: { path } });
const [redacted] = await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId), eq(activityLog.action, "redaction_fixture")));
expect(redacted!.details!.path).not.toBe(path);
await preserveNativeHarnessFilesystemIndependence(db, f.input);
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const receipt = current!.runnerProfileJson!.nativeHarnessFilesystemIndependence as Record<string, unknown>;
const audits = await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")));
const audit = audits.find(row => row.details?.previousReceiptDigest === first.receiptDigest)!.details!;
const reconstructed = audit.previousReceipt ?? { ...receipt, ...audit.previousReceiptState as Record<string, unknown> };
expect(nativeSha256(reconstructed)).toBe(first.receiptDigest);
expect(reconstructed).toEqual(previous);
expect(audit).not.toHaveProperty("previousReceipt");
expect(audit.previousReceiptState).toEqual({ factsDigest: previous.factsDigest,
localFactsDigest: previous.localFactsDigest, sourceCapabilityFingerprint: previous.sourceCapabilityFingerprint, observedAt: previous.observedAt });
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...current!.runnerProfileJson, retainedWarning: "another metadata update" } })
.where(eq(heartbeatRuns.id, f.runId));
await preserveNativeHarnessFilesystemIndependence(db, f.input);
const [latest] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const history = await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")));
expect(history).toHaveLength(3);
for (const row of history.filter(row => row.details?.previousReceiptState)) {
expect(nativeSha256({ ...latest!.runnerProfileJson!.nativeHarnessFilesystemIndependence as Record<string, unknown>,
...row.details!.previousReceiptState as Record<string, unknown> })).toBe(row.details!.previousReceiptDigest);
}
} finally { await db.update(instanceSettings).set({ general: settings!.general }).where(eq(instanceSettings.id, settings!.id)); }
});
it("refuses a WEB facts refresh that would drop unknown prior receipt fields", async () => {
const f = await staleWebFixture();
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const profile = { ...run!.runnerProfileJson, nativeHarnessFilesystemIndependence: { ...f.previous, unknownEvidence: "retain" } };
await db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId));
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow("native_harness_filesystem_independence_unverified");
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]!.runnerProfileJson).toEqual(profile);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(1);
});
it.each(["wrong-key", "unreadable-archive", "active-source"])("reverifies valid WEB coverage before idempotent preservation: %s", async control => {
const f = await webFixture(); await preserveNativeHarnessFilesystemIndependence(db, f.input);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
if (control === "wrong-key") vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "62".repeat(32));
if (control === "unreadable-archive") vi.spyOn(f.storage, "getObject").mockRejectedValue(new Error("archive unavailable"));
if (control === "active-source") {
const get = f.storage.getObject.bind(f.storage); let changed = false;
vi.spyOn(f.storage, "getObject").mockImplementation(async (...args) => {
if (!changed) { changed = true; await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) })
.where(eq(heartbeatRuns.id, f.runId)); }
return get(...args);
});
}
const before = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow();
const after = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
expect(after[0]!.runnerProfileJson).toEqual(before[0]!.runnerProfileJson);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(1);
});
it.each(["active-source", "changed-source", "changed-mode", "added-file", "wrong-key", "unreadable-archive", "aborted", "replaced-volume"])(
"retains a stale WEB receipt when fresh proof fails: %s", async control => {
const f = await staleWebFixture();
if (control === "active-source") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) })
.where(eq(heartbeatRuns.id, f.runId));
if (control === "changed-source") await writeFile(f.supervisorPath, `${f.supervisor} `);
if (control === "changed-mode") { const mode = (await lstat(f.supervisorPath)).mode & 0o777;
await chmod(f.supervisorPath, mode === 0o400 ? 0o600 : 0o400); }
if (control === "added-file") await writeFile(join(f.sourceRoot, "codex-home/uncovered.txt"), "new bytes");
if (control === "wrong-key") vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "62".repeat(32));
if (control === "unreadable-archive") vi.spyOn(f.storage, "getObject").mockRejectedValue(new Error("archive unavailable"));
if (control === "replaced-volume") vi.stubEnv("RAILWAY_VOLUME_ID", randomUUID());
const before = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const backups = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId));
const put = vi.spyOn(f.storage, "putFile");
await expect(preserveNativeHarnessFilesystemIndependence(db, { ...f.input,
...(control === "aborted" ? { signal: AbortSignal.abort() } : {}) })).rejects.toThrow();
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId))).toEqual(before);
expect(await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId))).toEqual(backups);
expect(put).not.toHaveBeenCalled();
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(1);
});
it.each(["matching-winner", "changed-receipt", "active-source", "changed-byte", "touched-source"])(
"guards stale WEB refresh publication after concurrent %s", async control => {
const f = await staleWebFixture();
let release!: () => void, entered!: () => void, delay = true;
const blocked = new Promise<void>(resolve => { release = resolve; }), ready = new Promise<void>(resolve => { entered = resolve; });
const get = f.storage.getObject.bind(f.storage);
vi.spyOn(f.storage, "getObject").mockImplementation(async (...args) => {
if (delay) { delay = false; entered(); await blocked; } return get(...args);
});
const late = preserveNativeHarnessFilesystemIndependence(db, f.input);
try {
await ready;
const winner = control === "matching-winner" ? await preserveNativeHarnessFilesystemIndependence(db, f.input) : null;
if (control === "changed-receipt") {
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run!.runnerProfileJson,
nativeHarnessFilesystemIndependence: { ...f.previous, observedAt: new Date().toISOString() } } }).where(eq(heartbeatRuns.id, f.runId));
}
if (control === "active-source") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) })
.where(eq(heartbeatRuns.id, f.runId));
if (control === "changed-byte") await writeFile(f.supervisorPath, `${f.supervisor} `);
if (control === "touched-source") { const stat = await lstat(f.supervisorPath);
await utimes(f.supervisorPath, stat.atime, new Date(stat.mtimeMs + 5_000)); }
const before = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
release();
if (winner) expect((await late).receiptDigest).toBe(winner.receiptDigest);
else await expect(late).rejects.toThrow();
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId))).toEqual(before);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(winner ? 2 : 1);
} finally { release(); await late.catch(() => undefined); }
});
it.each(["active-web", "owned-finalizer", "physical-writer", "foreign-supervisor", "missing-manifest", "symlink", "foreign-controller"])(
"refuses WEB evidence with %s and retains the original recovery state", async control => {
const f = await webFixture(), where = eq(heartbeatRuns.id, f.runId);
if (control === "active-web") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) }).where(where);
if (control === "owned-finalizer") await db.update(nativeRunFinalizations).set({ leaseOwner: "expired-but-still-owned", leaseExpiresAt: new Date(0) })
.where(eq(nativeRunFinalizations.runId, f.runId));
if (control === "physical-writer") { const [run] = await db.select().from(heartbeatRuns).where(where);
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run!.runnerProfileJson, nativeWorkspaceFinalizationOwner: { token: "owned" } } }).where(where); }
if (control === "foreign-supervisor") { const saved = JSON.parse(f.supervisor); saved.snapshot.identity.runId = randomUUID();
await writeFile(f.supervisorPath, JSON.stringify(saved)); }
if (control === "foreign-controller") { const saved = JSON.parse(f.raw); saved.identity.runId = randomUUID(); await writeFile(f.controllerPath, JSON.stringify(saved)); }
if (control === "missing-manifest") { if (!("runtimeContext" in f.execution)) throw new Error("fixture runtime context missing");
await rm(join(f.mount, "instances/default/runtime-context-assets/manifests", `${f.execution.runtimeContext.skills[0]!.bundle.digest}.json`)); }
if (control === "symlink") { const file = join(f.sourceRoot, "codex-home/history.jsonl"); await symlink(f.controllerPath, file); }
try {
const before = await db.select().from(heartbeatRuns).where(where);
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow();
expect(await db.select().from(heartbeatRuns).where(where)).toEqual(before);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
} finally { if (control === "symlink") await rm(join(f.sourceRoot, "codex-home/history.jsonl")); }
});
it.each(["changed-byte", "changed-mode", "added-file", "controller-appeared", "changed-source"])(
"does not publish WEB evidence after %s changes during encrypted upload", async control => {
const f = await webFixture();
const originalMode = (await lstat(f.supervisorPath)).mode & 0o777;
if (control === "controller-appeared") await rm(f.controllerPath);
const put = f.storage.putFile.bind(f.storage); let changed = false;
vi.spyOn(f.storage, "putFile").mockImplementation(async (...args) => {
const result = await put(...args);
if (!changed) { changed = true;
if (control === "changed-byte") await writeFile(f.supervisorPath, `${f.supervisor} `);
if (control === "changed-mode") await chmod(f.supervisorPath, originalMode === 0o400 ? 0o600 : 0o400);
if (control === "added-file") await writeFile(join(f.sourceRoot, "codex-home/history.jsonl"), "new uncovered bytes");
if (control === "controller-appeared") await writeFile(f.controllerPath, f.raw);
if (control === "changed-source") { const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run!.runnerProfileJson, retainedWarning: "new source binding" } }).where(eq(heartbeatRuns.id, f.runId)); }
} return result;
});
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow();
expect(changed).toBe(true);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]!.runnerProfileJson)
.not.toHaveProperty("nativeHarnessFilesystemIndependence");
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
});
it.each([false, true])("retains the authenticated WEB publication winner after a late upload (invalid: %s)", async invalid => {
const f = await webFixture();
let release!: () => void, entered!: () => void, delay = true;
const blocked = new Promise<void>(resolve => { release = resolve; }), ready = new Promise<void>(resolve => { entered = resolve; });
const get = f.storage.getObject.bind(f.storage);
vi.spyOn(f.storage, "getObject").mockImplementation(async (...args) => {
if (delay) { delay = false; entered(); await blocked; } return get(...args);
});
const late = preserveNativeHarnessFilesystemIndependence(db, f.input);
try {
await ready;
const winner = await preserveNativeHarnessFilesystemIndependence(db, f.input);
let [published] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
if (invalid) {
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...published!.runnerProfileJson,
nativeHarnessFilesystemIndependence: { ...(published!.runnerProfileJson!.nativeHarnessFilesystemIndependence as Record<string, unknown>), assetsDigest: "0".repeat(64) } } })
.where(eq(heartbeatRuns.id, f.runId));
[published] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
}
release();
if (invalid) await expect(late).rejects.toThrow("unverified");
else expect((await late).receiptDigest).toBe(winner.receiptDigest);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(published);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(1);
} finally { release(); await late.catch(() => undefined); }
});
it.each(["missing-entry", "changed-source"])("refuses existing WEB coverage after %s without replacing its receipt", async control => {
const f = await webFixture(); await preserveNativeHarnessFilesystemIndependence(db, f.input);
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const receipt = run!.runnerProfileJson!.nativeHarnessFilesystemIndependence as Record<string, unknown> & { web: { entries: Array<{ path: string }> } };
const profile = control === "missing-entry" ? { ...run!.runnerProfileJson, nativeHarnessFilesystemIndependence: { ...receipt,
web: { ...receipt.web, entries: receipt.web.entries.filter(entry => entry.path !== "web-supervisor.json") } } }
: { ...run!.runnerProfileJson, sessionCheckpoint: { changed: true } };
await db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId));
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]!.runnerProfileJson).toEqual(profile);
});
async function historyFixture(keepHistoricalHeaders = false, includeRunnerState = false) {
const f = await controllerFixture(true);
const get = f.storage.getObject.bind(f.storage);
const held = vi.spyOn(f.storage, "getObject").mockImplementation(async (...args) => {
if (args[1].includes("/skill-packages/")) throw new Error("fixture package verification held");
return get(...args);
});
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow("fixture package verification held");
held.mockRestore();
const [currentOnly] = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.runId, f.runId));
expect(currentOnly?.status).toBe("current");
const [anchor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
const entries: Array<{ path: string; contents: string; runId: string; header: typeof nativeRunnerCheckpoints.$inferSelect;
runner?: { path: string; contents: string } }> = [];
for (let index = 1; index <= 3; index++) {
const runId = randomUUID(), bootId = randomUUID(), leaseOwner = randomUUID(), leaseId = randomUUID();
const execution = parseNativeExecutionInput({ ...f.execution, binding: { ...f.execution.binding, runId } });
const snapshot = { ...(anchor!.runnerProfileJson!.sessionCheckpoint as Record<string, unknown>),
identity: { companyId: f.input.companyId, agentId: f.agentId, issueId: f.issueId, runId,
sessionId: execution.session.normalizedSessionId } };
await db.insert(heartbeatRuns).values({ ...anchor!, id: runId, status: "running", finishedAt: null,
createdAt: new Date(anchor!.createdAt.getTime() - index * 60_000),
runnerProfileJson: { ...anchor!.runnerProfileJson, nativeExecutionInput: execution, sessionCheckpoint: snapshot } });
await db.insert(nativeRunFinalizations).values({ companyId: f.input.companyId, issueId: f.issueId, runId,
phase: "observed", controllerBootId: bootId, controllerGeneration: 1, leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000) });
const identity = { ...f.identity, environmentLeaseId: execution.binding.executionWorkspaceId, runId, turnId: randomUUID(), itemId: randomUUID() };
let epoch = "epoch-" + createHash("sha256").update(JSON.stringify(identity)).digest("hex").slice(0, 24);
let path = join("authority-epochs", epoch, "control-plane/control-plane-state.json");
const control = new DurablePrpControlPlane({ stateDirectory: join(f.sourceRoot, "authority-epochs", epoch, "control-plane"),
identity, expectedRunnerVersion: "raw-evidence-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
for (const type of ["run.prepare", "session.open", "turn.start", "runner.suspend"]) {
Object.assign(control.queueCommand(type), { status: "completed", result: { status: "completed" } });
}
control.store.state.committedEvents.push({ sourceSeq: 1, sourceEventId: `retired-stream-${index}`, eventType: "item.delta", priority: 1,
envelope: { ...identity, text: `Retained historical stream ${index}` }, deliveryCount: 1, logicalEffectCount: 1 });
control.store.state.ackedSourceSeq = 1;
const contents = JSON.stringify(control.store.state);
control.setCheckpointPersistence(nativeRunnerCheckpointWriter(db, { companyId: f.input.companyId, issueId: f.issueId,
runId, authorityRunId: runId, bootId, generation: 1, leaseOwner }));
await control.flushCheckpoint(); await control.stop();
const canonicalEpoch = "epoch-" + createHash("sha256").update(JSON.stringify(JSON.parse(contents).identity)).digest("hex").slice(0, 24);
if (canonicalEpoch !== epoch) {
await rename(join(f.sourceRoot, "authority-epochs", epoch), join(f.sourceRoot, "authority-epochs", canonicalEpoch));
epoch = canonicalEpoch; path = join("authority-epochs", epoch, "control-plane/control-plane-state.json");
}
await writeFile(join(f.sourceRoot, path), contents, { mode: 0o600 });
const runner = includeRunnerState ? { path: join("authority-epochs", epoch, "runner-state.json"), contents: JSON.stringify({
schema: "paperclip.runner.durable.state.v1", ...JSON.parse(contents).identity, lifecycle: "suspended",
nextSourceSeq: 2, ackedSourceSeq: 1, lastControllerCommandSeq: 4, reconnectCount: 0,
outbox: [{ sourceSeq: 2, sourceEventId: `retired-outbox-${index}`, eventType: "item.delta",
payload: { text: `Suspended historical outbox ${index}` } }], processedCommands: {}, diagnostics: ["retained suspended authority"],
}) } : undefined;
if (runner) await writeFile(join(f.sourceRoot, runner.path), runner.contents, { mode: 0o600 });
const [result] = await db.insert(nativeRunResults).values({ companyId: f.input.companyId, issueId: f.issueId, runId,
completionContractId: anchor!.completionContractId!, serverFingerprint: "historical-evidence-fixture", schemaStatus: "accepted",
resultJson: {}, canonicalSha256: "a".repeat(64) }).returning();
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, runId));
await db.update(nativeRunFinalizations).set({ phase: "committed", resultId: result!.id, leaseOwner: null, leaseExpiresAt: null })
.where(eq(nativeRunFinalizations.runId, runId));
const [header] = await db.select().from(nativeRunnerCheckpoints).where(and(eq(nativeRunnerCheckpoints.runId, runId), eq(nativeRunnerCheckpoints.part, "header")));
expect(header).toBeDefined();
if (!keepHistoricalHeaders) await db.delete(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, runId));
const providerLeaseId = `destroyed-${randomUUID()}`;
await db.insert(environmentLeases).values({ ...lease!, id: leaseId, heartbeatRunId: runId, providerLeaseId,
metadata: { ...lease!.metadata, nativeHarnessBackup: { ...(lease!.metadata!.nativeHarnessBackup as Record<string, unknown>),
sourceProviderLeaseId: providerLeaseId, authorizedProviderLeaseId: providerLeaseId },
remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId: f.input.companyId, runId, leaseId,
provider: "daytona", providerLeaseId, state: "destroyed", confirmedAt: new Date().toISOString() } } });
entries.push({ path, contents, runId, header: header!, runner });
}
return { ...f, entries, currentOnly: currentOnly! };
}
it.each([false, true])("preserves a four-run session's three authority epochs with suspended Runner state %s and validates without the original volume", async includeRunnerState => {
const f = await historyFixture(false, includeRunnerState);
const beforeRuns = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId));
const beforeLeases = await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId));
for (const entry of f.entries) expect(await db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, entry.runId))).toEqual([]);
const result = await preserveNativeHarnessFilesystemIndependence(db, f.input);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
const [anchor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const receipt = anchor!.runnerProfileJson!.nativeHarnessFilesystemIndependence as { controller: { archive: { id: string } } };
const [archive] = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, receipt.controller.archive.id));
expect(archive!.id).not.toBe(f.currentOnly.id);
expect((await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, f.currentOnly.id)))[0]).toEqual(f.currentOnly);
const target = join(f.mount, "isolated-authority-evidence");
expect(await restoreDurableNativeHarnessBackup({ db, storage: f.storage, companyId: f.input.companyId,
scopeDigest: archive!.scopeDigest, backupRoot: target })).toBe(true);
for (const entry of f.entries) {
for (const file of [entry, ...(entry.runner ? [entry.runner] : [])]) {
expect(await readFile(join(target, "current", file.path), "utf8")).toBe(file.contents);
expect((await lstat(join(target, "current", file.path))).mode & 0o777).toBe(0o600);
expect(await readFile(join(f.sourceRoot, file.path), "utf8")).toBe(file.contents);
}
}
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId))).sort((a, b) => a.id.localeCompare(b.id)))
.toEqual(beforeRuns.map(run => run.id === f.runId ? { ...run, runnerProfileJson: { ...run.runnerProfileJson,
nativeHarnessFilesystemIndependence: anchor!.runnerProfileJson!.nativeHarnessFilesystemIndependence } } : run).sort((a, b) => a.id.localeCompare(b.id)));
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId))).toEqual(beforeLeases);
vi.stubEnv("PAPERCLIP_HOME", join(f.mount, "absent-replacement")); vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", join(f.mount, "absent-replacement/state"));
vi.stubEnv("RAILWAY_VOLUME_ID", ""); vi.spyOn(f.storage, "getObject").mockRejectedValue(new Error("replacement must not read storage"));
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
expect(result.receiptDigest).toMatch(/^[a-f0-9]{64}$/);
});
it.each(["active", "foreign-identity"])("retains historical Runner state without publication for %s", async control => {
const f = await historyFixture(false, true), runner = f.entries[0]!.runner!, value = JSON.parse(runner.contents);
if (control === "active") value.lifecycle = "active";
else value.runId = randomUUID();
const contents = JSON.stringify(value), path = join(f.sourceRoot, runner.path);
await writeFile(path, contents);
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow("native_harness_filesystem_independence_unverified");
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]!.runnerProfileJson)
.not.toHaveProperty("nativeHarnessFilesystemIndependence");
expect((await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, f.currentOnly.id)))[0]).toEqual(f.currentOnly);
expect(await readFile(path, "utf8")).toBe(contents);
expect((await lstat(path)).mode & 0o777).toBe(0o600);
});
it("bounds historical evidence at sixteen MiB per file and sixty-four MiB total", () => {
const limit = 16 * 1024 * 1024;
const descriptor = (sizes: number[]) => nativeControllerEvidenceDescriptor({ companyId: randomUUID(), issueId: randomUUID(),
runId: randomUUID(), controllerDigest: "a".repeat(64), supervisorDigest: null, purpose: "filesystem_independence",
historyBindingDigest: "b".repeat(64), history: [{ path: "authority-epochs", mode: 0o700, type: "directory" },
...sizes.flatMap((size, index) => {
const root = `authority-epochs/epoch-${index.toString(16).padStart(24, "0")}`;
return [{ path: root, mode: 0o700, type: "directory" as const },
{ path: `${root}/control-plane`, mode: 0o700, type: "directory" as const },
{ path: `${root}/control-plane/control-plane-state.json`, mode: 0o600, type: "file" as const, size, sha256: "c".repeat(64) }];
})] });
expect(() => descriptor([limit])).not.toThrow();
expect(() => descriptor([limit + 1])).toThrow("native_controller_archive_authority_invalid");
expect(() => descriptor([limit, limit, limit, limit])).not.toThrow();
expect(() => descriptor([limit, limit, limit, limit, 1])).toThrow("native_controller_archive_authority_invalid");
});
it("retains an oversized historical file and its existing archive without publication", async () => {
const f = await historyFixture(), path = join(f.sourceRoot, f.entries[0]!.path);
await truncate(path, 20 * 1024 * 1024);
const put = vi.spyOn(f.storage, "putFile");
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow("native_harness_filesystem_independence_unverified");
expect(put).not.toHaveBeenCalled();
expect((await lstat(path)).size).toBe(20 * 1024 * 1024);
expect((await lstat(path)).mode & 0o777).toBe(0o600);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]!.runnerProfileJson)
.not.toHaveProperty("nativeHarnessFilesystemIndependence");
expect((await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, f.currentOnly.id)))[0]).toEqual(f.currentOnly);
});
it.each(["foreign", "unknown", "active-owner", "changed-after-upload", "added-after-upload"])("retains authority history without publication for %s", async control => {
const f = await historyFixture(), path = join(f.sourceRoot, f.entries[0]!.path);
if (control === "foreign") {
const value = JSON.parse(f.entries[0]!.contents); value.identity.runId = randomUUID();
await writeFile(path, JSON.stringify(value));
}
if (control === "unknown") await writeFile(join(path, "..", "unknown.json"), "uncovered retained data");
if (control === "active-owner") await db.update(nativeRunFinalizations).set({ leaseOwner: "still-owned", leaseExpiresAt: new Date(0) })
.where(eq(nativeRunFinalizations.runId, f.entries[0]!.runId));
if (control.endsWith("after-upload")) {
const put = f.storage.putFile.bind(f.storage);
vi.spyOn(f.storage, "putFile").mockImplementation(async args => {
const result = await put(args);
if (control === "changed-after-upload") await writeFile(path, f.entries[0]!.contents + "\n");
else await writeFile(join(path, "..", "new-uncovered.json"), "new retained data");
return result;
});
}
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]!.runnerProfileJson)
.not.toHaveProperty("nativeHarnessFilesystemIndependence");
expect((await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, f.currentOnly.id)))[0]).toEqual(f.currentOnly);
expect((await lstat(path)).isFile()).toBe(true);
});
it("invalidates authority history coverage when a retained historical header disappears", async () => {
const f = await historyFixture(true);
await preserveNativeHarnessFilesystemIndependence(db, f.input);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
await db.delete(nativeRunnerCheckpoints).where(and(eq(nativeRunnerCheckpoints.runId, f.entries[0]!.runId), eq(nativeRunnerCheckpoints.part, "header")));
vi.stubEnv("PAPERCLIP_HOME", join(f.mount, "absent-replacement")); vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", join(f.mount, "absent-replacement/state"));
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
expect(await readFile(join(f.sourceRoot, f.entries[0]!.path), "utf8")).toBe(f.entries[0]!.contents);
});
it.each(["new-header", "receipt-identity", "missing-history"])("invalidates published authority history after %s changes", async control => {
const f = await historyFixture();
await preserveNativeHarnessFilesystemIndependence(db, f.input);
if (control === "new-header") await db.insert(nativeRunnerCheckpoints).values(f.entries[0]!.header);
else {
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const profile = structuredClone(run!.runnerProfileJson!);
const receipt = profile.nativeHarnessFilesystemIndependence as { controller: { history?: { epochs: Array<{ identity: { turnId: string }; identityDigest: string }> } } };
if (control === "missing-history") delete receipt.controller.history;
else {
receipt.controller.history!.epochs[0]!.identity.turnId = randomUUID();
receipt.controller.history!.epochs[0]!.identityDigest = nativeSha256(receipt.controller.history!.epochs[0]!.identity);
}
await db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId));
}
vi.stubEnv("PAPERCLIP_HOME", join(f.mount, "absent-replacement")); vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", join(f.mount, "absent-replacement/state"));
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
expect(await readFile(join(f.sourceRoot, f.entries[0]!.path), "utf8")).toBe(f.entries[0]!.contents);
});
it.each(["malformed", "retired", "unknown-override"])("refuses historical authority coverage with %s identity evidence", async control => {
const f = await historyFixture(control !== "unknown-override"), entry = f.entries[0]!;
if (control === "malformed") await db.update(nativeRunnerCheckpoints).set({ material: { schema: "unreadable" } })
.where(and(eq(nativeRunnerCheckpoints.runId, entry.runId), eq(nativeRunnerCheckpoints.part, "header")));
if (control === "retired") await db.update(nativeRunnerCheckpoints).set({ material: nativeCheckpointRetirement({ companyId: f.input.companyId,
agentId: f.agentId, sessionId: f.execution.session.normalizedSessionId!, runIds: [entry.runId] }) })
.where(and(eq(nativeRunnerCheckpoints.runId, entry.runId), eq(nativeRunnerCheckpoints.part, "header")));
if (control === "unknown-override") {
const value = JSON.parse(entry.contents); value.identity.environmentLeaseId = randomUUID();
await writeFile(join(f.sourceRoot, entry.path), JSON.stringify(value));
const previous = entry.path.split("/")[1]!, next = "epoch-" + createHash("sha256").update(JSON.stringify(value.identity)).digest("hex").slice(0, 24);
await rename(join(f.sourceRoot, "authority-epochs", previous), join(f.sourceRoot, "authority-epochs", next));
}
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]!.runnerProfileJson)
.not.toHaveProperty("nativeHarnessFilesystemIndependence");
expect((await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, f.currentOnly.id)))[0]).toEqual(f.currentOnly);
});
it("retains the first complete authority-history archive and receipt when an upload finishes late", async () => {
const f = await historyFixture();
let release!: () => void, entered!: () => void, delay = true;
const blocked = new Promise<void>(resolve => { release = resolve; }), ready = new Promise<void>(resolve => { entered = resolve; });
const put = f.storage.putFile.bind(f.storage);
vi.spyOn(f.storage, "putFile").mockImplementation(async args => {
if (delay) { delay = false; entered(); await blocked; }
return put(args);
});
const late = preserveNativeHarnessFilesystemIndependence(db, f.input);
void late.catch(() => undefined);
try {
await ready;
const winner = await preserveNativeHarnessFilesystemIndependence(db, f.input);
const [published] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
release(); expect((await late).receiptDigest).toBe(winner.receiptDigest);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(published);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(1);
} finally { release(); await late.catch(() => undefined); }
});
it("publishes only independence and validates on an empty replacement without the source capability", async () => {
const f = await fixture();
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
expect(await hasLocalNativeHarnessObligations(db)).toBe(true);
const [unrelated] = await db.insert(heartbeatRuns).values({ companyId: f.input.companyId, agentId: f.agentId, status: "running", runtimeMode: "native" }).returning();
await preserveNativeHarnessFilesystemIndependence(db, f.input);
expect(await hasLocalNativeHarnessObligations(db)).toBe(false);
const [after] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
expect(after).toEqual({ ...before, runnerProfileJson: { ...before!.runnerProfileJson,
nativeHarnessFilesystemIndependence: after!.runnerProfileJson!.nativeHarnessFilesystemIndependence } });
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)))[0]).toEqual(lease);
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
.toMatchObject({ phase: "terminal_failure", recoveryState: "blocked", leaseOwner: null });
expect(await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId))).toEqual([]);
expect(await readFile(join(f.sourceRoot, "codex-home/skills/retained/SKILL.md"), "utf8")).toBe("# original immutable skill\n");
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, unrelated!.id)))[0]!.status).toBe("running");
const replacement = await realpath(await mkdtemp(join(tmpdir(), "native-harness-replacement-"))); roots.push(replacement);
vi.stubEnv("PAPERCLIP_HOME", replacement); vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", join(replacement, "state"));
vi.stubEnv("RAILWAY_VOLUME_ID", ""); vi.stubEnv("RAILWAY_DEPLOYMENT_ID", randomUUID());
const get = vi.spyOn(f.storage, "getObject").mockRejectedValue(new Error("startup must not read objects"));
expect(await hasLocalNativeHarnessObligations(db)).toBe(false); expect(get).not.toHaveBeenCalled();
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow();
expect((await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified"))))).toHaveLength(1);
});
it.each(["history", "changed-skill", "backup-present", "unknown-provider", "owned-finalizer", "physical-writer", "storage-failure", "changed-binding"])("refuses %s without publishing or changing recovery", async control => {
const f = await fixture(), where = eq(heartbeatRuns.id, f.runId);
if (control === "history") await writeFile(join(f.sourceRoot, "codex-home/history.jsonl"), "retained conversation");
if (control === "changed-skill") { await chmod(join(f.sourceRoot, "codex-home/skills/retained"), 0o700); await writeFile(join(f.sourceRoot, "codex-home/skills/retained/extra"), "uncovered user bytes"); }
if (control === "backup-present") await mkdir(join(f.sourceRoot, "failover-backups/current"), { recursive: true });
if (control === "unknown-provider") await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, f.leaseId));
if (control === "owned-finalizer") await db.update(nativeRunFinalizations).set({ leaseOwner: "expired-but-owned", leaseExpiresAt: new Date(0) }).where(eq(nativeRunFinalizations.runId, f.runId));
if (control === "physical-writer") { const [r] = await db.select().from(heartbeatRuns).where(where); await db.update(heartbeatRuns).set({ runnerProfileJson: { ...r!.runnerProfileJson, nativeWorkspaceFinalizationOwner: { token: "still-owned" } } }).where(where); }
if (control === "storage-failure") vi.spyOn(f.storage, "getObject").mockRejectedValue(new Error("unavailable object"));
if (control === "changed-binding") { const get = f.storage.getObject.bind(f.storage); let changed = false;
vi.spyOn(f.storage, "getObject").mockImplementation(async (...args) => { const result = await get(...args); if (!changed) { changed = true;
const [row] = await db.select().from(heartbeatRuns).where(where); await db.update(heartbeatRuns).set({ runnerProfileJson: { ...row!.runnerProfileJson, retainedWarning: "newer warning" } }).where(where); } return result; }); }
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow();
expect((await db.select().from(heartbeatRuns).where(where))[0]!.runnerProfileJson).not.toHaveProperty("nativeHarnessFilesystemIndependence");
expect(await hasLocalNativeHarnessObligations(db)).toBe(true);
});
it.each(["checkpoint", "lease", "package", "retained-selection", "malformed-stamp"])("invalidates a published receipt after changed %s", async control => {
const f = await fixture(); await preserveNativeHarnessFilesystemIndependence(db, f.input);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
if (control === "checkpoint") { const [r] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...r!.runnerProfileJson, sessionCheckpoint: { changed: true } } }).where(eq(heartbeatRuns.id, f.runId)); }
if (control === "lease") await db.update(environmentLeases).set({ metadata: {} }).where(eq(environmentLeases.id, f.leaseId));
if (control === "package") await db.update(companySkillRepositorySnapshots).set({ storage: null }).where(eq(companySkillRepositorySnapshots.companyId, f.input.companyId));
if (control === "retained-selection") await db.update(agentTaskSessions).set({ lastRunId: null }).where(eq(agentTaskSessions.id, f.input.selectedSessionId));
if (control === "malformed-stamp") {
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
const stamp = { ...(lease!.metadata!.nativeHarnessBackup as Record<string, unknown>) }; delete stamp.sessionScopeSha256;
await db.update(environmentLeases).set({ metadata: { ...lease!.metadata, nativeHarnessBackup: stamp } }).where(eq(environmentLeases.id, f.leaseId));
expect(await hasLocalNativeHarnessObligations(db)).toBe(true);
}
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
});
it.each(["repository", "package"] as const)("uses the exact retained %s version rather than today's head", async kind => {
const f = await fixture(kind);
await preserveNativeHarnessFilesystemIndependence(db, f.input);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]!.runnerProfileJson)
.not.toHaveProperty("nativeRuntimeSkillPreservation");
await db.delete(companySkillVersions).where(eq(companySkillVersions.companyId, f.input.companyId));
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
});
it("does not publish across the actual physical-finalization lock", async () => {
const f = await fixture("none");
let release!: () => void, acquired!: () => void;
const held = new Promise<void>(resolve => { release = resolve; });
const ready = new Promise<void>(resolve => { acquired = resolve; });
const owner = db.transaction(async tx => { expect(await tryLockNativeWorkspaceFinalization(tx as unknown as typeof db,
{ companyId: f.input.companyId, runId: f.runId })).toBe(true); acquired(); await held; });
try { await ready; await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow("unverified"); }
finally { release(); await owner; }
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]!.runnerProfileJson)
.not.toHaveProperty("nativeHarnessFilesystemIndependence");
});
it("retains the first publication through concurrent attempts and an ordinary retry", async () => {
const f = await fixture();
const results = await Promise.allSettled([preserveNativeHarnessFilesystemIndependence(db, f.input), preserveNativeHarnessFilesystemIndependence(db, f.input)]);
expect(results.filter(result => result.status === "fulfilled").length).toBeGreaterThan(0);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
await preserveNativeHarnessFilesystemIndependence(db, f.input);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(before);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(1);
});
it.each([false, true])("retains or refuses the existing receipt when an earlier capture finishes late (invalid: %s)", async invalid => {
const f = await fixture();
let release!: () => void, entered!: () => void, delay = true;
const blocked = new Promise<void>(resolve => { release = resolve; });
const ready = new Promise<void>(resolve => { entered = resolve; });
const get = f.storage.getObject.bind(f.storage);
vi.spyOn(f.storage, "getObject").mockImplementation(async (...args) => {
if (delay) { delay = false; entered(); await blocked; }
return get(...args);
});
const late = preserveNativeHarnessFilesystemIndependence(db, f.input);
try {
await ready;
const winner = await preserveNativeHarnessFilesystemIndependence(db, f.input);
let [published] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
if (invalid) {
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...published!.runnerProfileJson,
nativeHarnessFilesystemIndependence: { ...(published!.runnerProfileJson!.nativeHarnessFilesystemIndependence as Record<string, unknown>), assetsDigest: "0".repeat(64) } } }).where(eq(heartbeatRuns.id, f.runId));
[published] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
}
release();
if (invalid) await expect(late).rejects.toThrow("unverified");
else expect((await late).receiptDigest).toBe(winner.receiptDigest);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(published);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(1);
} finally { release(); await late.catch(() => undefined); }
});
it("archives the successful source's raw controller and supervisor without changing execution authority", async () => {
const f = await controllerFixture();
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const [owner] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
const checkpoint = await loadNativeRunnerCheckpoint(db, { companyId: f.input.companyId, issueId: f.issueId, runId: f.runId });
const captured = captureLegacyDurablePrpCheckpoint(f.raw, { runId: f.runId, runnerInstanceId: before!.runnerInstanceId!,
normalizedSessionId: before!.nativeSessionId! }, { version: checkpoint!.runnerVersion, digest: checkpoint!.runnerDigest });
expect([...captured.records].map(([key, value]) => [key, JSON.parse(value)])).toEqual(
[...checkpoint!.records].map(([key, value]) => [key, JSON.parse(value)]));
await preserveNativeHarnessFilesystemIndependence(db, f.input);
const rows = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId));
expect(rows).toHaveLength(1); expect(rows[0]!.scopeDigest).not.toBe(f.input.scopeDigest);
expect(rows[0]!.status).toBe("current");
const restored = join(f.mount, "isolated-raw-evidence");
expect(await restoreDurableNativeHarnessBackup({ db, storage: f.storage, companyId: f.input.companyId,
scopeDigest: rows[0]!.scopeDigest, backupRoot: restored })).toBe(true);
expect(await readFile(join(restored, "current/control-plane-state.json"), "utf8")).toBe(f.raw);
expect(await readFile(join(restored, "current/supervisor.json"), "utf8")).toBe(f.supervisor);
expect(JSON.parse(await readFile(join(restored, "current/manifest.json"), "utf8")))
.toMatchObject({ schema: "paperclip.legacy-controller-archive.v1", reason: "original_volume_source_evidence" });
expect(await readFile(join(restored, "current/manifest.json"), "utf8")).toBe(JSON.stringify({ schema: "paperclip.legacy-controller-archive.v1",
companyId: f.input.companyId, issueId: f.issueId, runId: f.runId, reason: "original_volume_source_evidence",
files: { "control-plane-state.json": createHash("sha256").update(f.raw).digest("hex"),
"supervisor.json": createHash("sha256").update(f.supervisor).digest("hex") } }));
expect(await readFile(f.controllerPath, "utf8")).toBe(f.raw);
expect(await readFile(f.supervisorPath, "utf8")).toBe(f.supervisor);
const [after] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
expect((after!.runnerProfileJson!.nativeHarnessFilesystemIndependence as { controller: unknown }).controller).not.toHaveProperty("history");
expect(after).toEqual({ ...before, runnerProfileJson: { ...before!.runnerProfileJson,
nativeHarnessFilesystemIndependence: after!.runnerProfileJson!.nativeHarnessFilesystemIndependence } });
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(owner);
expect(await hasLocalNativeHarnessObligations(db)).toBe(false);
});
it.each(["codex-home", "opencode", "acpx"])("retains omitted %s history after switching to a preconfigured home", async directory => {
const f = await fixture("none");
const execution = parseNativeExecutionInput({ ...f.execution, schema: "paperclip.native-execution-input.v7", homeMode: "preconfigured",
provider: { kind: "codex", model: null, approvalPolicy: "never" } });
expect(scopedRunnerdStateRoot(execution)).toBe(f.sourceRoot);
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...before!.runnerProfileJson,
nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, f.runId));
const history = join(f.sourceRoot, directory, "history.json");
await mkdir(join(history, ".."), { recursive: true, mode: 0o700 });
await writeFile(history, "original provider history", { mode: 0o600 });
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow("unverified");
expect(await readFile(history, "utf8")).toBe("original provider history");
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]!.runnerProfileJson)
.not.toHaveProperty("nativeHarnessFilesystemIndependence");
expect(await hasLocalNativeHarnessObligations(db)).toBe(true);
});
it("permits a preconfigured home only when no omitted provider directory remains", async () => {
const f = await fixture("none");
const execution = parseNativeExecutionInput({ ...f.execution, schema: "paperclip.native-execution-input.v7", homeMode: "preconfigured",
provider: { kind: "codex", model: null, approvalPolicy: "never" } });
expect(scopedRunnerdStateRoot(execution)).toBe(f.sourceRoot);
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...before!.runnerProfileJson,
nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, f.runId));
await preserveNativeHarnessFilesystemIndependence(db, f.input);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
expect(await hasLocalNativeHarnessObligations(db)).toBe(false);
});
it("retains the first authenticated controller archive when a capture uploads late", async () => {
const f = await controllerFixture();
let release!: () => void, entered!: () => void, delay = true;
const blocked = new Promise<void>(resolve => { release = resolve; });
const ready = new Promise<void>(resolve => { entered = resolve; });
const put = f.storage.putFile.bind(f.storage);
vi.spyOn(f.storage, "putFile").mockImplementation(async args => {
const result = await put(args);
if (delay) { delay = false; entered(); await blocked; }
return result;
});
const late = preserveNativeHarnessFilesystemIndependence(db, f.input);
try {
await ready;
const winner = await preserveNativeHarnessFilesystemIndependence(db, f.input);
const [published] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const [archive] = await db.select().from(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.companyId, f.input.companyId),
eq(nativeHarnessBackups.status, "current")));
release();
expect((await late).receiptDigest).toBe(winner.receiptDigest);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(published);
expect(await db.select().from(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.companyId, f.input.companyId),
eq(nativeHarnessBackups.status, "current")))).toEqual([archive]);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
eq(activityLog.action, "native_controller_source_evidence_preserved")))).toHaveLength(1);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
} finally { release(); await late.catch(() => undefined); }
});
it.each(["checkpoint", "archive"])("invalidates preserved raw coverage when its %s binding changes", async binding => {
const f = await controllerFixture();
await preserveNativeHarnessFilesystemIndependence(db, f.input);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
if (binding === "checkpoint") await db.delete(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId));
else await db.update(nativeHarnessBackups).set({ status: "retired" }).where(eq(nativeHarnessBackups.companyId, f.input.companyId));
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
expect(await hasLocalNativeHarnessObligations(db)).toBe(true);
});
it.each(["missing-checkpoint", "foreign-controller", "active-source", "changed-after-upload"])("holds raw controller evidence for %s", async control => {
const f = await controllerFixture();
if (control === "missing-checkpoint") await db.delete(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId));
if (control === "foreign-controller") { const raw = JSON.parse(f.raw); raw.identity.runId = randomUUID(); await writeFile(f.controllerPath, JSON.stringify(raw)); }
if (control === "active-source") await db.update(heartbeatRuns).set({ status: "running", finishedAt: null }).where(eq(heartbeatRuns.id, f.runId));
if (control === "changed-after-upload") { const put = f.storage.putFile.bind(f.storage); vi.spyOn(f.storage, "putFile").mockImplementation(async args => {
const result = await put(args); await writeFile(f.controllerPath, f.raw + "\n"); return result;
}); }
await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]!.runnerProfileJson)
.not.toHaveProperty("nativeHarnessFilesystemIndependence");
const rows = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId));
expect(rows.every(row => row.status === "pending")).toBe(true);
});
});