fix: preserve compatible historical Codex WEB evidence (#20665)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native session migration preserves local evidence in encrypted
archives.
> - Each WEB receipt binds the exact provider directory and exclusion
policy.
> - The Codex credential refresh change added two excluded files and
changed that policy hash.
> - A new reader then rejected old receipts even when their facts,
files, skills, and authority still matched.
> - This pull request accepts the exact prior Codex policy while
checking archive paths against the current credential policy.
> - Operators can keep valid saved evidence across the upgrade without
copying it again.

## Linked Issues or Issue Description

Refs: #19814, #20646.

**What happened?**

A reader upgrade rejected saved Codex WEB evidence because the profile
hash changed when `auth-refresh.json` and `auth-refresh.json.tmp` became
excluded credentials. The other receipt bindings still matched. This
blocked the native harness migration gate.

**Expected behavior**

The exact known prior profile remains readable when the current archive
path checks accept every saved entry. An archive that includes either
newly excluded credential file must still fail. Unknown profiles and
changed authority must fail.

**Steps to reproduce**

1. Preserve WEB evidence with the frozen prior Codex profile and no
credential refresh files.
2. Read its unchanged receipt with the current profile.
3. Observe that the old reader returns false solely because the profile
hash changed.

**Paperclip version or commit**

Reproduced against `a3b34a9d88`; final branch is based on `e9343d8501`.

## What Changed

- Accept the exact prior Codex profile hash only when the complete
current profile has the reviewed successor hash and the provider is
Codex.
- Keep the current entry and excluded-path checks. Keep facts,
controller, archive, skill, and ownership checks.
- Add a frozen prior policy fixture with its literal hash. Test old
receipt reads, both credential files, unknown policies, changed
authority, and current writes.

## Verification

- Real PostgreSQL baseline: the frozen old-policy receipt test failed
because the reader returned false. The unchanged fixture passes with
this fix.
- All 18 selected PostgreSQL cases pass on final head after the master
rebase: six new cases and twelve existing WEB archive and authority
guards (14.93 seconds).
- Full layered server typecheck passes on final head after the rebase.
The module boundary check passes.
- A single read-only native-only candidate check on the deployed
dependency graph validated all 54 retained scopes. The 29 prior failures
passed, and the other 25 stayed valid. The check made no storage calls
or data writes. It did not replace final deployed enrollment acceptance.
- Final head `14112d0e3abb6e318a86fe9799d95bddd2dbf76b` passes all
[required CI
checks](https://github.com/paperclipai/paperclip/actions/runs/42919867108):
55 successful checks, four expected skips, and one neutral review. The
legacy security status also passes. Exact-head Greptile review is 5/5
with no open review threads.

## Risks

The compatibility branch accepts only the two reviewed policy hashes. A
future policy change can require another explicit compatibility
decision. The reader still rejects saved credential refresh files. It
does not rewrite receipts or change execution authority. No stored
schema or operator command changes.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window capacity are not exposed in this session. Used reasoning,
source editing, shell execution, and tool calls.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-10 17:22:08 -06:00
1 parent aa2a7be8b4
commit de7ceac0de
2 files changed
+76 -1

No files matched your search

@@ -16,6 +16,7 @@ import { materializeAsset } from "./runtime-context.js";
import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js";
import { preserveRuntimeSkillAssets } from "./runtime-skill-preservation.js";
import { nativeControllerMigrationDescriptor, scopedRunnerdStateRoot } from "./native-session-executor.js";
import * as nativeSessionExecutor from "./native-session-executor.js";
import { hasLocalNativeHarnessObligations } from "./native-harness-backup-migration.js";
import { preserveNativeHarnessFilesystemIndependence, hasValidNativeHarnessFilesystemIndependence } from "./native-harness-filesystem-independence.js";
import * as filesystemIndependence from "./native-harness-filesystem-independence.js";
@@ -39,6 +40,20 @@ import { LEGACY_WORKSPACE_RECOVERY_MIGRATION } from "../legacy-workspace-recover
vi.mock("../../build-commit.js", () => ({ readBuildCommit: () => "a".repeat(40) }));
const CODEX_WEB_PROFILE_8E4 = {
providerKind: "codex", driverKind: "codex_app_server",
directories: [
{ name: "runner", location: "runner", excludeEntries: [] },
{ name: "codex-home", location: "filesystem", excludeEntries: [
"tmp", ".tmp", "auth.json", "config.toml", ".remote-plugin-install-staging", "cache/remote_plugin_catalog",
"plugins/.remote-plugin-install-staging", "plugins/cache/openai-curated-remote", "plugins/cache/created-by-me-remote",
"plugins/cache/workspace-directory", "plugins/cache/workspace-shared-with-me",
"plugins/cache/workspace-shared-with-me-private", "plugins/cache/workspace-shared-with-me-unlisted",
] },
],
} as const;
const CODEX_WEB_PROFILE_8E4_DIGEST = "6f10c4b4424e84defcdb4c7e932921d09ea73dc0f39c96e61777c8b20c5f637b";
describeEmbeddedPostgres("owner-verified native harness filesystem independence", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>, db: ReturnType<typeof createDb>;
const roots: string[] = [];
@@ -198,6 +213,60 @@ describeEmbeddedPostgres("owner-verified native harness filesystem independence"
return { ...f, execution, controllerPath, raw, supervisor, supervisorPath: descriptor.supervisorPath, identity };
}
it("reads the frozen pre-auth-refresh Codex WEB proof without changing authority", async () => {
const f = await controllerFixture(true, "none");
expect(nativeSha256(CODEX_WEB_PROFILE_8E4.directories)).toBe(CODEX_WEB_PROFILE_8E4_DIGEST);
const oldPolicy = vi.spyOn(nativeSessionExecutor, "resolveNativeHarnessPersistenceProfile").mockReturnValue(structuredClone(CODEX_WEB_PROFILE_8E4));
try { await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, mode: "web_evidence" }); }
finally { oldPolicy.mockRestore(); }
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
expect((before!.runnerProfileJson!.nativeHarnessFilesystemIndependence as { web: { profileDigest: string } }).web.profileDigest)
.toBe(CODEX_WEB_PROFILE_8E4_DIGEST);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(before);
await db.update(nativeRunFinalizations).set({ leaseOwner: randomUUID(), leaseExpiresAt: new Date(Date.now() + 60_000) })
.where(eq(nativeRunFinalizations.runId, f.runId));
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
});
it.each(["auth-refresh.json", "auth-refresh.json.tmp"])("refuses a legacy WEB archive that contains %s", async name => {
const f = await controllerFixture(true, "none");
await mkdir(join(f.sourceRoot, "codex-home"), { mode: 0o700 });
await writeFile(join(f.sourceRoot, "codex-home", name), "private credential fixture", { mode: 0o600 });
const oldPolicy = vi.spyOn(nativeSessionExecutor, "resolveNativeHarnessPersistenceProfile").mockReturnValue(structuredClone(CODEX_WEB_PROFILE_8E4));
try { await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, mode: "web_evidence" }); }
finally { oldPolicy.mockRestore(); }
const [row] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const web = (row!.runnerProfileJson!.nativeHarnessFilesystemIndependence as { web: { entries: Array<{ path: string }> } }).web;
expect(web.entries.some(entry => entry.path === `web-home/codex-home/${name}`)).toBe(true);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
expect(await readFile(join(f.sourceRoot, "codex-home", name), "utf8")).toBe("private credential fixture");
});
it.each(["extra-exclusion", "missing-existing-exclusion"])("refuses an unknown historical Codex WEB policy: %s", async change => {
const f = await controllerFixture(true, "none");
const exclusions = change === "extra-exclusion" ? [...CODEX_WEB_PROFILE_8E4.directories[1].excludeEntries, "unrecognized-cache"]
: CODEX_WEB_PROFILE_8E4.directories[1].excludeEntries.filter(entry => entry !== "auth.json");
const unknownPolicy = vi.spyOn(nativeSessionExecutor, "resolveNativeHarnessPersistenceProfile").mockReturnValue({
...CODEX_WEB_PROFILE_8E4, directories: [CODEX_WEB_PROFILE_8E4.directories[0], { ...CODEX_WEB_PROFILE_8E4.directories[1], excludeEntries: exclusions }],
});
try { await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, mode: "web_evidence" }); }
finally { unknownPolicy.mockRestore(); }
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
});
it("keeps current Codex WEB writers on the current credential exclusion policy", async () => {
const f = await controllerFixture(true, "none");
await mkdir(join(f.sourceRoot, "codex-home"), { mode: 0o700 });
await writeFile(join(f.sourceRoot, "codex-home/auth-refresh.json"), "private credential fixture", { mode: 0o600 });
await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, mode: "web_evidence" });
const [row] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const web = (row!.runnerProfileJson!.nativeHarnessFilesystemIndependence as { web: { profileDigest: string; excluded: Array<{ path: string }> } }).web;
expect(web.profileDigest).toBe("e5248e68685fb06ae32336d9517d7054e8aff785bdfb220b4481823a82ff8e78");
expect(web.excluded.some(entry => entry.path === "codex-home/auth-refresh.json")).toBe(true);
expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
});
it("validates retained WEB receipts with bounded parallel database proofs", async () => {
const fixtures = [];
for (let index = 0; index < 5; index++) {
@@ -38,6 +38,8 @@ import { verifyDurableNativeHarnessBackup } from "./native-harness-backup-storag
const KEY = "nativeHarnessFilesystemIndependence";
const SCHEMA = "paperclip.native-harness-filesystem-independence.v1";
const WEB_SCHEMA = "paperclip.native-harness-web-evidence.v1";
const CODEX_WEB_PROFILE_BEFORE_AUTH_REFRESH = "6f10c4b4424e84defcdb4c7e932921d09ea73dc0f39c96e61777c8b20c5f637b";
const CODEX_WEB_PROFILE_WITH_AUTH_REFRESH = "e5248e68685fb06ae32336d9517d7054e8aff785bdfb220b4481823a82ff8e78";
const terminal = ["succeeded", "failed", "cancelled", "interrupted", "timed_out"];
const sha = (bytes: string | Buffer) => createHash("sha256").update(bytes).digest("hex");
const digest = (value: unknown) => nativeSha256(JSON.parse(JSON.stringify(value)));
@@ -495,8 +497,12 @@ function validWebBinding(facts: Awaited<ReturnType<typeof databaseFacts>>, web:
if (!web || typeof web.rootPresent !== "boolean" || typeof web.controllerAbsent !== "boolean" || !Array.isArray(web.entries)
|| !Array.isArray(web.skills) || !Array.isArray(web.excluded)
|| web.entries.length + web.excluded.length > NATIVE_CONTROLLER_WEB_LIMITS.entries
|| web.profileDigest !== digest(webProfile(facts)) || digest(web.skills) !== digest(webSkills(facts))) return false;
|| digest(web.skills) !== digest(webSkills(facts))) return false;
const profiles = webProfile(facts), manifests = new Map(web.skills.map(skill => [skill.bundleDigest, skill.manifestDigest]));
const profileDigest = digest(profiles);
if (web.profileDigest !== profileDigest && !(resolveNativeHarnessPersistenceProfile(facts.execution).providerKind === "codex"
&& profileDigest === CODEX_WEB_PROFILE_WITH_AUTH_REFRESH
&& web.profileDigest === CODEX_WEB_PROFILE_BEFORE_AUTH_REFRESH)) return false;
for (const [bundle, manifest] of manifests) if (!web.entries.some(entry => entry.path === `web-manifests/${bundle}.json`
&& entry.type === "file" && entry.sha256 === manifest)) return false;
if (web.entries.some(entry => {