fix(native): reprove stale filesystem preservation receipts (#20438)

## Thinking Path

> - Paperclip manages agent work and preserves saved native sessions
across releases.
> - Filesystem independence receipts bind database facts to preserved
controller and skill evidence.
> - A workspace migration can add run metadata without changing those
preserved bytes.
> - That change makes the saved facts digest stale. Full native
preservation then refuses the existing receipt, even after fresh source
checks.
> - WEB preservation already supports a full current proof and guarded
receipt refresh.
> - This pull request applies that pattern to full native receipts,
reuses the authenticated archive, and keeps the existing ownership and
destruction checks.
> - The benefit is a current receipt without a new archive or provider
replay.

## Linked Issues or Issue Description

Refs: #20322.

**What happened?**

Adding the canonical `legacyWorkspaceRecoveryMigration` marker to a
settled native run makes its saved receipt stale. The full native
preservation owner checks current evidence but then refuses the old
facts digest.

**Expected behavior**

The owner can refresh the proof metadata after it proves all current
facts and preserved bindings again. Changed evidence or authority must
leave the prior receipt intact.

**Steps to reproduce**

Use the added PostgreSQL regression. It saves a valid receipt for a
succeeded source with 453 checkpoint parts, adds the producer-shaped
migration marker, and calls preservation again. The unchanged owner
fails with `native_harness_filesystem_independence_unverified`.

## What Changed

- Reuse the prior controller archive through the existing authenticated
readback path.
- Require the old receipt to match every current immutable binding
before a facts refresh. Reconstruct its exact digest from unchanged
bindings and the prior proof fields.
- Recheck current database facts, assets, controller and local files
under the existing finalizer, selection, task and run fences.
- Publish with an exact prior-receipt comparison. Keep a valid
concurrent winner and refuse changed receipts or new owners.
- Audit the prior receipt digest and four proof fields. Add focused
source, integrity, race and DB-only validation tests, and update the
database guide.

## Verification

- Local Node24 canonical preflight and plugin SDK prerequisites passed.
The Runner build came from the content cache. After a normal merge of
fresh master, preflight confirmed unchanged build inputs.
- Genuine RED on `3040448da5`: the saved receipt is valid before the
migration marker and stale afterward. Reproof fails at the old receipt
guard (one failed, 102 intentionally not selected). Both retained RED
attempts reached that same guard.
- GREEN on final local merge `00f4b5d678083106afc88fb2f7e0588961b46d6d`:
19 PostgreSQL controls passed, 91 intentionally not selected (17.75s).
The wrapper completed cleanup.
- The positive preserves the same archive, controller bytes, assets,
local proof and all 453 checkpoint parts. It performs authenticated
object reads and zero archive PUTs. Runs, results, finalizers and leases
remain unchanged except for the receipt. The audit reconstructs the
exact prior receipt.
- DB-only validation passes with the original home, state directory and
volume absent, while storage reads throw.
- Refusal and race controls cover changed asset bindings, raw controller
bytes, missing destruction evidence, unknown receipt fields, a valid
concurrent winner, changed receipts and a late active owner. Existing
WEB refresh, physical finalization-lock and saved-binding controls also
pass.
- Server TypeScript passed, including the adjacent integration test.
Fresh Greptile is 5/5 on exact head
`00f4b5d678083106afc88fb2f7e0588961b46d6d`, with no review threads
([review](https://github.com/paperclipai/paperclip/pull/20438#issuecomment-6756548830)).
Final-head CI is terminal: 56 success, 4 skipped and 1 neutral. The
aggregate `ci / verify` passed
([run](https://github.com/paperclipai/paperclip/actions/runs/42737704110)).
- The first Runner-support job had one unrelated process-inspection
nonpass: `server-stop.test.ts` expected a successful wrapper close, but
its bounded process inspector returned no table. Its cause remains
unknown. The unchanged test, process helpers, workflow and dependency
inputs match the passed master control. One authorized same-head job
retry passed all 328 files / 6,927 tests (4 skipped); the exact
previously failed case passed in 665ms
([retry](https://github.com/paperclipai/paperclip/actions/runs/42737704110/job/129576420893)).
No timeout, assertion or source change was made for the retry.

```sh
node scripts/agent-preflight.mjs --skip-rust-debug
pnpm --filter @paperclipai/plugin-sdk ensure-build-deps
node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts -t "reproves stale native facts|retains stale native proof|guards stale native reproof|invalidates a published receipt after changed|actual physical-finalization lock|refreshes stale WEB facts|refuses a WEB facts refresh|retains or refuses the existing receipt when an earlier capture finishes late"
pnpm --filter @paperclipai/server exec tsc --noEmit
```

## Risks

- Old receipt schemas and immutable bindings remain required. Full fresh
proof carries compatible saved receipts forward; unknown fields or
changed evidence refuse refresh and preserve the old receipt.
- Full native preservation still requires canonical destroyed-provider
receipts. The change retains the existing source capability,
cancellation, finalizer and publication checks.
- Only proof metadata changes. No provider operation, archive
replacement, hosted recovery or deployment was performed for this PR.
Live qualification remains with the operator after delivery.

## Model Used

OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does
not expose a more specific model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

doc/DATABASE.md

@@ -122,8 +122,8 @@ original-volume anchor or filesystem read. These validations add bounded
 per-retained-scope database reads during startup. They do not add a worker or a
 poll. Missing or invalid byte evidence remains an enrollment blocker.
 
-The original-volume owner can refresh a WEB receipt whose database facts digest
+The original-volume owner can refresh a WEB or full native receipt whose database
-has become stale only when fresh source inspection, current-key checkpoint
+facts digest has become stale only when fresh source inspection, current-key checkpoint
 validation and ciphertext hash/size readback prove the preserved bindings unchanged.
 It reuses the same archive and takes the existing finalizer, task and run fences before
 comparing the prior receipt digest and replacing it. The company activity log
@@ -133,6 +133,10 @@ the exact prior proof from the unchanged current bindings before publication. No
 enter the audit delta; normal activity redaction remains enabled. Changed
 evidence, concurrent invalid receipts and active sources remain blocked; remote
 recovery obligations are unchanged.
+Full native reproof still requires canonical destroyed-provider receipts. A
+database-only legacy workspace migration marker can stale the facts digest while
+the preserved controller, skill assets and local byte proof remain unchanged;
+refresh authenticates that same object and does not upload a replacement archive.
 
 Local inode and timestamp observations may change between preservation attempts
 without changing byte coverage. Valid receipts remain idempotent after those

server/src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts

@@ -34,6 +34,7 @@ import { nativeCheckpointRetirement } from "./native-runner-checkpoint-retiremen
 import { nativeSha256 } from "./canonical.js";
 import { nativeControllerEvidenceDescriptor, NATIVE_CONTROLLER_WEB_LIMITS, type NativeControllerHistoryManifestEntry } from "./native-controller-archive.js";
 import { logActivity } from "../activity-log.js";
+import { LEGACY_WORKSPACE_RECOVERY_MIGRATION } from "../legacy-workspace-recovery-migration.js";
 
 vi.mock("../../build-commit.js", () => ({ readBuildCommit: () => "a".repeat(40) }));
 
@@ -389,6 +390,130 @@ describeEmbeddedPostgres("owner-verified native harness filesystem independence"
     expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
   });
 
+  async function staleNativeFixture(checkpointParts?: number) {
+    const f = await controllerFixture(true, "unversioned", checkpointParts);
+    const first = await preserveNativeHarnessFilesystemIndependence(db, f.input);
+    expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
+    const [saved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
+    const previous = saved!.runnerProfileJson!.nativeHarnessFilesystemIndependence as Record<string, unknown>;
+    const marker = { schema: LEGACY_WORKSPACE_RECOVERY_MIGRATION, operationId: randomUUID(),
+      workspaceId: f.execution.binding.executionWorkspaceId };
+    await db.update(heartbeatRuns).set({ runnerProfileJson: { ...saved!.runnerProfileJson,
+      legacyWorkspaceRecoveryMigration: marker }, updatedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId));
+    expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(false);
+    return { ...f, first, previous };
+  }
+
+  it("reproves stale native facts after a legacy migration marker without replacing preserved evidence", async () => {
+    const f = await staleNativeFixture(453), { first, previous } = f;
+    const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId));
+    const leases = await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId));
+    const finalizers = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, f.input.companyId));
+    const results = await db.select().from(nativeRunResults).where(eq(nativeRunResults.companyId, f.input.companyId));
+    const parts = await db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId));
+    const backups = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId));
+    const packages = await db.select().from(companySkillRepositorySnapshots).where(eq(companySkillRepositorySnapshots.companyId, f.input.companyId));
+    const versions = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, f.input.companyId));
+    const put = vi.spyOn(f.storage, "putFile"), get = vi.spyOn(f.storage, "getObject");
+    const result = await preserveNativeHarnessFilesystemIndependence(db, f.input);
+    const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
+    const receipt = current!.runnerProfileJson!.nativeHarnessFilesystemIndependence as Record<string, unknown>;
+    expect(receipt).toEqual({ ...previous, factsDigest: expect.any(String), observedAt: expect.any(String) });
+    expect(receipt.factsDigest).not.toBe(previous.factsDigest);
+    expect(result.receiptDigest).not.toBe(first.receiptDigest);
+    expect(put).not.toHaveBeenCalled(); expect(get).toHaveBeenCalled();
+    expect(await readFile(f.controllerPath, "utf8")).toBe(f.raw);
+    expect(await readFile(f.supervisorPath, "utf8")).toBe(f.supervisor);
+    expect(parts).toHaveLength(453);
+    expect(await db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId))).toEqual(parts);
+    expect(await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId))).toEqual(backups);
+    expect(await db.select().from(companySkillRepositorySnapshots).where(eq(companySkillRepositorySnapshots.companyId, f.input.companyId))).toEqual(packages);
+    expect(await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, f.input.companyId))).toEqual(versions);
+    expect(await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId))).toEqual(leases);
+    expect(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, f.input.companyId))).toEqual(finalizers);
+    expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.companyId, f.input.companyId))).toEqual(results);
+    expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId)))
+      .toEqual(runs.map(run => ({ ...run, runnerProfileJson: { ...run.runnerProfileJson, nativeHarnessFilesystemIndependence: receipt } })));
+    const audits = await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
+      eq(activityLog.action, "native_harness_filesystem_independence_verified")));
+    expect(audits).toHaveLength(2);
+    const audit = audits.find(row => row.details?.previousReceiptDigest === first.receiptDigest)!.details!;
+    expect(nativeSha256({ ...receipt, ...audit.previousReceiptState as Record<string, unknown> })).toBe(first.receiptDigest);
+    expect((await preserveNativeHarnessFilesystemIndependence(db, f.input)).receiptDigest).toBe(result.receiptDigest);
+    expect(put).not.toHaveBeenCalled();
+    vi.stubEnv("PAPERCLIP_HOME", join(f.mount, "absent-native-reproof"));
+    vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", join(f.mount, "absent-native-reproof/state")); vi.stubEnv("RAILWAY_VOLUME_ID", "");
+    get.mockRejectedValue(new Error("DB-only validation must not read storage"));
+    expect(await hasValidNativeHarnessFilesystemIndependence(db, f.input)).toBe(true);
+  }, 30_000);
+
+  it.each(["assets-binding", "raw-controller", "termination", "unknown-field"])(
+    "retains stale native proof when reproof rejects %s", async control => {
+      const f = await staleNativeFixture();
+      if (control === "raw-controller") {
+        const changed = f.raw.replace("Raw stream bytes absent from control-v1 records", "Changed raw streaming evidence");
+        expect(changed).not.toBe(f.raw); await writeFile(f.controllerPath, changed);
+      } else if (control === "termination") {
+        const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
+        const metadata = { ...lease!.metadata }; delete metadata.remoteExecutionTermination;
+        await db.update(environmentLeases).set({ metadata }).where(eq(environmentLeases.id, f.leaseId));
+      } else {
+        const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
+        const receipt = { ...f.previous, ...(control === "assets-binding" ? { assetsDigest: "0".repeat(64) } : { unknownEvidence: "retain" }) };
+        await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run!.runnerProfileJson,
+          nativeHarnessFilesystemIndependence: receipt } }).where(eq(heartbeatRuns.id, f.runId));
+      }
+      const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId));
+      const leases = await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId));
+      const backups = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId));
+      const put = vi.spyOn(f.storage, "putFile");
+      await expect(preserveNativeHarnessFilesystemIndependence(db, f.input)).rejects.toThrow("unverified");
+      expect(put).not.toHaveBeenCalled();
+      expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId))).toEqual(runs);
+      expect(await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.input.companyId))).toEqual(leases);
+      expect(await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId))).toEqual(backups);
+      expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
+        eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(1);
+    });
+
+  it.each(["matching-winner", "changed-receipt", "active-owner"])(
+    "guards stale native reproof publication after %s", async control => {
+      const f = await staleNativeFixture();
+      let release!: () => void, entered!: () => void, delay = true;
+      const blocked = new Promise<void>(resolve => { release = resolve; });
+      const ready = new Promise<void>(resolve => { entered = resolve; });
+      const get = f.storage.getObject.bind(f.storage), put = vi.spyOn(f.storage, "putFile");
+      vi.spyOn(f.storage, "getObject").mockImplementation(async (...args) => {
+        if (delay) { delay = false; entered(); await blocked; }
+        return get(...args);
+      });
+      const late = preserveNativeHarnessFilesystemIndependence(db, f.input);
+      try {
+        await ready;
+        const winner = control === "matching-winner" ? await preserveNativeHarnessFilesystemIndependence(db, f.input) : null;
+        if (control === "changed-receipt") {
+          const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
+          await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run!.runnerProfileJson,
+            nativeHarnessFilesystemIndependence: { ...f.previous, observedAt: new Date(0).toISOString() } } })
+            .where(eq(heartbeatRuns.id, f.runId));
+        }
+        if (control === "active-owner") await db.update(nativeRunFinalizations).set({ leaseOwner: "new-owner",
+          leaseExpiresAt: new Date(Date.now() + 60_000) }).where(eq(nativeRunFinalizations.runId, f.runId));
+        const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId));
+        const finalizers = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, f.input.companyId));
+        const backups = await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId));
+        release();
+        if (winner) expect((await late).receiptDigest).toBe(winner.receiptDigest);
+        else await expect(late).rejects.toThrow("unverified");
+        expect(put).not.toHaveBeenCalled();
+        expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.input.companyId))).toEqual(runs);
+        expect(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, f.input.companyId))).toEqual(finalizers);
+        expect(await db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.companyId, f.input.companyId))).toEqual(backups);
+        expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.input.companyId),
+          eq(activityLog.action, "native_harness_filesystem_independence_verified")))).toHaveLength(winner ? 2 : 1);
+      } finally { release(); await late.catch(() => undefined); }
+    });
+
   async function staleWebFixture() {
     const f = await webFixture(), first = await preserveNativeHarnessFilesystemIndependence(db, f.input);
     const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));

server/src/services/native-runtime/native-harness-filesystem-independence.ts

@@ -781,17 +781,27 @@ export async function preserveNativeHarnessFilesystemIndependence(db: Db, input:
   if (input.storage.provider !== "s3") fail();
   if (input.mode === "web_evidence") return preserveWebEvidence(db, input);
   const facts = await databaseFacts(db, input, false), assets = await assetBindings(db, facts.anchor, facts.source.id);
+  const previous = facts.anchor.runnerProfileJson?.[KEY] as Receipt | undefined;
+  const previousDigest = previous == null ? null : digest(previous);
+  if (previous != null && (previous.schema !== SCHEMA || !previous.controller)) fail();
   const history = await captureAuthorityHistory(db, facts, input.signal);
-  const controller = await preserveControllerEvidence(db, facts, input, history);
+  const controller = await preserveControllerEvidence(db, facts, input, history, undefined, previous?.controller);
   const local = await localFacts(facts, assets, input.storage, db, true, input.signal, controller), localFactsDigest = local.digest;
-  const previous = facts.anchor.runnerProfileJson?.[KEY];
   if (previous != null) {
-    if (!await hasValidNativeHarnessFilesystemIndependence(db, input)) fail();
+    if (!/^[a-f0-9]{64}$/.test(previous.factsDigest)
-    return { scopeDigest: input.scopeDigest, sourceRunId: facts.source.id, receiptDigest: digest(previous) };
+      || !matchingReceipt(previous, facts, input, assets.bindings, controller, undefined, previous.factsDigest)) fail();
+    if (matchingReceipt(previous, facts, input, assets.bindings, controller)) {
+      if (!await hasValidNativeHarnessFilesystemIndependence(db, input)) fail();
+      await assertOriginalHarnessVolumeSource(input.source); input.signal.throwIfAborted();
+      return { scopeDigest: input.scopeDigest, sourceRunId: facts.source.id, receiptDigest: previousDigest! };
+    }
   }
   const receipt: Receipt = { schema: SCHEMA, selectedSessionId: input.selectedSessionId, scopeDigest: input.scopeDigest,
     lastRunId: facts.anchor.id, sourceRunId: facts.source.id, factsDigest: facts.factsDigest,
     sourceCapabilityFingerprint: input.source.fingerprint, localFactsDigest, assetsDigest: digest(assets.bindings), ...assets.bindings, controller, observedAt: new Date().toISOString() };
+  const previousReceiptState = previous ? { factsDigest: previous.factsDigest, localFactsDigest: previous.localFactsDigest,
+    sourceCapabilityFingerprint: previous.sourceCapabilityFingerprint, observedAt: previous.observedAt } : null;
+  if (previousReceiptState && digest({ ...receipt, ...previousReceiptState }) !== previousDigest) fail();
   let publishedReceipt = receipt;
   const publications: ActivityPublication[] = [];
   await db.transaction(async transaction => {
@@ -815,16 +825,16 @@ export async function preserveNativeHarnessFilesystemIndependence(db: Db, input:
     const existing = current.anchor.runnerProfileJson?.[KEY];
     if (existing != null) {
       const winner = matchingReceipt(existing, current, input, currentAssets.bindings, currentController);
-      if (!winner) fail();
+      if (winner) { publishedReceipt = winner; return; }
-      publishedReceipt = winner;
-      return;
     }
+    if ((existing == null ? null : digest(existing)) !== previousDigest) fail();
     await tx.update(heartbeatRuns).set({ runnerProfileJson: { ...current.anchor.runnerProfileJson, [KEY]: receipt } })
       .where(and(eq(heartbeatRuns.id, current.anchor.id), eq(heartbeatRuns.companyId, input.companyId)));
     await logActivity(tx, { companyId: input.companyId, actorType: "system", actorId: "native-harness-preservation",
       action: "native_harness_filesystem_independence_verified", entityType: "heartbeat_run", entityId: current.anchor.id,
       agentId: current.anchor.agentId, runId: current.anchor.id, details: { scopeDigest: input.scopeDigest,
-        sourceRunId: current.source.id, sourceCapabilityFingerprint: receipt.sourceCapabilityFingerprint, factsDigest: receipt.factsDigest } }, publications);
+        sourceRunId: current.source.id, sourceCapabilityFingerprint: receipt.sourceCapabilityFingerprint, factsDigest: receipt.factsDigest,
+        ...(previousReceiptState ? { previousReceiptDigest: previousDigest, previousReceiptState } : {}) } }, publications);
   });
   for (const publication of publications) publishActivity(publication);
   return { scopeDigest: input.scopeDigest, sourceRunId: facts.source.id, receiptDigest: digest(publishedReceipt) };