fix(native): preserve pack proof across interrupted reattachment (#21246)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server can connect again to a live Runner after a restart.
> - Each new controller claim changes provider-pack admission to
pending.
> - The claim kept the previous verified proof only in local memory.
> - If recovery failed before its callback, the next claim lost that
proof.
> - This pull request saves a small reference to the verified proof with
the claim.
> - The next claim can check the same receipt without weakening
admission checks.

## Linked Issues or Issue Description

**What happened?**
A restart recovery attempt timed out before its admission callback. The
next attempt refused the saved pack with `source_state_pending`. It then
failed with `native_provider_artifacts_admission_unverified`.

**Expected behavior**
The next attempt can reuse the previous verified proof for the same
receipt, session, Runner, and lease. Current admission must remain
pending until its normal callback succeeds.

**Steps to reproduce**
1. Save verified provider-pack admission for a run.
2. Claim that run again. Commit the pending admission state.
3. Interrupt recovery before the session admission callback.
4. Claim the same run again and reconnect to the same Runner.
5. Observe that the old code refuses the saved receipt.

Related work: #20682 and #20850 address lock waits. #21085 adds refusal
logs. #21161 addresses command waits after restart. This change
addresses lost admission proof across interrupted claims. The original
transport timeout remains a separate investigation.

## What Changed

- The admission module prepares the claim patch and its recovery source.
- The claim saves a versioned reference with the source run, receipt
digest, and previous verified state. It adds no copy of provider files
or credentials.
- A later pending claim can use that reference only if the receipt is
unchanged. Normal company, agent, session, Runner, and lease checks
still apply.
- Missing proof, unknown state, a changed receipt, or a different Runner
stays refused. The callback clears the reference when it records its
decision.
- The executor uses this helper under its existing claim lock. Existing
artifact-retention and ownership checks stay in place.
- The execution guide states the previous-release behavior. Old verified
records seed the reference. Old pending records without a reference
remain refused. Older readers ignore the field and keep their existing
refusal behavior. No migration is needed.

## Verification

- The new interrupted-claim regression failed before the fix: admission
returned false instead of true.
- Focused admission, PostgreSQL fencing, and executor lease tests passed
after the rebase: 61 tests on commit c8123cc35d.
- The database case saves two interrupted claims, reads their persisted
state, checks a different Runner is refused, and publishes verified
admission for the original Runner.
- Negative cases cover missing proof, unknown state, a changed receipt,
a different source run, a different Runner, and a different lease.
Existing company and claim fences also pass.
- `pnpm verify` returned exit 2 because the execution guide requires
broader CI. Required CI must cover the final commit.
- Local server typecheck was killed with SIGKILL, including its
single-thread retry. This is an environment gap. Required CI must
provide typecheck and build results.
- Repository preflight installed dependencies and built shared packages.
It could not finish Runner preparation because cargo is absent.
- A live OpenCode restart workflow and recovery of the original task are
not verified. No task replay or deployment was performed.

## Risks

- The added profile field carries historical evidence. It does not mark
the current claim verified. Its use is limited to an unchanged receipt
with the normal identity checks.
- Existing pending or unknown incidents without this evidence cannot
gain proof from this fix. They keep their normal recovery path.
- No provider attempt limit, approval gate, workspace hold, or company
boundary changes.

## Model Used

OpenAI GPT-6, through Codex, with code search, tool use, and local test
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-13 12:56:57 -06:00
1 parent 9d9ee92d15
commit 6ad4529c5a
5 files changed
+113 -10

No files matched your search

+14
View File
@@ -6204,6 +6204,20 @@ can be accepted without a terminal acknowledgement. The selected run and its
history refresh after either outcome. Stop does not resume the provider or change
a manual task hold.
### Provider-pack admission during restart recovery
Provider-pack admission keeps a digest-bound historical proof when a controller
claim changes its current admission to pending. If a restart or a failed recovery
barrier interrupts that claim before its admission callback, the next same-run
claim can reuse that proof only for the unchanged receipt. Normal session,
company, agent, Runner and environment-lease checks still apply. Current artifact
retention remains blocked until the new claim publishes verified admission.
Unknown admission state discards the historical proof; a callback clears it when
it records its decision. This adds no background work or provider replay.
Previous-release verified receipts seed this proof on their first new claim.
Older pending or unknown records without it remain unverified, as before. Older
readers ignore the added profile field and retain their existing refusal behavior.
### Native session diagnostic ownership
Native session diagnostics follow the current execution's recorder epoch, including
@@ -5,7 +5,7 @@ import { agents, companies, createDb, heartbeatRuns, issues, nativeRunFinalizati
import type { NativeSession } from "../../vendor/paperclip-runner/index.js";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { canonicalProviderPackJson, type ProviderPackIdentity } from "./remote-provider-pack-verification.js";
import { admittedNativeProviderPack, nativeProviderPackAdmissionPending, recordNativeProviderPackAdmission,
import { admittedNativeProviderPack, nativeProviderPackAdmissionPending, prepareNativeProviderPackAdmission, recordNativeProviderPackAdmission,
rememberAdmittedNativeProviderPack, rememberRecoveredNativeProviderPack } from "./native-provider-pack-admission.js";
const hash = (value: string) => `sha256:${createHash("sha256").update(value).digest("hex")}`;
@@ -32,6 +32,41 @@ function receiptDb(receipt: unknown, state: unknown = verified): Db {
}
describe("provider pack admission identity", () => {
it("reattaches after a previous claim was interrupted before its admission callback", async () => {
const original = sessionFixture(); remember(original.session);
const receipt = admittedNativeProviderPack(original.session, initial)!;
const profile = { nativeProviderPackAdmission: receipt, nativeProviderPackAdmissionState: verified };
const first = prepareNativeProviderPackAdmission(initial.runId, profile, randomUUID());
// The claim commits, then snapshot/drain fails before the callback. The
// replacement controller has only persisted data, not the first's locals.
const pending = JSON.parse(JSON.stringify({ ...profile, ...first.patch }));
expect(pending.nativeProviderPackAdmissionState.state).toBe("pending");
const second = prepareNativeProviderPackAdmission(initial.runId, pending, randomUUID());
const recovered = sessionFixture();
await expect(rememberRecoveredNativeProviderPack(receiptDb(receipt, pending.nativeProviderPackAdmissionState), recovered.session,
{ identity: initial, sourceRunId: initial.runId, ...binding, source: second.source })).resolves.toBe(true);
expect(admittedNativeProviderPack(recovered.session, initial)).toEqual(receipt);
});
it.each(["unknown", "missing", "different receipt", "different run", "different runner", "different lease"])(
"does not reuse a pending recovery proof with %s", async change => {
const original = sessionFixture(); remember(original.session);
const receipt = admittedNativeProviderPack(original.session, initial)!;
const profile = { nativeProviderPackAdmission: receipt, nativeProviderPackAdmissionState: verified };
const first = prepareNativeProviderPackAdmission(initial.runId, profile, randomUUID());
const pending = JSON.parse(JSON.stringify({ ...profile, ...first.patch }));
if (change === "unknown") pending.nativeProviderPackAdmissionState.state = "unknown";
if (change === "missing") delete pending.nativeProviderPackAdmissionRecovery;
if (change === "different receipt") pending.nativeProviderPackAdmission.providerPackDigest = hash("replacement");
if (change === "different run") pending.nativeProviderPackAdmissionRecovery.runId = "other-run";
const next = prepareNativeProviderPackAdmission(initial.runId, pending, randomUUID());
const recovered = sessionFixture();
await expect(rememberRecoveredNativeProviderPack(receiptDb(null), recovered.session, {
identity: initial, sourceRunId: initial.runId, ...binding, source: next.source,
...(change === "different runner" ? { runnerInstanceId: "replacement" } : {}),
...(change === "different lease" ? { environmentLeaseId: "replacement" } : {}),
})).resolves.toBe(false);
expect(admittedNativeProviderPack(recovered.session, initial)).toBeNull();
});
it("does not publish selected configuration before successful session admission", async () => {
const { session } = sessionFixture();
expect(admittedNativeProviderPack(session, initial)).toBeNull();
@@ -179,6 +214,32 @@ describe("provider pack admission PostgreSQL fencing", () => {
await expect(recordNativeProviderPackAdmission(db, { ...f, claim: { ...f.claim, [key]: changed } as typeof f.claim })).resolves.toBe(false);
expect((await read(f.identity.runId)).runnerProfileJson).toEqual(f.profile);
});
it("republishes admission after two persisted interrupted claims without admitting a different Runner", async () => {
const f = await fixture();
await expect(recordNativeProviderPackAdmission(db, f)).resolves.toBe(true);
for (let i = 0; i < 2; i++) {
const prior = (await read(f.identity.runId)).runnerProfileJson!;
const claim = prepareNativeProviderPackAdmission(f.identity.runId, prior, randomUUID());
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...prior, ...claim.patch } }).where(eq(heartbeatRuns.id, f.identity.runId));
}
const prior = (await read(f.identity.runId)).runnerProfileJson!;
const claim = prepareNativeProviderPackAdmission(f.identity.runId, prior, randomUUID());
const recovered = sessionFixture(f.identity);
await expect(rememberRecoveredNativeProviderPack(db, recovered.session, {
identity: f.identity, sourceRunId: f.identity.runId, ...binding, source: claim.source,
runnerInstanceId: "replacement",
})).resolves.toBe(false);
await expect(rememberRecoveredNativeProviderPack(db, recovered.session, {
identity: f.identity, sourceRunId: f.identity.runId, ...binding, source: claim.source,
})).resolves.toBe(true);
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...prior, ...claim.patch } }).where(eq(heartbeatRuns.id, f.identity.runId));
await expect(recordNativeProviderPackAdmission(db, { ...f, session: recovered.session,
admissionEpoch: claim.patch.nativeProviderPackAdmissionState.admissionEpoch })).resolves.toBe(true);
const final = (await read(f.identity.runId)).runnerProfileJson!;
expect(final.nativeProviderPackAdmissionState).toMatchObject({ state: "verified" });
expect(final.nativeProviderPackAdmission).toEqual(prior.nativeProviderPackAdmission);
expect(final.nativeProviderPackAdmissionRecovery).toBeNull();
});
it("rejects an expired claim and a terminal run", async () => {
const f = await fixture();
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(0) }).where(eq(nativeRunFinalizations.runId, f.identity.runId));
@@ -41,11 +41,39 @@ export function nativeProviderPackAdmissionPending(admissionEpoch: string) {
return { schema: "paperclip.native-provider-pack-admission-state.v1" as const, state: "pending" as const, admissionEpoch };
}
/** Called under the executor's claim lock, before provider admission. */
export function prepareNativeProviderPackAdmission(runId: string, profile: Record<string, unknown> | null | undefined, admissionEpoch: string) {
const receipt = profile?.nativeProviderPackAdmission;
let state = profile?.nativeProviderPackAdmissionState;
const saved = profile?.nativeProviderPackAdmissionRecovery as Record<string, unknown> | null | undefined;
const receiptDigest = receipt && typeof receipt === "object" && !Array.isArray(receipt)
? `sha256:${createHash("sha256").update(canonicalProviderPackJson(receipt)).digest("hex")}` : null;
// A pending claim has not selected a different pack. It can carry forward
// the exact verified proof that the previous claim captured under its lock.
// Unknown state means selection was refused and must never regain that proof.
if (admissionState(state, "pending")
&& saved?.schema === "paperclip.native-provider-pack-admission-recovery.v1"
&& Object.keys(saved).sort().join(",") === "receiptDigest,runId,schema,state"
&& saved.runId === runId && receiptDigest !== null && saved.receiptDigest === receiptDigest
&& verifiedState(saved.state)) state = saved.state;
const recovery = receiptDigest !== null && verifiedState(state)
? { schema: "paperclip.native-provider-pack-admission-recovery.v1", runId, receiptDigest, state } : null;
return {
source: { runId, receipt, state, artifacts: profile?.nativeProviderArtifacts },
patch: { nativeProviderPackAdmissionState: nativeProviderPackAdmissionPending(admissionEpoch),
nativeProviderPackAdmissionRecovery: recovery },
};
}
function verifiedState(value: unknown): boolean {
return admissionState(value, "verified");
}
function admissionState(value: unknown, expected: "pending" | "verified"): boolean {
if (!value || typeof value !== "object" || Array.isArray(value)) return false;
const state = value as Record<string, unknown>;
return Object.keys(state).sort().join(",") === "admissionEpoch,schema,state"
&& state.schema === "paperclip.native-provider-pack-admission-state.v1" && state.state === "verified"
&& state.schema === "paperclip.native-provider-pack-admission-state.v1" && state.state === expected
&& typeof state.admissionEpoch === "string" && /^[a-f0-9-]{36}$/.test(state.admissionEpoch);
}
@@ -210,6 +238,7 @@ async function recordNativeProviderPackAdmissionOnce(db: Db, input: AdmissionInp
=== canonicalProviderPackJson({ ...receipt, inheritedFromRunId: null }));
if (verified && current.state === "verified") return true;
const patch = { ...(verified && previous == null ? { nativeProviderPackAdmission: receipt } : {}),
...(run.profile?.nativeProviderPackAdmissionRecovery != null ? { nativeProviderPackAdmissionRecovery: null } : {}),
nativeProviderPackAdmissionState: { ...nativeProviderPackAdmissionPending(input.admissionEpoch), state: verified ? "verified" : "unknown" } };
const saved = await tx.update(heartbeatRuns).set({
runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) || ${JSON.stringify(patch)}::jsonb`,
@@ -10683,9 +10683,10 @@ describe("native session execution lease fencing", () => {
});
await executePaperclipNativeSession({ db: leaseDb(execution, {}, {}, updates, profile), execution, runnerInstanceId: "runner" });
expect(state.execute).toHaveBeenCalledOnce();
expect(observed).toEqual({ nativeProviderPackAdmissionState: { schema: "paperclip.native-provider-pack-admission-state.v1",
expect(observed).toMatchObject({ nativeProviderPackAdmissionState: { schema: "paperclip.native-provider-pack-admission-state.v1",
state: "pending", admissionEpoch: expect.any(String) } });
expect(JSON.stringify(observed)).not.toContain(profile.nativeProviderPackAdmissionState.admissionEpoch);
expect((observed as unknown as Record<string, { admissionEpoch: string }>).nativeProviderPackAdmissionState.admissionEpoch)
.not.toBe(profile.nativeProviderPackAdmissionState.admissionEpoch);
expect(profile.nativeProviderPackAdmissionState.state).toBe("verified");
});
it("does not admit a session if mandatory claim invalidation fails", async () => {
@@ -45,7 +45,7 @@ import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
import { providerPackWorkspaceId, recordAdmittedRemoteProviderPack, resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js";
import { nativeProviderPackAdmissionFailureCode, nativeProviderPackAdmissionPending, rememberAdmittedNativeProviderPack, rememberRecoveredNativeProviderPack, recordNativeProviderPackAdmission, type NativeProviderPackAdmissionSource } from "./native-provider-pack-admission.js";
import { nativeProviderPackAdmissionFailureCode, prepareNativeProviderPackAdmission, rememberAdmittedNativeProviderPack, rememberRecoveredNativeProviderPack, recordNativeProviderPackAdmission, type NativeProviderPackAdmissionSource } from "./native-provider-pack-admission.js";
import { type ProviderPackInstallation, verifyRemoteProviderPack as verifyRemoteProviderPackIdentity } from "./remote-provider-pack-verification.js";
import { writeEnvironmentCleanup } from "../environment-cleanup-notifications.js";
import { NATIVE_SESSION_EXECUTION_LEASE_TTL_MS, lockNativeExecutionLeaseForReacquire, nativeExecutionLeaseHeldCondition, releaseUnstartedNativeExecutionClaim, startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal } from "./native-execution-lease.js";
@@ -11259,10 +11259,8 @@ async function executePaperclipNativeSessionAttempt(
.returning({ runId: nativeRunFinalizations.runId })
.then((rows) => rows[0] ?? null);
if (!claimed) throw new Error("native_session_lease_lost");
providerPackAdmissionSource = { runId: coordinator.runId,
receipt: boundRun.runnerProfileJson?.nativeProviderPackAdmission,
state: boundRun.runnerProfileJson?.nativeProviderPackAdmissionState,
artifacts: boundRun.runnerProfileJson?.nativeProviderArtifacts };
const providerPackAdmission = prepareNativeProviderPackAdmission(coordinator.runId, boundRun.runnerProfileJson, providerPackAdmissionEpoch);
providerPackAdmissionSource = providerPackAdmission.source;
controllerGeneration = nextControllerGeneration;
claimedStopCleanup = stopRecovery;
await tx
@@ -11270,7 +11268,7 @@ async function executePaperclipNativeSessionAttempt(
.set({
nativePhase: "observed",
nativePhaseUpdatedAt: leaseNow,
runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) || ${JSON.stringify({ nativeProviderPackAdmissionState: nativeProviderPackAdmissionPending(providerPackAdmissionEpoch),
runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) || ${JSON.stringify({ ...providerPackAdmission.patch,
...(claimedStartupContext ? { nativeStartupContext: { schema: "paperclip.native-startup-context.v1",
companyId: input.execution.binding.companyId, agentId: input.execution.binding.agentId, runId: input.execution.binding.runId,
contextId: claimedStartupContext.id, deadlineAtMs: claimedStartupContext.deadlineAtMs, leaseId: target?.kind === "remote" ? target.leaseId : null,