fix(lifecycle): never repeat a passed harness verification; settle a dead verifier's lease at once (#20030)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A new or reconfigured agent runs a harness verification (a hello
probe in a test sandbox) before it is ready, and its runs wait until it
passes
> - In the final Daytona restart chaos wave, a Codex agent's runs waited
about 417 s for their first sandbox, while boot took only about 32 s
> - Each control-plane restart killed the verifier. The next process
waited out the 120 s lifecycle lease and then booted a fresh sandbox.
Nothing recorded a configuration that had already passed.
> - This pull request records a passed verification by what makes it
valid and reuses it, and settles a dead verifier's lease on the shared
ownership path at once
> - The benefit is that a restart never repeats a passed verification,
and never leaves runs waiting out a dead process's lease

## Linked Issues or Issue Description

**What happened?**
In the final Daytona chaos wave, the Codex agent was created at 09:55:46
UTC and its three runs were queued. Verification ran four times, in four
sandboxes, because faults K1, K4 and K2 each killed the verifying
server. After K2 the server was ready at 10:00:20, but verification
restarted only at 10:02:13: it waited out the 120 s lifecycle lease. The
first run sandboxes were created at 10:02:43, 417 s after launch.

**Steps to reproduce**
1. Create a Codex agent and assign it a task at once, so the task waits
for harness verification.
2. Kill the control plane (SIGKILL) while the agent is verifying.
3. The restarted server waits out the dead verifier's 120 s lifecycle
lease, then verifies again in a new test target. The task's first token
arrives about 130 s after the fault.

**Expected behavior**
- A passed verification of the same effective configuration in the same
runner image is never repeated.
- A verification is never restarted from scratch while its holder
process is alive. When the holder is proven gone, its lease is settled
at once.
- Runs gate only on a configuration that has never passed.

**Root cause**
`verify()` (`server/src/services/agent-harness-verification.ts`) leased
a fresh sandbox on every attempt and recorded nothing on a pass. The
lifecycle lease (`modules/agent-lifecycle/adapters/postgres.ts`) named
no holder process, so a restart could only wait for it to expire.

## What Changed

- Restart chaos TTFT driver: a `verify` phase, reported in its own rows,
and a `--verify-fault K1|K2|K6` option. The fault fires 1 s after
`verifying` is observed, so it interrupts a held lease. The seed keeps
each agent's creation body, so the driver can hire an identical agent.

- **Persisted pass:** new columns `agents.harness_verified_digest` and
`harness_verified_at` (migration 0448). The digest is an HMAC, keyed
with the instance's decision-signing secret, over:
- the fully resolved effective configuration: every value the test used,
including `cwd`, resolved credentials, and agent-over-environment
precedence;
- the selected AI account and its credential generation (connection and
grant ids and `updatedAt`, never the credential);
  - the environment and the runner image.

The key means the stored value cannot confirm a guessed credential
offline. For a sandbox, the runner image is an immutable image digest or
a qualified Daytona snapshot ID (`sandboxBinding`, now exported). For
local runs it is the server version. A mutable image or snapshot name,
or a local environment redirected to the managed sandbox, never reuses a
pass.
- **Server-only:** `normalizeAgentBaseRow` and
`redactAgentRowForResponse` strip the digest and its timestamp from
agent API responses.
- `verify()` still runs every account, policy and secret check, then
returns complete on a matching digest without booting a test target. It
records the digest on a pass before the lifecycle step completes, so a
restart in between reuses it.
- **Lease holder:** the lifecycle `claim()` stamps the claiming
controller process on the lease (`lifecycleOperation.leaseHolder`,
supplied through the lifecycle effects port) and clears the stamp with
the lease. The compare-and-swap release lives in the lifecycle module
(`releaseAgentLifecycleLeaseOfHolderInTransaction`).
- **Shared ownership path:** a new `agent_lifecycle_lease` kind with
`settleAgentLifecycleLease`.
- When the reconciler proves the holder gone, settle clears exactly that
lease (compare-and-swap on the holder), notifies lifecycle work, and
logs `agent.lifecycle_lease_reconciled`.
- A live holder is never taken over. The lifecycle state, operation and
progress are untouched.

## Verification

- `restart-verification.test.ts` (embedded Postgres, real lifecycle
store and worker, real reconciler pass):
- A verification held by a dead process is claimed and completed at once
after the startup pass, with the agent ready in under 10 s instead of
waiting out the 120 s lease. It fails without the new kind.
  - A live holder is never taken over.
  - The claim stamps the holder, and `defer` clears it.
- `agent-harness-verification-preflight.test.ts`:
- A second `verify()` of the same configuration allocates no target. A
revoked account still fails before reuse. A changed model, `cwd` or
literal credential tests again. It fails without the reuse.
  - A failed verification records nothing.
- Ownership report, all lifecycle module tests, remote, provider-API and
unit harness verification tests, and crash-point call sites: 13 files,
220 tests pass.
- Targeted local run: the TTFT driver's new `verify` phase (`--phases
verify --verify-fault K1`, Codex, 2 samples) hires a new agent, assigns
it a task at once, kills the control plane (K1) while it verifies, and
times fault to first agent token:
- Before (master server code): 129.8 s and 129.8 s. The restarted server
waited out the dead verifier's 120 s lease. Run-log TTFT from run
creation was only 4.1 s.
- After (this branch): 15.8 s and 17.2 s. The startup reconciler pass
settled `agent_lifecycle_lease` and verification re-ran at once.

## Risks

- **A stale pass:** the digest covers every resolved value, the selected
account generation, the environment and an immutable runner image, so
any change tests again. Rotating the instance signing secret invalidates
every digest, so each agent verifies once more.
- **Migration:** two nullable columns, no backfill. Existing agents
verify once more, then reuse.
- **Settle:** it runs only on a `gone` proof, compare-and-swap on the
exact holder. It never changes the lifecycle state.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/modules/agent-lifecycle/__tests__/restart-verification.test.ts

@@ -0,0 +1,97 @@
+import { randomUUID } from "node:crypto";
+import { agents, companies, createDb, type Db } from "@paperclipai/db";
+import { eq, sql } from "drizzle-orm";
+import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
+import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js";
+import { createAgentLifecycle, configureAgentLifecycle, createAgentLifecycleEffects } from "../../../services/agent-lifecycle.js";
+import { startAgentLifecycle } from "../index.js";
+import { createLifecycleStore } from "../adapters/postgres.js";
+import type { LifecycleAgent } from "../application/ports.js";
+import { createOwnershipReportState, runOwnershipReconciliationPass } from "../../../services/recovery/ownership-invariant-report.js";
+import {
+  currentControllerProcessStamp,
+  currentProcessSpaceHash,
+  currentProcessStartedAtMs,
+} from "../../../services/controller-process-identity.js";
+
+const support = await getEmbeddedPostgresTestSupport();
+(support.supported ? describe : describe.skip)("harness verification across a control-plane restart", () => {
+  let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
+  let db: Db;
+  let companyId: string;
+  const workers: Array<{ stop(): Promise<void> }> = [];
+  beforeAll(async () => {
+    database = await startEmbeddedPostgresTestDatabase("paperclip-restart-verification-");
+    db = createDb(database.connectionString, { maxConnections: 4 });
+  }, 90_000);
+  afterAll(async () => { await database?.cleanup(); });
+  beforeEach(async () => {
+    companyId = randomUUID();
+    await db.insert(companies).values({ id: companyId, name: "Restart verification", issuePrefix: `R${companyId.slice(0, 7)}` });
+  });
+  afterEach(async () => {
+    for (const worker of workers.splice(0)) await worker.stop();
+    await db.update(agents).set({ lifecycleState: "terminated", status: "terminated" }).where(eq(agents.companyId, companyId));
+  });
+  const current = async (id: string) => (await db.select().from(agents).where(eq(agents.id, id)))[0]!;
+
+  /** An agent whose verification a controller claimed and was running when the controller died. */
+  async function verifyingHeldBy(holder: Record<string, unknown>) {
+    const runHost = vi.fn(async (agent: LifecycleAgent) => agent.lifecycleState === "verifying" ? "pending" as const : "complete" as const);
+    const before = configureAgentLifecycle(db, { requiredPluginIds: async () => [], runHost, runPlugin: async () => "complete" as const });
+    const agent = await createAgentLifecycle(db).requestHire(companyId, { name: "Verifier", adapterType: "process", adapterConfig: { command: "true" } });
+    await before.process(agent.id);
+    await before.stop();
+    expect((await current(agent.id)).lifecycleState).toBe("verifying");
+    await db.update(agents).set({ lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt'` }).where(eq(agents.id, agent.id));
+    const store = createLifecycleStore(db, createAgentLifecycleEffects());
+    expect(await store.claim(agent.id, "verifier-before-restart", new Date())).not.toBeNull();
+    await db.update(agents).set({ lifecycleOperation: sql`${agents.lifecycleOperation} || ${JSON.stringify({ leaseHolder: holder })}::jsonb` })
+      .where(eq(agents.id, agent.id));
+    return agent.id;
+  }
+
+  /** The restarted process: one startup reconciliation pass, then its lifecycle sweep. */
+  async function restart() {
+    const verify = vi.fn(async (agent: LifecycleAgent) => { void agent; return "complete" as const; });
+    const restarted = startAgentLifecycle(db, createAgentLifecycleEffects(),
+      { requiredPluginIds: async () => [], runHost: verify, runPlugin: async () => "complete" as const }, () => true);
+    workers.push(restarted);
+    const startedAt = performance.now();
+    await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
+    await restarted.sweep();
+    return { verify, queuedMs: performance.now() - startedAt };
+  }
+
+  it("verifies again at once when the verifier's process is proven gone, not after the lease", async () => {
+    const id = await verifyingHeldBy({ bootId: randomUUID(), pid: 2_000_000_001, hostHash: currentProcessSpaceHash(),
+      processStartedAt: new Date(Date.now() - 60_000).toISOString() });
+    const { verify, queuedMs } = await restart();
+    expect(verify).toHaveBeenCalledOnce();
+    // Runs gate on readiness; before this change they waited out the 120 s lease.
+    expect((await current(id)).lifecycleState).toBe("ready");
+    expect(queuedMs).toBeLessThan(10_000);
+  });
+
+  it("never takes over a verification whose process is alive", async () => {
+    const id = await verifyingHeldBy({ ...currentControllerProcessStamp(), bootId: randomUUID(), pid: process.pid,
+      processStartedAt: new Date((await currentProcessStartedAtMs())!).toISOString() });
+    const before = await current(id);
+    const { verify } = await restart();
+    expect(verify).not.toHaveBeenCalled();
+    const after = await current(id);
+    expect(after.lifecycleState).toBe("verifying");
+    expect(after.lifecycleOperation?.leaseOwner).toBe(before.lifecycleOperation?.leaseOwner);
+  });
+
+  it("stamps the claiming process on the lease and clears it with the lease", async () => {
+    const id = await verifyingHeldBy(currentControllerProcessStamp() as unknown as Record<string, unknown>);
+    const store = createLifecycleStore(db, createAgentLifecycleEffects());
+    await db.update(agents).set({ lifecycleOperation: sql`${agents.lifecycleOperation} - 'leaseOwner' - 'leaseUntil' - 'leaseHolder'` })
+      .where(eq(agents.id, id));
+    const claimed = await store.claim(id, "owner", new Date());
+    expect(claimed?.lifecycleOperation?.leaseHolder).toMatchObject({ pid: process.pid, bootId: currentControllerProcessStamp().bootId });
+    expect(await store.defer(claimed!, "owner", null, new Date())).toBe(true);
+    expect((await current(id)).lifecycleOperation).not.toHaveProperty("leaseHolder");
+  });
+});

server/src/modules/agent-lifecycle/adapters/configuration.ts

@@ -48,7 +48,7 @@ export async function updateAgentConfiguration(tx: Db, effects: LifecycleEffects
     ...(!prepareSetup && current.lifecycleHolds.includes("setup") && ["preparing", "verifying"].includes(current.lifecycleState)
       ? { lifecycleState: "pausing" as const } : {}),
     lifecycleOperation: { ...current.lifecycleOperation!, id: randomUUID(), hostComplete: false,
-      completedPluginIds: [], attempts: 0, leaseOwner: undefined, leaseUntil: undefined,
+      completedPluginIds: [], attempts: 0, leaseOwner: undefined, leaseUntil: undefined, leaseHolder: undefined,
       retryAt: undefined, retryRequired: undefined, verificationFailures: undefined,
       waitingForSignal: undefined, recheckRequested: undefined },
   } : {}) }).where(eq(agents.id, id)).returning();

server/src/modules/agent-lifecycle/adapters/effects.ts

@@ -10,6 +10,8 @@ export interface LifecycleEffects {
     profile: Pick<CreateAgentData, "adapterType" | "adapterConfig">): Promise<boolean>;
   environmentRequiresNewPlugin(db: Db, environmentId: string | null | undefined, selectedPluginIds: string[] | null): Promise<boolean>;
   notifyWork(db: Db): Promise<void>;
+  /** The calling controller process, stamped on a lifecycle lease so restart recovery can settle a dead holder. */
+  leaseHolder?(): Record<string, unknown> | null;
   beforeStatusChange(db: Db, companyId: string, agentId: string, before: string, after: string): Promise<void>;
   beforeConnectionEligibilityRestore?(db: Db, companyId: string, agentId: string): Promise<void>;
   reportFailure(context: LifecycleFailure, error: unknown): void;

server/src/modules/agent-lifecycle/adapters/postgres.ts

@@ -30,6 +30,32 @@ export const lifecycleSignalWait = and(
 
 /** The producer owns this outer transaction. Register its intent before either
  * the recheck write or the corresponding external fact commits. */
+/**
+ * Releases a lifecycle lease whose holder process restart recovery proved gone.
+ * Compare-and-swap on that exact holder and lease owner; the lifecycle state,
+ * operation and progress are untouched, so the worker claims the step again.
+ */
+export async function releaseAgentLifecycleLeaseOfHolderInTransaction(
+  tx: Pick<Db, "select" | "update">,
+  input: { agentId: string; holder: { bootId: string | null; pid: number | null } },
+  notify: () => Promise<void>,
+): Promise<{ released: false; reason: string } | { released: true; agent: LifecycleRow; leaseUntil: string }> {
+  if (!("nestedIndex" in tx)) throw new AgentLifecycleConflict("Lifecycle lease release requires the caller transaction");
+  const [agent] = await tx.select().from(agents).where(eq(agents.id, input.agentId)).for("update");
+  const operation = agent?.lifecycleOperation;
+  if (!agent || !operation?.leaseOwner || !operation.leaseUntil) return { released: false, reason: "lease_no_longer_held" };
+  if (operation.leaseHolder?.bootId !== input.holder.bootId || operation.leaseHolder?.pid !== input.holder.pid) {
+    return { released: false, reason: "owner_changed_since_proof" };
+  }
+  // Register the wake intent before the write so drain and sleep checks see it.
+  await notify();
+  const [row] = await tx.update(agents).set({
+    lifecycleOperation: sql`${agents.lifecycleOperation} - 'leaseOwner' - 'leaseUntil' - 'leaseHolder'`,
+  }).where(and(eq(agents.id, agent.id), sql`${agents.lifecycleOperation}->>'leaseOwner' = ${operation.leaseOwner}`))
+    .returning({ id: agents.id });
+  return row ? { released: true, agent, leaseUntil: operation.leaseUntil } : { released: false, reason: "lease_no_longer_held" };
+}
+
 export async function requestAgentLifecycleRecheckInTransaction(
   tx: Pick<Db, "select" | "update">,
   input: { companyId: string; agentId: string; expectedVersion?: number; onlyIfWaiting?: boolean },
@@ -75,7 +101,7 @@ async function changeLockedAgent(tx: Db, effects: LifecycleEffects, agent: Lifec
     }
     await effects.notifyWork(tx);
     const [updated] = await tx.update(agents).set({ lifecycleError: null,
-      lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt' - 'retryRequired' - 'verificationFailures' - 'leaseOwner' - 'leaseUntil' - 'waitingForSignal' - 'recheckRequested'`, updatedAt: new Date() }).where(eq(agents.id, agent.id)).returning();
+      lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt' - 'retryRequired' - 'verificationFailures' - 'leaseOwner' - 'leaseUntil' - 'leaseHolder' - 'waitingForSignal' - 'recheckRequested'`, updatedAt: new Date() }).where(eq(agents.id, agent.id)).returning();
     return updated;
   }
   if (command === "resume" && !["ready", "paused", "pausing", "resuming"].includes(agent.lifecycleState)) transition(agent.lifecycleState, command);
@@ -180,10 +206,13 @@ export function createLifecycleStore(db: Db, effects: LifecycleEffects): Lifecyc
       });
     },
     async claim(id, owner, now) {
+      // leaseHolder names the claiming process, so restart recovery settles a dead holder's lease at once.
+      const leaseHolder = effects.leaseHolder?.() ?? null;
       const [agent] = await db.update(agents).set({
         lifecycleOperation: sql`(coalesce(${agents.lifecycleOperation}, '{"id":"", "hostComplete":false, "completedPluginIds":[], "attempts":0}'::jsonb) - 'recheckRequested')
           || jsonb_build_object('leaseOwner', ${owner}::text, 'leaseUntil', ${(new Date(now.getTime() + 120_000)).toISOString()}::text,
-            'attempts', coalesce((${agents.lifecycleOperation}->>'attempts')::integer, 0) + 1)`,
+            'attempts', coalesce((${agents.lifecycleOperation}->>'attempts')::integer, 0) + 1)
+          || ${JSON.stringify(leaseHolder ? { leaseHolder } : {})}::jsonb`,
       }).where(and(eq(agents.id, id), pendingLifecycleWork,
         sql`(${agents.lifecycleState} in ('pausing', 'terminating', 'cleaning_up') or exists (
           select 1 from ${companies} where ${companies.id} = ${agents.companyId} and ${companies.status} = 'active'))`,
@@ -224,7 +253,7 @@ export function createLifecycleStore(db: Db, effects: LifecycleEffects): Lifecyc
     async defer(agent, owner, error, now, options) {
       if (options?.waitForSignal && agent.lifecycleState === "verifying") {
         const rows = await db.update(agents).set({ lifecycleError: null,
-          lifecycleOperation: sql`(${agents.lifecycleOperation} - 'leaseOwner' - 'leaseUntil' - 'retryAt' - 'waitingForSignal' - 'recheckRequested')
+          lifecycleOperation: sql`(${agents.lifecycleOperation} - 'leaseOwner' - 'leaseUntil' - 'leaseHolder' - 'retryAt' - 'waitingForSignal' - 'recheckRequested')
             || case when coalesce(${agents.lifecycleOperation}->>'recheckRequested', 'false') = 'true'
               then '{}'::jsonb else '{"waitingForSignal":true}'::jsonb end`,
           updatedAt: now,
@@ -240,7 +269,7 @@ export function createLifecycleStore(db: Db, effects: LifecycleEffects): Lifecyc
       }
       const rows = await db.update(agents).set({
         lifecycleError: error,
-        lifecycleOperation: sql`(${agents.lifecycleOperation} - 'leaseOwner' - 'leaseUntil' - 'retryAt' - 'retryRequired' - 'waitingForSignal' - 'recheckRequested')
+        lifecycleOperation: sql`(${agents.lifecycleOperation} - 'leaseOwner' - 'leaseUntil' - 'leaseHolder' - 'retryAt' - 'retryRequired' - 'waitingForSignal' - 'recheckRequested')
           || ${JSON.stringify(retry)}::jsonb`,
         updatedAt: now,
       }).where(and(eq(agents.id, agent.id), eq(agents.lifecycleVersion, agent.lifecycleVersion),

server/src/modules/agent-lifecycle/index.ts

@@ -13,7 +13,7 @@ import { hasCompanyHoldMismatch, projectCompanyHolds } from "./adapters/company-
 import { createLifecycleWorker } from "./application/worker.js";
 import type { LifecycleDriver } from "./application/ports.js";
 
-export { applyAgentPolicyHoldsInTransaction, requestAgentLifecycleRecheckInTransaction } from "./adapters/postgres.js";
+export { applyAgentPolicyHoldsInTransaction, releaseAgentLifecycleLeaseOfHolderInTransaction, requestAgentLifecycleRecheckInTransaction } from "./adapters/postgres.js";
 export { companyHoldMismatch, fenceActiveCompanyAgentStopsInTransaction } from "./adapters/company-policy.js";
 
 export { AgentLifecycleConflict, canConfigureAgentConnection, isAgentAwaitingSetup, isAgentSetupDraft } from "./domain/policy.js";