fix(lifecycle): align stop proof for pre-dispatch resource waits (#20384)

## Thinking Path

> - Paperclip manages AI agents and controls when their work can run.
> - An agent pause waits until every earlier run has stopped.
> - Recovery already treats verified workspace and AI-capacity waits as
pre-dispatch outcomes.
> - The stop proof did not accept those same receipts. Historical waits
could keep an agent pausing forever.
> - This pull request shares the resource-wait classification and the
existing execution-event check.
> - The benefit is consistent stop evidence without a new lifecycle
command, database backfill, or host-boot assumption.

## Linked Issues or Issue Description

Refs #18147 and #17718.

This fixes the verified resource-wait case of #18147. It does not
certify ambiguous historical runs that have no positive non-execution
evidence.

An agent can have an old cancelled legacy run with `errorCode:
workspace_busy`, `executionStage: preparing`, and `executionRecovery: {
kind: workspace_wait, providerWorkStarted: false }`. No process, native
coordinator, or retained resource exists. Recovery recognizes this as a
resource wait, but pause previously required a stop acknowledgment from
an executor that never started.

### Credit and related work

Thanks to @br1an-c0des (#17692) and @crownlayerapp (#15383) for the
focused preparation and shared-admission analysis, @lucasweismann-ai
(#18072) for the durable dispatch-evidence and race analysis, and
@Pdesengrini (#18905) for the cancellation-receipt and process-identity
guard cases. We reviewed these alternatives before implementing this
smaller change. No contributor commits were cherry-picked.

The design uses receipts already written by core. It does not adopt a
reboot-time bypass, a new adapter-entry journal, or a pause-abort route.
#17236 and #20037 address the separate cost of history reads.

## What Changed

- Add one shared classifier for matching workspace and AI-capacity wait
receipts. Use it in legacy recovery and cancellation stop proof.
- Reuse the existing dispatch-event predicate instead of maintaining
another event list.
- Retain checks for native execution references, unresolved workspace
obligations, controller leases, and resource cleanup. A live controller
or cleanup still in flight cannot pass the proof.
- Add a focused PostgreSQL suite. It drives the real pause worker with
historical wait records and checks Retry eligibility against the same
evidence.
- Document the contract. No schema, API, UI, dependency, or scheduling
change is required.

## Verification

Source checkout on macOS arm64, Node 26.12.0, isolated local PostgreSQL
18, synthetic data only.

```sh
node node_modules/vitest/vitest.mjs run --root server \
  src/modules/agent-lifecycle/__tests__/resource-wait-stop.test.ts \
  src/services/legacy-execution-recovery.test.ts
```

- 61 tests passed across these two files.
- All eight positive regressions failed on unchanged master. The fix
restores them.
- After rebasing onto current master, the 61 focused tests passed again.
Another 53 selected tests passed in
`explicit-native-continuation.test.ts` with `-t 'pre-dispatch review
wait|unclaimed workspace retry|historical native preparation'`. The
other 361 cases in that file were not run.
- Guard cases cover wrong/missing receipts, dispatch events, process
identity, native ownership, live controller leases, workspace recovery,
cleanup in flight, unreceipted remote resources, and an unknown
historical HTTP dispatch.
- A remote lease passes only after its exact termination receipt is
stored.
- Secret/PII diff scan and `git diff --check` passed.
- Standard Greptile reviewed commit
`201a61d2bb63290ca9b9b9d524e6c6715a872310`: [5/5 with no actionable
findings](https://github.com/paperclipai/paperclip/pull/20384#issuecomment-6753165178).
No review threads are unresolved.
- Local testing is limited to affected cases. GitHub CI owns broad
typecheck, build, and test coverage.

Verification setup failures remain accounted for: worktree preflight
could not complete because Cargo is unavailable; the first bundled
PostgreSQL run failed because its macOS binary lacks a shared library.
The focused tests passed on a separate, temporary local PostgreSQL 18
cluster. An early local server typecheck could not resolve the unbuilt
Runner package; it is not claimed as a pass.

## Risks

- Stop proof now accepts existing positive resource-wait receipts.
Missing PIDs, terminal status, elapsed time, and a different boot ID
remain insufficient.
- The receipt must match its error code. The runtime must remain
unresolved, with a preparation or settled stage and no conflicting
native or workspace references.
- The shared cancellation proof is also used by explicit continuation
and Retry. Those paths keep their existing user-intent, ownership,
pause, budget, and authorization gates. A stop proof does not itself
start another run.
- Runs that lack usable evidence remain held. This PR does not claim to
repair every run shape reported on #18147.
- No migration or live data edit is needed. No hosted deployment is
included.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, code execution,
PostgreSQL regression tests, and GitHub tools. The exact serving
revision, context window, and internal thinking configuration are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky LeachandPaperclip authored and GitHub committed 2026-11-09 14:40:14 -08:00
1 parent 9c634e42e4
commit 2550f399d9
5 files changed
+178 -18

No files matched your search

@@ -0,0 +1,142 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { agents, applyPendingMigrations, closeRegisteredClients, companies, createDb, environmentLeases, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "../../../__tests__/helpers/external-test-database.js";
import { createAgentLifecycle, configureAgentLifecycle } from "../../../services/agent-lifecycle.js";
import { createLifecycleDriver } from "../../../services/agent-lifecycle-driver.js";
import { agentExecutionsHaveStopped } from "../../../services/agent-execution-stop.js";
import { canRetryStoppedRun } from "../../../services/cancelled-native-startup.js";
import { legacyExecutionNeedsReconciliation } from "../../../services/legacy-execution-recovery.js";
import { legacyControllerBootId } from "../../../services/legacy-controller-lease.js";
import { PROCESS_START_REQUESTED } from "../../../services/native-local-process-stop.js";
import { remoteTerminationReceipt } from "../../../services/remote-execution-termination.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_AGENT_LIFECYCLE_TEST_DATABASE_URL");
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("pre-dispatch resource wait stop proof", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: Db;
beforeAll(async () => {
if (external) { await applyPendingMigrations(external); db = createDb(external); }
else { database = await startEmbeddedPostgresTestDatabase("paperclip-resource-wait-stop-"); db = createDb(database.connectionString); }
}, 90_000);
afterAll(async () => { if (external) await closeRegisteredClients(external); else await database?.cleanup(); });
async function fixture(errorCode = "workspace_busy", executionStage = "preparing") {
const id = randomUUID();
const [company] = await db.insert(companies).values({ name: "Resource wait", issuePrefix: `W${id.slice(0, 7)}` }).returning();
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Migrated agent",
adapterType: "paperclip_runner", status: "idle", lifecycleState: "ready" }).returning();
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id,
status: "cancelled", invocationSource: "on_demand", errorCode, runtimeMode: "legacy", executionStage,
startedAt: new Date("2000-01-01T00:00:00Z"), finishedAt: new Date("2000-01-01T00:00:01Z"),
controllerBootId: legacyControllerBootId, controllerLeaseExpiresAt: new Date("2000-01-01T00:01:00Z"),
runnerProfileJson: { adapterDispatch: { adapterType: "claude_local" } },
resultJson: {
executionRecovery: { kind: errorCode === "workspace_busy" ? "workspace_wait" : "ai_connection_wait", providerWorkStarted: false },
cancellation: { source: "control_plane", expected: true, initiator: { type: "system" },
reason: "Waiting for a resource", recordedAt: "2000-01-01T00:00:01Z" },
},
}).returning();
return { agent, run };
}
it.each(["workspace_busy", "ai_connection_busy", "ai_connection_pool_exhausted", "provider_limit_wait"].flatMap(errorCode =>
["preparing", "settled"].map(stage => [errorCode, stage])))("finishes a pause after a %s wait at %s", async (errorCode, stage) => {
const { agent, run } = await fixture(errorCode, stage);
const work = configureAgentLifecycle(db, {
requiredPluginIds: async () => [], runPlugin: async () => "complete",
runHost: createLifecycleDriver(db, {} as never).runHost,
});
try {
expect(legacyExecutionNeedsReconciliation(run)).toBe(false);
expect(await canRetryStoppedRun(db, run)).toBe(true);
await createAgentLifecycle(db).pauseAgent(agent.id);
await work.process(agent.id);
const [paused] = await db.select().from(agents).where(eq(agents.id, agent.id));
expect(paused).toMatchObject({ status: "paused", lifecycleState: "paused", lifecycleHolds: ["manual"] });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agent.id)))).toHaveLength(1);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("cancelled");
} finally { await work.stop(); }
});
it.each([
{ resultJson: {} },
{ resultJson: { executionRecovery: { kind: "workspace_wait", providerWorkStarted: true } } },
{ resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } },
{ errorCode: "cancelled" }, { executionStage: "dispatching" }, { executionStage: null },
{ runtimeMode: "native" }, { runtimeModeResolvedAt: new Date("2000-01-01T00:00:00Z") },
{ processPid: process.pid }, { processStartedAt: new Date("2000-01-01T00:00:00Z") },
{ nativeSessionId: randomUUID() }, { nativeIssueId: randomUUID() }, { sessionIdAfter: "provider-session" },
{ controllerLeaseExpiresAt: new Date("2099-01-01T00:00:00Z") },
{ runnerProfileJson: { nativeExecutionInput: {} } }, { runnerProfileJson: { remoteWorkspace: {} } },
{ runnerProfileJson: { nativeWorkspaceSync: {} } },
])("rejects contradictory run evidence: %j", async patch => {
const { agent, run } = await fixture();
await db.update(heartbeatRuns).set(patch).where(eq(heartbeatRuns.id, run.id));
const [changed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
expect(await canRetryStoppedRun(db, changed)).toBe(false);
expect(await agentExecutionsHaveStopped(db, [agent.id])).toBe(false);
});
it.each(["restore_unsafe_archive", "restore_lock_timeout"])("preserves an outstanding %s workspace obligation", async workspaceRestoreFailure => {
const { agent, run } = await fixture();
const [changed] = await db.update(heartbeatRuns).set({ resultJson: { ...run.resultJson, workspaceRestoreFailure,
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1" } } }).where(eq(heartbeatRuns.id, run.id)).returning();
expect(await canRetryStoppedRun(db, changed)).toBe(false);
expect(await agentExecutionsHaveStopped(db, [agent.id])).toBe(false);
});
it.each(["adapter.invoke", PROCESS_START_REQUESTED, "session.started", "provider.event", "tool.execution.started", "source_event"])(
"retains a wait with a recorded %s event", async eventType => {
const { agent, run } = await fixture();
await db.insert(heartbeatRunEvents).values({ companyId: run.companyId, agentId: agent.id, runId: run.id,
seq: 1, eventType, ...(eventType === "source_event" ? { sourceEventId: "provider-event" } : {}) });
expect(await canRetryStoppedRun(db, run)).toBe(false);
expect(await agentExecutionsHaveStopped(db, [agent.id])).toBe(false);
},
);
it.each([
{ provider: "local", status: "active" },
{ provider: "local", status: "pending_cleanup", releasedAt: new Date(0) },
{ provider: "local", status: "released", cleanupStatus: "failed", releasedAt: new Date(0) },
{ provider: "local", status: "released", cleanupStatus: "success", releasedAt: new Date(0), metadata: { pendingCleanupInFlight: true } },
{ provider: "ssh", status: "released", cleanupStatus: "success", releasedAt: new Date(0), providerLeaseId: "synthetic-lease" },
])("requires resource cleanup proof: %j", async lease => {
const { agent, run } = await fixture();
await db.insert(environmentLeases).values({ companyId: run.companyId, heartbeatRunId: run.id, ...lease });
expect(await canRetryStoppedRun(db, run)).toBe(false);
expect(await agentExecutionsHaveStopped(db, [agent.id])).toBe(false);
});
it("rejects a native coordinator even without a current owner", async () => {
const { agent, run } = await fixture();
const [issue] = await db.insert(issues).values({ companyId: run.companyId, title: "Native ownership" }).returning();
const [changed] = await db.update(heartbeatRuns).set({ nativeIssueId: issue.id }).where(eq(heartbeatRuns.id, run.id)).returning();
await db.insert(nativeRunFinalizations).values({ companyId: run.companyId, runId: run.id, issueId: issue.id, phase: "observed" });
expect(await canRetryStoppedRun(db, changed)).toBe(false);
expect(await agentExecutionsHaveStopped(db, [agent.id])).toBe(false);
});
it("accepts a remote resource only after its exact termination receipt is saved", async () => {
const { agent, run } = await fixture();
const [lease] = await db.insert(environmentLeases).values({ companyId: run.companyId, heartbeatRunId: run.id,
provider: "ssh", providerLeaseId: "synthetic-remote", status: "released", releasedAt: new Date(0), cleanupStatus: "success" }).returning();
expect(await canRetryStoppedRun(db, run)).toBe(false);
await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination:
remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "stopped" }) } }).where(eq(environmentLeases.id, lease.id));
expect(await canRetryStoppedRun(db, run)).toBe(true);
expect(await agentExecutionsHaveStopped(db, [agent.id])).toBe(true);
});
it("does not mistake an unproven HTTP dispatch for a resource wait", async () => {
const { agent, run } = await fixture("cancelled", "dispatching");
const [changed] = await db.update(heartbeatRuns).set({ resultJson: null,
runnerProfileJson: { adapterDispatch: { adapterType: "http" } } }).where(eq(heartbeatRuns.id, run.id)).returning();
expect(await canRetryStoppedRun(db, changed)).toBe(false);
expect(await agentExecutionsHaveStopped(db, [agent.id])).toBe(false);
});
});