Files
PaperClipAI/server/src/services/pre-dispatch-execution-evidence.ts
T
Nicky LeachandPaperclip 2550f399d9 fix(lifecycle): align stop proof for pre-dispatch resource waits (#20384)
## Thinking Path

> - Paperclip manages AI agents and controls when their work can run.
> - An agent pause waits until every earlier run has stopped.
> - Recovery already treats verified workspace and AI-capacity waits as
pre-dispatch outcomes.
> - The stop proof did not accept those same receipts. Historical waits
could keep an agent pausing forever.
> - This pull request shares the resource-wait classification and the
existing execution-event check.
> - The benefit is consistent stop evidence without a new lifecycle
command, database backfill, or host-boot assumption.

## Linked Issues or Issue Description

Refs #18147 and #17718.

This fixes the verified resource-wait case of #18147. It does not
certify ambiguous historical runs that have no positive non-execution
evidence.

An agent can have an old cancelled legacy run with `errorCode:
workspace_busy`, `executionStage: preparing`, and `executionRecovery: {
kind: workspace_wait, providerWorkStarted: false }`. No process, native
coordinator, or retained resource exists. Recovery recognizes this as a
resource wait, but pause previously required a stop acknowledgment from
an executor that never started.

### Credit and related work

Thanks to @br1an-c0des (#17692) and @crownlayerapp (#15383) for the
focused preparation and shared-admission analysis, @lucasweismann-ai
(#18072) for the durable dispatch-evidence and race analysis, and
@Pdesengrini (#18905) for the cancellation-receipt and process-identity
guard cases. We reviewed these alternatives before implementing this
smaller change. No contributor commits were cherry-picked.

The design uses receipts already written by core. It does not adopt a
reboot-time bypass, a new adapter-entry journal, or a pause-abort route.
#17236 and #20037 address the separate cost of history reads.

## What Changed

- Add one shared classifier for matching workspace and AI-capacity wait
receipts. Use it in legacy recovery and cancellation stop proof.
- Reuse the existing dispatch-event predicate instead of maintaining
another event list.
- Retain checks for native execution references, unresolved workspace
obligations, controller leases, and resource cleanup. A live controller
or cleanup still in flight cannot pass the proof.
- Add a focused PostgreSQL suite. It drives the real pause worker with
historical wait records and checks Retry eligibility against the same
evidence.
- Document the contract. No schema, API, UI, dependency, or scheduling
change is required.

## Verification

Source checkout on macOS arm64, Node 26.12.0, isolated local PostgreSQL
18, synthetic data only.

```sh
node node_modules/vitest/vitest.mjs run --root server \
  src/modules/agent-lifecycle/__tests__/resource-wait-stop.test.ts \
  src/services/legacy-execution-recovery.test.ts
```

- 61 tests passed across these two files.
- All eight positive regressions failed on unchanged master. The fix
restores them.
- After rebasing onto current master, the 61 focused tests passed again.
Another 53 selected tests passed in
`explicit-native-continuation.test.ts` with `-t 'pre-dispatch review
wait|unclaimed workspace retry|historical native preparation'`. The
other 361 cases in that file were not run.
- Guard cases cover wrong/missing receipts, dispatch events, process
identity, native ownership, live controller leases, workspace recovery,
cleanup in flight, unreceipted remote resources, and an unknown
historical HTTP dispatch.
- A remote lease passes only after its exact termination receipt is
stored.
- Secret/PII diff scan and `git diff --check` passed.
- Standard Greptile reviewed commit
`201a61d2bb63290ca9b9b9d524e6c6715a872310`: [5/5 with no actionable
findings](https://github.com/paperclipai/paperclip/pull/20384#issuecomment-6753165178).
No review threads are unresolved.
- Local testing is limited to affected cases. GitHub CI owns broad
typecheck, build, and test coverage.

Verification setup failures remain accounted for: worktree preflight
could not complete because Cargo is unavailable; the first bundled
PostgreSQL run failed because its macOS binary lacks a shared library.
The focused tests passed on a separate, temporary local PostgreSQL 18
cluster. An early local server typecheck could not resolve the unbuilt
Runner package; it is not claimed as a pass.

## Risks

- Stop proof now accepts existing positive resource-wait receipts.
Missing PIDs, terminal status, elapsed time, and a different boot ID
remain insufficient.
- The receipt must match its error code. The runtime must remain
unresolved, with a preparation or settled stage and no conflicting
native or workspace references.
- The shared cancellation proof is also used by explicit continuation
and Retry. Those paths keep their existing user-intent, ownership,
pause, budget, and authorization gates. A stop proof does not itself
start another run.
- Runs that lack usable evidence remain held. This PR does not claim to
repair every run shape reported on #18147.
- No migration or live data edit is needed. No hosted deployment is
included.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, code execution,
PostgreSQL regression tests, and GitHub tools. The exact serving
revision, context window, and internal thinking configuration are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 14:40:14 -08:00

42 lines
3.0 KiB
TypeScript

import { and, eq, inArray, isNotNull, isNull, or, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations } from "@paperclipai/db";
import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-local-process-stop.js";
import { LEGACY_WORKSPACE_RECOVERY_SCHEMA } from "./workspace-restore-recovery-state.js";
export function isPreDispatchResourceWait(run: Pick<typeof heartbeatRuns.$inferSelect,
"runtimeMode" | "status" | "errorCode" | "resultJson">): boolean {
const evidence = run.resultJson?.executionRecovery as Record<string, unknown> | undefined;
return run.runtimeMode === "legacy" && run.status === "cancelled" && evidence?.providerWorkStarted === false &&
(run.errorCode === "workspace_busy" && evidence.kind === "workspace_wait" ||
["ai_connection_busy", "ai_connection_pool_exhausted", "provider_limit_wait"].includes(run.errorCode ?? "") &&
evidence.kind === "ai_connection_wait");
}
export function executionStartedEventPredicate() {
return or(isNotNull(heartbeatRunEvents.sourceEventId), inArray(heartbeatRunEvents.eventType,
["adapter.invoke", PROCESS_START_REQUESTED, PROCESS_IDENTITY_RECORDED, "harness.ready", "session.started",
"session.resumed", "session.updated", "turn.started", "provider.event", "provider.rpc_result", "tool.execution.started"]));
}
/** Both dispatch cancellation policies require this same retained-evidence
* proof. Each caller still owns its terminal status and admission receipt. */
export function noRetainedExecutionPredicate() {
return and(
isNull(heartbeatRuns.startedAt), isNull(heartbeatRuns.runtimeModeResolvedAt),
isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId), isNull(heartbeatRuns.processStartedAt),
isNull(heartbeatRuns.nativeIssueId), isNull(heartbeatRuns.nativeSessionId), isNull(heartbeatRuns.sessionIdAfter),
isNull(heartbeatRuns.controllerBootId), isNull(heartbeatRuns.controllerLeaseExpiresAt), isNull(heartbeatRuns.executionStage),
sql`coalesce(${heartbeatRuns.resultJson}->>'workspaceRestoreFailure', '') <> 'restore_unsafe_archive'`,
sql`coalesce(${heartbeatRuns.resultJson}->'workspaceRestoreRecovery'->>'schema', '') <> ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`,
sql`not (coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) ?| array[
'nativeExecutionInput', 'nativeWorkspaceSync', 'remoteWorkspace', 'nativeWorkspaceFinalizationOwner', 'sessionCheckpoint'])`,
sql`not exists (select 1 from ${nativeRunFinalizations}
where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId} and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id})`,
sql`not exists (select 1 from ${environmentLeases}
where ${environmentLeases.companyId} = ${heartbeatRuns.companyId} and ${environmentLeases.heartbeatRunId} = ${heartbeatRuns.id})`,
sql`not exists (select 1 from ${heartbeatRunEvents}
where ${and(eq(heartbeatRunEvents.companyId, heartbeatRuns.companyId),
eq(heartbeatRunEvents.runId, heartbeatRuns.id), executionStartedEventPredicate())})`,
);
}