Files
PaperClipAI/server/src/services/plugin-task-resume.ts
T
Nicky LeachandClaude Opus 5.5 afe86e9ded fix(runtime): record a plugin task's stop receipt when its stop is proven (#21237)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin environment runs each Paperclip Runner turn as a provider
task, and a follow-up run can continue the same Claude/ACPX conversation
from the saved checkpoint
> - Core authorizes that continuation only when the predecessor run's
remote execution has stopped (`remoteExecutionHasStopped`), so two
Runners never write one checkpoint
> - The predecessor's own settlement already proves the stop: the
provider reports `executionStopped`, and Core then completes the task
> - On a per-turn (ephemeral) lease, Core records the stop receipt only
when it releases the lease, and that release can be held for a long time
(for example by a pending workspace export)
> - In that window, an ordinary follow-up fails with `adapter_failed:
Plugin Runner retained session is not authorized`, even though the stop
is already proven
> - This pull request records the stop receipt when the stop is proven,
and the stop check accepts it before the release
> - The benefit is that follow-ups continue as soon as the predecessor
is proven stopped, and the release gates keep their meaning

## Linked Issues or Issue Description

Fixes #21211

Refs #21214 and #21223. Those cover the creation fence for same-run
recovery, which still requires a released lease.

Production evidence for this change: in both observed failures, the
predecessor was a per-turn plugin run. Its `task.settle` completed with
all provider calls successful. Core then did not release its lease for
30 minutes or more, and no release call was made in that time. The
follow-up failed in checkpoint recovery (`recoverPluginRunnerCheckpoint`
→ `resolvePluginTaskResume`) within seconds of admission.

## What Changed

- `server/src/services/environment-task-runtime.ts`
- After a recorded provider stop, `complete` writes the bound stop
receipt (`remoteTerminationReceipt(..., { state: "stopped" })`) on the
current lease for every lease policy. Before, it did this only for
`reuse_by_environment` leases. Reusable tasks still confirm every owner
and still move retained owners to released.
- New work on a lease (`submit`, `copilot_metadata`, first portable
resume) removes the stop receipt in the same write that clears
`runnerTaskExecutionStopped`. A replacement physical task does not copy
the old receipt.
- `server/src/services/remote-execution-termination.ts`
- New `hasBoundRemoteStopReceipt` checks only the receipt binding
(company, run, lease, provider and physical task).
`hasRemoteTerminationReceipt` uses it and keeps its released-only
semantics. Resource cleanup gates do not change.
- `remoteExecutionHasStopped` and `stoppedRemoteCleanupScopes` accept a
lease that is released with its receipt, or that is still `active` (not
released) with a bound stop receipt. A lease whose release started and
did not settle does not count. After release starts, the release owns
the receipt.
  - New `remoteExecutionHasReleased` gives the release-based answer.
- `server/src/services/environment-run-orchestrator.ts`: same-run plugin
recovery uses `remoteExecutionHasReleased`. The same-run resume check
after it requires released receipts, so its behavior does not change.
- `resolvePluginTaskResume` keeps its checks and their order. Its
refusal now names the first failing check: `Plugin Runner retained
session is not authorized (reason: <code>)`, for example
`predecessor_not_stopped`, `scope_mismatch` or `identity_mismatch`. The
old message stays as the prefix, and no ids go into it. The
stale-availability recheck in `environment-task-runtime.ts` names its
refusal the same way. Production telemetry for #21211 could not tell
which check fired. The next occurrence will show it.
- `doc/execution-semantics.md` and
`doc/testing-native-remote-continuity.md` describe the stop receipt and
when it is recorded.

Caller audit for the changed meaning of `remoteExecutionHasStopped`:

| Caller | Needs | Decision |
| --- | --- | --- |
| `agent-instruction-working-copies.ts` | stop | Accepts. Directory
cleanup still requires a released "destroyed" receipt. |
| `environment-run-orchestrator.ts` | release | Moved to
`remoteExecutionHasReleased`. |
| `plugin-task-resume.ts` | stop | Accepts. This is the fix. |
| `native-runtime/native-runner-ownership.ts` | stop | Accepts.
`hasPendingRunCleanup` still counts the active lease as pending. |
| `native-runtime/preserved-native-result-finalization.ts` | stop |
Accepts. Workspace recovery has its own gate. |
| `heartbeat/case-handoff-lifecycle.ts` | stop | Accepts.
`hasPendingRunCleanup` still holds active leases. |
| `heartbeat/run-control.ts` (3 sites) | stop | Accepts. Stop
acknowledgement and quarantined ownership retirement. Holds and blockers
still apply on the normal path. |
| `heartbeat/runtime-execution.ts` | stop | Accepts. Release errors
still fail it. |

## Verification

From `server/`, after `node scripts/agent-preflight.mjs`:

- `npx vitest run
src/__tests__/environment-task-authority.integration.test.ts`: 10/10
pass. The 2 new tests fail without the fix.
- The plugin task and environment runtime suites
(`environment-task-authority.integration`,
`plugin-runner-task.integration`, `plugin-warm-tasks.integration`,
`environment-task-runtime`, `plugin-task-source-recovery.integration`,
`environment-runtime`): 6 files, 366 pass.
- The audited caller suites (`agent-instruction-working-copies`, case
handoff legacy and native, `environment-run-orchestrator`,
`heartbeat-case-handoff`, `explicit-native-continuation`, `run-control`,
`native-remote-retry-admission`, `preserved-native-response-wait`): 9
files, 806 pass.
- `pnpm --filter @paperclipai/server typecheck`: pass.

The new integration tests cover:

- `complete` writes a bound stop receipt on a still-active ephemeral
lease. `remoteExecutionHasStopped` is true and
`hasRemoteTerminationReceipt` is still false.
- `resolvePluginTaskResume` (inspect and launch) authorizes a
follow-up's new task, and the real host `read_state` checkpoint read
succeeds, while the predecessor's release is held.
- Refusals: no provider stop, a receipt for another provider task, an
unstopped sibling lease, and a release in progress.
- New work on the lease removes the receipt. Release overwrites the
receipt with its own receipt. Warm owners keep their behavior.

I did not add an executor-level test. `native-session-executor.test.ts`
mocks `resolvePluginTaskResume`, so the integration test runs the real
checkpoint read path instead. I did not run the full suite locally. CI
runs it.

## Risks

- `remoteExecutionHasStopped` now returns true earlier for a run whose
leases are still held. Each caller is audited above. Callers that need
released resources use the release-based predicates.
- The stop receipt can now be on an `active` lease. The SQL readers of
`remoteExecutionTermination` (cost accounting, execution blockers,
idle-work proof, orphan settlement) also require a released status.
`pinFailedObservedWorkspace` treats an active ephemeral lease with a
receipt as not pinnable. That path applies only to the remote-workspace
flow and falls back to `preserveNativeFailedWorkspace`.
- Leases that settled before this change have no early receipt. They
continue to wait for their release.
- Nothing compares the refusal message exactly. Matchers use substrings,
and run failure classification does not use the message.
- A follow-up still fails closed if the predecessor's stop was never
proven. This change does not add a recoverable wait.
- The reason the release was held in production is not addressed here.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents, Honeycomb trace queries) and extended
thinking. One sub-agent wrote the change. A second session reviewed it
before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:20:45 -08:00

94 lines
5.9 KiB
TypeScript

import { and, eq, or } from "drizzle-orm";
import { environmentLeases, heartbeatRuns, type Db } from "@paperclipai/db";
import { hasRemoteTerminationReceipt, remoteExecutionHasStopped } from "./remote-execution-termination.js";
/** The first check that refused a retained plugin task resume, named in the error. */
export type PluginTaskResumeRefusal =
| "next_lease_missing"
| "predecessor_candidate_missing"
| "predecessor_candidate_ambiguous"
| "predecessor_task_missing"
| "identity_mismatch"
| "provider_mismatch"
| "next_task_missing"
| "same_provider_task"
| "predecessor_run_active"
| "scope_mismatch"
| "next_lease_inactive"
| "next_run_not_running"
| "predecessor_not_stopped"
| "checkpoint_status_unconfirmed"
| "resume_task_mismatch";
/** Keeps the original message as a prefix so existing matchers still apply. */
export function pluginTaskResumeRefused(reason: PluginTaskResumeRefusal) {
return new Error(`Plugin Runner retained session is not authorized (reason: ${reason})`);
}
type ResumeRow = { lease: typeof environmentLeases.$inferSelect; run: typeof heartbeatRuns.$inferSelect };
type PriorIdentity = { runnerInstanceId: string; environmentLeaseId: string; runId: string; normalizedSessionId: string };
const TERMINAL_RUN_STATUSES = ["succeeded", "interrupted", "failed", "cancelled", "timed_out"];
/** Cheap, synchronous checks in evaluation order; the stop proof is read only after these pass. */
function resumeRefusal(priorIdentity: PriorIdentity, next: ResumeRow | undefined, candidates: ResumeRow[]): PluginTaskResumeRefusal | null {
if (!next) return "next_lease_missing";
if (candidates.length === 0) return "predecessor_candidate_missing";
if (candidates.length > 1) return "predecessor_candidate_ambiguous";
const old = candidates[0]!;
const identity = old.lease.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined;
if (!old.lease.providerLeaseId) return "predecessor_task_missing";
if (!identity || identity.runnerId !== priorIdentity.runnerInstanceId || identity.leaseId !== priorIdentity.environmentLeaseId ||
identity.sessionId !== priorIdentity.normalizedSessionId || identity.runId !== priorIdentity.runId) return "identity_mismatch";
if (old.lease.metadata?.driver !== "plugin" || next.lease.metadata?.driver !== "plugin") return "provider_mismatch";
if (!next.lease.providerLeaseId) return "next_task_missing";
if (next.lease.providerLeaseId === old.lease.providerLeaseId) return "same_provider_task";
if (old.run.id !== next.run.id && !TERMINAL_RUN_STATUSES.includes(old.run.status)) return "predecessor_run_active";
if (old.run.agentId !== next.run.agentId || old.lease.issueId !== next.lease.issueId ||
old.lease.executionWorkspaceId !== next.lease.executionWorkspaceId ||
!old.lease.environmentId || old.lease.environmentId !== next.lease.environmentId) return "scope_mismatch";
if (!next.lease.provider?.startsWith("plugin:") || old.lease.provider !== next.lease.provider ||
old.lease.metadata?.pluginId !== next.lease.metadata?.pluginId ||
old.lease.metadata?.driverKey !== next.lease.metadata?.driverKey) return "provider_mismatch";
if (next.lease.status !== "active" || next.lease.releasedAt ||
(next.lease.expiresAt && next.lease.expiresAt.getTime() <= Date.now())) return "next_lease_inactive";
if (next.run.status !== "running") return "next_run_not_running";
return null;
}
/** A saved provider task can resume only within the same authorized agent workspace. */
export async function resolvePluginTaskResume(db: Db, input: {
companyId: string; leaseId: string;
priorIdentity: PriorIdentity;
/** Read-only discovery; launching still requires an attested retained checkpoint. */
inspectCheckpoint?: boolean;
}): Promise<string> {
const rows = await db.select({ lease: environmentLeases, run: heartbeatRuns }).from(environmentLeases)
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), eq(heartbeatRuns.companyId, input.companyId)))
.where(and(eq(environmentLeases.companyId, input.companyId),
or(eq(environmentLeases.id, input.leaseId), eq(heartbeatRuns.id, input.priorIdentity.runId))));
const next = rows.find(row => row.lease.id === input.leaseId);
// A retained PRP lease is logical; later provider tasks have different physical lease IDs.
const candidates = rows.filter(row => {
const identity = row.lease.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined;
const consumer = row.lease.metadata?.runnerTaskSourceConsumer as { leaseId?: unknown; runId?: unknown; resumeFrom?: unknown } | null | undefined;
return row.lease.id !== input.leaseId && row.run.id === input.priorIdentity.runId &&
row.lease.metadata?.portableTaskSuccessor === undefined &&
(!Object.hasOwn(row.lease.metadata ?? {}, "runnerTaskSourceConsumer") ||
consumer?.leaseId === input.leaseId && consumer.runId === next?.run.id && consumer.resumeFrom === row.lease.providerLeaseId) &&
(input.inspectCheckpoint || row.lease.metadata?.runnerTaskSessionRetained === true) &&
identity?.runnerId === input.priorIdentity.runnerInstanceId && identity.leaseId === input.priorIdentity.environmentLeaseId &&
identity.sessionId === input.priorIdentity.normalizedSessionId && identity.runId === input.priorIdentity.runId &&
!rows.some(successor => successor.lease.id !== input.leaseId &&
successor.lease.metadata?.runnerTaskResumeFrom === row.lease.providerLeaseId);
});
const refusal = resumeRefusal(input.priorIdentity, next, candidates);
if (refusal) throw pluginTaskResumeRefused(refusal);
const old = candidates[0]!;
const stopped = old.run.id === next!.run.id
? rows.filter(row => row.lease.id !== next!.lease.id).every(row => hasRemoteTerminationReceipt(row.lease))
: await remoteExecutionHasStopped(db, input.companyId, old.run.id);
if (!stopped) throw pluginTaskResumeRefused("predecessor_not_stopped");
return old.lease.providerLeaseId!;
}