Files
PaperClipAI/server/src/services/remote-execution-termination.ts
T
Nicky LeachandClaude Opus 5.5 afe86e9ded fix(runtime): record a plugin task's stop receipt when its stop is proven (#21237)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin environment runs each Paperclip Runner turn as a provider
task, and a follow-up run can continue the same Claude/ACPX conversation
from the saved checkpoint
> - Core authorizes that continuation only when the predecessor run's
remote execution has stopped (`remoteExecutionHasStopped`), so two
Runners never write one checkpoint
> - The predecessor's own settlement already proves the stop: the
provider reports `executionStopped`, and Core then completes the task
> - On a per-turn (ephemeral) lease, Core records the stop receipt only
when it releases the lease, and that release can be held for a long time
(for example by a pending workspace export)
> - In that window, an ordinary follow-up fails with `adapter_failed:
Plugin Runner retained session is not authorized`, even though the stop
is already proven
> - This pull request records the stop receipt when the stop is proven,
and the stop check accepts it before the release
> - The benefit is that follow-ups continue as soon as the predecessor
is proven stopped, and the release gates keep their meaning

## Linked Issues or Issue Description

Fixes #21211

Refs #21214 and #21223. Those cover the creation fence for same-run
recovery, which still requires a released lease.

Production evidence for this change: in both observed failures, the
predecessor was a per-turn plugin run. Its `task.settle` completed with
all provider calls successful. Core then did not release its lease for
30 minutes or more, and no release call was made in that time. The
follow-up failed in checkpoint recovery (`recoverPluginRunnerCheckpoint`
→ `resolvePluginTaskResume`) within seconds of admission.

## What Changed

- `server/src/services/environment-task-runtime.ts`
- After a recorded provider stop, `complete` writes the bound stop
receipt (`remoteTerminationReceipt(..., { state: "stopped" })`) on the
current lease for every lease policy. Before, it did this only for
`reuse_by_environment` leases. Reusable tasks still confirm every owner
and still move retained owners to released.
- New work on a lease (`submit`, `copilot_metadata`, first portable
resume) removes the stop receipt in the same write that clears
`runnerTaskExecutionStopped`. A replacement physical task does not copy
the old receipt.
- `server/src/services/remote-execution-termination.ts`
- New `hasBoundRemoteStopReceipt` checks only the receipt binding
(company, run, lease, provider and physical task).
`hasRemoteTerminationReceipt` uses it and keeps its released-only
semantics. Resource cleanup gates do not change.
- `remoteExecutionHasStopped` and `stoppedRemoteCleanupScopes` accept a
lease that is released with its receipt, or that is still `active` (not
released) with a bound stop receipt. A lease whose release started and
did not settle does not count. After release starts, the release owns
the receipt.
  - New `remoteExecutionHasReleased` gives the release-based answer.
- `server/src/services/environment-run-orchestrator.ts`: same-run plugin
recovery uses `remoteExecutionHasReleased`. The same-run resume check
after it requires released receipts, so its behavior does not change.
- `resolvePluginTaskResume` keeps its checks and their order. Its
refusal now names the first failing check: `Plugin Runner retained
session is not authorized (reason: <code>)`, for example
`predecessor_not_stopped`, `scope_mismatch` or `identity_mismatch`. The
old message stays as the prefix, and no ids go into it. The
stale-availability recheck in `environment-task-runtime.ts` names its
refusal the same way. Production telemetry for #21211 could not tell
which check fired. The next occurrence will show it.
- `doc/execution-semantics.md` and
`doc/testing-native-remote-continuity.md` describe the stop receipt and
when it is recorded.

Caller audit for the changed meaning of `remoteExecutionHasStopped`:

| Caller | Needs | Decision |
| --- | --- | --- |
| `agent-instruction-working-copies.ts` | stop | Accepts. Directory
cleanup still requires a released "destroyed" receipt. |
| `environment-run-orchestrator.ts` | release | Moved to
`remoteExecutionHasReleased`. |
| `plugin-task-resume.ts` | stop | Accepts. This is the fix. |
| `native-runtime/native-runner-ownership.ts` | stop | Accepts.
`hasPendingRunCleanup` still counts the active lease as pending. |
| `native-runtime/preserved-native-result-finalization.ts` | stop |
Accepts. Workspace recovery has its own gate. |
| `heartbeat/case-handoff-lifecycle.ts` | stop | Accepts.
`hasPendingRunCleanup` still holds active leases. |
| `heartbeat/run-control.ts` (3 sites) | stop | Accepts. Stop
acknowledgement and quarantined ownership retirement. Holds and blockers
still apply on the normal path. |
| `heartbeat/runtime-execution.ts` | stop | Accepts. Release errors
still fail it. |

## Verification

From `server/`, after `node scripts/agent-preflight.mjs`:

- `npx vitest run
src/__tests__/environment-task-authority.integration.test.ts`: 10/10
pass. The 2 new tests fail without the fix.
- The plugin task and environment runtime suites
(`environment-task-authority.integration`,
`plugin-runner-task.integration`, `plugin-warm-tasks.integration`,
`environment-task-runtime`, `plugin-task-source-recovery.integration`,
`environment-runtime`): 6 files, 366 pass.
- The audited caller suites (`agent-instruction-working-copies`, case
handoff legacy and native, `environment-run-orchestrator`,
`heartbeat-case-handoff`, `explicit-native-continuation`, `run-control`,
`native-remote-retry-admission`, `preserved-native-response-wait`): 9
files, 806 pass.
- `pnpm --filter @paperclipai/server typecheck`: pass.

The new integration tests cover:

- `complete` writes a bound stop receipt on a still-active ephemeral
lease. `remoteExecutionHasStopped` is true and
`hasRemoteTerminationReceipt` is still false.
- `resolvePluginTaskResume` (inspect and launch) authorizes a
follow-up's new task, and the real host `read_state` checkpoint read
succeeds, while the predecessor's release is held.
- Refusals: no provider stop, a receipt for another provider task, an
unstopped sibling lease, and a release in progress.
- New work on the lease removes the receipt. Release overwrites the
receipt with its own receipt. Warm owners keep their behavior.

I did not add an executor-level test. `native-session-executor.test.ts`
mocks `resolvePluginTaskResume`, so the integration test runs the real
checkpoint read path instead. I did not run the full suite locally. CI
runs it.

## Risks

- `remoteExecutionHasStopped` now returns true earlier for a run whose
leases are still held. Each caller is audited above. Callers that need
released resources use the release-based predicates.
- The stop receipt can now be on an `active` lease. The SQL readers of
`remoteExecutionTermination` (cost accounting, execution blockers,
idle-work proof, orphan settlement) also require a released status.
`pinFailedObservedWorkspace` treats an active ephemeral lease with a
receipt as not pinnable. That path applies only to the remote-workspace
flow and falls back to `preserveNativeFailedWorkspace`.
- Leases that settled before this change have no early receipt. They
continue to wait for their release.
- Nothing compares the refusal message exactly. Matchers use substrings,
and run failure classification does not use the message.
- A follow-up still fails closed if the predecessor's stop was never
proven. This change does not add a recoverable wait.
- The reason the release was held in production is not addressed here.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents, Honeycomb trace queries) and extended
thinking. One sub-agent wrote the change. A second session reviewed it
before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:20:45 -08:00

132 lines
7.4 KiB
TypeScript

import { and, eq } from "drizzle-orm";
import { environmentLeases, type Db } from "@paperclipai/db";
import { retiredReusableLeasePredecessors } from "./environment-lease-handoff.js";
type Lease = typeof environmentLeases.$inferSelect;
type LeaseIdentity = {
id: string; companyId: string; heartbeatRunId: string | null;
provider: string | null; providerLeaseId: string | null;
};
/** Bind a provider receipt to the exact host-owned lease and run. Old plugins
* return void; that remains supported but grants no continuation authority. */
export function remoteTerminationReceipt(lease: LeaseIdentity, value: unknown) {
const receipt = value as { providerLeaseId?: unknown; state?: unknown } | null;
if (!lease.heartbeatRunId || !lease.provider || lease.provider === "local" ||
!lease.providerLeaseId || receipt?.providerLeaseId !== lease.providerLeaseId ||
!["stopped", "destroyed"].includes(String(receipt?.state))) return undefined;
return {
schema: "paperclip.remote-termination.v1", companyId: lease.companyId,
runId: lease.heartbeatRunId, leaseId: lease.id, provider: lease.provider,
providerLeaseId: lease.providerLeaseId, state: receipt!.state,
confirmedAt: new Date().toISOString(),
};
}
type ReceiptLease = LeaseIdentity & {
releasedAt: unknown; cleanupStatus: string | null; status: string;
metadata: Record<string, unknown> | null;
};
/** The lease carries a provider stop receipt bound to its exact company, run,
* lease, provider and physical task. It proves execution there stopped, not
* that the lease's resources were released. */
export function hasBoundRemoteStopReceipt(lease: Pick<ReceiptLease, keyof LeaseIdentity | "metadata">): boolean {
const receipt = lease.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined;
return Boolean(receipt && receipt.schema === "paperclip.remote-termination.v1" &&
receipt.companyId === lease.companyId && receipt.runId === lease.heartbeatRunId &&
receipt.leaseId === lease.id && receipt.provider === lease.provider &&
remoteTerminationReceipt(lease, receipt));
}
/** The lease was released with successful cleanup and its bound stop receipt.
* Resource cleanup gates use this; it never accepts an unreleased lease. */
export function hasRemoteTerminationReceipt(lease: ReceiptLease): boolean {
return Boolean(lease.releasedAt && lease.cleanupStatus === "success" &&
["released", "expired", "failed"].includes(lease.status) && hasBoundRemoteStopReceipt(lease));
}
/** Release or destroy of this lease has started: plugin release and destroy
* record their cleanup method first, and task settlement, startup cleanup and
* interrupted-creation cleanup mark the task retiring. The provider can then be
* releasing the task's resources while the lease row is still active. */
function leaseReleaseStarted(lease: Pick<ReceiptLease, "metadata">): boolean {
return lease.metadata?.pluginLeaseCleanupMethod !== undefined || lease.metadata?.runnerTaskRetiring === true;
}
/** Execution on this lease has stopped: it was released with its termination
* receipt, or it is still active, its release has not started, and its plugin
* task recorded the provider's stop when it settled. Release can be held long
* after that stop (for example by a pending workspace export); once release
* starts, it owns the receipt until the lease is released. */
function remoteLeaseExecutionStopped(lease: ReceiptLease): boolean {
return hasRemoteTerminationReceipt(lease) ||
lease.status === "active" && !lease.releasedAt && !leaseReleaseStarted(lease) && hasBoundRemoteStopReceipt(lease);
}
/** Workspace recovery saved this run's files, re-stopped its sandbox, and then
* queued the stopped sandbox for destroy. The destroy is resource cleanup: the
* stop receipt it carried forward still proves no provider can run there. */
export function hasRecoveredCleanupStopReceipt(lease: LeaseIdentity & {
releasedAt: unknown; status: string; metadata: Record<string, unknown> | null;
}): boolean {
const cleanup = lease.metadata?.workspaceRecoveryCleanup as Record<string, unknown> | undefined;
if (!cleanup || cleanup.runId !== lease.heartbeatRunId || !lease.releasedAt ||
!["pending_cleanup", "released", "expired", "failed"].includes(lease.status)) return false;
return hasRemoteTerminationReceipt({ ...lease, status: "released", cleanupStatus: "success",
metadata: { remoteExecutionTermination: cleanup.stopReceipt } });
}
export function remoteLeaseCleanupScope(lease: Pick<LeaseIdentity, "provider" | "providerLeaseId">) {
return lease.provider && lease.provider !== "local" && lease.providerLeaseId
? JSON.stringify([lease.provider, lease.providerLeaseId]) : undefined;
}
export async function resolveRemoteTerminationLease(db: Db, lease: Lease): Promise<Lease> {
if (hasRemoteTerminationReceipt(lease) || lease.status !== "expired" ||
lease.leasePolicy !== "reuse_by_environment" || !lease.provider || !lease.providerLeaseId) return lease;
const history = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.provider, lease.provider), eq(environmentLeases.providerLeaseId, lease.providerLeaseId),
));
const current = history.find(row => row.id === lease.id);
if (!current || current.companyId !== lease.companyId || current.heartbeatRunId !== lease.heartbeatRunId) return lease;
const tips = history.filter(row => !Object.hasOwn(row.metadata ?? {}, "reusableLeaseSuccessor"));
if (tips.length !== 1) return current;
const tip = tips[0]!;
const receipt = tip.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined;
if (!hasRemoteTerminationReceipt(tip) || receipt?.state !== "destroyed") return current;
const predecessors = retiredReusableLeasePredecessors(tip, history.filter(row => row.id !== tip.id));
if (!predecessors?.some(row => row.id === current.id)) return current;
const termination = remoteTerminationReceipt(current, { providerLeaseId: current.providerLeaseId, state: "destroyed" });
if (!termination) return current;
return { ...current, metadata: { ...current.metadata, remoteExecutionTermination: {
...termination, confirmedAt: receipt.confirmedAt,
} } };
}
async function runRemoteLeases(db: Db, companyId: string, runId: string) {
const leases = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId),
));
return Promise.all(leases.map(lease => resolveRemoteTerminationLease(db, lease)));
}
/** Cleanup scopes of a run whose every lease has stopped execution, or null. */
export async function stoppedRemoteCleanupScopes(db: Db, companyId: string, runId: string) {
const leases = await runRemoteLeases(db, companyId, runId);
if (leases.length === 0 || !leases.every(remoteLeaseExecutionStopped)) return null;
return [...new Set(leases.map(lease => remoteLeaseCleanupScope(lease)!))];
}
/** Every lease of the run has stopped execution. Its resources can still be held. */
export async function remoteExecutionHasStopped(db: Db, companyId: string, runId: string) {
return await stoppedRemoteCleanupScopes(db, companyId, runId) !== null;
}
/** Every lease of the run was released with its termination receipt. */
export async function remoteExecutionHasReleased(db: Db, companyId: string, runId: string) {
const leases = await runRemoteLeases(db, companyId, runId);
return leases.length > 0 && leases.every(hasRemoteTerminationReceipt);
}