mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A plugin environment runs each Paperclip Runner turn as a provider task, and a follow-up run can continue the same Claude/ACPX conversation from the saved checkpoint > - Core authorizes that continuation only when the predecessor run's remote execution has stopped (`remoteExecutionHasStopped`), so two Runners never write one checkpoint > - The predecessor's own settlement already proves the stop: the provider reports `executionStopped`, and Core then completes the task > - On a per-turn (ephemeral) lease, Core records the stop receipt only when it releases the lease, and that release can be held for a long time (for example by a pending workspace export) > - In that window, an ordinary follow-up fails with `adapter_failed: Plugin Runner retained session is not authorized`, even though the stop is already proven > - This pull request records the stop receipt when the stop is proven, and the stop check accepts it before the release > - The benefit is that follow-ups continue as soon as the predecessor is proven stopped, and the release gates keep their meaning ## Linked Issues or Issue Description Fixes #21211 Refs #21214 and #21223. Those cover the creation fence for same-run recovery, which still requires a released lease. Production evidence for this change: in both observed failures, the predecessor was a per-turn plugin run. Its `task.settle` completed with all provider calls successful. Core then did not release its lease for 30 minutes or more, and no release call was made in that time. The follow-up failed in checkpoint recovery (`recoverPluginRunnerCheckpoint` → `resolvePluginTaskResume`) within seconds of admission. ## What Changed - `server/src/services/environment-task-runtime.ts` - After a recorded provider stop, `complete` writes the bound stop receipt (`remoteTerminationReceipt(..., { state: "stopped" })`) on the current lease for every lease policy. Before, it did this only for `reuse_by_environment` leases. Reusable tasks still confirm every owner and still move retained owners to released. - New work on a lease (`submit`, `copilot_metadata`, first portable resume) removes the stop receipt in the same write that clears `runnerTaskExecutionStopped`. A replacement physical task does not copy the old receipt. - `server/src/services/remote-execution-termination.ts` - New `hasBoundRemoteStopReceipt` checks only the receipt binding (company, run, lease, provider and physical task). `hasRemoteTerminationReceipt` uses it and keeps its released-only semantics. Resource cleanup gates do not change. - `remoteExecutionHasStopped` and `stoppedRemoteCleanupScopes` accept a lease that is released with its receipt, or that is still `active` (not released) with a bound stop receipt. A lease whose release started and did not settle does not count. After release starts, the release owns the receipt. - New `remoteExecutionHasReleased` gives the release-based answer. - `server/src/services/environment-run-orchestrator.ts`: same-run plugin recovery uses `remoteExecutionHasReleased`. The same-run resume check after it requires released receipts, so its behavior does not change. - `resolvePluginTaskResume` keeps its checks and their order. Its refusal now names the first failing check: `Plugin Runner retained session is not authorized (reason: <code>)`, for example `predecessor_not_stopped`, `scope_mismatch` or `identity_mismatch`. The old message stays as the prefix, and no ids go into it. The stale-availability recheck in `environment-task-runtime.ts` names its refusal the same way. Production telemetry for #21211 could not tell which check fired. The next occurrence will show it. - `doc/execution-semantics.md` and `doc/testing-native-remote-continuity.md` describe the stop receipt and when it is recorded. Caller audit for the changed meaning of `remoteExecutionHasStopped`: | Caller | Needs | Decision | | --- | --- | --- | | `agent-instruction-working-copies.ts` | stop | Accepts. Directory cleanup still requires a released "destroyed" receipt. | | `environment-run-orchestrator.ts` | release | Moved to `remoteExecutionHasReleased`. | | `plugin-task-resume.ts` | stop | Accepts. This is the fix. | | `native-runtime/native-runner-ownership.ts` | stop | Accepts. `hasPendingRunCleanup` still counts the active lease as pending. | | `native-runtime/preserved-native-result-finalization.ts` | stop | Accepts. Workspace recovery has its own gate. | | `heartbeat/case-handoff-lifecycle.ts` | stop | Accepts. `hasPendingRunCleanup` still holds active leases. | | `heartbeat/run-control.ts` (3 sites) | stop | Accepts. Stop acknowledgement and quarantined ownership retirement. Holds and blockers still apply on the normal path. | | `heartbeat/runtime-execution.ts` | stop | Accepts. Release errors still fail it. | ## Verification From `server/`, after `node scripts/agent-preflight.mjs`: - `npx vitest run src/__tests__/environment-task-authority.integration.test.ts`: 10/10 pass. The 2 new tests fail without the fix. - The plugin task and environment runtime suites (`environment-task-authority.integration`, `plugin-runner-task.integration`, `plugin-warm-tasks.integration`, `environment-task-runtime`, `plugin-task-source-recovery.integration`, `environment-runtime`): 6 files, 366 pass. - The audited caller suites (`agent-instruction-working-copies`, case handoff legacy and native, `environment-run-orchestrator`, `heartbeat-case-handoff`, `explicit-native-continuation`, `run-control`, `native-remote-retry-admission`, `preserved-native-response-wait`): 9 files, 806 pass. - `pnpm --filter @paperclipai/server typecheck`: pass. The new integration tests cover: - `complete` writes a bound stop receipt on a still-active ephemeral lease. `remoteExecutionHasStopped` is true and `hasRemoteTerminationReceipt` is still false. - `resolvePluginTaskResume` (inspect and launch) authorizes a follow-up's new task, and the real host `read_state` checkpoint read succeeds, while the predecessor's release is held. - Refusals: no provider stop, a receipt for another provider task, an unstopped sibling lease, and a release in progress. - New work on the lease removes the receipt. Release overwrites the receipt with its own receipt. Warm owners keep their behavior. I did not add an executor-level test. `native-session-executor.test.ts` mocks `resolvePluginTaskResume`, so the integration test runs the real checkpoint read path instead. I did not run the full suite locally. CI runs it. ## Risks - `remoteExecutionHasStopped` now returns true earlier for a run whose leases are still held. Each caller is audited above. Callers that need released resources use the release-based predicates. - The stop receipt can now be on an `active` lease. The SQL readers of `remoteExecutionTermination` (cost accounting, execution blockers, idle-work proof, orphan settlement) also require a released status. `pinFailedObservedWorkspace` treats an active ephemeral lease with a receipt as not pinnable. That path applies only to the remote-workspace flow and falls back to `preserveNativeFailedWorkspace`. - Leases that settled before this change have no early receipt. They continue to wait for their release. - Nothing compares the refusal message exactly. Matchers use substrings, and run failure classification does not use the message. - A follow-up still fails closed if the predecessor's stop was never proven. This change does not add a recoverable wait. - The reason the release was held in production is not addressed here. - No migration and no API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, sub-agents, Honeycomb trace queries) and extended thinking. One sub-agent wrote the change. A second session reviewed it before this pull request. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
132 lines
7.4 KiB
TypeScript
132 lines
7.4 KiB
TypeScript
import { and, eq } from "drizzle-orm";
|
|
import { environmentLeases, type Db } from "@paperclipai/db";
|
|
import { retiredReusableLeasePredecessors } from "./environment-lease-handoff.js";
|
|
|
|
type Lease = typeof environmentLeases.$inferSelect;
|
|
|
|
type LeaseIdentity = {
|
|
id: string; companyId: string; heartbeatRunId: string | null;
|
|
provider: string | null; providerLeaseId: string | null;
|
|
};
|
|
|
|
/** Bind a provider receipt to the exact host-owned lease and run. Old plugins
|
|
* return void; that remains supported but grants no continuation authority. */
|
|
export function remoteTerminationReceipt(lease: LeaseIdentity, value: unknown) {
|
|
const receipt = value as { providerLeaseId?: unknown; state?: unknown } | null;
|
|
if (!lease.heartbeatRunId || !lease.provider || lease.provider === "local" ||
|
|
!lease.providerLeaseId || receipt?.providerLeaseId !== lease.providerLeaseId ||
|
|
!["stopped", "destroyed"].includes(String(receipt?.state))) return undefined;
|
|
return {
|
|
schema: "paperclip.remote-termination.v1", companyId: lease.companyId,
|
|
runId: lease.heartbeatRunId, leaseId: lease.id, provider: lease.provider,
|
|
providerLeaseId: lease.providerLeaseId, state: receipt!.state,
|
|
confirmedAt: new Date().toISOString(),
|
|
};
|
|
}
|
|
|
|
type ReceiptLease = LeaseIdentity & {
|
|
releasedAt: unknown; cleanupStatus: string | null; status: string;
|
|
metadata: Record<string, unknown> | null;
|
|
};
|
|
|
|
/** The lease carries a provider stop receipt bound to its exact company, run,
|
|
* lease, provider and physical task. It proves execution there stopped, not
|
|
* that the lease's resources were released. */
|
|
export function hasBoundRemoteStopReceipt(lease: Pick<ReceiptLease, keyof LeaseIdentity | "metadata">): boolean {
|
|
const receipt = lease.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined;
|
|
return Boolean(receipt && receipt.schema === "paperclip.remote-termination.v1" &&
|
|
receipt.companyId === lease.companyId && receipt.runId === lease.heartbeatRunId &&
|
|
receipt.leaseId === lease.id && receipt.provider === lease.provider &&
|
|
remoteTerminationReceipt(lease, receipt));
|
|
}
|
|
|
|
/** The lease was released with successful cleanup and its bound stop receipt.
|
|
* Resource cleanup gates use this; it never accepts an unreleased lease. */
|
|
export function hasRemoteTerminationReceipt(lease: ReceiptLease): boolean {
|
|
return Boolean(lease.releasedAt && lease.cleanupStatus === "success" &&
|
|
["released", "expired", "failed"].includes(lease.status) && hasBoundRemoteStopReceipt(lease));
|
|
}
|
|
|
|
/** Release or destroy of this lease has started: plugin release and destroy
|
|
* record their cleanup method first, and task settlement, startup cleanup and
|
|
* interrupted-creation cleanup mark the task retiring. The provider can then be
|
|
* releasing the task's resources while the lease row is still active. */
|
|
function leaseReleaseStarted(lease: Pick<ReceiptLease, "metadata">): boolean {
|
|
return lease.metadata?.pluginLeaseCleanupMethod !== undefined || lease.metadata?.runnerTaskRetiring === true;
|
|
}
|
|
|
|
/** Execution on this lease has stopped: it was released with its termination
|
|
* receipt, or it is still active, its release has not started, and its plugin
|
|
* task recorded the provider's stop when it settled. Release can be held long
|
|
* after that stop (for example by a pending workspace export); once release
|
|
* starts, it owns the receipt until the lease is released. */
|
|
function remoteLeaseExecutionStopped(lease: ReceiptLease): boolean {
|
|
return hasRemoteTerminationReceipt(lease) ||
|
|
lease.status === "active" && !lease.releasedAt && !leaseReleaseStarted(lease) && hasBoundRemoteStopReceipt(lease);
|
|
}
|
|
|
|
/** Workspace recovery saved this run's files, re-stopped its sandbox, and then
|
|
* queued the stopped sandbox for destroy. The destroy is resource cleanup: the
|
|
* stop receipt it carried forward still proves no provider can run there. */
|
|
export function hasRecoveredCleanupStopReceipt(lease: LeaseIdentity & {
|
|
releasedAt: unknown; status: string; metadata: Record<string, unknown> | null;
|
|
}): boolean {
|
|
const cleanup = lease.metadata?.workspaceRecoveryCleanup as Record<string, unknown> | undefined;
|
|
if (!cleanup || cleanup.runId !== lease.heartbeatRunId || !lease.releasedAt ||
|
|
!["pending_cleanup", "released", "expired", "failed"].includes(lease.status)) return false;
|
|
return hasRemoteTerminationReceipt({ ...lease, status: "released", cleanupStatus: "success",
|
|
metadata: { remoteExecutionTermination: cleanup.stopReceipt } });
|
|
}
|
|
|
|
export function remoteLeaseCleanupScope(lease: Pick<LeaseIdentity, "provider" | "providerLeaseId">) {
|
|
return lease.provider && lease.provider !== "local" && lease.providerLeaseId
|
|
? JSON.stringify([lease.provider, lease.providerLeaseId]) : undefined;
|
|
}
|
|
|
|
export async function resolveRemoteTerminationLease(db: Db, lease: Lease): Promise<Lease> {
|
|
if (hasRemoteTerminationReceipt(lease) || lease.status !== "expired" ||
|
|
lease.leasePolicy !== "reuse_by_environment" || !lease.provider || !lease.providerLeaseId) return lease;
|
|
const history = await db.select().from(environmentLeases).where(and(
|
|
eq(environmentLeases.provider, lease.provider), eq(environmentLeases.providerLeaseId, lease.providerLeaseId),
|
|
));
|
|
const current = history.find(row => row.id === lease.id);
|
|
if (!current || current.companyId !== lease.companyId || current.heartbeatRunId !== lease.heartbeatRunId) return lease;
|
|
const tips = history.filter(row => !Object.hasOwn(row.metadata ?? {}, "reusableLeaseSuccessor"));
|
|
if (tips.length !== 1) return current;
|
|
const tip = tips[0]!;
|
|
const receipt = tip.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined;
|
|
if (!hasRemoteTerminationReceipt(tip) || receipt?.state !== "destroyed") return current;
|
|
const predecessors = retiredReusableLeasePredecessors(tip, history.filter(row => row.id !== tip.id));
|
|
if (!predecessors?.some(row => row.id === current.id)) return current;
|
|
const termination = remoteTerminationReceipt(current, { providerLeaseId: current.providerLeaseId, state: "destroyed" });
|
|
if (!termination) return current;
|
|
return { ...current, metadata: { ...current.metadata, remoteExecutionTermination: {
|
|
...termination, confirmedAt: receipt.confirmedAt,
|
|
} } };
|
|
}
|
|
|
|
async function runRemoteLeases(db: Db, companyId: string, runId: string) {
|
|
const leases = await db.select().from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId),
|
|
));
|
|
return Promise.all(leases.map(lease => resolveRemoteTerminationLease(db, lease)));
|
|
}
|
|
|
|
/** Cleanup scopes of a run whose every lease has stopped execution, or null. */
|
|
export async function stoppedRemoteCleanupScopes(db: Db, companyId: string, runId: string) {
|
|
const leases = await runRemoteLeases(db, companyId, runId);
|
|
if (leases.length === 0 || !leases.every(remoteLeaseExecutionStopped)) return null;
|
|
return [...new Set(leases.map(lease => remoteLeaseCleanupScope(lease)!))];
|
|
}
|
|
|
|
/** Every lease of the run has stopped execution. Its resources can still be held. */
|
|
export async function remoteExecutionHasStopped(db: Db, companyId: string, runId: string) {
|
|
return await stoppedRemoteCleanupScopes(db, companyId, runId) !== null;
|
|
}
|
|
|
|
/** Every lease of the run was released with its termination receipt. */
|
|
export async function remoteExecutionHasReleased(db: Db, companyId: string, runId: string) {
|
|
const leases = await runRemoteLeases(db, companyId, runId);
|
|
return leases.length > 0 && leases.every(hasRemoteTerminationReceipt);
|
|
}
|