mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A plugin environment runs each Paperclip Runner turn as a provider task, and a follow-up run can continue the same Claude/ACPX conversation from the saved checkpoint > - Core authorizes that continuation only when the predecessor run's remote execution has stopped (`remoteExecutionHasStopped`), so two Runners never write one checkpoint > - The predecessor's own settlement already proves the stop: the provider reports `executionStopped`, and Core then completes the task > - On a per-turn (ephemeral) lease, Core records the stop receipt only when it releases the lease, and that release can be held for a long time (for example by a pending workspace export) > - In that window, an ordinary follow-up fails with `adapter_failed: Plugin Runner retained session is not authorized`, even though the stop is already proven > - This pull request records the stop receipt when the stop is proven, and the stop check accepts it before the release > - The benefit is that follow-ups continue as soon as the predecessor is proven stopped, and the release gates keep their meaning ## Linked Issues or Issue Description Fixes #21211 Refs #21214 and #21223. Those cover the creation fence for same-run recovery, which still requires a released lease. Production evidence for this change: in both observed failures, the predecessor was a per-turn plugin run. Its `task.settle` completed with all provider calls successful. Core then did not release its lease for 30 minutes or more, and no release call was made in that time. The follow-up failed in checkpoint recovery (`recoverPluginRunnerCheckpoint` → `resolvePluginTaskResume`) within seconds of admission. ## What Changed - `server/src/services/environment-task-runtime.ts` - After a recorded provider stop, `complete` writes the bound stop receipt (`remoteTerminationReceipt(..., { state: "stopped" })`) on the current lease for every lease policy. Before, it did this only for `reuse_by_environment` leases. Reusable tasks still confirm every owner and still move retained owners to released. - New work on a lease (`submit`, `copilot_metadata`, first portable resume) removes the stop receipt in the same write that clears `runnerTaskExecutionStopped`. A replacement physical task does not copy the old receipt. - `server/src/services/remote-execution-termination.ts` - New `hasBoundRemoteStopReceipt` checks only the receipt binding (company, run, lease, provider and physical task). `hasRemoteTerminationReceipt` uses it and keeps its released-only semantics. Resource cleanup gates do not change. - `remoteExecutionHasStopped` and `stoppedRemoteCleanupScopes` accept a lease that is released with its receipt, or that is still `active` (not released) with a bound stop receipt. A lease whose release started and did not settle does not count. After release starts, the release owns the receipt. - New `remoteExecutionHasReleased` gives the release-based answer. - `server/src/services/environment-run-orchestrator.ts`: same-run plugin recovery uses `remoteExecutionHasReleased`. The same-run resume check after it requires released receipts, so its behavior does not change. - `resolvePluginTaskResume` keeps its checks and their order. Its refusal now names the first failing check: `Plugin Runner retained session is not authorized (reason: <code>)`, for example `predecessor_not_stopped`, `scope_mismatch` or `identity_mismatch`. The old message stays as the prefix, and no ids go into it. The stale-availability recheck in `environment-task-runtime.ts` names its refusal the same way. Production telemetry for #21211 could not tell which check fired. The next occurrence will show it. - `doc/execution-semantics.md` and `doc/testing-native-remote-continuity.md` describe the stop receipt and when it is recorded. Caller audit for the changed meaning of `remoteExecutionHasStopped`: | Caller | Needs | Decision | | --- | --- | --- | | `agent-instruction-working-copies.ts` | stop | Accepts. Directory cleanup still requires a released "destroyed" receipt. | | `environment-run-orchestrator.ts` | release | Moved to `remoteExecutionHasReleased`. | | `plugin-task-resume.ts` | stop | Accepts. This is the fix. | | `native-runtime/native-runner-ownership.ts` | stop | Accepts. `hasPendingRunCleanup` still counts the active lease as pending. | | `native-runtime/preserved-native-result-finalization.ts` | stop | Accepts. Workspace recovery has its own gate. | | `heartbeat/case-handoff-lifecycle.ts` | stop | Accepts. `hasPendingRunCleanup` still holds active leases. | | `heartbeat/run-control.ts` (3 sites) | stop | Accepts. Stop acknowledgement and quarantined ownership retirement. Holds and blockers still apply on the normal path. | | `heartbeat/runtime-execution.ts` | stop | Accepts. Release errors still fail it. | ## Verification From `server/`, after `node scripts/agent-preflight.mjs`: - `npx vitest run src/__tests__/environment-task-authority.integration.test.ts`: 10/10 pass. The 2 new tests fail without the fix. - The plugin task and environment runtime suites (`environment-task-authority.integration`, `plugin-runner-task.integration`, `plugin-warm-tasks.integration`, `environment-task-runtime`, `plugin-task-source-recovery.integration`, `environment-runtime`): 6 files, 366 pass. - The audited caller suites (`agent-instruction-working-copies`, case handoff legacy and native, `environment-run-orchestrator`, `heartbeat-case-handoff`, `explicit-native-continuation`, `run-control`, `native-remote-retry-admission`, `preserved-native-response-wait`): 9 files, 806 pass. - `pnpm --filter @paperclipai/server typecheck`: pass. The new integration tests cover: - `complete` writes a bound stop receipt on a still-active ephemeral lease. `remoteExecutionHasStopped` is true and `hasRemoteTerminationReceipt` is still false. - `resolvePluginTaskResume` (inspect and launch) authorizes a follow-up's new task, and the real host `read_state` checkpoint read succeeds, while the predecessor's release is held. - Refusals: no provider stop, a receipt for another provider task, an unstopped sibling lease, and a release in progress. - New work on the lease removes the receipt. Release overwrites the receipt with its own receipt. Warm owners keep their behavior. I did not add an executor-level test. `native-session-executor.test.ts` mocks `resolvePluginTaskResume`, so the integration test runs the real checkpoint read path instead. I did not run the full suite locally. CI runs it. ## Risks - `remoteExecutionHasStopped` now returns true earlier for a run whose leases are still held. Each caller is audited above. Callers that need released resources use the release-based predicates. - The stop receipt can now be on an `active` lease. The SQL readers of `remoteExecutionTermination` (cost accounting, execution blockers, idle-work proof, orphan settlement) also require a released status. `pinFailedObservedWorkspace` treats an active ephemeral lease with a receipt as not pinnable. That path applies only to the remote-workspace flow and falls back to `preserveNativeFailedWorkspace`. - Leases that settled before this change have no early receipt. They continue to wait for their release. - Nothing compares the refusal message exactly. Matchers use substrings, and run failure classification does not use the message. - A follow-up still fails closed if the predecessor's stop was never proven. This change does not add a recoverable wait. - The reason the release was held in production is not addressed here. - No migration and no API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, sub-agents, Honeycomb trace queries) and extended thinking. One sub-agent wrote the change. A second session reviewed it before this pull request. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
94 lines
5.9 KiB
TypeScript
94 lines
5.9 KiB
TypeScript
import { and, eq, or } from "drizzle-orm";
|
|
import { environmentLeases, heartbeatRuns, type Db } from "@paperclipai/db";
|
|
import { hasRemoteTerminationReceipt, remoteExecutionHasStopped } from "./remote-execution-termination.js";
|
|
|
|
/** The first check that refused a retained plugin task resume, named in the error. */
|
|
export type PluginTaskResumeRefusal =
|
|
| "next_lease_missing"
|
|
| "predecessor_candidate_missing"
|
|
| "predecessor_candidate_ambiguous"
|
|
| "predecessor_task_missing"
|
|
| "identity_mismatch"
|
|
| "provider_mismatch"
|
|
| "next_task_missing"
|
|
| "same_provider_task"
|
|
| "predecessor_run_active"
|
|
| "scope_mismatch"
|
|
| "next_lease_inactive"
|
|
| "next_run_not_running"
|
|
| "predecessor_not_stopped"
|
|
| "checkpoint_status_unconfirmed"
|
|
| "resume_task_mismatch";
|
|
|
|
/** Keeps the original message as a prefix so existing matchers still apply. */
|
|
export function pluginTaskResumeRefused(reason: PluginTaskResumeRefusal) {
|
|
return new Error(`Plugin Runner retained session is not authorized (reason: ${reason})`);
|
|
}
|
|
|
|
type ResumeRow = { lease: typeof environmentLeases.$inferSelect; run: typeof heartbeatRuns.$inferSelect };
|
|
type PriorIdentity = { runnerInstanceId: string; environmentLeaseId: string; runId: string; normalizedSessionId: string };
|
|
|
|
const TERMINAL_RUN_STATUSES = ["succeeded", "interrupted", "failed", "cancelled", "timed_out"];
|
|
|
|
/** Cheap, synchronous checks in evaluation order; the stop proof is read only after these pass. */
|
|
function resumeRefusal(priorIdentity: PriorIdentity, next: ResumeRow | undefined, candidates: ResumeRow[]): PluginTaskResumeRefusal | null {
|
|
if (!next) return "next_lease_missing";
|
|
if (candidates.length === 0) return "predecessor_candidate_missing";
|
|
if (candidates.length > 1) return "predecessor_candidate_ambiguous";
|
|
const old = candidates[0]!;
|
|
const identity = old.lease.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined;
|
|
if (!old.lease.providerLeaseId) return "predecessor_task_missing";
|
|
if (!identity || identity.runnerId !== priorIdentity.runnerInstanceId || identity.leaseId !== priorIdentity.environmentLeaseId ||
|
|
identity.sessionId !== priorIdentity.normalizedSessionId || identity.runId !== priorIdentity.runId) return "identity_mismatch";
|
|
if (old.lease.metadata?.driver !== "plugin" || next.lease.metadata?.driver !== "plugin") return "provider_mismatch";
|
|
if (!next.lease.providerLeaseId) return "next_task_missing";
|
|
if (next.lease.providerLeaseId === old.lease.providerLeaseId) return "same_provider_task";
|
|
if (old.run.id !== next.run.id && !TERMINAL_RUN_STATUSES.includes(old.run.status)) return "predecessor_run_active";
|
|
if (old.run.agentId !== next.run.agentId || old.lease.issueId !== next.lease.issueId ||
|
|
old.lease.executionWorkspaceId !== next.lease.executionWorkspaceId ||
|
|
!old.lease.environmentId || old.lease.environmentId !== next.lease.environmentId) return "scope_mismatch";
|
|
if (!next.lease.provider?.startsWith("plugin:") || old.lease.provider !== next.lease.provider ||
|
|
old.lease.metadata?.pluginId !== next.lease.metadata?.pluginId ||
|
|
old.lease.metadata?.driverKey !== next.lease.metadata?.driverKey) return "provider_mismatch";
|
|
if (next.lease.status !== "active" || next.lease.releasedAt ||
|
|
(next.lease.expiresAt && next.lease.expiresAt.getTime() <= Date.now())) return "next_lease_inactive";
|
|
if (next.run.status !== "running") return "next_run_not_running";
|
|
return null;
|
|
}
|
|
|
|
/** A saved provider task can resume only within the same authorized agent workspace. */
|
|
export async function resolvePluginTaskResume(db: Db, input: {
|
|
companyId: string; leaseId: string;
|
|
priorIdentity: PriorIdentity;
|
|
/** Read-only discovery; launching still requires an attested retained checkpoint. */
|
|
inspectCheckpoint?: boolean;
|
|
}): Promise<string> {
|
|
const rows = await db.select({ lease: environmentLeases, run: heartbeatRuns }).from(environmentLeases)
|
|
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), eq(heartbeatRuns.companyId, input.companyId)))
|
|
.where(and(eq(environmentLeases.companyId, input.companyId),
|
|
or(eq(environmentLeases.id, input.leaseId), eq(heartbeatRuns.id, input.priorIdentity.runId))));
|
|
const next = rows.find(row => row.lease.id === input.leaseId);
|
|
// A retained PRP lease is logical; later provider tasks have different physical lease IDs.
|
|
const candidates = rows.filter(row => {
|
|
const identity = row.lease.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined;
|
|
const consumer = row.lease.metadata?.runnerTaskSourceConsumer as { leaseId?: unknown; runId?: unknown; resumeFrom?: unknown } | null | undefined;
|
|
return row.lease.id !== input.leaseId && row.run.id === input.priorIdentity.runId &&
|
|
row.lease.metadata?.portableTaskSuccessor === undefined &&
|
|
(!Object.hasOwn(row.lease.metadata ?? {}, "runnerTaskSourceConsumer") ||
|
|
consumer?.leaseId === input.leaseId && consumer.runId === next?.run.id && consumer.resumeFrom === row.lease.providerLeaseId) &&
|
|
(input.inspectCheckpoint || row.lease.metadata?.runnerTaskSessionRetained === true) &&
|
|
identity?.runnerId === input.priorIdentity.runnerInstanceId && identity.leaseId === input.priorIdentity.environmentLeaseId &&
|
|
identity.sessionId === input.priorIdentity.normalizedSessionId && identity.runId === input.priorIdentity.runId &&
|
|
!rows.some(successor => successor.lease.id !== input.leaseId &&
|
|
successor.lease.metadata?.runnerTaskResumeFrom === row.lease.providerLeaseId);
|
|
});
|
|
const refusal = resumeRefusal(input.priorIdentity, next, candidates);
|
|
if (refusal) throw pluginTaskResumeRefused(refusal);
|
|
const old = candidates[0]!;
|
|
const stopped = old.run.id === next!.run.id
|
|
? rows.filter(row => row.lease.id !== next!.lease.id).every(row => hasRemoteTerminationReceipt(row.lease))
|
|
: await remoteExecutionHasStopped(db, input.companyId, old.run.id);
|
|
if (!stopped) throw pluginTaskResumeRefused("predecessor_not_stopped");
|
|
return old.lease.providerLeaseId!;
|
|
}
|