mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(runtime): record a plugin task's stop receipt when its stop is proven (#21237)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A plugin environment runs each Paperclip Runner turn as a provider task, and a follow-up run can continue the same Claude/ACPX conversation from the saved checkpoint > - Core authorizes that continuation only when the predecessor run's remote execution has stopped (`remoteExecutionHasStopped`), so two Runners never write one checkpoint > - The predecessor's own settlement already proves the stop: the provider reports `executionStopped`, and Core then completes the task > - On a per-turn (ephemeral) lease, Core records the stop receipt only when it releases the lease, and that release can be held for a long time (for example by a pending workspace export) > - In that window, an ordinary follow-up fails with `adapter_failed: Plugin Runner retained session is not authorized`, even though the stop is already proven > - This pull request records the stop receipt when the stop is proven, and the stop check accepts it before the release > - The benefit is that follow-ups continue as soon as the predecessor is proven stopped, and the release gates keep their meaning ## Linked Issues or Issue Description Fixes #21211 Refs #21214 and #21223. Those cover the creation fence for same-run recovery, which still requires a released lease. Production evidence for this change: in both observed failures, the predecessor was a per-turn plugin run. Its `task.settle` completed with all provider calls successful. Core then did not release its lease for 30 minutes or more, and no release call was made in that time. The follow-up failed in checkpoint recovery (`recoverPluginRunnerCheckpoint` → `resolvePluginTaskResume`) within seconds of admission. ## What Changed - `server/src/services/environment-task-runtime.ts` - After a recorded provider stop, `complete` writes the bound stop receipt (`remoteTerminationReceipt(..., { state: "stopped" })`) on the current lease for every lease policy. Before, it did this only for `reuse_by_environment` leases. Reusable tasks still confirm every owner and still move retained owners to released. - New work on a lease (`submit`, `copilot_metadata`, first portable resume) removes the stop receipt in the same write that clears `runnerTaskExecutionStopped`. A replacement physical task does not copy the old receipt. - `server/src/services/remote-execution-termination.ts` - New `hasBoundRemoteStopReceipt` checks only the receipt binding (company, run, lease, provider and physical task). `hasRemoteTerminationReceipt` uses it and keeps its released-only semantics. Resource cleanup gates do not change. - `remoteExecutionHasStopped` and `stoppedRemoteCleanupScopes` accept a lease that is released with its receipt, or that is still `active` (not released) with a bound stop receipt. A lease whose release started and did not settle does not count. After release starts, the release owns the receipt. - New `remoteExecutionHasReleased` gives the release-based answer. - `server/src/services/environment-run-orchestrator.ts`: same-run plugin recovery uses `remoteExecutionHasReleased`. The same-run resume check after it requires released receipts, so its behavior does not change. - `resolvePluginTaskResume` keeps its checks and their order. Its refusal now names the first failing check: `Plugin Runner retained session is not authorized (reason: <code>)`, for example `predecessor_not_stopped`, `scope_mismatch` or `identity_mismatch`. The old message stays as the prefix, and no ids go into it. The stale-availability recheck in `environment-task-runtime.ts` names its refusal the same way. Production telemetry for #21211 could not tell which check fired. The next occurrence will show it. - `doc/execution-semantics.md` and `doc/testing-native-remote-continuity.md` describe the stop receipt and when it is recorded. Caller audit for the changed meaning of `remoteExecutionHasStopped`: | Caller | Needs | Decision | | --- | --- | --- | | `agent-instruction-working-copies.ts` | stop | Accepts. Directory cleanup still requires a released "destroyed" receipt. | | `environment-run-orchestrator.ts` | release | Moved to `remoteExecutionHasReleased`. | | `plugin-task-resume.ts` | stop | Accepts. This is the fix. | | `native-runtime/native-runner-ownership.ts` | stop | Accepts. `hasPendingRunCleanup` still counts the active lease as pending. | | `native-runtime/preserved-native-result-finalization.ts` | stop | Accepts. Workspace recovery has its own gate. | | `heartbeat/case-handoff-lifecycle.ts` | stop | Accepts. `hasPendingRunCleanup` still holds active leases. | | `heartbeat/run-control.ts` (3 sites) | stop | Accepts. Stop acknowledgement and quarantined ownership retirement. Holds and blockers still apply on the normal path. | | `heartbeat/runtime-execution.ts` | stop | Accepts. Release errors still fail it. | ## Verification From `server/`, after `node scripts/agent-preflight.mjs`: - `npx vitest run src/__tests__/environment-task-authority.integration.test.ts`: 10/10 pass. The 2 new tests fail without the fix. - The plugin task and environment runtime suites (`environment-task-authority.integration`, `plugin-runner-task.integration`, `plugin-warm-tasks.integration`, `environment-task-runtime`, `plugin-task-source-recovery.integration`, `environment-runtime`): 6 files, 366 pass. - The audited caller suites (`agent-instruction-working-copies`, case handoff legacy and native, `environment-run-orchestrator`, `heartbeat-case-handoff`, `explicit-native-continuation`, `run-control`, `native-remote-retry-admission`, `preserved-native-response-wait`): 9 files, 806 pass. - `pnpm --filter @paperclipai/server typecheck`: pass. The new integration tests cover: - `complete` writes a bound stop receipt on a still-active ephemeral lease. `remoteExecutionHasStopped` is true and `hasRemoteTerminationReceipt` is still false. - `resolvePluginTaskResume` (inspect and launch) authorizes a follow-up's new task, and the real host `read_state` checkpoint read succeeds, while the predecessor's release is held. - Refusals: no provider stop, a receipt for another provider task, an unstopped sibling lease, and a release in progress. - New work on the lease removes the receipt. Release overwrites the receipt with its own receipt. Warm owners keep their behavior. I did not add an executor-level test. `native-session-executor.test.ts` mocks `resolvePluginTaskResume`, so the integration test runs the real checkpoint read path instead. I did not run the full suite locally. CI runs it. ## Risks - `remoteExecutionHasStopped` now returns true earlier for a run whose leases are still held. Each caller is audited above. Callers that need released resources use the release-based predicates. - The stop receipt can now be on an `active` lease. The SQL readers of `remoteExecutionTermination` (cost accounting, execution blockers, idle-work proof, orphan settlement) also require a released status. `pinFailedObservedWorkspace` treats an active ephemeral lease with a receipt as not pinnable. That path applies only to the remote-workspace flow and falls back to `preserveNativeFailedWorkspace`. - Leases that settled before this change have no early receipt. They continue to wait for their release. - Nothing compares the refusal message exactly. Matchers use substrings, and run failure classification does not use the message. - A follow-up still fails closed if the predecessor's stop was never proven. This change does not add a recoverable wait. - The reason the release was held in production is not addressed here. - No migration and no API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, sub-agents, Honeycomb trace queries) and extended thinking. One sub-agent wrote the change. A second session reviewed it before this pull request. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b9308f9828
commit
afe86e9ded
7 files changed
+396
-59
No files matched your search
@@ -3483,6 +3483,23 @@ agent, workspace, retained session, and termination of prior execution. The
|
||||
checkpoint must match the exact Runner, lease, run, session, turn, and item.
|
||||
A missing restart claim does not remove these checks or authorize a fresh session.
|
||||
|
||||
A plugin task records its stop receipt when it settles. When the provider
|
||||
reports that the task stopped, completing the task writes a receipt bound to
|
||||
the exact company, run, lease, provider and task, even before the lease is
|
||||
released. Lease release can be held long after that, for example by a pending
|
||||
workspace export. A per-turn follow-up in a later run can then resume the
|
||||
retained checkpoint: for continuation, every lease of the predecessor run must
|
||||
be released with its termination receipt or still be active with this stop
|
||||
receipt and its release not started. Once release or destroy of a lease starts,
|
||||
release owns the receipt: the lease does not count until release settles with
|
||||
its own receipt, because the provider can be releasing the task's resources
|
||||
while the lease is still active. Dispatching new work on a lease removes its
|
||||
stop receipt. Resource cleanup gates, and a same-run retry's replacement of earlier
|
||||
leases, still require the released termination receipt.
|
||||
When the server refuses the resume, the error keeps the message
|
||||
`Plugin Runner retained session is not authorized` and names the first failed
|
||||
check, for example `(reason: predecessor_not_stopped)`.
|
||||
|
||||
These checks can refuse the exact checkpoint before the retry claims a new
|
||||
provider attempt. For example, the interrupted attempt can leave the provider
|
||||
state not suspended. The run error then names the interrupted attempt first, as
|
||||
|
||||
@@ -80,7 +80,13 @@ The executor tests cover Claude and Codex, including reopening blocked work.
|
||||
The lease-authority integration test uses PostgreSQL to check that checkpoint
|
||||
discovery cannot launch work or bypass a termination receipt. A stale negative
|
||||
availability flag requires a fresh provider status before the checkpoint can
|
||||
be read for continuation.
|
||||
be read for continuation. It also checks that a follow-up can resume while the
|
||||
predecessor's lease release is held, using the stop receipt that the task
|
||||
recorded when it settled. A receipt for another task, a second predecessor
|
||||
lease without stop proof, or a release that has started and not settled still
|
||||
refuses continuation. The test holds a real lease release inside the provider
|
||||
call and checks that the follow-up is refused until that release settles.
|
||||
Each refusal is checked by the reason that it names.
|
||||
|
||||
The executor can restore one quarantined journal in the same session scope.
|
||||
It requires a terminal prior run, completed commands, a terminal event, and an
|
||||
|
||||
@@ -3,7 +3,9 @@ import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { agents, companies, createDb, environmentLeases, environments, heartbeatRuns, plugins, sandboxCreationOperations } from "@paperclipai/db";
|
||||
import { resolvePluginTaskResume } from "../services/environment-task-runtime.js";
|
||||
import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
|
||||
import type { PluginTaskResumeRefusal } from "../services/plugin-task-resume.js";
|
||||
import { hasBoundRemoteStopReceipt, hasRemoteTerminationReceipt, remoteExecutionHasReleased, remoteExecutionHasStopped,
|
||||
remoteTerminationReceipt } from "../services/remote-execution-termination.js";
|
||||
import { environmentRuntimeService } from "../services/environment-runtime.js";
|
||||
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
|
||||
import { createPluginRunnerTaskOwner } from "../services/native-runtime/plugin-runner-task.js";
|
||||
@@ -12,6 +14,8 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const databaseSuite = support.supported ? describe : describe.skip;
|
||||
/** A resume refusal keeps its original message and names the check that failed. */
|
||||
const refused = (reason: PluginTaskResumeRefusal) => `Plugin Runner retained session is not authorized (reason: ${reason})`;
|
||||
|
||||
databaseSuite("environment task lease authority", () => {
|
||||
let db: ReturnType<typeof createDb>;
|
||||
@@ -227,17 +231,29 @@ databaseSuite("environment task lease authority", () => {
|
||||
await expect(authorize()).resolves.toBe("old-task");
|
||||
for (const field of ["runnerInstanceId", "environmentLeaseId", "normalizedSessionId", "runId"]) {
|
||||
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId,
|
||||
priorIdentity: { ...priorIdentity, [field]: randomUUID() } })).rejects.toThrow("not authorized");
|
||||
priorIdentity: { ...priorIdentity, [field]: randomUUID() } })).rejects.toThrow(refused("predecessor_candidate_missing"));
|
||||
}
|
||||
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, oldRunId));
|
||||
await expect(authorize()).rejects.toThrow("not authorized");
|
||||
await expect(authorize()).rejects.toThrow(refused("predecessor_run_active"));
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, oldRunId));
|
||||
await expect(resolvePluginTaskResume(db, { companyId: randomUUID(), leaseId: nextLeaseId, priorIdentity })).rejects.toThrow("not authorized");
|
||||
await expect(resolvePluginTaskResume(db, { companyId: randomUUID(), leaseId: nextLeaseId, priorIdentity })).rejects.toThrow(refused("next_lease_missing"));
|
||||
await db.update(environmentLeases).set({ metadata: { ...pin, pluginId: randomUUID() } }).where(eq(environmentLeases.id, nextLeaseId));
|
||||
await expect(authorize()).rejects.toThrow("not authorized");
|
||||
await expect(authorize()).rejects.toThrow(refused("provider_mismatch"));
|
||||
await db.update(environmentLeases).set({ metadata: pin }).where(eq(environmentLeases.id, nextLeaseId));
|
||||
await db.update(environmentLeases).set({ providerLeaseId: "old-task" }).where(eq(environmentLeases.id, nextLeaseId));
|
||||
await expect(authorize()).rejects.toThrow(refused("same_provider_task"));
|
||||
await db.update(environmentLeases).set({ providerLeaseId: "new-task" }).where(eq(environmentLeases.id, nextLeaseId));
|
||||
const otherEnvironmentId = randomUUID();
|
||||
await db.insert(environments).values({ id: otherEnvironmentId, name: `Fixture ${otherEnvironmentId}`, driver: "plugin", config: {} });
|
||||
await db.update(environmentLeases).set({ environmentId: otherEnvironmentId }).where(eq(environmentLeases.id, nextLeaseId));
|
||||
await expect(authorize()).rejects.toThrow(refused("scope_mismatch"));
|
||||
await db.update(environmentLeases).set({ environmentId }).where(eq(environmentLeases.id, nextLeaseId));
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, nextRunId));
|
||||
await expect(authorize()).rejects.toThrow(refused("next_run_not_running"));
|
||||
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, nextRunId));
|
||||
await expect(authorize()).resolves.toBe("old-task");
|
||||
await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, oldLeaseId));
|
||||
await expect(authorize()).rejects.toThrow("not authorized");
|
||||
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
|
||||
await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, oldLeaseId));
|
||||
let sessionStateAvailable = false;
|
||||
const operations: string[] = [];
|
||||
@@ -259,8 +275,8 @@ databaseSuite("environment task lease authority", () => {
|
||||
operation: { kind: "ensure_ownership", operationId: portableTargetLeaseId, sourceTaskId: "old-task",
|
||||
sourceGeneration: "source-generation", targetGeneration: "target-generation", policy: "ordinary", identity: checkpointIdentity },
|
||||
} } }).where(eq(environmentLeases.id, oldLeaseId));
|
||||
await expect(authorize()).rejects.toThrow("not authorized");
|
||||
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).rejects.toThrow("not authorized");
|
||||
await expect(authorize()).rejects.toThrow(refused("predecessor_candidate_missing"));
|
||||
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).rejects.toThrow(refused("predecessor_candidate_missing"));
|
||||
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
|
||||
kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task",
|
||||
} })).rejects.toThrow("not authorized");
|
||||
@@ -269,19 +285,19 @@ databaseSuite("environment task lease authority", () => {
|
||||
} })).rejects.toThrow("not authorized");
|
||||
expect(call).not.toHaveBeenCalled();
|
||||
await db.update(environmentLeases).set({ metadata: { ...source.metadata, runnerTaskSessionRetained: false } }).where(eq(environmentLeases.id, oldLeaseId));
|
||||
await expect(authorize()).rejects.toThrow("not authorized");
|
||||
await expect(authorize()).rejects.toThrow(refused("predecessor_candidate_missing"));
|
||||
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).resolves.toBe("old-task");
|
||||
const inspect = () => runtime.task({ companyId, leaseId: nextLeaseId, operation: {
|
||||
kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task",
|
||||
} });
|
||||
await expect(inspect()).rejects.toThrow("not authorized");
|
||||
await expect(authorize()).rejects.toThrow("not authorized");
|
||||
await expect(inspect()).rejects.toThrow(refused("checkpoint_status_unconfirmed"));
|
||||
await expect(authorize()).rejects.toThrow(refused("predecessor_candidate_missing"));
|
||||
// Discovery cannot launch, and cannot bypass the original termination receipt.
|
||||
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
|
||||
kind: "submit", preparedLaunchId: "fixture", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task",
|
||||
} })).rejects.toThrow("resume identity mismatch");
|
||||
await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, oldLeaseId));
|
||||
await expect(inspect()).rejects.toThrow("not authorized");
|
||||
await expect(inspect()).rejects.toThrow(refused("predecessor_not_stopped"));
|
||||
expect(operations).toEqual(["status"]);
|
||||
await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, oldLeaseId));
|
||||
sessionStateAvailable = true;
|
||||
@@ -350,4 +366,215 @@ databaseSuite("environment task lease authority", () => {
|
||||
|
||||
});
|
||||
|
||||
async function seedTaskPlugin(name: string) {
|
||||
const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), pluginId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name, issuePrefix: `T${companyId.slice(0, 7).replace(/-/g, "").toUpperCase()}` });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", adapterType: "paperclip_runner" });
|
||||
await db.insert(environments).values({ id: environmentId, name: `Fixture ${environmentId}`, driver: "plugin",
|
||||
config: { pluginKey: pluginId, driverKey: "tasks", driverConfig: {} } });
|
||||
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: `fixture-${name}`, version: "1.0.0", status: "ready", manifestJson: {
|
||||
capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }],
|
||||
} as never });
|
||||
const pin = { driver: "plugin", pluginId, driverKey: "tasks", environmentId, runnerTaskLease: true,
|
||||
runnerTaskProjectAuthValidation: { configurationId: "fixture", provider: "openai", projectIds: [] } };
|
||||
return { companyId, agentId, environmentId, pin };
|
||||
}
|
||||
|
||||
it("records a per-turn task's stop receipt when it settles, before its held lease release", async () => {
|
||||
const { companyId, agentId, environmentId, pin } = await seedTaskPlugin("per-turn-stop");
|
||||
const oldRunId = randomUUID(), nextRunId = randomUUID(), oldLeaseId = randomUUID(), nextLeaseId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values([
|
||||
{ id: oldRunId, companyId, agentId, invocationSource: "manual", status: "running" },
|
||||
{ id: nextRunId, companyId, agentId, invocationSource: "manual", status: "running" },
|
||||
]);
|
||||
const prior = { id: oldLeaseId, companyId, environmentId, heartbeatRunId: oldRunId, provider: "plugin:fixture:tasks", providerLeaseId: "old-task" };
|
||||
const oldIdentity = { runnerId: "runner", leaseId: oldLeaseId, runId: oldRunId, sessionId: "session", turnId: "prior-turn", itemId: "prior-item" };
|
||||
await db.insert(environmentLeases).values([
|
||||
{ ...prior, status: "active", leasePolicy: "ephemeral", metadata: { ...pin, runnerTaskDispatched: true,
|
||||
runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false, runnerTaskIdentity: oldIdentity } },
|
||||
{ id: nextLeaseId, companyId, environmentId, heartbeatRunId: nextRunId, provider: prior.provider, providerLeaseId: "new-task",
|
||||
status: "active", leasePolicy: "ephemeral", metadata: pin },
|
||||
]);
|
||||
let executionStopped = false;
|
||||
const call = vi.fn(async (_id: string, method: string, params: any) => method !== "environmentTask" ? undefined
|
||||
: ["read_state", "archive_state"].includes(params.operation.kind)
|
||||
? { kind: "state", taskId: params.taskId, state: { schema: "paperclip.runner.durable.state.v1", lifecycle: "suspended",
|
||||
...(params.operation.identity ?? params.operation.priorIdentity) } }
|
||||
: params.operation.kind === "status"
|
||||
? { kind: "status", taskId: params.taskId, phase: executionStopped ? "completed" : "running", executionStopped, sessionStateAvailable: executionStopped }
|
||||
: { kind: "accepted", taskId: params.taskId });
|
||||
const workers = { isRunning: () => true, getWorker: () => ({ supportedMethods: ["environmentTask", "environmentReleaseLease"] }), call } as unknown as PluginWorkerManager;
|
||||
const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers });
|
||||
const readLease = async (id: string) => (await db.select().from(environmentLeases).where(eq(environmentLeases.id, id)))[0]!;
|
||||
const priorIdentity = { runnerInstanceId: "runner", normalizedSessionId: "session", environmentLeaseId: oldLeaseId, runId: oldRunId };
|
||||
const checkpointIdentity = { ...priorIdentity, turnId: "prior-turn", itemId: "prior-item" };
|
||||
const runner = { runnerId: "runner", leaseId: oldLeaseId, sessionId: "session", runId: nextRunId,
|
||||
turnId: "turn", itemId: "item", harness: "codex", protocolMin: 1, protocolMax: 3 };
|
||||
const authorize = (inspectCheckpoint?: boolean) => resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint });
|
||||
|
||||
// Neither a running status nor a completion without a provider stop is stop proof.
|
||||
await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "status" } });
|
||||
await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "complete" } });
|
||||
expect((await readLease(oldLeaseId)).metadata).not.toHaveProperty("remoteExecutionTermination");
|
||||
expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(false);
|
||||
|
||||
// The turn checkpoint stops and completes the task; release of the lease is still held.
|
||||
executionStopped = true;
|
||||
await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "stop" } });
|
||||
await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "status" } });
|
||||
await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "complete" } });
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, oldRunId));
|
||||
const settled = await readLease(oldLeaseId);
|
||||
expect(settled).toMatchObject({ status: "active", releasedAt: null, cleanupStatus: null, metadata: {
|
||||
runnerTaskExecutionStopped: true, runnerTaskSessionRetained: true, runnerTaskDispatched: false,
|
||||
remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId, runId: oldRunId, leaseId: oldLeaseId,
|
||||
provider: prior.provider, providerLeaseId: "old-task", state: "stopped" } } });
|
||||
expect(hasBoundRemoteStopReceipt(settled)).toBe(true);
|
||||
expect(hasRemoteTerminationReceipt(settled)).toBe(false);
|
||||
expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(true);
|
||||
expect(await remoteExecutionHasReleased(db, companyId, oldRunId)).toBe(false);
|
||||
|
||||
// The follow-up can inspect and launch from the checkpoint while the predecessor is unreleased.
|
||||
await expect(authorize(true)).resolves.toBe("old-task");
|
||||
await expect(authorize()).resolves.toBe("old-task");
|
||||
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
|
||||
kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task",
|
||||
} })).resolves.toMatchObject({ kind: "state", state: { ...checkpointIdentity, lifecycle: "suspended" } });
|
||||
|
||||
// A receipt for another physical task, a second unstopped predecessor lease, or a release
|
||||
// that started without settling still refuses authority.
|
||||
await db.update(environmentLeases).set({ metadata: { ...settled.metadata, remoteExecutionTermination:
|
||||
remoteTerminationReceipt({ ...prior, providerLeaseId: "other-task" }, { providerLeaseId: "other-task", state: "stopped" }) } })
|
||||
.where(eq(environmentLeases.id, oldLeaseId));
|
||||
await expect(authorize(true)).rejects.toThrow(refused("predecessor_not_stopped"));
|
||||
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
|
||||
await db.update(environmentLeases).set({ metadata: settled.metadata }).where(eq(environmentLeases.id, oldLeaseId));
|
||||
const siblingId = randomUUID();
|
||||
await db.insert(environmentLeases).values({ id: siblingId, companyId, environmentId, heartbeatRunId: oldRunId,
|
||||
provider: prior.provider, providerLeaseId: "sibling-task", status: "active", leasePolicy: "ephemeral", metadata: pin });
|
||||
await expect(authorize(true)).rejects.toThrow(refused("predecessor_not_stopped"));
|
||||
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
|
||||
await db.delete(environmentLeases).where(eq(environmentLeases.id, siblingId));
|
||||
await db.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", releasedAt: new Date() })
|
||||
.where(eq(environmentLeases.id, oldLeaseId));
|
||||
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
|
||||
await db.update(environmentLeases).set({ status: "active", cleanupStatus: null, releasedAt: null }).where(eq(environmentLeases.id, oldLeaseId));
|
||||
await expect(authorize()).resolves.toBe("old-task");
|
||||
|
||||
// Dispatching new execution invalidates a stop receipt on that lease.
|
||||
const next = await readLease(nextLeaseId);
|
||||
await db.update(environmentLeases).set({ metadata: { ...next.metadata, runnerTaskExecutionStopped: true,
|
||||
remoteExecutionTermination: remoteTerminationReceipt(next, { providerLeaseId: "new-task", state: "stopped" }) } })
|
||||
.where(eq(environmentLeases.id, nextLeaseId));
|
||||
expect(await remoteExecutionHasStopped(db, companyId, nextRunId)).toBe(true);
|
||||
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
|
||||
kind: "submit", preparedLaunchId: "fixture", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task",
|
||||
} })).resolves.toMatchObject({ kind: "accepted" });
|
||||
expect((await readLease(nextLeaseId)).metadata).not.toHaveProperty("remoteExecutionTermination");
|
||||
expect(await remoteExecutionHasStopped(db, companyId, nextRunId)).toBe(false);
|
||||
|
||||
// Release still writes its own receipt over the settlement receipt.
|
||||
const releaseErrors: unknown[] = [];
|
||||
await runtime.releaseRunLeases(oldRunId, "released", (_id, error) => { releaseErrors.push(error); });
|
||||
expect(releaseErrors).toEqual([]);
|
||||
const released = await readLease(oldLeaseId);
|
||||
expect(released).toMatchObject({ status: "released", cleanupStatus: "success",
|
||||
metadata: { remoteExecutionTermination: { leaseId: oldLeaseId, providerLeaseId: "old-task", state: "stopped" } } });
|
||||
expect(hasRemoteTerminationReceipt(released)).toBe(true);
|
||||
expect(await remoteExecutionHasReleased(db, companyId, oldRunId)).toBe(true);
|
||||
});
|
||||
|
||||
it("stops counting a settled task's stop receipt once its lease release starts", async () => {
|
||||
const { companyId, agentId, environmentId, pin } = await seedTaskPlugin("release-owns-receipt");
|
||||
const oldRunId = randomUUID(), nextRunId = randomUUID(), oldLeaseId = randomUUID(), nextLeaseId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values([
|
||||
{ id: oldRunId, companyId, agentId, invocationSource: "manual", status: "succeeded" },
|
||||
{ id: nextRunId, companyId, agentId, invocationSource: "manual", status: "running" },
|
||||
]);
|
||||
const prior = { id: oldLeaseId, companyId, environmentId, heartbeatRunId: oldRunId, provider: "plugin:fixture:tasks", providerLeaseId: "old-task" };
|
||||
// The predecessor's task settled in its run: stopped, session retained, stop receipt recorded.
|
||||
const settledMetadata = { ...pin, runnerTaskDispatched: false, runnerTaskExecutionStopped: true, runnerTaskSessionRetained: true,
|
||||
runnerTaskIdentity: { runnerId: "runner", leaseId: oldLeaseId, runId: oldRunId, sessionId: "session", turnId: "turn", itemId: "item" },
|
||||
remoteExecutionTermination: remoteTerminationReceipt(prior, { providerLeaseId: "old-task", state: "stopped" }) };
|
||||
await db.insert(environmentLeases).values([
|
||||
{ ...prior, status: "active", leasePolicy: "ephemeral", metadata: settledMetadata },
|
||||
{ id: nextLeaseId, companyId, environmentId, heartbeatRunId: nextRunId, provider: prior.provider, providerLeaseId: "new-task",
|
||||
status: "active", leasePolicy: "ephemeral", metadata: pin },
|
||||
]);
|
||||
let releaseEntered!: () => void, finishRelease!: () => void;
|
||||
const entered = new Promise<void>(resolve => { releaseEntered = resolve; });
|
||||
const held = new Promise<void>(resolve => { finishRelease = resolve; });
|
||||
const call = vi.fn(async (_id: string, method: string) => {
|
||||
if (method !== "environmentReleaseLease") throw new Error(`unexpected ${method}`);
|
||||
releaseEntered();
|
||||
await held;
|
||||
});
|
||||
const workers = { isRunning: () => true, getWorker: () => ({ supportedMethods: ["environmentTask", "environmentReleaseLease"] }), call } as unknown as PluginWorkerManager;
|
||||
const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers });
|
||||
const priorIdentity = { runnerInstanceId: "runner", normalizedSessionId: "session", environmentLeaseId: oldLeaseId, runId: oldRunId };
|
||||
const authorize = (inspectCheckpoint?: boolean) => resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint });
|
||||
await expect(authorize()).resolves.toBe("old-task");
|
||||
|
||||
// Each marker that release, destroy, or startup and creation cleanup write first takes the receipt over.
|
||||
for (const marker of [{ pluginLeaseCleanupMethod: "release" }, { pluginLeaseCleanupMethod: "destroy" }, { runnerTaskRetiring: true }]) {
|
||||
await db.update(environmentLeases).set({ metadata: { ...settledMetadata, ...marker } }).where(eq(environmentLeases.id, oldLeaseId));
|
||||
expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(false);
|
||||
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
|
||||
}
|
||||
await db.update(environmentLeases).set({ metadata: settledMetadata }).where(eq(environmentLeases.id, oldLeaseId));
|
||||
|
||||
// Hold the real release inside the provider call: the lease is still active with its settlement receipt.
|
||||
const releaseErrors: unknown[] = [];
|
||||
const releasing = runtime.releaseRunLeases(oldRunId, "released", (_id, error) => { releaseErrors.push(error); });
|
||||
await entered;
|
||||
const inFlight = (await db.select().from(environmentLeases).where(eq(environmentLeases.id, oldLeaseId)))[0]!;
|
||||
expect(inFlight).toMatchObject({ status: "active", releasedAt: null,
|
||||
metadata: { pluginLeaseCleanupMethod: "release", runnerTaskRetiring: true } });
|
||||
expect(hasBoundRemoteStopReceipt(inFlight)).toBe(true);
|
||||
expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(false);
|
||||
await expect(authorize(true)).rejects.toThrow(refused("predecessor_not_stopped"));
|
||||
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
|
||||
|
||||
// Release settles with its own receipt, which authorizes the follow-up.
|
||||
finishRelease();
|
||||
await releasing;
|
||||
expect(releaseErrors).toEqual([]);
|
||||
const released = (await db.select().from(environmentLeases).where(eq(environmentLeases.id, oldLeaseId)))[0]!;
|
||||
expect(released).toMatchObject({ status: "released", cleanupStatus: "success" });
|
||||
expect(hasRemoteTerminationReceipt(released)).toBe(true);
|
||||
expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(true);
|
||||
await expect(authorize(true)).resolves.toBe("old-task");
|
||||
await expect(authorize()).resolves.toBe("old-task");
|
||||
});
|
||||
|
||||
it("still records stop receipts for every warm owner and retires retained owners", async () => {
|
||||
const { companyId, agentId, environmentId, pin } = await seedTaskPlugin("warm-stop");
|
||||
const priorRunId = randomUUID(), currentRunId = randomUUID(), priorLeaseId = randomUUID(), currentLeaseId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values([
|
||||
{ id: priorRunId, companyId, agentId, invocationSource: "manual", status: "succeeded" },
|
||||
{ id: currentRunId, companyId, agentId, invocationSource: "manual", status: "running" },
|
||||
]);
|
||||
const shared = { companyId, environmentId, provider: "plugin:fixture:tasks", providerLeaseId: "warm-task", leasePolicy: "reuse_by_environment" as const };
|
||||
await db.insert(environmentLeases).values([
|
||||
{ ...shared, id: priorLeaseId, heartbeatRunId: priorRunId, status: "retained", metadata: pin },
|
||||
{ ...shared, id: currentLeaseId, heartbeatRunId: currentRunId, status: "active", metadata: { ...pin, runnerTaskDispatched: true } },
|
||||
]);
|
||||
const call = vi.fn(async (_id: string, _method: string, params: any) => params.operation.kind === "status"
|
||||
? { kind: "status", taskId: params.taskId, phase: "completed", executionStopped: true, sessionStateAvailable: false }
|
||||
: { kind: "accepted", taskId: params.taskId });
|
||||
const runtime = environmentRuntimeService(db, { pluginWorkerManager: { getWorker: () => ({ supportedMethods: ["environmentTask"] }), call } as unknown as PluginWorkerManager });
|
||||
await runtime.task({ companyId, leaseId: currentLeaseId, operation: { kind: "status" } });
|
||||
await runtime.task({ companyId, leaseId: currentLeaseId, operation: { kind: "complete" } });
|
||||
const leases = await db.select().from(environmentLeases).where(eq(environmentLeases.providerLeaseId, "warm-task"));
|
||||
const priorLease = leases.find(lease => lease.id === priorLeaseId)!, currentLease = leases.find(lease => lease.id === currentLeaseId)!;
|
||||
expect(priorLease).toMatchObject({ status: "released", cleanupStatus: "success",
|
||||
metadata: { remoteExecutionTermination: { leaseId: priorLeaseId, runId: priorRunId, state: "stopped" } } });
|
||||
expect(currentLease).toMatchObject({ status: "active",
|
||||
metadata: { remoteExecutionTermination: { leaseId: currentLeaseId, runId: currentRunId, state: "stopped" } } });
|
||||
expect(hasRemoteTerminationReceipt(priorLease)).toBe(true);
|
||||
expect(await remoteExecutionHasStopped(db, companyId, priorRunId)).toBe(true);
|
||||
expect(await remoteExecutionHasStopped(db, companyId, currentRunId)).toBe(true);
|
||||
expect(await remoteExecutionHasReleased(db, companyId, currentRunId)).toBe(false);
|
||||
});
|
||||
|
||||
});
|
||||
@@ -20,7 +20,7 @@ import { prepareRemoteWorkspace, readRemoteWorkspaceAuthority, remoteWorkspaceRe
|
||||
|
||||
import { environmentLeases, type Db } from "@paperclipai/db";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { remoteExecutionHasStopped } from "./remote-execution-termination.js";
|
||||
import { remoteExecutionHasReleased } from "./remote-execution-termination.js";
|
||||
import { remoteRecoveryLeaseMismatch, type RemoteRecoveryLeaseMismatchField } from "./native-runtime/remote-recovery-lease-mismatch.js";
|
||||
import { EnvironmentProviderUnavailableError, readTransientLeaseAcquisitionFailure } from "./environment-provider-unavailable-error.js";
|
||||
import { injectLeaseAcquisitionFault } from "./environment-lease-fault-injection.js";
|
||||
@@ -347,7 +347,9 @@ export function environmentRunOrchestrator(
|
||||
lease.metadata?.agentId !== input.agentId)) throw new Error("native_plugin_recovery_lease_mismatch");
|
||||
const errors: unknown[] = [];
|
||||
await environmentRuntime.releaseRunLeases(input.heartbeatRunId, "released", (_id, error) => { errors.push(error); });
|
||||
if (errors.length || (prior.length && !await remoteExecutionHasStopped(db, input.companyId, input.heartbeatRunId))) {
|
||||
// Same-run task resume requires every prior lease released with its receipt;
|
||||
// a stop recorded before a held release is not enough here.
|
||||
if (errors.length || (prior.length && !await remoteExecutionHasReleased(db, input.companyId, input.heartbeatRunId))) {
|
||||
throw new Error("native_plugin_recovery_termination_unconfirmed", { cause: errors[0] });
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { resolvePluginTaskResume } from "./plugin-task-resume.js";
|
||||
import { pluginTaskResumeRefused, resolvePluginTaskResume } from "./plugin-task-resume.js";
|
||||
export { resolvePluginTaskResume } from "./plugin-task-resume.js";
|
||||
import { attachSandboxCreation, environmentService } from "./environments.js";
|
||||
import { createOwnedSandbox, queueSandboxCreationCleanup, recordSandboxCreationResource } from "./sandbox-creation-operations.js";
|
||||
@@ -198,11 +198,13 @@ export async function executeEnvironmentTask(db: Db, workers: Pick<PluginWorkerM
|
||||
if (inspectCheckpoint && sources[0]!.metadata?.runnerTaskSessionRetained !== true) {
|
||||
const status = await executeEnvironmentTask(db, workers, { companyId: input.companyId,
|
||||
leaseId: sources[0]!.id, operation: { kind: "status" } });
|
||||
if (status.kind !== "status" || status.executionStopped !== true || status.sessionStateAvailable !== true ||
|
||||
resumeFrom !== await resolvePluginTaskResume(db, { companyId: input.companyId, leaseId: lease.id,
|
||||
priorIdentity: { runnerInstanceId: runner.runnerId, environmentLeaseId: runner.leaseId,
|
||||
runId: priorRunId, normalizedSessionId: runner.sessionId } })) {
|
||||
throw new Error("Plugin Runner retained session is not authorized");
|
||||
if (status.kind !== "status" || status.executionStopped !== true || status.sessionStateAvailable !== true) {
|
||||
throw pluginTaskResumeRefused("checkpoint_status_unconfirmed");
|
||||
}
|
||||
if (resumeFrom !== await resolvePluginTaskResume(db, { companyId: input.companyId, leaseId: lease.id,
|
||||
priorIdentity: { runnerInstanceId: runner.runnerId, environmentLeaseId: runner.leaseId,
|
||||
runId: priorRunId, normalizedSessionId: runner.sessionId } })) {
|
||||
throw pluginTaskResumeRefused("resume_task_mismatch");
|
||||
}
|
||||
}
|
||||
stoppedSourceCheckpoint = true;
|
||||
@@ -310,6 +312,7 @@ export async function executeEnvironmentTask(db: Db, workers: Pick<PluginWorkerM
|
||||
}
|
||||
if (operation.kind === "submit" || operation.kind === "copilot_metadata" || firstPortableResume) {
|
||||
// No ticket, headers, environment values, or launch paths belong in metadata.
|
||||
// New execution invalidates this lease's earlier stop receipt with its stop flag.
|
||||
const dispatched = {
|
||||
runnerTaskLease: true, runnerTaskDispatched: true,
|
||||
runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false,
|
||||
@@ -320,7 +323,7 @@ export async function executeEnvironmentTask(db: Db, workers: Pick<PluginWorkerM
|
||||
} : {}),
|
||||
};
|
||||
const marked = await db.update(environmentLeases).set({
|
||||
metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskAdmissionRejection') || ${JSON.stringify(dispatched)}::jsonb`,
|
||||
metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskAdmissionRejection' - 'remoteExecutionTermination') || ${JSON.stringify(dispatched)}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
|
||||
eq(environmentLeases.status, "active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`,
|
||||
requiredProjectAuthProof ? sql`${environmentLeases.metadata}->'runnerTaskProjectAuthValidation' = ${JSON.stringify(requiredProjectAuthProof)}::jsonb` : undefined,
|
||||
@@ -420,7 +423,7 @@ export async function executeEnvironmentTask(db: Db, workers: Pick<PluginWorkerM
|
||||
});
|
||||
if (!fresh.providerLeaseId || fresh.providerLeaseId === taskId) throw new Error("Plugin replacement task identity unavailable");
|
||||
const {runnerTaskIdentity:_identity,runnerTaskPriorIdentity:_prior,runnerTaskCredentialHandoff:_handoff,
|
||||
runnerTaskResumeFrom:_resume,runnerTaskPreviousLeaseId:_previous,...rest}=metadata;
|
||||
runnerTaskResumeFrom:_resume,runnerTaskPreviousLeaseId:_previous,remoteExecutionTermination:_stopped,...rest}=metadata;
|
||||
const replacementMetadata={...rest,providerMetadata:fresh.metadata ?? {},runnerTaskReused:false,runnerTaskDispatched:false,
|
||||
runnerTaskExecutionStopped:false,runnerTaskSessionRetained:false};
|
||||
try {
|
||||
@@ -557,10 +560,12 @@ export async function executeEnvironmentTask(db: Db, workers: Pick<PluginWorkerM
|
||||
if (operation.kind === "complete") {
|
||||
if (freshWorkspace && metadata.runnerTaskExecutionStopped === true)
|
||||
await releaseProviderRecoveryTransfers(input.companyId, freshWorkspace);
|
||||
// A physical task can back several run leases. Confirm retirement for every prior owner.
|
||||
if (lease.leasePolicy === "reuse_by_environment" && metadata.runnerTaskExecutionStopped === true) {
|
||||
const owners=await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId,input.companyId),
|
||||
eq(environmentLeases.provider,lease.provider!),eq(environmentLeases.providerLeaseId,taskId)));
|
||||
// The provider attested this task stopped: record the stop receipt now, not when
|
||||
// the lease's resources are released. A reusable physical task can back several
|
||||
// run leases, so confirm retirement for every owner.
|
||||
if (metadata.runnerTaskExecutionStopped === true) {
|
||||
const owners=lease.leasePolicy === "reuse_by_environment" ? await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId,input.companyId),
|
||||
eq(environmentLeases.provider,lease.provider!),eq(environmentLeases.providerLeaseId,taskId))) : [lease];
|
||||
for(const owner of owners) {
|
||||
const receipt=remoteTerminationReceipt(owner,{providerLeaseId:taskId,state:"stopped"});
|
||||
if(!receipt) continue;
|
||||
|
||||
@@ -2,10 +2,64 @@ import { and, eq, or } from "drizzle-orm";
|
||||
import { environmentLeases, heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
import { hasRemoteTerminationReceipt, remoteExecutionHasStopped } from "./remote-execution-termination.js";
|
||||
|
||||
/** The first check that refused a retained plugin task resume, named in the error. */
|
||||
export type PluginTaskResumeRefusal =
|
||||
| "next_lease_missing"
|
||||
| "predecessor_candidate_missing"
|
||||
| "predecessor_candidate_ambiguous"
|
||||
| "predecessor_task_missing"
|
||||
| "identity_mismatch"
|
||||
| "provider_mismatch"
|
||||
| "next_task_missing"
|
||||
| "same_provider_task"
|
||||
| "predecessor_run_active"
|
||||
| "scope_mismatch"
|
||||
| "next_lease_inactive"
|
||||
| "next_run_not_running"
|
||||
| "predecessor_not_stopped"
|
||||
| "checkpoint_status_unconfirmed"
|
||||
| "resume_task_mismatch";
|
||||
|
||||
/** Keeps the original message as a prefix so existing matchers still apply. */
|
||||
export function pluginTaskResumeRefused(reason: PluginTaskResumeRefusal) {
|
||||
return new Error(`Plugin Runner retained session is not authorized (reason: ${reason})`);
|
||||
}
|
||||
|
||||
type ResumeRow = { lease: typeof environmentLeases.$inferSelect; run: typeof heartbeatRuns.$inferSelect };
|
||||
type PriorIdentity = { runnerInstanceId: string; environmentLeaseId: string; runId: string; normalizedSessionId: string };
|
||||
|
||||
const TERMINAL_RUN_STATUSES = ["succeeded", "interrupted", "failed", "cancelled", "timed_out"];
|
||||
|
||||
/** Cheap, synchronous checks in evaluation order; the stop proof is read only after these pass. */
|
||||
function resumeRefusal(priorIdentity: PriorIdentity, next: ResumeRow | undefined, candidates: ResumeRow[]): PluginTaskResumeRefusal | null {
|
||||
if (!next) return "next_lease_missing";
|
||||
if (candidates.length === 0) return "predecessor_candidate_missing";
|
||||
if (candidates.length > 1) return "predecessor_candidate_ambiguous";
|
||||
const old = candidates[0]!;
|
||||
const identity = old.lease.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined;
|
||||
if (!old.lease.providerLeaseId) return "predecessor_task_missing";
|
||||
if (!identity || identity.runnerId !== priorIdentity.runnerInstanceId || identity.leaseId !== priorIdentity.environmentLeaseId ||
|
||||
identity.sessionId !== priorIdentity.normalizedSessionId || identity.runId !== priorIdentity.runId) return "identity_mismatch";
|
||||
if (old.lease.metadata?.driver !== "plugin" || next.lease.metadata?.driver !== "plugin") return "provider_mismatch";
|
||||
if (!next.lease.providerLeaseId) return "next_task_missing";
|
||||
if (next.lease.providerLeaseId === old.lease.providerLeaseId) return "same_provider_task";
|
||||
if (old.run.id !== next.run.id && !TERMINAL_RUN_STATUSES.includes(old.run.status)) return "predecessor_run_active";
|
||||
if (old.run.agentId !== next.run.agentId || old.lease.issueId !== next.lease.issueId ||
|
||||
old.lease.executionWorkspaceId !== next.lease.executionWorkspaceId ||
|
||||
!old.lease.environmentId || old.lease.environmentId !== next.lease.environmentId) return "scope_mismatch";
|
||||
if (!next.lease.provider?.startsWith("plugin:") || old.lease.provider !== next.lease.provider ||
|
||||
old.lease.metadata?.pluginId !== next.lease.metadata?.pluginId ||
|
||||
old.lease.metadata?.driverKey !== next.lease.metadata?.driverKey) return "provider_mismatch";
|
||||
if (next.lease.status !== "active" || next.lease.releasedAt ||
|
||||
(next.lease.expiresAt && next.lease.expiresAt.getTime() <= Date.now())) return "next_lease_inactive";
|
||||
if (next.run.status !== "running") return "next_run_not_running";
|
||||
return null;
|
||||
}
|
||||
|
||||
/** A saved provider task can resume only within the same authorized agent workspace. */
|
||||
export async function resolvePluginTaskResume(db: Db, input: {
|
||||
companyId: string; leaseId: string;
|
||||
priorIdentity: { runnerInstanceId: string; environmentLeaseId: string; runId: string; normalizedSessionId: string };
|
||||
priorIdentity: PriorIdentity;
|
||||
/** Read-only discovery; launching still requires an attested retained checkpoint. */
|
||||
inspectCheckpoint?: boolean;
|
||||
}): Promise<string> {
|
||||
@@ -28,25 +82,12 @@ export async function resolvePluginTaskResume(db: Db, input: {
|
||||
!rows.some(successor => successor.lease.id !== input.leaseId &&
|
||||
successor.lease.metadata?.runnerTaskResumeFrom === row.lease.providerLeaseId);
|
||||
});
|
||||
const old = candidates.length === 1 ? candidates[0] : undefined;
|
||||
const identity = old?.lease.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined;
|
||||
if (!old || !next || !old.lease.providerLeaseId || !identity ||
|
||||
identity.runnerId !== input.priorIdentity.runnerInstanceId || identity.leaseId !== input.priorIdentity.environmentLeaseId ||
|
||||
identity.sessionId !== input.priorIdentity.normalizedSessionId || identity.runId !== input.priorIdentity.runId ||
|
||||
old.lease.metadata?.driver !== "plugin" || next.lease.metadata?.driver !== "plugin" ||
|
||||
!next.lease.providerLeaseId || next.lease.providerLeaseId === old.lease.providerLeaseId ||
|
||||
(old.run.id !== next.run.id && !["succeeded", "interrupted", "failed", "cancelled", "timed_out"].includes(old.run.status)) ||
|
||||
old.run.agentId !== next.run.agentId || old.lease.issueId !== next.lease.issueId ||
|
||||
old.lease.executionWorkspaceId !== next.lease.executionWorkspaceId ||
|
||||
!old.lease.environmentId || old.lease.environmentId !== next.lease.environmentId ||
|
||||
!next.lease.provider?.startsWith("plugin:") || old.lease.provider !== next.lease.provider ||
|
||||
old.lease.metadata?.pluginId !== next.lease.metadata?.pluginId || old.lease.metadata?.driverKey !== next.lease.metadata?.driverKey ||
|
||||
next.lease.status !== "active" || next.lease.releasedAt || next.run.status !== "running" ||
|
||||
(next.lease.expiresAt && next.lease.expiresAt.getTime() <= Date.now()) ||
|
||||
!(old.run.id === next.run.id
|
||||
? rows.filter(row => row.lease.id !== next.lease.id).every(row => hasRemoteTerminationReceipt(row.lease))
|
||||
: await remoteExecutionHasStopped(db, input.companyId, old.run.id))) {
|
||||
throw new Error("Plugin Runner retained session is not authorized");
|
||||
}
|
||||
return old.lease.providerLeaseId;
|
||||
const refusal = resumeRefusal(input.priorIdentity, next, candidates);
|
||||
if (refusal) throw pluginTaskResumeRefused(refusal);
|
||||
const old = candidates[0]!;
|
||||
const stopped = old.run.id === next!.run.id
|
||||
? rows.filter(row => row.lease.id !== next!.lease.id).every(row => hasRemoteTerminationReceipt(row.lease))
|
||||
: await remoteExecutionHasStopped(db, input.companyId, old.run.id);
|
||||
if (!stopped) throw pluginTaskResumeRefused("predecessor_not_stopped");
|
||||
return old.lease.providerLeaseId!;
|
||||
}
|
||||
@@ -24,19 +24,47 @@ export function remoteTerminationReceipt(lease: LeaseIdentity, value: unknown) {
|
||||
};
|
||||
}
|
||||
|
||||
export function hasRemoteTerminationReceipt(lease: LeaseIdentity & {
|
||||
type ReceiptLease = LeaseIdentity & {
|
||||
releasedAt: unknown; cleanupStatus: string | null; status: string;
|
||||
metadata: Record<string, unknown> | null;
|
||||
}): boolean {
|
||||
};
|
||||
|
||||
/** The lease carries a provider stop receipt bound to its exact company, run,
|
||||
* lease, provider and physical task. It proves execution there stopped, not
|
||||
* that the lease's resources were released. */
|
||||
export function hasBoundRemoteStopReceipt(lease: Pick<ReceiptLease, keyof LeaseIdentity | "metadata">): boolean {
|
||||
const receipt = lease.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined;
|
||||
return Boolean(lease.releasedAt && lease.cleanupStatus === "success" &&
|
||||
["released", "expired", "failed"].includes(lease.status) && receipt &&
|
||||
receipt.schema === "paperclip.remote-termination.v1" &&
|
||||
return Boolean(receipt && receipt.schema === "paperclip.remote-termination.v1" &&
|
||||
receipt.companyId === lease.companyId && receipt.runId === lease.heartbeatRunId &&
|
||||
receipt.leaseId === lease.id && receipt.provider === lease.provider &&
|
||||
remoteTerminationReceipt(lease, receipt));
|
||||
}
|
||||
|
||||
/** The lease was released with successful cleanup and its bound stop receipt.
|
||||
* Resource cleanup gates use this; it never accepts an unreleased lease. */
|
||||
export function hasRemoteTerminationReceipt(lease: ReceiptLease): boolean {
|
||||
return Boolean(lease.releasedAt && lease.cleanupStatus === "success" &&
|
||||
["released", "expired", "failed"].includes(lease.status) && hasBoundRemoteStopReceipt(lease));
|
||||
}
|
||||
|
||||
/** Release or destroy of this lease has started: plugin release and destroy
|
||||
* record their cleanup method first, and task settlement, startup cleanup and
|
||||
* interrupted-creation cleanup mark the task retiring. The provider can then be
|
||||
* releasing the task's resources while the lease row is still active. */
|
||||
function leaseReleaseStarted(lease: Pick<ReceiptLease, "metadata">): boolean {
|
||||
return lease.metadata?.pluginLeaseCleanupMethod !== undefined || lease.metadata?.runnerTaskRetiring === true;
|
||||
}
|
||||
|
||||
/** Execution on this lease has stopped: it was released with its termination
|
||||
* receipt, or it is still active, its release has not started, and its plugin
|
||||
* task recorded the provider's stop when it settled. Release can be held long
|
||||
* after that stop (for example by a pending workspace export); once release
|
||||
* starts, it owns the receipt until the lease is released. */
|
||||
function remoteLeaseExecutionStopped(lease: ReceiptLease): boolean {
|
||||
return hasRemoteTerminationReceipt(lease) ||
|
||||
lease.status === "active" && !lease.releasedAt && !leaseReleaseStarted(lease) && hasBoundRemoteStopReceipt(lease);
|
||||
}
|
||||
|
||||
/** Workspace recovery saved this run's files, re-stopped its sandbox, and then
|
||||
* queued the stopped sandbox for destroy. The destroy is resource cleanup: the
|
||||
* stop receipt it carried forward still proves no provider can run there. */
|
||||
@@ -77,16 +105,27 @@ export async function resolveRemoteTerminationLease(db: Db, lease: Lease): Promi
|
||||
} } };
|
||||
}
|
||||
|
||||
export async function stoppedRemoteCleanupScopes(db: Db, companyId: string, runId: string) {
|
||||
async function runRemoteLeases(db: Db, companyId: string, runId: string) {
|
||||
const leases = await db.select().from(environmentLeases).where(and(
|
||||
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId),
|
||||
));
|
||||
if (leases.length === 0) return null;
|
||||
const resolved = await Promise.all(leases.map(lease => resolveRemoteTerminationLease(db, lease)));
|
||||
if (!resolved.every(hasRemoteTerminationReceipt)) return null;
|
||||
return [...new Set(resolved.map(lease => remoteLeaseCleanupScope(lease)!))];
|
||||
return Promise.all(leases.map(lease => resolveRemoteTerminationLease(db, lease)));
|
||||
}
|
||||
|
||||
/** Cleanup scopes of a run whose every lease has stopped execution, or null. */
|
||||
export async function stoppedRemoteCleanupScopes(db: Db, companyId: string, runId: string) {
|
||||
const leases = await runRemoteLeases(db, companyId, runId);
|
||||
if (leases.length === 0 || !leases.every(remoteLeaseExecutionStopped)) return null;
|
||||
return [...new Set(leases.map(lease => remoteLeaseCleanupScope(lease)!))];
|
||||
}
|
||||
|
||||
/** Every lease of the run has stopped execution. Its resources can still be held. */
|
||||
export async function remoteExecutionHasStopped(db: Db, companyId: string, runId: string) {
|
||||
return await stoppedRemoteCleanupScopes(db, companyId, runId) !== null;
|
||||
}
|
||||
|
||||
/** Every lease of the run was released with its termination receipt. */
|
||||
export async function remoteExecutionHasReleased(db: Db, companyId: string, runId: string) {
|
||||
const leases = await runRemoteLeases(db, companyId, runId);
|
||||
return leases.length > 0 && leases.every(hasRemoteTerminationReceipt);
|
||||
}
|
||||
Reference in new issue
Block a user