fix(runtime): record a plugin task's stop receipt when its stop is proven (#21237)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin environment runs each Paperclip Runner turn as a provider
task, and a follow-up run can continue the same Claude/ACPX conversation
from the saved checkpoint
> - Core authorizes that continuation only when the predecessor run's
remote execution has stopped (`remoteExecutionHasStopped`), so two
Runners never write one checkpoint
> - The predecessor's own settlement already proves the stop: the
provider reports `executionStopped`, and Core then completes the task
> - On a per-turn (ephemeral) lease, Core records the stop receipt only
when it releases the lease, and that release can be held for a long time
(for example by a pending workspace export)
> - In that window, an ordinary follow-up fails with `adapter_failed:
Plugin Runner retained session is not authorized`, even though the stop
is already proven
> - This pull request records the stop receipt when the stop is proven,
and the stop check accepts it before the release
> - The benefit is that follow-ups continue as soon as the predecessor
is proven stopped, and the release gates keep their meaning

## Linked Issues or Issue Description

Fixes #21211

Refs #21214 and #21223. Those cover the creation fence for same-run
recovery, which still requires a released lease.

Production evidence for this change: in both observed failures, the
predecessor was a per-turn plugin run. Its `task.settle` completed with
all provider calls successful. Core then did not release its lease for
30 minutes or more, and no release call was made in that time. The
follow-up failed in checkpoint recovery (`recoverPluginRunnerCheckpoint`
→ `resolvePluginTaskResume`) within seconds of admission.

## What Changed

- `server/src/services/environment-task-runtime.ts`
- After a recorded provider stop, `complete` writes the bound stop
receipt (`remoteTerminationReceipt(..., { state: "stopped" })`) on the
current lease for every lease policy. Before, it did this only for
`reuse_by_environment` leases. Reusable tasks still confirm every owner
and still move retained owners to released.
- New work on a lease (`submit`, `copilot_metadata`, first portable
resume) removes the stop receipt in the same write that clears
`runnerTaskExecutionStopped`. A replacement physical task does not copy
the old receipt.
- `server/src/services/remote-execution-termination.ts`
- New `hasBoundRemoteStopReceipt` checks only the receipt binding
(company, run, lease, provider and physical task).
`hasRemoteTerminationReceipt` uses it and keeps its released-only
semantics. Resource cleanup gates do not change.
- `remoteExecutionHasStopped` and `stoppedRemoteCleanupScopes` accept a
lease that is released with its receipt, or that is still `active` (not
released) with a bound stop receipt. A lease whose release started and
did not settle does not count. After release starts, the release owns
the receipt.
  - New `remoteExecutionHasReleased` gives the release-based answer.
- `server/src/services/environment-run-orchestrator.ts`: same-run plugin
recovery uses `remoteExecutionHasReleased`. The same-run resume check
after it requires released receipts, so its behavior does not change.
- `resolvePluginTaskResume` keeps its checks and their order. Its
refusal now names the first failing check: `Plugin Runner retained
session is not authorized (reason: <code>)`, for example
`predecessor_not_stopped`, `scope_mismatch` or `identity_mismatch`. The
old message stays as the prefix, and no ids go into it. The
stale-availability recheck in `environment-task-runtime.ts` names its
refusal the same way. Production telemetry for #21211 could not tell
which check fired. The next occurrence will show it.
- `doc/execution-semantics.md` and
`doc/testing-native-remote-continuity.md` describe the stop receipt and
when it is recorded.

Caller audit for the changed meaning of `remoteExecutionHasStopped`:

| Caller | Needs | Decision |
| --- | --- | --- |
| `agent-instruction-working-copies.ts` | stop | Accepts. Directory
cleanup still requires a released "destroyed" receipt. |
| `environment-run-orchestrator.ts` | release | Moved to
`remoteExecutionHasReleased`. |
| `plugin-task-resume.ts` | stop | Accepts. This is the fix. |
| `native-runtime/native-runner-ownership.ts` | stop | Accepts.
`hasPendingRunCleanup` still counts the active lease as pending. |
| `native-runtime/preserved-native-result-finalization.ts` | stop |
Accepts. Workspace recovery has its own gate. |
| `heartbeat/case-handoff-lifecycle.ts` | stop | Accepts.
`hasPendingRunCleanup` still holds active leases. |
| `heartbeat/run-control.ts` (3 sites) | stop | Accepts. Stop
acknowledgement and quarantined ownership retirement. Holds and blockers
still apply on the normal path. |
| `heartbeat/runtime-execution.ts` | stop | Accepts. Release errors
still fail it. |

## Verification

From `server/`, after `node scripts/agent-preflight.mjs`:

- `npx vitest run
src/__tests__/environment-task-authority.integration.test.ts`: 10/10
pass. The 2 new tests fail without the fix.
- The plugin task and environment runtime suites
(`environment-task-authority.integration`,
`plugin-runner-task.integration`, `plugin-warm-tasks.integration`,
`environment-task-runtime`, `plugin-task-source-recovery.integration`,
`environment-runtime`): 6 files, 366 pass.
- The audited caller suites (`agent-instruction-working-copies`, case
handoff legacy and native, `environment-run-orchestrator`,
`heartbeat-case-handoff`, `explicit-native-continuation`, `run-control`,
`native-remote-retry-admission`, `preserved-native-response-wait`): 9
files, 806 pass.
- `pnpm --filter @paperclipai/server typecheck`: pass.

The new integration tests cover:

- `complete` writes a bound stop receipt on a still-active ephemeral
lease. `remoteExecutionHasStopped` is true and
`hasRemoteTerminationReceipt` is still false.
- `resolvePluginTaskResume` (inspect and launch) authorizes a
follow-up's new task, and the real host `read_state` checkpoint read
succeeds, while the predecessor's release is held.
- Refusals: no provider stop, a receipt for another provider task, an
unstopped sibling lease, and a release in progress.
- New work on the lease removes the receipt. Release overwrites the
receipt with its own receipt. Warm owners keep their behavior.

I did not add an executor-level test. `native-session-executor.test.ts`
mocks `resolvePluginTaskResume`, so the integration test runs the real
checkpoint read path instead. I did not run the full suite locally. CI
runs it.

## Risks

- `remoteExecutionHasStopped` now returns true earlier for a run whose
leases are still held. Each caller is audited above. Callers that need
released resources use the release-based predicates.
- The stop receipt can now be on an `active` lease. The SQL readers of
`remoteExecutionTermination` (cost accounting, execution blockers,
idle-work proof, orphan settlement) also require a released status.
`pinFailedObservedWorkspace` treats an active ephemeral lease with a
receipt as not pinnable. That path applies only to the remote-workspace
flow and falls back to `preserveNativeFailedWorkspace`.
- Leases that settled before this change have no early receipt. They
continue to wait for their release.
- Nothing compares the refusal message exactly. Matchers use substrings,
and run failure classification does not use the message.
- A follow-up still fails closed if the predecessor's stop was never
proven. This change does not add a recoverable wait.
- The reason the release was held in production is not addressed here.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents, Honeycomb trace queries) and extended
thinking. One sub-agent wrote the change. A second session reviewed it
before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Nicky LeachandClaude Opus 5.5 authored and GitHub committed 2026-11-12 20:20:45 -08:00
1 parent b9308f9828
commit afe86e9ded
7 files changed
+396 -59

No files matched your search

+17
View File
@@ -3483,6 +3483,23 @@ agent, workspace, retained session, and termination of prior execution. The
checkpoint must match the exact Runner, lease, run, session, turn, and item.
A missing restart claim does not remove these checks or authorize a fresh session.
A plugin task records its stop receipt when it settles. When the provider
reports that the task stopped, completing the task writes a receipt bound to
the exact company, run, lease, provider and task, even before the lease is
released. Lease release can be held long after that, for example by a pending
workspace export. A per-turn follow-up in a later run can then resume the
retained checkpoint: for continuation, every lease of the predecessor run must
be released with its termination receipt or still be active with this stop
receipt and its release not started. Once release or destroy of a lease starts,
release owns the receipt: the lease does not count until release settles with
its own receipt, because the provider can be releasing the task's resources
while the lease is still active. Dispatching new work on a lease removes its
stop receipt. Resource cleanup gates, and a same-run retry's replacement of earlier
leases, still require the released termination receipt.
When the server refuses the resume, the error keeps the message
`Plugin Runner retained session is not authorized` and names the first failed
check, for example `(reason: predecessor_not_stopped)`.
These checks can refuse the exact checkpoint before the retry claims a new
provider attempt. For example, the interrupted attempt can leave the provider
state not suspended. The run error then names the interrupted attempt first, as
+7 -1
View File
@@ -80,7 +80,13 @@ The executor tests cover Claude and Codex, including reopening blocked work.
The lease-authority integration test uses PostgreSQL to check that checkpoint
discovery cannot launch work or bypass a termination receipt. A stale negative
availability flag requires a fresh provider status before the checkpoint can
be read for continuation.
be read for continuation. It also checks that a follow-up can resume while the
predecessor's lease release is held, using the stop receipt that the task
recorded when it settled. A receipt for another task, a second predecessor
lease without stop proof, or a release that has started and not settled still
refuses continuation. The test holds a real lease release inside the provider
call and checks that the follow-up is refused until that release settles.
Each refusal is checked by the reason that it names.
The executor can restore one quarantined journal in the same session scope.
It requires a terminal prior run, completed commands, a terminal event, and an
@@ -3,7 +3,9 @@ import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { eq } from "drizzle-orm";
import { agents, companies, createDb, environmentLeases, environments, heartbeatRuns, plugins, sandboxCreationOperations } from "@paperclipai/db";
import { resolvePluginTaskResume } from "../services/environment-task-runtime.js";
import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
import type { PluginTaskResumeRefusal } from "../services/plugin-task-resume.js";
import { hasBoundRemoteStopReceipt, hasRemoteTerminationReceipt, remoteExecutionHasReleased, remoteExecutionHasStopped,
remoteTerminationReceipt } from "../services/remote-execution-termination.js";
import { environmentRuntimeService } from "../services/environment-runtime.js";
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
import { createPluginRunnerTaskOwner } from "../services/native-runtime/plugin-runner-task.js";
@@ -12,6 +14,8 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro
const support = await getEmbeddedPostgresTestSupport();
const databaseSuite = support.supported ? describe : describe.skip;
/** A resume refusal keeps its original message and names the check that failed. */
const refused = (reason: PluginTaskResumeRefusal) => `Plugin Runner retained session is not authorized (reason: ${reason})`;
databaseSuite("environment task lease authority", () => {
let db: ReturnType<typeof createDb>;
@@ -227,17 +231,29 @@ databaseSuite("environment task lease authority", () => {
await expect(authorize()).resolves.toBe("old-task");
for (const field of ["runnerInstanceId", "environmentLeaseId", "normalizedSessionId", "runId"]) {
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId,
priorIdentity: { ...priorIdentity, [field]: randomUUID() } })).rejects.toThrow("not authorized");
priorIdentity: { ...priorIdentity, [field]: randomUUID() } })).rejects.toThrow(refused("predecessor_candidate_missing"));
}
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, oldRunId));
await expect(authorize()).rejects.toThrow("not authorized");
await expect(authorize()).rejects.toThrow(refused("predecessor_run_active"));
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, oldRunId));
await expect(resolvePluginTaskResume(db, { companyId: randomUUID(), leaseId: nextLeaseId, priorIdentity })).rejects.toThrow("not authorized");
await expect(resolvePluginTaskResume(db, { companyId: randomUUID(), leaseId: nextLeaseId, priorIdentity })).rejects.toThrow(refused("next_lease_missing"));
await db.update(environmentLeases).set({ metadata: { ...pin, pluginId: randomUUID() } }).where(eq(environmentLeases.id, nextLeaseId));
await expect(authorize()).rejects.toThrow("not authorized");
await expect(authorize()).rejects.toThrow(refused("provider_mismatch"));
await db.update(environmentLeases).set({ metadata: pin }).where(eq(environmentLeases.id, nextLeaseId));
await db.update(environmentLeases).set({ providerLeaseId: "old-task" }).where(eq(environmentLeases.id, nextLeaseId));
await expect(authorize()).rejects.toThrow(refused("same_provider_task"));
await db.update(environmentLeases).set({ providerLeaseId: "new-task" }).where(eq(environmentLeases.id, nextLeaseId));
const otherEnvironmentId = randomUUID();
await db.insert(environments).values({ id: otherEnvironmentId, name: `Fixture ${otherEnvironmentId}`, driver: "plugin", config: {} });
await db.update(environmentLeases).set({ environmentId: otherEnvironmentId }).where(eq(environmentLeases.id, nextLeaseId));
await expect(authorize()).rejects.toThrow(refused("scope_mismatch"));
await db.update(environmentLeases).set({ environmentId }).where(eq(environmentLeases.id, nextLeaseId));
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, nextRunId));
await expect(authorize()).rejects.toThrow(refused("next_run_not_running"));
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, nextRunId));
await expect(authorize()).resolves.toBe("old-task");
await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, oldLeaseId));
await expect(authorize()).rejects.toThrow("not authorized");
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, oldLeaseId));
let sessionStateAvailable = false;
const operations: string[] = [];
@@ -259,8 +275,8 @@ databaseSuite("environment task lease authority", () => {
operation: { kind: "ensure_ownership", operationId: portableTargetLeaseId, sourceTaskId: "old-task",
sourceGeneration: "source-generation", targetGeneration: "target-generation", policy: "ordinary", identity: checkpointIdentity },
} } }).where(eq(environmentLeases.id, oldLeaseId));
await expect(authorize()).rejects.toThrow("not authorized");
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).rejects.toThrow("not authorized");
await expect(authorize()).rejects.toThrow(refused("predecessor_candidate_missing"));
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).rejects.toThrow(refused("predecessor_candidate_missing"));
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task",
} })).rejects.toThrow("not authorized");
@@ -269,19 +285,19 @@ databaseSuite("environment task lease authority", () => {
} })).rejects.toThrow("not authorized");
expect(call).not.toHaveBeenCalled();
await db.update(environmentLeases).set({ metadata: { ...source.metadata, runnerTaskSessionRetained: false } }).where(eq(environmentLeases.id, oldLeaseId));
await expect(authorize()).rejects.toThrow("not authorized");
await expect(authorize()).rejects.toThrow(refused("predecessor_candidate_missing"));
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).resolves.toBe("old-task");
const inspect = () => runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task",
} });
await expect(inspect()).rejects.toThrow("not authorized");
await expect(authorize()).rejects.toThrow("not authorized");
await expect(inspect()).rejects.toThrow(refused("checkpoint_status_unconfirmed"));
await expect(authorize()).rejects.toThrow(refused("predecessor_candidate_missing"));
// Discovery cannot launch, and cannot bypass the original termination receipt.
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "submit", preparedLaunchId: "fixture", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task",
} })).rejects.toThrow("resume identity mismatch");
await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, oldLeaseId));
await expect(inspect()).rejects.toThrow("not authorized");
await expect(inspect()).rejects.toThrow(refused("predecessor_not_stopped"));
expect(operations).toEqual(["status"]);
await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, oldLeaseId));
sessionStateAvailable = true;
@@ -350,4 +366,215 @@ databaseSuite("environment task lease authority", () => {
});
async function seedTaskPlugin(name: string) {
const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), pluginId = randomUUID();
await db.insert(companies).values({ id: companyId, name, issuePrefix: `T${companyId.slice(0, 7).replace(/-/g, "").toUpperCase()}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", adapterType: "paperclip_runner" });
await db.insert(environments).values({ id: environmentId, name: `Fixture ${environmentId}`, driver: "plugin",
config: { pluginKey: pluginId, driverKey: "tasks", driverConfig: {} } });
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: `fixture-${name}`, version: "1.0.0", status: "ready", manifestJson: {
capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }],
} as never });
const pin = { driver: "plugin", pluginId, driverKey: "tasks", environmentId, runnerTaskLease: true,
runnerTaskProjectAuthValidation: { configurationId: "fixture", provider: "openai", projectIds: [] } };
return { companyId, agentId, environmentId, pin };
}
it("records a per-turn task's stop receipt when it settles, before its held lease release", async () => {
const { companyId, agentId, environmentId, pin } = await seedTaskPlugin("per-turn-stop");
const oldRunId = randomUUID(), nextRunId = randomUUID(), oldLeaseId = randomUUID(), nextLeaseId = randomUUID();
await db.insert(heartbeatRuns).values([
{ id: oldRunId, companyId, agentId, invocationSource: "manual", status: "running" },
{ id: nextRunId, companyId, agentId, invocationSource: "manual", status: "running" },
]);
const prior = { id: oldLeaseId, companyId, environmentId, heartbeatRunId: oldRunId, provider: "plugin:fixture:tasks", providerLeaseId: "old-task" };
const oldIdentity = { runnerId: "runner", leaseId: oldLeaseId, runId: oldRunId, sessionId: "session", turnId: "prior-turn", itemId: "prior-item" };
await db.insert(environmentLeases).values([
{ ...prior, status: "active", leasePolicy: "ephemeral", metadata: { ...pin, runnerTaskDispatched: true,
runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false, runnerTaskIdentity: oldIdentity } },
{ id: nextLeaseId, companyId, environmentId, heartbeatRunId: nextRunId, provider: prior.provider, providerLeaseId: "new-task",
status: "active", leasePolicy: "ephemeral", metadata: pin },
]);
let executionStopped = false;
const call = vi.fn(async (_id: string, method: string, params: any) => method !== "environmentTask" ? undefined
: ["read_state", "archive_state"].includes(params.operation.kind)
? { kind: "state", taskId: params.taskId, state: { schema: "paperclip.runner.durable.state.v1", lifecycle: "suspended",
...(params.operation.identity ?? params.operation.priorIdentity) } }
: params.operation.kind === "status"
? { kind: "status", taskId: params.taskId, phase: executionStopped ? "completed" : "running", executionStopped, sessionStateAvailable: executionStopped }
: { kind: "accepted", taskId: params.taskId });
const workers = { isRunning: () => true, getWorker: () => ({ supportedMethods: ["environmentTask", "environmentReleaseLease"] }), call } as unknown as PluginWorkerManager;
const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers });
const readLease = async (id: string) => (await db.select().from(environmentLeases).where(eq(environmentLeases.id, id)))[0]!;
const priorIdentity = { runnerInstanceId: "runner", normalizedSessionId: "session", environmentLeaseId: oldLeaseId, runId: oldRunId };
const checkpointIdentity = { ...priorIdentity, turnId: "prior-turn", itemId: "prior-item" };
const runner = { runnerId: "runner", leaseId: oldLeaseId, sessionId: "session", runId: nextRunId,
turnId: "turn", itemId: "item", harness: "codex", protocolMin: 1, protocolMax: 3 };
const authorize = (inspectCheckpoint?: boolean) => resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint });
// Neither a running status nor a completion without a provider stop is stop proof.
await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "status" } });
await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "complete" } });
expect((await readLease(oldLeaseId)).metadata).not.toHaveProperty("remoteExecutionTermination");
expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(false);
// The turn checkpoint stops and completes the task; release of the lease is still held.
executionStopped = true;
await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "stop" } });
await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "status" } });
await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "complete" } });
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, oldRunId));
const settled = await readLease(oldLeaseId);
expect(settled).toMatchObject({ status: "active", releasedAt: null, cleanupStatus: null, metadata: {
runnerTaskExecutionStopped: true, runnerTaskSessionRetained: true, runnerTaskDispatched: false,
remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId, runId: oldRunId, leaseId: oldLeaseId,
provider: prior.provider, providerLeaseId: "old-task", state: "stopped" } } });
expect(hasBoundRemoteStopReceipt(settled)).toBe(true);
expect(hasRemoteTerminationReceipt(settled)).toBe(false);
expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(true);
expect(await remoteExecutionHasReleased(db, companyId, oldRunId)).toBe(false);
// The follow-up can inspect and launch from the checkpoint while the predecessor is unreleased.
await expect(authorize(true)).resolves.toBe("old-task");
await expect(authorize()).resolves.toBe("old-task");
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task",
} })).resolves.toMatchObject({ kind: "state", state: { ...checkpointIdentity, lifecycle: "suspended" } });
// A receipt for another physical task, a second unstopped predecessor lease, or a release
// that started without settling still refuses authority.
await db.update(environmentLeases).set({ metadata: { ...settled.metadata, remoteExecutionTermination:
remoteTerminationReceipt({ ...prior, providerLeaseId: "other-task" }, { providerLeaseId: "other-task", state: "stopped" }) } })
.where(eq(environmentLeases.id, oldLeaseId));
await expect(authorize(true)).rejects.toThrow(refused("predecessor_not_stopped"));
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
await db.update(environmentLeases).set({ metadata: settled.metadata }).where(eq(environmentLeases.id, oldLeaseId));
const siblingId = randomUUID();
await db.insert(environmentLeases).values({ id: siblingId, companyId, environmentId, heartbeatRunId: oldRunId,
provider: prior.provider, providerLeaseId: "sibling-task", status: "active", leasePolicy: "ephemeral", metadata: pin });
await expect(authorize(true)).rejects.toThrow(refused("predecessor_not_stopped"));
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
await db.delete(environmentLeases).where(eq(environmentLeases.id, siblingId));
await db.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", releasedAt: new Date() })
.where(eq(environmentLeases.id, oldLeaseId));
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
await db.update(environmentLeases).set({ status: "active", cleanupStatus: null, releasedAt: null }).where(eq(environmentLeases.id, oldLeaseId));
await expect(authorize()).resolves.toBe("old-task");
// Dispatching new execution invalidates a stop receipt on that lease.
const next = await readLease(nextLeaseId);
await db.update(environmentLeases).set({ metadata: { ...next.metadata, runnerTaskExecutionStopped: true,
remoteExecutionTermination: remoteTerminationReceipt(next, { providerLeaseId: "new-task", state: "stopped" }) } })
.where(eq(environmentLeases.id, nextLeaseId));
expect(await remoteExecutionHasStopped(db, companyId, nextRunId)).toBe(true);
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "submit", preparedLaunchId: "fixture", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task",
} })).resolves.toMatchObject({ kind: "accepted" });
expect((await readLease(nextLeaseId)).metadata).not.toHaveProperty("remoteExecutionTermination");
expect(await remoteExecutionHasStopped(db, companyId, nextRunId)).toBe(false);
// Release still writes its own receipt over the settlement receipt.
const releaseErrors: unknown[] = [];
await runtime.releaseRunLeases(oldRunId, "released", (_id, error) => { releaseErrors.push(error); });
expect(releaseErrors).toEqual([]);
const released = await readLease(oldLeaseId);
expect(released).toMatchObject({ status: "released", cleanupStatus: "success",
metadata: { remoteExecutionTermination: { leaseId: oldLeaseId, providerLeaseId: "old-task", state: "stopped" } } });
expect(hasRemoteTerminationReceipt(released)).toBe(true);
expect(await remoteExecutionHasReleased(db, companyId, oldRunId)).toBe(true);
});
it("stops counting a settled task's stop receipt once its lease release starts", async () => {
const { companyId, agentId, environmentId, pin } = await seedTaskPlugin("release-owns-receipt");
const oldRunId = randomUUID(), nextRunId = randomUUID(), oldLeaseId = randomUUID(), nextLeaseId = randomUUID();
await db.insert(heartbeatRuns).values([
{ id: oldRunId, companyId, agentId, invocationSource: "manual", status: "succeeded" },
{ id: nextRunId, companyId, agentId, invocationSource: "manual", status: "running" },
]);
const prior = { id: oldLeaseId, companyId, environmentId, heartbeatRunId: oldRunId, provider: "plugin:fixture:tasks", providerLeaseId: "old-task" };
// The predecessor's task settled in its run: stopped, session retained, stop receipt recorded.
const settledMetadata = { ...pin, runnerTaskDispatched: false, runnerTaskExecutionStopped: true, runnerTaskSessionRetained: true,
runnerTaskIdentity: { runnerId: "runner", leaseId: oldLeaseId, runId: oldRunId, sessionId: "session", turnId: "turn", itemId: "item" },
remoteExecutionTermination: remoteTerminationReceipt(prior, { providerLeaseId: "old-task", state: "stopped" }) };
await db.insert(environmentLeases).values([
{ ...prior, status: "active", leasePolicy: "ephemeral", metadata: settledMetadata },
{ id: nextLeaseId, companyId, environmentId, heartbeatRunId: nextRunId, provider: prior.provider, providerLeaseId: "new-task",
status: "active", leasePolicy: "ephemeral", metadata: pin },
]);
let releaseEntered!: () => void, finishRelease!: () => void;
const entered = new Promise<void>(resolve => { releaseEntered = resolve; });
const held = new Promise<void>(resolve => { finishRelease = resolve; });
const call = vi.fn(async (_id: string, method: string) => {
if (method !== "environmentReleaseLease") throw new Error(`unexpected ${method}`);
releaseEntered();
await held;
});
const workers = { isRunning: () => true, getWorker: () => ({ supportedMethods: ["environmentTask", "environmentReleaseLease"] }), call } as unknown as PluginWorkerManager;
const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers });
const priorIdentity = { runnerInstanceId: "runner", normalizedSessionId: "session", environmentLeaseId: oldLeaseId, runId: oldRunId };
const authorize = (inspectCheckpoint?: boolean) => resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint });
await expect(authorize()).resolves.toBe("old-task");
// Each marker that release, destroy, or startup and creation cleanup write first takes the receipt over.
for (const marker of [{ pluginLeaseCleanupMethod: "release" }, { pluginLeaseCleanupMethod: "destroy" }, { runnerTaskRetiring: true }]) {
await db.update(environmentLeases).set({ metadata: { ...settledMetadata, ...marker } }).where(eq(environmentLeases.id, oldLeaseId));
expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(false);
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
}
await db.update(environmentLeases).set({ metadata: settledMetadata }).where(eq(environmentLeases.id, oldLeaseId));
// Hold the real release inside the provider call: the lease is still active with its settlement receipt.
const releaseErrors: unknown[] = [];
const releasing = runtime.releaseRunLeases(oldRunId, "released", (_id, error) => { releaseErrors.push(error); });
await entered;
const inFlight = (await db.select().from(environmentLeases).where(eq(environmentLeases.id, oldLeaseId)))[0]!;
expect(inFlight).toMatchObject({ status: "active", releasedAt: null,
metadata: { pluginLeaseCleanupMethod: "release", runnerTaskRetiring: true } });
expect(hasBoundRemoteStopReceipt(inFlight)).toBe(true);
expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(false);
await expect(authorize(true)).rejects.toThrow(refused("predecessor_not_stopped"));
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
// Release settles with its own receipt, which authorizes the follow-up.
finishRelease();
await releasing;
expect(releaseErrors).toEqual([]);
const released = (await db.select().from(environmentLeases).where(eq(environmentLeases.id, oldLeaseId)))[0]!;
expect(released).toMatchObject({ status: "released", cleanupStatus: "success" });
expect(hasRemoteTerminationReceipt(released)).toBe(true);
expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(true);
await expect(authorize(true)).resolves.toBe("old-task");
await expect(authorize()).resolves.toBe("old-task");
});
it("still records stop receipts for every warm owner and retires retained owners", async () => {
const { companyId, agentId, environmentId, pin } = await seedTaskPlugin("warm-stop");
const priorRunId = randomUUID(), currentRunId = randomUUID(), priorLeaseId = randomUUID(), currentLeaseId = randomUUID();
await db.insert(heartbeatRuns).values([
{ id: priorRunId, companyId, agentId, invocationSource: "manual", status: "succeeded" },
{ id: currentRunId, companyId, agentId, invocationSource: "manual", status: "running" },
]);
const shared = { companyId, environmentId, provider: "plugin:fixture:tasks", providerLeaseId: "warm-task", leasePolicy: "reuse_by_environment" as const };
await db.insert(environmentLeases).values([
{ ...shared, id: priorLeaseId, heartbeatRunId: priorRunId, status: "retained", metadata: pin },
{ ...shared, id: currentLeaseId, heartbeatRunId: currentRunId, status: "active", metadata: { ...pin, runnerTaskDispatched: true } },
]);
const call = vi.fn(async (_id: string, _method: string, params: any) => params.operation.kind === "status"
? { kind: "status", taskId: params.taskId, phase: "completed", executionStopped: true, sessionStateAvailable: false }
: { kind: "accepted", taskId: params.taskId });
const runtime = environmentRuntimeService(db, { pluginWorkerManager: { getWorker: () => ({ supportedMethods: ["environmentTask"] }), call } as unknown as PluginWorkerManager });
await runtime.task({ companyId, leaseId: currentLeaseId, operation: { kind: "status" } });
await runtime.task({ companyId, leaseId: currentLeaseId, operation: { kind: "complete" } });
const leases = await db.select().from(environmentLeases).where(eq(environmentLeases.providerLeaseId, "warm-task"));
const priorLease = leases.find(lease => lease.id === priorLeaseId)!, currentLease = leases.find(lease => lease.id === currentLeaseId)!;
expect(priorLease).toMatchObject({ status: "released", cleanupStatus: "success",
metadata: { remoteExecutionTermination: { leaseId: priorLeaseId, runId: priorRunId, state: "stopped" } } });
expect(currentLease).toMatchObject({ status: "active",
metadata: { remoteExecutionTermination: { leaseId: currentLeaseId, runId: currentRunId, state: "stopped" } } });
expect(hasRemoteTerminationReceipt(priorLease)).toBe(true);
expect(await remoteExecutionHasStopped(db, companyId, priorRunId)).toBe(true);
expect(await remoteExecutionHasStopped(db, companyId, currentRunId)).toBe(true);
expect(await remoteExecutionHasReleased(db, companyId, currentRunId)).toBe(false);
});
});
@@ -20,7 +20,7 @@ import { prepareRemoteWorkspace, readRemoteWorkspaceAuthority, remoteWorkspaceRe
import { environmentLeases, type Db } from "@paperclipai/db";
import { and, eq } from "drizzle-orm";
import { remoteExecutionHasStopped } from "./remote-execution-termination.js";
import { remoteExecutionHasReleased } from "./remote-execution-termination.js";
import { remoteRecoveryLeaseMismatch, type RemoteRecoveryLeaseMismatchField } from "./native-runtime/remote-recovery-lease-mismatch.js";
import { EnvironmentProviderUnavailableError, readTransientLeaseAcquisitionFailure } from "./environment-provider-unavailable-error.js";
import { injectLeaseAcquisitionFault } from "./environment-lease-fault-injection.js";
@@ -347,7 +347,9 @@ export function environmentRunOrchestrator(
lease.metadata?.agentId !== input.agentId)) throw new Error("native_plugin_recovery_lease_mismatch");
const errors: unknown[] = [];
await environmentRuntime.releaseRunLeases(input.heartbeatRunId, "released", (_id, error) => { errors.push(error); });
if (errors.length || (prior.length && !await remoteExecutionHasStopped(db, input.companyId, input.heartbeatRunId))) {
// Same-run task resume requires every prior lease released with its receipt;
// a stop recorded before a held release is not enough here.
if (errors.length || (prior.length && !await remoteExecutionHasReleased(db, input.companyId, input.heartbeatRunId))) {
throw new Error("native_plugin_recovery_termination_unconfirmed", { cause: errors[0] });
}
}
+17 -12
View File
@@ -1,4 +1,4 @@
import { resolvePluginTaskResume } from "./plugin-task-resume.js";
import { pluginTaskResumeRefused, resolvePluginTaskResume } from "./plugin-task-resume.js";
export { resolvePluginTaskResume } from "./plugin-task-resume.js";
import { attachSandboxCreation, environmentService } from "./environments.js";
import { createOwnedSandbox, queueSandboxCreationCleanup, recordSandboxCreationResource } from "./sandbox-creation-operations.js";
@@ -198,11 +198,13 @@ export async function executeEnvironmentTask(db: Db, workers: Pick<PluginWorkerM
if (inspectCheckpoint && sources[0]!.metadata?.runnerTaskSessionRetained !== true) {
const status = await executeEnvironmentTask(db, workers, { companyId: input.companyId,
leaseId: sources[0]!.id, operation: { kind: "status" } });
if (status.kind !== "status" || status.executionStopped !== true || status.sessionStateAvailable !== true ||
resumeFrom !== await resolvePluginTaskResume(db, { companyId: input.companyId, leaseId: lease.id,
priorIdentity: { runnerInstanceId: runner.runnerId, environmentLeaseId: runner.leaseId,
runId: priorRunId, normalizedSessionId: runner.sessionId } })) {
throw new Error("Plugin Runner retained session is not authorized");
if (status.kind !== "status" || status.executionStopped !== true || status.sessionStateAvailable !== true) {
throw pluginTaskResumeRefused("checkpoint_status_unconfirmed");
}
if (resumeFrom !== await resolvePluginTaskResume(db, { companyId: input.companyId, leaseId: lease.id,
priorIdentity: { runnerInstanceId: runner.runnerId, environmentLeaseId: runner.leaseId,
runId: priorRunId, normalizedSessionId: runner.sessionId } })) {
throw pluginTaskResumeRefused("resume_task_mismatch");
}
}
stoppedSourceCheckpoint = true;
@@ -310,6 +312,7 @@ export async function executeEnvironmentTask(db: Db, workers: Pick<PluginWorkerM
}
if (operation.kind === "submit" || operation.kind === "copilot_metadata" || firstPortableResume) {
// No ticket, headers, environment values, or launch paths belong in metadata.
// New execution invalidates this lease's earlier stop receipt with its stop flag.
const dispatched = {
runnerTaskLease: true, runnerTaskDispatched: true,
runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false,
@@ -320,7 +323,7 @@ export async function executeEnvironmentTask(db: Db, workers: Pick<PluginWorkerM
} : {}),
};
const marked = await db.update(environmentLeases).set({
metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskAdmissionRejection') || ${JSON.stringify(dispatched)}::jsonb`,
metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskAdmissionRejection' - 'remoteExecutionTermination') || ${JSON.stringify(dispatched)}::jsonb`,
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.status, "active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`,
requiredProjectAuthProof ? sql`${environmentLeases.metadata}->'runnerTaskProjectAuthValidation' = ${JSON.stringify(requiredProjectAuthProof)}::jsonb` : undefined,
@@ -420,7 +423,7 @@ export async function executeEnvironmentTask(db: Db, workers: Pick<PluginWorkerM
});
if (!fresh.providerLeaseId || fresh.providerLeaseId === taskId) throw new Error("Plugin replacement task identity unavailable");
const {runnerTaskIdentity:_identity,runnerTaskPriorIdentity:_prior,runnerTaskCredentialHandoff:_handoff,
runnerTaskResumeFrom:_resume,runnerTaskPreviousLeaseId:_previous,...rest}=metadata;
runnerTaskResumeFrom:_resume,runnerTaskPreviousLeaseId:_previous,remoteExecutionTermination:_stopped,...rest}=metadata;
const replacementMetadata={...rest,providerMetadata:fresh.metadata ?? {},runnerTaskReused:false,runnerTaskDispatched:false,
runnerTaskExecutionStopped:false,runnerTaskSessionRetained:false};
try {
@@ -557,10 +560,12 @@ export async function executeEnvironmentTask(db: Db, workers: Pick<PluginWorkerM
if (operation.kind === "complete") {
if (freshWorkspace && metadata.runnerTaskExecutionStopped === true)
await releaseProviderRecoveryTransfers(input.companyId, freshWorkspace);
// A physical task can back several run leases. Confirm retirement for every prior owner.
if (lease.leasePolicy === "reuse_by_environment" && metadata.runnerTaskExecutionStopped === true) {
const owners=await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId,input.companyId),
eq(environmentLeases.provider,lease.provider!),eq(environmentLeases.providerLeaseId,taskId)));
// The provider attested this task stopped: record the stop receipt now, not when
// the lease's resources are released. A reusable physical task can back several
// run leases, so confirm retirement for every owner.
if (metadata.runnerTaskExecutionStopped === true) {
const owners=lease.leasePolicy === "reuse_by_environment" ? await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId,input.companyId),
eq(environmentLeases.provider,lease.provider!),eq(environmentLeases.providerLeaseId,taskId))) : [lease];
for(const owner of owners) {
const receipt=remoteTerminationReceipt(owner,{providerLeaseId:taskId,state:"stopped"});
if(!receipt) continue;
+63 -22
View File
@@ -2,10 +2,64 @@ import { and, eq, or } from "drizzle-orm";
import { environmentLeases, heartbeatRuns, type Db } from "@paperclipai/db";
import { hasRemoteTerminationReceipt, remoteExecutionHasStopped } from "./remote-execution-termination.js";
/** The first check that refused a retained plugin task resume, named in the error. */
export type PluginTaskResumeRefusal =
| "next_lease_missing"
| "predecessor_candidate_missing"
| "predecessor_candidate_ambiguous"
| "predecessor_task_missing"
| "identity_mismatch"
| "provider_mismatch"
| "next_task_missing"
| "same_provider_task"
| "predecessor_run_active"
| "scope_mismatch"
| "next_lease_inactive"
| "next_run_not_running"
| "predecessor_not_stopped"
| "checkpoint_status_unconfirmed"
| "resume_task_mismatch";
/** Keeps the original message as a prefix so existing matchers still apply. */
export function pluginTaskResumeRefused(reason: PluginTaskResumeRefusal) {
return new Error(`Plugin Runner retained session is not authorized (reason: ${reason})`);
}
type ResumeRow = { lease: typeof environmentLeases.$inferSelect; run: typeof heartbeatRuns.$inferSelect };
type PriorIdentity = { runnerInstanceId: string; environmentLeaseId: string; runId: string; normalizedSessionId: string };
const TERMINAL_RUN_STATUSES = ["succeeded", "interrupted", "failed", "cancelled", "timed_out"];
/** Cheap, synchronous checks in evaluation order; the stop proof is read only after these pass. */
function resumeRefusal(priorIdentity: PriorIdentity, next: ResumeRow | undefined, candidates: ResumeRow[]): PluginTaskResumeRefusal | null {
if (!next) return "next_lease_missing";
if (candidates.length === 0) return "predecessor_candidate_missing";
if (candidates.length > 1) return "predecessor_candidate_ambiguous";
const old = candidates[0]!;
const identity = old.lease.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined;
if (!old.lease.providerLeaseId) return "predecessor_task_missing";
if (!identity || identity.runnerId !== priorIdentity.runnerInstanceId || identity.leaseId !== priorIdentity.environmentLeaseId ||
identity.sessionId !== priorIdentity.normalizedSessionId || identity.runId !== priorIdentity.runId) return "identity_mismatch";
if (old.lease.metadata?.driver !== "plugin" || next.lease.metadata?.driver !== "plugin") return "provider_mismatch";
if (!next.lease.providerLeaseId) return "next_task_missing";
if (next.lease.providerLeaseId === old.lease.providerLeaseId) return "same_provider_task";
if (old.run.id !== next.run.id && !TERMINAL_RUN_STATUSES.includes(old.run.status)) return "predecessor_run_active";
if (old.run.agentId !== next.run.agentId || old.lease.issueId !== next.lease.issueId ||
old.lease.executionWorkspaceId !== next.lease.executionWorkspaceId ||
!old.lease.environmentId || old.lease.environmentId !== next.lease.environmentId) return "scope_mismatch";
if (!next.lease.provider?.startsWith("plugin:") || old.lease.provider !== next.lease.provider ||
old.lease.metadata?.pluginId !== next.lease.metadata?.pluginId ||
old.lease.metadata?.driverKey !== next.lease.metadata?.driverKey) return "provider_mismatch";
if (next.lease.status !== "active" || next.lease.releasedAt ||
(next.lease.expiresAt && next.lease.expiresAt.getTime() <= Date.now())) return "next_lease_inactive";
if (next.run.status !== "running") return "next_run_not_running";
return null;
}
/** A saved provider task can resume only within the same authorized agent workspace. */
export async function resolvePluginTaskResume(db: Db, input: {
companyId: string; leaseId: string;
priorIdentity: { runnerInstanceId: string; environmentLeaseId: string; runId: string; normalizedSessionId: string };
priorIdentity: PriorIdentity;
/** Read-only discovery; launching still requires an attested retained checkpoint. */
inspectCheckpoint?: boolean;
}): Promise<string> {
@@ -28,25 +82,12 @@ export async function resolvePluginTaskResume(db: Db, input: {
!rows.some(successor => successor.lease.id !== input.leaseId &&
successor.lease.metadata?.runnerTaskResumeFrom === row.lease.providerLeaseId);
});
const old = candidates.length === 1 ? candidates[0] : undefined;
const identity = old?.lease.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined;
if (!old || !next || !old.lease.providerLeaseId || !identity ||
identity.runnerId !== input.priorIdentity.runnerInstanceId || identity.leaseId !== input.priorIdentity.environmentLeaseId ||
identity.sessionId !== input.priorIdentity.normalizedSessionId || identity.runId !== input.priorIdentity.runId ||
old.lease.metadata?.driver !== "plugin" || next.lease.metadata?.driver !== "plugin" ||
!next.lease.providerLeaseId || next.lease.providerLeaseId === old.lease.providerLeaseId ||
(old.run.id !== next.run.id && !["succeeded", "interrupted", "failed", "cancelled", "timed_out"].includes(old.run.status)) ||
old.run.agentId !== next.run.agentId || old.lease.issueId !== next.lease.issueId ||
old.lease.executionWorkspaceId !== next.lease.executionWorkspaceId ||
!old.lease.environmentId || old.lease.environmentId !== next.lease.environmentId ||
!next.lease.provider?.startsWith("plugin:") || old.lease.provider !== next.lease.provider ||
old.lease.metadata?.pluginId !== next.lease.metadata?.pluginId || old.lease.metadata?.driverKey !== next.lease.metadata?.driverKey ||
next.lease.status !== "active" || next.lease.releasedAt || next.run.status !== "running" ||
(next.lease.expiresAt && next.lease.expiresAt.getTime() <= Date.now()) ||
!(old.run.id === next.run.id
? rows.filter(row => row.lease.id !== next.lease.id).every(row => hasRemoteTerminationReceipt(row.lease))
: await remoteExecutionHasStopped(db, input.companyId, old.run.id))) {
throw new Error("Plugin Runner retained session is not authorized");
}
return old.lease.providerLeaseId;
const refusal = resumeRefusal(input.priorIdentity, next, candidates);
if (refusal) throw pluginTaskResumeRefused(refusal);
const old = candidates[0]!;
const stopped = old.run.id === next!.run.id
? rows.filter(row => row.lease.id !== next!.lease.id).every(row => hasRemoteTerminationReceipt(row.lease))
: await remoteExecutionHasStopped(db, input.companyId, old.run.id);
if (!stopped) throw pluginTaskResumeRefused("predecessor_not_stopped");
return old.lease.providerLeaseId!;
}
@@ -24,19 +24,47 @@ export function remoteTerminationReceipt(lease: LeaseIdentity, value: unknown) {
};
}
export function hasRemoteTerminationReceipt(lease: LeaseIdentity & {
type ReceiptLease = LeaseIdentity & {
releasedAt: unknown; cleanupStatus: string | null; status: string;
metadata: Record<string, unknown> | null;
}): boolean {
};
/** The lease carries a provider stop receipt bound to its exact company, run,
* lease, provider and physical task. It proves execution there stopped, not
* that the lease's resources were released. */
export function hasBoundRemoteStopReceipt(lease: Pick<ReceiptLease, keyof LeaseIdentity | "metadata">): boolean {
const receipt = lease.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined;
return Boolean(lease.releasedAt && lease.cleanupStatus === "success" &&
["released", "expired", "failed"].includes(lease.status) && receipt &&
receipt.schema === "paperclip.remote-termination.v1" &&
return Boolean(receipt && receipt.schema === "paperclip.remote-termination.v1" &&
receipt.companyId === lease.companyId && receipt.runId === lease.heartbeatRunId &&
receipt.leaseId === lease.id && receipt.provider === lease.provider &&
remoteTerminationReceipt(lease, receipt));
}
/** The lease was released with successful cleanup and its bound stop receipt.
* Resource cleanup gates use this; it never accepts an unreleased lease. */
export function hasRemoteTerminationReceipt(lease: ReceiptLease): boolean {
return Boolean(lease.releasedAt && lease.cleanupStatus === "success" &&
["released", "expired", "failed"].includes(lease.status) && hasBoundRemoteStopReceipt(lease));
}
/** Release or destroy of this lease has started: plugin release and destroy
* record their cleanup method first, and task settlement, startup cleanup and
* interrupted-creation cleanup mark the task retiring. The provider can then be
* releasing the task's resources while the lease row is still active. */
function leaseReleaseStarted(lease: Pick<ReceiptLease, "metadata">): boolean {
return lease.metadata?.pluginLeaseCleanupMethod !== undefined || lease.metadata?.runnerTaskRetiring === true;
}
/** Execution on this lease has stopped: it was released with its termination
* receipt, or it is still active, its release has not started, and its plugin
* task recorded the provider's stop when it settled. Release can be held long
* after that stop (for example by a pending workspace export); once release
* starts, it owns the receipt until the lease is released. */
function remoteLeaseExecutionStopped(lease: ReceiptLease): boolean {
return hasRemoteTerminationReceipt(lease) ||
lease.status === "active" && !lease.releasedAt && !leaseReleaseStarted(lease) && hasBoundRemoteStopReceipt(lease);
}
/** Workspace recovery saved this run's files, re-stopped its sandbox, and then
* queued the stopped sandbox for destroy. The destroy is resource cleanup: the
* stop receipt it carried forward still proves no provider can run there. */
@@ -77,16 +105,27 @@ export async function resolveRemoteTerminationLease(db: Db, lease: Lease): Promi
} } };
}
export async function stoppedRemoteCleanupScopes(db: Db, companyId: string, runId: string) {
async function runRemoteLeases(db: Db, companyId: string, runId: string) {
const leases = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId),
));
if (leases.length === 0) return null;
const resolved = await Promise.all(leases.map(lease => resolveRemoteTerminationLease(db, lease)));
if (!resolved.every(hasRemoteTerminationReceipt)) return null;
return [...new Set(resolved.map(lease => remoteLeaseCleanupScope(lease)!))];
return Promise.all(leases.map(lease => resolveRemoteTerminationLease(db, lease)));
}
/** Cleanup scopes of a run whose every lease has stopped execution, or null. */
export async function stoppedRemoteCleanupScopes(db: Db, companyId: string, runId: string) {
const leases = await runRemoteLeases(db, companyId, runId);
if (leases.length === 0 || !leases.every(remoteLeaseExecutionStopped)) return null;
return [...new Set(leases.map(lease => remoteLeaseCleanupScope(lease)!))];
}
/** Every lease of the run has stopped execution. Its resources can still be held. */
export async function remoteExecutionHasStopped(db: Db, companyId: string, runId: string) {
return await stoppedRemoteCleanupScopes(db, companyId, runId) !== null;
}
/** Every lease of the run was released with its termination receipt. */
export async function remoteExecutionHasReleased(db: Db, companyId: string, runId: string) {
const leases = await runRemoteLeases(db, companyId, runId);
return leases.length > 0 && leases.every(hasRemoteTerminationReceipt);
}