Files
PaperClipAI/server/src/__tests__/environment-task-authority.integration.test.ts
T
Nicky LeachandClaude Opus 5.5 afe86e9ded fix(runtime): record a plugin task's stop receipt when its stop is proven (#21237)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin environment runs each Paperclip Runner turn as a provider
task, and a follow-up run can continue the same Claude/ACPX conversation
from the saved checkpoint
> - Core authorizes that continuation only when the predecessor run's
remote execution has stopped (`remoteExecutionHasStopped`), so two
Runners never write one checkpoint
> - The predecessor's own settlement already proves the stop: the
provider reports `executionStopped`, and Core then completes the task
> - On a per-turn (ephemeral) lease, Core records the stop receipt only
when it releases the lease, and that release can be held for a long time
(for example by a pending workspace export)
> - In that window, an ordinary follow-up fails with `adapter_failed:
Plugin Runner retained session is not authorized`, even though the stop
is already proven
> - This pull request records the stop receipt when the stop is proven,
and the stop check accepts it before the release
> - The benefit is that follow-ups continue as soon as the predecessor
is proven stopped, and the release gates keep their meaning

## Linked Issues or Issue Description

Fixes #21211

Refs #21214 and #21223. Those cover the creation fence for same-run
recovery, which still requires a released lease.

Production evidence for this change: in both observed failures, the
predecessor was a per-turn plugin run. Its `task.settle` completed with
all provider calls successful. Core then did not release its lease for
30 minutes or more, and no release call was made in that time. The
follow-up failed in checkpoint recovery (`recoverPluginRunnerCheckpoint`
→ `resolvePluginTaskResume`) within seconds of admission.

## What Changed

- `server/src/services/environment-task-runtime.ts`
- After a recorded provider stop, `complete` writes the bound stop
receipt (`remoteTerminationReceipt(..., { state: "stopped" })`) on the
current lease for every lease policy. Before, it did this only for
`reuse_by_environment` leases. Reusable tasks still confirm every owner
and still move retained owners to released.
- New work on a lease (`submit`, `copilot_metadata`, first portable
resume) removes the stop receipt in the same write that clears
`runnerTaskExecutionStopped`. A replacement physical task does not copy
the old receipt.
- `server/src/services/remote-execution-termination.ts`
- New `hasBoundRemoteStopReceipt` checks only the receipt binding
(company, run, lease, provider and physical task).
`hasRemoteTerminationReceipt` uses it and keeps its released-only
semantics. Resource cleanup gates do not change.
- `remoteExecutionHasStopped` and `stoppedRemoteCleanupScopes` accept a
lease that is released with its receipt, or that is still `active` (not
released) with a bound stop receipt. A lease whose release started and
did not settle does not count. After release starts, the release owns
the receipt.
  - New `remoteExecutionHasReleased` gives the release-based answer.
- `server/src/services/environment-run-orchestrator.ts`: same-run plugin
recovery uses `remoteExecutionHasReleased`. The same-run resume check
after it requires released receipts, so its behavior does not change.
- `resolvePluginTaskResume` keeps its checks and their order. Its
refusal now names the first failing check: `Plugin Runner retained
session is not authorized (reason: <code>)`, for example
`predecessor_not_stopped`, `scope_mismatch` or `identity_mismatch`. The
old message stays as the prefix, and no ids go into it. The
stale-availability recheck in `environment-task-runtime.ts` names its
refusal the same way. Production telemetry for #21211 could not tell
which check fired. The next occurrence will show it.
- `doc/execution-semantics.md` and
`doc/testing-native-remote-continuity.md` describe the stop receipt and
when it is recorded.

Caller audit for the changed meaning of `remoteExecutionHasStopped`:

| Caller | Needs | Decision |
| --- | --- | --- |
| `agent-instruction-working-copies.ts` | stop | Accepts. Directory
cleanup still requires a released "destroyed" receipt. |
| `environment-run-orchestrator.ts` | release | Moved to
`remoteExecutionHasReleased`. |
| `plugin-task-resume.ts` | stop | Accepts. This is the fix. |
| `native-runtime/native-runner-ownership.ts` | stop | Accepts.
`hasPendingRunCleanup` still counts the active lease as pending. |
| `native-runtime/preserved-native-result-finalization.ts` | stop |
Accepts. Workspace recovery has its own gate. |
| `heartbeat/case-handoff-lifecycle.ts` | stop | Accepts.
`hasPendingRunCleanup` still holds active leases. |
| `heartbeat/run-control.ts` (3 sites) | stop | Accepts. Stop
acknowledgement and quarantined ownership retirement. Holds and blockers
still apply on the normal path. |
| `heartbeat/runtime-execution.ts` | stop | Accepts. Release errors
still fail it. |

## Verification

From `server/`, after `node scripts/agent-preflight.mjs`:

- `npx vitest run
src/__tests__/environment-task-authority.integration.test.ts`: 10/10
pass. The 2 new tests fail without the fix.
- The plugin task and environment runtime suites
(`environment-task-authority.integration`,
`plugin-runner-task.integration`, `plugin-warm-tasks.integration`,
`environment-task-runtime`, `plugin-task-source-recovery.integration`,
`environment-runtime`): 6 files, 366 pass.
- The audited caller suites (`agent-instruction-working-copies`, case
handoff legacy and native, `environment-run-orchestrator`,
`heartbeat-case-handoff`, `explicit-native-continuation`, `run-control`,
`native-remote-retry-admission`, `preserved-native-response-wait`): 9
files, 806 pass.
- `pnpm --filter @paperclipai/server typecheck`: pass.

The new integration tests cover:

- `complete` writes a bound stop receipt on a still-active ephemeral
lease. `remoteExecutionHasStopped` is true and
`hasRemoteTerminationReceipt` is still false.
- `resolvePluginTaskResume` (inspect and launch) authorizes a
follow-up's new task, and the real host `read_state` checkpoint read
succeeds, while the predecessor's release is held.
- Refusals: no provider stop, a receipt for another provider task, an
unstopped sibling lease, and a release in progress.
- New work on the lease removes the receipt. Release overwrites the
receipt with its own receipt. Warm owners keep their behavior.

I did not add an executor-level test. `native-session-executor.test.ts`
mocks `resolvePluginTaskResume`, so the integration test runs the real
checkpoint read path instead. I did not run the full suite locally. CI
runs it.

## Risks

- `remoteExecutionHasStopped` now returns true earlier for a run whose
leases are still held. Each caller is audited above. Callers that need
released resources use the release-based predicates.
- The stop receipt can now be on an `active` lease. The SQL readers of
`remoteExecutionTermination` (cost accounting, execution blockers,
idle-work proof, orphan settlement) also require a released status.
`pinFailedObservedWorkspace` treats an active ephemeral lease with a
receipt as not pinnable. That path applies only to the remote-workspace
flow and falls back to `preserveNativeFailedWorkspace`.
- Leases that settled before this change have no early receipt. They
continue to wait for their release.
- Nothing compares the refusal message exactly. Matchers use substrings,
and run failure classification does not use the message.
- A follow-up still fails closed if the predecessor's stop was never
proven. This change does not add a recoverable wait.
- The reason the release was held in production is not addressed here.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents, Honeycomb trace queries) and extended
thinking. One sub-agent wrote the change. A second session reviewed it
before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:20:45 -08:00

581 lines
48 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { eq } from "drizzle-orm";
import { agents, companies, createDb, environmentLeases, environments, heartbeatRuns, plugins, sandboxCreationOperations } from "@paperclipai/db";
import { resolvePluginTaskResume } from "../services/environment-task-runtime.js";
import type { PluginTaskResumeRefusal } from "../services/plugin-task-resume.js";
import { hasBoundRemoteStopReceipt, hasRemoteTerminationReceipt, remoteExecutionHasReleased, remoteExecutionHasStopped,
remoteTerminationReceipt } from "../services/remote-execution-termination.js";
import { environmentRuntimeService } from "../services/environment-runtime.js";
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
import { createPluginRunnerTaskOwner } from "../services/native-runtime/plugin-runner-task.js";
import { NativeExecutionContext } from "../services/native-runtime/native-execution-context.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
const support = await getEmbeddedPostgresTestSupport();
const databaseSuite = support.supported ? describe : describe.skip;
/** A resume refusal keeps its original message and names the check that failed. */
const refused = (reason: PluginTaskResumeRefusal) => `Plugin Runner retained session is not authorized (reason: ${reason})`;
databaseSuite("environment task lease authority", () => {
let db: ReturnType<typeof createDb>;
let stopDb: (() => Promise<void>) | undefined;
beforeAll(async () => {
const started = await startEmbeddedPostgresTestDatabase("environment-task-authority");
stopDb = started.cleanup; db = createDb(started.connectionString);
});
afterAll(async () => { await stopDb?.(); });
it.each(["dispatch", "writeback"])("fences stale owner %s through maintained physical task replacement", async mode => {
const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), pluginId = randomUUID(), runId = randomUUID(), leaseId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Warm candidate", issuePrefix: `T${companyId.slice(0, 7).replace(/-/g, "").toUpperCase()}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", adapterType: "paperclip_runner" });
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "fixture-warm-replacement", version: "1", status: "ready",
manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }] } as never });
await db.insert(environments).values({ id: environmentId, name: `Fixture ${environmentId}`, driver: "plugin",
config: { pluginKey: pluginId, driverKey: "tasks", driverConfig: {} } });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "manual", status: "running" });
await db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId, heartbeatRunId: runId,
provider: "plugin:fixture:tasks", providerLeaseId: "stopped-task", status: "active", leasePolicy: "reuse_by_environment",
metadata: { driver: "plugin", pluginId, pluginKey: pluginId, driverKey: "tasks", runnerTaskLease: true, runnerTaskReused: true } });
const calls: string[] = [], stoppedTasks: string[] = [];
let holdStatus = false, releaseStatus!: () => void, enterStatus!: () => void;
const statusEntered = new Promise<void>(resolve => { enterStatus = resolve; });
const statusReleased = new Promise<void>(resolve => { releaseStatus = resolve; });
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask", "environmentAcquireLease"] }),
call: async (_plugin: string, method: string, params: any, _timeout?: number, _sink?: unknown, settled?: () => void) => {
calls.push(method === "environmentTask" ? params.operation.kind : method);
if (method === "environmentTask" && params.operation.kind === "stop") stoppedTasks.push(params.taskId);
if (method === "environmentTask" && params.operation.kind === "status" && holdStatus) {
holdStatus = false; enterStatus(); await statusReleased;
}
try {
if (method === "environmentAcquireLease") return { providerLeaseId: "replacement-task" };
if (params.operation.kind === "status") return { kind: "status", taskId: params.taskId, phase: "completed", executionStopped: true };
if (params.operation.kind === "describe") return { kind: "description", taskId: params.taskId,
runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` }, executionContextVersion: 1 };
return { kind: "accepted", taskId: params.taskId };
} finally { settled?.(); }
},
} as unknown as PluginWorkerManager;
const context = new NativeExecutionContext({ binding: { companyId, agentId, runId } } as never, 1_000);
const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers });
const task = (operation: unknown, options?: { expectedTaskId?: string }) => runtime.task({ companyId, leaseId,
operation: operation as never, ...options });
const identity = { runnerInstanceId: "runner", environmentLeaseId: leaseId, runId, normalizedSessionId: "session", turnId: "turn", itemId: "item" };
const oldOwner = createPluginRunnerTaskOwner({ taskId: "stopped-task", task, projectIds: [], harness: "codex", adoptIdentity: identity });
const oldRegistration = await oldOwner.registration({} as never, runId);
holdStatus = mode === "writeback";
const lateStatus = holdStatus ? task({ kind: "status" }, { expectedTaskId: "stopped-task" }) : Promise.resolve();
if (holdStatus) await statusEntered;
try {
await expect(environmentRuntimeService(db, { pluginWorkerManager: workers }).task({ companyId, leaseId,
operation: { kind: "describe" }, executionContext: context })).resolves.toMatchObject({ kind: "description", taskId: "replacement-task" });
expect(calls).toEqual([...(mode === "writeback" ? ["status"] : []), "status", "complete", "environmentAcquireLease", "describe"]);
expect(context.pending).toBe(false);
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId));
expect(lease.providerLeaseId).toBe("replacement-task");
expect(lease.metadata?.runnerTaskReused).toBe(false);
expect(JSON.stringify(lease.metadata)).not.toContain(context.id);
releaseStatus(); await lateStatus;
const [afterLateStatus] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId));
expect(afterLateStatus.metadata?.runnerTaskExecutionStopped).toBe(false);
const failedCleanup = await oldRegistration.checkpoint("unsettled").then(() => undefined, error => error);
expect(stoppedTasks).not.toContain("replacement-task");
expect(failedCleanup).toBeInstanceOf(Error);
const nextOwner = createPluginRunnerTaskOwner({ taskId: "replacement-task", task, projectIds: [], harness: "codex", adoptIdentity: identity });
const nextRegistration = await nextOwner.registration({} as never, runId);
try { await expect(nextRegistration.checkpoint("unsettled")).resolves.toBeUndefined(); }
finally { await nextRegistration.release("detach"); }
expect(stoppedTasks).toEqual(["replacement-task"]);
} finally { releaseStatus(); await lateStatus.catch(() => undefined); context.finish(); await oldRegistration.release("detach"); }
});
it.each(["late", "post-result", "retiring"])("queues a replacement instead of attaching after %s startup cancellation", async mode => {
const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), pluginId = randomUUID(), runId = randomUUID(), leaseId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Warm candidate", issuePrefix: `T${companyId.slice(0, 7).replace(/-/g, "").toUpperCase()}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", adapterType: "paperclip_runner" });
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "fixture-warm-replacement", version: "1", status: "ready",
manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }] } as never });
await db.insert(environments).values({ id: environmentId, name: `Fixture ${environmentId}`, driver: "plugin",
config: { pluginKey: pluginId, driverKey: "tasks", driverConfig: {} } });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "manual", status: "running" });
await db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId, heartbeatRunId: runId,
provider: "plugin:fixture:tasks", providerLeaseId: "stopped-task", status: "active", leasePolicy: "reuse_by_environment",
metadata: { driver: "plugin", pluginId, pluginKey: pluginId, driverKey: "tasks", runnerTaskLease: true, runnerTaskReused: true } });
const calls: string[] = [];
let reply!: () => Promise<void>, acquiring!: () => void;
const entered = new Promise<void>(resolve => { acquiring = resolve; });
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask", "environmentAcquireLease"] }),
call: async (_plugin: string, method: string, params: any, _timeout?: number, _sink?: unknown, settled?: () => void, _contextId?: string, observe?: (result: any) => Promise<void>) => {
calls.push(method === "environmentTask" ? params.operation.kind : method);
try {
if (method === "environmentAcquireLease") return await new Promise(resolve => {
reply = async () => { const result = { providerLeaseId: "replacement-task", metadata: { binding: "fresh" } };
await observe?.(result);
if (mode === "retiring") await db.update(environmentLeases).set({ metadata: { driver: "plugin", pluginId, driverKey: "tasks", runnerTaskRetiring: true } }).where(eq(environmentLeases.id, leaseId));
resolve(result); };
acquiring();
});
if (params.operation.kind === "status") return { kind: "status", taskId: params.taskId, phase: "completed", executionStopped: true };
if (params.operation.kind === "describe") return { kind: "description", taskId: params.taskId,
runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` }, executionContextVersion: 1 };
return { kind: "accepted", taskId: params.taskId };
} finally { settled?.(); }
},
} as unknown as PluginWorkerManager;
const context = new NativeExecutionContext({ binding: { companyId, agentId, runId } } as never, 1_000);
if (mode === "post-result") {
const wait = context.wait.bind(context);
vi.spyOn(context, "wait").mockImplementation(operation => wait(async () => {
const value = await operation(); context.abort(new Error("Cancelled after recording")); return value;
}));
}
try {
const request = environmentRuntimeService(db, { pluginWorkerManager: workers }).task({ companyId, leaseId,
operation: { kind: "describe" }, executionContext: context });
const rejected = expect(request).rejects.toThrow("operation unavailable");
await entered;
if (mode === "late") {
context.abort(new Error("Startup cancelled"));
await rejected;
expect(context.pending).toBe(true);
}
await reply(); await rejected;
expect(context.pending).toBe(false);
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId));
expect(lease.providerLeaseId).toBe("stopped-task");
const [creation] = await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.runId, runId));
expect(creation.state).toBe("cleanup_pending");
expect(creation.metadata.runnerTaskCreationContextId).toBe(context.id);
const [cleanup] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, creation.leaseId!));
expect(cleanup).toMatchObject({ heartbeatRunId: runId, providerLeaseId: "replacement-task", status: "pending_cleanup" });
expect(cleanup.metadata).toMatchObject({ runnerTaskLease: true, providerMetadata: { binding: "fresh" } });
expect(calls).not.toContain("describe");
} finally { context.finish(); }
});
it.each([
{ retained: false, replaced: true },
{ retained: true, replaced: false },
])("replaces a stopped physical task for a later same-run attempt unless it retains a session (retained=$retained)", async ({ retained, replaced }) => {
const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), pluginId = randomUUID(), runId = randomUUID(), leaseId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Retry candidate", issuePrefix: `T${companyId.slice(0, 7).replace(/-/g, "").toUpperCase()}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", adapterType: "paperclip_runner" });
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "fixture-stopped-replacement", version: "1", status: "ready",
manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }] } as never });
await db.insert(environments).values({ id: environmentId, name: `Fixture ${environmentId}`, driver: "plugin",
config: { pluginKey: pluginId, driverKey: "tasks", driverConfig: {} } });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "manual", status: "running" });
await db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId, heartbeatRunId: runId,
provider: "plugin:fixture:tasks", providerLeaseId: "tombstoned-task", status: "active", leasePolicy: "reuse_by_environment",
metadata: { driver: "plugin", pluginId, pluginKey: pluginId, driverKey: "tasks", runnerTaskLease: true, runnerTaskReused: false,
runnerTaskDispatched: false, runnerTaskExecutionStopped: true, runnerTaskSessionRetained: retained,
runnerTaskSourceRecovery: { schema: "fixture" } } });
const calls: string[] = [];
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask", "environmentAcquireLease"] }),
call: async (_plugin: string, method: string, params: any, _timeout?: number, _sink?: unknown, settled?: () => void) => {
calls.push(method === "environmentTask" ? `${params.operation.kind}:${params.taskId}` : method);
try {
if (method === "environmentAcquireLease") return { providerLeaseId: "replacement-task" };
if (params.operation.kind === "status") return { kind: "status", taskId: params.taskId, phase: "cancelled",
executionStopped: true, sessionStateAvailable: retained };
if (params.operation.kind === "describe") return { kind: "description", taskId: params.taskId,
runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` }, executionContextVersion: 1 };
return { kind: "accepted", taskId: params.taskId };
} finally { settled?.(); }
},
} as unknown as PluginWorkerManager;
const context = new NativeExecutionContext({ binding: { companyId, agentId, runId } } as never, 1_000);
try {
await expect(environmentRuntimeService(db, { pluginWorkerManager: workers }).task({ companyId, leaseId,
operation: { kind: "describe" }, executionContext: context })).resolves.toMatchObject({ kind: "description",
taskId: replaced ? "replacement-task" : "tombstoned-task" });
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId));
if (replaced) {
expect(calls).toEqual(["status:tombstoned-task", "complete:tombstoned-task", "environmentAcquireLease", "describe:replacement-task"]);
expect(lease.providerLeaseId).toBe("replacement-task");
expect(lease.metadata).toMatchObject({ runnerTaskExecutionStopped: false, runnerTaskDispatched: false,
runnerTaskSessionRetained: false, runnerTaskSourceRecovery: { schema: "fixture" } });
} else {
expect(calls).toEqual(["describe:tombstoned-task"]);
expect(lease.providerLeaseId).toBe("tombstoned-task");
}
} finally { context.finish(); }
});
it("authorizes a new task lease only after scoped prior execution has stopped", async () => {
const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), pluginId = randomUUID();
const oldRunId = randomUUID(), nextRunId = randomUUID(), oldLeaseId = randomUUID(), nextLeaseId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Lease transition", issuePrefix: `T${companyId.slice(0, 7).replace(/-/g, "").toUpperCase()}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", role: "engineer", adapterType: "paperclip_runner" });
await db.insert(environments).values({ id: environmentId, name: `Fixture ${environmentId}`, driver: "plugin", config: {} });
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "fixture-continuation", version: "1.0.0", status: "ready", manifestJson: {
capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }],
} as never });
await db.insert(heartbeatRuns).values([
{ id: oldRunId, companyId, agentId, invocationSource: "manual", status: "succeeded" },
{ id: nextRunId, companyId, agentId, invocationSource: "manual", status: "running" },
]);
const pin = { driver: "plugin", pluginId, driverKey: "tasks", environmentId, runnerTaskLease: true,
runnerTaskProjectAuthValidation: { configurationId: "fixture", provider: "openai", projectIds: [] } };
const prior = { id: oldLeaseId, companyId, environmentId, heartbeatRunId: oldRunId, provider: "plugin:fixture:tasks", providerLeaseId: "old-task" };
await db.insert(environmentLeases).values([
{ ...prior, status: "released", leasePolicy: "ephemeral", cleanupStatus: "success", releasedAt: new Date(),
metadata: { ...pin, runnerTaskSessionRetained: true, runnerTaskIdentity: { runnerId: "runner", leaseId: oldLeaseId, runId: oldRunId, sessionId: "session", turnId: "prior-turn", itemId: "prior-item" }, remoteExecutionTermination: remoteTerminationReceipt(prior, { providerLeaseId: "old-task", state: "stopped" }) } },
{ id: nextLeaseId, companyId, environmentId, heartbeatRunId: nextRunId, provider: prior.provider, providerLeaseId: "new-task",
status: "active", leasePolicy: "ephemeral", metadata: pin },
]);
const priorIdentity = { runnerInstanceId: "runner", normalizedSessionId: "session", environmentLeaseId: oldLeaseId, runId: oldRunId };
const authorize = () => resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity });
await expect(authorize()).resolves.toBe("old-task");
for (const field of ["runnerInstanceId", "environmentLeaseId", "normalizedSessionId", "runId"]) {
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId,
priorIdentity: { ...priorIdentity, [field]: randomUUID() } })).rejects.toThrow(refused("predecessor_candidate_missing"));
}
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, oldRunId));
await expect(authorize()).rejects.toThrow(refused("predecessor_run_active"));
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, oldRunId));
await expect(resolvePluginTaskResume(db, { companyId: randomUUID(), leaseId: nextLeaseId, priorIdentity })).rejects.toThrow(refused("next_lease_missing"));
await db.update(environmentLeases).set({ metadata: { ...pin, pluginId: randomUUID() } }).where(eq(environmentLeases.id, nextLeaseId));
await expect(authorize()).rejects.toThrow(refused("provider_mismatch"));
await db.update(environmentLeases).set({ metadata: pin }).where(eq(environmentLeases.id, nextLeaseId));
await db.update(environmentLeases).set({ providerLeaseId: "old-task" }).where(eq(environmentLeases.id, nextLeaseId));
await expect(authorize()).rejects.toThrow(refused("same_provider_task"));
await db.update(environmentLeases).set({ providerLeaseId: "new-task" }).where(eq(environmentLeases.id, nextLeaseId));
const otherEnvironmentId = randomUUID();
await db.insert(environments).values({ id: otherEnvironmentId, name: `Fixture ${otherEnvironmentId}`, driver: "plugin", config: {} });
await db.update(environmentLeases).set({ environmentId: otherEnvironmentId }).where(eq(environmentLeases.id, nextLeaseId));
await expect(authorize()).rejects.toThrow(refused("scope_mismatch"));
await db.update(environmentLeases).set({ environmentId }).where(eq(environmentLeases.id, nextLeaseId));
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, nextRunId));
await expect(authorize()).rejects.toThrow(refused("next_run_not_running"));
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, nextRunId));
await expect(authorize()).resolves.toBe("old-task");
await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, oldLeaseId));
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, oldLeaseId));
let sessionStateAvailable = false;
const operations: string[] = [];
const call = vi.fn(async (_id: string, _method: string, params: any) => ["read_state", "archive_state"].includes(params.operation.kind)
? { kind: "state", taskId: params.taskId, state: { schema: "paperclip.runner.durable.state.v1", lifecycle: "suspended",
...(params.operation.identity ?? params.operation.priorIdentity) } }
: params.operation.kind === "status"
? (operations.push("status"), { kind: "status", taskId: params.taskId, phase: "completed", executionStopped: true, sessionStateAvailable })
: { kind: "accepted", taskId: params.taskId });
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }), call } as unknown as PluginWorkerManager;
const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers });
const runner = { runnerId: "runner", leaseId: oldLeaseId, sessionId: "session", runId: nextRunId,
turnId: "turn", itemId: "item", harness: "codex", protocolMin: 1, protocolMax: 3 };
const checkpointIdentity = { ...priorIdentity, turnId: "prior-turn", itemId: "prior-item" };
const [source] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, oldLeaseId));
const portableTargetLeaseId = randomUUID();
await db.update(environmentLeases).set({ metadata: { ...source.metadata, portableTaskSuccessor: {
sourceLeaseId: oldLeaseId, targetLeaseId: portableTargetLeaseId,
operation: { kind: "ensure_ownership", operationId: portableTargetLeaseId, sourceTaskId: "old-task",
sourceGeneration: "source-generation", targetGeneration: "target-generation", policy: "ordinary", identity: checkpointIdentity },
} } }).where(eq(environmentLeases.id, oldLeaseId));
await expect(authorize()).rejects.toThrow(refused("predecessor_candidate_missing"));
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).rejects.toThrow(refused("predecessor_candidate_missing"));
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task",
} })).rejects.toThrow("not authorized");
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "submit", preparedLaunchId: "fixture", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task",
} })).rejects.toThrow("not authorized");
expect(call).not.toHaveBeenCalled();
await db.update(environmentLeases).set({ metadata: { ...source.metadata, runnerTaskSessionRetained: false } }).where(eq(environmentLeases.id, oldLeaseId));
await expect(authorize()).rejects.toThrow(refused("predecessor_candidate_missing"));
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).resolves.toBe("old-task");
const inspect = () => runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task",
} });
await expect(inspect()).rejects.toThrow(refused("checkpoint_status_unconfirmed"));
await expect(authorize()).rejects.toThrow(refused("predecessor_candidate_missing"));
// Discovery cannot launch, and cannot bypass the original termination receipt.
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "submit", preparedLaunchId: "fixture", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task",
} })).rejects.toThrow("resume identity mismatch");
await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, oldLeaseId));
await expect(inspect()).rejects.toThrow(refused("predecessor_not_stopped"));
expect(operations).toEqual(["status"]);
await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, oldLeaseId));
sessionStateAvailable = true;
await expect(inspect()).resolves.toMatchObject({ kind: "state" });
expect(operations).toEqual(["status", "status"]);
await expect(authorize()).resolves.toBe("old-task");
for (const operation of [
{ kind: "read_state" as const, identity: checkpointIdentity, resumeFrom: "old-task" },
{ kind: "archive_state" as const, priorIdentity: checkpointIdentity, archiveKey: "a".repeat(24), resumeFrom: "old-task" },
]) {
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation })).resolves.toMatchObject({ kind: "state" });
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: { ...operation, resumeFrom: "other-task" } })).rejects.toThrow("resume identity mismatch");
}
for (const key of ["turnId", "itemId"]) {
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "read_state", identity: { ...checkpointIdentity, [key]: "foreign" }, resumeFrom: "old-task",
} })).rejects.toThrow("resume identity mismatch");
}
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "read_state", identity: { ...checkpointIdentity, runId: nextRunId }, resumeFrom: "old-task",
} })).rejects.toThrow("resume identity mismatch");
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "submit", preparedLaunchId: "fixture", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task",
} })).resolves.toMatchObject({ kind: "accepted" });
const [second] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, nextLeaseId));
await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success", releasedAt: new Date(),
metadata: { ...second.metadata, runnerTaskSessionRetained: true,
remoteExecutionTermination: remoteTerminationReceipt(second, { providerLeaseId: "new-task", state: "stopped" }) },
}).where(eq(environmentLeases.id, nextLeaseId));
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, nextRunId));
const thirdRunId = randomUUID(), thirdLeaseId = randomUUID();
await db.insert(heartbeatRuns).values({ id: thirdRunId, companyId, agentId, invocationSource: "manual", status: "running" });
await db.insert(environmentLeases).values({ id: thirdLeaseId, companyId, environmentId, heartbeatRunId: thirdRunId,
status: "active", provider: prior.provider, providerLeaseId: "third-task", metadata: pin });
await expect(resolvePluginTaskResume(db, { companyId, leaseId: thirdLeaseId,
priorIdentity: { ...priorIdentity, runId: nextRunId } })).resolves.toBe("new-task");
const submit = { kind: "submit", preparedLaunchId: "fixture", projectIds: [], bootstrapTicket: "fixture", runner: { ...runner, runId: thirdRunId }, resumeFrom: "new-task" } as const;
await expect(runtime.task({ companyId, leaseId: thirdLeaseId, operation: { ...submit, projectIds: [] } })).resolves.toMatchObject({ kind: "accepted" });
await expect(runtime.task({ companyId, leaseId: thirdLeaseId, operation: { ...submit, projectIds: [], runner } })).rejects.toThrow("identity mismatch");
// Same-run recovery must choose the latest task in the saved lineage, not an earlier physical lease.
const [third] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, thirdLeaseId));
await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success", releasedAt: new Date(),
metadata: { ...third.metadata, runnerTaskSessionRetained: true,
remoteExecutionTermination: remoteTerminationReceipt(third, { providerLeaseId: "third-task", state: "stopped" }) },
}).where(eq(environmentLeases.id, thirdLeaseId));
const retryLeaseId = randomUUID();
await db.insert(environmentLeases).values({ id: retryLeaseId, companyId, environmentId, heartbeatRunId: thirdRunId,
status: "active", provider: prior.provider, providerLeaseId: "retry-task", metadata: pin });
const retryIdentity = { ...checkpointIdentity, runId: thirdRunId, turnId: "turn", itemId: "item" };
await expect(resolvePluginTaskResume(db, { companyId, leaseId: retryLeaseId,
priorIdentity: retryIdentity })).resolves.toBe("third-task");
await expect(runtime.task({ companyId, leaseId: retryLeaseId, operation: {
kind: "read_state", identity: retryIdentity, resumeFrom: "third-task",
} })).resolves.toMatchObject({ kind: "state", state: { ...retryIdentity, lifecycle: "suspended" } });
await expect(runtime.task({ companyId, leaseId: retryLeaseId, operation: { ...submit, projectIds: [], resumeFrom: "third-task" } })).resolves.toMatchObject({ kind: "accepted" });
const [retry] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, retryLeaseId));
await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success", releasedAt: new Date(),
metadata: { ...retry.metadata, runnerTaskSessionRetained: true,
remoteExecutionTermination: remoteTerminationReceipt(retry, { providerLeaseId: "retry-task", state: "stopped" }) },
}).where(eq(environmentLeases.id, retryLeaseId));
const finalLeaseId = randomUUID();
await db.insert(environmentLeases).values({ id: finalLeaseId, companyId, environmentId, heartbeatRunId: thirdRunId,
status: "active", provider: prior.provider, providerLeaseId: "final-task", metadata: pin });
await expect(resolvePluginTaskResume(db, { companyId, leaseId: finalLeaseId,
priorIdentity: { ...priorIdentity, runId: thirdRunId } })).resolves.toBe("retry-task");
});
async function seedTaskPlugin(name: string) {
const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), pluginId = randomUUID();
await db.insert(companies).values({ id: companyId, name, issuePrefix: `T${companyId.slice(0, 7).replace(/-/g, "").toUpperCase()}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", adapterType: "paperclip_runner" });
await db.insert(environments).values({ id: environmentId, name: `Fixture ${environmentId}`, driver: "plugin",
config: { pluginKey: pluginId, driverKey: "tasks", driverConfig: {} } });
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: `fixture-${name}`, version: "1.0.0", status: "ready", manifestJson: {
capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }],
} as never });
const pin = { driver: "plugin", pluginId, driverKey: "tasks", environmentId, runnerTaskLease: true,
runnerTaskProjectAuthValidation: { configurationId: "fixture", provider: "openai", projectIds: [] } };
return { companyId, agentId, environmentId, pin };
}
it("records a per-turn task's stop receipt when it settles, before its held lease release", async () => {
const { companyId, agentId, environmentId, pin } = await seedTaskPlugin("per-turn-stop");
const oldRunId = randomUUID(), nextRunId = randomUUID(), oldLeaseId = randomUUID(), nextLeaseId = randomUUID();
await db.insert(heartbeatRuns).values([
{ id: oldRunId, companyId, agentId, invocationSource: "manual", status: "running" },
{ id: nextRunId, companyId, agentId, invocationSource: "manual", status: "running" },
]);
const prior = { id: oldLeaseId, companyId, environmentId, heartbeatRunId: oldRunId, provider: "plugin:fixture:tasks", providerLeaseId: "old-task" };
const oldIdentity = { runnerId: "runner", leaseId: oldLeaseId, runId: oldRunId, sessionId: "session", turnId: "prior-turn", itemId: "prior-item" };
await db.insert(environmentLeases).values([
{ ...prior, status: "active", leasePolicy: "ephemeral", metadata: { ...pin, runnerTaskDispatched: true,
runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false, runnerTaskIdentity: oldIdentity } },
{ id: nextLeaseId, companyId, environmentId, heartbeatRunId: nextRunId, provider: prior.provider, providerLeaseId: "new-task",
status: "active", leasePolicy: "ephemeral", metadata: pin },
]);
let executionStopped = false;
const call = vi.fn(async (_id: string, method: string, params: any) => method !== "environmentTask" ? undefined
: ["read_state", "archive_state"].includes(params.operation.kind)
? { kind: "state", taskId: params.taskId, state: { schema: "paperclip.runner.durable.state.v1", lifecycle: "suspended",
...(params.operation.identity ?? params.operation.priorIdentity) } }
: params.operation.kind === "status"
? { kind: "status", taskId: params.taskId, phase: executionStopped ? "completed" : "running", executionStopped, sessionStateAvailable: executionStopped }
: { kind: "accepted", taskId: params.taskId });
const workers = { isRunning: () => true, getWorker: () => ({ supportedMethods: ["environmentTask", "environmentReleaseLease"] }), call } as unknown as PluginWorkerManager;
const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers });
const readLease = async (id: string) => (await db.select().from(environmentLeases).where(eq(environmentLeases.id, id)))[0]!;
const priorIdentity = { runnerInstanceId: "runner", normalizedSessionId: "session", environmentLeaseId: oldLeaseId, runId: oldRunId };
const checkpointIdentity = { ...priorIdentity, turnId: "prior-turn", itemId: "prior-item" };
const runner = { runnerId: "runner", leaseId: oldLeaseId, sessionId: "session", runId: nextRunId,
turnId: "turn", itemId: "item", harness: "codex", protocolMin: 1, protocolMax: 3 };
const authorize = (inspectCheckpoint?: boolean) => resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint });
// Neither a running status nor a completion without a provider stop is stop proof.
await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "status" } });
await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "complete" } });
expect((await readLease(oldLeaseId)).metadata).not.toHaveProperty("remoteExecutionTermination");
expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(false);
// The turn checkpoint stops and completes the task; release of the lease is still held.
executionStopped = true;
await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "stop" } });
await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "status" } });
await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "complete" } });
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, oldRunId));
const settled = await readLease(oldLeaseId);
expect(settled).toMatchObject({ status: "active", releasedAt: null, cleanupStatus: null, metadata: {
runnerTaskExecutionStopped: true, runnerTaskSessionRetained: true, runnerTaskDispatched: false,
remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId, runId: oldRunId, leaseId: oldLeaseId,
provider: prior.provider, providerLeaseId: "old-task", state: "stopped" } } });
expect(hasBoundRemoteStopReceipt(settled)).toBe(true);
expect(hasRemoteTerminationReceipt(settled)).toBe(false);
expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(true);
expect(await remoteExecutionHasReleased(db, companyId, oldRunId)).toBe(false);
// The follow-up can inspect and launch from the checkpoint while the predecessor is unreleased.
await expect(authorize(true)).resolves.toBe("old-task");
await expect(authorize()).resolves.toBe("old-task");
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task",
} })).resolves.toMatchObject({ kind: "state", state: { ...checkpointIdentity, lifecycle: "suspended" } });
// A receipt for another physical task, a second unstopped predecessor lease, or a release
// that started without settling still refuses authority.
await db.update(environmentLeases).set({ metadata: { ...settled.metadata, remoteExecutionTermination:
remoteTerminationReceipt({ ...prior, providerLeaseId: "other-task" }, { providerLeaseId: "other-task", state: "stopped" }) } })
.where(eq(environmentLeases.id, oldLeaseId));
await expect(authorize(true)).rejects.toThrow(refused("predecessor_not_stopped"));
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
await db.update(environmentLeases).set({ metadata: settled.metadata }).where(eq(environmentLeases.id, oldLeaseId));
const siblingId = randomUUID();
await db.insert(environmentLeases).values({ id: siblingId, companyId, environmentId, heartbeatRunId: oldRunId,
provider: prior.provider, providerLeaseId: "sibling-task", status: "active", leasePolicy: "ephemeral", metadata: pin });
await expect(authorize(true)).rejects.toThrow(refused("predecessor_not_stopped"));
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
await db.delete(environmentLeases).where(eq(environmentLeases.id, siblingId));
await db.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", releasedAt: new Date() })
.where(eq(environmentLeases.id, oldLeaseId));
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
await db.update(environmentLeases).set({ status: "active", cleanupStatus: null, releasedAt: null }).where(eq(environmentLeases.id, oldLeaseId));
await expect(authorize()).resolves.toBe("old-task");
// Dispatching new execution invalidates a stop receipt on that lease.
const next = await readLease(nextLeaseId);
await db.update(environmentLeases).set({ metadata: { ...next.metadata, runnerTaskExecutionStopped: true,
remoteExecutionTermination: remoteTerminationReceipt(next, { providerLeaseId: "new-task", state: "stopped" }) } })
.where(eq(environmentLeases.id, nextLeaseId));
expect(await remoteExecutionHasStopped(db, companyId, nextRunId)).toBe(true);
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "submit", preparedLaunchId: "fixture", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task",
} })).resolves.toMatchObject({ kind: "accepted" });
expect((await readLease(nextLeaseId)).metadata).not.toHaveProperty("remoteExecutionTermination");
expect(await remoteExecutionHasStopped(db, companyId, nextRunId)).toBe(false);
// Release still writes its own receipt over the settlement receipt.
const releaseErrors: unknown[] = [];
await runtime.releaseRunLeases(oldRunId, "released", (_id, error) => { releaseErrors.push(error); });
expect(releaseErrors).toEqual([]);
const released = await readLease(oldLeaseId);
expect(released).toMatchObject({ status: "released", cleanupStatus: "success",
metadata: { remoteExecutionTermination: { leaseId: oldLeaseId, providerLeaseId: "old-task", state: "stopped" } } });
expect(hasRemoteTerminationReceipt(released)).toBe(true);
expect(await remoteExecutionHasReleased(db, companyId, oldRunId)).toBe(true);
});
it("stops counting a settled task's stop receipt once its lease release starts", async () => {
const { companyId, agentId, environmentId, pin } = await seedTaskPlugin("release-owns-receipt");
const oldRunId = randomUUID(), nextRunId = randomUUID(), oldLeaseId = randomUUID(), nextLeaseId = randomUUID();
await db.insert(heartbeatRuns).values([
{ id: oldRunId, companyId, agentId, invocationSource: "manual", status: "succeeded" },
{ id: nextRunId, companyId, agentId, invocationSource: "manual", status: "running" },
]);
const prior = { id: oldLeaseId, companyId, environmentId, heartbeatRunId: oldRunId, provider: "plugin:fixture:tasks", providerLeaseId: "old-task" };
// The predecessor's task settled in its run: stopped, session retained, stop receipt recorded.
const settledMetadata = { ...pin, runnerTaskDispatched: false, runnerTaskExecutionStopped: true, runnerTaskSessionRetained: true,
runnerTaskIdentity: { runnerId: "runner", leaseId: oldLeaseId, runId: oldRunId, sessionId: "session", turnId: "turn", itemId: "item" },
remoteExecutionTermination: remoteTerminationReceipt(prior, { providerLeaseId: "old-task", state: "stopped" }) };
await db.insert(environmentLeases).values([
{ ...prior, status: "active", leasePolicy: "ephemeral", metadata: settledMetadata },
{ id: nextLeaseId, companyId, environmentId, heartbeatRunId: nextRunId, provider: prior.provider, providerLeaseId: "new-task",
status: "active", leasePolicy: "ephemeral", metadata: pin },
]);
let releaseEntered!: () => void, finishRelease!: () => void;
const entered = new Promise<void>(resolve => { releaseEntered = resolve; });
const held = new Promise<void>(resolve => { finishRelease = resolve; });
const call = vi.fn(async (_id: string, method: string) => {
if (method !== "environmentReleaseLease") throw new Error(`unexpected ${method}`);
releaseEntered();
await held;
});
const workers = { isRunning: () => true, getWorker: () => ({ supportedMethods: ["environmentTask", "environmentReleaseLease"] }), call } as unknown as PluginWorkerManager;
const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers });
const priorIdentity = { runnerInstanceId: "runner", normalizedSessionId: "session", environmentLeaseId: oldLeaseId, runId: oldRunId };
const authorize = (inspectCheckpoint?: boolean) => resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint });
await expect(authorize()).resolves.toBe("old-task");
// Each marker that release, destroy, or startup and creation cleanup write first takes the receipt over.
for (const marker of [{ pluginLeaseCleanupMethod: "release" }, { pluginLeaseCleanupMethod: "destroy" }, { runnerTaskRetiring: true }]) {
await db.update(environmentLeases).set({ metadata: { ...settledMetadata, ...marker } }).where(eq(environmentLeases.id, oldLeaseId));
expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(false);
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
}
await db.update(environmentLeases).set({ metadata: settledMetadata }).where(eq(environmentLeases.id, oldLeaseId));
// Hold the real release inside the provider call: the lease is still active with its settlement receipt.
const releaseErrors: unknown[] = [];
const releasing = runtime.releaseRunLeases(oldRunId, "released", (_id, error) => { releaseErrors.push(error); });
await entered;
const inFlight = (await db.select().from(environmentLeases).where(eq(environmentLeases.id, oldLeaseId)))[0]!;
expect(inFlight).toMatchObject({ status: "active", releasedAt: null,
metadata: { pluginLeaseCleanupMethod: "release", runnerTaskRetiring: true } });
expect(hasBoundRemoteStopReceipt(inFlight)).toBe(true);
expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(false);
await expect(authorize(true)).rejects.toThrow(refused("predecessor_not_stopped"));
await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped"));
// Release settles with its own receipt, which authorizes the follow-up.
finishRelease();
await releasing;
expect(releaseErrors).toEqual([]);
const released = (await db.select().from(environmentLeases).where(eq(environmentLeases.id, oldLeaseId)))[0]!;
expect(released).toMatchObject({ status: "released", cleanupStatus: "success" });
expect(hasRemoteTerminationReceipt(released)).toBe(true);
expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(true);
await expect(authorize(true)).resolves.toBe("old-task");
await expect(authorize()).resolves.toBe("old-task");
});
it("still records stop receipts for every warm owner and retires retained owners", async () => {
const { companyId, agentId, environmentId, pin } = await seedTaskPlugin("warm-stop");
const priorRunId = randomUUID(), currentRunId = randomUUID(), priorLeaseId = randomUUID(), currentLeaseId = randomUUID();
await db.insert(heartbeatRuns).values([
{ id: priorRunId, companyId, agentId, invocationSource: "manual", status: "succeeded" },
{ id: currentRunId, companyId, agentId, invocationSource: "manual", status: "running" },
]);
const shared = { companyId, environmentId, provider: "plugin:fixture:tasks", providerLeaseId: "warm-task", leasePolicy: "reuse_by_environment" as const };
await db.insert(environmentLeases).values([
{ ...shared, id: priorLeaseId, heartbeatRunId: priorRunId, status: "retained", metadata: pin },
{ ...shared, id: currentLeaseId, heartbeatRunId: currentRunId, status: "active", metadata: { ...pin, runnerTaskDispatched: true } },
]);
const call = vi.fn(async (_id: string, _method: string, params: any) => params.operation.kind === "status"
? { kind: "status", taskId: params.taskId, phase: "completed", executionStopped: true, sessionStateAvailable: false }
: { kind: "accepted", taskId: params.taskId });
const runtime = environmentRuntimeService(db, { pluginWorkerManager: { getWorker: () => ({ supportedMethods: ["environmentTask"] }), call } as unknown as PluginWorkerManager });
await runtime.task({ companyId, leaseId: currentLeaseId, operation: { kind: "status" } });
await runtime.task({ companyId, leaseId: currentLeaseId, operation: { kind: "complete" } });
const leases = await db.select().from(environmentLeases).where(eq(environmentLeases.providerLeaseId, "warm-task"));
const priorLease = leases.find(lease => lease.id === priorLeaseId)!, currentLease = leases.find(lease => lease.id === currentLeaseId)!;
expect(priorLease).toMatchObject({ status: "released", cleanupStatus: "success",
metadata: { remoteExecutionTermination: { leaseId: priorLeaseId, runId: priorRunId, state: "stopped" } } });
expect(currentLease).toMatchObject({ status: "active",
metadata: { remoteExecutionTermination: { leaseId: currentLeaseId, runId: currentRunId, state: "stopped" } } });
expect(hasRemoteTerminationReceipt(priorLease)).toBe(true);
expect(await remoteExecutionHasStopped(db, companyId, priorRunId)).toBe(true);
expect(await remoteExecutionHasStopped(db, companyId, currentRunId)).toBe(true);
expect(await remoteExecutionHasReleased(db, companyId, currentRunId)).toBe(false);
});
});