mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
fix(native): stop or release unclosable idle warm Runners on environment delete, and say why a close failed (#20957)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown
## Linked Issues or Issue Description
**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.
**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:
1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.
The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.
**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.
**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.
## What Changed
- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.
## Verification
- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches 7e8901aa60, not current master, so each wave
ran this change cherry-picked onto 7e8901aa60:
- `w-20261111T210049-s6111` (first commit only): the close reason is now
logged: `close failed: Daytona sandbox lease 192606aa… is no longer
active.` The delete stayed 409 with that reason. The new check failed
the wave correctly.
- `w-20261111T222826-s6112` (all commits): two W2Q owners were released
with `sandbox was already destroyed`. The environment was deleted. 0
sandboxes left running. The warm-shutdown check passed. No 409 from a
warm session.
- Not verified: a wave where the graceful close fails and the forced
Launcher stop succeeds on a live sandbox (that path is covered by the
unit test). The wave's task failures (`T:first_run_succeeded`,
`R3:retry_needed`) are other open defects. They are also on master.
## Risks
- Environment delete now stops an idle Runner whose close did not
settle. The delete already has the operator's consent to destroy those
sandboxes. The workspace-export and restore-source guards on sandbox
destruction are unchanged. Instruction files are collected after the
confirmed stop, as in #20844.
- An owner is released without contact only when a lease for the same
company, environment and sandbox has a valid `destroyed` termination
receipt. A `stopped` receipt is not enough, because a stopped sandbox
can be resumed.
- `closeIdleWarmNativeSessions` results have a new `failures` field.
Callers that read only the counts are unchanged.
- No persisted state changes. Warm owners live only in controller
memory, so older saved state and mid-run deploys are not affected.
- The new harness check can fail a warm restart-fault wave that leaves a
warm session unclosed at controller shutdown. That is a real leaked
Runner.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs, chaos waves) in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
068d814735
commit
c45f54ee74
11 files changed
+410
-44
No files matched your search
@@ -11433,7 +11433,7 @@ describe("native warm session supervision", () => {
|
||||
expect(collectStopped).not.toHaveBeenCalled();
|
||||
expect(readNativeSessionIdleWork()).toBe("present");
|
||||
close.mockResolvedValue(undefined);
|
||||
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0 });
|
||||
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
expect(collectStopped).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
@@ -11459,7 +11459,7 @@ describe("native warm session supervision", () => {
|
||||
expect(close).toHaveBeenCalledOnce();
|
||||
expect(collectStopped).toHaveBeenCalledOnce();
|
||||
expect(close.mock.invocationCallOrder[0]).toBeLessThan(collectStopped.mock.invocationCallOrder[0]!);
|
||||
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 0, busy: 0, failed: 0 });
|
||||
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 0, busy: 0, failed: 0, failures: [] });
|
||||
});
|
||||
|
||||
it.each(["failed", "cancelled"] as const)("collects edits written during %s provider shutdown after the warm checkpoint", async (runTerminalState) => {
|
||||
@@ -11716,7 +11716,7 @@ describe("native warm session supervision", () => {
|
||||
releaseClose();
|
||||
close.mockResolvedValue(undefined);
|
||||
expect(await closeWarmNativeSessionsForEnvironment({ environmentId: identity, reason: "test cleanup" }))
|
||||
.toEqual({ closed: changed && !closeFails ? 0 : 1, busy: 0, failed: 0 });
|
||||
.toEqual({ closed: changed && !closeFails ? 0 : 1, busy: 0, failed: 0, failures: [] });
|
||||
expect(collectStopped).toHaveBeenCalledOnce();
|
||||
}
|
||||
});
|
||||
@@ -12292,6 +12292,122 @@ describe("native warm session supervision", () => {
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["contained", "handoff-unconfirmed"] as const)(
|
||||
"stops an idle warm owner whose close cannot settle when its environment is deleted (%s)", async mode => {
|
||||
const { base } = fixture(`idle-unsettled-${mode}`);
|
||||
const environmentId = `idle-unsettled-env-${mode}`;
|
||||
const identity = { runnerInstanceId: "runner", environmentLeaseId: "lease", runId: base.binding.runId,
|
||||
normalizedSessionId: base.session.normalizedSessionId!, turnId: "turn", itemId: "item" };
|
||||
let stopped = false;
|
||||
const task = vi.fn(async (operation: any) => {
|
||||
if (operation.kind === "describe") return { kind: "description", taskId: "parked-task", launchPreparationVersion: 1,
|
||||
projectAuthCheckVersion: 1, runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` } };
|
||||
if (operation.kind === "stop") stopped = true;
|
||||
if (operation.kind === "status") return { kind: "status", taskId: "parked-task", phase: stopped ? "cancelled" : "running", executionStopped: stopped };
|
||||
if (operation.kind === "complete" && mode === "handoff-unconfirmed") throw new Error("handoff unconfirmed");
|
||||
return { kind: "accepted", taskId: "parked-task" };
|
||||
});
|
||||
const createOwner = pluginTaskRuntime.createPluginRunnerTaskOwner;
|
||||
let owner!: ReturnType<typeof createOwner>;
|
||||
const ownerFactory = vi.spyOn(pluginTaskRuntime, "createPluginRunnerTaskOwner").mockImplementation(input => {
|
||||
owner = createOwner({ ...input, adoptIdentity: identity });
|
||||
return owner;
|
||||
});
|
||||
onTestFinished(() => ownerFactory.mockRestore());
|
||||
const backend = await createRunnerdBackend({ db: leaseDb(base),
|
||||
execution: { ...base, session: { ...base.session, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } } },
|
||||
runnerInstanceId: "runner", runnerIngressAuthorized: true,
|
||||
runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".", leaseId: "lease", runnerTask: task } });
|
||||
const registration = await owner.registration({} as never, base.binding.runId);
|
||||
const failure = new NativeSessionCloseUnrecoverableError({ runnerSuspended: false, providerDrained: true,
|
||||
semanticTools: { pending: [{ callId: "question-call" }] } }, false);
|
||||
let closePromise: Promise<void> | undefined;
|
||||
const close = vi.fn(() => closePromise ??= (async () => {
|
||||
await registration.release();
|
||||
await registration.checkpoint("unsettled").catch(() => undefined);
|
||||
throw failure;
|
||||
})());
|
||||
const session = { close };
|
||||
backend.bindManagedSession(session as never);
|
||||
const collectStopped = vi.fn(async () => { expect(owner.isRetired()).toBe(true); });
|
||||
const retirementFailed = vi.fn(async () => {});
|
||||
state.execute.mockReset().mockImplementationOnce(async options => {
|
||||
await options.onSession?.(session); return result;
|
||||
});
|
||||
try {
|
||||
await executePaperclipNativeSession({ db: leaseDb(base), execution: base, runnerInstanceId: "runner",
|
||||
instructionWorkingCopy: { hasChanges: async () => false, collectStopped, retirementFailed },
|
||||
runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId, remoteCwd: `/tmp/${environmentId}`,
|
||||
sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: "parked-sandbox" } } });
|
||||
const graceful = await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "idle sweep" });
|
||||
expect(graceful).toMatchObject({ closed: 0, failed: 1, failures: [{ runId: base.binding.runId,
|
||||
providerLeaseId: "parked-sandbox", environmentId }] });
|
||||
expect(graceful.failures[0]!.reason).toContain("native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint");
|
||||
expect(graceful.failures[0]!.reason).toContain("question-call");
|
||||
expect(collectStopped).not.toHaveBeenCalled();
|
||||
const deleted = await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "environment deleted", forceStopUnsettled: true });
|
||||
expect(task.mock.calls.filter(([operation]) => operation.kind === "stop")).toHaveLength(1);
|
||||
expect(task.mock.calls.some(([operation]) => operation.kind === "read_state")).toBe(false);
|
||||
if (mode === "contained") {
|
||||
expect(deleted).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
expect(collectStopped).toHaveBeenCalledOnce();
|
||||
expect(await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "environment deleted", forceStopUnsettled: true }))
|
||||
.toEqual({ closed: 0, busy: 0, failed: 0, failures: [] });
|
||||
} else {
|
||||
expect(deleted).toMatchObject({ closed: 0, failed: 1, failures: [{ runId: base.binding.runId }] });
|
||||
expect(deleted.failures[0]!.reason).toContain("runner did not durably suspend before checkpoint");
|
||||
expect(deleted.failures[0]!.reason).toContain("forced Runner stop failed: handoff unconfirmed");
|
||||
expect(collectStopped).not.toHaveBeenCalled();
|
||||
}
|
||||
} finally {
|
||||
close.mockResolvedValue(undefined);
|
||||
collectStopped.mockResolvedValue(undefined);
|
||||
expect(await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "test cleanup" })).toMatchObject({ failed: 0 });
|
||||
ownerFactory.mockRestore();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["destroyed", "stopped", "unproven"] as const)(
|
||||
"releases an idle warm owner whose sandbox lease is gone when its environment is deleted (%s)", async receipt => {
|
||||
const { base } = fixture(`idle-gone-${receipt}`);
|
||||
const environmentId = `idle-gone-env-${receipt}`;
|
||||
const inactive = new Error("Daytona sandbox lease gone-sandbox is no longer active.");
|
||||
const close = vi.fn(async (): Promise<void> => { throw inactive; });
|
||||
const retirementFailed = vi.fn(async () => {});
|
||||
state.execute.mockReset().mockImplementationOnce(async options => {
|
||||
await options.onSession?.({ close }); return result;
|
||||
});
|
||||
const lease = { id: "lease-gone", companyId: base.binding.companyId, environmentId, heartbeatRunId: "pending-cleanup-run",
|
||||
provider: "daytona", providerLeaseId: "gone-sandbox", status: "released", releasedAt: new Date(), cleanupStatus: "success",
|
||||
leasePolicy: "reuse_by_environment",
|
||||
metadata: receipt === "unproven" ? {} : { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1",
|
||||
companyId: base.binding.companyId, runId: "pending-cleanup-run", leaseId: "lease-gone", provider: "daytona",
|
||||
providerLeaseId: "gone-sandbox", state: receipt, confirmedAt: new Date().toISOString() } } };
|
||||
const where = vi.fn(async () => [lease]);
|
||||
const db = { select: () => ({ from: () => ({ where }) }) } as never;
|
||||
try {
|
||||
await executePaperclipNativeSession({ db: leaseDb(base), execution: base, runnerInstanceId: "runner",
|
||||
instructionWorkingCopy: { hasChanges: async () => false, collectStopped: async () => {}, retirementFailed },
|
||||
runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId, remoteCwd: `/tmp/${environmentId}`,
|
||||
sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: "gone-sandbox" } } });
|
||||
const deleted = await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "environment deleted", forceStopUnsettled: true, db });
|
||||
expect(where).toHaveBeenCalledOnce();
|
||||
if (receipt === "destroyed") {
|
||||
expect(deleted).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
expect(await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "environment deleted", forceStopUnsettled: true, db }))
|
||||
.toEqual({ closed: 0, busy: 0, failed: 0, failures: [] });
|
||||
} else {
|
||||
expect(deleted).toMatchObject({ closed: 0, failed: 1, failures: [{ runId: base.binding.runId, providerLeaseId: "gone-sandbox",
|
||||
reason: "close failed: Daytona sandbox lease gone-sandbox is no longer active." }] });
|
||||
}
|
||||
} finally {
|
||||
close.mockResolvedValue(undefined);
|
||||
await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "test cleanup" });
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("retains failed retirement for retry and never launches over an uncontained owner", async () => {
|
||||
const { base, next, run } = fixture("identity-close-failure");
|
||||
const close = vi.fn().mockRejectedValueOnce(new Error("containment unavailable")).mockResolvedValue(undefined);
|
||||
@@ -12474,7 +12590,7 @@ describe("native warm session supervision", () => {
|
||||
await executePaperclipNativeSession({ db: leaseDb(warmExecution), execution: warmExecution, runnerInstanceId: "runner" });
|
||||
expect(readNativeSessionIdleWork()).toBe("present");
|
||||
if (retirement === "controller restart") {
|
||||
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 0, busy: 0, failed: 1 });
|
||||
expect(await closeIdleWarmNativeSessionsForRestart()).toMatchObject({ closed: 0, busy: 0, failed: 1 });
|
||||
} else {
|
||||
await vi.waitFor(() => expect(close).toHaveBeenCalledOnce());
|
||||
}
|
||||
@@ -12486,7 +12602,7 @@ describe("native warm session supervision", () => {
|
||||
expect(state.execute).toHaveBeenCalledOnce();
|
||||
expect(readNativeSessionIdleWork()).toBe("present");
|
||||
close.mockResolvedValue(undefined);
|
||||
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0 });
|
||||
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
expect(close).toHaveBeenCalledTimes(3);
|
||||
expect(readNativeSessionIdleWork()).toBe("none");
|
||||
} finally {
|
||||
@@ -12521,18 +12637,18 @@ describe("native warm session supervision", () => {
|
||||
it("does no work when no warm owner exists", async () => {
|
||||
await closeIdleWarmNativeSessionsForRestart();
|
||||
expect(await prepareSleep({ canRetire: () => true }))
|
||||
.toEqual({ closed: 0, busy: 0, failed: 0 });
|
||||
.toEqual({ closed: 0, busy: 0, failed: 0, failures: [] });
|
||||
expect(readNativeSessionIdleWork()).toBe("none");
|
||||
});
|
||||
|
||||
it("leaves an idle owner intact without current drain authority", async () => {
|
||||
const f = await start("sleep-stale-hold");
|
||||
expect(await prepareSleep({ canRetire: () => false }))
|
||||
.toEqual({ closed: 0, busy: 0, failed: 0 });
|
||||
.toEqual({ closed: 0, busy: 0, failed: 0, failures: [] });
|
||||
expect(f.close).not.toHaveBeenCalled();
|
||||
expect(readNativeSessionIdleWork()).toBe("present");
|
||||
expect(await prepareSleep({ canRetire: () => true }))
|
||||
.toEqual({ closed: 1, busy: 0, failed: 0 });
|
||||
.toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
expect(f.close).toHaveBeenCalledExactlyOnceWith({ reason: "tenant idle sleep" });
|
||||
expect(readNativeSessionIdleWork()).toBe("none");
|
||||
});
|
||||
@@ -12547,7 +12663,7 @@ describe("native warm session supervision", () => {
|
||||
heartbeat.getTaskDrainStatus();
|
||||
expect(f.close).not.toHaveBeenCalled();
|
||||
expect(await heartbeat.prepareIdleWarmNativeSessionsForSleep({ canRetire: () => true }))
|
||||
.toEqual({ closed: 1, busy: 0, failed: 0 });
|
||||
.toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
expect(readNativeSessionIdleWork()).toBe("none");
|
||||
});
|
||||
|
||||
@@ -12563,11 +12679,11 @@ describe("native warm session supervision", () => {
|
||||
expect(readNativeSessionIdleWork()).toBe("present");
|
||||
current = false;
|
||||
finish();
|
||||
expect(await preparation).toEqual({ closed: 1, busy: 0, failed: 0 });
|
||||
expect(await preparation).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
expect(second.close).not.toHaveBeenCalled();
|
||||
expect(readNativeSessionIdleWork()).toBe("present");
|
||||
expect(await prepareSleep({ canRetire: () => true }))
|
||||
.toEqual({ closed: 1, busy: 0, failed: 0 });
|
||||
.toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
expect(readNativeSessionIdleWork()).toBe("none");
|
||||
} finally { finish(); await preparation; }
|
||||
});
|
||||
@@ -12576,13 +12692,13 @@ describe("native warm session supervision", () => {
|
||||
const close = vi.fn(async () => undefined);
|
||||
await start("sleep-busy-turn", close, async () => {
|
||||
expect(await prepareSleep({ canRetire: () => true }))
|
||||
.toEqual({ closed: 0, busy: 1, failed: 0 });
|
||||
.toEqual({ closed: 0, busy: 1, failed: 0, failures: [] });
|
||||
expect(close).not.toHaveBeenCalled();
|
||||
});
|
||||
expect(close).not.toHaveBeenCalled();
|
||||
expect(readNativeSessionIdleWork()).toBe("present");
|
||||
expect(await prepareSleep({ canRetire: () => true }))
|
||||
.toEqual({ closed: 1, busy: 0, failed: 0 });
|
||||
.toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
});
|
||||
|
||||
it("collects managed edits after the provider checkpoint before clearing the owner", async () => {
|
||||
@@ -12604,7 +12720,7 @@ describe("native warm session supervision", () => {
|
||||
runId: "sleep-files", root, preparationKey: "sleep-files", hasChanges: async () => false, collectStopped,
|
||||
});
|
||||
expect(await prepareSleep({ canRetire: () => true }))
|
||||
.toEqual({ closed: 1, busy: 0, failed: 0 });
|
||||
.toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
expect(await readFile(saved, "utf8")).toBe("Edits collected from stopped provider");
|
||||
expect(JSON.parse(await readFile(providerCheckpoint, "utf8"))).toEqual({ session: "sleep-files", resume: "retained" });
|
||||
expect(collectStopped).toHaveBeenCalledOnce();
|
||||
@@ -12620,10 +12736,10 @@ describe("native warm session supervision", () => {
|
||||
const close = vi.fn(async () => undefined).mockRejectedValueOnce(failure);
|
||||
const f = await start("sleep-retry", close);
|
||||
expect(await prepareSleep({ canRetire: () => true }))
|
||||
.toEqual({ closed: 0, busy: 0, failed: 1 });
|
||||
.toMatchObject({ closed: 0, busy: 0, failed: 1 });
|
||||
expect(readNativeSessionIdleWork()).toBe("present");
|
||||
expect(await prepareSleep({ canRetire: () => true }))
|
||||
.toEqual({ closed: 1, busy: 0, failed: 0 });
|
||||
.toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
const next = { ...f.current, binding: { ...f.current.binding, runId: "sleep-retry-next" } };
|
||||
state.execute.mockImplementationOnce(async options => {
|
||||
expect(options.existingSession).toBeUndefined();
|
||||
@@ -12690,7 +12806,7 @@ describe("native warm session supervision", () => {
|
||||
environmentId: "other-environment",
|
||||
reason: "environment deleted",
|
||||
}),
|
||||
).resolves.toEqual({ closed: 0, busy: 0, failed: 0 });
|
||||
).resolves.toEqual({ closed: 0, busy: 0, failed: 0, failures: [] });
|
||||
const closeResult = shutdownKind === "controller restart"
|
||||
? closeIdleWarmNativeSessionsForRestart()
|
||||
: closeWarmNativeSessionsForEnvironment({
|
||||
@@ -12918,9 +13034,9 @@ describe("native warm session supervision", () => {
|
||||
if (checkpointFails) {
|
||||
expect(readNativeSessionIdleWork()).toBe("present");
|
||||
close.mockResolvedValue(undefined);
|
||||
await expect(closeIdleWarmNativeSessionsForRestart()).resolves.toEqual({ closed: 1, busy: 0, failed: 0 });
|
||||
await expect(closeIdleWarmNativeSessionsForRestart()).resolves.toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
} else {
|
||||
await expect(closeIdleWarmNativeSessionsForRestart()).resolves.toEqual({ closed: 0, busy: 0, failed: 0 });
|
||||
await expect(closeIdleWarmNativeSessionsForRestart()).resolves.toEqual({ closed: 0, busy: 0, failed: 0, failures: [] });
|
||||
}
|
||||
});
|
||||
|
||||
@@ -13753,7 +13869,7 @@ describe("native warm session supervision", () => {
|
||||
await writeFile(join(durableRoot, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(durableIdentity)));
|
||||
await mkdir(join(durableRoot, "runner"), { recursive: true });
|
||||
await writeFile(join(durableRoot, "runner", "runner-state.json"), JSON.stringify(durableRunnerState(durableIdentity, "ready")));
|
||||
if (path === "adoption") expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0 });
|
||||
if (path === "adoption") expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
const continuation = executePaperclipNativeSession({
|
||||
db: { ...leaseDb(second), select: () => ({ from: () => ({ where: () => ({ limit: () =>
|
||||
Promise.resolve([{ status: "succeeded", runnerProfileJson: { nativeExecutionInput: first } }]) }) }) }) } as unknown as Db,
|
||||
@@ -17837,7 +17953,7 @@ describe("runnerd provider runtime wiring", () => {
|
||||
}
|
||||
if (retirement === "environment deletion") {
|
||||
expect(await closeWarmNativeSessionsForEnvironment({ environmentId: target.environmentId, reason: "environment deleted" }))
|
||||
.toEqual({ closed: 1, busy: 0, failed: 0 });
|
||||
.toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
expect(close).toHaveBeenCalledExactlyOnceWith({ reason: "environment deleted" });
|
||||
expect(detachControllerForRestart).not.toHaveBeenCalled();
|
||||
return;
|
||||
@@ -17845,7 +17961,7 @@ describe("runnerd provider runtime wiring", () => {
|
||||
if (retirement === "detach failure") {
|
||||
detachControllerForRestart.mockRejectedValue(new Error("fixture journal drain failed"));
|
||||
for (let attempt = 0; attempt < 2; attempt++) {
|
||||
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 0, busy: 0, failed: 1 });
|
||||
expect(await closeIdleWarmNativeSessionsForRestart()).toMatchObject({ closed: 0, busy: 0, failed: 1 });
|
||||
expect(close).not.toHaveBeenCalled();
|
||||
expect(readNativeSessionIdleWork()).toBe("present");
|
||||
expect(JSON.parse(await readFile(path, "utf8")).releasedRunner).toEqual(envelope.releasedRunner);
|
||||
@@ -17853,8 +17969,8 @@ describe("runnerd provider runtime wiring", () => {
|
||||
detachControllerForRestart.mockResolvedValue(undefined);
|
||||
}
|
||||
if (retirement !== "idle timeout") {
|
||||
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0 });
|
||||
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 0, busy: 0, failed: 0 });
|
||||
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 0, busy: 0, failed: 0, failures: [] });
|
||||
} else await vi.waitFor(() => expect(detachControllerForRestart).toHaveBeenCalledOnce());
|
||||
expect(readNativeSessionIdleWork()).toBe("none");
|
||||
expect(close).not.toHaveBeenCalled();
|
||||
|
||||
Reference in new issue
Block a user