mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(native): close idle warm sessions in parallel across agents at shutdown (#21597)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - At shutdown, the server closes its idle warm Runner sessions before
it prepares the hot-restart snapshot
> - It closed them one at a time, and each full close (checkpoint, then
two stopped-copy collections) takes 3-5 s
> - In a zero-downtime swap test, five closes used the whole 20 s
pre-drain budget. Hot-restart preparation timed out, a checkpoint was
cut mid-way, and shutdown logged "hot-restart shutdown preparation
failed"
> - The same test logged 34 "Delivery reconciliation failed" errors, all
from directory cleanup retries that found the agent directory lock busy
> - This pull request closes idle warm sessions in parallel across
agents, one at a time within an agent, and treats a busy directory lock
as a quiet retry
> - The benefit is that shutdown preparation fits its budget with
several warm sessions, and expected lock contention no longer looks like
a failure
## Linked Issues or Issue Description
**What happened?**
`closeIdleWarmNativeSessions` awaited each warm session's full close in
turn. In swap 2 of a release-swap test, five idle sessions closed at
22:28:42, :46, :51, :56 and :59 (one "transport_close_requested"
diagnostic each). The `hot_restart_prepare` step then timed out at 19.9
s. Separately, `recoverDirectoryRelease` threw
`AgentDirectoryOperationBusyError` when another close held the agent
directory lock. The delivery coordinator logged each one at ERROR level.
I first checked if these sessions should have taken the cheap
released-session detach. They could not: `entry.released` is set only
after a successful `session.release()` through a release/resume-capable
plugin Runner task. The Runner state files from all sandboxes in the
test recorded only `run.prepare`, `session.open`, `turn.start`,
`runner.drain` and `runner.suspend` commands, and no `runner.release`.
The Daytona daemon logs show only `/process/execute` and
`/process/session` calls, and no Runner-task API. So no session in that
test was ever released, and a full close was the only correct shutdown
for them.
**Expected behavior**
Idle warm sessions of different agents close at the same time. Two
sessions of the same agent never close together, because they collide on
the agent directory advisory lock. A cleanup retry that finds that lock
busy stays pending until its claimed retry time and logs no error.
**Steps to reproduce**
1. Keep five or more idle warm sessions on remote sandboxes.
2. Send SIGTERM.
3. The closes run one after another and `hot_restart_prepare` times out.
**Paperclip version or commit**
`master` at bbed32450c.
## What Changed
- `closeIdleWarmNativeSessions` groups sessions by company and agent. Up
to four agents close at once, and each agent's sessions close one at a
time. The per-session close logic is unchanged.
- Idle-sleep retirement (`prepareIdleWarmNativeSessionsForSleep`) keeps
a concurrency of one, so it still checks drain authority before each
retirement.
- `recoverDirectoryRelease` catches `AgentDirectoryOperationBusyError`,
keeps the deferred retry time, and returns the pending entry instead of
throwing.
## Verification
- Red, then green (each failed against `master` source and passes with
the fix):
- `closes idle warm sessions of different agents in parallel and one at
a time within an agent` in
`server/src/services/native-runtime/native-session-executor.test.ts`.
Before the fix, the second agent's close was never started while the
first agent's close was pending.
- `keeps a cleanup pending without an error while another server holds
the agent directory lock` in
`server/src/__tests__/agent-directory-working-copies.test.ts`. It holds
the real Postgres advisory lock from a second connection. Before the
fix, the coordinator received an `AggregateError` with
`AgentDirectoryOperationBusyError`.
- `native-session-executor.test.ts` (1502 passed) and
`agent-directory-working-copies.test.ts` pass in full.
- `pnpm typecheck` in `server` passes.
## Risks
- Up to four warm closes now run at once, so shutdown makes more Runner
and database calls at the same time. The cap keeps this small.
- The order of entries in `failures` can change when several closes
fail.
- A busy release now waits for its deferred retry silently. A lock that
stays busy for a long time shows as a stuck pending cleanup, not as
repeated errors.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
12ac7cfd9f
commit
aaec67b891
4 files changed
+100
-12
No files matched your search
@@ -355,6 +355,31 @@ describe("persistent agent directories", () => {
|
||||
expect(worker.nextWakeAt()).toBeNull();
|
||||
} finally { release(); await foreground; removes.mockRestore(); await worker.stop(); }
|
||||
});
|
||||
it("keeps a cleanup pending without an error while another server holds the agent directory lock", async () => {
|
||||
const copy = await run();
|
||||
await db.update(agentInstructionWorkingCopies).set({ location: `remote:${randomUUID()}` })
|
||||
.where(eq(agentInstructionWorkingCopies.runId, copy.runId));
|
||||
await stopped(copy, new Date(Date.now() - 1000));
|
||||
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
|
||||
const held = await db.$client.reserve();
|
||||
const lockKey = `agent-directory:${companyId}:${agentId}`;
|
||||
await held`select pg_advisory_lock(hashtextextended(${lockKey}, 0))`;
|
||||
const errors: unknown[] = [], worker = coordinator(() => true, () => true, error => errors.push(error));
|
||||
try {
|
||||
const before = Date.now();
|
||||
await copies.registerDirectoryCleanupWork(worker, () => true, async () => true).ready;
|
||||
await expect.poll(() => idleWorkSnapshot().active).toBe(0);
|
||||
expect(errors).toEqual([]);
|
||||
const pending = (await copies.get(companyId, copy.runId))!;
|
||||
expect(pending.receipt?.cleanupPending).not.toBe(false);
|
||||
expect(pending.nextAttemptAt!.getTime()).toBeGreaterThan(before);
|
||||
} finally {
|
||||
await worker.stop();
|
||||
await held`select pg_advisory_unlock(hashtextextended(${lockKey}, 0))`;
|
||||
held.release();
|
||||
vi.unstubAllEnvs();
|
||||
}
|
||||
});
|
||||
it("reactivates from the real worktree setting without polling disabled policy", async () => {
|
||||
const copy = await run(); await stopped(copy);
|
||||
const runtimeEnv = { PAPERCLIP_IN_WORKTREE: "true", PAPERCLIP_INSTANCE_ID: "sleep41-directory-activation" };
|
||||
|
||||
@@ -5,7 +5,7 @@ import { notifyInstructionCommit, type InstructionCommitScope } from "./instruct
|
||||
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
|
||||
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
||||
import { registerScopedDeadlineWork } from "./scoped-deadline-work.js";
|
||||
import { agentDirectoryWorkingCopyService, isAgentDirectoryCopy } from "./agent-directory-working-copies.js";
|
||||
import { AgentDirectoryOperationBusyError, agentDirectoryWorkingCopyService, isAgentDirectoryCopy } from "./agent-directory-working-copies.js";
|
||||
import { createHash } from "node:crypto";
|
||||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
@@ -564,7 +564,8 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen
|
||||
if (claimed) {
|
||||
try { await directories.release(claimed); }
|
||||
catch (error) {
|
||||
await deferCleanup();
|
||||
const deferred = await deferCleanup();
|
||||
if (error instanceof AgentDirectoryOperationBusyError) return directoryCleanupEntry(deferred);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -13262,6 +13262,50 @@ describe("native warm session supervision", () => {
|
||||
expect(onGoalCheckpoint).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("closes idle warm sessions of different agents in parallel and one at a time within an agent", async () => {
|
||||
await closeIdleWarmNativeSessionsForRestart();
|
||||
const owners = [["agent-close-a", "first"], ["agent-close-a", "second"], ["agent-close-b", "only"]] as const;
|
||||
const finishers = new Map<string, () => void>();
|
||||
let released = false;
|
||||
const closes = new Map<string, ReturnType<typeof vi.fn<() => Promise<void>>>>();
|
||||
try {
|
||||
for (const [agentId, label] of owners) {
|
||||
const name = `parallel-close-${agentId}-${label}`;
|
||||
const close = vi.fn(() => released ? Promise.resolve() : new Promise<void>(resolve => { finishers.set(name, resolve); }));
|
||||
closes.set(name, close);
|
||||
const warmExecution = { ...execution, binding: { ...execution.binding, agentId, issueId: name, runId: name, executionWorkspaceId: name },
|
||||
session: { ...execution.session, normalizedSessionId: name,
|
||||
lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } } } as NativeExecutionInputV1;
|
||||
state.execute.mockReset().mockImplementationOnce(async options => {
|
||||
await options.onSession?.({ close });
|
||||
return { result: { summary: "Completed" }, terminal: { runTerminalState: "succeeded" },
|
||||
turnId: name, normalizedSessionId: name, providerSessionId: name, driverKind: "test", driverVersion: "1",
|
||||
nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
|
||||
});
|
||||
await executePaperclipNativeSession({ db: leaseDb(warmExecution), execution: warmExecution, runnerInstanceId: name });
|
||||
}
|
||||
const first = closes.get("parallel-close-agent-close-a-first")!;
|
||||
const second = closes.get("parallel-close-agent-close-a-second")!;
|
||||
const other = closes.get("parallel-close-agent-close-b-only")!;
|
||||
const closing = closeIdleWarmNativeSessionsForRestart();
|
||||
await vi.waitFor(() => {
|
||||
expect(first).toHaveBeenCalledOnce();
|
||||
expect(other).toHaveBeenCalledOnce();
|
||||
});
|
||||
await new Promise(resolve => setTimeout(resolve, 50));
|
||||
expect(second).not.toHaveBeenCalled();
|
||||
finishers.get("parallel-close-agent-close-a-first")!();
|
||||
await vi.waitFor(() => expect(second).toHaveBeenCalledOnce());
|
||||
finishers.get("parallel-close-agent-close-a-second")!();
|
||||
finishers.get("parallel-close-agent-close-b-only")!();
|
||||
expect(await closing).toEqual({ closed: 3, busy: 0, failed: 0, failures: [] });
|
||||
} finally {
|
||||
released = true;
|
||||
for (const finish of finishers.values()) finish();
|
||||
await closeIdleWarmNativeSessionsForRestart();
|
||||
}
|
||||
});
|
||||
|
||||
it("blocks idle sleep for a released local warm session until its close checkpoint finishes", async () => {
|
||||
await closeIdleWarmNativeSessionsForRestart();
|
||||
const { reconcilePersistedRuntimeServicesOnStartup, readWorkspaceRuntimeIdleWork } = await import("../workspace-runtime.js");
|
||||
|
||||
@@ -747,6 +747,7 @@ type WarmNativeSession = {
|
||||
};
|
||||
|
||||
const WARM_CLOSE_FAILURE_DETAIL_MAX = 2_000;
|
||||
const WARM_NATIVE_SESSION_CLOSE_CONCURRENCY = 4;
|
||||
|
||||
export function describeWarmCloseFailure(error: unknown): string {
|
||||
const describe = (value: unknown): string => {
|
||||
@@ -1141,11 +1142,8 @@ async function closeIdleWarmNativeSessions(input: {
|
||||
let busy = 0;
|
||||
let failed = 0;
|
||||
const failures: WarmNativeSessionCloseFailure[] = [];
|
||||
for (const [sessionId, entry] of [...warmNativeSessions]) {
|
||||
if (input.canRetire && !input.canRetire()) break;
|
||||
if (input.environmentId !== undefined && entry.environmentId !== input.environmentId) {
|
||||
continue;
|
||||
}
|
||||
let stopped = false;
|
||||
const closeEntry = async (sessionId: string, entry: WarmNativeSession) => {
|
||||
if (entry.busy || entry.preparingRunId || executingRunnerdSessionScopes.has(sessionId)
|
||||
|| (input.canRetire && (entry.instructionPreparationRunId || closingWarmNativeSessions.has(sessionId)))) {
|
||||
// A busy turn can complete while another idle session is checkpointing.
|
||||
@@ -1153,10 +1151,10 @@ async function closeIdleWarmNativeSessions(input: {
|
||||
// owner behind after the shutdown sweep has already passed it.
|
||||
if (input.closeBusyOnRelease) entry.closeOnReleaseReason = input.reason;
|
||||
busy += 1;
|
||||
continue;
|
||||
return;
|
||||
}
|
||||
// The host owns retention after release; a quiet local handle needs no retirement for sleep.
|
||||
if (input.canRetire && entry.released && !entry.closeOnReleaseReason) continue;
|
||||
if (input.canRetire && entry.released && !entry.closeOnReleaseReason) return;
|
||||
const detach = input.detachReleased === true && entry.released === true && !entry.closeOnReleaseReason;
|
||||
if (entry.idleTimer !== null) clearTimeout(entry.idleTimer);
|
||||
// Keep a fenced owner until both retirement and stopped collection succeed.
|
||||
@@ -1179,7 +1177,7 @@ async function closeIdleWarmNativeSessions(input: {
|
||||
if (detach) entry.closeOnReleaseReason = "released Runner local detach is unconfirmed";
|
||||
if (!detach && error instanceof AgentDirectoryOperationBusyError) {
|
||||
busy += 1;
|
||||
continue;
|
||||
return;
|
||||
}
|
||||
const destroyed = !detach && input.forceStopUnsettled && input.db
|
||||
? await warmSandboxDestroyed(input.db, entry).catch(proofError => {
|
||||
@@ -1193,7 +1191,7 @@ async function closeIdleWarmNativeSessions(input: {
|
||||
"warm native session's sandbox was already destroyed; released its owner");
|
||||
forgetWarmNativeSession(sessionId, entry);
|
||||
closed += 1;
|
||||
continue;
|
||||
return;
|
||||
}
|
||||
failed += 1;
|
||||
const forced = forcedStopFailure === undefined ? "" : `; forced Runner stop failed: ${describeWarmCloseFailure(forcedStopFailure)}`;
|
||||
@@ -1207,8 +1205,28 @@ async function closeIdleWarmNativeSessions(input: {
|
||||
}
|
||||
releaseIdleSessionToolAuthority(sessionId);
|
||||
}
|
||||
if (input.canRetire && !input.canRetire()) break;
|
||||
};
|
||||
const agentQueues = new Map<string, [string, WarmNativeSession][]>();
|
||||
for (const [sessionId, entry] of [...warmNativeSessions]) {
|
||||
if (input.environmentId !== undefined && entry.environmentId !== input.environmentId) continue;
|
||||
const agentKey = `${entry.companyId}:${entry.agentId}`;
|
||||
const queue = agentQueues.get(agentKey);
|
||||
if (queue) queue.push([sessionId, entry]); else agentQueues.set(agentKey, [[sessionId, entry]]);
|
||||
}
|
||||
const pendingAgents = [...agentQueues.values()];
|
||||
const closeAgentQueues = async () => {
|
||||
for (let queue = pendingAgents.shift(); queue; queue = pendingAgents.shift()) {
|
||||
for (const [sessionId, entry] of queue) {
|
||||
if (stopped || (input.canRetire && !input.canRetire())) {
|
||||
stopped = true;
|
||||
return;
|
||||
}
|
||||
await closeEntry(sessionId, entry);
|
||||
}
|
||||
}
|
||||
};
|
||||
const concurrency = input.canRetire ? 1 : WARM_NATIVE_SESSION_CLOSE_CONCURRENCY;
|
||||
await Promise.all(Array.from({ length: Math.min(concurrency, pendingAgents.length) }, closeAgentQueues));
|
||||
if (failures.length > 0) logger.warn({ reason: input.reason, failures }, "idle warm native sessions did not close");
|
||||
return { closed, busy, failed, failures };
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user