mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 04:21:24 +01:00
fix(native): close idle warm sessions in parallel across agents at shutdown (#21597)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - At shutdown, the server closes its idle warm Runner sessions before
it prepares the hot-restart snapshot
> - It closed them one at a time, and each full close (checkpoint, then
two stopped-copy collections) takes 3-5 s
> - In a zero-downtime swap test, five closes used the whole 20 s
pre-drain budget. Hot-restart preparation timed out, a checkpoint was
cut mid-way, and shutdown logged "hot-restart shutdown preparation
failed"
> - The same test logged 34 "Delivery reconciliation failed" errors, all
from directory cleanup retries that found the agent directory lock busy
> - This pull request closes idle warm sessions in parallel across
agents, one at a time within an agent, and treats a busy directory lock
as a quiet retry
> - The benefit is that shutdown preparation fits its budget with
several warm sessions, and expected lock contention no longer looks like
a failure
## Linked Issues or Issue Description
**What happened?**
`closeIdleWarmNativeSessions` awaited each warm session's full close in
turn. In swap 2 of a release-swap test, five idle sessions closed at
22:28:42, :46, :51, :56 and :59 (one "transport_close_requested"
diagnostic each). The `hot_restart_prepare` step then timed out at 19.9
s. Separately, `recoverDirectoryRelease` threw
`AgentDirectoryOperationBusyError` when another close held the agent
directory lock. The delivery coordinator logged each one at ERROR level.
I first checked if these sessions should have taken the cheap
released-session detach. They could not: `entry.released` is set only
after a successful `session.release()` through a release/resume-capable
plugin Runner task. The Runner state files from all sandboxes in the
test recorded only `run.prepare`, `session.open`, `turn.start`,
`runner.drain` and `runner.suspend` commands, and no `runner.release`.
The Daytona daemon logs show only `/process/execute` and
`/process/session` calls, and no Runner-task API. So no session in that
test was ever released, and a full close was the only correct shutdown
for them.
**Expected behavior**
Idle warm sessions of different agents close at the same time. Two
sessions of the same agent never close together, because they collide on
the agent directory advisory lock. A cleanup retry that finds that lock
busy stays pending until its claimed retry time and logs no error.
**Steps to reproduce**
1. Keep five or more idle warm sessions on remote sandboxes.
2. Send SIGTERM.
3. The closes run one after another and `hot_restart_prepare` times out.
**Paperclip version or commit**
`master` at bbed32450c.
## What Changed
- `closeIdleWarmNativeSessions` groups sessions by company and agent. Up
to four agents close at once, and each agent's sessions close one at a
time. The per-session close logic is unchanged.
- Idle-sleep retirement (`prepareIdleWarmNativeSessionsForSleep`) keeps
a concurrency of one, so it still checks drain authority before each
retirement.
- `recoverDirectoryRelease` catches `AgentDirectoryOperationBusyError`,
keeps the deferred retry time, and returns the pending entry instead of
throwing.
## Verification
- Red, then green (each failed against `master` source and passes with
the fix):
- `closes idle warm sessions of different agents in parallel and one at
a time within an agent` in
`server/src/services/native-runtime/native-session-executor.test.ts`.
Before the fix, the second agent's close was never started while the
first agent's close was pending.
- `keeps a cleanup pending without an error while another server holds
the agent directory lock` in
`server/src/__tests__/agent-directory-working-copies.test.ts`. It holds
the real Postgres advisory lock from a second connection. Before the
fix, the coordinator received an `AggregateError` with
`AgentDirectoryOperationBusyError`.
- `native-session-executor.test.ts` (1502 passed) and
`agent-directory-working-copies.test.ts` pass in full.
- `pnpm typecheck` in `server` passes.
## Risks
- Up to four warm closes now run at once, so shutdown makes more Runner
and database calls at the same time. The cap keeps this small.
- The order of entries in `failures` can change when several closes
fail.
- A busy release now waits for its deferred retry silently. A lock that
stays busy for a long time shows as a stuck pending cleanup, not as
repeated errors.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
12ac7cfd9f
commit
aaec67b891
4 files changed
+100
-12
No files matched your search
@@ -355,6 +355,31 @@ describe("persistent agent directories", () => {
|
|||||||
expect(worker.nextWakeAt()).toBeNull();
|
expect(worker.nextWakeAt()).toBeNull();
|
||||||
} finally { release(); await foreground; removes.mockRestore(); await worker.stop(); }
|
} finally { release(); await foreground; removes.mockRestore(); await worker.stop(); }
|
||||||
});
|
});
|
||||||
|
it("keeps a cleanup pending without an error while another server holds the agent directory lock", async () => {
|
||||||
|
const copy = await run();
|
||||||
|
await db.update(agentInstructionWorkingCopies).set({ location: `remote:${randomUUID()}` })
|
||||||
|
.where(eq(agentInstructionWorkingCopies.runId, copy.runId));
|
||||||
|
await stopped(copy, new Date(Date.now() - 1000));
|
||||||
|
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
|
||||||
|
const held = await db.$client.reserve();
|
||||||
|
const lockKey = `agent-directory:${companyId}:${agentId}`;
|
||||||
|
await held`select pg_advisory_lock(hashtextextended(${lockKey}, 0))`;
|
||||||
|
const errors: unknown[] = [], worker = coordinator(() => true, () => true, error => errors.push(error));
|
||||||
|
try {
|
||||||
|
const before = Date.now();
|
||||||
|
await copies.registerDirectoryCleanupWork(worker, () => true, async () => true).ready;
|
||||||
|
await expect.poll(() => idleWorkSnapshot().active).toBe(0);
|
||||||
|
expect(errors).toEqual([]);
|
||||||
|
const pending = (await copies.get(companyId, copy.runId))!;
|
||||||
|
expect(pending.receipt?.cleanupPending).not.toBe(false);
|
||||||
|
expect(pending.nextAttemptAt!.getTime()).toBeGreaterThan(before);
|
||||||
|
} finally {
|
||||||
|
await worker.stop();
|
||||||
|
await held`select pg_advisory_unlock(hashtextextended(${lockKey}, 0))`;
|
||||||
|
held.release();
|
||||||
|
vi.unstubAllEnvs();
|
||||||
|
}
|
||||||
|
});
|
||||||
it("reactivates from the real worktree setting without polling disabled policy", async () => {
|
it("reactivates from the real worktree setting without polling disabled policy", async () => {
|
||||||
const copy = await run(); await stopped(copy);
|
const copy = await run(); await stopped(copy);
|
||||||
const runtimeEnv = { PAPERCLIP_IN_WORKTREE: "true", PAPERCLIP_INSTANCE_ID: "sleep41-directory-activation" };
|
const runtimeEnv = { PAPERCLIP_IN_WORKTREE: "true", PAPERCLIP_INSTANCE_ID: "sleep41-directory-activation" };
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ import { notifyInstructionCommit, type InstructionCommitScope } from "./instruct
|
|||||||
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
|
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
|
||||||
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
||||||
import { registerScopedDeadlineWork } from "./scoped-deadline-work.js";
|
import { registerScopedDeadlineWork } from "./scoped-deadline-work.js";
|
||||||
import { agentDirectoryWorkingCopyService, isAgentDirectoryCopy } from "./agent-directory-working-copies.js";
|
import { AgentDirectoryOperationBusyError, agentDirectoryWorkingCopyService, isAgentDirectoryCopy } from "./agent-directory-working-copies.js";
|
||||||
import { createHash } from "node:crypto";
|
import { createHash } from "node:crypto";
|
||||||
import { execFile as execFileCallback } from "node:child_process";
|
import { execFile as execFileCallback } from "node:child_process";
|
||||||
import { promisify } from "node:util";
|
import { promisify } from "node:util";
|
||||||
@@ -564,7 +564,8 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen
|
|||||||
if (claimed) {
|
if (claimed) {
|
||||||
try { await directories.release(claimed); }
|
try { await directories.release(claimed); }
|
||||||
catch (error) {
|
catch (error) {
|
||||||
await deferCleanup();
|
const deferred = await deferCleanup();
|
||||||
|
if (error instanceof AgentDirectoryOperationBusyError) return directoryCleanupEntry(deferred);
|
||||||
throw error;
|
throw error;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -13262,6 +13262,50 @@ describe("native warm session supervision", () => {
|
|||||||
expect(onGoalCheckpoint).toHaveBeenCalledOnce();
|
expect(onGoalCheckpoint).toHaveBeenCalledOnce();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("closes idle warm sessions of different agents in parallel and one at a time within an agent", async () => {
|
||||||
|
await closeIdleWarmNativeSessionsForRestart();
|
||||||
|
const owners = [["agent-close-a", "first"], ["agent-close-a", "second"], ["agent-close-b", "only"]] as const;
|
||||||
|
const finishers = new Map<string, () => void>();
|
||||||
|
let released = false;
|
||||||
|
const closes = new Map<string, ReturnType<typeof vi.fn<() => Promise<void>>>>();
|
||||||
|
try {
|
||||||
|
for (const [agentId, label] of owners) {
|
||||||
|
const name = `parallel-close-${agentId}-${label}`;
|
||||||
|
const close = vi.fn(() => released ? Promise.resolve() : new Promise<void>(resolve => { finishers.set(name, resolve); }));
|
||||||
|
closes.set(name, close);
|
||||||
|
const warmExecution = { ...execution, binding: { ...execution.binding, agentId, issueId: name, runId: name, executionWorkspaceId: name },
|
||||||
|
session: { ...execution.session, normalizedSessionId: name,
|
||||||
|
lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } } } as NativeExecutionInputV1;
|
||||||
|
state.execute.mockReset().mockImplementationOnce(async options => {
|
||||||
|
await options.onSession?.({ close });
|
||||||
|
return { result: { summary: "Completed" }, terminal: { runTerminalState: "succeeded" },
|
||||||
|
turnId: name, normalizedSessionId: name, providerSessionId: name, driverKind: "test", driverVersion: "1",
|
||||||
|
nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
|
||||||
|
});
|
||||||
|
await executePaperclipNativeSession({ db: leaseDb(warmExecution), execution: warmExecution, runnerInstanceId: name });
|
||||||
|
}
|
||||||
|
const first = closes.get("parallel-close-agent-close-a-first")!;
|
||||||
|
const second = closes.get("parallel-close-agent-close-a-second")!;
|
||||||
|
const other = closes.get("parallel-close-agent-close-b-only")!;
|
||||||
|
const closing = closeIdleWarmNativeSessionsForRestart();
|
||||||
|
await vi.waitFor(() => {
|
||||||
|
expect(first).toHaveBeenCalledOnce();
|
||||||
|
expect(other).toHaveBeenCalledOnce();
|
||||||
|
});
|
||||||
|
await new Promise(resolve => setTimeout(resolve, 50));
|
||||||
|
expect(second).not.toHaveBeenCalled();
|
||||||
|
finishers.get("parallel-close-agent-close-a-first")!();
|
||||||
|
await vi.waitFor(() => expect(second).toHaveBeenCalledOnce());
|
||||||
|
finishers.get("parallel-close-agent-close-a-second")!();
|
||||||
|
finishers.get("parallel-close-agent-close-b-only")!();
|
||||||
|
expect(await closing).toEqual({ closed: 3, busy: 0, failed: 0, failures: [] });
|
||||||
|
} finally {
|
||||||
|
released = true;
|
||||||
|
for (const finish of finishers.values()) finish();
|
||||||
|
await closeIdleWarmNativeSessionsForRestart();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
it("blocks idle sleep for a released local warm session until its close checkpoint finishes", async () => {
|
it("blocks idle sleep for a released local warm session until its close checkpoint finishes", async () => {
|
||||||
await closeIdleWarmNativeSessionsForRestart();
|
await closeIdleWarmNativeSessionsForRestart();
|
||||||
const { reconcilePersistedRuntimeServicesOnStartup, readWorkspaceRuntimeIdleWork } = await import("../workspace-runtime.js");
|
const { reconcilePersistedRuntimeServicesOnStartup, readWorkspaceRuntimeIdleWork } = await import("../workspace-runtime.js");
|
||||||
|
|||||||
@@ -747,6 +747,7 @@ type WarmNativeSession = {
|
|||||||
};
|
};
|
||||||
|
|
||||||
const WARM_CLOSE_FAILURE_DETAIL_MAX = 2_000;
|
const WARM_CLOSE_FAILURE_DETAIL_MAX = 2_000;
|
||||||
|
const WARM_NATIVE_SESSION_CLOSE_CONCURRENCY = 4;
|
||||||
|
|
||||||
export function describeWarmCloseFailure(error: unknown): string {
|
export function describeWarmCloseFailure(error: unknown): string {
|
||||||
const describe = (value: unknown): string => {
|
const describe = (value: unknown): string => {
|
||||||
@@ -1141,11 +1142,8 @@ async function closeIdleWarmNativeSessions(input: {
|
|||||||
let busy = 0;
|
let busy = 0;
|
||||||
let failed = 0;
|
let failed = 0;
|
||||||
const failures: WarmNativeSessionCloseFailure[] = [];
|
const failures: WarmNativeSessionCloseFailure[] = [];
|
||||||
for (const [sessionId, entry] of [...warmNativeSessions]) {
|
let stopped = false;
|
||||||
if (input.canRetire && !input.canRetire()) break;
|
const closeEntry = async (sessionId: string, entry: WarmNativeSession) => {
|
||||||
if (input.environmentId !== undefined && entry.environmentId !== input.environmentId) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (entry.busy || entry.preparingRunId || executingRunnerdSessionScopes.has(sessionId)
|
if (entry.busy || entry.preparingRunId || executingRunnerdSessionScopes.has(sessionId)
|
||||||
|| (input.canRetire && (entry.instructionPreparationRunId || closingWarmNativeSessions.has(sessionId)))) {
|
|| (input.canRetire && (entry.instructionPreparationRunId || closingWarmNativeSessions.has(sessionId)))) {
|
||||||
// A busy turn can complete while another idle session is checkpointing.
|
// A busy turn can complete while another idle session is checkpointing.
|
||||||
@@ -1153,10 +1151,10 @@ async function closeIdleWarmNativeSessions(input: {
|
|||||||
// owner behind after the shutdown sweep has already passed it.
|
// owner behind after the shutdown sweep has already passed it.
|
||||||
if (input.closeBusyOnRelease) entry.closeOnReleaseReason = input.reason;
|
if (input.closeBusyOnRelease) entry.closeOnReleaseReason = input.reason;
|
||||||
busy += 1;
|
busy += 1;
|
||||||
continue;
|
return;
|
||||||
}
|
}
|
||||||
// The host owns retention after release; a quiet local handle needs no retirement for sleep.
|
// The host owns retention after release; a quiet local handle needs no retirement for sleep.
|
||||||
if (input.canRetire && entry.released && !entry.closeOnReleaseReason) continue;
|
if (input.canRetire && entry.released && !entry.closeOnReleaseReason) return;
|
||||||
const detach = input.detachReleased === true && entry.released === true && !entry.closeOnReleaseReason;
|
const detach = input.detachReleased === true && entry.released === true && !entry.closeOnReleaseReason;
|
||||||
if (entry.idleTimer !== null) clearTimeout(entry.idleTimer);
|
if (entry.idleTimer !== null) clearTimeout(entry.idleTimer);
|
||||||
// Keep a fenced owner until both retirement and stopped collection succeed.
|
// Keep a fenced owner until both retirement and stopped collection succeed.
|
||||||
@@ -1179,7 +1177,7 @@ async function closeIdleWarmNativeSessions(input: {
|
|||||||
if (detach) entry.closeOnReleaseReason = "released Runner local detach is unconfirmed";
|
if (detach) entry.closeOnReleaseReason = "released Runner local detach is unconfirmed";
|
||||||
if (!detach && error instanceof AgentDirectoryOperationBusyError) {
|
if (!detach && error instanceof AgentDirectoryOperationBusyError) {
|
||||||
busy += 1;
|
busy += 1;
|
||||||
continue;
|
return;
|
||||||
}
|
}
|
||||||
const destroyed = !detach && input.forceStopUnsettled && input.db
|
const destroyed = !detach && input.forceStopUnsettled && input.db
|
||||||
? await warmSandboxDestroyed(input.db, entry).catch(proofError => {
|
? await warmSandboxDestroyed(input.db, entry).catch(proofError => {
|
||||||
@@ -1193,7 +1191,7 @@ async function closeIdleWarmNativeSessions(input: {
|
|||||||
"warm native session's sandbox was already destroyed; released its owner");
|
"warm native session's sandbox was already destroyed; released its owner");
|
||||||
forgetWarmNativeSession(sessionId, entry);
|
forgetWarmNativeSession(sessionId, entry);
|
||||||
closed += 1;
|
closed += 1;
|
||||||
continue;
|
return;
|
||||||
}
|
}
|
||||||
failed += 1;
|
failed += 1;
|
||||||
const forced = forcedStopFailure === undefined ? "" : `; forced Runner stop failed: ${describeWarmCloseFailure(forcedStopFailure)}`;
|
const forced = forcedStopFailure === undefined ? "" : `; forced Runner stop failed: ${describeWarmCloseFailure(forcedStopFailure)}`;
|
||||||
@@ -1207,8 +1205,28 @@ async function closeIdleWarmNativeSessions(input: {
|
|||||||
}
|
}
|
||||||
releaseIdleSessionToolAuthority(sessionId);
|
releaseIdleSessionToolAuthority(sessionId);
|
||||||
}
|
}
|
||||||
if (input.canRetire && !input.canRetire()) break;
|
};
|
||||||
|
const agentQueues = new Map<string, [string, WarmNativeSession][]>();
|
||||||
|
for (const [sessionId, entry] of [...warmNativeSessions]) {
|
||||||
|
if (input.environmentId !== undefined && entry.environmentId !== input.environmentId) continue;
|
||||||
|
const agentKey = `${entry.companyId}:${entry.agentId}`;
|
||||||
|
const queue = agentQueues.get(agentKey);
|
||||||
|
if (queue) queue.push([sessionId, entry]); else agentQueues.set(agentKey, [[sessionId, entry]]);
|
||||||
}
|
}
|
||||||
|
const pendingAgents = [...agentQueues.values()];
|
||||||
|
const closeAgentQueues = async () => {
|
||||||
|
for (let queue = pendingAgents.shift(); queue; queue = pendingAgents.shift()) {
|
||||||
|
for (const [sessionId, entry] of queue) {
|
||||||
|
if (stopped || (input.canRetire && !input.canRetire())) {
|
||||||
|
stopped = true;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
await closeEntry(sessionId, entry);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
const concurrency = input.canRetire ? 1 : WARM_NATIVE_SESSION_CLOSE_CONCURRENCY;
|
||||||
|
await Promise.all(Array.from({ length: Math.min(concurrency, pendingAgents.length) }, closeAgentQueues));
|
||||||
if (failures.length > 0) logger.warn({ reason: input.reason, failures }, "idle warm native sessions did not close");
|
if (failures.length > 0) logger.warn({ reason: input.reason, failures }, "idle warm native sessions did not close");
|
||||||
return { closed, busy, failed, failures };
|
return { closed, busy, failed, failures };
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in new issue
Block a user