From 4d954646eb213c2ba22915df9d547afc32e09336 Mon Sep 17 00:00:00 2001 From: paperclipbugbot Date: Fri, 20 Nov 2026 03:58:32 -0800 Subject: [PATCH] fix(native-runtime): refresh a runtime asset lease once for concurrent turns (#22384) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a turn starts, the server copies each assigned skill into a content-addressed runtime asset bundle. Retention garbage collection can delete old bundles. > - `withRuntimeAssetUse` keeps a bundle safe from deletion with an hourly access lease. It records the lease under the same directory publication lock that deletion uses. The lease is held in memory, so it is cold after a server restart and again each hour. > - When the lease is cold, each concurrent turn that uses the same bundle takes the exclusive lock in sequence and verifies the full bundle again. One large skill has 1,350 files (24 MB). On a busy server, the turns at the end of that queue wait more than the 30 s lock budget. The run then fails with "Timed out waiting for workspace restore lock". The lock diagnostics show that the same process holds the lock (`knownLocalHolder: true`). > - This pull request lets concurrent uses of one digest share one locked lease refresh. > - The benefit is that a restart or an hourly lease expiry no longer makes the preparation of parallel turns fail. ## Linked Issues or Issue Description No public issue exists. This is the bug description. **What happened?** Agent runs on a hosted staging server failed during preparation with `ERR_WORKSPACE_RESTORE_LOCK_TIMEOUT` ("Timed out waiting for workspace restore lock at /locks/directory-merge/.lock"). The stack is `materializeSelectedSkills` (Promise.all) -> `materializeSkillAsset` -> `withCanonicalDirectoryMergeLock` -> `acquireDirectoryMergeLock` -> `waitForLock`. The lock diagnostics were `knownLocalHolder: true` and `waitMs: 30052`, and the owner sidecar read timed out. The server process had started 48 minutes before the failure. Eight events occurred in two days. One agent ran many turns in parallel with the same large skill. **Expected behavior** Concurrent turns that use the same runtime asset bundle start without waiting for each other to verify the bundle again. **Steps to reproduce** 1. Start a server with an empty in-memory lease map (a new process). 2. Start many turns at the same time. Assign the same large skill to all of them. 3. Each turn takes the publication lock for that bundle in sequence. The last turns time out after 30 s. The new unit test shows the cause directly: 8 concurrent cold uses of one digest take the lock 8 times on `master`. With this change, they take it once. **Paperclip version or commit** Release `24132dac169fafc616615b4f3144e921aae8ad54`. The code is unchanged on `master`. **Deployment mode** Hosted (staging). **Agent adapter(s) involved** Not adapter-specific (core bug). ## What Changed - `withRuntimeAssetUse` (`server/src/services/native-runtime/runtime-asset-retention.ts`): the first non-mutation use of a digest with a cold lease runs the locked refresh. It registers the refresh in a per-process map before its first `await`. Concurrent non-mutation uses of the same digest wait for that refresh. When the refresh records the lease, they use the asset without the lock, as warm turns already do. When it records no lease (for example, the refresh failed or no manifest exists), each waiter takes the lock as before. - Mutation uses (`captureSkills`, rebuilds) always take the lock. Their behavior does not change. - The locked body moved to a `useLocked` helper without changes. - Two regression tests in `runtime-asset-retention.test.ts`: one refresh serves 8 concurrent cold uses, and waiters fall back to the lock when the shared refresh records no lease. ## Verification - Red: on `master` the new test fails with `expected [ true, true, true, true, true, …(3) ] to have a length of 1 but got 8`. - Green: `npx vitest run src/services/native-runtime/runtime-asset-retention.test.ts` passes 4/4 with this change. - Related suites pass: `runtime-context.test.ts`, `runtime-skill-preservation.test.ts`, `managed-skill-content.test.ts`, `ephemeral-server-runtime-skill-startup.test.ts` (with the retention suite: 103 of 104 tests pass). The single failure is `remote-launch-assets.test.ts` "delivers the selected Codex skills…" (`expected 256 to be 292`). It fails the same way on unmodified `master` in this sandbox because the sandbox umask is 077. - `node server/scripts/typecheck-layers.mjs`: all layers pass (below, cycle, above, tests, test-tree-0..2), exit 0. ## Risks - Low risk. The change only removes repeated lock acquisitions while one refresh for the same digest runs in the same process. The lease semantics and the deletion lock do not change. - A waiter uses the asset without the lock only after the shared refresh recorded a fresh lease. This is the same condition that warm turns use today. - If a refresh never settles, its waiters wait with it. Before this change they waited on the lock and failed after 30 s. The lock acquisition in the refresh keeps its own 30 s budget, so a refresh cannot wait for the lock without limit. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 --- .../runtime-asset-retention.test.ts | 32 ++++++++++++++++++- .../native-runtime/runtime-asset-retention.ts | 25 +++++++++++++-- 2 files changed, 53 insertions(+), 4 deletions(-) diff --git a/server/src/services/native-runtime/runtime-asset-retention.test.ts b/server/src/services/native-runtime/runtime-asset-retention.test.ts index 2d8444c0184..313e924b488 100644 --- a/server/src/services/native-runtime/runtime-asset-retention.test.ts +++ b/server/src/services/native-runtime/runtime-asset-retention.test.ts @@ -5,7 +5,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import type { Db } from "@paperclipai/db"; import { resolvePaperclipInstanceRoot } from "../../home-paths.js"; import { materializeAsset } from "./runtime-context.js"; -import { removeRuntimeAssetTree, runtimeAssetRetention, RUNTIME_ASSET_RETENTION_MS } from "./runtime-asset-retention.js"; +import { removeRuntimeAssetTree, runtimeAssetRetention, RUNTIME_ASSET_RETENTION_MS, withRuntimeAssetUse } from "./runtime-asset-retention.js"; let home: string; const originalHome = process.env.PAPERCLIP_HOME; @@ -54,4 +54,34 @@ describe("runtime asset retention", () => { expect((await fs.stat(asset.rootPath)).isDirectory()).toBe(true); expect(worker.nextRunAt()).toBeNull(); }); + it("refreshes a cold lease once for concurrent turns instead of queueing each on the lock", async () => { + const digest = "a".repeat(64); + const manifests = path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets", "manifests"); + await fs.mkdir(manifests, { recursive: true }); + await fs.writeFile(path.join(manifests, `${digest}.json`), "{}"); + const modes: boolean[] = []; + const use = () => withRuntimeAssetUse(digest, async locked => { + modes.push(locked); + if (locked) await new Promise(resolve => setTimeout(resolve, 50)); + return locked; + }); + const results = await Promise.all(Array.from({ length: 8 }, use)); + expect(modes.filter(Boolean)).toHaveLength(1); + expect(results.filter(Boolean)).toHaveLength(1); + expect(await withRuntimeAssetUse(digest, async locked => locked, true)).toBe(true); + }); + it("lets waiters take the lock themselves when the shared refresh records no lease", async () => { + const digest = "b".repeat(64); + const modes: boolean[] = []; + let first = true; + const use = () => withRuntimeAssetUse(digest, async locked => { + modes.push(locked); + if (first) { first = false; await new Promise(resolve => setTimeout(resolve, 20)); throw new Error("refresh failed"); } + return locked; + }); + const [leader, ...waiters] = await Promise.allSettled(Array.from({ length: 4 }, use)); + expect(leader.status).toBe("rejected"); + expect(waiters).toEqual(Array.from({ length: 3 }, () => ({ status: "fulfilled", value: true }))); + expect(modes).toEqual([true, true, true, true]); + }); }); diff --git a/server/src/services/native-runtime/runtime-asset-retention.ts b/server/src/services/native-runtime/runtime-asset-retention.ts index 2e74881888b..a511a486205 100644 --- a/server/src/services/native-runtime/runtime-asset-retention.ts +++ b/server/src/services/native-runtime/runtime-asset-retention.ts @@ -12,6 +12,7 @@ import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; export const RUNTIME_ASSET_RETENTION_MS = 30 * 24 * 60 * 60_000; const TOUCH_MS = 60 * 60_000; const touched = new Map(); +const refreshing = new Map>(); const observers = new Set<() => void>(); const root = () => path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets"); function paths(digest: string) { @@ -24,9 +25,27 @@ function paths(digest: string) { * protects the short interval between preparation and durable run pinning. */ export async function withRuntimeAssetUse(digest: string, consume: (locked: boolean) => Promise, mutation = false): Promise { const target = paths(digest); - const last = touched.get(target.manifest); - const fresh = last !== undefined && Date.now() - last < TOUCH_MS; - if (fresh && !mutation) return consume(false); + if (!mutation) { + if (leaseFresh(target.manifest)) return consume(false); + const inflight = refreshing.get(target.manifest); + if (inflight) { + await inflight.catch(() => undefined); + if (leaseFresh(target.manifest)) return consume(false); + } else { + const refresh = useLocked(target, consume, false); + refreshing.set(target.manifest, refresh); + try { return await refresh; } finally { if (refreshing.get(target.manifest) === refresh) refreshing.delete(target.manifest); } + } + } + return useLocked(target, consume, leaseFresh(target.manifest)); +} + +function leaseFresh(manifest: string) { + const last = touched.get(manifest); + return last !== undefined && Date.now() - last < TOUCH_MS; +} + +async function useLocked(target: ReturnType, consume: (locked: boolean) => Promise, fresh: boolean): Promise { await ensureSkillCacheDirectory(path.dirname(target.bundle), true); return withDirectoryPublicationLock(target.bundle, async () => { const result = await consume(true);