mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
fix(native-runtime): refresh a runtime asset lease once for concurrent turns (#22384)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a turn starts, the server copies each assigned skill into a
content-addressed runtime asset bundle. Retention garbage collection can
delete old bundles.
> - `withRuntimeAssetUse` keeps a bundle safe from deletion with an
hourly access lease. It records the lease under the same directory
publication lock that deletion uses. The lease is held in memory, so it
is cold after a server restart and again each hour.
> - When the lease is cold, each concurrent turn that uses the same
bundle takes the exclusive lock in sequence and verifies the full bundle
again. One large skill has 1,350 files (24 MB). On a busy server, the
turns at the end of that queue wait more than the 30 s lock budget. The
run then fails with "Timed out waiting for workspace restore lock". The
lock diagnostics show that the same process holds the lock
(`knownLocalHolder: true`).
> - This pull request lets concurrent uses of one digest share one
locked lease refresh.
> - The benefit is that a restart or an hourly lease expiry no longer
makes the preparation of parallel turns fail.
## Linked Issues or Issue Description
No public issue exists. This is the bug description.
**What happened?**
Agent runs on a hosted staging server failed during preparation with
`ERR_WORKSPACE_RESTORE_LOCK_TIMEOUT` ("Timed out waiting for workspace
restore lock at <instance>/locks/directory-merge/<sha256>.lock"). The
stack is `materializeSelectedSkills` (Promise.all) ->
`materializeSkillAsset` -> `withCanonicalDirectoryMergeLock` ->
`acquireDirectoryMergeLock` -> `waitForLock`. The lock diagnostics were
`knownLocalHolder: true` and `waitMs: 30052`, and the owner sidecar read
timed out. The server process had started 48 minutes before the failure.
Eight events occurred in two days. One agent ran many turns in parallel
with the same large skill.
**Expected behavior**
Concurrent turns that use the same runtime asset bundle start without
waiting for each other to verify the bundle again.
**Steps to reproduce**
1. Start a server with an empty in-memory lease map (a new process).
2. Start many turns at the same time. Assign the same large skill to all
of them.
3. Each turn takes the publication lock for that bundle in sequence. The
last turns time out after 30 s.
The new unit test shows the cause directly: 8 concurrent cold uses of
one digest take the lock 8 times on `master`. With this change, they
take it once.
**Paperclip version or commit**
Release `24132dac169fafc616615b4f3144e921aae8ad54`. The code is
unchanged on `master`.
**Deployment mode**
Hosted (staging).
**Agent adapter(s) involved**
Not adapter-specific (core bug).
## What Changed
- `withRuntimeAssetUse`
(`server/src/services/native-runtime/runtime-asset-retention.ts`): the
first non-mutation use of a digest with a cold lease runs the locked
refresh. It registers the refresh in a per-process map before its first
`await`. Concurrent non-mutation uses of the same digest wait for that
refresh. When the refresh records the lease, they use the asset without
the lock, as warm turns already do. When it records no lease (for
example, the refresh failed or no manifest exists), each waiter takes
the lock as before.
- Mutation uses (`captureSkills`, rebuilds) always take the lock. Their
behavior does not change.
- The locked body moved to a `useLocked` helper without changes.
- Two regression tests in `runtime-asset-retention.test.ts`: one refresh
serves 8 concurrent cold uses, and waiters fall back to the lock when
the shared refresh records no lease.
## Verification
- Red: on `master` the new test fails with `expected [ true, true, true,
true, true, …(3) ] to have a length of 1 but got 8`.
- Green: `npx vitest run
src/services/native-runtime/runtime-asset-retention.test.ts` passes 4/4
with this change.
- Related suites pass: `runtime-context.test.ts`,
`runtime-skill-preservation.test.ts`, `managed-skill-content.test.ts`,
`ephemeral-server-runtime-skill-startup.test.ts` (with the retention
suite: 103 of 104 tests pass). The single failure is
`remote-launch-assets.test.ts` "delivers the selected Codex skills…"
(`expected 256 to be 292`). It fails the same way on unmodified `master`
in this sandbox because the sandbox umask is 077.
- `node server/scripts/typecheck-layers.mjs`: all layers pass (below,
cycle, above, tests, test-tree-0..2), exit 0.
## Risks
- Low risk. The change only removes repeated lock acquisitions while one
refresh for the same digest runs in the same process. The lease
semantics and the deletion lock do not change.
- A waiter uses the asset without the lock only after the shared refresh
recorded a fresh lease. This is the same condition that warm turns use
today.
- If a refresh never settles, its waiters wait with it. Before this
change they waited on the lock and failed after 30 s. The lock
acquisition in the refresh keeps its own 30 s budget, so a refresh
cannot wait for the lock without limit.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2 files changed
+53
-4
No files matched your search
@@ -5,7 +5,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { resolvePaperclipInstanceRoot } from "../../home-paths.js";
|
||||
import { materializeAsset } from "./runtime-context.js";
|
||||
import { removeRuntimeAssetTree, runtimeAssetRetention, RUNTIME_ASSET_RETENTION_MS } from "./runtime-asset-retention.js";
|
||||
import { removeRuntimeAssetTree, runtimeAssetRetention, RUNTIME_ASSET_RETENTION_MS, withRuntimeAssetUse } from "./runtime-asset-retention.js";
|
||||
|
||||
let home: string;
|
||||
const originalHome = process.env.PAPERCLIP_HOME;
|
||||
@@ -54,4 +54,34 @@ describe("runtime asset retention", () => {
|
||||
expect((await fs.stat(asset.rootPath)).isDirectory()).toBe(true);
|
||||
expect(worker.nextRunAt()).toBeNull();
|
||||
});
|
||||
it("refreshes a cold lease once for concurrent turns instead of queueing each on the lock", async () => {
|
||||
const digest = "a".repeat(64);
|
||||
const manifests = path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets", "manifests");
|
||||
await fs.mkdir(manifests, { recursive: true });
|
||||
await fs.writeFile(path.join(manifests, `${digest}.json`), "{}");
|
||||
const modes: boolean[] = [];
|
||||
const use = () => withRuntimeAssetUse(digest, async locked => {
|
||||
modes.push(locked);
|
||||
if (locked) await new Promise(resolve => setTimeout(resolve, 50));
|
||||
return locked;
|
||||
});
|
||||
const results = await Promise.all(Array.from({ length: 8 }, use));
|
||||
expect(modes.filter(Boolean)).toHaveLength(1);
|
||||
expect(results.filter(Boolean)).toHaveLength(1);
|
||||
expect(await withRuntimeAssetUse(digest, async locked => locked, true)).toBe(true);
|
||||
});
|
||||
it("lets waiters take the lock themselves when the shared refresh records no lease", async () => {
|
||||
const digest = "b".repeat(64);
|
||||
const modes: boolean[] = [];
|
||||
let first = true;
|
||||
const use = () => withRuntimeAssetUse(digest, async locked => {
|
||||
modes.push(locked);
|
||||
if (first) { first = false; await new Promise(resolve => setTimeout(resolve, 20)); throw new Error("refresh failed"); }
|
||||
return locked;
|
||||
});
|
||||
const [leader, ...waiters] = await Promise.allSettled(Array.from({ length: 4 }, use));
|
||||
expect(leader.status).toBe("rejected");
|
||||
expect(waiters).toEqual(Array.from({ length: 3 }, () => ({ status: "fulfilled", value: true })));
|
||||
expect(modes).toEqual([true, true, true, true]);
|
||||
});
|
||||
});
|
||||
@@ -12,6 +12,7 @@ import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
|
||||
export const RUNTIME_ASSET_RETENTION_MS = 30 * 24 * 60 * 60_000;
|
||||
const TOUCH_MS = 60 * 60_000;
|
||||
const touched = new Map<string, number>();
|
||||
const refreshing = new Map<string, Promise<unknown>>();
|
||||
const observers = new Set<() => void>();
|
||||
const root = () => path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets");
|
||||
function paths(digest: string) {
|
||||
@@ -24,9 +25,27 @@ function paths(digest: string) {
|
||||
* protects the short interval between preparation and durable run pinning. */
|
||||
export async function withRuntimeAssetUse<T>(digest: string, consume: (locked: boolean) => Promise<T>, mutation = false): Promise<T> {
|
||||
const target = paths(digest);
|
||||
const last = touched.get(target.manifest);
|
||||
const fresh = last !== undefined && Date.now() - last < TOUCH_MS;
|
||||
if (fresh && !mutation) return consume(false);
|
||||
if (!mutation) {
|
||||
if (leaseFresh(target.manifest)) return consume(false);
|
||||
const inflight = refreshing.get(target.manifest);
|
||||
if (inflight) {
|
||||
await inflight.catch(() => undefined);
|
||||
if (leaseFresh(target.manifest)) return consume(false);
|
||||
} else {
|
||||
const refresh = useLocked(target, consume, false);
|
||||
refreshing.set(target.manifest, refresh);
|
||||
try { return await refresh; } finally { if (refreshing.get(target.manifest) === refresh) refreshing.delete(target.manifest); }
|
||||
}
|
||||
}
|
||||
return useLocked(target, consume, leaseFresh(target.manifest));
|
||||
}
|
||||
|
||||
function leaseFresh(manifest: string) {
|
||||
const last = touched.get(manifest);
|
||||
return last !== undefined && Date.now() - last < TOUCH_MS;
|
||||
}
|
||||
|
||||
async function useLocked<T>(target: ReturnType<typeof paths>, consume: (locked: boolean) => Promise<T>, fresh: boolean): Promise<T> {
|
||||
await ensureSkillCacheDirectory(path.dirname(target.bundle), true);
|
||||
return withDirectoryPublicationLock(target.bundle, async () => {
|
||||
const result = await consume(true);
|
||||
|
||||
Reference in new issue
Block a user