mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
0eacad0d0cb22a68e86d6bce20d52230ccc0b5c3
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
51c352ec2e |
fix(skills): release a runtime skill-cache lock whose holder provably exited (#19849)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a run starts, the server builds a read-only cache of each company skill under a cross-process file lock > - The lock records only a pid and a hostname, so a crash leaves a lock that nothing can release > - If the holder pid is dead on the same host, the next run fails and asks for manual cleanup. If the host name differs, the next run waits 60 s and then fails > - This is on the turn path, so one crash during a skill build breaks later runs that use that skill > - This pull request records the shared controller process identity in the lock and releases the lock at once when `controllerProcessExited` proves the holder is gone > - The benefit is that a crash during a skill build no longer blocks later runs, and the release never relies on lock age or another host ## Linked Issues or Issue Description **Bug description** The Phase 0 ownership inventory (finding 4) lists the runtime skill-cache lock (`runtime-skill-cache.ts`) as a record that nothing releases automatically. When a publisher crashes while it holds `<skillId>.lock`, every later preparation of that skill fails. **Steps to reproduce** 1. Start a skill cache build, then kill the server process while it holds the lock. 2. Restart and start a run that uses the same skill. 3. Preparation fails with "Runtime skill cache publisher exited; remove its stale publication lock before retrying". For a holder whose host name differs, it fails after 60 s with "publisher is busy". **Expected behavior** A lock whose holder has provably exited is released, and the build continues without waiting. A holder that cannot be proven gone is still waited for. Refs #19846 ## What Changed - `controller-process-identity.ts`: add `currentControllerProcessStamp()`. It returns this process's identity with no I/O, so the turn path gets no new wait. - `runtime-skill-cache.ts`: the lock owner file now also stores `controller` (boot id, pid, PID-space hash, start time if already known). `pid` and `host` stay so older readers still work. - New `runtimeSkillCacheLockHolderExited`: true only when `controllerProcessExited` proves the holder gone. Another host or PID namespace, an unknown space, a legacy lock without `controller`, a live pid, and this process all return false. - On proof, waiters race for one exclusive claim named by the stale holder's unique token (`<lock>.released-<token>`). Only the winner unlinks the lock, and only while it still has the stale inode. The claim is kept, so a late waiter does nothing. The waiter then retries the link right away instead of after the 20 ms poll. - A live holder is re-proved at most once a second. `controllerProcessExited` skips the start-time probe (`ps` on macOS) when no start time was recorded. - The busy error now names the lock path. The same-host "publisher exited" hard failure is removed. ## Verification - `npx vitest run src/__tests__/runtime-skill-cache.test.ts src/services/controller-process-identity.test.ts`: 28 passed, rebased on master after #19848. - New case "takes over a release claim whose claimer exited before removing the stale lock" covers the second Greptile finding. - New case "never removes a newer lock for a waiter that read a stale one late" covers the Greptile P1. - New case "releases a publication lock at once when its holder provably exited". On the old code it fails with "Runtime skill cache publisher exited; remove its stale publication lock before retrying". On the new code it builds in well under 5 s and leaves no lock. - New case "never treats age, another host, or a legacy holder as proof": a different host hash, a legacy `{pid, host}` lock, a live pid, this process, and malformed content all stay held. - Server `tsc --noEmit` shows no errors in the changed files. ## Risks - Previous-release story: a lock written by the previous release has no `controller` field. It is unprovable. The waiter waits 60 s and fails as before, and the error now names the path to remove. A new lock still carries `pid` and `host`, so a previous-release reader behaves as before. - Lock release race: while a stale lock exists, only the single claim winner can remove it, and the inode check cannot match a newer lock because the stale holder's owner file stays and pins that inode. A claim whose claimer crashed is itself released by the same proof (depth bounded at 2). - Each released lock leaves one small claim file and the dead holder's owner file next to it. - No new I/O before `turn.start`. The stamp is synchronous. The proof runs only when the lock is contended. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
799e4d556f |
fix: make accounting durable and synchronize cost reporting (#14997)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7ed122911b |
Add end-to-end session goals to Paperclip Runner
Add capability-aware slash-goal controls, durable provider goal state, PRP v2 negotiation, autonomous goal execution, and safe local session recovery. Integrate with current master, preserve provider session identity, and verify the browser goal/chat/replacement/clear workflow and unsupported-agent rejection. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c723bb4dfc |
fix(skills): reuse validated runtime revisions during preparation (#13042)
## Thinking Path > - Paperclip manages AI agents and prepares their runtime inputs before each turn. > - Shared company skills are part of those inputs for native and legacy adapters. > - Runtime materialization refreshed the full inventory again for every declared file. > - Remote skill directories were also downloaded and rebuilt on every turn. > - Measured preparation took 42–73 seconds while runner execution took 7–9 seconds. > - This change reads the inventory once and reuses validated installed revisions. > - Agents retain their selected skills while repeated preparation avoids upstream work. ## Linked Issues or Issue Description **What happened?** One 114-skill preparation performed 407 inventory refreshes, 48 directory rebuilds, and 388 GitHub file fetches. Reusing existing local copies took 151 ms. **Expected behavior** Each listing refreshes inventory once. Unchanged installed remote revisions reuse complete, validated local copies. Local edits remain visible. Explicit updates select new revisions. **Steps to reproduce** 1. Import GitHub skills with supporting files. 2. Run an agent turn, then run another with the same installed revisions. 3. Observe repeated inventory scans, downloads, and runtime directory replacement before execution. Related prior attempts: #2330 and #9268 (still open; #9268 last updated July 9). Those use a marker compared with `updatedAt`. This patch follows the required content validation, immutable revision, company isolation, atomic publication, and read-only semantics, and removes refresh-per-file multiplication. ## What Changed - Split public file reading from reading an already loaded skill. Runtime listing refreshes inventory once. - Add a company-scoped revision cache with file manifests outside the delivered skill directory. Fingerprints omit cosmetic metadata. - Validate exact file inventory, sizes, and hashes before warm reuse. Reject traversal and symlinks. Stage complete builds and serialize atomic publication across processes. - Preserve local/catalog direct sources, stored Markdown fallback, explicit version snapshots, and legacy mutable-ref compatibility. Report missing supporting files and keep older valid revisions readable. - Clean both runtime layouts on rename/removal and record `skills.prepare` under preparation timing. - Add service/cache regressions and an isolated 114-skill benchmark, including a new-process warm run. ## Verification - Final targeted skill-service/cache/trace validation: 86 tests pass (61 embedded-PostgreSQL service tests, 19 cache tests, 6 trace tests). Database tests executed rather than skipped. Focused skill routes, adapter selection, and native runtime context also pass. - `pnpm -r typecheck` and `pnpm build` pass locally at `22caa1fe4`. - The full `pnpm test:run` matrix passes on supported Linux CI at the final head: [CI run](https://github.com/paperclipai/paperclip/actions/runs/34236097762). Local full-suite execution encountered PostgreSQL startup contention, a random allocated-port boundary, and a socket hang-up; every affected suite passed on an isolated rerun. The interrupted local serialized run is not claimed as a complete local pass. - Repeatable benchmark: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-skill-preparation.ts`. Mixed 114-skill inventory with 429 remote files on Linux: cold 286 ms, warm median 96 ms / maximum 153 ms including a new process. Every warm sample performs one refresh, zero upstream fetches/rebuilds, and reports no missing entries; content assertions pass. - Controlled deployment against the previously deployed revision completed with zero lost runs. Real inventory: 114 skills, 670 declared files; 402 cached files match the prior installed copies byte-for-byte. Ten post-deployment warm preparations: median 129 ms / maximum 208 ms; new-process warm 194 ms, zero downloads/rebuilds/missing entries. - Five sequential real browser questions persisted in 10.6–20.7 s (median 12.2 s), versus 50–83 s before. Skill preparation median 240 ms, with one 2.37 s outlier. Total preparation median 3.337 s / maximum 8.728 s **does not fully meet** the <3 s / <5 s target. The excluded historical-run redaction query takes about 1.36 s per scan at two preparation call sites; wider application latency coincided with the outlier, without a cache rebuild. These residuals are reported rather than discarded. - Disposable skill reimport verified through actual selected-skill runs: the next run read the changed code. Fixture removed and agent configuration verified unchanged. - Greptile 5/5, zero unresolved review threads, all final-head CI checks green. ## Risks - Cold preparation still requires upstream availability for supporting files. An unavailable revision is reported missing and never falls back to an older revision. - Valid older revisions and quarantined invalid entries consume additive disk space until skill cleanup. An abruptly killed publisher can leave a lock that requires operator cleanup after confirming its PID is dead. - Warm validation reads all cached file bytes. Very large inventories still have proportional local I/O cost. - No HTTP API, schema, agent configuration, or first-party Telemetry changes. OpenTelemetry retains its operator endpoint gate. ## Model Used OpenAI GPT-6 in Codex, with reasoning, repository inspection, code editing, and test execution. The exact serving snapshot and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted and isolated reruns; full Linux CI matrix passes, local full-run caveats above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |