5 Commits

Include renames

Author	SHA1	Message	Date
3 people	b06ea111a2	fix(server): schedule skill cache retention from changes and saved deadlines (#17923) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/83/changes/paperclip/83 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server caches saved skill versions on disk so runs can restore their exact instructions. > - The old retention owner scans company rows every hour, even after all caches are protected or removed. > - Cache publication, version changes, and pin release already have local or transaction-owned boundaries. > - Grace windows and terminal controller leases already have saved timestamps. > - This pull request uses those changes and deadlines to drive the existing bounded prune service. > - Idle retention stops SQL and directory reads while pending cleanup remains recoverable. ## Linked Issues or Issue Description Stack slice 64. Parent: #17876 (`codex/sleep-63-agent-timer-deadlines`). Related: #15625 establishes transaction-owned work signals. #17907 independently keeps an active cache-lock transaction alive; this PR preserves that lock and does not duplicate that repair. **What happened?** At parent `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`, skill-version retention has an hourly company scan, a startup timeout, and separate continuation timers. Empty or permanently protected caches still cause recurring SQL. **Expected behavior** Reconstruct once, inspect changed companies, and schedule saved access/staging grace or terminal controller deadlines. Stop recurring work when no timed or uncertain obligation remains. Keep current/newest/agent/run protection, restorable-byte validation, cache locking, final pin revalidation, atomic rename, and trash recovery. **Steps to reproduce** Start an enabled server with no cached versions, or only current/pinned versions. After the initial scan, advance an hour. The old interval selects companies again. The new worker has no deadline and performs no retention SQL or directory scan in that state. ## What Changed - Signal the missing-local-source reconciliation deletion in its transaction. Limit fresh scans of a successfully observed company to one per thirty seconds. Keep one-second page continuations and actual saved deadlines independent. Empty or uncertain passes do not delay first publication. - Restart a company scan if drain interrupts inspection or post-rename trash cleanup. Restore bounded candidate/byte logging with dry-run mode. Three controlled cases failed before repair and the complete 34-case retention suite passes afterward. - Register retention through the delivery coordinator and shared scheduler. Remove the hourly, startup, and continuation timers from `index.ts`. Track startup work without delaying readiness. Stop delivery work before closing retained iterators. - Keep one active company cursor and bounded 50-entry passes. Preserve observed deadlines across pages and bound company discovery to 51 rows per page. Retain concrete company failures and incomplete pins for retry. - Report saved grace and terminal controller deadlines from the existing cache owner. PostgreSQL remains the controller-lease authority. Current/newest/agent pins do not arm unrelated lease checks. - Wake on cache publication, rejected materialization, and access markers. Signal version creation/deletion and agent configuration/hire changes in their owning transactions. Extend existing instruction, finalization/environment, and retained-session fanouts with company hints. - Route reused provider-task prior-owner retirement through `writeEnvironmentCleanup`, so that supported pin-release write also settles a notification. - Add an optional scope callback to `registerDeadlineWork`. Existing consumers omit it and keep their behavior. The feature owns persistence predicates; startup only registers it. Add no table or migration. Full producer audit, shared-contract explanation, lifecycle evidence, and limitations: [`doc/plans/2026-10-27-skill-version-retention-sleep-audit.md`](doc/plans/2026-10-27-skill-version-retention-sleep-audit.md). ## Verification - Reconciled all existing sleep changes with actual master `7e27bc5007c8f9cb8cdce06c24f9813e27f7120f` through normal merges. All 24 retention-owned and 95 unrelated upstream paths are preserved; package, UI, E2E and map trees equal master. - Combined-source qualification: **1,009 TypeScript cases across 42 complete files**, plus **28 Rust cases** in two targets. One intentional Linux-inotify-only adapter case is skipped on this macOS host. Coverage includes actual retention/cache/producer/runtime behavior, startup, shared deadline consumers, changed OpenCode/observability and native execution boundaries. - After the subsequent task-index/task-read upstream merges, **85 cases across six complete files** passed on final source `d9277b1858a2230bdb726974c90a0c9bc5c5ebe6`: real retention/cache/work, task reads/query behavior and concurrent migration repair. Counts overlap earlier coverage. Fresh PostgreSQL 17 schemas apply all 413 journal entries through index 414. - Database/adapter/Runner builds, production server and Runner typechecking, module boundaries, target migration order and generated TypeScript/Rust release-profile consistency pass. Selected source hashes and temporary fixture imports restore exactly. - Reproduced and repaired Greptile's own-company due-cleanup P2 with identical old-worker-failing/new-worker-passing assertions. The 30-second ordinary scan floor remains; already-due cleanup receives its owed scan and disarms afterward. Earlier 799-case and fresh-blank-plugin acceptance remain in the audit. - **Inherited validation exception:** the separate Copilot source-identity check fails after upstream OpenCode reserved-environment changes; all 72 identity inputs equal current master. No provider release attestation is regenerated or new live qualification claimed here. Required exact-head CI/security and Greptile 5/5 still must pass. - Historical results, repair details and limitations remain in [the retention audit](doc/plans/2026-10-27-skill-version-retention-sleep-audit.md). No full local repository-suite, live provider, production-load, external-alarm, measured suspend/wake or fleet-savings claim. - CI on the prior published head found four connection-review assertions racing the concealed conversation after reload. Its fresh Greptile/Ponytail reviews were both 5/5, but the failing browser gate was not waived or retried unchanged. The fixture now foregrounds the task tab and waits for visible history before retaining all existing saved-result assertions. The full file passes four journeys locally; the controlled-restart case is intentionally skipped by the default configuration. Original local journeys also passed unchanged, so no deterministic local red reproduction is claimed. See the audit for CI failure artifacts and scope. Runtime and prior sleep-test sources remain unchanged. ## Risks - Supported live writers are the existing service/API owners in one process. Independent SQL/FS writers need an ingress contract. Startup reconstruction covers restarts. - Notifications cause a bounded inspection; they do not authorize deletion. Durable pin predicates and the cache lock remain authoritative. - A database clock behind application time can require another one-second check of the concrete expired controller lease. Unknown pins, lock contention, and failed inspection retain retries until resolved. - The in-process scheduler cannot wake stopped compute. This PR preserves Cloud sleep-safety gates and external-alarm coverage. It does not prove deployed suspension or fleet savings. Cache reclamation can wait until a supported wake when compute is stopped. - Native-harness backup retention and the other remaining duties are separate stack slices. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, normal Git merges, composition proofs, a controlled red/green regression and complete PostgreSQL/filesystem integration suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 10:14:12 -05:00
DottaandPaperclip	4970d5ed3d	fix(skills): unblock agent attachment and preserve Latest selections (#16899) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators attach company skills to agents from the skill library. > - Attachment first reads the agent's skill snapshot to preserve other assignments. > - That read materialized and validated repository files for the entire library, even in read-only mode. > - The response also treated resolved runtime versions as stored pins. Saving could then fail with beta pins disabled. > - This pull request makes version-cache inspection lightweight and returns the stored assignment choices. > - The benefit is a responsive attachment flow that preserves Latest and explicit pins. ## Linked Issues or Issue Description **What happened?** Adding a repository skill to an agent could stall before the save request. The prerequisite snapshot read projected or checked every file in every installed repository skill. Once it returned, an existing Latest assignment could become a version pin in the save payload. The server then rejected that save when beta skill pins were disabled. **Expected behavior** Reading assignments must not prepare all repository snapshots for execution. Adding a skill must preserve the other assignments and their Latest or pinned choices. **Steps to reproduce** 1. Install repository-backed skills with stored immutable versions and cold disk caches. 2. Assign one repository skill to a Claude agent with Latest selected. Leave beta skill pins disabled. 3. Open another skill, select Add to agent, choose the same agent, and save. 4. Observe the expensive prerequisite read and the beta-pin rejection after that read completes. **Paperclip version or commit** Reproduced from master at `ac9459bae3329395f10bc81c638a374eef152baa`. **Deployment mode** Observed in hosted staging. Reproduced with a disposable local instance, embedded PostgreSQL, and the Claude local adapter on macOS arm64 with Node 26.4.0. Related: https://github.com/paperclipai/paperclip/pull/9995 adds a separate read-only assignment endpoint for peer-agent access. This fix keeps the existing configuration-gated snapshot and its permissions. ## What Changed - Respect read-only listing for current and pinned version caches. Check directory and entrypoint safety without projecting repository contents, reading all support files, or updating cache access records. - Keep complete snapshot restoration and byte validation at run preparation. Missing caches are restored from stored data. Corrupted files are repaired there. - Return stored assignment version choices in both list and sync responses. Keep runtime version details in the snapshot entries. - Add database-backed regressions for cold and warm reads, pinned versions, execution repair, and unsafe entrypoints. Add HTTP regressions for save round trips and explicit pins. - Update developer documentation and verification recipes. ## Verification - Red: both latest and pinned read-only cache regressions failed on the original code. The original code created caches during listing. - Red: the HTTP save-roundtrip regression returned 400 with the beta-pin error. The explicit-pin regression also failed. - Green: the updated API and cache suites passed all 145 tests. The three related company-skill service suites also passed. - Real browser: attached a repository skill to a paused test agent with another repository skill already set to Latest and beta pins disabled. The success toast appeared. The skill's Agents tab retained the assignment after reload. API reads confirmed both assignments remained unpinned and no repository version caches were created. - Local synthetic fixture: six skills share a 606-file repository snapshot. A cold HTTP skill-list request completed in 241 ms. This is local evidence, not a staging latency measurement. - Real service comparison on that fixture: the baseline cold list wrote 3,648 version/cache files. A warm baseline list read 3,642 version files and wrote six access records (457 ms). The fixed warm list read and wrote zero version/cache files (61 ms). Both services returned the same 16 library entries. - `pnpm -r typecheck` passed. The final API edit also passed the server typecheck. - `pnpm build` passed. - `node scripts/check-maps.mjs` and `git diff --check` passed. - The broad `pnpm test:run` invocation reported six failures in the untouched `tool-gateway-service.test.ts` suite. An isolated rerun passed all 59 tests. The broad run was stopped after recording those failures, so it is not a green full-local result. The PR CI matrix separately passed those suites. Greptile completed on the current commit with 5/5, zero comments, and no open threads. - One CI shard was interrupted by runner shutdown, with no failed assertion in its log. A single failed-job retry passed on the same commit. The aggregate CI gate passed. All current-head PR checks are complete with no failing or pending checks. - This PR does not claim that the fix has been deployed to staging. ## Risks - Settings reads check cache locations and entrypoints. They defer full support-file validation to execution. Regression tests verify that execution still repairs corrupt bytes and restores missing versions. - The response now distinguishes stored pins from resolved runtime versions. Explicit historical pins remain intact. Company, configuration, connector, and beta-pin authorization gates keep their existing behavior. - No schema migration or dependency change. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, code execution, and browser tools. The exact runtime build ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-19 16:33:02 -05:00
DottaandPaperclip	06c5bc471d	fix(skills): unblock retention for legacy run selections (#16852) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent runs can select saved skill versions. > - The server keeps those files in a cache that must protect active and unresolved runs. > - Older runs can lack the newer pin map even when their saved execution input records the full selection. > - Finished interrupted runs can also keep a company blocked after all execution and cleanup have settled. > - This pull request uses existing selection evidence and recognizes settled interruptions so safe cache expiration can resume. ## Linked Issues or Issue Description Related: #16433. **What happened?** Skill cache retention reports `unknown_run_pins` for a whole company when it finds a finished interrupted run without pins, or an older native run whose complete selection is stored only in its execution input. Old cache directories cannot expire. **Expected behavior** Release pins from fully settled interruptions. Preserve exact versions from valid saved native selections. Continue to block cleanup when selection evidence is missing, invalid, or belongs to another run. **Steps to reproduce** 1. Materialize an old, restorable skill version and let its idle grace period expire. 2. Add a finished interrupted run with settled ownership and cleanup, or an unresolved native run with a valid saved execution input but no modern pin map. 3. Run the retention service. Before this change, it skips the company with `unknown_run_pins`. ## What Changed - Treat `interrupted` as terminal only when the existing finish, controller, finalization, and environment cleanup checks allow it. - For native runs with no modern pin-map field, parse the saved execution input with the existing recovery parser. Validate its digests and company, run, agent, and issue binding. - Keep captured version IDs protected. Unversioned skills use separate captured asset bundles. Do not replace a malformed modern map or write any recovery records. - Add database regressions and document the fallback. Remove an unused status list that no longer represented the actual eligibility query. ## Verification - Passed 41 focused tests: `skill-version-cache.test.ts`, `skill-version-retention.test.ts`, and `skill-version-retention-config.test.ts`. - The tests perform real PostgreSQL reads, directory eviction, and restoration from saved bytes with upstream unavailable. They cover leases, unfinished cleanup, invalid digests, invalid versions, and mismatched identities. - Passed `node scripts/check-module-boundaries.mjs` and `git diff --check`. - A read-only evaluation of the candidate selector against existing legacy data cleared the unknown-pin blocker while retaining exact version pins. - Passed `pnpm -r --workspace-concurrency=2 typecheck` and `pnpm build`. The full remote CI suite passed, including all server, workspace, Runner, and end-to-end shards. The full local test command remains in progress; its focused cache suites passed as recorded above. - Greptile reviewed this head at 5/5 with no actionable comments or unresolved threads. - Initial local database attempts did not run the cases: sandbox access prevented a listener, then an install with scripts disabled lacked PostgreSQL library symlinks. After running the package's symlink setup, the binary started and all focused tests passed. ## Risks The main risk is releasing a cache still in use. The existing ownership and cleanup checks remain in force. The native fallback applies only when the modern field is absent and the saved input passes the same parser used for recovery. Invalid or missing evidence still blocks the company. Current versions and explicit agent pins stay protected. Saved version rows and runtime asset bundles are not deleted. There is no migration. The selector reads execution inputs only for legacy native runs that need the fallback. Normal runs still load only their pin maps and binding columns. A representative read-only selector check completed in 98 ms; this is a sample, not a latency guarantee. ## Model Used OpenAI Codex, GPT-6 family, with tool-assisted coding and verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-19 10:10:49 -05:00
DottaandPaperclip	06689c1b34	perf(skills): coalesce inventory before run cache locking (#16576) ... Share each active company skill-inventory refresh before taking the version-cache lock. Read actor visibility under the lock and keep snapshot publication protected through durable run pinning. Preserve empty-selection behavior and the list API. Validation: 176 affected cases, full repository typecheck/build, fresh full CI/E2E, security and exact-head Apex 5/5 pass. The local six-reader latency experiment improves from 50.4s to 18.3s; live response latency is a separate acceptance check. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-16 10:39:02 -05:00
3 people	86c965ce7a	fix(skills): expire unused version caches and restore saved contents (#16433) ... Expire idle skill-version caches with durable runtime pins, bounded discovery, conservative retention, and database-backed restoration. Co-Authored-By: Dotta <cryppadotta@users.noreply.github.com> Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-15 12:20:57 -05:00