3 Commits

Include renames

Author	SHA1	Message	Date
3 people	3cd3af5d75	Trace native preparation and OpenCode execution phases (#17918) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Each native run prepares its workspace and skills, then starts its provider. > - A long preparation span does not show whether inventory refresh, a cache lock, repository reads, or file checks caused the delay. > - One hosted run spent 81 seconds in this phase. The same deployed service and pinned files took 2.55 seconds in a separate diagnostic process. > - Another hosted turn spent 69 seconds before its first output. A matched public-workspace local model replay took 7.64 seconds. > - We need skill preparation and provider-internal measurements before choosing another fix. > - This pull request adds skill spans, a real-service benchmark, and opt-in filtered provider timing. > - The benefit is a faster and more precise diagnosis loop. ## Linked Issues or Issue Description Refs #17620. That PR owns the batched read optimization. This PR adds diagnostics and does not change its read policy. Refs #17907. That PR protects long cache lock transactions from the database idle timeout. **What happened?** Native skill configuration can take much longer than matched standalone resolution. Provider execution also has long gaps with no internal timings. Existing spans cannot separate inventory refresh, cache acquisition, repository reads, context loading, Git snapshots, or model streaming. **Expected behavior** An operator-configured trace shows these phases under the existing run context. Without an OTLP endpoint, the new spans stay off. A local benchmark can reproduce the service path with real PostgreSQL, files, an agent actor, and a responsible user. **Steps to reproduce** 1. Run `pnpm --filter @paperclipai/server exec tsx scripts/benchmark-native-skill-preparation.ts --actor=agent --staging-shape=true`. 2. Inspect cold and warm phase timings, SQL counts, and file verification results. 3. Set the existing operator OTLP configuration and add `--export-otlp=true` to inspect the same hierarchy in the collector. ## What Changed - Add bounded skill inventory, refresh wait, cache acquisition, row, version, repository, file check, and consumer spans. - Keep the existing endpoint gate, attribute allowlist, cache locks, authorization, file validation, and durable pin boundary. - Add a standalone benchmark with a full repository fixture, immutable snapshot input, calibrated local transport delay, and optional OTLP export. - Verify every delivered file and executable bit outside the timed section. Create its fixture agent through the lifecycle API. - Add an opt-in authenticated loopback sink for OpenCode. Discard raw attributes, logs and content. Retain only closed phase names and timestamps. - Carry one bounded diagnostic batch through the real proxy, Rust normalization, durable journal and server trace projection. Group provider phases under their execution parent. - Freeze the Date clock in an existing delegated Slack rate-limit test. This fixes a CI failure when consecutive requests cross a rate-window boundary. It does not change rate-limit behavior. - Document phase boundaries and the difference between SDK startup and confirmed ingestion. ## Verification - Skill service, runtime selection, concurrency, cache and timing suites: 122 tests passed with real PostgreSQL and migrations. Enabled-span integration tests also passed. They verify parent relationships and a real advisory-lock timeout. - A captured 208-file repository produced 209 verified runtime files. The benchmark uses real agent and responsible-user authorization. The lifecycle fixture correction passed with the staging-shaped inventory. - Affected OpenCode driver, collector and transport tests: 72 passed. The tests cover authentication, content removal, size/count bounds, disabled behavior and the rebuilt native Runner boundary. One Rust projection test and 18 server trace tests also passed. A held run-log sink proves that a full 128-span diagnostic batch adds no duplicate database writes before terminal consumption. - The two CI-failing delegated Slack rate-limit cases passed against real PostgreSQL with only the Date clock frozen. - Runner TypeScript and server TypeScript checks passed. Rust formatting and module-boundary checks passed. - Real Honeycomb readback confirmed the 14-span skill waterfall and a 40-span provider waterfall. Provider phases have the correct execution parent. The provider export uses synthetic model replies and is instrumentation proof only. - Earlier full local checks failed on missing package-local executables, then on the default Node heap and a missing generated service worker. The dependency links were repaired. The server check passed with a larger heap. The full local test attempt could not start embedded PostgreSQL on this host and was stopped after retaining its failures; real database coverage uses an isolated database on an existing local server. These attempts do not count as a full local pass. - CI on the prior revision passed typecheck, build, Runner and all E2E shards. It failed the benchmark lifecycle boundary and one existing rate-window test. Those fixes passed CI. A later Runner assertion exposed the Copilot source identity pin on the shared transport file. The diagnostic flag now uses the existing configured-environment projection, preserving that file byte-for-byte. All 66 selected installation and native transport tests passed. New-head CI and review remain required. - The under-15-second hosted response target remains unmet. This PR adds diagnostics and does not claim a hosted latency fix. ## Risks - Skill spans and Honeycomb export retain the operator endpoint gate. OpenCode phase collection requires a separate explicit diagnostic flag. Without an endpoint, its closed measurements stay in the instance run log. - Raw OpenCode OTLP can contain prompts and authorization attributes. The new sink rejects unauthenticated requests and discards all attributes before any measurement reaches PRP or the operator collector. Request, event and per-phase limits bound memory and journal growth. - Collection does not wait for a provider flush. Late or oversized exports can be absent; these measurements are not a complete profiling guarantee. - The benchmark's inventory contents and transport delay are explicit fixtures. They do not reproduce server contention or prove hosted response time. - The one durable diagnostic batch remains the local evidence. Its child span export does not add per-phase run-log rows. - No database migration, authorization policy, model selection or inference transport changes. The Rust Runner and OpenCode proxy must be shipped together for the new opt-in diagnostic. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, shell execution, browser tools, and Honeycomb queries. The session does not expose a verifiable backend deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Paperclip <paperclip@paperclip.ing>	2026-10-27 14:53:32 -05:00
DottaandPaperclip	06c5bc471d	fix(skills): unblock retention for legacy run selections (#16852) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent runs can select saved skill versions. > - The server keeps those files in a cache that must protect active and unresolved runs. > - Older runs can lack the newer pin map even when their saved execution input records the full selection. > - Finished interrupted runs can also keep a company blocked after all execution and cleanup have settled. > - This pull request uses existing selection evidence and recognizes settled interruptions so safe cache expiration can resume. ## Linked Issues or Issue Description Related: #16433. **What happened?** Skill cache retention reports `unknown_run_pins` for a whole company when it finds a finished interrupted run without pins, or an older native run whose complete selection is stored only in its execution input. Old cache directories cannot expire. **Expected behavior** Release pins from fully settled interruptions. Preserve exact versions from valid saved native selections. Continue to block cleanup when selection evidence is missing, invalid, or belongs to another run. **Steps to reproduce** 1. Materialize an old, restorable skill version and let its idle grace period expire. 2. Add a finished interrupted run with settled ownership and cleanup, or an unresolved native run with a valid saved execution input but no modern pin map. 3. Run the retention service. Before this change, it skips the company with `unknown_run_pins`. ## What Changed - Treat `interrupted` as terminal only when the existing finish, controller, finalization, and environment cleanup checks allow it. - For native runs with no modern pin-map field, parse the saved execution input with the existing recovery parser. Validate its digests and company, run, agent, and issue binding. - Keep captured version IDs protected. Unversioned skills use separate captured asset bundles. Do not replace a malformed modern map or write any recovery records. - Add database regressions and document the fallback. Remove an unused status list that no longer represented the actual eligibility query. ## Verification - Passed 41 focused tests: `skill-version-cache.test.ts`, `skill-version-retention.test.ts`, and `skill-version-retention-config.test.ts`. - The tests perform real PostgreSQL reads, directory eviction, and restoration from saved bytes with upstream unavailable. They cover leases, unfinished cleanup, invalid digests, invalid versions, and mismatched identities. - Passed `node scripts/check-module-boundaries.mjs` and `git diff --check`. - A read-only evaluation of the candidate selector against existing legacy data cleared the unknown-pin blocker while retaining exact version pins. - Passed `pnpm -r --workspace-concurrency=2 typecheck` and `pnpm build`. The full remote CI suite passed, including all server, workspace, Runner, and end-to-end shards. The full local test command remains in progress; its focused cache suites passed as recorded above. - Greptile reviewed this head at 5/5 with no actionable comments or unresolved threads. - Initial local database attempts did not run the cases: sandbox access prevented a listener, then an install with scripts disabled lacked PostgreSQL library symlinks. After running the package's symlink setup, the binary started and all focused tests passed. ## Risks The main risk is releasing a cache still in use. The existing ownership and cleanup checks remain in force. The native fallback applies only when the modern field is absent and the saved input passes the same parser used for recovery. Invalid or missing evidence still blocks the company. Current versions and explicit agent pins stay protected. Saved version rows and runtime asset bundles are not deleted. There is no migration. The selector reads execution inputs only for legacy native runs that need the fallback. Normal runs still load only their pin maps and binding columns. A representative read-only selector check completed in 98 ms; this is a sample, not a latency guarantee. ## Model Used OpenAI Codex, GPT-6 family, with tool-assisted coding and verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-19 10:10:49 -05:00
3 people	86c965ce7a	fix(skills): expire unused version caches and restore saved contents (#16433) ... Expire idle skill-version caches with durable runtime pins, bounded discovery, conservative retention, and database-backed restoration. Co-Authored-By: Dotta <cryppadotta@users.noreply.github.com> Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-15 12:20:57 -05:00