mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 05:16:50 +01:00
DottaandPaperclip
f1911a4c53
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/205/changes/paperclip/205 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents receive selected company skills before each turn. > - Imported skills can need files from the whole source repository. > - The current path stores those files in Postgres and reads large JSON payloads during preparation. > - This pull request stores verified packages in the configured file or object store and keeps preparation queries small. > - It also converts old rows automatically and restores missing files from the exact saved commit. > - The result reduces database traffic while preserving skill versions and repository dependencies. ## Linked Issues or Issue Description **What existing behavior does this improve?** Company skill import, version storage, and preparation before agent turns. **Current behavior** Repository snapshot bodies live in JSONB. Skill preparation can read those bodies again even when the files already exist on disk. Historical packages and native assets also need a bounded cleanup path. **Proposed behavior** Store package archives and manifests through the existing storage service. Keep version identities, receipts, hashes, and recovery metadata in Postgres. Convert legacy rows after the stored package passes read-back verification. A missing package can use legacy bytes or its exact Git commit under current repository authorization. No separate data-migration command is required. **Reason and benefit** The warm path sends no repository bodies through SQL and reads no object bytes. Cleanup runs through the existing deadline coordinator. Current versions, active runs, explicit pins, and authored content retain their protection. **Breaking changes** The normal application schema upgrade is required. Single-version and skill-file responses keep their existing shape. Version lists, current-version summaries, and test-run version summaries now contain file names and kinds without file bodies. Diff and restore fetch selected versions through the existing single-version endpoint. Older server builds must not write after conversion unless they support the new package format. The storage override and rollback limits are documented in `doc/skill-package-storage.md`. **Additional context** Related work: #15731 preserves repository dependencies; #17923 changes version-cache scheduling; #18080 changes delivery to remote Codex homes. This change owns package persistence, acquisition, and expiry. It does not replace those delivery or scheduler changes. ## What Changed - Add company-scoped archive receipts, exact-commit recovery recipes, migration deadlines, and a durable object-deletion queue. - Add bounded archive upload, read-back verification, extraction, concurrent request coalescing, and verified local caches. - Convert old snapshots and versions automatically. Keep legacy bytes when verification fails. - Recover missing files through current source authorization and the original commit. Validate repository identity, file hashes, modes, and the aggregate hash. - Expire unused recoverable Git packages and native assets after 30 days. Preserve active pins and the durable copy of authored or unproven content. - Keep warm preparation queries metadata-only. Bound file digest caches and avoid repeated body hashing when file identities are unchanged. - Extend the existing storage contract with cancellation. Register package work with the existing coordinator. Optional conversion retries retain the original bytes and may wait for ingress without blocking sleep. - Add regression tests, operating guidance, a deep trace, benchmark receipts, and an optional maintenance command. ## Verification ### Final master integration and review repair — October 29, 2026 Current head: `a6065424249e763dfeac37161f1b52a9f25bd9b7`. Commit `f8bbdd3dbd` resolves the conflicts with master `76a796e964`; the final two commits repair the Apex deletion findings below. - Keep both package-maintenance and version-retention notifications in their write transactions. - Keep the shared run-pin predicate and master's controller-lease deadline metadata. Package expiry reads the updated protected-version result. - Preserve every master migration entry. Generate migration **0415 (`0415_zippy_zemo.sql`)** after master's 0414. Its skill SQL is byte-for-byte unchanged from the previously tested idempotent migration. Retain only the five newest schema snapshots. Migration numbering and safety checks pass. - Repair failed file/folder deletion: preserve a complete temporary subtree until transaction commit, restore on upload or SQL failure, and restore callback failures before releasing the mutation lock. Both PostgreSQL regressions reproduced missing files before the fix and now pass. They cover binary data, executable permissions, empty folders, unlisted files, unchanged database/version rows, successful retry after upload-claim expiry, and historical reads. - Reconcile lost commit replies under the same advisory and row locks. Check the saved version and source identity before restoring. Keep committed deletions intact and preserve backups when the database outcome is unknown or a newer edit exists. Four real-PostgreSQL outcome tests pass; the prior code failed three of them. - Full workspace typecheck and production build pass on the final commit. All **204** focused PostgreSQL package/cache/retention/authoring and filesystem-boundary tests pass, with none skipped. - Apex review **18 passes 5/5**, with zero new comments on the exact final head. All **34** review conversations are resolved. - [Final-head CI](https://github.com/paperclipai/paperclip/actions/runs/40914116508) completes successfully on attempt 2. The head has **55 successful checks**, two neutral reports, two skipped checks, and a successful Snyk status. No checks are pending or failing. All eight browser shards pass. - Runner Vitest 2/2 timed out once in the unchanged ACPX controller-replacement fixture. The same job passed on the preceding commit. One unchanged CI retry passes all five ACPX recovery cases, **2,253** Runner package cases and **126** native server integration cases; their standard one and five skips remain. The initial timeout's root cause is not established. Local diagnosis also hit the fixture's eight-second session.open timeout with an authenticated Runner; no local Runner pass is claimed. - Both full local `pnpm test:run` attempts were stopped when Apex identified deletion defects and source repair became necessary; neither is claimed as a passing full local run. The final **204** focused tests, full workspace typecheck, and production build pass. Full-suite acceptance uses the final-head CI jobs. The superseded merge-head CI included runners terminated by shutdown signals; each source repair started a new full run. - Auto-merge is enabled for squash merge. The final head is conflict-free and all checks pass. GitHub's active master ruleset still requires a code-owner approval for the dependency files. The existing review requests are with `devinfoley`, `nickyleach`, and `forgottendev`; no policy bypass was used. - The earlier hosted stress and full native receipts below remain tied to `a2e597d800`; they are not relabeled as a deployment of this final integration. The earlier local PostgreSQL startup failure remains recorded below; PostgreSQL ran successfully for this integration verification. ### Full native skill execution — October 29, 2026 The full native-agent acceptance gap is closed on candidate `a2e597d800eb989d565780e7aa947d5368676444`. Two qualifying native turns used OpenAI API / `gpt-5.4-mini` in real Daytona sandboxes, on the same disposable hosted TEST tenant as the stress test. A follow-up on the same task consumed a newly saved skill version. The task text asked for the assigned skill workflow. It did not contain the proof nonce or expected receipt. The assigned authored skill contained a Python verifier and a random package-local nonce. The agent read the actual installed skill files and executed that verifier against the imported, automatically converted 1,360-file repository package. It checked the repository file count and reference/binary bytes, and directly executed the repository's executable probe. Independent verification matched the emitted JSON to the nonce and expected byte hashes. The agent persisted the same receipt as its task response; native finalization committed, workspace finalization succeeded, and task state became Done. | Qualifying turn | End-to-end run time | Recorded skills.prepare | Repository files | Binary hex | Shell receipt SHA-256 | |---|---:|---:|---:|---|---| | Initial task | 47.881 s | 1822 ms | 1360 | `00ff07800a` | `a1deb30dc095f2306c7aa9cc5ec396da12c68961cf5b809f4d7c20f7fc68493b` | | Corrected follow-up, new nonce/version | 37.974 s | 377 ms | 1360 | `00ff07800a` | `dbcdc33fa1376b4edc136feae1cdf770560a73292467e93619efdda20e4282a7` | Both successful shell receipts reported `probe: STRESS18082` and `assetSha256: e8ea7a2f15f7bb3b332a46adbef848465884c8996595aa234748befd544d1f3a`. Their proof nonces, selected version IDs and receipt hashes differed, and both ran on the same task. These are observed timings, not latency SLOs; they include real provider work. The service stress-test timings remain a separate measurement boundary. Earlier failures remain in the evidence: - The original five-minute Daytona allocation timeout never reached model execution. After normal owned-stack restore, its durable cleanup record was `expired` / `success`. The cause of that original allocation timeout remains unknown; a later success establishes recovery only. - Initial continuation setup used unsupported generic agent-update fields and returned HTTP 409 before dispatch. Setup was corrected to use the supported skill-assignment and versioned instruction commands. - The intervening completed native turn read repository bytes and ran the probe, but omitted the verifier JSON under the older test agent's overly-specific instruction file. Its strict receipt validator failed; it is not counted as a qualifying turn. Correcting the test agent's instruction file and sending an explicit task-scoped wake produced the qualifying follow-up above. - The pinned application's immutable rate card omits `gpt-5.4-mini`, so completed usage triggered the normal block-on-unpriced budget policy. Supported append-only accounting corrections estimated the three completed runs using the [official OpenAI model rate card](https://developers.openai.com/api/docs/models/gpt-5.4-mini): $0.75 input, $0.075 cached input, $4.50 output per million tokens, assuming standard processing. Estimates totaled 12.482655 cents. Original usage and missing-price records are preserved; the $1 hard stop and unpriced-usage guard stayed active. - Model shell attempts included a missing `xxd` command and incorrect guessed file paths. Those failed calls and subsequent successful actual verifier calls remain visible in complete event streams. No zero-tool-error claim is made. No product source changed during this continuation. The previously measured contention, cold-start and memory limits remain. The later merge resolution is covered by the separate integration checks above; these hosted receipts qualify their recorded candidate. ### Hosted stress test — October 29, 2026 **Hosted acceptance:** full native-agent execution and a changed-skill follow-up pass on the recorded hosted candidate. The final integration section above records the subsequent conflict resolution and its separate checks. Tested baseline `929e840595c8933ca485ac7a2e99413e1e11ca64` and this exact candidate sequentially on one disposable TEST tenant, with external Neon PostgreSQL and actual S3. The fixture contained **1,360 files / 22.4 MB / ten skills sharing one repository**. Measurements cover deployed skill-preparation and native-context services in an adjacent diagnostic process, not a complete agent turn. The container had **24 vCPU / 24 GB**. | Warm workload | Median before → after | p95 before → after | DB returned value bytes/preparation before → after | |---|---:|---:|---:| | One skill, serial | 1,060 → 125 ms | 1,303 → 155 ms | 22,610,389 → 1,771 | | One skill, four concurrent | 4,415 → 297 ms | 5,143 → 328 ms | 22,610,389 → 1,771 | | Ten skills, serial | 5,723 → 535 ms | 6,057 → 600 ms | 224,739,030 → 10,671 | | Ten skills, four concurrent | 24,410 → 1,986 ms | 24,833 → 2,221 ms | 224,739,030 → 10,671 | - Candidate warm preparation performed **zero S3 GETs**. Ordinary-pool queries fell from 16 to 9 for one skill and 61 to 9 for ten, excluding dedicated lock connections. DB numbers measure returned values before Drizzle decoding, not wire bytes. Baseline/candidate sample counts were 30/100, 32/100, 5/50 and 8/48 respectively; the small baseline ten-skill samples are comparative evidence, not SLO estimates. - **Application-UID soak:** 800 ten-skill preparations over 10.3 minutes, four workers paced by 2.5 seconds; **zero failures**, median **559 ms**, p95 **775 ms**, p99 **910 ms**, zero warm S3 reads. All 726 accompanying health/company-list requests returned HTTP 200. Their end-to-end p95 was 569/502 ms, including network latency. The probe overlaps most of the soak. Soak DB-response capture was limited to its first 60 seconds. - **Important remaining costs:** an unpaced 16-way ten-skill burst had p95 **8.32 seconds** because company-level preparation serializes. Cold preparation took approximately **1.1–2.5 seconds** for one skill and **7.4–10.0 seconds** for ten. Ten-skill diagnostic RSS remained about **1.17 GiB** (baseline 1.18 GiB), and the soak peaked at **1.35 GiB**. A concurrent whole-container sample was 3.45 GB. This does not establish suitability for a small-memory host or eliminate per-entrypoint native asset duplication. - **Upgrade without a manual data migration passed:** legacy snapshot JSON shrank from 22,465,691 bytes to `[]`, with a stored package and zero pending conversions. - **Recovery passed:** sixteen concurrent cold requests coalesced into two S3 GETs; full bytes/binary/executable modes matched; same-size corruption repaired; an injected storage failure preserved warm access and failed cold access safely. A disabled public-Git source denied recovery, and after the actual five-minute backoff its exact SHA reconstructed all 229 files. Cross-company descriptor access was denied. Private-Git credential revocation was not tested. - **Retention passed:** reclaimed 27,754,496 allocated cache bytes, deleted an unreferenced package row and both objects, preserved aged authored history, and expired/reacquired reconstructible Git content. A version pinned by the failed native run was correctly retained. Only owned fixture timestamps were aged. - **Browser UI passed:** authored skill creation, saved edit surviving reload, version history, and restoration of earlier content. - **Initial native attempt (superseded by the qualifying turns above):** one real run failed at Daytona allocation after five minutes (`setup_failed`), before any model execution. Provider cleanup also reported failure. That attempt does not count as a passing native turn. Its receipt and all evidence remain preserved; its cleanup later recorded success after normal restoration. - Initial A/B commands ran as SSH root. Diagnostic-owned lock/cache permissions caused one setup HTTP 500 and the first UID-1000 soak to fail; exact fixture paths were repaired. The final soak, retention and recovery ran as application UID 1000. A fresh UID-1000 one-skill batch also passed (30 samples, median 116 ms, p95 321 ms, 1,771 DB value bytes/preparation). Earlier harness-export, SSH, npm provenance, backoff and UI-wait failures remain recorded alongside their successful retries. - Private evidence includes raw timings, assertions, HTTP probes, screenshots and failure/cleanup receipts. Initial failed-campaign cleanup suspended and retained the owned TEST tenant and released the shared QA account lease at **15:22:41 UTC**. The tenant was subsequently resumed under its existing ownership for the native execution qualification above. Final cleanup completed at **15:53:09 UTC**: the account lease is released and the tenant is suspended/retained under the original October 30, 12:24:52 UTC retention deadline. Its separate stack lease remains held; no active or pending-cleanup environment lease remained. The original retention window is deliberately preserved even after successful continuation. - Earlier hosted-tested head: `a2e597d800eb989d565780e7aa947d5368676444`, integrated with master `929e840595`. The earlier merge conflicts were resolved at this head. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/40884087629) passed; all checks are complete with 55 successes, two neutral reports, two skips, and no failures. [Apex review 15](https://github.com/paperclipai/paperclip/pull/18082#issuecomment-6480257258) passed 5/5 with zero findings on this exact head. All review conversations are resolved. - Ten real-filesystem boundary cases and server typecheck pass locally for the security fix. Coverage includes cache roots, descendants, individual files, symlink aliases, recursive ancestors, a future concurrent cache fill, and allowed sibling directories. [Current-head private-project integration tests](https://github.com/paperclipai/paperclip/actions/runs/40884087629/job/123483306405) passed all 29 cases, including authorized and unauthorized imports of a cached private package. Full typecheck, build, server suites, all eight browser shards, and Runner checks passed in current-head CI. Local database tests could not run because PostgreSQL could not initialize; this limitation is recorded below. - The earlier generated migration preserved the newest five snapshots. The final integration renumbers its unchanged SQL to 0415. Migration safety/numbering checks pass. Its SQL is byte-for-byte identical to the previously verified idempotent skill migration; repeated application on this latest schema was blocked by the same local database startup failure. - The broader verification before the final migration-only rebase passed 1,025 package, lifecycle, remote-launch, native-session, and asset-retention tests, 52 focused UI tests, full workspace typecheck/build, token gates, and map checks. The linked complete CI run passed before the latest focused fixes. - A disposable MinIO instance passed multipart upload, verified read-back, and cold restoration of an 18 MB archive. - Twenty warm samples with a synthetic 1,349-file, 22.8 MB repository retained five metadata queries and zero object reads. Median preparation was 256 ms for one selected skill and 691 ms for ten, with 5 ms emulated network delay each way. These are local measurements, not production latency guarantees. - No production instance was changed. Earlier verification failures and their disposition are retained below. <details> <summary>Earlier failures, review findings, fixes, and rerun evidence</summary> - The first CI pass found a stale inventory-span expectation and a missing repository-read span. Both are fixed; all three observability tests pass. One browser test copied page text into its YAML parser after reload; its first unchanged-head retry passed, but the race recurred after rebasing. The test now waits for the renamed URL and completed reload before copying the editor, retaining its original assertions. The earlier full local suite did not complete cleanly: it reported setup timeouts and assertions, while isolated relevant reruns passed. The complete current-head CI run closes that broad verification gap. No production instance was changed. - The first Apex review scored 2/5 and identified six issues. The fixes protect authored history even when it shares an in-flight Git upload, recheck upload expiry during orphan deletion, bound failed-upload cleanup, retain per-version kind/encoding metadata, add sanitized failure diagnostics, and correct the migration guide. The final focused run passes all 140 tests; the preceding eight-file run passed all 149 tests, including local storage and S3 diagnostics. Server typecheck and migration safety/numbering checks pass after the fixes. The second Apex review scored 1/5 and found unbounded maintenance storage calls and mismatched package/Git batch limits. Those are fixed with a 30-second cancellable cleanup deadline, shared 100 MiB/10,000-file scanner and storage limits, and bounded Git read batches. The proposed manual lockfile edit was rejected with evidence of the enforced CI-owned lockfile policy. All 19 storage/recovery unit tests and server typechecking pass. The new database-backed cleanup/conversion tests were locally skipped because the host exhausted its PostgreSQL semaphore capacity (confirmed directly by initdb). That gap is closed by current-head CI: [package-store tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109300889) pass all 18 cases and [package-worker tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109301157) pass all 7, with no skips in either suite. The corrected YAML browser shard passes. A separate planning-fixture test failed its cleanup DELETE with an activity-log foreign key violation after its behavior assertions passed; its one unchanged-head retry passed. The third Apex review scored 1/5 and identified a one-slot pool deadlock, projected path limits, cleanup holding the preparation lock during archive reads, and a publication/expiry race. These are fixed with transaction-local reads, prefix-aware bounds, isolated verification, and shared package row locks with a publication grace. The separate Paperclip review also found stale historical inspection, eager history downloads, and conversion retries blocking sleep. Those are fixed. Local PostgreSQL capacity recovered: the new database regressions ran, including the one-slot fork, paused storage during a warm turn, and a locked publication during expiry. The expanded run passed 144/145 tests; its remaining assertion still expected file bodies from the version list and was updated to fetch the selected version. The corrected company-skill suite passes all 81 tests, and the final package/projection run passes all 27. All 52 focused UI tests pass after updating their query-provider fixtures; the first UI run exposed a fixture remount, and the follow-up exposed an editing syntax error, both corrected. Full workspace typecheck and build, token gates, and map checks pass. The sandbox initially blocked the build tool’s local IPC socket; the authorized build rerun passed. Master then claimed migration 0401. The branch is rebased onf8226b0515, the unchanged SQL is regenerated as 0402, and repeated application passed twice on the combined schema. Post-rebase workspace typecheck passes. All 1,025 package, lifecycle, remote-launch, native-session, and asset-retention tests pass on the combined head. All CI checks passed for head 9d80a3b52d. The fourth Apex review scored 3/5 and found a save/edit lock-order cycle, a retry retaining an uncommitted source ID in its recovery recipe, and a summary read racing conversion. Commit b7d9237c6a fixes all three. The new concurrency and retry tests pass within all 61 package/source tests, and server typecheck passes. Master then claimed migration 0402. The branch is now rebased on65fa1cd617and the unchanged SQL is regenerated as 0403. All 68 package, source, and worker regressions, server typecheck, migration safety/numbering, and repeated migration application pass on current head62553671a0. Subsequent verification and review outcomes are recorded below. - The fifth Apex review scored 4/5 and found cached version reads unnecessarily depending on object storage and conversion reads missing the coordinator cancellation signal. Commit1eb90d80befixes both: cached reads verify bytes without the preparation lock; manifest and archive reads carry cancellation through conversion. The first focused run passed 70/71 tests, including all three new cases, but the existing diagnostic test observed no warning. Its fixture now sets an explicitly expired migration deadline; the final worker suite passes all nine cases. Server typecheck passes. - The sixth Apex review scored 4/5 with one finding: concurrent cached inventories were unbounded. Commit4b7589e14cgives cached inventories and previews a two-reader/32-waiter limit independent of cold storage. The first expanded pressure test observed a third reader before the preview path had the same bound; both paths now share the local limit. All 73 focused cases pass, followed by all 29 package cases after the final overload adjustment; server typecheck passes. The separate Paperclip review found no must-fix or should-fix issues. Its optional suggestion to persist durability verification was considered: cleanup intentionally verifies current external bytes before discarding the last local authored copy, and the missing-object regression requires that check. Shared-package eviction happens on cold fills with at most two attempts and a ten-second budget. - The seventh Apex review reached 5/5 with one remaining P2: timeout-bearing manifest reads bypassed request coalescing. The security review also found repeated archive acquisition after a cold administrative read. Commit79a80a39d2shares manifest loads with per-caller cancellation and cancels the underlying work when the last caller leaves; coalesces identical cold version/file requests; and publishes verified restored packages into the shared cache. Publication uses a short nonblocking lock, while eviction verification runs outside it. The first expanded run passed 73/74 tests and exposed a missing originating DB for cache publication inside read-only inspection; transaction readers now pass that owner explicitly, keeping metadata on the transaction. The corrected run passed 75/75. A final eviction test initially expected the shared package to remain after only its separate runtime projection was reused; it now explicitly accesses the shared package during verification. All 32 final package tests pass, including independent caller cancellation, one acquisition across concurrent/repeated reads, and warm preparation during stalled eviction verification. Server typecheck passes. - The eighth Apex review scored 4/5 with one finding: decoded cold inventories outlived the storage semaphore while waiting for cleanup. Commit9d7921d1ecadds a separate two-response/32-waiter lifetime bound held through cleanup, preserving the independent warm-read queue and forwarding cancellation through eviction verification. The new regression pauses cleanup after two responses decode, verifies a third archive stays queued, and completes warm preparation before releasing cleanup. All 77 package/source/worker cases and server typecheck pass. On the preceding head, serialized server shard 6/9 was interrupted with “The operation was canceled” after 57 seconds of tests; the log contains no failing assertion. Fresh latest-head CI reruns that shard along with the other gates. - The ninth Apex review scored 4/5 with one P1: cold responses could own both permits while waiting on a one-slot ordinary database pool held by a queued transaction reader. Commita94f0e409freads caller-scoped metadata before admission and isolates all acquisition/cleanup SQL on a private connection. The new real-PostgreSQL regression reproduces the deadlock against the previous head and passes with the fix; all 78 focused cases and server typecheck pass. The previous CI run completed with 54 successful checks, including both formerly interrupted/stuck serialized shards. - The tenth Apex review scored 4/5 with one P1: preloading legacy bodies before admission could retain many full repositories in queued requests. Commit276e353a37removes that preload and also bounds legacy manifest derivation/fallback, using private connections inside the I/O slot. All 80 focused tests pass; the final synchronized package suite passes all 36 tests and server typecheck passes. - Apex review 11 returned 5/5 with zero comments on276e353a37. While it ran, master8bc051245fadded migration 0403, creating a metadata conflict. Merge commite1783764d9integrates master and generates 0404_clumsy_leo.sql; its skill SQL is byte-for-byte identical to the previously verified migration. The combined schema passes migration safety/numbering, repeated SQL application twice, all 80 focused PostgreSQL tests, full workspace typecheck, and full build. The precedinga94f0e409fCI run was superseded and cancelled while its final runner job was active; no old-head success is claimed for that run. - Final acceptance: Apex review 12 returned 5/5 with zero comments one1783764d9. All 28 review threads are resolved. The final head has 55 successful checks, two neutral reports, two skipped checks, and no pending or failing checks; the PR workflow completed successfully. Superseded run 40789121902 was cancelled to free its workflow slot for the final head; normal cancellation did not finish promptly, so force cancellation completed it. - October 29 conflict resolution (867c00673b): master929e840595added installed-selection reuse and bundled relocation, routine cleanup notifications, a hardened YAML copy helper, and migrations through 0413. The merge retains those behaviors together with package read scoping and the public-selection fast path; restricted selections still reauthorize and the full row read rechecks scope. Generated migration 0414 retains the existing SQL unchanged. Workspace typecheck/build and migration checks pass. Local database verification is unavailable: initdb reports semget / “could not create semaphores: No space left on device”; all 168 cases skipped, and repeated migration application failed at startup. Docker returned HTTP 500 during container creation, so no Docker test pass is claimed. Current-head CI must close this gap. - Apex review 14 scored 3/5 with one P1: a local import could copy cached private package files into a company-wide skill. Commita2e597d800rejects cache trees and ancestors through a canonical filesystem boundary, including aliases and ancestors checked before a concurrent fill. Ten local filesystem tests and server typecheck pass. The existing private-project integration case now rejects package-cache imports for both authorized and unauthorized users. The preceding merge commit’s complete CI run passed, and logs confirm all 168 locally skipped cases executed successfully: package store 36, sources 35, package worker 9, runtime selection 4, observability 3, authoring 81. - Apex review 15 returned 5/5 with zero new findings ona2e597d800. The separate Paperclip review found no must-fix or should-fix issues. Its optional suggestions were considered: unexpected filesystem cleanup failures remain visible, the durable marker cannot replace checking that bytes still exist before deleting a last local authored copy, and administrative history batching remains outside the pre-turn optimization. The cache-path literal centralization suggestion is also optional; the current producer and guard use the same directory, covered by the new integration regression. All review conversations are resolved. - Earlier October 29 CI/Apex acceptance:a2e597d800was mergeable at that check, Apex review 15 passed 5/5 with zero findings, and all review conversations were resolved. The hosted verification above records the later acceptance limits and renewed conflicts. Workflow 40884087629 completed successfully, with 55 successful checks, two neutral reports, two skips, and no pending or failed checks. The project-skill-sources job log confirms all 29 cases ran and passed, closing the latest integration gap. The preceding merge workflow also confirmed all 168 focused database cases ran and passed. No local PostgreSQL or Docker pass is claimed. </details> ## Risks - First use after eviction can require storage or Git access. Revoked grants, removed sources, and unreachable commits can prevent reconstruction. Recovery never substitutes a newer commit. - Historical packages without a proven recipe keep durable bytes. Cleanup favors retention when a run pin cannot be interpreted safely. - New uploads and conversion use bounded work and persisted retries. Large initial conversions still add temporary storage and database load. - Optional cleanup can wait until the next server wake. Supported notifications assume one owning server process. - Per-entrypoint native tree duplication and cold preparation cost remain. The final ten-skill cold sample took 47.6 seconds on the shared host. - Generic authored edits can still hold an existing edit transaction while a package uploads. - Explicit file/folder deletion uses temporary disk space for rollback. An uncertain commit with a changed skill or unavailable database retains that backup and logs its location for recovery. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and tests. The exact served model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
Paperclip maps
The maps describe the same capabilities from two perspectives:
- Feature map: what a person can do, where to reach it, and how to verify the user journey. Preserve its Poteto format.
- Agent access map: the APIs and agent entry points for those capabilities, their permission boundaries, and access verification.
The chat scenario catalog adds natural-language requests, starting context and expected conversational outcomes for the same capability IDs. It lives with Product E2E as planning data; a scenario reference is neither a registered execution nor evidence that a journey passed.
These are sibling directories under maps/. Each feature file has a same-named
agent access companion. Reuse the product map's sub-feature IDs, so tasks/create
identifies the same capability in both maps. Product behavior remains governed by
the implementation spec; a map cannot grant
authority or redefine a permission denial as success.
Capability identifiers
The feature ID is the recipe filename without .md, such as onboarding.
The full capability ID joins that feature ID and a Sub-features key with /:
the company key in feature-map/onboarding.md is onboarding/company.
Its agent-access-map/onboarding.md companion uses the same local company key.
The feature-map index lists the existing
feature and full capability IDs.
Each child page states Feature ID: `onboarding`. after its introduction or
product-map link. In feature recipes, show the full ID beside the parsed local
key: - `company`: `onboarding/company` — create a company. In access
companions and Storybook verification guidance, keep the existing local heading
and put Capability ID: `onboarding/company`. beneath it. These labels make
the existing identifiers visible without changing keys, anchors, or ID generation.
Use full capability IDs in cross-feature references, Storybook annotations, tests, issues, PRs, and verification evidence. Both segments use lowercase letters, digits, and hyphens. Keep filenames and keys stable when display titles or navigation labels change; renaming either changes the capability ID.
Entry-point IDs describe a surface, such as web-wizard, and are scoped to their
recipe. Reference them with a recipe link, such as
onboarding.md#web-wizard, and record the
capability exercised separately. Navigation section labels are not capability IDs.
See IDs and references for examples.
Access contract
Every business operation needs both an API and a usable agent entry point for an
actor authorized to perform it. A dedicated tool is optional when REST, the CLI,
an assigned MCP client, or native search_api / call_api already provides the
operation. Record which agent surfaces actually have that path; an installed CLI
or optional MCP package cannot be assumed available to every native/remote run.
Track these independently:
| Dimension | Values and meaning |
|---|---|
| API | present, missing, not-applicable, or unreviewed. Use exact methods/paths; distinguish protocols from REST. |
| Agent access | available, partial, missing, restricted, operator-only, not-applicable, or unreviewed. Name the supported surface and actual entry point. |
| Policy | Actor, action/grant, company/resource/ownership scope, responsible-user intersection, run requirements, work modes, and gates relevant to the operation. |
| Evidence | source-only, verified, failed, blocked, or not-run. Source presence and successful board calls are not agent verification. |
| Review depth | baseline records representative operations and known gaps; operation-reviewed enumerates the capability's meaningful operations and their individual access contracts. |
Broad capabilities such as task properties must be split into operations before they count as operation-reviewed: renaming, metadata edits, assignment, dependencies, and lifecycle changes can use different tools and permission rules. An available read path cannot certify a write path. List/get/create/update/delete, execution/control, credential handling, and resolution are separate where their authority or transport differs. Do not duplicate request schemas from OpenAPI.
Exceptions and restrictions
Presentation-only state, static developer fixtures, and pre-start/local operator
work may use not-applicable for a domain API, with a concrete reason and the
existing local path. When a capability combines shared data with presentation,
split the operations: browser recents do not exempt resource lookup from API
coverage. Promoting local state to a shared feature requires revisiting the entry.
A board-only endpoint is restricted for agent identity, not an automatic
exception or a missing API. Cite the enforced policy and record whether the
agent can request the authorized human's action and observe its result. An
unresolved delegation question remains in Gaps. Company access, a job title,
a personal assistant's OAuth grant, or an accepted interaction does not grant
board identity. Keep budget, approval, secret and execution authority intact.
Check actual route/service authorization and production tool admission. OpenAPI actor labels and scenario/evaluation tool declarations are supporting metadata; record disagreements rather than using them as proof of access. Native API fallback restrictions, tool field/scope limits, and rollout settings are part of the agent path, even when a legacy agent can call the REST endpoint directly.
Companion file format
Each agent-access-map/<area>.md has an H1, a relative link to its product file,
and its feature ID, followed by one H2 backticked local key per product
sub-feature. Each capability starts with its full capability ID and contains
these fields:
- Review:
baselineoroperation-reviewed. - API: presence state and exact operation IDs, or a reasoned exception.
- Agent access: availability state and concrete entry points by surface.
- Policy: relevant permission and runtime boundaries, with source references.
- Evidence: evidence state, reviewed commit and sources or executed proof.
- Verification: allowed/denied assertions and observable result; record what has actually run separately from the proposed recipe.
- Gaps: missing APIs/paths, discovery drift, unreviewed operations or policy
decisions; use
noneonly for the enumerated scope that was reviewed.
For an operation-reviewed capability, use H3 backticked operation IDs and repeat
API, Agent access, Policy, Evidence, Verification and Gaps for each operation.
Use small operation names such as create, update, read, or run-now.
The enclosing capability has Review: operation-reviewed only after its
meaningful operations have been enumerated. The following excerpt shows one
operation; include the others before using that review label:
## `definition`
Review: `operation-reviewed`.
### `create`
API: `present` — POST /api/companies/{companyId}/routines.
Agent access: `available` — agent REST, CLI routine create and native call_api
under the enabled rollout and active-run/mode gates.
Policy: same company; assigneeAgentId must be the calling agent; cite the
routine authorization and native API policy sources.
Evidence: `source-only` — reviewed commit and source links; no live proof.
Verification: `not-run` — create an owned routine; reject assignment to another
agent; inspect saved configuration, attribution and retry behavior.
Gaps: verify ownership and responsible-user resolution with a permitted agent
context; source presence alone does not qualify this operation.
Keeping the maps synchronized
When a change adds/removes a capability or changes its behavior, API, permission, tool binding, CLI support, or work mode, update the affected companion and product recipe as appropriate in the same change. Preserve IDs across renames where the capability is unchanged; update the affected cross-links together. Indexes discover recipes automatically. An API-only change may update the companion without rewriting unchanged user instructions. Update the corresponding chat scenarios when their request, assumptions or expected outcome changes; use multiple cases for meaningfully different starting conditions rather than treating an API path as conversational coverage.
maintain-verification-skill owns reconciliation of both maps. It follows changed source and authorization, checks real agent reachability, and records product gaps without implementing them during a maintenance pass. The creation skill uses this structure. verify-paperclip consumes the relevant recipes and reports evidence/drift; map editing belongs to maintenance.
Run the structural check after map changes:
node scripts/check-maps.mjs
It checks paired areas/IDs, navigation metadata, required fields, and local references. It does not certify endpoint authorization, production bindings, operation completeness, or live behavior. Source audits must check those against the existing API and tool contracts; avoid a second schema catalog or runtime.
For live access proof, use Paperclip's existing API/runner/MCP fixtures or an owned
isolated instance. Use the actual agent identity, target and work mode. Assert the
allowed result and relevant denial (another company/private resource, wrong
owner, missing grant, invalid run, restricted mode or resolver). Mutations also
need persisted state/attribution and retry behavior. Capture evidence using the
existing artifact workflow; keep executed run results
separate from generated maps/generated/coverage.json, which lists page references.
Local authorship and generated indexes
Change only the recipes, access contracts and stories whose behavior or intent
changed. Feature recipes own navigation metadata; stories own small
parameters.paperclip annotations. Keep the feature-map README's readable ID
index aligned with those sources; it does not register entries for generation.
Do not refresh counts in shared READMEs, inventories, fingerprints or review
journals.
node scripts/storybook-catalog.mjs validates and generates aggregate navigation,
Storybook data, verification routes and the page inventory. All outputs live in
ignored maps/generated/, ui/storybook/generated/, and the ignored Storybook
preview.tsx wrapper. Start/build and UI typechecking generate their own inputs.
A clean merged checkout therefore assembles both branches' changes automatically.
Git provides history and diffs. PRs, issues and CI artifacts retain review/run evidence for the named revision. Capability associations do not create whole-file review dependencies. Keep durable scenario limits in the affected recipe or story.