mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 05:16:50 +01:00
DottaandPaperclip
f1911a4c53
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/205/changes/paperclip/205 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents receive selected company skills before each turn. > - Imported skills can need files from the whole source repository. > - The current path stores those files in Postgres and reads large JSON payloads during preparation. > - This pull request stores verified packages in the configured file or object store and keeps preparation queries small. > - It also converts old rows automatically and restores missing files from the exact saved commit. > - The result reduces database traffic while preserving skill versions and repository dependencies. ## Linked Issues or Issue Description **What existing behavior does this improve?** Company skill import, version storage, and preparation before agent turns. **Current behavior** Repository snapshot bodies live in JSONB. Skill preparation can read those bodies again even when the files already exist on disk. Historical packages and native assets also need a bounded cleanup path. **Proposed behavior** Store package archives and manifests through the existing storage service. Keep version identities, receipts, hashes, and recovery metadata in Postgres. Convert legacy rows after the stored package passes read-back verification. A missing package can use legacy bytes or its exact Git commit under current repository authorization. No separate data-migration command is required. **Reason and benefit** The warm path sends no repository bodies through SQL and reads no object bytes. Cleanup runs through the existing deadline coordinator. Current versions, active runs, explicit pins, and authored content retain their protection. **Breaking changes** The normal application schema upgrade is required. Single-version and skill-file responses keep their existing shape. Version lists, current-version summaries, and test-run version summaries now contain file names and kinds without file bodies. Diff and restore fetch selected versions through the existing single-version endpoint. Older server builds must not write after conversion unless they support the new package format. The storage override and rollback limits are documented in `doc/skill-package-storage.md`. **Additional context** Related work: #15731 preserves repository dependencies; #17923 changes version-cache scheduling; #18080 changes delivery to remote Codex homes. This change owns package persistence, acquisition, and expiry. It does not replace those delivery or scheduler changes. ## What Changed - Add company-scoped archive receipts, exact-commit recovery recipes, migration deadlines, and a durable object-deletion queue. - Add bounded archive upload, read-back verification, extraction, concurrent request coalescing, and verified local caches. - Convert old snapshots and versions automatically. Keep legacy bytes when verification fails. - Recover missing files through current source authorization and the original commit. Validate repository identity, file hashes, modes, and the aggregate hash. - Expire unused recoverable Git packages and native assets after 30 days. Preserve active pins and the durable copy of authored or unproven content. - Keep warm preparation queries metadata-only. Bound file digest caches and avoid repeated body hashing when file identities are unchanged. - Extend the existing storage contract with cancellation. Register package work with the existing coordinator. Optional conversion retries retain the original bytes and may wait for ingress without blocking sleep. - Add regression tests, operating guidance, a deep trace, benchmark receipts, and an optional maintenance command. ## Verification ### Final master integration and review repair — October 29, 2026 Current head: `a6065424249e763dfeac37161f1b52a9f25bd9b7`. Commit `f8bbdd3dbd` resolves the conflicts with master `76a796e964`; the final two commits repair the Apex deletion findings below. - Keep both package-maintenance and version-retention notifications in their write transactions. - Keep the shared run-pin predicate and master's controller-lease deadline metadata. Package expiry reads the updated protected-version result. - Preserve every master migration entry. Generate migration **0415 (`0415_zippy_zemo.sql`)** after master's 0414. Its skill SQL is byte-for-byte unchanged from the previously tested idempotent migration. Retain only the five newest schema snapshots. Migration numbering and safety checks pass. - Repair failed file/folder deletion: preserve a complete temporary subtree until transaction commit, restore on upload or SQL failure, and restore callback failures before releasing the mutation lock. Both PostgreSQL regressions reproduced missing files before the fix and now pass. They cover binary data, executable permissions, empty folders, unlisted files, unchanged database/version rows, successful retry after upload-claim expiry, and historical reads. - Reconcile lost commit replies under the same advisory and row locks. Check the saved version and source identity before restoring. Keep committed deletions intact and preserve backups when the database outcome is unknown or a newer edit exists. Four real-PostgreSQL outcome tests pass; the prior code failed three of them. - Full workspace typecheck and production build pass on the final commit. All **204** focused PostgreSQL package/cache/retention/authoring and filesystem-boundary tests pass, with none skipped. - Apex review **18 passes 5/5**, with zero new comments on the exact final head. All **34** review conversations are resolved. - [Final-head CI](https://github.com/paperclipai/paperclip/actions/runs/40914116508) completes successfully on attempt 2. The head has **55 successful checks**, two neutral reports, two skipped checks, and a successful Snyk status. No checks are pending or failing. All eight browser shards pass. - Runner Vitest 2/2 timed out once in the unchanged ACPX controller-replacement fixture. The same job passed on the preceding commit. One unchanged CI retry passes all five ACPX recovery cases, **2,253** Runner package cases and **126** native server integration cases; their standard one and five skips remain. The initial timeout's root cause is not established. Local diagnosis also hit the fixture's eight-second session.open timeout with an authenticated Runner; no local Runner pass is claimed. - Both full local `pnpm test:run` attempts were stopped when Apex identified deletion defects and source repair became necessary; neither is claimed as a passing full local run. The final **204** focused tests, full workspace typecheck, and production build pass. Full-suite acceptance uses the final-head CI jobs. The superseded merge-head CI included runners terminated by shutdown signals; each source repair started a new full run. - Auto-merge is enabled for squash merge. The final head is conflict-free and all checks pass. GitHub's active master ruleset still requires a code-owner approval for the dependency files. The existing review requests are with `devinfoley`, `nickyleach`, and `forgottendev`; no policy bypass was used. - The earlier hosted stress and full native receipts below remain tied to `a2e597d800`; they are not relabeled as a deployment of this final integration. The earlier local PostgreSQL startup failure remains recorded below; PostgreSQL ran successfully for this integration verification. ### Full native skill execution — October 29, 2026 The full native-agent acceptance gap is closed on candidate `a2e597d800eb989d565780e7aa947d5368676444`. Two qualifying native turns used OpenAI API / `gpt-5.4-mini` in real Daytona sandboxes, on the same disposable hosted TEST tenant as the stress test. A follow-up on the same task consumed a newly saved skill version. The task text asked for the assigned skill workflow. It did not contain the proof nonce or expected receipt. The assigned authored skill contained a Python verifier and a random package-local nonce. The agent read the actual installed skill files and executed that verifier against the imported, automatically converted 1,360-file repository package. It checked the repository file count and reference/binary bytes, and directly executed the repository's executable probe. Independent verification matched the emitted JSON to the nonce and expected byte hashes. The agent persisted the same receipt as its task response; native finalization committed, workspace finalization succeeded, and task state became Done. | Qualifying turn | End-to-end run time | Recorded skills.prepare | Repository files | Binary hex | Shell receipt SHA-256 | |---|---:|---:|---:|---|---| | Initial task | 47.881 s | 1822 ms | 1360 | `00ff07800a` | `a1deb30dc095f2306c7aa9cc5ec396da12c68961cf5b809f4d7c20f7fc68493b` | | Corrected follow-up, new nonce/version | 37.974 s | 377 ms | 1360 | `00ff07800a` | `dbcdc33fa1376b4edc136feae1cdf770560a73292467e93619efdda20e4282a7` | Both successful shell receipts reported `probe: STRESS18082` and `assetSha256: e8ea7a2f15f7bb3b332a46adbef848465884c8996595aa234748befd544d1f3a`. Their proof nonces, selected version IDs and receipt hashes differed, and both ran on the same task. These are observed timings, not latency SLOs; they include real provider work. The service stress-test timings remain a separate measurement boundary. Earlier failures remain in the evidence: - The original five-minute Daytona allocation timeout never reached model execution. After normal owned-stack restore, its durable cleanup record was `expired` / `success`. The cause of that original allocation timeout remains unknown; a later success establishes recovery only. - Initial continuation setup used unsupported generic agent-update fields and returned HTTP 409 before dispatch. Setup was corrected to use the supported skill-assignment and versioned instruction commands. - The intervening completed native turn read repository bytes and ran the probe, but omitted the verifier JSON under the older test agent's overly-specific instruction file. Its strict receipt validator failed; it is not counted as a qualifying turn. Correcting the test agent's instruction file and sending an explicit task-scoped wake produced the qualifying follow-up above. - The pinned application's immutable rate card omits `gpt-5.4-mini`, so completed usage triggered the normal block-on-unpriced budget policy. Supported append-only accounting corrections estimated the three completed runs using the [official OpenAI model rate card](https://developers.openai.com/api/docs/models/gpt-5.4-mini): $0.75 input, $0.075 cached input, $4.50 output per million tokens, assuming standard processing. Estimates totaled 12.482655 cents. Original usage and missing-price records are preserved; the $1 hard stop and unpriced-usage guard stayed active. - Model shell attempts included a missing `xxd` command and incorrect guessed file paths. Those failed calls and subsequent successful actual verifier calls remain visible in complete event streams. No zero-tool-error claim is made. No product source changed during this continuation. The previously measured contention, cold-start and memory limits remain. The later merge resolution is covered by the separate integration checks above; these hosted receipts qualify their recorded candidate. ### Hosted stress test — October 29, 2026 **Hosted acceptance:** full native-agent execution and a changed-skill follow-up pass on the recorded hosted candidate. The final integration section above records the subsequent conflict resolution and its separate checks. Tested baseline `929e840595c8933ca485ac7a2e99413e1e11ca64` and this exact candidate sequentially on one disposable TEST tenant, with external Neon PostgreSQL and actual S3. The fixture contained **1,360 files / 22.4 MB / ten skills sharing one repository**. Measurements cover deployed skill-preparation and native-context services in an adjacent diagnostic process, not a complete agent turn. The container had **24 vCPU / 24 GB**. | Warm workload | Median before → after | p95 before → after | DB returned value bytes/preparation before → after | |---|---:|---:|---:| | One skill, serial | 1,060 → 125 ms | 1,303 → 155 ms | 22,610,389 → 1,771 | | One skill, four concurrent | 4,415 → 297 ms | 5,143 → 328 ms | 22,610,389 → 1,771 | | Ten skills, serial | 5,723 → 535 ms | 6,057 → 600 ms | 224,739,030 → 10,671 | | Ten skills, four concurrent | 24,410 → 1,986 ms | 24,833 → 2,221 ms | 224,739,030 → 10,671 | - Candidate warm preparation performed **zero S3 GETs**. Ordinary-pool queries fell from 16 to 9 for one skill and 61 to 9 for ten, excluding dedicated lock connections. DB numbers measure returned values before Drizzle decoding, not wire bytes. Baseline/candidate sample counts were 30/100, 32/100, 5/50 and 8/48 respectively; the small baseline ten-skill samples are comparative evidence, not SLO estimates. - **Application-UID soak:** 800 ten-skill preparations over 10.3 minutes, four workers paced by 2.5 seconds; **zero failures**, median **559 ms**, p95 **775 ms**, p99 **910 ms**, zero warm S3 reads. All 726 accompanying health/company-list requests returned HTTP 200. Their end-to-end p95 was 569/502 ms, including network latency. The probe overlaps most of the soak. Soak DB-response capture was limited to its first 60 seconds. - **Important remaining costs:** an unpaced 16-way ten-skill burst had p95 **8.32 seconds** because company-level preparation serializes. Cold preparation took approximately **1.1–2.5 seconds** for one skill and **7.4–10.0 seconds** for ten. Ten-skill diagnostic RSS remained about **1.17 GiB** (baseline 1.18 GiB), and the soak peaked at **1.35 GiB**. A concurrent whole-container sample was 3.45 GB. This does not establish suitability for a small-memory host or eliminate per-entrypoint native asset duplication. - **Upgrade without a manual data migration passed:** legacy snapshot JSON shrank from 22,465,691 bytes to `[]`, with a stored package and zero pending conversions. - **Recovery passed:** sixteen concurrent cold requests coalesced into two S3 GETs; full bytes/binary/executable modes matched; same-size corruption repaired; an injected storage failure preserved warm access and failed cold access safely. A disabled public-Git source denied recovery, and after the actual five-minute backoff its exact SHA reconstructed all 229 files. Cross-company descriptor access was denied. Private-Git credential revocation was not tested. - **Retention passed:** reclaimed 27,754,496 allocated cache bytes, deleted an unreferenced package row and both objects, preserved aged authored history, and expired/reacquired reconstructible Git content. A version pinned by the failed native run was correctly retained. Only owned fixture timestamps were aged. - **Browser UI passed:** authored skill creation, saved edit surviving reload, version history, and restoration of earlier content. - **Initial native attempt (superseded by the qualifying turns above):** one real run failed at Daytona allocation after five minutes (`setup_failed`), before any model execution. Provider cleanup also reported failure. That attempt does not count as a passing native turn. Its receipt and all evidence remain preserved; its cleanup later recorded success after normal restoration. - Initial A/B commands ran as SSH root. Diagnostic-owned lock/cache permissions caused one setup HTTP 500 and the first UID-1000 soak to fail; exact fixture paths were repaired. The final soak, retention and recovery ran as application UID 1000. A fresh UID-1000 one-skill batch also passed (30 samples, median 116 ms, p95 321 ms, 1,771 DB value bytes/preparation). Earlier harness-export, SSH, npm provenance, backoff and UI-wait failures remain recorded alongside their successful retries. - Private evidence includes raw timings, assertions, HTTP probes, screenshots and failure/cleanup receipts. Initial failed-campaign cleanup suspended and retained the owned TEST tenant and released the shared QA account lease at **15:22:41 UTC**. The tenant was subsequently resumed under its existing ownership for the native execution qualification above. Final cleanup completed at **15:53:09 UTC**: the account lease is released and the tenant is suspended/retained under the original October 30, 12:24:52 UTC retention deadline. Its separate stack lease remains held; no active or pending-cleanup environment lease remained. The original retention window is deliberately preserved even after successful continuation. - Earlier hosted-tested head: `a2e597d800eb989d565780e7aa947d5368676444`, integrated with master `929e840595`. The earlier merge conflicts were resolved at this head. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/40884087629) passed; all checks are complete with 55 successes, two neutral reports, two skips, and no failures. [Apex review 15](https://github.com/paperclipai/paperclip/pull/18082#issuecomment-6480257258) passed 5/5 with zero findings on this exact head. All review conversations are resolved. - Ten real-filesystem boundary cases and server typecheck pass locally for the security fix. Coverage includes cache roots, descendants, individual files, symlink aliases, recursive ancestors, a future concurrent cache fill, and allowed sibling directories. [Current-head private-project integration tests](https://github.com/paperclipai/paperclip/actions/runs/40884087629/job/123483306405) passed all 29 cases, including authorized and unauthorized imports of a cached private package. Full typecheck, build, server suites, all eight browser shards, and Runner checks passed in current-head CI. Local database tests could not run because PostgreSQL could not initialize; this limitation is recorded below. - The earlier generated migration preserved the newest five snapshots. The final integration renumbers its unchanged SQL to 0415. Migration safety/numbering checks pass. Its SQL is byte-for-byte identical to the previously verified idempotent skill migration; repeated application on this latest schema was blocked by the same local database startup failure. - The broader verification before the final migration-only rebase passed 1,025 package, lifecycle, remote-launch, native-session, and asset-retention tests, 52 focused UI tests, full workspace typecheck/build, token gates, and map checks. The linked complete CI run passed before the latest focused fixes. - A disposable MinIO instance passed multipart upload, verified read-back, and cold restoration of an 18 MB archive. - Twenty warm samples with a synthetic 1,349-file, 22.8 MB repository retained five metadata queries and zero object reads. Median preparation was 256 ms for one selected skill and 691 ms for ten, with 5 ms emulated network delay each way. These are local measurements, not production latency guarantees. - No production instance was changed. Earlier verification failures and their disposition are retained below. <details> <summary>Earlier failures, review findings, fixes, and rerun evidence</summary> - The first CI pass found a stale inventory-span expectation and a missing repository-read span. Both are fixed; all three observability tests pass. One browser test copied page text into its YAML parser after reload; its first unchanged-head retry passed, but the race recurred after rebasing. The test now waits for the renamed URL and completed reload before copying the editor, retaining its original assertions. The earlier full local suite did not complete cleanly: it reported setup timeouts and assertions, while isolated relevant reruns passed. The complete current-head CI run closes that broad verification gap. No production instance was changed. - The first Apex review scored 2/5 and identified six issues. The fixes protect authored history even when it shares an in-flight Git upload, recheck upload expiry during orphan deletion, bound failed-upload cleanup, retain per-version kind/encoding metadata, add sanitized failure diagnostics, and correct the migration guide. The final focused run passes all 140 tests; the preceding eight-file run passed all 149 tests, including local storage and S3 diagnostics. Server typecheck and migration safety/numbering checks pass after the fixes. The second Apex review scored 1/5 and found unbounded maintenance storage calls and mismatched package/Git batch limits. Those are fixed with a 30-second cancellable cleanup deadline, shared 100 MiB/10,000-file scanner and storage limits, and bounded Git read batches. The proposed manual lockfile edit was rejected with evidence of the enforced CI-owned lockfile policy. All 19 storage/recovery unit tests and server typechecking pass. The new database-backed cleanup/conversion tests were locally skipped because the host exhausted its PostgreSQL semaphore capacity (confirmed directly by initdb). That gap is closed by current-head CI: [package-store tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109300889) pass all 18 cases and [package-worker tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109301157) pass all 7, with no skips in either suite. The corrected YAML browser shard passes. A separate planning-fixture test failed its cleanup DELETE with an activity-log foreign key violation after its behavior assertions passed; its one unchanged-head retry passed. The third Apex review scored 1/5 and identified a one-slot pool deadlock, projected path limits, cleanup holding the preparation lock during archive reads, and a publication/expiry race. These are fixed with transaction-local reads, prefix-aware bounds, isolated verification, and shared package row locks with a publication grace. The separate Paperclip review also found stale historical inspection, eager history downloads, and conversion retries blocking sleep. Those are fixed. Local PostgreSQL capacity recovered: the new database regressions ran, including the one-slot fork, paused storage during a warm turn, and a locked publication during expiry. The expanded run passed 144/145 tests; its remaining assertion still expected file bodies from the version list and was updated to fetch the selected version. The corrected company-skill suite passes all 81 tests, and the final package/projection run passes all 27. All 52 focused UI tests pass after updating their query-provider fixtures; the first UI run exposed a fixture remount, and the follow-up exposed an editing syntax error, both corrected. Full workspace typecheck and build, token gates, and map checks pass. The sandbox initially blocked the build tool’s local IPC socket; the authorized build rerun passed. Master then claimed migration 0401. The branch is rebased onf8226b0515, the unchanged SQL is regenerated as 0402, and repeated application passed twice on the combined schema. Post-rebase workspace typecheck passes. All 1,025 package, lifecycle, remote-launch, native-session, and asset-retention tests pass on the combined head. All CI checks passed for head 9d80a3b52d. The fourth Apex review scored 3/5 and found a save/edit lock-order cycle, a retry retaining an uncommitted source ID in its recovery recipe, and a summary read racing conversion. Commit b7d9237c6a fixes all three. The new concurrency and retry tests pass within all 61 package/source tests, and server typecheck passes. Master then claimed migration 0402. The branch is now rebased on65fa1cd617and the unchanged SQL is regenerated as 0403. All 68 package, source, and worker regressions, server typecheck, migration safety/numbering, and repeated migration application pass on current head62553671a0. Subsequent verification and review outcomes are recorded below. - The fifth Apex review scored 4/5 and found cached version reads unnecessarily depending on object storage and conversion reads missing the coordinator cancellation signal. Commit1eb90d80befixes both: cached reads verify bytes without the preparation lock; manifest and archive reads carry cancellation through conversion. The first focused run passed 70/71 tests, including all three new cases, but the existing diagnostic test observed no warning. Its fixture now sets an explicitly expired migration deadline; the final worker suite passes all nine cases. Server typecheck passes. - The sixth Apex review scored 4/5 with one finding: concurrent cached inventories were unbounded. Commit4b7589e14cgives cached inventories and previews a two-reader/32-waiter limit independent of cold storage. The first expanded pressure test observed a third reader before the preview path had the same bound; both paths now share the local limit. All 73 focused cases pass, followed by all 29 package cases after the final overload adjustment; server typecheck passes. The separate Paperclip review found no must-fix or should-fix issues. Its optional suggestion to persist durability verification was considered: cleanup intentionally verifies current external bytes before discarding the last local authored copy, and the missing-object regression requires that check. Shared-package eviction happens on cold fills with at most two attempts and a ten-second budget. - The seventh Apex review reached 5/5 with one remaining P2: timeout-bearing manifest reads bypassed request coalescing. The security review also found repeated archive acquisition after a cold administrative read. Commit79a80a39d2shares manifest loads with per-caller cancellation and cancels the underlying work when the last caller leaves; coalesces identical cold version/file requests; and publishes verified restored packages into the shared cache. Publication uses a short nonblocking lock, while eviction verification runs outside it. The first expanded run passed 73/74 tests and exposed a missing originating DB for cache publication inside read-only inspection; transaction readers now pass that owner explicitly, keeping metadata on the transaction. The corrected run passed 75/75. A final eviction test initially expected the shared package to remain after only its separate runtime projection was reused; it now explicitly accesses the shared package during verification. All 32 final package tests pass, including independent caller cancellation, one acquisition across concurrent/repeated reads, and warm preparation during stalled eviction verification. Server typecheck passes. - The eighth Apex review scored 4/5 with one finding: decoded cold inventories outlived the storage semaphore while waiting for cleanup. Commit9d7921d1ecadds a separate two-response/32-waiter lifetime bound held through cleanup, preserving the independent warm-read queue and forwarding cancellation through eviction verification. The new regression pauses cleanup after two responses decode, verifies a third archive stays queued, and completes warm preparation before releasing cleanup. All 77 package/source/worker cases and server typecheck pass. On the preceding head, serialized server shard 6/9 was interrupted with “The operation was canceled” after 57 seconds of tests; the log contains no failing assertion. Fresh latest-head CI reruns that shard along with the other gates. - The ninth Apex review scored 4/5 with one P1: cold responses could own both permits while waiting on a one-slot ordinary database pool held by a queued transaction reader. Commita94f0e409freads caller-scoped metadata before admission and isolates all acquisition/cleanup SQL on a private connection. The new real-PostgreSQL regression reproduces the deadlock against the previous head and passes with the fix; all 78 focused cases and server typecheck pass. The previous CI run completed with 54 successful checks, including both formerly interrupted/stuck serialized shards. - The tenth Apex review scored 4/5 with one P1: preloading legacy bodies before admission could retain many full repositories in queued requests. Commit276e353a37removes that preload and also bounds legacy manifest derivation/fallback, using private connections inside the I/O slot. All 80 focused tests pass; the final synchronized package suite passes all 36 tests and server typecheck passes. - Apex review 11 returned 5/5 with zero comments on276e353a37. While it ran, master8bc051245fadded migration 0403, creating a metadata conflict. Merge commite1783764d9integrates master and generates 0404_clumsy_leo.sql; its skill SQL is byte-for-byte identical to the previously verified migration. The combined schema passes migration safety/numbering, repeated SQL application twice, all 80 focused PostgreSQL tests, full workspace typecheck, and full build. The precedinga94f0e409fCI run was superseded and cancelled while its final runner job was active; no old-head success is claimed for that run. - Final acceptance: Apex review 12 returned 5/5 with zero comments one1783764d9. All 28 review threads are resolved. The final head has 55 successful checks, two neutral reports, two skipped checks, and no pending or failing checks; the PR workflow completed successfully. Superseded run 40789121902 was cancelled to free its workflow slot for the final head; normal cancellation did not finish promptly, so force cancellation completed it. - October 29 conflict resolution (867c00673b): master929e840595added installed-selection reuse and bundled relocation, routine cleanup notifications, a hardened YAML copy helper, and migrations through 0413. The merge retains those behaviors together with package read scoping and the public-selection fast path; restricted selections still reauthorize and the full row read rechecks scope. Generated migration 0414 retains the existing SQL unchanged. Workspace typecheck/build and migration checks pass. Local database verification is unavailable: initdb reports semget / “could not create semaphores: No space left on device”; all 168 cases skipped, and repeated migration application failed at startup. Docker returned HTTP 500 during container creation, so no Docker test pass is claimed. Current-head CI must close this gap. - Apex review 14 scored 3/5 with one P1: a local import could copy cached private package files into a company-wide skill. Commita2e597d800rejects cache trees and ancestors through a canonical filesystem boundary, including aliases and ancestors checked before a concurrent fill. Ten local filesystem tests and server typecheck pass. The existing private-project integration case now rejects package-cache imports for both authorized and unauthorized users. The preceding merge commit’s complete CI run passed, and logs confirm all 168 locally skipped cases executed successfully: package store 36, sources 35, package worker 9, runtime selection 4, observability 3, authoring 81. - Apex review 15 returned 5/5 with zero new findings ona2e597d800. The separate Paperclip review found no must-fix or should-fix issues. Its optional suggestions were considered: unexpected filesystem cleanup failures remain visible, the durable marker cannot replace checking that bytes still exist before deleting a last local authored copy, and administrative history batching remains outside the pre-turn optimization. The cache-path literal centralization suggestion is also optional; the current producer and guard use the same directory, covered by the new integration regression. All review conversations are resolved. - Earlier October 29 CI/Apex acceptance:a2e597d800was mergeable at that check, Apex review 15 passed 5/5 with zero findings, and all review conversations were resolved. The hosted verification above records the later acceptance limits and renewed conflicts. Workflow 40884087629 completed successfully, with 55 successful checks, two neutral reports, two skips, and no pending or failed checks. The project-skill-sources job log confirms all 29 cases ran and passed, closing the latest integration gap. The preceding merge workflow also confirmed all 168 focused database cases ran and passed. No local PostgreSQL or Docker pass is claimed. </details> ## Risks - First use after eviction can require storage or Git access. Revoked grants, removed sources, and unreachable commits can prevent reconstruction. Recovery never substitutes a newer commit. - Historical packages without a proven recipe keep durable bytes. Cleanup favors retention when a run pin cannot be interpreted safely. - New uploads and conversion use bounded work and persisted retries. Large initial conversions still add temporary storage and database load. - Optional cleanup can wait until the next server wake. Supported notifications assume one owning server process. - Per-entrypoint native tree duplication and cold preparation cost remain. The final ten-skill cold sample took 47.6 seconds on the shared host. - Generic authored edits can still hold an existing edit transaction while a package uploads. - Explicit file/folder deletion uses temporary disk space for rollback. An uncertain commit with a changed skill or unavailable database retains that backup and logs its location for recovery. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and tests. The exact served model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
Quickstart · Docs · GitHub · Discord · Twitter · Website
Sign up for the Paperclip Cloud waitlist →
Paperclip is the app people use to manage AI agents for work.
Open-source orchestration for teams of AI agents.
If OpenClaw is an employee, Paperclip is the company.
Paperclip is a Node.js server and React UI that orchestrates a team of AI agents to run a business. Bring your own agents, assign goals, and track work and costs from one dashboard. Choose models and harnesses per agent while keeping your team's tasks, skills, permissions, and history in one place.
It looks like a task manager. Under the hood: org charts, budgets, governance, goal alignment, and agent coordination.
Manage business goals, not pull requests.
| Step | Example | |
|---|---|---|
| 01 | Define the goal | "Build the #1 AI note-taking app to $1M MRR." |
| 02 | Hire the team | CEO, CTO, engineers, designers, marketers — any bot, any provider. |
| 03 | Approve and run | Review strategy. Set budgets. Hit go. Monitor from the dashboard. |
| Works with |
 OpenClaw |
 Claude Code |
 Codex |
 Cursor + Cloud |
 Gemini CLI |
 OpenCode |
 Pi |
 Hermes + Gateway |
 Grok Build |
 Kimi Code |
If it can receive a heartbeat, it's hired.
Custom processes, HTTP endpoints, and external adapter packages extend the roster. See the adapter overview for setup and capabilities.
Paperclip is right for you if
- ✅ You want to build autonomous AI organizations
- ✅ You coordinate many different agents (OpenClaw, Codex, Claude, Cursor) toward a common goal
- ✅ You have 20 simultaneous Claude Code terminals open and lose track of what everyone is doing
- ✅ You want agents running autonomously 24/7, but still want to audit work and chime in when needed
- ✅ You want to monitor costs and enforce budgets
- ✅ You want a process for managing agents that feels like using a task manager
- ✅ You want to manage your autonomous businesses from your phone
The four pillars
Four things have to work for an organization of AI agents to actually produce: the tasks, the org, the training, and the infrastructure. Paperclip is built around exactly those four pillars.
| Pillar | Built for | What it covers |
|---|---|---|
| Agentic Task Manager — Declare intent. Agents work. You verify the output. | Everyone, daily | Tasks, approvals & review gates · proactive agent coworkers · auditable routines & workflows · verify from diffs, screenshots & tests |
| Org Chart for Agents — Roles, permissions & boundaries for humans and agents. | Managers | Mixed human + agent org chart · responsibilities, delegation, specialization · governance: who can do what · scoped secrets & company boundaries · connection permissions & responsible-user identities |
| Agent Employee Training — Design, train & evaluate your AI employees. | Enablers | Skill Studio & shared org-wide skills · evals & saved test runs · active learning loops & quality metrics · performance reviews for agents · saved test inputs · skill version history & restore · reusable team templates |
| Agentic OS — The infrastructure that makes the work run. | IT & platform | Cross-provider runtime: any model, any agent · sandboxing, integrations & MCP servers · SSO, GRC, RBAC & cost controls · data privacy, internal trace collection, compounding data value · personal & shared app connections · run history & opt-in tracing |
Features
🔌 Bring Your Own AgentAny agent, any runtime, one org chart. If it can receive a heartbeat, it's hired. |
🎯 Goal AlignmentLink tasks and projects to your organization goals. Agents receive the goal context behind their work. |
💓 HeartbeatsAgents wake for assigned work, follow-up messages, or configured schedules. Delegation flows up and down the org chart. |
💰 Cost ControlCompany, agent, and project budgets. Track reported spend, get threshold alerts, and pause work at configured limits. |
🏢 Multi-OrganizationOne deployment, many organizations. Separate tasks, agents, permissions, and activity histories for each. |
🎫 Task ThreadsKeep conversations, plans, blockers, files, and run history attached to the work. Assign tasks to agents or people. |
🛡️ GovernanceConfigure review and approval stages, approve hires, and pause, reassign, or stop work when needed. |
📊 Org ChartHierarchies, roles, reporting lines. Your agents have a boss, a title, and a job description. |
📱 Mobile ReadyMonitor and manage your autonomous businesses from anywhere. |
🔗 Apps & ConnectionsConnect services such as GitHub, Notion, and Railway, or your own MCP server. Set gateway actions to Allowed, Ask first, or Off. |
👥 Shared Agents, Personal AccountsChoose who can use a connection and which agents can access it. Managed GitHub operations can use the account of the person directing the work. |
🧠 Skills & Skill StudioInstall or write shared skills, test them with saved inputs, inspect results, and restore earlier versions. |
📅 Scheduled RoutinesRun recurring work on a schedule or trigger it through an API or webhook. Each run has a task, an owner, and a history. |
📎 Artifacts & FeedbackFind the files and documents agents produce. Preview supported formats and leave comments on specific passages in documents. |
📦 Ready-Made TeamsPreview and install teams with roles, skills, projects, and routines. Choose their runtimes and make the setup your own. |
Experimental Agent Chat and chat/email connectors add conversations with agents in Paperclip and through configured services such as Slack, Discord, Telegram, and AgentMail. Enable the relevant instance settings to try them.
Problems Paperclip solves
| Without Paperclip | With Paperclip |
|---|---|
| ❌ You have 20 Claude Code tabs open and can't track which one does what. On reboot you lose everything. | ✅ Tasks are ticket-based, conversations are threaded, sessions persist across reboots. |
| ❌ You manually gather context from several places to remind your bot what you're actually doing. | ✅ Context flows from the task up through the project and company goals — your agent always knows what to do and why. |
| ❌ Folders of agent configs are disorganized and you're re-inventing task management, communication, and coordination between agents. | ✅ Paperclip gives you org charts, ticketing, delegation, and governance out of the box — so you run a company, not a pile of scripts. |
| ❌ Runaway loops waste hundreds of dollars of tokens and max your quota before you even know what happened. | ✅ Spend tracking, budget alerts, and automatic pauses help you control the cost of ongoing work. |
| ❌ You have recurring jobs (customer support, social, reports) and have to remember to manually kick them off. | ✅ Routines create assigned tasks on a schedule, with outputs and run history you can inspect. |
| ❌ You have an idea, you have to find your repo, fire up Claude Code, keep a tab open, and babysit it. | ✅ Add a task in Paperclip. Your coding agent works on it until it's done. Management reviews their work. |
Why Paperclip is special
Paperclip handles the hard orchestration details correctly.
| Atomic task checkout. | A single assignee and execution locks prevent competing runs from claiming the same task. |
| Persistent work context. | Tasks, comments, and documents stay in Paperclip. Supporting adapters resume saved sessions across runs. |
| Runtime skill injection. | Agents can learn Paperclip workflows and project context at runtime, without retraining. |
| Governance with rollback. | Approval gates are enforced, config changes are revisioned, and bad changes can be rolled back safely. |
| Accountable connections. | Human access, agent eligibility, and gateway action permissions are separate controls. Approve a call once or save a revocable rule. |
| Goal-aware execution. | Linked tasks and projects carry goal ancestry so agents see the "why," not just a title. |
| Portable company templates. | Export/import orgs, agents, and skills with secret scrubbing and collision handling. |
| Organization boundaries. | Company-scoped access checks keep each organization's work, agents, and activity separate within one deployment. |
What's Under the Hood
Paperclip is a full control plane, not a wrapper. Before you build any of this yourself, know that it already exists:
┌──────────────────────────────────────────────────────────────┐
│ PAPERCLIP SERVER │
│ │
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
│ │Identity & │ │ Work & │ │ Heartbeat │ │Governance │ │
│ │ Access │ │ Tasks │ │ Execution │ │& Approvals│ │
│ └───────────┘ └───────────┘ └───────────┘ └───────────┘ │
│ │
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
│ │ Org Chart │ │Workspaces │ │ Plugins │ │ Budget │ │
│ │ & Agents │ │ & Runtime │ │ │ │ & Costs │ │
│ └───────────┘ └───────────┘ └───────────┘ └───────────┘ │
│ │
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
│ │ Routines │ │ Secrets & │ │ Activity │ │ Company │ │
│ │& Schedules│ │ Storage │ │ & Events │ │Portability│ │
│ └───────────┘ └───────────┘ └───────────┘ └───────────┘ │
└──────────────────────────────────────────────────────────────┘
▲ ▲ ▲ ▲
┌─────┴─────┐ ┌─────┴─────┐ ┌─────┴─────┐ ┌─────┴─────┐
│ Claude │ │ Codex │ │ CLI │ │ HTTP/web │
│ Code │ │ │ │ agents │ │ bots │
└───────────┘ └───────────┘ └───────────┘ └───────────┘
The Systems
|
Identity & Access — Two deployment modes (trusted local or authenticated), human roles and permissions, agent API keys, short-lived run JWTs, company memberships, and invite flows. Responsible-user attribution follows work through delegation and supported managed connections. |
Org Chart & Agents — Agents have roles, titles, reporting lines, permissions, and budgets. Adapter examples match the diagram: Claude Code, Codex, CLI agents such as Cursor/Gemini/bash, HTTP/webhook bots such as OpenClaw, and external adapter plugins. If it can receive a heartbeat, it's hired. |
|
Work & Task System — Issues carry company/project/goal/parent links, atomic checkout with execution locks, first-class blocker dependencies, comments, documents, attachments, work products, labels, and inbox state. Search across work, review document revisions, and leave anchored feedback. |
Heartbeat Execution — DB-backed wakeup queue with coalescing, budget checks, workspace resolution, secret injection, skill loading, and adapter invocation. Runs produce logs, usage records, and adapter-specific session state. Bounded recovery handles supported failures and surfaces cases that need human action. |
|
Workspaces & Runtime — Project repositories and workspaces, optional isolated execution workspaces (git worktrees, operator branches), and runtime services (dev servers, preview URLs). Sandbox providers extend execution beyond the local host; availability depends on the configured environment and adapter. |
Governance & Approvals — Board approval workflows, execution policies with review/approval stages, decision tracking, budget hard-stops, and agent pause/resume/terminate. Configured task reviews govern completion; connection action approvals govern calls through the tool gateway. |
|
Budget & Cost Control — Token and cost tracking by company, agent, project, goal, issue, provider, and model. Scoped budget policies with warning thresholds and hard stops. Enforcement uses recorded spend; usage reporting and in-flight work can delay a stop. |
Routines & Schedules — Recurring tasks with cron, webhook, and API triggers. Concurrency and catch-up policies. Each routine execution creates a tracked issue and wakes the assigned agent — no manual kick-offs needed. |
|
Plugins — Instance-wide plugin system with out-of-process workers, capability-gated host services, job scheduling, tool exposure, and UI contributions. Extend Paperclip without forking it. |
Secrets & Storage — Company secrets and per-person secret values, encrypted credential storage, local or S3-compatible file storage, attachments, and work products. Secret references supply credentials to authorized runs without copying values into ordinary agent configuration. |
|
Activity & Events — Mutating actions, heartbeat state changes, cost events, approvals, comments, and work products are recorded as durable activity so operators can audit what happened and why. |
Company Portability — Preview, export, and import organization packages with agents, skills, and optional projects, routines, tasks, and attachments. Referenced secret values are omitted; review packages before sharing because plain environment values and local paths can remain. Packages share an operating setup; full-instance recovery uses backups. |
What Paperclip is not
| Not just a chatbot. | Conversations stay attached to tasks, plans, decisions, and outputs. Experimental Agent Chat can hand work off to assigned tasks. |
| Not an agent framework. | We don't tell you how to build agents. We tell you how to run a company made of them. |
| Not just a workflow builder. | Routines operate within an organization, with roles, goals, budgets, and governance. |
| Not a prompt manager. | Agents bring their own prompts, models, and runtimes. Paperclip manages the organization they work in. |
| Not limited to one agent. | Start with one agent and grow into a team with shared skills, delegation, and review. |
| Not only for code review. | Coding and PR review fit alongside research, operations, content, and other work. |
Quickstart
Open source. Self-hosted. No Paperclip account required. Follow the guided quickstart to set up your first agent.
Just ask your agent to install Paperclip
Share the installation guide with your agent.
Or install it yourself
With Node.js 24.11 or newer installed:
npx paperclipai@latest onboard --yes
The CLI runs from npm's cache; your instance configuration and data persist locally.
See the installation guide for managed installs, pinned versions, canary and git-ref installs, updates, rollback, service management, and uninstalling.
For an isolated manual test instance that is already initialized with a CEO
agent, use test-drive. It stays in the foreground, never installs a service
or creates a first task, and opens the browser only after setup succeeds:
ANTHROPIC_API_KEY=... npx paperclipai test-drive
OPENAI_API_KEY=... npx paperclipai test-drive --harness codex
OPENROUTER_API_KEY=... npx paperclipai test-drive \
--harness opencode \
--model openrouter/anthropic/claude-sonnet-4.5
Each run without --data-dir gets a unique, retained temporary directory; its
absolute path is printed at startup. Pass --data-dir to reuse one, or
--no-browser to leave the initialized instance unopened. When invoked from a
linked Git worktree, test-drive also enables task execution in that worktree.
See the test-drive guide for credential and
reuse behavior.
Troubleshooting: private npm registry
.npmrcIf this fails with an
E404forpaperclipai(or similar) and you use a private npm registry (for example GitHub Packages) via a global~/.npmrc,npxmay be resolvingpaperclipaiagainst that private registry instead of the public npm registry.Diagnostic:
npm config get registryWorkaround (cross-platform; force the public npm registry for this command):
npx --registry https://registry.npmjs.org paperclipai@latest onboard --yes
That quickstart path now defaults to trusted local loopback mode for the fastest first run. To start in authenticated/private mode instead, choose a bind preset explicitly:
npx paperclipai@latest onboard --yes --bind lan
# or:
npx paperclipai@latest onboard --yes --bind tailnet
If you already have Paperclip configured, rerunning onboard keeps the existing config in place. Use npx paperclipai configure to edit settings.
Or manually:
git clone https://github.com/paperclipai/paperclip.git
cd paperclip
pnpm install
pnpm dev
This starts the UI and API at http://localhost:3100. An embedded PostgreSQL database is created automatically — no setup required.
Requirements: Node.js 24.11+, pnpm 9.15+
Local Claude and Codex subscription sign-in also needs Python 3 and the corresponding provider CLI on the Paperclip host. The Docker image includes them.
Source development also builds the native Paperclip Runner when enabled (the self-hosted default). Install a Rust toolchain, or set PAPERCLIP_RUNNER_BINARY to a compatible prebuilt runner.
FAQ
Q: Is this project maintained or just slop?
A: Paperclip is maintained by the Paperclip team. We've merged over 2,700 pull requests.
Q: What does a typical setup look like?
A: Locally, a single Node.js process manages an embedded Postgres and local file storage. For production, point it at your own Postgres and deploy however you like. Configure projects, agents, and goals — the agents take care of the rest.
For remote access, use authenticated mode with a private-network bind such as Tailscale, or deploy the persistent server with Docker. See deployment modes and the Docker guide.
Q: Can I run multiple companies?
A: Yes. A single deployment can host multiple organizations with company-scoped data and access checks.
Q: How is Paperclip different from agents like OpenClaw or Claude Code?
A: Paperclip uses those agents. It orchestrates them into a company — with org charts, budgets, goals, governance, and accountability.
Q: Why should I use Paperclip instead of just pointing my OpenClaw to Asana or Trello?
A: Agent orchestration has subtleties in how you coordinate who has work checked out, how to maintain sessions, monitoring costs, establishing governance - Paperclip does this for you.
(Bring-your-own-ticket-system is on the Roadmap)
Q: Do agents run continuously?
A: Agents wake for assigned work and follow-up messages. Optional timer heartbeats let them check for work periodically; routines create recurring tasks on their own schedules. You can also connect externally running agents such as OpenClaw. A mention alone does not assign work or wake another agent.
Development
pnpm dev # Full dev (API + UI, watch mode)
pnpm dev:once # Full dev without file watching
pnpm dev:server # Server only
pnpm dev:mobile # Serve prebuilt UI on :3101 for phones/tablets (proxies /api → :3100)
pnpm dev:both # Run `pnpm dev` and `pnpm dev:mobile` together
pnpm build # Build all
pnpm typecheck # Type checking
pnpm test # Cheap default test run (Vitest only)
pnpm test:watch # Vitest watch mode
pnpm test:e2e # Playwright browser suite
pnpm db:generate # Generate DB migration
pnpm db:migrate # Apply migrations
pnpm test does not run Playwright. Browser suites stay separate and are typically run only when working on those flows or in CI.
See doc/DEVELOPING.md for the full development guide.
Roadmap
- ✅ Plugin system (e.g. add a knowledge base, custom tracing, queues, etc)
- ✅ Get OpenClaw / claw-style agent employees
- ✅ companies.sh - import and export entire organizations
- ✅ Easy AGENTS.md configurations
- ✅ Skills Manager, Skill Studio & Skills Store
- ✅ Scheduled Routines
- ✅ Better Budgeting
- ✅ Agent Reviews and Approvals
- ✅ Multiple Human Users
- ✅ Cloud / Sandbox agents (e2b, Cloudflare, Daytona, Modal, Novita, boxd, self-hosted Kubernetes)
- ✅ Artifacts & Work Products
- ✅ Deep Planning (planning mode, revisioned plans, plan approvals)
- ✅ Enforced Outcomes (watchdogs, recovery actions, review gates)
- ✅ MCP Tool Gateway & Apps (governed tool access)
- ✅ Secrets Manager with per-agent access
- ✅ Activity log & action attribution
- ✅ Self-healing runs & automatic recovery
- ✅ Agent evals & feedback
- ✅ Connected Apps
- ✅ Personal & Shared AI Accounts
- ✅ Shared Agents Use Personal GitHub Identities
- ✅ Skill Version History & Restore
- ✅ Document Comments & Revision History
- ✅ Company-Wide Search
- ✅ Multi-Model & Multi-Harness Teams
- 🟡 Memory / Knowledge
- ⚪ MAXIMIZER MODE
- ⚪ Work Queues
- ⚪ Self-Organization
- ⚪ Automatic Organizational Learning
- 🟡 Agent Chat
- 🟡 Cloud deployments
- ⚪ Desktop App
- ⚪ Bring-your-own-ticket-system (Asana / Linear / Jira as on-ramps)
This is the short roadmap preview. See the full roadmap in ROADMAP.md.
Community & Plugins
Find Plugins and more at awesome-paperclip
Observability
Paperclip ships with opt-in OpenTelemetry auto-instrumentation for the server (traces only). It activates when OTEL_EXPORTER_OTLP_ENDPOINT is set and supports grpc, http/protobuf, and http/json via the standard OTEL_EXPORTER_OTLP_PROTOCOL env var. @opentelemetry/api is a normal server dependency; the SDK, auto-instrumentation, and exporter packages are optional peer dependencies — install them only if you want tracing. See doc/observability.md for install commands and the full env-var reference.
Paperclip also ships with opt-in Sentry error monitoring for the server and the browser. Set SENTRY_DSN_FRONTEND to activate it for the browser and SENTRY_DSN_BACKEND to activate it for the server — each variable is optional, and the legacy SENTRY_DSN variable still works as a fallback for either component. The supported server SDK version is @sentry/node@10.71.0; it is an optional peer dependency for the server, so install it only if you want error monitoring. The browser SDK, @sentry/browser, is pinned to the same exact version. See doc/observability.md for the install command, the privacy settings, and the full default capture set.
Telemetry
Paperclip collects anonymous usage telemetry to help us understand how the product is used and improve it. No personal information, issue content, prompts, file paths, or secrets are ever collected. Private repository references are hashed with a per-install salt before being sent.
Contributors changing emitted telemetry events should follow the Telemetry Data Contract. For proposed first-party events that are not in the generated contract yet, follow Telemetry Workflow.
Telemetry is enabled by default and can be disabled with any of the following:
| Method | How |
|---|---|
| Environment variable | PAPERCLIP_TELEMETRY_DISABLED=1 |
| Standard convention | DO_NOT_TRACK=1 |
| CI environments | Automatically disabled when CI=true |
| Config file | Set telemetry.enabled: false in your Paperclip config |
Contributing
We welcome contributions. See the contributing guide for details.
Community
- Discord — Join the community
- Twitter / X — Follow updates and announcements
- GitHub Issues — bugs and feature requests
- GitHub Discussions — ideas and RFC
License
MIT © 2026 Paperclip Labs, Inc
Star History
Open source under MIT. Built for people who want to get work done, not babysit agents.