mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 01:27:09 +01:00
4e97daafa06a54f02027335437c4f48e50b84236
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
9f3fc2094f |
Fix shared workspace seed capacity and trace deferred copyback (#18181)
Apply the existing company quota to company-shared workspace seed buckets. Preserve individual seed limits and recheck size after verification. Trace each deferred native workspace finalization check and its resumed copyback under the same hashed run ID. Keep existing admission, ownership and retry behavior. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
27fbdb2a74 |
perf(workspaces): reuse verified seeds across task branches (#18159)
## Thinking Path > - Paperclip manages AI agents and their workspaces. > - Remote runs stage a recoverable copy of the selected host workspace. > - New tasks use separate workspace IDs and Git branches. > - The verified seed cache currently treats those identities as different content. > - This causes repeated clones and seed verification when task files and Git commits are identical. > - This pull request shares immutable seed content within a company and applies each task's branch to a private archive. > - It also removes credential broker calls for two local Git snapshot commands. ## Linked Issues or Issue Description **What happened?** New remote tasks rebuild the same Git seed under different workspace IDs and branches. In a measured staging run, preparation took 29.7 seconds. Git clone took 6.1 seconds and optional seed publication took 7.8 seconds. Publication overlapped the 4.4-second transfer, so those values cannot be added. Local `diff-files` and `write-tree` commands also request GitHub credentials. **What did you expect?** Reuse a verified seed when filesystem contents, Git HEAD, origin, and nested repository layout match. Preserve each task's branch, recovery archive, ignored files, and copyback behavior. Keep local snapshot plumbing independent of GitHub credential capture. **Steps to reproduce** 1. Configure a remote native environment and a Git project workspace. 2. Create two tasks from the same source commit with different task branches and workspace IDs. 3. Inspect the workspace preparation spans. Both tasks currently clone and publish the same seed content. **Paperclip version and deployment mode** Reproduced on Core `25f1070fbdabd96b26cdf199e624a77905f3ca37`, with a managed remote sandbox and native Runner. The local comparison used a public repository at a fixed commit with roughly 13,000 entries. Refs #17886, #16398, #16399, #17357. The Git launcher extension builds on Jamie Buchanan's work in #17886. The other PRs establish the seed verification, recovery, and eviction contracts preserved here. A search found no open PR for sharing these verified seeds across task branches. ## What Changed - Use a company-scoped cache bucket for native host-current preparation. Keep the existing bucket and company storage limits and eviction rules. - Remove branch names from the shared generation identity. Keep file contents, Git HEAD, origin, and nested layout in that identity. Keep the default workspace-scoped identity unchanged for other callers. - Store a detached Git archive in the shared cache. Copy it and append the selected branch metadata before upload and private per-run persistence. Existing durable-seed replay can read the private archive without the new cache option. - Forward the scope through the execution target and both command-runtime paths. - Treat `diff-files` and `write-tree` as local Git commands. Network commands, unknown commands, and managed commit identity keep their existing credential path. - Test new workspace IDs, named and detached branches, nested repositories, dirty and ignored files, symlinks, recovery, corrupt caches, refused publication, and copyback. ## Verification - Current head `6087bac274d50804d1b332646e43f3b8d41d798f`: 53 checks passed, two optional checks skipped, Greptile 5/5, no unresolved review threads. Linux CI passed. - Adapter-utils TypeScript 7.0.2 typecheck passed. - Server package typecheck passed, including its Runner and plugin build prerequisites. - 140 seed-cache, seed-reuse, and sandbox-runtime tests passed after rebasing. - 44 Git launcher and command-runtime tests passed after rebasing. - 30 native workspace preparation, warm history, and retention-configuration tests passed. All 56 retention tests also passed (5.72 seconds) after repairing missing library aliases inside the installed macOS Postgres package. That local dependency repair changed no repository source. The earlier startup timeouts occurred before test assertions. - The new seed reuse regression failed before the change because the second branch cloned again. It passes through the execution target and both command-runtime paths after the change. - Clean local preparation comparison: two alternating measured pairs, with an existing verified company seed and new task branches. Baseline: 19.63 and 9.85 seconds. Shared cache: 5.18 and 4.80 seconds. Median: 14.74 to 4.99 seconds. Each baseline cloned and published; each shared-cache run skipped both. Branch and file checks passed. The complete comparison loop took 58 seconds. - The comparison used real Git, archives, snapshot workers, and local file transport. It used no model calls. It does not establish a complete staging response time. The first generation still pays the cold preparation cost. ## Risks - A shared cache hit requires an identical generation that is still present. The company bucket retains the existing 16-generation/1 GiB bucket limit; busy companies may miss more often. - The branch binding appends entries to a private tar archive. Tests cover the host platform; Linux CI must also pass before merge. - A cache miss, corruption, or capacity refusal must retain the private recovery path. Existing tests exercise those cases. - Git plumbing can still encounter repository-specific filters or missing objects. This extends the existing local-command policy by two commands; it does not grant network credentials to them. - The latest measured real staging reply took 77.8 seconds. This change targets a bounded part of that path and does not claim the under-15-second response goal is met. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and repository tools. The session does not expose the exact model revision or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
813e808af4 |
Remove filesystem history authorization from workspace preparation (#16963)
Remove physical-root source/reader authorization and sticky directory classifications from workspace preparation. Preserve current task/run authority, app-record privacy, credential authorization, containment, and lifecycle/publication integrity.
Validate replacement selections against persisted new inputs and refresh verified checkout ownership so bounded reuse survives earlier workspace relocation. Add real PostgreSQL/Git regressions, a reproducible history-scaling benchmark, and documentation.
Full CI passed all 48 jobs and Greptile scored 5/5 at
|
||
|
DottaandPaperclip
|
ce0eff7d1f |
perf: reduce fresh workspace clone and sandbox staging waits (#16487)
## Thinking Path > - Paperclip manages AI agents and their work. > - Each run needs a private workspace and safe persistence. > - Fresh isolated repository sources download upstream Git history again. > - Sandbox staging also verifies an optional seed cache before it uploads the workspace. > - That verification holds a company lock during a full extraction and hash scan. > - This pull request reuses pristine Git objects through an authenticated remote clone. > - It also overlaps seed verification with upload and moves verification outside the publication lock. > - The benefit is less repeated transfer and less serial preparation work, with the same task isolation and recovery contract. ## Linked Issues or Issue Description **What existing behavior does this improve?** Fresh task-private repository preparation and inbound sandbox workspace staging. **Current behavior** Each isolated remote source clones the full upstream history. Optional seed verification then extracts and hashes another full tree before upload. One workspace's verification can hold the company cache lock while other workspaces wait. **Proposed behavior** Use a company and credential scoped cache of pristine upstream objects for GitHub HTTPS sources. Each task still clones from the current remote. It disconnects from the cache before publication. Run seed verification alongside upload. Hold the company lock only for final capacity checks and publication. **Reason and benefit** These operations lie on the workspace preparation path. Removing repeated Git pack transfer and serial verification can reduce startup time. Local measurements below cover these boundaries. They do not establish hosted agent latency. Related: #16425, #16454, #16398, #16461, #16468, #16473. The cache inspection and durable folder proposal in #13376 covers separate product work. ## What Changed - Add a pristine upstream Git object cache for fresh isolated GitHub HTTPS sources. Key it by company, repository and current credential fingerprint. Do not store credentials. - Keep online clone authorization and current remote refs. Use `--reference-if-able --dissociate` so task objects remain independent. A reference failure retries once online without the cache and resolves credentials again. - Cap cache storage at 64 entries and 4 GiB per company, with 2 GiB per entry. Skip contended publication. Bound priming by a cooperative five-second budget and a 4,096-entry scan allowance. Keep the existing free-space reserve. - Overlap optional seed verification and required workspace upload. Settle both before returning or deleting temporary files. Recheck quotas and company directory identity under the publication lock. - Keep the independent fresh upstream reproducibility proof, task commits, dirty source copies, ignored files, nested repositories and durable recovery behavior. - Suppress optional Git server-info generation during dissociation so generated storage passes the existing safe-expiration checks. This uses [the supported Git setting](https://git-scm.com/docs/git-config/2.43.0#Documentation/git-config.txt-repackupdateServerInfo). - Add bounded timing phases, regression coverage and two repeatable local benchmarks. No first-party Telemetry or dependency changes. ## Verification - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - Affected adapter suites: 240 tests passed. Native workspace persistence/history and tracing: 33 tests passed. Server checkout/isolation/tracing batch: 53 tests passed. Final object-cache authority, path and capacity suite: 12 tests passed. These batches overlap. - Cache and expiration/restoration lifecycle batch: 34 tests passed. A direct generated-storage check covers cached clones. - Real HTTP Git fixture: a matching cache does not bypass revoked remote access. Other fixtures cover current refs/tags, hidden cached commits, independent objects after cache removal, corruption fallback, credential/company scope, quota limits and publication contention. - Initial restricted runs hit SQLite write and Unix socket permission failures. The adapter batch passed with disposable writable instance state and socket access. The first typecheck rejected the test use of `Promise.withResolvers`; typed deferred gates now use the supported library contract. - Repository benchmark on final head `9047368bf3b7fa2061e00d698b8b51cf4ae1da32`: five alternating samples, real local `file://` Git transport, synthetic 32 MiB history, no injected latency. Median 1143.8 ms to 240.5 ms; ranges 1085.7–1486.6 ms and 188.0–254.8 ms. Received Git pack bytes fell from 33,569,788 to zero. Initial cache priming took 1208.4 ms and is reported separately. - Sandbox benchmark, five alternating samples per mode on 10,000 files with real packing, extraction and verification: with a controlled 3-second transfer delay, median 9.450 s to 7.620 s (ranges 9.280–12.115 s and 7.296–7.983 s). With no injected delay, median 6.808 s to 6.450 s (ranges 6.255–7.029 s and 5.709–6.902 s). The no-delay ranges overlap; local copying competes with verification for disk. Both modes verify final tree contents, ignored-file exclusion and Git HEAD. These are synthetic local fixtures, not hosted agent runs. - Reproduce with `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-repository-object-cache.ts` and `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-workspace-staging.ts --baseline-ref=307eb16c0c --transfer-delay-ms=3000`. - Prior head `af766bc820` received Greptile 5/5 with no findings. Its E2E shard 7 failed in `plan-task-fixture` cleanup: company deletion raced an `activity_log` foreign key. That fixture has no provider run or Git workspace. The failure is retained here; final-head CI must pass. The corrected corruption fixture now forces a pack and verifies a real damaged archive. Current head `9047368bf3b7fa2061e00d698b8b51cf4ae1da32` has Greptile 5/5 with no review threads or findings. Final head has 55 passing checks and two optional Storybook skips. [The first workspace-shard attempt](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116655485350) passed all 1,778 cases, then reported a late `EPIPE` from unchanged `server-utils.ts` while the ACP reuse test was active. That case mocks workspace preparation and passed locally in isolation. [Its single retry on the same commit](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116659728034) passed, as did the dependent verification gate. The earlier E2E cleanup fixture passed on this final head. Both initial failures remain recorded here. A full local suite was not duplicated; the affected boundaries were run locally and the complete suite runs in CI. ## Risks - First cache population adds local copying. Cold clones, credential rotation, capacity exhaustion and a corrupt cache retain the online clone path. No eviction is added. Cached data is bounded, but many companies can each consume their allowance. - Individual filesystem calls and cleanup can exceed the cooperative priming budget. Failed reference clones make one additional online attempt. - Parallel verification and upload compete for disk and CPU. The local no-delay benchmark shows a smaller gain than a slow-transfer case. Verification still checks contents before cache publication, and required upload failures remain failures. - Local fixtures do not prove hosted throughput or the 15–30-second end-to-end target. The fresh checkout still writes its working tree, and model startup, copyback and other preparation costs remain. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context window are not exposed in this session. Used reasoning, source editing, shell tools and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal task identifiers - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c68a483cd9 |
fix: replay prepared warm seeds when generation caching is unavailable (#16399)
Replay this preparation’s verified private workspace seed after failed warm validation, including when the generation cache refuses publication. Preserve host-current semantics and rebuild if the private seed is missing or changed. Verified 89 focused cases, full typecheck/build, exact-head CI/E2E/security and Apex 5/5. Live PAP latency acceptance remains separate. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5bc4e878e7 |
fix: reuse verified seeds after warm workspace fallback (#16398)
Preserve the verified company/workspace seed-cache scope when a warm sandbox requires host-current staging. Keep remote identity checks, archive digests and rebuilding for missing or corrupt cache generations. 85 focused cases and full repository typecheck/build pass. Exact-head full CI/E2E, Superagent, legacy Snyk and Apex 5/5 pass with no unresolved findings. No staging response-time claim is made. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f329b24ee1 |
feat: give tasks durable project-optional workspaces (#15749)
Give tasks durable company-scoped workspace bindings independent of project assignment, channel defaults and authorized repository preparation. Reuse verified input seeds and sparse output recovery while preserving trust, privacy and controller provisioning boundaries.
Full CI and E2E, Superagent, legacy checks and Greptile Apex 5/5 passed on reviewed head
|