mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-17 20:09:05 +02:00
3081b7480fb2c402bfe2cc53fa99e643a4441349
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Nicky LeachandPaperclip
|
233be4b36c |
feat: parallelize sandbox file-sync behind a provider opt-in capability (#11736)
## Thinking Path > - Paperclip runs AI agents through local and remote execution adapters. > - Sandbox providers move workspace and asset files before and after agent runs. > - Serial file transfers delay startup and teardown when several operations do not depend on each other. > - Providers need an opt-in contract so existing providers keep their serial behavior. > - This pull request adds a bounded scheduler and routes inbound and outbound sync operations through it. > - The benefit is shorter sandbox setup and teardown with stable errors, clear telemetry, and a safe opt-in path. ## Linked Issues or Issue Description **Subsystem affected** Cross-cutting (multiple of the above): packages/shared, packages/adapter-utils, packages/plugins, and server. **Problem or motivation** Sandbox sync processes the workspace, assets, and referenced projects in series. This adds avoidable wait time to agent startup and teardown. **Proposed solution** Add a fail-closed provider capability named concurrentSyncOperations. Use a bounded scheduler with a limit of four operations. Preserve operation order for error reporting. Keep non-opted-in providers on the serial path. **Alternatives considered** Increase the serial transfer speed or add provider-specific schedulers. Those options do not provide one shared contract or stable behavior across providers. **Roadmap alignment** ROADMAP.md lists cloud and sandbox agents as a product area. This change improves sandbox execution without changing the control-plane contract. **Additional context** The Daytona provider opts in. Board trials on this commit showed overlap for inbound sync and outbound restore, with no referenced-project staging failures. ## What Changed - Add the concurrentSyncOperations sandbox capability and fail-closed parsing. - Add a bounded settle-all scheduler with stable input-order errors. - Parallelize inbound workspace, asset, and referenced-project sync operations when the provider opts in. - Parallelize outbound workspace and asset restore operations when the provider opts in. - Surface referenced-project failure text in run logs and server telemetry. - Add Daytona sync spans and the capability declaration. - Preserve in-flight upload scratch tarballs during workspace wipe. - Add unit and regression tests for the scheduler, coordinators, provider behavior, telemetry, and wipe race. ## Verification - Run the adapter-utils and server type checks. - Run the targeted adapter-utils, server, and Daytona test suites. - Run the full automated sweep. - Review six cold Daytona trials, with three serial and three parallel runs. - Confirm that parallel trials show inbound overlap and outbound restore overlap. - Confirm that providers without the capability keep serial behavior. ## Risks - Providers must opt in only when their file operations can run safely at the same time. - A provider that declares the capability incorrectly can expose transfer races. - The scheduler keeps a limit of four to bound resource use. - Providers without the capability keep the prior serial behavior. ## Model Used OpenAI GPT-5 in the Codex runtime. The model used tool calls, code inspection, and GitHub workflow support. The model did not author the implementation commits. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub #NNN / github.com/paperclipai/paperclip URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
9ace548fd2 |
feat(observability): rename sandbox provider spans and add run-time wrapper spans (#10999)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip uses adapter and sandbox code to start agents and run sandbox work > - The current sandbox spans use mixed names and do not group related run-time work > - Mixed names make traces harder to read and compare across providers > - This pull request renames provider spans, adds run-time wrapper spans, and keeps the host allowlist closed > - The benefit is clearer traces with the same sandbox behavior and trust boundary ## Linked Issues or Issue Description **What existing behavior does this improve?** This improves OpenTelemetry span names and grouping for sandbox startup, execution, callback relay, and agent session work. **Subsystem affected** Cross-cutting (multiple of the above): adapter utilities, sandbox providers, shared telemetry documentation, and server instrumentation. **Current behavior** Sandbox provider spans use mixed names. Related run-time operations expose inner `sandbox.exec` spans without a named wrapper span. The host mapper uses a closed allowlist for provider span names. **Proposed behavior** Use descriptive provider-scoped span names. Add wrapper spans for agent session input, agent session output polling, and callback relay. Keep the host mapper allowlist closed and map unknown names to `other`. **Reason and benefit** Clear names make traces easier to read and reduce ambiguity during sandbox operation analysis. Wrapper spans show the full operation while preserving the inner execution spans. **Breaking changes** None. This change updates telemetry span names and grouping only. It does not change sandbox behavior, endpoint behavior, or the host trust boundary. **Additional context** Related prior work: [#10758](https://github.com/paperclipai/paperclip/pull/10758). ## What Changed - Rename Daytona provider sync and session spans with descriptive provider-scoped names. - Add three run-time wrapper spans for agent session input, output polling, and callback relay. - Add a shared span runner that preserves no-op behavior without a real tracer. - Keep the host mapper allowlist closed and map unknown names to `other`. - Update telemetry documentation and span-name tests. ## Verification - Focused adapter-utils span tests pass for startup timing, callback relay, and sandbox execution. - Focused Daytona plugin span tests pass for renamed leaf spans and session open or close spans. - Focused server tests pass for host mapping and instrumentation. - The stacked diff contains one commit on top of `feat/daytona-persistent-session-model`. ## Risks - Span names change for existing telemetry consumers. - The wrapper spans add trace structure but do not change sandbox execution. - The host mapper keeps the existing closed allowlist and `other` bucket. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI GPT-5 (Codex agent); exact deployment revision and context window are not exposed in this run; tool use and code execution enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
cfed36ea6b |
feat(plugin-daytona): persistent session model with plain command dispatch (#10941)
## Thinking Path > - Paperclip is the control plane for autonomous AI companies > - One core subsystem runs agent work inside sandboxes > - The Daytona provider uses that path to run user commands > - The current one-shot model does not keep a shell alive across commands > - This pull request adds an opt-in persistent session model for Daytona > - The benefit is faster command dispatch with the same sandbox boundaries ## Linked Issues or Issue Description ### Subsystem affected Cross-cutting. This change touches `packages/adapters`, provider tests, span names, and sandbox command behavior. ### Problem or motivation The Daytona provider needs a persistent shell for repeated command dispatch. The old advisory wrapper path does not reach that goal. It also adds cost and removes the session speed gain. ### Proposed solution Add a `useSessions` driver flag. Keep it off by default. Open one Daytona session per lease when the flag is on. Send each user command into that session. Read stdout and stderr from the session logs endpoint. Run each command in a subshell so `exit` does not stop the shell. Remove the advisory `bwrap` wrapper path and its lease metadata. Add session setup and teardown spans. Keep a hard delete on teardown. ### Alternatives considered Keep the advisory `bwrap` wrapper. That path does not give a real persistent session. It also keeps extra command overhead. Keep a one-shot fallback for user commands. That would weaken the session model and hide a missing session case. ### Roadmap alignment This work fits the `Cloud / Sandbox agents` milestone in `ROADMAP.md`. It also supports the control plane goal of safe remote sandbox execution. ### Additional context The handoff verification reported `tsc --noEmit` clean and 119 Daytona unit tests passing. The handoff also reported a clean host span allowlist test and five expected commits on the branch. The security review gate remains required before merge. ## What Changed - Added an opt-in persistent session model for the Daytona sandbox provider. - Routed user commands through `executeSessionCommand` when sessions are enabled. - Removed the advisory `bwrap` command wrapper path and the lease metadata it used. - Added session lifecycle spans and span allowlist coverage. - Documented the leak bound in `DIRECTORY-CONSTRAINT-FINDINGS.md`. ## Verification - `tsc --noEmit` clean for the Daytona plugin, per handoff verification. - Daytona unit suite passes, with 119 tests, per handoff verification. - Host span allowlist test passes, per handoff verification. ## Risks - Persistent sessions can leak if teardown fails. - Session logs must keep stdout and stderr separate. - The flag stays off by default to limit rollout risk. ## Model Used OpenAI GPT-5, Codex, tool use enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
d114c4925e |
feat(observability): give sandbox sync spans true wall-clock width (#10864)
## Thinking Path > - Paperclip keeps company work visible and governed. > - Sandbox agents run serial sync work across worker and host boundaries. > - The current span path hid real wall-clock time for that sync work. > - The host needs safe timestamps if it wants true span width. > - This pull request carries worker timestamps, validates them, and records the real duration. > - The benefit is clearer operator visibility for sandbox sync work. ## Linked Issues or Issue Description **Subsystem affected** Cross-cutting. This touches `packages/plugins`, `server`, and the Daytona plugin test surface. **Problem or motivation** Sandbox sync spans opened and closed in one host call. The native width stayed near zero, so the real time spent in serial round trips was hard to see. **Proposed solution** Carry worker start and end times across the span record protocol. Validate the pair at the host boundary. Record the host span with the true duration when the pair is safe. **Alternatives considered** Keep the numeric duration only. That keeps the data, but it does not widen the span and it does not show the real wall-clock time. **Roadmap alignment** This fits the `Cloud / Sandbox agents` and `Artifacts & Work Products` areas in `ROADMAP.md`. I found no other roadmap item that covers this span-width gap. **Additional context** The host allowlist stays narrow. Unknown names still map to `sandbox.provider.other`. Invalid timestamp pairs still fall back to the synchronous path. Related public PRs: none found. ## What Changed - Added optional `startTimeMs` and `endTimeMs` fields to the `span.record` protocol. - Captured start and end times in the worker tracer and sent them to the host. - Validated host timestamps with finite, ordered, bounded checks before span reconstruction. - Extended the host allowlist to the sandbox sync command names. - Wrapped each inbound sync round trip in its own named span. - Added tests for the worker path, host boundary, host recorder, and Daytona sync flow. ## Verification - `pnpm --filter @paperclipai/plugins-sdk test` - `pnpm --filter @paperclipai/server test` - `pnpm --filter @paperclipai/daytona-plugin test` - `pnpm --filter @paperclipai/server tsc --noEmit` still shows pre-existing `drizzle-orm` duplicate-declaration errors in this sandbox. The changed files do not touch those lines. - GitHub checks are green. - Greptile review is 5/5. - No open review threads remain. ## Risks - A bad timestamp pair can fall back to the synchronous path. - The host clock gate can reject spans if the pair is stale, reversed, or too large. - The new worker fields change the wire protocol, but the public plugin tracer contract stays the same. ## Model Used OpenAI GPT-5, tool-enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues with `Fixes: #` / `Closes #` / `Refs #` or described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
af6b32d82a |
feat(observability): add provider OTel spans - cache-hit flag, plugin tracer, Daytona pack/transfer spans (#10764)
## Thinking Path > - Paperclip uses pull requests to ship control-plane changes with review and audit. > - This change extends sandbox provider telemetry. > - The first PR added startup and exec spans. > - This PR adds provider spans for cache hits, plugin tracer flow, and Daytona file sync steps. > - The host must keep the trust boundary and reject unsafe span data. > - The result is fuller OTel coverage with bounded labels and safe parentage. ## Linked Issues or Issue Description **Problem or motivation** Sandbox provider telemetry lacks an explicit cache-hit signal, plugin span context, and file-sync pack and transfer spans. The current proxy for a cache hit is fragile. The plugin SDK also needs a safe tracer surface and a per-call parent context. **Proposed solution** Add an explicit `cacheHit` flag from the Daytona sandbox handle lookup. Expose `ctx.tracer` on the plugin context and pass a `traceparent` string with each call. Record provider spans on the host with allowlist clamping and capability checks. Wrap Daytona file sync pack and transfer steps in spans. **Alternatives considered** Keep the old `providerGetMs == 0` proxy. Reject that path because the cache-hit decision belongs at the handle lookup, not in a timing proxy. The host stays the trust boundary for worker span data. It clamps labels, rejects bad parent data, and gates the worker-to-host span RPC by capability. A security review completed before this PR opened. ## What Changed - Added an explicit `cacheHit` flag from the Daytona sandbox handle lookup. - Added `ctx.tracer` on the plugin context and a `traceparent` field in the per-call context. - Added host-side span recording with attribute clamping and capability checks. - Wrapped Daytona file sync pack and transfer steps in spans. - Added tests for the host trust boundary, the plugin tracer no-op path, and the Daytona span paths. ## Verification - `pnpm --filter @paperclipai/plugin-daytona exec vitest run` - `pnpm --filter @paperclipai/plugin-sdk exec vitest run` - `pnpm --filter @paperclipai/server exec vitest run plugin-host-services environment-execution-target instrumentation plugin-worker-manager` - `tsc --noEmit` for server, plugin-sdk, and adapter-utils ## Risks - Span data now crosses the worker boundary, so the host allowlist and capability gate must stay strict. - The `traceparent` path must stay valid and host-owned. - Daytona file sync spans must not add new round trips. ## Model Used OpenAI Codex, GPT-5, tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues with `Fixes:` / `Closes:` / `Refs:` or described the issue in the PR body - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |