mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 21:07:40 +01:00
feat: parallelize sandbox file-sync behind a provider opt-in capability (#11736)
## Thinking Path > - Paperclip runs AI agents through local and remote execution adapters. > - Sandbox providers move workspace and asset files before and after agent runs. > - Serial file transfers delay startup and teardown when several operations do not depend on each other. > - Providers need an opt-in contract so existing providers keep their serial behavior. > - This pull request adds a bounded scheduler and routes inbound and outbound sync operations through it. > - The benefit is shorter sandbox setup and teardown with stable errors, clear telemetry, and a safe opt-in path. ## Linked Issues or Issue Description **Subsystem affected** Cross-cutting (multiple of the above): packages/shared, packages/adapter-utils, packages/plugins, and server. **Problem or motivation** Sandbox sync processes the workspace, assets, and referenced projects in series. This adds avoidable wait time to agent startup and teardown. **Proposed solution** Add a fail-closed provider capability named concurrentSyncOperations. Use a bounded scheduler with a limit of four operations. Preserve operation order for error reporting. Keep non-opted-in providers on the serial path. **Alternatives considered** Increase the serial transfer speed or add provider-specific schedulers. Those options do not provide one shared contract or stable behavior across providers. **Roadmap alignment** ROADMAP.md lists cloud and sandbox agents as a product area. This change improves sandbox execution without changing the control-plane contract. **Additional context** The Daytona provider opts in. Board trials on this commit showed overlap for inbound sync and outbound restore, with no referenced-project staging failures. ## What Changed - Add the concurrentSyncOperations sandbox capability and fail-closed parsing. - Add a bounded settle-all scheduler with stable input-order errors. - Parallelize inbound workspace, asset, and referenced-project sync operations when the provider opts in. - Parallelize outbound workspace and asset restore operations when the provider opts in. - Surface referenced-project failure text in run logs and server telemetry. - Add Daytona sync spans and the capability declaration. - Preserve in-flight upload scratch tarballs during workspace wipe. - Add unit and regression tests for the scheduler, coordinators, provider behavior, telemetry, and wipe race. ## Verification - Run the adapter-utils and server type checks. - Run the targeted adapter-utils, server, and Daytona test suites. - Run the full automated sweep. - Review six cold Daytona trials, with three serial and three parallel runs. - Confirm that parallel trials show inbound overlap and outbound restore overlap. - Confirm that providers without the capability keep serial behavior. ## Risks - Providers must opt in only when their file operations can run safely at the same time. - A provider that declares the capability incorrectly can expose transfer races. - The scheduler keeps a limit of four to bound resource use. - Providers without the capability keep the prior serial behavior. ## Model Used OpenAI GPT-5 in the Codex runtime. The model used tool calls, code inspection, and GitHub workflow support. The model did not author the implementation commits. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub #NNN / github.com/paperclipai/paperclip URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
a5f3ac6b48
commit
233be4b36c
30 files changed
+3032
-478
No files matched your search
@@ -313,18 +313,29 @@ describe("sandbox provider capability declaration validators", () => {
|
||||
expect(rejected.success).toBe(false);
|
||||
});
|
||||
|
||||
it("test_removed_concurrency_capabilities_are_rejected_as_unknown_keys", () => {
|
||||
// The concurrency flags left the public contract because no runtime path
|
||||
// enforced them. The strict schema now rejects them, so a manifest cannot
|
||||
// declare a capability the host does not honor.
|
||||
for (const key of ["concurrentSyncAndExec", "concurrentSyncOperations"]) {
|
||||
const rejected = pluginManifestV1Schema.safeParse(
|
||||
buildSandboxProviderManifest({
|
||||
sandboxCapabilities: { [key]: true },
|
||||
}),
|
||||
);
|
||||
expect(rejected.success).toBe(false);
|
||||
}
|
||||
it("test_manifest_accepts_concurrent_sync_operations_capability", () => {
|
||||
// A provider opts in to parallel bidirectional file sync with this key. The
|
||||
// strict schema accepts it and keeps the declared value.
|
||||
const parsed = pluginManifestV1Schema.parse(
|
||||
buildSandboxProviderManifest({
|
||||
sandboxCapabilities: { concurrentSyncOperations: true },
|
||||
}),
|
||||
);
|
||||
|
||||
expect(parsed.environmentDrivers?.[0]?.sandboxCapabilities).toEqual({
|
||||
concurrentSyncOperations: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("test_manifest_rejects_unknown_sync_concurrency_capability_key", () => {
|
||||
// A neighboring but unknown concurrency key must fail validation, not drop
|
||||
// silently. The strict schema rejects a capability the host does not honor.
|
||||
const rejected = pluginManifestV1Schema.safeParse(
|
||||
buildSandboxProviderManifest({
|
||||
sandboxCapabilities: { concurrentSyncAndExec: true },
|
||||
}),
|
||||
);
|
||||
expect(rejected.success).toBe(false);
|
||||
});
|
||||
|
||||
it("test_supports_reusable_leases_compat_maps_to_reusable_leases", () => {
|
||||
|
||||
Reference in new issue
Block a user