mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-22 00:30:19 +02:00
## Thinking Path > - Paperclip is an open-source AI-agent management platform; agents run tasks inside sandboxed environments (Daytona, Kubernetes, E2B, etc.) > - The control-plane ↔ sandbox file-transfer path flows through the `environmentExecute` seam in `protocol.ts` — the only verb available to plugins — which forces a base64-over-exec chunked loop for every file move: workspace files, assets, Codex home sync > - This transport is correct and safe, but it bypasses provider-native bulk/streaming APIs (Daytona `uploadFiles`, K8s `FastUploadInterceptor` / volume mounts), leaving significant throughput on the table for large workspaces > - The right fix is an opt-in seam extension: providers with faster native transfer declare two optional verbs; providers that do not opt in stay on the existing fallback with zero code or behavior change required > - This PR adds the first layer of that extension — two optional verbs (`environmentSyncIn` / `environmentSyncOut`) in the plugin SDK, the runtime plumbing to prefer the native path for the two clean destroy-then-replace cases, and a doc for the contract > - The core correctness invariant is byte-identical fallback: if no provider opts in, execution is exactly what ships today; `assertSyncOperationsConfined` enforces host-side path confinement for providers that do opt in > - No provider advertises the verbs yet → zero production behavior change; future PRs wire up Daytona and K8s providers against this contract ## Linked Issues or Issue Description No public GitHub issue exists for this feature. Description follows the `feature_request` issue template: **Subsystem affected:** packages/plugins — plugin system; packages/adapter-utils — adapter runtime; server/ — EnvironmentRuntimeService **Problem or motivation:** Sandbox file transfers currently always use a base64-over-exec chunked loop regardless of what the underlying provider supports. For workspaces larger than a few MB this becomes the dominant wall-clock cost of every sandbox run, and it bypasses bulk/stream APIs that providers like Daytona already expose natively. **Proposed solution:** Add two optional, opt-in plugin hooks — `onEnvironmentSyncIn` / `onEnvironmentSyncOut` — to the plugin SDK. When a provider defines both hooks and both are advertised via the existing `supportedMethods` negotiation, the runtime prefers the native path for the two clean destroy-then-replace transfer cases; all other cases fall back to the existing byte-identical base64 transport. **Alternatives considered:** An unconditional verb would require every provider to implement or stub the verb. The opt-in / `METHOD_NOT_IMPLEMENTED` pattern (already used by `environmentExecute`) preserves backward compatibility with zero provider changes required. **Roadmap alignment:** Consistent with the ✅ "Cloud / Sandbox agents" and ✅ "Plugin system" milestones; extends the plugin seam rather than adding control-plane-level logic. **Additional context:** Searched open pull requests and issues for duplicate sandbox file-sync / native-transfer work; none found. ## What Changed - **`packages/plugins/sdk`** - `protocol.ts`: two new optional `HostToWorkerMethods` — `environmentSyncIn` / `environmentSyncOut` — plus generic `SyncOperation`, `SyncFileMapping`, and `SyncOutcome` types - `define-plugin.ts`: optional `onEnvironmentSyncIn` / `onEnvironmentSyncOut` fields on `PluginDefinition`; worker advertises each verb only when its hook is defined (else `METHOD_NOT_IMPLEMENTED`, mirroring `environmentExecute`) - `worker-rpc-host.ts`: route new verbs to plugin hooks - `index.ts`: re-export new public types - **`packages/adapter-utils`** - `command-managed-runtime.ts`: expose optional `syncIn` / `syncOut` on `CommandManagedRuntimeRunner` (available only when both verbs are advertised); add `assertSyncOperationsConfined` host-side path-confinement guard - `sandbox-managed-runtime.ts`: `SandboxManagedRuntimeClient` gains optional `syncIn` / `syncOut`; orchestrator prefers native path for default-provision asset inbound and workspace-download-into-fresh-dir outbound; all other paths keep the existing base64 fallback - `sandbox-file-sync.test.ts` (new): 234-line characterization suite — native-opt-in branch, fallback branch, `assertSyncOperationsConfined` escape-path rejection, `followSymlinks` → tar `-h` - `command-managed-runtime.test.ts`: negotiation + native-sync + confinement tests - **`server/src/services/environment-runtime.ts`**: `EnvironmentRuntimeService` delegates to `syncIn` / `syncOut`, gated on advertised support - **`server/src/services/environment-execution-target.ts`**: minor typing fix alongside the new verbs - **`doc/plugins/SANDBOX_FILE_SYNC_HOOKS.md`** (new): documents the full contract — opt-in / no-op guarantee, operation ordering, provider-may-tar, atomicity, `followSymlinks`, secret modes (0600, no window), path confinement, `operationId` opacity, resource bounds, shell-quoting ## Verification ```bash # SDK suite pnpm --filter packages/plugins/sdk test # Adapter-utils suite (includes new sandbox-file-sync characterization tests) pnpm --filter packages/adapter-utils test # Expected: 255 pass / 4 skip # Type-check across affected packages pnpm --filter packages/plugins/sdk typecheck pnpm --filter packages/adapter-utils typecheck # Server changed-file spot check: cd server && npx tsc --noEmit --skipLibCheck 2>&1 | grep -E "environment-(runtime|execution-target)" | head -20 ``` Key behavioral invariant to spot-check: with no provider opting in (the current state), run any sandbox task and confirm file-transfer behavior is byte-for-byte identical to what the pre-PR code produces. The characterization tests assert this at the unit level. ## Risks - **Zero production risk today**: no provider advertises `environmentSyncIn` / `environmentSyncOut`, so the new code paths are unreachable in production; all real traffic stays on the existing base64 fallback - **Path confinement**: `assertSyncOperationsConfined` rejects any `targetPath` that escapes the declared root — this is the primary security boundary for future providers. The test suite covers escape-path rejection - **Atomicity**: the contract delegates atomicity to providers; the doc explicitly calls out that directory-level ops are not guaranteed atomic - **Secret transport**: credential assets (e.g., Codex `auth.json`, directory mappings) continue to use the existing tar path — they do not go through the new verbs in any current provider > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Provider: Anthropic Model: `claude-sonnet-4-6` (Claude Sonnet 4.6) Context window: 200 K tokens Capabilities: extended tool use, multi-file code generation, agentic reasoning via the Paperclip agent framework ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Harold Kim <harold@paperclip.ing> Co-authored-by: Paperclip <noreply@paperclip.ing>
192 lines
6.5 KiB
TypeScript
192 lines
6.5 KiB
TypeScript
import { createInterface } from "node:readline";
|
|
import { PassThrough } from "node:stream";
|
|
|
|
import { describe, expect, it } from "vitest";
|
|
|
|
import { definePlugin } from "../src/define-plugin.js";
|
|
import {
|
|
createRequest,
|
|
isJsonRpcResponse,
|
|
parseMessage,
|
|
PLUGIN_RPC_ERROR_CODES,
|
|
serializeMessage,
|
|
type JsonRpcResponse,
|
|
type PluginEnvironmentSyncInParams,
|
|
type PluginEnvironmentSyncOutParams,
|
|
type PluginEnvironmentSyncResult,
|
|
} from "../src/protocol.js";
|
|
import { startWorkerRpcHost } from "../src/worker-rpc-host.js";
|
|
|
|
const MANIFEST = {
|
|
id: "paperclip.sync-negotiation-test",
|
|
apiVersion: 1,
|
|
version: "1.0.0",
|
|
displayName: "Sync Negotiation Test",
|
|
description: "Test plugin",
|
|
author: "Paperclip",
|
|
categories: ["automation"],
|
|
capabilities: [],
|
|
entrypoints: {},
|
|
} as const;
|
|
|
|
function startTestWorker(plugin: ReturnType<typeof definePlugin>) {
|
|
const hostToWorker = new PassThrough();
|
|
const workerToHost = new PassThrough();
|
|
const hostReadline = createInterface({ input: workerToHost });
|
|
const pending = new Map<string, (response: JsonRpcResponse) => void>();
|
|
let nextRequestId = 1;
|
|
|
|
hostReadline.on("line", (line) => {
|
|
const message = parseMessage(line);
|
|
if (!isJsonRpcResponse(message)) return;
|
|
pending.get(String(message.id))?.(message);
|
|
pending.delete(String(message.id));
|
|
});
|
|
|
|
const worker = startWorkerRpcHost({ plugin, stdin: hostToWorker, stdout: workerToHost });
|
|
|
|
function callWorker<T = unknown>(method: string, params: unknown): Promise<T> {
|
|
const id = `host-${nextRequestId++}`;
|
|
const result = new Promise<T>((resolve, reject) => {
|
|
pending.set(id, (response) => {
|
|
if ("error" in response && response.error) {
|
|
reject(Object.assign(new Error(response.error.message), { code: response.error.code }));
|
|
return;
|
|
}
|
|
resolve((response as { result?: T }).result as T);
|
|
});
|
|
});
|
|
hostToWorker.write(serializeMessage(createRequest(method, params, id)));
|
|
return result;
|
|
}
|
|
|
|
function stop() {
|
|
worker.stop();
|
|
hostReadline.close();
|
|
hostToWorker.destroy();
|
|
workerToHost.destroy();
|
|
}
|
|
|
|
return { callWorker, stop };
|
|
}
|
|
|
|
describe("environment sync verb negotiation", () => {
|
|
it("advertises environmentSyncIn/environmentSyncOut only when the hooks are defined", async () => {
|
|
const withHooks = startTestWorker(
|
|
definePlugin({
|
|
async setup() {},
|
|
async onEnvironmentSyncIn(): Promise<PluginEnvironmentSyncResult> {
|
|
return { operations: [] };
|
|
},
|
|
async onEnvironmentSyncOut(): Promise<PluginEnvironmentSyncResult> {
|
|
return { operations: [] };
|
|
},
|
|
}),
|
|
);
|
|
try {
|
|
const result = await withHooks.callWorker<{ ok: boolean; supportedMethods: string[] }>(
|
|
"initialize",
|
|
{ manifest: MANIFEST, config: {}, databaseNamespace: null },
|
|
);
|
|
expect(result.supportedMethods).toContain("environmentSyncIn");
|
|
expect(result.supportedMethods).toContain("environmentSyncOut");
|
|
} finally {
|
|
withHooks.stop();
|
|
}
|
|
|
|
const withoutHooks = startTestWorker(definePlugin({ async setup() {} }));
|
|
try {
|
|
const result = await withoutHooks.callWorker<{ ok: boolean; supportedMethods: string[] }>(
|
|
"initialize",
|
|
{ manifest: MANIFEST, config: {}, databaseNamespace: null },
|
|
);
|
|
expect(result.supportedMethods).not.toContain("environmentSyncIn");
|
|
expect(result.supportedMethods).not.toContain("environmentSyncOut");
|
|
} finally {
|
|
withoutHooks.stop();
|
|
}
|
|
});
|
|
|
|
it("routes environmentSyncIn/environmentSyncOut to the hooks when defined", async () => {
|
|
const seen: string[] = [];
|
|
const worker = startTestWorker(
|
|
definePlugin({
|
|
async setup() {},
|
|
async onEnvironmentSyncIn(params): Promise<PluginEnvironmentSyncResult> {
|
|
seen.push("in");
|
|
return {
|
|
operations: params.operations.map((op) => ({
|
|
operationId: op.operationId,
|
|
filesTransferred: op.files.length,
|
|
bytesTransferred: 0,
|
|
})),
|
|
};
|
|
},
|
|
async onEnvironmentSyncOut(params): Promise<PluginEnvironmentSyncResult> {
|
|
seen.push("out");
|
|
return {
|
|
operations: params.operations.map((op) => ({
|
|
operationId: op.operationId,
|
|
filesTransferred: op.files.length,
|
|
bytesTransferred: 0,
|
|
})),
|
|
};
|
|
},
|
|
}),
|
|
);
|
|
try {
|
|
await worker.callWorker("initialize", { manifest: MANIFEST, config: {}, databaseNamespace: null });
|
|
const baseParams = {
|
|
driverKey: "sandbox",
|
|
companyId: "company",
|
|
environmentId: "env",
|
|
config: {},
|
|
lease: { providerLeaseId: "lease-1" },
|
|
};
|
|
const inParams: PluginEnvironmentSyncInParams = {
|
|
...baseParams,
|
|
operations: [
|
|
{ operationId: "op-a", files: [{ sourcePath: "/host/a", targetPath: "/remote/a", kind: "file" }] },
|
|
],
|
|
};
|
|
const inResult = await worker.callWorker<PluginEnvironmentSyncResult>("environmentSyncIn", inParams);
|
|
expect(inResult.operations[0]).toMatchObject({ operationId: "op-a", filesTransferred: 1 });
|
|
|
|
const outParams: PluginEnvironmentSyncOutParams = {
|
|
...baseParams,
|
|
operations: [
|
|
{ operationId: "op-b", files: [{ sourcePath: "/remote/b", targetPath: "/host/b", kind: "directory" }] },
|
|
],
|
|
};
|
|
const outResult = await worker.callWorker<PluginEnvironmentSyncResult>("environmentSyncOut", outParams);
|
|
expect(outResult.operations[0]).toMatchObject({ operationId: "op-b", filesTransferred: 1 });
|
|
expect(seen).toEqual(["in", "out"]);
|
|
} finally {
|
|
worker.stop();
|
|
}
|
|
});
|
|
|
|
it("throws METHOD_NOT_IMPLEMENTED when the sync hooks are absent", async () => {
|
|
const worker = startTestWorker(definePlugin({ async setup() {} }));
|
|
try {
|
|
await worker.callWorker("initialize", { manifest: MANIFEST, config: {}, databaseNamespace: null });
|
|
const params = {
|
|
driverKey: "sandbox",
|
|
companyId: "company",
|
|
environmentId: "env",
|
|
config: {},
|
|
lease: { providerLeaseId: "lease-1" },
|
|
operations: [],
|
|
};
|
|
await expect(worker.callWorker("environmentSyncIn", params)).rejects.toMatchObject({
|
|
code: PLUGIN_RPC_ERROR_CODES.METHOD_NOT_IMPLEMENTED,
|
|
});
|
|
await expect(worker.callWorker("environmentSyncOut", params)).rejects.toMatchObject({
|
|
code: PLUGIN_RPC_ERROR_CODES.METHOD_NOT_IMPLEMENTED,
|
|
});
|
|
} finally {
|
|
worker.stop();
|
|
}
|
|
});
|
|
});
|