Files
PaperClipAI/packages/adapter-utils/src/remote-managed-runtime.ts
Devin FoleyandPaperclip 07e98d2b2c feat(adapter-utils): add observable sandbox sync progress (#8395)
## Thinking Path

> - Paperclip is the control plane for running AI-agent companies, so
long-running remote work needs to stay observable to human operators.
> - Cloud / sandbox agents are an active roadmap area, and their
workspace sync path is part of the runtime substrate every remote coding
run depends on.
> - In the sandbox and SSH execution-target flows, Paperclip logged that
sync had started, then often went silent for the full transfer window.
> - That made large remote syncs feel stalled and also hid a real
performance problem in the command-managed sandbox upload path.
> - The first part of this pull request threads a throttled
progress-reporting surface through the adapter execution-target stack so
sync and restore work can emit meaningful updates.
> - The second part fixes the command-managed sandbox transport itself:
it removes the old serial 32KB append bottleneck, but also falls back
away from the single-stream path when a provider-backed sandbox runner
cannot surface mid-flight stdin progress.
> - The result is that sandbox and SSH transfers are both faster and
more observable, including the live Daytona-style sandbox case that
previously only emitted `0%` and `100%`.

## Linked Issues or Issue Description

No public GitHub issue exists for this bug, so it is described inline
below following the bug report template.

### What happened

- Remote sandbox and SSH workspace syncs could spend a long time
transferring data while only logging a start line (`Syncing workspace
and runtime assets to sandbox environment`) and, at best, a terminal
line.
- In the command-managed sandbox path, the original upload
implementation also paid a large performance cost by appending base64
data in many small sequential remote writes (thousands of serial 32KB
round-trips on a large workspace).
- After the initial transport rewrite, live provider-backed sandbox runs
still only emitted `0%` and `100%` because the single-stream stdin RPC
buffered progress until completion.

### Expected behavior

- Long-running sandbox and SSH syncs should periodically report how much
of the transfer is complete (a percentage and/or MB transferred) so an
operator can tell the run is healthy and making progress rather than
stuck.
- The main sandbox upload path should not be artificially slow.
- A transfer that fails partway should leave an explicit failure marker
in the log rather than a dangling intermediate percentage.

### Steps to reproduce

1. Run an agent against a sandbox (command-managed) or SSH
(remote-managed) execution target with a non-trivial workspace.
2. Watch the run log during the workspace/runtime asset sync phase.
3. Observe that the log shows the sync start line and then stays silent
for the full transfer (live provider-backed sandbox runs only show `0%`
then `100%`).

### Paperclip version or commit

- Branch `PAPA-825-provide-status-updates-when-syncing-sandboxes` off
`master`.

### Deployment mode

- Self-hosted / local instance using sandbox (command-managed) and SSH
(remote-managed) execution targets, including provider-backed sandbox
runners.

## What Changed

- Added shared throttled runtime progress reporting and threaded
`onProgress` through the adapter execution-target surface and adapter
`execute.ts` entrypoints.
- Added sync and restore progress reporting for the command-managed
sandbox path and the SSH/remote-managed path, including git
import/export progress where totals are known.
- Reworked command-managed sandbox transfer behavior so uploads use the
faster single-stream path when appropriate, but fall back to chunked
progress-emitting writes when the runner cannot expose mid-stream stdin
progress.
- Marked provider-backed environment sandbox runners as not supporting
single-stream stdin progress so live sandbox runs emit meaningful
intermediate updates instead of only `0%` and `100%`.
- Emit an explicit terminal failure marker (`failed at NN% (x/y MB)`)
when an SSH/tar transfer rejects, so a failed sync no longer leaves a
dangling intermediate percentage in the log.
- Run the SSH sync/restore size estimate (local directory walk / remote
`du` probe) concurrently with the transfer instead of awaiting it before
opening the pipe, so progress instrumentation no longer adds startup
latency proportional to workspace file count.
- Added and extended focused regression coverage for runtime progress
throttling and the new failure marker, command-managed sandbox
transfers, sandbox orchestration, SSH transfer progress, and environment
execution-target wiring.

## Verification

- `pnpm exec vitest run
packages/adapter-utils/src/runtime-progress.test.ts
packages/adapter-utils/src/ssh-fixture.test.ts
packages/adapter-utils/src/command-managed-runtime.test.ts
packages/adapter-utils/src/sandbox-managed-runtime.test.ts`
- `pnpm exec vitest run
packages/adapter-utils/src/command-managed-runtime.test.ts
server/src/__tests__/environment-execution-target.test.ts`
- `npx tsc --noEmit` for `packages/adapter-utils`

## Risks

- The provider-backed sandbox fallback now prefers chunked
command-managed writes when progress hooks are active, so
small-to-medium uploads may trade some raw throughput for observable
intermediate progress on runtimes that cannot surface true mid-stream
stdin progress.
- Progress percentages on tar-based transfers still depend on estimates
in some cases, so operators may briefly see MB-only lines before the
estimate resolves, then near-final clamping before the terminal `100%`
line.
- This PR changes shared execution-target behavior used by multiple
adapters, so regressions would most likely appear in remote runtime
setup/teardown flows rather than in a single adapter.

## Model Used

- Initial implementation: OpenAI GPT-5.4 via Codex local agent
(`codex_local`), high reasoning mode.
- Observability follow-ups (failure marker, concurrent size estimate,
added tests): Claude Opus 4.8 via Claude Code (`claude_local`).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-06-20 13:03:42 -07:00

143 lines
4.4 KiB
TypeScript

import path from "node:path";
import {
type SshRemoteExecutionSpec,
prepareWorkspaceForSshExecution,
restoreWorkspaceFromSshExecution,
syncDirectoryToSsh,
} from "./ssh.js";
import { captureDirectorySnapshot } from "./workspace-restore-merge.js";
import type { RuntimeProgressSink } from "./runtime-progress.js";
export interface RemoteManagedRuntimeAsset {
key: string;
localDir: string;
followSymlinks?: boolean;
exclude?: string[];
}
export interface PreparedRemoteManagedRuntime {
spec: SshRemoteExecutionSpec;
workspaceLocalDir: string;
workspaceRemoteDir: string;
runtimeRootDir: string;
assetDirs: Record<string, string>;
restoreWorkspace(onProgress?: RuntimeProgressSink): Promise<void>;
}
function asObject(value: unknown): Record<string, unknown> {
return value && typeof value === "object" && !Array.isArray(value)
? (value as Record<string, unknown>)
: {};
}
function asString(value: unknown): string {
return typeof value === "string" ? value : "";
}
function asNumber(value: unknown): number {
return typeof value === "number" ? value : Number(value);
}
export function buildRemoteExecutionSessionIdentity(spec: SshRemoteExecutionSpec | null) {
if (!spec) return null;
return {
transport: "ssh",
host: spec.host,
port: spec.port,
username: spec.username,
remoteCwd: spec.remoteCwd,
} as const;
}
export function remoteExecutionSessionMatches(saved: unknown, current: SshRemoteExecutionSpec | null): boolean {
const currentIdentity = buildRemoteExecutionSessionIdentity(current);
if (!currentIdentity) return false;
const parsedSaved = asObject(saved);
return (
asString(parsedSaved.transport) === currentIdentity.transport &&
asString(parsedSaved.host) === currentIdentity.host &&
asNumber(parsedSaved.port) === currentIdentity.port &&
asString(parsedSaved.username) === currentIdentity.username &&
asString(parsedSaved.remoteCwd) === currentIdentity.remoteCwd
);
}
export async function prepareRemoteManagedRuntime(input: {
spec: SshRemoteExecutionSpec;
runId: string;
adapterKey: string;
workspaceLocalDir: string;
workspaceRemoteDir?: string;
assets?: RemoteManagedRuntimeAsset[];
// Upload progress sink. Threaded for the byte-counting transport rewrite; the
// child task wires it into the workspace/asset transfers.
onProgress?: RuntimeProgressSink;
}): Promise<PreparedRemoteManagedRuntime> {
const baseWorkspaceRemoteDir = input.workspaceRemoteDir ?? input.spec.remoteCwd;
const workspaceRemoteDir = path.posix.join(
baseWorkspaceRemoteDir,
".paperclip-runtime",
"runs",
input.runId,
"workspace",
);
const runtimeRootDir = path.posix.join(workspaceRemoteDir, ".paperclip-runtime", input.adapterKey);
const preparedWorkspace = await prepareWorkspaceForSshExecution({
spec: input.spec,
localDir: input.workspaceLocalDir,
remoteDir: workspaceRemoteDir,
onProgress: input.onProgress,
});
const restoreExclude = preparedWorkspace.gitBacked ? [".git", ".paperclip-runtime"] : [".paperclip-runtime"];
const baselineSnapshot = await captureDirectorySnapshot(input.workspaceLocalDir, {
exclude: restoreExclude,
});
const assetDirs: Record<string, string> = {};
try {
for (const asset of input.assets ?? []) {
const remoteDir = path.posix.join(runtimeRootDir, asset.key);
assetDirs[asset.key] = remoteDir;
await syncDirectoryToSsh({
spec: input.spec,
localDir: asset.localDir,
remoteDir,
followSymlinks: asset.followSymlinks,
exclude: asset.exclude,
onProgress: input.onProgress,
progressLabel: asset.key,
});
}
} catch (error) {
await restoreWorkspaceFromSshExecution({
spec: input.spec,
localDir: input.workspaceLocalDir,
remoteDir: workspaceRemoteDir,
baselineSnapshot,
restoreGitHistory: preparedWorkspace.gitBacked,
onProgress: input.onProgress,
});
throw error;
}
return {
spec: input.spec,
workspaceLocalDir: input.workspaceLocalDir,
workspaceRemoteDir,
runtimeRootDir,
assetDirs,
restoreWorkspace: async (onProgress?: RuntimeProgressSink) => {
await restoreWorkspaceFromSshExecution({
spec: input.spec,
localDir: input.workspaceLocalDir,
remoteDir: workspaceRemoteDir,
baselineSnapshot,
restoreGitHistory: preparedWorkspace.gitBacked,
onProgress,
});
},
};
}