mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - OpenCode agents in remote sandboxes must have a saved, verified copy of their launch files before their first turn (#19622, #21287) > - The first runs of a company share one copy of these files per server. Other runs wait for it > - In a restart chaos wave, the shared copy out of one sandbox stalled for 330 s. Seven other runs waited silently for it. Two runs then failed before their first turn with the cause `unknown` > - The copy speed out of a sandbox differs from one sandbox to the next. When the first copy failed, the other sandboxes copied the same 314 MB in 15 to 24 s > - This pull request lets every waiting run use the first copy that any run saves, starts a second copy when the first one is slow, and names the cause of a stalled copy > - The benefit is a shorter cold start for the first runs of a company, no failed run when another run already saved the files, and a named cause instead of `unknown` ## Linked Issues or Issue Description Refs #19622 Refs #21287 **What happened?** A restart chaos wave started 15 OpenCode runs on new Daytona sandboxes (1 vCPU) in a new company. The first run (A) started the shared copy of the 314 MB launch files out of its sandbox. The copy did not finish. The plugin call timed out after 330 s, and run A failed before its first turn. The other runs waited for that copy without any run event, so the task looked dead for 284 to 343 s. When run A failed, every waiting run started its own copy at the same time. Five copies finished in 15 to 24 s. One copy (run B) stalled again and failed after 330 s, although a saved copy existed after 15 s. One copy (run C) took 5 min and started its turn 10 min after the run started, although it could have used the saved copy 5 min earlier. Both failures had the technical cause `unknown`. Each failure also logged "native result could not be marked workspace_failed; the reconciliation loop will retry persisted results", but the run had no result, so there was nothing to retry. **Expected behavior** A run uses the first saved copy as soon as one exists. A slow copy does not hold all first runs. A stalled copy out of the sandbox has a named cause. A run with no result does not log a result-finalization failure. **Steps to reproduce** 1. Use a company with remote OpenCode agents on Daytona sandboxes with durable storage and no saved launch-file copy. 2. Start 8 or more runs together on new sandboxes. 3. Make the copy out of the first sandbox stall (for example, a slow sandbox link). 4. See that all other runs wait for the full plugin timeout, and see the failures with the cause `unknown`. **Agent adapter(s) involved** OpenCode (native runtime, remote sandbox). ## What Changed - `shareNativeProviderArtifactCapture` (`native-provider-artifact-capture.ts`): - The first capture that saves a ready copy ends the wait for every run. A run whose own capture is still copying uses the saved copy and aborts its own copy. - A run whose own capture fails uses a saved copy when one exists, and fails only when none exists. - When the running capture is slower than 90 s, one waiting run starts one more capture from its own sandbox. At most two captures run at once. A capture keeps its slot until its copy out of the sandbox ends. - The start decision is owned by one `SharedArtifactCapture` coordinator (`tryStart()` returns a release lease). - When the shared capture fails, one waiting run starts the next capture. The other runs keep waiting for it. Before, every waiting run started its own copy at the same time. - The capture callback now gets its own abort signal, so a superseded copy stops before upload. - A failed copy out of the sandbox is now `NativeProviderArtifactExportError` with the code `native_provider_artifacts_export_timeout` (plugin call timed out) or `native_provider_artifacts_export_failed`. The run's technical cause shows this code instead of `unknown`. A timeout shows "Setting up the computer took too long ...". - `runtime-execution.ts`: do not try to mark the workspace as failed on a native result when the run has no result. Before, this always threw `native_finalization_missing` and logged a false "will retry" warning. - `doc/opencode-cold-recovery.md`: describe the new sharing, the second capture and the new cause codes. ## Verification - Red/green: with the old `native-provider-artifact-capture.ts`, these 5 new tests fail. With this change, they pass: - `native-provider-artifact-gate.test.ts`: "starts a run's turn from another run's saved artifact while its own capture is still copying", "uses a saved artifact when the run's own capture fails after another run saved one", "starts one more capture when the shared capture runs past the hedge delay, not one per waiting run", "does not start a capture per waiting run when the shared capture fails", "names a stalled copy out of the sandbox instead of an unknown cause". - Red/green: `heartbeat-native-predispatch-retry.test.ts` "continues on a fresh ... sandbox when the provider artifacts fail to save before any turn" now also asserts that no "could not be marked workspace_failed" warning is logged. It fails before the `runtime-execution.ts` change and passes after it. - `cd server && npx vitest run src/services/native-runtime/native-provider-artifact-gate.test.ts src/services/native-runtime/native-provider-artifact-capture-confinement.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-harness-backup-storage.integration.test.ts src/services/native-runtime/native-provider-artifact-capture.test.ts src/services/heartbeat/runtime-execution.test.ts src/services/task-interruption-presentation.test.ts src/__tests__/heartbeat-native-predispatch-retry.test.ts`: all pass. - `npx tsc --noEmit -p server`: pass. ## Risks - A slow first capture can now cause one more 314 MB copy after 90 s. At most two copies run at once for one company and manifest. A run whose saved copy cannot be reused (for example, an ownership fence mismatch) starts its own copy at once. Before, a failed burst started one copy per waiting run at once, which was more. - A superseded copy never uploads, but the provider call that copies files out of the sandbox continues until it ends or times out. Its local temporary directory is removed when it ends. - Sharing is still per server. Two servers can still copy the same files. - No change to the rule from #19622: a verified saved copy still exists before the first turn. - No runner or image change. No migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, test runs). Default context window; extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
323 lines
12 KiB
TypeScript
323 lines
12 KiB
TypeScript
import { logger } from "../../middleware/logger.js";
|
|
import { randomUUID } from "node:crypto";
|
|
import { mkdtemp, mkdir } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import { join, dirname, posix } from "node:path";
|
|
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
|
|
import type { NativeProviderArtifactSource } from "./native-provider-artifacts.js";
|
|
import type { OpenCodeLaunchArtifacts } from "./remote-opencode-artifacts.js";
|
|
import type { NativeExecutionContext } from "./native-execution-context.js";
|
|
import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js";
|
|
import { NativeProviderArtifactTransferTimeoutError } from "./native-provider-artifacts.js";
|
|
import { isPluginRpcTimeout } from "./remote-runner-recovery-probe.js";
|
|
|
|
const names = ["nodeCommand", "opencodeProxy", "opencodeExecutable"] as const;
|
|
export const NATIVE_PROVIDER_ARTIFACT_PUBLISH_ATTEMPTS = 3;
|
|
export const NATIVE_PROVIDER_ARTIFACT_CAPTURE_HEDGE_MS = 90_000;
|
|
|
|
type Latch = { promise: Promise<void>; resolve: () => void };
|
|
|
|
function latch(): Latch {
|
|
let resolve!: () => void;
|
|
const promise = new Promise<void>(done => { resolve = done; });
|
|
return { promise, resolve };
|
|
}
|
|
|
|
export type ShareNativeProviderArtifactCaptureInput<T> = {
|
|
key: string;
|
|
signal: AbortSignal;
|
|
reuse: () => Promise<T | null>;
|
|
capture: (signal: AbortSignal, hold: (work: Promise<unknown>) => void) => Promise<T>;
|
|
hedgeAfterMs?: number;
|
|
};
|
|
|
|
const MAX_CONCURRENT_CAPTURES = 2;
|
|
|
|
class SharedArtifactCapture {
|
|
private readonly state = { saved: false, running: 0, lastStartedAtMs: 0 };
|
|
private changed = latch();
|
|
|
|
constructor(private readonly key: string) {}
|
|
|
|
get saved() {
|
|
return this.state.saved;
|
|
}
|
|
|
|
tryStart(hedgeAfterMs: number, options: { afterSavedMiss: boolean }): (() => void) | null {
|
|
if (this.state.running >= MAX_CONCURRENT_CAPTURES) return null;
|
|
if (!options.afterSavedMiss && this.state.running > 0 && this.state.lastStartedAtMs + hedgeAfterMs > Date.now()) return null;
|
|
this.state.running += 1;
|
|
this.state.lastStartedAtMs = Date.now();
|
|
let released = false;
|
|
return () => {
|
|
if (released) return;
|
|
released = true;
|
|
this.state.running -= 1;
|
|
this.notify();
|
|
if (this.state.running === 0 && sharedCaptures.get(this.key) === this) sharedCaptures.delete(this.key);
|
|
};
|
|
}
|
|
|
|
async waitForChange(hedgeAfterMs: number, stopped: Promise<never>, options: { afterSavedMiss: boolean }) {
|
|
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
const waits: Promise<unknown>[] = [this.changed.promise, stopped];
|
|
const hedgeInMs = this.state.lastStartedAtMs + hedgeAfterMs - Date.now();
|
|
if (!options.afterSavedMiss && this.state.running < MAX_CONCURRENT_CAPTURES && hedgeInMs > 0) {
|
|
waits.push(new Promise<void>(resolve => { timer = setTimeout(resolve, hedgeInMs); }));
|
|
}
|
|
try {
|
|
await Promise.race(waits);
|
|
} finally {
|
|
clearTimeout(timer);
|
|
}
|
|
}
|
|
|
|
async savedCopy() {
|
|
while (!this.state.saved) await this.changed.promise;
|
|
}
|
|
|
|
markSaved() {
|
|
this.state.saved = true;
|
|
this.notify();
|
|
}
|
|
|
|
private notify() {
|
|
const changed = this.changed;
|
|
this.changed = latch();
|
|
changed.resolve();
|
|
}
|
|
}
|
|
|
|
const sharedCaptures = new Map<string, SharedArtifactCapture>();
|
|
|
|
function sharedCaptureFor(key: string) {
|
|
let shared = sharedCaptures.get(key);
|
|
if (!shared) {
|
|
shared = new SharedArtifactCapture(key);
|
|
sharedCaptures.set(key, shared);
|
|
}
|
|
return shared;
|
|
}
|
|
|
|
function stopWith(signal: AbortSignal) {
|
|
let stop!: () => void;
|
|
const promise = new Promise<never>((_resolve, reject) => {
|
|
stop = () => reject(signal.reason);
|
|
signal.addEventListener("abort", stop, { once: true });
|
|
if (signal.aborted) stop();
|
|
});
|
|
promise.catch(() => undefined);
|
|
return { promise, dispose: () => signal.removeEventListener("abort", stop) };
|
|
}
|
|
|
|
export async function shareNativeProviderArtifactCapture<T>(input: ShareNativeProviderArtifactCaptureInput<T>): Promise<T> {
|
|
const hedgeAfterMs = input.hedgeAfterMs ?? NATIVE_PROVIDER_ARTIFACT_CAPTURE_HEDGE_MS;
|
|
const stopped = stopWith(input.signal);
|
|
try {
|
|
let afterSavedMiss = false;
|
|
for (;;) {
|
|
const shared = sharedCaptureFor(input.key);
|
|
const release = shared.tryStart(hedgeAfterMs, { afterSavedMiss });
|
|
if (release) return await captureOrReuse(input, shared, release, stopped.promise);
|
|
if (afterSavedMiss || !shared.saved) await shared.waitForChange(hedgeAfterMs, stopped.promise, { afterSavedMiss });
|
|
const reused = await input.reuse();
|
|
if (reused) {
|
|
shared.markSaved();
|
|
return reused;
|
|
}
|
|
afterSavedMiss ||= shared.saved;
|
|
}
|
|
} finally {
|
|
stopped.dispose();
|
|
}
|
|
}
|
|
|
|
async function captureOrReuse<T>(
|
|
input: ShareNativeProviderArtifactCaptureInput<T>,
|
|
shared: SharedArtifactCapture,
|
|
release: () => void,
|
|
stopped: Promise<never>,
|
|
): Promise<T> {
|
|
const superseded = new AbortController();
|
|
const held: Promise<unknown>[] = [];
|
|
const own = Promise.resolve().then(() => input.capture(AbortSignal.any([input.signal, superseded.signal]), work => held.push(work)));
|
|
void own.then(() => undefined, () => undefined).then(() => Promise.allSettled(held)).then(release);
|
|
try {
|
|
const first = await Promise.race([own.then(value => ({ value })), shared.savedCopy().then(() => null), stopped]);
|
|
if (first) {
|
|
shared.markSaved();
|
|
return first.value;
|
|
}
|
|
const reused = await input.reuse();
|
|
if (!reused) return await own;
|
|
superseded.abort(new Error("native_provider_artifacts_capture_superseded"));
|
|
return reused;
|
|
} catch (error) {
|
|
if (input.signal.aborted) throw error;
|
|
const reused = await input.reuse().catch(() => null);
|
|
if (!reused) throw error;
|
|
shared.markSaved();
|
|
return reused;
|
|
}
|
|
}
|
|
|
|
export class NativeProviderArtifactExportError extends Error {
|
|
readonly code: "native_provider_artifacts_export_timeout" | "native_provider_artifacts_export_failed";
|
|
constructor(cause: unknown) {
|
|
const code = isPluginRpcTimeout(cause) ? "native_provider_artifacts_export_timeout" : "native_provider_artifacts_export_failed";
|
|
super(`${code}: ${cause instanceof Error ? cause.message : String(cause)}`, { cause });
|
|
this.name = "NativeProviderArtifactExportError";
|
|
this.code = code;
|
|
}
|
|
}
|
|
|
|
// Copies the qualified launch files into a fresh export directory. The launch
|
|
// root sits under the sandbox account's home, outside the workspace remote dir,
|
|
// and sandbox providers confine bulk-export sources to that workspace. The copy
|
|
// refuses symlinks, so the confined export can only carry these regular files;
|
|
// Core still verifies every byte against the manifest digests after download.
|
|
const stageExportScript = [
|
|
"set -eu",
|
|
"umask 077",
|
|
'parent=$1; target=$2; shift 2',
|
|
'mkdir -p -- "$parent"',
|
|
'{ [ -d "$parent" ] && [ ! -L "$parent" ]; } || exit 41',
|
|
'mkdir -- "$target"',
|
|
"for name in nodeCommand opencodeProxy opencodeExecutable; do",
|
|
' source=$1; shift',
|
|
' { [ -f "$source" ] && [ ! -L "$source" ]; } || exit 42',
|
|
' cp -P -- "$source" "$target/$name"',
|
|
' { [ -f "$target/$name" ] && [ ! -L "$target/$name" ]; } || exit 43',
|
|
"done",
|
|
].join("\n");
|
|
const removeExportScript = 'set -eu\nrm -rf -- "$1"';
|
|
|
|
async function runExportCommand(runner: CommandManagedRuntimeRunner, script: string, args: string[]) {
|
|
const result = await runner.execute({
|
|
command: "sh",
|
|
args: ["-c", script, "sh", ...args],
|
|
bypassSession: true,
|
|
timeoutMs: 120_000,
|
|
});
|
|
if (result.exitCode !== 0 || result.timedOut) {
|
|
throw new Error("native_provider_artifacts_remote_unavailable");
|
|
}
|
|
}
|
|
|
|
export async function captureRemoteNativeProviderArtifacts<T>(
|
|
input: {
|
|
runner: CommandManagedRuntimeRunner;
|
|
/** Workspace remote dir: the only root a sandbox bulk export may read. */
|
|
remoteCwd: string;
|
|
launch: OpenCodeLaunchArtifacts;
|
|
manifest: NativeProviderArtifactSource["manifest"];
|
|
signal: AbortSignal;
|
|
context?: NativeExecutionContext;
|
|
hold?: (work: Promise<unknown>) => void;
|
|
},
|
|
publish: (source: NativeProviderArtifactSource) => Promise<T>,
|
|
): Promise<T> {
|
|
const root = await mkdtemp(join(tmpdir(), "paperclip-provider-launch-"));
|
|
let ownedByOperation = false;
|
|
try {
|
|
const exportParent = posix.join(
|
|
input.remoteCwd,
|
|
".paperclip-runtime",
|
|
"paperclip-runner",
|
|
"provider-artifact-export",
|
|
);
|
|
const exportRoot = posix.join(exportParent, randomUUID());
|
|
const files = await Promise.all(
|
|
names.map(
|
|
async (name) => {
|
|
const targetPath = join(
|
|
root,
|
|
input.manifest.payload.artifacts[name].path,
|
|
);
|
|
await mkdir(dirname(targetPath), { recursive: true, mode: 0o700 });
|
|
return {
|
|
sourcePath: posix.join(exportRoot, name),
|
|
targetPath,
|
|
kind: "file" as const,
|
|
mode: 0o500,
|
|
};
|
|
},
|
|
),
|
|
);
|
|
input.signal.throwIfAborted();
|
|
const settled = input.context?.trackWorker();
|
|
ownedByOperation = true;
|
|
const operation = Promise.resolve()
|
|
.then(async () => {
|
|
input.signal.throwIfAborted();
|
|
input.context?.assertOpen();
|
|
try {
|
|
await runExportCommand(input.runner, stageExportScript, [
|
|
exportParent,
|
|
exportRoot,
|
|
...names.map((name) => input.launch.paths[name]),
|
|
]);
|
|
input.signal.throwIfAborted();
|
|
input.context?.assertOpen();
|
|
await input.runner.syncOut!([
|
|
{ operationId: `provider-artifacts-${randomUUID()}`, files },
|
|
]).catch((error: unknown) => {
|
|
throw new NativeProviderArtifactExportError(error);
|
|
});
|
|
} finally {
|
|
// The export copy is scratch under the excluded runtime dir. A failed
|
|
// removal must not replace the transfer outcome; it is logged instead.
|
|
await runExportCommand(input.runner, removeExportScript, [exportRoot]).catch((err) => {
|
|
logger.warn({ err }, "native provider artifact export cleanup failed");
|
|
});
|
|
}
|
|
for (let attempt = 1; ; attempt++) {
|
|
input.signal.throwIfAborted();
|
|
input.context?.assertOpen();
|
|
try {
|
|
return await publish({
|
|
sourceRoot: root,
|
|
manifest: input.manifest,
|
|
launchRoot: input.launch.root,
|
|
accountUid: input.launch.accountUid,
|
|
});
|
|
} catch (error) {
|
|
if (!(error instanceof NativeProviderArtifactTransferTimeoutError) || attempt >= NATIVE_PROVIDER_ARTIFACT_PUBLISH_ATTEMPTS) throw error;
|
|
logger.warn({ err: error, attempt }, "native provider artifact transfer timed out; retrying the transfer");
|
|
}
|
|
}
|
|
})
|
|
.finally(async () => {
|
|
try {
|
|
await removeNativeHarnessBackup(root);
|
|
} catch (err) {
|
|
logger.warn(
|
|
{ err },
|
|
"native provider artifact capture cleanup failed",
|
|
);
|
|
throw err;
|
|
} finally {
|
|
settled?.();
|
|
}
|
|
});
|
|
void operation.catch(() => undefined);
|
|
input.hold?.(operation);
|
|
let abort!: () => void;
|
|
const stopped = new Promise<never>((_resolve, reject) => {
|
|
abort = () => reject(input.signal.reason);
|
|
input.signal.addEventListener("abort", abort, { once: true });
|
|
if (input.signal.aborted) abort();
|
|
});
|
|
try {
|
|
return await Promise.race([
|
|
input.context ? input.context.wait(() => operation) : operation,
|
|
stopped,
|
|
]);
|
|
} finally {
|
|
input.signal.removeEventListener("abort", abort);
|
|
}
|
|
} finally {
|
|
if (!ownedByOperation) await removeNativeHarnessBackup(root);
|
|
}
|
|
}
|