Files
PaperClipAI/server/src/services/native-runtime/native-provider-artifact-capture.ts
T
DottaandClaude Opus 5.5 25b5cad7f9 fix(native-runtime): start first OpenCode turns from any saved launch-file copy instead of waiting on a stalled one (#22367)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agents in remote sandboxes must have a saved, verified copy
of their launch files before their first turn (#19622, #21287)
> - The first runs of a company share one copy of these files per
server. Other runs wait for it
> - In a restart chaos wave, the shared copy out of one sandbox stalled
for 330 s. Seven other runs waited silently for it. Two runs then failed
before their first turn with the cause `unknown`
> - The copy speed out of a sandbox differs from one sandbox to the
next. When the first copy failed, the other sandboxes copied the same
314 MB in 15 to 24 s
> - This pull request lets every waiting run use the first copy that any
run saves, starts a second copy when the first one is slow, and names
the cause of a stalled copy
> - The benefit is a shorter cold start for the first runs of a company,
no failed run when another run already saved the files, and a named
cause instead of `unknown`

## Linked Issues or Issue Description

Refs #19622
Refs #21287

**What happened?**
A restart chaos wave started 15 OpenCode runs on new Daytona sandboxes
(1 vCPU) in a new company. The first run (A) started the shared copy of
the 314 MB launch files out of its sandbox. The copy did not finish. The
plugin call timed out after 330 s, and run A failed before its first
turn. The other runs waited for that copy without any run event, so the
task looked dead for 284 to 343 s. When run A failed, every waiting run
started its own copy at the same time. Five copies finished in 15 to 24
s. One copy (run B) stalled again and failed after 330 s, although a
saved copy existed after 15 s. One copy (run C) took 5 min and started
its turn 10 min after the run started, although it could have used the
saved copy 5 min earlier. Both failures had the technical cause
`unknown`. Each failure also logged "native result could not be marked
workspace_failed; the reconciliation loop will retry persisted results",
but the run had no result, so there was nothing to retry.

**Expected behavior**
A run uses the first saved copy as soon as one exists. A slow copy does
not hold all first runs. A stalled copy out of the sandbox has a named
cause. A run with no result does not log a result-finalization failure.

**Steps to reproduce**
1. Use a company with remote OpenCode agents on Daytona sandboxes with
durable storage and no saved launch-file copy.
2. Start 8 or more runs together on new sandboxes.
3. Make the copy out of the first sandbox stall (for example, a slow
sandbox link).
4. See that all other runs wait for the full plugin timeout, and see the
failures with the cause `unknown`.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- `shareNativeProviderArtifactCapture`
(`native-provider-artifact-capture.ts`):
- The first capture that saves a ready copy ends the wait for every run.
A run whose own capture is still copying uses the saved copy and aborts
its own copy.
- A run whose own capture fails uses a saved copy when one exists, and
fails only when none exists.
- When the running capture is slower than 90 s, one waiting run starts
one more capture from its own sandbox. At most two captures run at once.
A capture keeps its slot until its copy out of the sandbox ends.
- The start decision is owned by one `SharedArtifactCapture` coordinator
(`tryStart()` returns a release lease).
- When the shared capture fails, one waiting run starts the next
capture. The other runs keep waiting for it. Before, every waiting run
started its own copy at the same time.
- The capture callback now gets its own abort signal, so a superseded
copy stops before upload.
- A failed copy out of the sandbox is now
`NativeProviderArtifactExportError` with the code
`native_provider_artifacts_export_timeout` (plugin call timed out) or
`native_provider_artifacts_export_failed`. The run's technical cause
shows this code instead of `unknown`. A timeout shows "Setting up the
computer took too long ...".
- `runtime-execution.ts`: do not try to mark the workspace as failed on
a native result when the run has no result. Before, this always threw
`native_finalization_missing` and logged a false "will retry" warning.
- `doc/opencode-cold-recovery.md`: describe the new sharing, the second
capture and the new cause codes.

## Verification

- Red/green: with the old `native-provider-artifact-capture.ts`, these 5
new tests fail. With this change, they pass:
- `native-provider-artifact-gate.test.ts`: "starts a run's turn from
another run's saved artifact while its own capture is still copying",
"uses a saved artifact when the run's own capture fails after another
run saved one", "starts one more capture when the shared capture runs
past the hedge delay, not one per waiting run", "does not start a
capture per waiting run when the shared capture fails", "names a stalled
copy out of the sandbox instead of an unknown cause".
- Red/green: `heartbeat-native-predispatch-retry.test.ts` "continues on
a fresh ... sandbox when the provider artifacts fail to save before any
turn" now also asserts that no "could not be marked workspace_failed"
warning is logged. It fails before the `runtime-execution.ts` change and
passes after it.
- `cd server && npx vitest run
src/services/native-runtime/native-provider-artifact-gate.test.ts
src/services/native-runtime/native-provider-artifact-capture-confinement.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts
src/services/native-runtime/native-provider-artifact-capture.test.ts
src/services/heartbeat/runtime-execution.test.ts
src/services/task-interruption-presentation.test.ts
src/__tests__/heartbeat-native-predispatch-retry.test.ts`: all pass.
- `npx tsc --noEmit -p server`: pass.

## Risks

- A slow first capture can now cause one more 314 MB copy after 90 s. At
most two copies run at once for one company and manifest. A run whose
saved copy cannot be reused (for example, an ownership fence mismatch)
starts its own copy at once. Before, a failed burst started one copy per
waiting run at once, which was more.
- A superseded copy never uploads, but the provider call that copies
files out of the sandbox continues until it ends or times out. Its local
temporary directory is removed when it ends.
- Sharing is still per server. Two servers can still copy the same
files.
- No change to the rule from #19622: a verified saved copy still exists
before the first turn.
- No runner or image change. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Default context
window; extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 21:11:30 -06:00

323 lines
12 KiB
TypeScript

import { logger } from "../../middleware/logger.js";
import { randomUUID } from "node:crypto";
import { mkdtemp, mkdir } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join, dirname, posix } from "node:path";
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
import type { NativeProviderArtifactSource } from "./native-provider-artifacts.js";
import type { OpenCodeLaunchArtifacts } from "./remote-opencode-artifacts.js";
import type { NativeExecutionContext } from "./native-execution-context.js";
import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js";
import { NativeProviderArtifactTransferTimeoutError } from "./native-provider-artifacts.js";
import { isPluginRpcTimeout } from "./remote-runner-recovery-probe.js";
const names = ["nodeCommand", "opencodeProxy", "opencodeExecutable"] as const;
export const NATIVE_PROVIDER_ARTIFACT_PUBLISH_ATTEMPTS = 3;
export const NATIVE_PROVIDER_ARTIFACT_CAPTURE_HEDGE_MS = 90_000;
type Latch = { promise: Promise<void>; resolve: () => void };
function latch(): Latch {
let resolve!: () => void;
const promise = new Promise<void>(done => { resolve = done; });
return { promise, resolve };
}
export type ShareNativeProviderArtifactCaptureInput<T> = {
key: string;
signal: AbortSignal;
reuse: () => Promise<T | null>;
capture: (signal: AbortSignal, hold: (work: Promise<unknown>) => void) => Promise<T>;
hedgeAfterMs?: number;
};
const MAX_CONCURRENT_CAPTURES = 2;
class SharedArtifactCapture {
private readonly state = { saved: false, running: 0, lastStartedAtMs: 0 };
private changed = latch();
constructor(private readonly key: string) {}
get saved() {
return this.state.saved;
}
tryStart(hedgeAfterMs: number, options: { afterSavedMiss: boolean }): (() => void) | null {
if (this.state.running >= MAX_CONCURRENT_CAPTURES) return null;
if (!options.afterSavedMiss && this.state.running > 0 && this.state.lastStartedAtMs + hedgeAfterMs > Date.now()) return null;
this.state.running += 1;
this.state.lastStartedAtMs = Date.now();
let released = false;
return () => {
if (released) return;
released = true;
this.state.running -= 1;
this.notify();
if (this.state.running === 0 && sharedCaptures.get(this.key) === this) sharedCaptures.delete(this.key);
};
}
async waitForChange(hedgeAfterMs: number, stopped: Promise<never>, options: { afterSavedMiss: boolean }) {
let timer: ReturnType<typeof setTimeout> | undefined;
const waits: Promise<unknown>[] = [this.changed.promise, stopped];
const hedgeInMs = this.state.lastStartedAtMs + hedgeAfterMs - Date.now();
if (!options.afterSavedMiss && this.state.running < MAX_CONCURRENT_CAPTURES && hedgeInMs > 0) {
waits.push(new Promise<void>(resolve => { timer = setTimeout(resolve, hedgeInMs); }));
}
try {
await Promise.race(waits);
} finally {
clearTimeout(timer);
}
}
async savedCopy() {
while (!this.state.saved) await this.changed.promise;
}
markSaved() {
this.state.saved = true;
this.notify();
}
private notify() {
const changed = this.changed;
this.changed = latch();
changed.resolve();
}
}
const sharedCaptures = new Map<string, SharedArtifactCapture>();
function sharedCaptureFor(key: string) {
let shared = sharedCaptures.get(key);
if (!shared) {
shared = new SharedArtifactCapture(key);
sharedCaptures.set(key, shared);
}
return shared;
}
function stopWith(signal: AbortSignal) {
let stop!: () => void;
const promise = new Promise<never>((_resolve, reject) => {
stop = () => reject(signal.reason);
signal.addEventListener("abort", stop, { once: true });
if (signal.aborted) stop();
});
promise.catch(() => undefined);
return { promise, dispose: () => signal.removeEventListener("abort", stop) };
}
export async function shareNativeProviderArtifactCapture<T>(input: ShareNativeProviderArtifactCaptureInput<T>): Promise<T> {
const hedgeAfterMs = input.hedgeAfterMs ?? NATIVE_PROVIDER_ARTIFACT_CAPTURE_HEDGE_MS;
const stopped = stopWith(input.signal);
try {
let afterSavedMiss = false;
for (;;) {
const shared = sharedCaptureFor(input.key);
const release = shared.tryStart(hedgeAfterMs, { afterSavedMiss });
if (release) return await captureOrReuse(input, shared, release, stopped.promise);
if (afterSavedMiss || !shared.saved) await shared.waitForChange(hedgeAfterMs, stopped.promise, { afterSavedMiss });
const reused = await input.reuse();
if (reused) {
shared.markSaved();
return reused;
}
afterSavedMiss ||= shared.saved;
}
} finally {
stopped.dispose();
}
}
async function captureOrReuse<T>(
input: ShareNativeProviderArtifactCaptureInput<T>,
shared: SharedArtifactCapture,
release: () => void,
stopped: Promise<never>,
): Promise<T> {
const superseded = new AbortController();
const held: Promise<unknown>[] = [];
const own = Promise.resolve().then(() => input.capture(AbortSignal.any([input.signal, superseded.signal]), work => held.push(work)));
void own.then(() => undefined, () => undefined).then(() => Promise.allSettled(held)).then(release);
try {
const first = await Promise.race([own.then(value => ({ value })), shared.savedCopy().then(() => null), stopped]);
if (first) {
shared.markSaved();
return first.value;
}
const reused = await input.reuse();
if (!reused) return await own;
superseded.abort(new Error("native_provider_artifacts_capture_superseded"));
return reused;
} catch (error) {
if (input.signal.aborted) throw error;
const reused = await input.reuse().catch(() => null);
if (!reused) throw error;
shared.markSaved();
return reused;
}
}
export class NativeProviderArtifactExportError extends Error {
readonly code: "native_provider_artifacts_export_timeout" | "native_provider_artifacts_export_failed";
constructor(cause: unknown) {
const code = isPluginRpcTimeout(cause) ? "native_provider_artifacts_export_timeout" : "native_provider_artifacts_export_failed";
super(`${code}: ${cause instanceof Error ? cause.message : String(cause)}`, { cause });
this.name = "NativeProviderArtifactExportError";
this.code = code;
}
}
// Copies the qualified launch files into a fresh export directory. The launch
// root sits under the sandbox account's home, outside the workspace remote dir,
// and sandbox providers confine bulk-export sources to that workspace. The copy
// refuses symlinks, so the confined export can only carry these regular files;
// Core still verifies every byte against the manifest digests after download.
const stageExportScript = [
"set -eu",
"umask 077",
'parent=$1; target=$2; shift 2',
'mkdir -p -- "$parent"',
'{ [ -d "$parent" ] && [ ! -L "$parent" ]; } || exit 41',
'mkdir -- "$target"',
"for name in nodeCommand opencodeProxy opencodeExecutable; do",
' source=$1; shift',
' { [ -f "$source" ] && [ ! -L "$source" ]; } || exit 42',
' cp -P -- "$source" "$target/$name"',
' { [ -f "$target/$name" ] && [ ! -L "$target/$name" ]; } || exit 43',
"done",
].join("\n");
const removeExportScript = 'set -eu\nrm -rf -- "$1"';
async function runExportCommand(runner: CommandManagedRuntimeRunner, script: string, args: string[]) {
const result = await runner.execute({
command: "sh",
args: ["-c", script, "sh", ...args],
bypassSession: true,
timeoutMs: 120_000,
});
if (result.exitCode !== 0 || result.timedOut) {
throw new Error("native_provider_artifacts_remote_unavailable");
}
}
export async function captureRemoteNativeProviderArtifacts<T>(
input: {
runner: CommandManagedRuntimeRunner;
/** Workspace remote dir: the only root a sandbox bulk export may read. */
remoteCwd: string;
launch: OpenCodeLaunchArtifacts;
manifest: NativeProviderArtifactSource["manifest"];
signal: AbortSignal;
context?: NativeExecutionContext;
hold?: (work: Promise<unknown>) => void;
},
publish: (source: NativeProviderArtifactSource) => Promise<T>,
): Promise<T> {
const root = await mkdtemp(join(tmpdir(), "paperclip-provider-launch-"));
let ownedByOperation = false;
try {
const exportParent = posix.join(
input.remoteCwd,
".paperclip-runtime",
"paperclip-runner",
"provider-artifact-export",
);
const exportRoot = posix.join(exportParent, randomUUID());
const files = await Promise.all(
names.map(
async (name) => {
const targetPath = join(
root,
input.manifest.payload.artifacts[name].path,
);
await mkdir(dirname(targetPath), { recursive: true, mode: 0o700 });
return {
sourcePath: posix.join(exportRoot, name),
targetPath,
kind: "file" as const,
mode: 0o500,
};
},
),
);
input.signal.throwIfAborted();
const settled = input.context?.trackWorker();
ownedByOperation = true;
const operation = Promise.resolve()
.then(async () => {
input.signal.throwIfAborted();
input.context?.assertOpen();
try {
await runExportCommand(input.runner, stageExportScript, [
exportParent,
exportRoot,
...names.map((name) => input.launch.paths[name]),
]);
input.signal.throwIfAborted();
input.context?.assertOpen();
await input.runner.syncOut!([
{ operationId: `provider-artifacts-${randomUUID()}`, files },
]).catch((error: unknown) => {
throw new NativeProviderArtifactExportError(error);
});
} finally {
// The export copy is scratch under the excluded runtime dir. A failed
// removal must not replace the transfer outcome; it is logged instead.
await runExportCommand(input.runner, removeExportScript, [exportRoot]).catch((err) => {
logger.warn({ err }, "native provider artifact export cleanup failed");
});
}
for (let attempt = 1; ; attempt++) {
input.signal.throwIfAborted();
input.context?.assertOpen();
try {
return await publish({
sourceRoot: root,
manifest: input.manifest,
launchRoot: input.launch.root,
accountUid: input.launch.accountUid,
});
} catch (error) {
if (!(error instanceof NativeProviderArtifactTransferTimeoutError) || attempt >= NATIVE_PROVIDER_ARTIFACT_PUBLISH_ATTEMPTS) throw error;
logger.warn({ err: error, attempt }, "native provider artifact transfer timed out; retrying the transfer");
}
}
})
.finally(async () => {
try {
await removeNativeHarnessBackup(root);
} catch (err) {
logger.warn(
{ err },
"native provider artifact capture cleanup failed",
);
throw err;
} finally {
settled?.();
}
});
void operation.catch(() => undefined);
input.hold?.(operation);
let abort!: () => void;
const stopped = new Promise<never>((_resolve, reject) => {
abort = () => reject(input.signal.reason);
input.signal.addEventListener("abort", abort, { once: true });
if (input.signal.aborted) abort();
});
try {
return await Promise.race([
input.context ? input.context.wait(() => operation) : operation,
stopped,
]);
} finally {
input.signal.removeEventListener("abort", abort);
}
} finally {
if (!ownedByOperation) await removeNativeHarnessBackup(root);
}
}