mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
fix(native-runtime): start first OpenCode turns from any saved launch-file copy instead of waiting on a stalled one (#22367)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - OpenCode agents in remote sandboxes must have a saved, verified copy of their launch files before their first turn (#19622, #21287) > - The first runs of a company share one copy of these files per server. Other runs wait for it > - In a restart chaos wave, the shared copy out of one sandbox stalled for 330 s. Seven other runs waited silently for it. Two runs then failed before their first turn with the cause `unknown` > - The copy speed out of a sandbox differs from one sandbox to the next. When the first copy failed, the other sandboxes copied the same 314 MB in 15 to 24 s > - This pull request lets every waiting run use the first copy that any run saves, starts a second copy when the first one is slow, and names the cause of a stalled copy > - The benefit is a shorter cold start for the first runs of a company, no failed run when another run already saved the files, and a named cause instead of `unknown` ## Linked Issues or Issue Description Refs #19622 Refs #21287 **What happened?** A restart chaos wave started 15 OpenCode runs on new Daytona sandboxes (1 vCPU) in a new company. The first run (A) started the shared copy of the 314 MB launch files out of its sandbox. The copy did not finish. The plugin call timed out after 330 s, and run A failed before its first turn. The other runs waited for that copy without any run event, so the task looked dead for 284 to 343 s. When run A failed, every waiting run started its own copy at the same time. Five copies finished in 15 to 24 s. One copy (run B) stalled again and failed after 330 s, although a saved copy existed after 15 s. One copy (run C) took 5 min and started its turn 10 min after the run started, although it could have used the saved copy 5 min earlier. Both failures had the technical cause `unknown`. Each failure also logged "native result could not be marked workspace_failed; the reconciliation loop will retry persisted results", but the run had no result, so there was nothing to retry. **Expected behavior** A run uses the first saved copy as soon as one exists. A slow copy does not hold all first runs. A stalled copy out of the sandbox has a named cause. A run with no result does not log a result-finalization failure. **Steps to reproduce** 1. Use a company with remote OpenCode agents on Daytona sandboxes with durable storage and no saved launch-file copy. 2. Start 8 or more runs together on new sandboxes. 3. Make the copy out of the first sandbox stall (for example, a slow sandbox link). 4. See that all other runs wait for the full plugin timeout, and see the failures with the cause `unknown`. **Agent adapter(s) involved** OpenCode (native runtime, remote sandbox). ## What Changed - `shareNativeProviderArtifactCapture` (`native-provider-artifact-capture.ts`): - The first capture that saves a ready copy ends the wait for every run. A run whose own capture is still copying uses the saved copy and aborts its own copy. - A run whose own capture fails uses a saved copy when one exists, and fails only when none exists. - When the running capture is slower than 90 s, one waiting run starts one more capture from its own sandbox. At most two captures run at once. A capture keeps its slot until its copy out of the sandbox ends. - The start decision is owned by one `SharedArtifactCapture` coordinator (`tryStart()` returns a release lease). - When the shared capture fails, one waiting run starts the next capture. The other runs keep waiting for it. Before, every waiting run started its own copy at the same time. - The capture callback now gets its own abort signal, so a superseded copy stops before upload. - A failed copy out of the sandbox is now `NativeProviderArtifactExportError` with the code `native_provider_artifacts_export_timeout` (plugin call timed out) or `native_provider_artifacts_export_failed`. The run's technical cause shows this code instead of `unknown`. A timeout shows "Setting up the computer took too long ...". - `runtime-execution.ts`: do not try to mark the workspace as failed on a native result when the run has no result. Before, this always threw `native_finalization_missing` and logged a false "will retry" warning. - `doc/opencode-cold-recovery.md`: describe the new sharing, the second capture and the new cause codes. ## Verification - Red/green: with the old `native-provider-artifact-capture.ts`, these 5 new tests fail. With this change, they pass: - `native-provider-artifact-gate.test.ts`: "starts a run's turn from another run's saved artifact while its own capture is still copying", "uses a saved artifact when the run's own capture fails after another run saved one", "starts one more capture when the shared capture runs past the hedge delay, not one per waiting run", "does not start a capture per waiting run when the shared capture fails", "names a stalled copy out of the sandbox instead of an unknown cause". - Red/green: `heartbeat-native-predispatch-retry.test.ts` "continues on a fresh ... sandbox when the provider artifacts fail to save before any turn" now also asserts that no "could not be marked workspace_failed" warning is logged. It fails before the `runtime-execution.ts` change and passes after it. - `cd server && npx vitest run src/services/native-runtime/native-provider-artifact-gate.test.ts src/services/native-runtime/native-provider-artifact-capture-confinement.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-harness-backup-storage.integration.test.ts src/services/native-runtime/native-provider-artifact-capture.test.ts src/services/heartbeat/runtime-execution.test.ts src/services/task-interruption-presentation.test.ts src/__tests__/heartbeat-native-predispatch-retry.test.ts`: all pass. - `npx tsc --noEmit -p server`: pass. ## Risks - A slow first capture can now cause one more 314 MB copy after 90 s. At most two copies run at once for one company and manifest. A run whose saved copy cannot be reused (for example, an ownership fence mismatch) starts its own copy at once. Before, a failed burst started one copy per waiting run at once, which was more. - A superseded copy never uploads, but the provider call that copies files out of the sandbox continues until it ends or times out. Its local temporary directory is removed when it ends. - Sharing is still per server. Two servers can still copy the same files. - No change to the rule from #19622: a verified saved copy still exists before the first turn. - No runner or image change. No migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, test runs). Default context window; extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
19a7381398
commit
25b5cad7f9
7 files changed
+430
-44
No files matched your search
@@ -43,14 +43,28 @@ sandbox, and nothing is encrypted, uploaded or downloaded. A durable artifact
|
||||
therefore still exists before the first turn. Concurrent first admissions on
|
||||
one server with the same company and manifest share one in-flight capture. The
|
||||
other admissions wait for it, then reuse its ready object under their own
|
||||
ownership fence. If the shared capture fails, each waiting admission captures on
|
||||
its own. Admissions on different servers may still upload the same closure
|
||||
ownership fence. If the shared capture fails, one waiting admission starts the
|
||||
next capture and the others keep waiting for it. When the running capture has
|
||||
not finished after 90 seconds, one more waiting admission starts a capture from
|
||||
its own sandbox, because the copy speed out of a sandbox differs from one
|
||||
sandbox to the next. At most two captures run at once for one company and
|
||||
manifest; a capture counts until its copy out of the sandbox ends. A run that
|
||||
finds a saved object it cannot reuse starts its own capture as soon as a slot
|
||||
is free, without waiting for the 90 seconds. The first capture that saves a ready object ends the wait
|
||||
for every admission: an admission whose own capture is still copying reuses that
|
||||
object and abandons its capture, and an admission whose own capture fails
|
||||
reuses it too. An abandoned capture never uploads or publishes. Its copy out of
|
||||
the sandbox is not interrupted: that provider call runs until it ends or reaches
|
||||
its own timeout, and its local files are then removed. Admissions on different servers may still upload the same closure
|
||||
independently.
|
||||
|
||||
Each capture has the 512 MiB limit; a run can retain at most three pending
|
||||
uploads. There is no single deadline for the whole capture. Each network step
|
||||
has its own bound instead: each export command has 120 seconds, and the sandbox
|
||||
transfer keeps its provider's command timeouts. The upload and the download that
|
||||
transfer keeps its provider's command timeouts. When the copy out of the
|
||||
sandbox fails, the failure is `native_provider_artifacts_export_timeout` if the
|
||||
provider call timed out, and `native_provider_artifacts_export_failed`
|
||||
otherwise. The upload and the download that
|
||||
verifies it each have the larger of 120 seconds and the time for the object at
|
||||
256 KiB per second. When an upload or download step exceeds its bound, the
|
||||
failure is `native_provider_artifacts_transfer_timeout` and names the step. The
|
||||
|
||||
@@ -11,6 +11,7 @@ import type { NativeRunIdentity, NativeSession, NativeSessionBackend, PrpEvent }
|
||||
import type { ExecutionContinuationEnvelope } from "@paperclipai/shared";
|
||||
import { renderPaperclipWakePrompt } from "@paperclipai/adapter-utils/server-utils";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { heartbeatService, type HeartbeatServiceOptions } from "../services/heartbeat.js";
|
||||
import { instanceSettingsService } from "../services/instance-settings.js";
|
||||
import { closeIdleWarmNativeSessionsForRestart, scopedRunnerdStateRoot } from "../services/native-runtime/native-session-executor.js";
|
||||
@@ -378,9 +379,12 @@ describe.skipIf(!support.supported)("heartbeat native pre-dispatch failure retry
|
||||
const failure: AdmissionFailure = { remaining: 1, turnNeverIssued: true, environmentId: environment!.id, provider,
|
||||
providerLeaseId: `sandbox-${provider}`, leasePolicy, failedRuns: [], sessionRoots: [] };
|
||||
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db, { admissionFailure: failure, turnMessages }) });
|
||||
const warn = vi.spyOn(logger, "warn");
|
||||
try {
|
||||
const { issueId, runId } = await startTask(db, heartbeat);
|
||||
expect(failure.failedRuns).toEqual([runId]);
|
||||
expect(warn.mock.calls.filter(([, message]) => typeof message === "string" && message.includes("could not be marked workspace_failed")))
|
||||
.toEqual([]);
|
||||
expect(turnMessages.has(runId)).toBe(false);
|
||||
expect(existsSync(failure.sessionRoots[0]!)).toBe(false);
|
||||
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
@@ -416,6 +420,7 @@ describe.skipIf(!support.supported)("heartbeat native pre-dispatch failure retry
|
||||
const [settled] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
expect(await taskInterruptionPresentation(db, settled!)).toMatchObject({ state: "resolved", canRetry: false });
|
||||
} finally {
|
||||
warn.mockRestore();
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
await closeIdleWarmNativeSessionsForRestart();
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and(
|
||||
|
||||
@@ -1315,20 +1315,22 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
|
||||
}
|
||||
return { dispatched: true, adapterResult: acceptedResult };
|
||||
}
|
||||
try {
|
||||
await measureRunPhase("execution.native_finalize", () => finalizeNativeRun({
|
||||
db,
|
||||
runId: run.id,
|
||||
workspaceFinalizeStatus: "failed",
|
||||
}));
|
||||
await dispatchPendingNativeStatusWakeups({
|
||||
companyId: run.companyId,
|
||||
});
|
||||
} catch (finalizeErr) {
|
||||
logger.warn(
|
||||
{ err: finalizeErr, runId: run.id },
|
||||
"native result could not be marked workspace_failed; the reconciliation loop will retry persisted results",
|
||||
);
|
||||
if (proposedResult) {
|
||||
try {
|
||||
await measureRunPhase("execution.native_finalize", () => finalizeNativeRun({
|
||||
db,
|
||||
runId: run.id,
|
||||
workspaceFinalizeStatus: "failed",
|
||||
}));
|
||||
await dispatchPendingNativeStatusWakeups({
|
||||
companyId: run.companyId,
|
||||
});
|
||||
} catch (finalizeErr) {
|
||||
logger.warn(
|
||||
{ err: finalizeErr, runId: run.id },
|
||||
"native result could not be marked workspace_failed; the reconciliation loop will retry persisted results",
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
throw adapterErr;
|
||||
|
||||
@@ -10,7 +10,7 @@ export type FailedBeforeProviderTurnRecord = {
|
||||
proof: FailedBeforeProviderTurnProof; technicalCause?: string;
|
||||
};
|
||||
|
||||
const TRANSFER_TIMEOUT_CODE = "native_provider_artifacts_transfer_timeout";
|
||||
const SLOW_SETUP_CODES = new Set(["native_provider_artifacts_transfer_timeout", "native_provider_artifacts_export_timeout"]);
|
||||
|
||||
function technicalCauseCode(cause: unknown): string {
|
||||
const code = cause && typeof cause === "object" ? (cause as { code?: unknown }).code : undefined;
|
||||
@@ -35,7 +35,7 @@ export class NativeFailedBeforeProviderTurnError extends Error {
|
||||
readonly technicalCause: string;
|
||||
constructor(readonly providerLeaseId: string | null, cause: unknown) {
|
||||
const technicalCause = technicalCauseCode(cause);
|
||||
super(technicalCause === TRANSFER_TIMEOUT_CODE
|
||||
super(SLOW_SETUP_CODES.has(technicalCause)
|
||||
? "Setting up the computer took too long, so the task is continuing on a fresh one."
|
||||
: "The computer could not be set up before the agent started, so the task is continuing on a fresh one.", { cause });
|
||||
this.name = "NativeFailedBeforeProviderTurnError";
|
||||
|
||||
@@ -9,39 +9,165 @@ import type { OpenCodeLaunchArtifacts } from "./remote-opencode-artifacts.js";
|
||||
import type { NativeExecutionContext } from "./native-execution-context.js";
|
||||
import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js";
|
||||
import { NativeProviderArtifactTransferTimeoutError } from "./native-provider-artifacts.js";
|
||||
import { isPluginRpcTimeout } from "./remote-runner-recovery-probe.js";
|
||||
|
||||
const names = ["nodeCommand", "opencodeProxy", "opencodeExecutable"] as const;
|
||||
export const NATIVE_PROVIDER_ARTIFACT_PUBLISH_ATTEMPTS = 3;
|
||||
const sharedCaptures = new Map<string, Promise<unknown>>();
|
||||
export const NATIVE_PROVIDER_ARTIFACT_CAPTURE_HEDGE_MS = 90_000;
|
||||
|
||||
export async function shareNativeProviderArtifactCapture<T>(input: {
|
||||
type Latch = { promise: Promise<void>; resolve: () => void };
|
||||
|
||||
function latch(): Latch {
|
||||
let resolve!: () => void;
|
||||
const promise = new Promise<void>(done => { resolve = done; });
|
||||
return { promise, resolve };
|
||||
}
|
||||
|
||||
export type ShareNativeProviderArtifactCaptureInput<T> = {
|
||||
key: string;
|
||||
signal: AbortSignal;
|
||||
reuse: () => Promise<T | null>;
|
||||
capture: () => Promise<T>;
|
||||
}): Promise<T> {
|
||||
const inFlight = sharedCaptures.get(input.key);
|
||||
if (inFlight) {
|
||||
let stop!: () => void;
|
||||
const stopped = new Promise<never>((_resolve, reject) => {
|
||||
stop = () => reject(input.signal.reason);
|
||||
input.signal.addEventListener("abort", stop, { once: true });
|
||||
if (input.signal.aborted) stop();
|
||||
});
|
||||
capture: (signal: AbortSignal, hold: (work: Promise<unknown>) => void) => Promise<T>;
|
||||
hedgeAfterMs?: number;
|
||||
};
|
||||
|
||||
const MAX_CONCURRENT_CAPTURES = 2;
|
||||
|
||||
class SharedArtifactCapture {
|
||||
private readonly state = { saved: false, running: 0, lastStartedAtMs: 0 };
|
||||
private changed = latch();
|
||||
|
||||
constructor(private readonly key: string) {}
|
||||
|
||||
get saved() {
|
||||
return this.state.saved;
|
||||
}
|
||||
|
||||
tryStart(hedgeAfterMs: number, options: { afterSavedMiss: boolean }): (() => void) | null {
|
||||
if (this.state.running >= MAX_CONCURRENT_CAPTURES) return null;
|
||||
if (!options.afterSavedMiss && this.state.running > 0 && this.state.lastStartedAtMs + hedgeAfterMs > Date.now()) return null;
|
||||
this.state.running += 1;
|
||||
this.state.lastStartedAtMs = Date.now();
|
||||
let released = false;
|
||||
return () => {
|
||||
if (released) return;
|
||||
released = true;
|
||||
this.state.running -= 1;
|
||||
this.notify();
|
||||
if (this.state.running === 0 && sharedCaptures.get(this.key) === this) sharedCaptures.delete(this.key);
|
||||
};
|
||||
}
|
||||
|
||||
async waitForChange(hedgeAfterMs: number, stopped: Promise<never>, options: { afterSavedMiss: boolean }) {
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
const waits: Promise<unknown>[] = [this.changed.promise, stopped];
|
||||
const hedgeInMs = this.state.lastStartedAtMs + hedgeAfterMs - Date.now();
|
||||
if (!options.afterSavedMiss && this.state.running < MAX_CONCURRENT_CAPTURES && hedgeInMs > 0) {
|
||||
waits.push(new Promise<void>(resolve => { timer = setTimeout(resolve, hedgeInMs); }));
|
||||
}
|
||||
try {
|
||||
await Promise.race([inFlight.catch(() => undefined), stopped]);
|
||||
await Promise.race(waits);
|
||||
} finally {
|
||||
input.signal.removeEventListener("abort", stop);
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
async savedCopy() {
|
||||
while (!this.state.saved) await this.changed.promise;
|
||||
}
|
||||
|
||||
markSaved() {
|
||||
this.state.saved = true;
|
||||
this.notify();
|
||||
}
|
||||
|
||||
private notify() {
|
||||
const changed = this.changed;
|
||||
this.changed = latch();
|
||||
changed.resolve();
|
||||
}
|
||||
}
|
||||
|
||||
const sharedCaptures = new Map<string, SharedArtifactCapture>();
|
||||
|
||||
function sharedCaptureFor(key: string) {
|
||||
let shared = sharedCaptures.get(key);
|
||||
if (!shared) {
|
||||
shared = new SharedArtifactCapture(key);
|
||||
sharedCaptures.set(key, shared);
|
||||
}
|
||||
return shared;
|
||||
}
|
||||
|
||||
function stopWith(signal: AbortSignal) {
|
||||
let stop!: () => void;
|
||||
const promise = new Promise<never>((_resolve, reject) => {
|
||||
stop = () => reject(signal.reason);
|
||||
signal.addEventListener("abort", stop, { once: true });
|
||||
if (signal.aborted) stop();
|
||||
});
|
||||
promise.catch(() => undefined);
|
||||
return { promise, dispose: () => signal.removeEventListener("abort", stop) };
|
||||
}
|
||||
|
||||
export async function shareNativeProviderArtifactCapture<T>(input: ShareNativeProviderArtifactCaptureInput<T>): Promise<T> {
|
||||
const hedgeAfterMs = input.hedgeAfterMs ?? NATIVE_PROVIDER_ARTIFACT_CAPTURE_HEDGE_MS;
|
||||
const stopped = stopWith(input.signal);
|
||||
try {
|
||||
let afterSavedMiss = false;
|
||||
for (;;) {
|
||||
const shared = sharedCaptureFor(input.key);
|
||||
const release = shared.tryStart(hedgeAfterMs, { afterSavedMiss });
|
||||
if (release) return await captureOrReuse(input, shared, release, stopped.promise);
|
||||
if (afterSavedMiss || !shared.saved) await shared.waitForChange(hedgeAfterMs, stopped.promise, { afterSavedMiss });
|
||||
const reused = await input.reuse();
|
||||
if (reused) {
|
||||
shared.markSaved();
|
||||
return reused;
|
||||
}
|
||||
afterSavedMiss ||= shared.saved;
|
||||
}
|
||||
} finally {
|
||||
stopped.dispose();
|
||||
}
|
||||
}
|
||||
|
||||
async function captureOrReuse<T>(
|
||||
input: ShareNativeProviderArtifactCaptureInput<T>,
|
||||
shared: SharedArtifactCapture,
|
||||
release: () => void,
|
||||
stopped: Promise<never>,
|
||||
): Promise<T> {
|
||||
const superseded = new AbortController();
|
||||
const held: Promise<unknown>[] = [];
|
||||
const own = Promise.resolve().then(() => input.capture(AbortSignal.any([input.signal, superseded.signal]), work => held.push(work)));
|
||||
void own.then(() => undefined, () => undefined).then(() => Promise.allSettled(held)).then(release);
|
||||
try {
|
||||
const first = await Promise.race([own.then(value => ({ value })), shared.savedCopy().then(() => null), stopped]);
|
||||
if (first) {
|
||||
shared.markSaved();
|
||||
return first.value;
|
||||
}
|
||||
const reused = await input.reuse();
|
||||
if (reused) return reused;
|
||||
if (!reused) return await own;
|
||||
superseded.abort(new Error("native_provider_artifacts_capture_superseded"));
|
||||
return reused;
|
||||
} catch (error) {
|
||||
if (input.signal.aborted) throw error;
|
||||
const reused = await input.reuse().catch(() => null);
|
||||
if (!reused) throw error;
|
||||
shared.markSaved();
|
||||
return reused;
|
||||
}
|
||||
const own = input.capture();
|
||||
sharedCaptures.set(input.key, own);
|
||||
try {
|
||||
return await own;
|
||||
} finally {
|
||||
if (sharedCaptures.get(input.key) === own) sharedCaptures.delete(input.key);
|
||||
}
|
||||
|
||||
export class NativeProviderArtifactExportError extends Error {
|
||||
readonly code: "native_provider_artifacts_export_timeout" | "native_provider_artifacts_export_failed";
|
||||
constructor(cause: unknown) {
|
||||
const code = isPluginRpcTimeout(cause) ? "native_provider_artifacts_export_timeout" : "native_provider_artifacts_export_failed";
|
||||
super(`${code}: ${cause instanceof Error ? cause.message : String(cause)}`, { cause });
|
||||
this.name = "NativeProviderArtifactExportError";
|
||||
this.code = code;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -87,6 +213,7 @@ export async function captureRemoteNativeProviderArtifacts<T>(
|
||||
manifest: NativeProviderArtifactSource["manifest"];
|
||||
signal: AbortSignal;
|
||||
context?: NativeExecutionContext;
|
||||
hold?: (work: Promise<unknown>) => void;
|
||||
},
|
||||
publish: (source: NativeProviderArtifactSource) => Promise<T>,
|
||||
): Promise<T> {
|
||||
@@ -134,7 +261,9 @@ export async function captureRemoteNativeProviderArtifacts<T>(
|
||||
input.context?.assertOpen();
|
||||
await input.runner.syncOut!([
|
||||
{ operationId: `provider-artifacts-${randomUUID()}`, files },
|
||||
]);
|
||||
]).catch((error: unknown) => {
|
||||
throw new NativeProviderArtifactExportError(error);
|
||||
});
|
||||
} finally {
|
||||
// The export copy is scratch under the excluded runtime dir. A failed
|
||||
// removal must not replace the transfer outcome; it is logged instead.
|
||||
@@ -172,6 +301,7 @@ export async function captureRemoteNativeProviderArtifacts<T>(
|
||||
}
|
||||
});
|
||||
void operation.catch(() => undefined);
|
||||
input.hold?.(operation);
|
||||
let abort!: () => void;
|
||||
const stopped = new Promise<never>((_resolve, reject) => {
|
||||
abort = () => reject(input.signal.reason);
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
|
||||
import type { NativeProviderArtifactSource } from "./native-provider-artifacts.js";
|
||||
import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES } from "@paperclipai/plugin-sdk";
|
||||
import { NativeProviderArtifactTransferTimeoutError } from "./native-provider-artifacts.js";
|
||||
import { NativeFailedBeforeProviderTurnError } from "./failed-before-provider-turn.js";
|
||||
import {
|
||||
captureRemoteNativeProviderArtifacts,
|
||||
shareNativeProviderArtifactCapture,
|
||||
@@ -52,6 +54,203 @@ describe("shared provider artifact capture", () => {
|
||||
expect(capture).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("starts a run's turn from another run's saved artifact while its own capture is still copying", async () => {
|
||||
const key = `company:${Math.random()}`;
|
||||
const hung = deferred<string>();
|
||||
const fast = deferred<string>();
|
||||
const signals: AbortSignal[] = [];
|
||||
let ready = false;
|
||||
const reuse = vi.fn(async () => ready ? "reused" : null);
|
||||
const capture = vi.fn()
|
||||
.mockImplementationOnce((signal: AbortSignal) => { signals.push(signal); return hung.promise; })
|
||||
.mockImplementationOnce((signal: AbortSignal) => { signals.push(signal); return fast.promise; });
|
||||
const slow = shareNativeProviderArtifactCapture({ key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 10 });
|
||||
const other = shareNativeProviderArtifactCapture({ key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 10 });
|
||||
await vi.waitFor(() => expect(capture).toHaveBeenCalledTimes(2), { interval: 1 });
|
||||
ready = true;
|
||||
fast.resolve("captured");
|
||||
await expect(other).resolves.toBe("captured");
|
||||
await expect(slow).resolves.toBe("reused");
|
||||
expect(signals[0]!.aborted).toBe(true);
|
||||
expect(signals[1]!.aborted).toBe(false);
|
||||
});
|
||||
|
||||
it("uses a saved artifact when the run's own capture fails after another run saved one", async () => {
|
||||
const key = `company:${Math.random()}`;
|
||||
let ready = false;
|
||||
const reuse = vi.fn(async () => ready ? "reused" : null);
|
||||
const capture = vi.fn(async () => {
|
||||
ready = true;
|
||||
throw Object.assign(new Error('RPC call "environmentSyncOut" timed out after 330000ms'), { code: -32003 });
|
||||
});
|
||||
await expect(shareNativeProviderArtifactCapture({ key, signal: new AbortController().signal, reuse, capture }))
|
||||
.resolves.toBe("reused");
|
||||
});
|
||||
|
||||
it("ends a still-running capture when another run's failed capture finds a saved copy", async () => {
|
||||
const key = `company:${Math.random()}`;
|
||||
const hung = deferred<string>();
|
||||
const signals: AbortSignal[] = [];
|
||||
let ready = false;
|
||||
const reuse = vi.fn(async () => ready ? "reused" : null);
|
||||
const capture = vi.fn()
|
||||
.mockImplementationOnce((signal: AbortSignal) => { signals.push(signal); return hung.promise; })
|
||||
.mockImplementationOnce(async () => {
|
||||
ready = true;
|
||||
throw new Error("copy failed after another server saved the files");
|
||||
});
|
||||
const slow = shareNativeProviderArtifactCapture({ key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 10 });
|
||||
const failing = shareNativeProviderArtifactCapture({ key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 10 });
|
||||
await expect(failing).resolves.toBe("reused");
|
||||
await expect(slow).resolves.toBe("reused");
|
||||
expect(signals[0]!.aborted).toBe(true);
|
||||
});
|
||||
|
||||
it("ends the slow capture when a waiting run finds a copy another server saved", async () => {
|
||||
const key = `company:${Math.random()}`;
|
||||
const hung = deferred<string>();
|
||||
const signals: AbortSignal[] = [];
|
||||
let ready = false;
|
||||
const reuse = vi.fn(async () => ready ? "reused" : null);
|
||||
const capture = vi.fn().mockImplementationOnce((signal: AbortSignal) => { signals.push(signal); return hung.promise; });
|
||||
const slow = shareNativeProviderArtifactCapture({ key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 10 });
|
||||
ready = true;
|
||||
const waiting = shareNativeProviderArtifactCapture({ key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 10 });
|
||||
await expect(waiting).resolves.toBe("reused");
|
||||
await expect(slow).resolves.toBe("reused");
|
||||
expect(signals[0]!.aborted).toBe(true);
|
||||
expect(capture).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("starts its own capture once instead of asking again when a saved copy cannot be reused", async () => {
|
||||
const key = `company:${Math.random()}`;
|
||||
const hung = deferred<string>();
|
||||
const reuse = vi.fn(async () => null);
|
||||
const capture = vi.fn()
|
||||
.mockImplementationOnce(() => hung.promise)
|
||||
.mockImplementationOnce(async () => "captured")
|
||||
.mockImplementationOnce(async () => "own");
|
||||
const slow = shareNativeProviderArtifactCapture({ key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 10 });
|
||||
const hedge = shareNativeProviderArtifactCapture({ key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 10 });
|
||||
await expect(hedge).resolves.toBe("captured");
|
||||
const late = shareNativeProviderArtifactCapture({ key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 10 });
|
||||
await expect(late).resolves.toBe("own");
|
||||
expect(reuse.mock.calls.length).toBeLessThan(6);
|
||||
hung.resolve("late");
|
||||
await slow;
|
||||
});
|
||||
|
||||
it("never runs more than two captures at once however many hedge windows pass", async () => {
|
||||
const key = `company:${Math.random()}`;
|
||||
const hung = deferred<string>();
|
||||
const hedge = deferred<string>();
|
||||
let ready = false;
|
||||
const reuse = vi.fn(async () => ready ? "reused" : null);
|
||||
const capture = vi.fn().mockImplementationOnce(() => hung.promise).mockImplementationOnce(() => hedge.promise);
|
||||
const runs = Array.from({ length: 6 }, () => shareNativeProviderArtifactCapture({
|
||||
key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 10 }));
|
||||
await new Promise(resolve => setTimeout(resolve, 80));
|
||||
expect(capture).toHaveBeenCalledTimes(2);
|
||||
expect(reuse.mock.calls.length).toBeLessThan(20);
|
||||
ready = true;
|
||||
hedge.resolve("captured");
|
||||
expect((await Promise.all(runs)).sort()).toEqual(["captured", ...Array(5).fill("reused")]);
|
||||
expect(capture).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("keeps a stopped capture's slot until its copy out of the sandbox ends", async () => {
|
||||
const key = `company:${Math.random()}`;
|
||||
const transfer = deferred<void>();
|
||||
const stop = new AbortController();
|
||||
const reuse = vi.fn(async () => null);
|
||||
const pending = () => new Promise<string>(() => undefined);
|
||||
const capture = vi.fn()
|
||||
.mockImplementationOnce((signal: AbortSignal, hold: (work: Promise<unknown>) => void) => {
|
||||
hold(transfer.promise);
|
||||
return new Promise<string>((_resolve, reject) => signal.addEventListener("abort", () => reject(signal.reason)));
|
||||
})
|
||||
.mockImplementation(pending);
|
||||
const stopped = shareNativeProviderArtifactCapture({ key, signal: stop.signal, reuse, capture, hedgeAfterMs: 0 });
|
||||
await vi.waitFor(() => expect(capture).toHaveBeenCalledTimes(1), { interval: 1 });
|
||||
stop.abort(new Error("controlled Stop"));
|
||||
await expect(stopped).rejects.toThrow("controlled Stop");
|
||||
void shareNativeProviderArtifactCapture({ key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 0 });
|
||||
void shareNativeProviderArtifactCapture({ key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 0 });
|
||||
await new Promise(resolve => setTimeout(resolve, 20));
|
||||
expect(capture).toHaveBeenCalledTimes(2);
|
||||
transfer.resolve();
|
||||
await vi.waitFor(() => expect(capture).toHaveBeenCalledTimes(3), { interval: 1 });
|
||||
});
|
||||
|
||||
it("keeps the two-capture cap for runs that cannot reuse the saved copy", async () => {
|
||||
const key = `company:${Math.random()}`;
|
||||
const releases: Array<() => void> = [];
|
||||
let active = 0, peak = 0;
|
||||
const reuse = vi.fn(async () => null);
|
||||
const capture = vi.fn((_signal: AbortSignal, hold: (work: Promise<unknown>) => void) => {
|
||||
const transfer = deferred<void>();
|
||||
active += 1;
|
||||
peak = Math.max(peak, active);
|
||||
hold(transfer.promise.finally(() => { active -= 1; }));
|
||||
releases.push(() => transfer.resolve());
|
||||
return capture.mock.calls.length === 2 ? Promise.resolve("captured") : new Promise<string>(() => undefined);
|
||||
});
|
||||
void shareNativeProviderArtifactCapture({ key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 10 });
|
||||
await expect(shareNativeProviderArtifactCapture({ key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 10 }))
|
||||
.resolves.toBe("captured");
|
||||
const misses = [0, 1].map(() => shareNativeProviderArtifactCapture({
|
||||
key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 10 }));
|
||||
await new Promise(resolve => setTimeout(resolve, 40));
|
||||
expect(capture).toHaveBeenCalledTimes(2);
|
||||
expect(peak).toBe(2);
|
||||
releases[1]!();
|
||||
await vi.waitFor(() => expect(capture).toHaveBeenCalledTimes(3), { interval: 1 });
|
||||
await new Promise(resolve => setTimeout(resolve, 20));
|
||||
expect(capture).toHaveBeenCalledTimes(3);
|
||||
expect(peak).toBe(2);
|
||||
expect(reuse.mock.calls.length).toBeLessThan(12);
|
||||
releases.forEach(release => release());
|
||||
void misses;
|
||||
});
|
||||
|
||||
it("starts one more capture when the shared capture runs past the hedge delay, not one per waiting run", async () => {
|
||||
const key = `company:${Math.random()}`;
|
||||
const hung = deferred<string>();
|
||||
const hedge = deferred<string>();
|
||||
let ready = false;
|
||||
const reuse = vi.fn(async () => ready ? "reused" : null);
|
||||
const capture = vi.fn().mockImplementationOnce(() => hung.promise).mockImplementationOnce(() => hedge.promise);
|
||||
const runs = Array.from({ length: 8 }, () => shareNativeProviderArtifactCapture({
|
||||
key, signal: new AbortController().signal, reuse, capture, hedgeAfterMs: 50 }));
|
||||
await vi.waitFor(() => expect(capture).toHaveBeenCalledTimes(2), { interval: 1 });
|
||||
await new Promise(resolve => setTimeout(resolve, 5));
|
||||
expect(capture).toHaveBeenCalledTimes(2);
|
||||
ready = true;
|
||||
hedge.resolve("captured");
|
||||
expect((await Promise.all(runs)).sort()).toEqual(["captured", ...Array(7).fill("reused")]);
|
||||
expect(capture).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("does not start a capture per waiting run when the shared capture fails", async () => {
|
||||
const key = `company:${Math.random()}`;
|
||||
const leader = deferred<string>();
|
||||
const replacement = deferred<string>();
|
||||
let ready = false;
|
||||
const reuse = vi.fn(async () => ready ? "reused" : null);
|
||||
const capture = vi.fn().mockImplementationOnce(() => leader.promise).mockImplementationOnce(() => replacement.promise);
|
||||
const runs = Promise.allSettled(Array.from({ length: 6 }, () => shareNativeProviderArtifactCapture({
|
||||
key, signal: new AbortController().signal, reuse, capture })));
|
||||
leader.reject(new Error("leader copy failed"));
|
||||
await vi.waitFor(() => expect(capture).toHaveBeenCalledTimes(2), { interval: 1 });
|
||||
await new Promise(resolve => setTimeout(resolve, 5));
|
||||
expect(capture).toHaveBeenCalledTimes(2);
|
||||
ready = true;
|
||||
replacement.resolve("captured");
|
||||
const settled = await runs;
|
||||
expect(settled.filter(outcome => outcome.status === "rejected")).toHaveLength(1);
|
||||
expect(capture).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("lets Stop end a wait for another run's capture", async () => {
|
||||
const key = `company:${Math.random()}`;
|
||||
const leader = deferred<string>();
|
||||
@@ -101,6 +300,41 @@ describe("provider artifact transfer step", () => {
|
||||
expect(publish).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
it("names a stalled copy out of the sandbox instead of an unknown cause", async () => {
|
||||
vi.mocked(runner.syncOut!).mockRejectedValueOnce(new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.TIMEOUT,
|
||||
message: 'RPC call "environmentSyncOut" timed out after 330000ms' }));
|
||||
const publish = vi.fn();
|
||||
const failure = await capture(publish).catch((error: unknown) => error);
|
||||
expect(failure).toMatchObject({ code: "native_provider_artifacts_export_timeout" });
|
||||
expect(new NativeFailedBeforeProviderTurnError(null, failure)).toMatchObject({
|
||||
technicalCause: "native_provider_artifacts_export_timeout",
|
||||
message: "Setting up the computer took too long, so the task is continuing on a fresh one.",
|
||||
});
|
||||
expect(publish).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("reports the copy out of the sandbox as held work that outlives a Stop", async () => {
|
||||
const transfer = deferred<{ operations: [] }>();
|
||||
vi.mocked(runner.syncOut!).mockImplementationOnce(() => transfer.promise);
|
||||
const stop = new AbortController();
|
||||
const held: Promise<unknown>[] = [];
|
||||
const capturing = captureRemoteNativeProviderArtifacts({
|
||||
runner, remoteCwd: "/workspace", manifest, signal: stop.signal, hold: work => held.push(work),
|
||||
launch: { root: "/owned/qualified", accountUid: 1000,
|
||||
paths: { nodeCommand: "/qualified/node", opencodeProxy: "/qualified/proxy", opencodeExecutable: "/qualified/opencode" } },
|
||||
}, vi.fn());
|
||||
await vi.waitFor(() => expect(runner.syncOut).toHaveBeenCalled(), { interval: 1 });
|
||||
stop.abort(new Error("controlled Stop"));
|
||||
await expect(capturing).rejects.toThrow("controlled Stop");
|
||||
expect(held).toHaveLength(1);
|
||||
let settled = false;
|
||||
void held[0]!.then(() => { settled = true; }, () => { settled = true; });
|
||||
await new Promise(resolve => setTimeout(resolve, 10));
|
||||
expect(settled).toBe(false);
|
||||
transfer.resolve({ operations: [] });
|
||||
await vi.waitFor(() => expect(settled).toBe(true), { interval: 1 });
|
||||
});
|
||||
|
||||
it("does not retry a failure that is not a transfer timeout", async () => {
|
||||
const publish = vi.fn().mockRejectedValue(new Error("native_provider_artifacts_owner_changed"));
|
||||
await expect(capture(publish)).rejects.toThrow("native_provider_artifacts_owner_changed");
|
||||
|
||||
@@ -19196,9 +19196,10 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
launchRoot: launch.root, accountUid: launch.accountUid });
|
||||
providerArtifacts = await reuse() ?? await shareNativeProviderArtifactCapture({
|
||||
key: `${input.execution.binding.companyId}:${manifest.digest}`, signal, reuse,
|
||||
capture: () => captureRemoteNativeProviderArtifacts({ runner: remoteCommandRunner,
|
||||
remoteCwd: remoteTarget.remoteCwd, launch, manifest, signal, context: input.executionContext },
|
||||
source => retainNativeProviderArtifacts({ db: input.db, storage: artifactStorage, owner: backupOwner, source, signal })),
|
||||
capture: (captureSignal, hold) => captureRemoteNativeProviderArtifacts({ runner: remoteCommandRunner,
|
||||
remoteCwd: remoteTarget.remoteCwd, launch, manifest, signal: captureSignal, context: input.executionContext, hold },
|
||||
source => retainNativeProviderArtifacts({ db: input.db, storage: artifactStorage, owner: backupOwner, source,
|
||||
signal: captureSignal })),
|
||||
});
|
||||
} : undefined,
|
||||
bindDiagnosticSession: session => {
|
||||
|
||||
Reference in new issue
Block a user