mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
fix(native): read settled harness state again after a dropped sandbox connection (#20690)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps one provider conversation across the per-turn runs of a task. After each turn, the controller reads the settled Runner and provider state out of the sandbox and saves it as the harness backup. > - A per-turn sandbox is released after the turn. The harness backup is then the only copy of the provider session. > - On staging, one dropped sandbox connection (`write ECONNRESET`) during that read lost the backup. The run itself succeeded, because its result was already accepted. > - The next run of the task could not resume the conversation: `PRP provider resume state is unavailable`. When the task was waiting for a reply, the run stopped at "Recovery needed" with no Retry. A person had to send a message to start a new conversation. > - This pull request reads the settled state again when the sandbox connection drops during that read. > - The benefit is that one network reset no longer loses a task's conversation. ## Linked Issues or Issue Description Refs #20627 (warm backup restore when the source sandbox is idle) and #20683 (relaunch of a settled runner). This PR changes a different step: the backup capture at the end of a turn. **What happened?** A per-turn OpenCode task ran on a remote Daytona sandbox with reuse disabled. 1. Run A accepted its result. The Runner suspended. 2. The controller read the settled state out of the sandbox. The read failed with `remote checkpoint failed: write ECONNRESET`. `harness_state.backup.persist` and `session.checkpoint.persist` failed. 3. Run A still finished as `succeeded`, because the result was durable. The sandbox was released. No harness backup of turn A existed. 4. Run B started in a new sandbox. It failed in 0.2 s with `native_session_recovery_failed: Error: PRP provider resume state is unavailable`. When the prior turn yielded for a reply, the checkpoint policy does not allow a replacement session, so the task showed "Recovery needed" with only inspect actions. On staging this happened on 4 tasks in 48 hours. In all 3 predecessor runs that I could inspect, the backup capture failed. 2 of them show the `ECONNRESET` in the run log. No deploy happened between run A and run B in one case, so a deploy is not the cause. **Expected behavior** A transient connection drop while reading settled state does not lose the provider conversation. The read is repeated, and the backup is saved. **Steps to reproduce** 1. Run the first durable turn of a remote OpenCode session. 2. Make the sandbox `syncOut` of the backup directories fail once with `ECONNRESET`. 3. Call the settled checkpoint. On `master` it throws `write ECONNRESET`, and no backup is saved. The new test `uses the bundled Linux image authority for remote opencode-new-durable-reset without controller overrides` runs these steps. **Paperclip version or commit** `master` at `7435abcc63`. ## What Changed - The settled-state capture (inspect plus `syncOut` of each persistence directory into a pending directory) is now one step that can repeat. The step writes nothing durable, so it is safe to repeat. - A dropped, reset or refused sandbox connection repeats the step, at most 3 attempts, with 1 s and 2 s delays. Each repeat clears the pending directory and inspects the live state again, so the copy always matches the inspected identity. - Any other failure fails at once, as before: a state mismatch, a missing directory, an unsafe archive, or a TLS failure. - The durable backup upload and the local publish run only after a complete capture. They did not change. - `isTransientRemoteCheckpointReadFailure` classifies the error with the existing `classifyRemoteConnectionError`, plus a message match for SDK errors that lose their `code`. - The first completeness inspection and its visibility-window re-reads use the same bounded retry (`retryTransientRemoteCheckpointRead`). A dropped provider-state read keeps its `provider_state_unreadable` refusal for every other caller, but the checkpoint retry looks through it to the dropped read. Undici `fetch failed` and `no response returned` also count as drops. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts`: 1343 passed. - Red on `master`: the new `opencode-new-durable-reset` case fails with `Error: write ECONNRESET`. Green with this change. - The `isTransientRemoteCheckpointReadFailure` test checks that state mismatches, unsafe archives and TLS failures do not repeat. - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low risk. The normal path has no added wait. A repeat adds at most 3 s, and only after a connection failure. - A repeated read inspects the live state again. If the state changed between attempts, the existing identity checks still refuse it. - A connection that stays down for all 3 attempts fails the same way as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed
1 parent
69c69e4c20
commit
a1ec05a1a4
2 files changed
+145
-20
No files matched your search
@@ -469,6 +469,9 @@ import {
|
||||
stageRemoteRunnerFile,
|
||||
steerNativeSession,
|
||||
syncRemoteRunnerDirectoryOut,
|
||||
isTransientRemoteCheckpointReadFailure,
|
||||
remoteProviderStateUnreadable,
|
||||
retryTransientRemoteCheckpointRead,
|
||||
verifyNativeHarnessBackup,
|
||||
shouldRestoreNativeHarnessBackupIntoSandbox,
|
||||
} from "./native-session-executor.js";
|
||||
@@ -2312,8 +2315,14 @@ describe("remote provider pack manifest", () => {
|
||||
return JSON.stringify(value);
|
||||
};
|
||||
|
||||
it.each(["opencode", "opencode-new-durable", "opencode-continuation", "opencode-continuation-missing", "opencode-continuation-no-receipt", "opencode-continuation-replacement", "opencode-continuation-cold", "opencode-continuation-legacy-durable", "opencode-dirty", "opencode-diagnostics", "opencode-diagnostics-stale", "cursor", "claude-x64", "claude-arm64"] as const)("uses the bundled Linux image authority for remote %s without controller overrides", async provider => {
|
||||
const firstDurable = provider === "opencode-new-durable";
|
||||
it.each(["opencode", "opencode-new-durable", "opencode-new-durable-reset", "opencode-new-durable-inspect-reset", "opencode-new-durable-provider-reset", "opencode-continuation", "opencode-continuation-missing", "opencode-continuation-no-receipt", "opencode-continuation-replacement", "opencode-continuation-cold", "opencode-continuation-legacy-durable", "opencode-dirty", "opencode-diagnostics", "opencode-diagnostics-stale", "cursor", "claude-x64", "claude-arm64"] as const)("uses the bundled Linux image authority for remote %s without controller overrides", async provider => {
|
||||
const firstDurable = provider.startsWith("opencode-new-durable");
|
||||
// A sandbox connection drops once while the settled state is read out.
|
||||
let checkpointReadResets = provider === "opencode-new-durable-reset" ? 1 : 0;
|
||||
// Or while the checkpoint inspects the runner state, or reads the provider state, before any copy starts.
|
||||
let inspectionResets = 0;
|
||||
const inspectionReset = provider === "opencode-new-durable-inspect-reset" ? "runner-state.json"
|
||||
: provider === "opencode-new-durable-provider-reset" ? "codex-provider-state.json" : null;
|
||||
const architecture = provider === "claude-arm64" ? "arm64" : "x64";
|
||||
const claude = provider.startsWith("claude-");
|
||||
const opencode = provider.startsWith("opencode");
|
||||
@@ -2372,6 +2381,10 @@ describe("remote provider pack manifest", () => {
|
||||
const suspendedState = durableRunnerState({ runId: remoteExecution.binding.runId, runnerInstanceId: "bundled-image-runner",
|
||||
environmentLeaseId: "lease", normalizedSessionId: remoteExecution.session.normalizedSessionId }, "suspended");
|
||||
const syncOut = vi.fn(async (operations: Array<{ files: Array<{ kind: string; targetPath: string }> }>) => {
|
||||
if (checkpointReadResets > 0 && operations.some(operation => operation.files.some(file => file.targetPath.includes("/.pending-")))) {
|
||||
checkpointReadResets--;
|
||||
throw Object.assign(new Error("write ECONNRESET"), { code: "ECONNRESET" });
|
||||
}
|
||||
for (const operation of operations) for (const file of operation.files) {
|
||||
if (file.kind === "directory") {
|
||||
await mkdir(file.targetPath, { recursive: true });
|
||||
@@ -2402,7 +2415,14 @@ describe("remote provider pack manifest", () => {
|
||||
});
|
||||
onTestFinished(() => { storage.mockRestore(); owner.mockRestore(); validate.mockRestore(); retain.mockRestore(); capture.mockRestore(); backup.mockRestore(); });
|
||||
}
|
||||
const execute = vi.fn(async (command: { command: string; args?: string[] }) => ({
|
||||
const execute = vi.fn(async (command: { command: string; args?: string[] }) => {
|
||||
const script = command.args?.[1] ?? "";
|
||||
if (inspectionResets > 0 && inspectionReset && script.includes("base64 <") && script.includes(inspectionReset)
|
||||
&& (inspectionReset === "runner-state.json" || !script.includes("runner-state.json"))) {
|
||||
inspectionResets--;
|
||||
throw Object.assign(new Error("read ECONNRESET"), { code: "ECONNRESET" });
|
||||
}
|
||||
return {
|
||||
exitCode: 0, signal: null, timedOut: false, stderr: "",
|
||||
stdout: command.args?.[0] === "--build-metadata" ? JSON.stringify({
|
||||
schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd",
|
||||
@@ -2412,7 +2432,8 @@ describe("remote provider pack manifest", () => {
|
||||
}) : firstDurable && command.args?.[1]?.startsWith("const os=require") ? JSON.stringify({ home: "/home/runtime-account", uid: 1000 })
|
||||
: firstDurable && command.args?.[1]?.includes("base64 <") ? Buffer.from(JSON.stringify(command.args[1].includes("codex-provider-state.json")
|
||||
&& !command.args[1].includes("runner-state.json") ? providerState : suspendedState)).toString("base64") : "",
|
||||
}));
|
||||
};
|
||||
});
|
||||
try {
|
||||
let coldAdmission: ReturnType<typeof admittedNativeProviderPack> = null;
|
||||
const prepare = createRunnerdBackend({ db: preparedDb, execution: remoteExecution,
|
||||
@@ -2492,8 +2513,11 @@ describe("remote provider pack manifest", () => {
|
||||
expect(exportSources.every(source => source.startsWith("/workspace/"))).toBe(true);
|
||||
expect(providerArtifactStorage.retainNativeProviderArtifacts).toHaveBeenCalledWith(expect.objectContaining({
|
||||
owner: expect.objectContaining({ runId: remoteExecution.binding.runId }), source: expect.objectContaining({ manifest: installed, launchRoot, accountUid: 1000 }) }));
|
||||
if (inspectionReset) inspectionResets = 1;
|
||||
await driver.mock.calls.at(-1)![0].checkpoint!("settled");
|
||||
expect(harnessStorage.persistNativeHarnessBackup).toHaveBeenCalledTimes(1);
|
||||
expect(checkpointReadResets).toBe(0);
|
||||
expect(inspectionResets).toBe(0);
|
||||
return;
|
||||
}
|
||||
expect(execute.mock.calls.some(([call]) => call.args?.[0] === "-e" || call.args?.[0] === "--version")).toBe(false);
|
||||
@@ -21223,3 +21247,32 @@ describe("HTTP Stop during remote runner staging", () => {
|
||||
} finally { release(); await outcome; await stopping; await bundle.transport.detachControllerForRestart?.().catch(() => {}); await temporary.cleanup(); await rm(root, { recursive: true, force: true }); }
|
||||
}, 120000);
|
||||
});
|
||||
|
||||
describe("isTransientRemoteCheckpointReadFailure", () => {
|
||||
it("repeats only a dropped sandbox connection, never a state or safety refusal", () => {
|
||||
expect(isTransientRemoteCheckpointReadFailure(Object.assign(new Error("write ECONNRESET"), { code: "ECONNRESET" }))).toBe(true);
|
||||
expect(isTransientRemoteCheckpointReadFailure(new Error("Daytona syncOut directory download failed: socket hang up"))).toBe(true);
|
||||
expect(isTransientRemoteCheckpointReadFailure(new Error("runner_harness_state_mismatch: checkpoint_identity_incomplete"))).toBe(false);
|
||||
expect(isTransientRemoteCheckpointReadFailure(new Error("Daytona syncOut refusing tarball member that escapes the extraction dir: ../x"))).toBe(false);
|
||||
expect(isTransientRemoteCheckpointReadFailure(Object.assign(new Error("certificate expired"), { code: "CERT_HAS_EXPIRED" }))).toBe(false);
|
||||
// A provider-state refusal is transient only when the read behind it dropped.
|
||||
expect(isTransientRemoteCheckpointReadFailure(remoteProviderStateUnreadable(Object.assign(new Error("read ECONNRESET"), { code: "ECONNRESET" })))).toBe(true);
|
||||
expect(isTransientRemoteCheckpointReadFailure(remoteProviderStateUnreadable(new Error("runner_remote_provider_state_unavailable")))).toBe(false);
|
||||
expect(isTransientRemoteCheckpointReadFailure(remoteProviderStateUnreadable(new SyntaxError("Unexpected token")))).toBe(false);
|
||||
expect(remoteProviderStateUnreadable(new Error("read ECONNRESET")).message).toBe("runner_harness_state_mismatch: provider_state_unreadable");
|
||||
});
|
||||
|
||||
it("stops after three reads, and never repeats a refusal", async () => {
|
||||
const reset = () => Object.assign(new Error("read ECONNRESET"), { code: "ECONNRESET" });
|
||||
const onRetry = vi.fn();
|
||||
const dropping = vi.fn(async () => { throw reset(); });
|
||||
await expect(retryTransientRemoteCheckpointRead(dropping, onRetry, 0)).rejects.toThrow("read ECONNRESET");
|
||||
expect(dropping).toHaveBeenCalledTimes(3);
|
||||
expect(onRetry.mock.calls).toEqual([[1, 3], [2, 3]]);
|
||||
const refusing = vi.fn(async () => { throw new Error("runner_harness_state_mismatch: runner_identity"); });
|
||||
await expect(retryTransientRemoteCheckpointRead(refusing, vi.fn(), 0)).rejects.toThrow("runner_identity");
|
||||
expect(refusing).toHaveBeenCalledTimes(1);
|
||||
let drops = 2;
|
||||
await expect(retryTransientRemoteCheckpointRead(async () => { if (drops-- > 0) throw reset(); return "settled"; }, vi.fn(), 0)).resolves.toBe("settled");
|
||||
});
|
||||
});
|
||||
@@ -295,6 +295,7 @@ import {
|
||||
} from "./status-arbiter.js";
|
||||
import { conflict, HttpError } from "../../errors.js";
|
||||
import { redactSensitiveText } from "../../redaction.js";
|
||||
import { classifyRemoteConnectionError } from "../remote-connection-failure.js";
|
||||
import { resolvePaperclipRunnerBinary, resolveRemotePaperclipRunnerBinary } from "./native-codex-runner.js";
|
||||
import {
|
||||
createNativeRunTrace,
|
||||
@@ -13735,6 +13736,55 @@ function assertSafeExtractedCheckpoint(root: string): void {
|
||||
}
|
||||
}
|
||||
|
||||
const REMOTE_CHECKPOINT_CAPTURE_ATTEMPTS = 3;
|
||||
const REMOTE_CHECKPOINT_CAPTURE_RETRY_MS = 1_000;
|
||||
const TRANSIENT_REMOTE_READ_MESSAGE = /\b(?:ECONNRESET|EPIPE|ETIMEDOUT|ECONNREFUSED|socket hang up|fetch failed|no response returned)\b/;
|
||||
/** `provider_state_unreadable` refusals whose underlying read was a dropped
|
||||
* connection. The refusal keeps its message for every other caller; only the
|
||||
* checkpoint read retry looks through it. */
|
||||
const providerStateReadTransportFailures = new WeakMap<Error, unknown>();
|
||||
|
||||
/** A dropped or refused sandbox connection while reading settled state. A
|
||||
* mismatch, a missing directory or an unsafe archive is never transient. */
|
||||
export function isTransientRemoteCheckpointReadFailure(error: unknown): boolean {
|
||||
if (error instanceof Error && providerStateReadTransportFailures.has(error)) {
|
||||
return isTransientRemoteCheckpointReadFailure(providerStateReadTransportFailures.get(error));
|
||||
}
|
||||
const reason = classifyRemoteConnectionError(error);
|
||||
if (reason !== null) return reason !== "tls_failure";
|
||||
return error instanceof Error && !error.message.startsWith("runner_")
|
||||
&& TRANSIENT_REMOTE_READ_MESSAGE.test(error.message);
|
||||
}
|
||||
|
||||
/** The refusal `inspectRemoteHarnessState` throws when the provider state
|
||||
* cannot be read, remembering the read failure behind it. */
|
||||
export function remoteProviderStateUnreadable(readFailure: unknown): Error {
|
||||
const refusal = new Error("runner_harness_state_mismatch: provider_state_unreadable");
|
||||
providerStateReadTransportFailures.set(refusal, readFailure);
|
||||
return refusal;
|
||||
}
|
||||
|
||||
/**
|
||||
* Repeats a read of settled state out of the sandbox after a dropped
|
||||
* connection, at most {@link REMOTE_CHECKPOINT_CAPTURE_ATTEMPTS} times in all.
|
||||
* Any other failure, and the last transient one, is thrown unchanged.
|
||||
*/
|
||||
export async function retryTransientRemoteCheckpointRead<T>(
|
||||
read: () => Promise<T>,
|
||||
onRetry: (attempt: number, attempts: number) => Promise<void> | void,
|
||||
retryDelayMs = REMOTE_CHECKPOINT_CAPTURE_RETRY_MS,
|
||||
): Promise<T> {
|
||||
for (let attempt = 1; ; attempt++) {
|
||||
try {
|
||||
return await read();
|
||||
} catch (error) {
|
||||
if (attempt >= REMOTE_CHECKPOINT_CAPTURE_ATTEMPTS || !isTransientRemoteCheckpointReadFailure(error)) throw error;
|
||||
await onRetry(attempt, REMOTE_CHECKPOINT_CAPTURE_ATTEMPTS);
|
||||
await new Promise<void>((resolveDelay) => setTimeout(resolveDelay, retryDelayMs * attempt));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Copy a remote runner directory into durable local state.
|
||||
*
|
||||
@@ -15754,8 +15804,8 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
stateDirectory: remoteStateDirectory,
|
||||
execution: input.execution,
|
||||
});
|
||||
} catch {
|
||||
throw new Error("runner_harness_state_mismatch: provider_state_unreadable");
|
||||
} catch (error) {
|
||||
throw remoteProviderStateUnreadable(error);
|
||||
}
|
||||
const providerSessionIdentity =
|
||||
providerSessionIdentityFromDurableProviderState({
|
||||
@@ -16301,14 +16351,23 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
// `runner_harness_state_mismatch`. Only checkpoint a harness that runnerd
|
||||
// has proved complete. A malformed or identity-conflicting state still
|
||||
// throws from inspectRemoteHarnessState and therefore fails closed.
|
||||
let checkpointable = await measureNativeRunnerSpan(input.trace, "harness_state.inspect", () => inspectRemoteHarnessState());
|
||||
// A dropped sandbox connection repeats the read (bounded) instead of
|
||||
// losing the only backup of the provider session.
|
||||
const inspectForCheckpoint = () => retryTransientRemoteCheckpointRead(
|
||||
() => measureNativeRunnerSpan(input.trace, "harness_state.inspect", () => inspectRemoteHarnessState()),
|
||||
(attempt, attempts) => input.onLog?.(
|
||||
"stderr",
|
||||
`[paperclip-runner] remote checkpoint inspection interrupted (attempt ${attempt} of ${attempts}); reading the settled state again\n`,
|
||||
),
|
||||
);
|
||||
let checkpointable = await inspectForCheckpoint();
|
||||
// The remote runner writes its suspended lifecycle and provider state
|
||||
// before the outer process-owner RPC necessarily observes completion.
|
||||
// Allow a very small bounded visibility window without ever accepting an
|
||||
// active, incomplete, malformed, or identity-conflicting checkpoint.
|
||||
for (let attempt = 1; !checkpointable.complete && attempt < 3; attempt++) {
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, 100));
|
||||
checkpointable = await measureNativeRunnerSpan(input.trace, "harness_state.inspect", () => inspectRemoteHarnessState());
|
||||
checkpointable = await inspectForCheckpoint();
|
||||
}
|
||||
if (!checkpointable.complete) {
|
||||
const incompleteFailure = remoteCheckpointIncompleteFailure(
|
||||
@@ -16338,24 +16397,27 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
input.trace,
|
||||
"harness_state.backup.persist",
|
||||
async () => {
|
||||
const verified = await measureNativeRunnerSpan(input.trace, "harness_state.inspect", () => inspectRemoteHarnessState());
|
||||
if (
|
||||
!verified.complete ||
|
||||
!verified.runnerState ||
|
||||
!verified.providerSessionIdentity
|
||||
) {
|
||||
throw new Error("runner_harness_state_mismatch: checkpoint_identity_incomplete");
|
||||
}
|
||||
const providerSessionIdentity = verified.providerSessionIdentity;
|
||||
|
||||
const backupRoot = harnessBackupRoot(root);
|
||||
mkdirSync(backupRoot, { recursive: true, mode: 0o700 });
|
||||
const pendingRoot = resolve(
|
||||
backupRoot,
|
||||
`.pending-${randomUUID()}`,
|
||||
);
|
||||
mkdirSync(pendingRoot, { recursive: true, mode: 0o700 });
|
||||
try {
|
||||
// Reading the settled state out of the sandbox writes nothing
|
||||
// durable, so a dropped sandbox connection can repeat the whole
|
||||
// read. Without this, one reset loses the only copy of the
|
||||
// provider session when the per-turn sandbox is released.
|
||||
const captureSettledState = async () => {
|
||||
rmSync(pendingRoot, { recursive: true, force: true });
|
||||
const verified = await measureNativeRunnerSpan(input.trace, "harness_state.inspect", () => inspectRemoteHarnessState());
|
||||
if (
|
||||
!verified.complete ||
|
||||
!verified.runnerState ||
|
||||
!verified.providerSessionIdentity
|
||||
) {
|
||||
throw new Error("runner_harness_state_mismatch: checkpoint_identity_incomplete");
|
||||
}
|
||||
mkdirSync(pendingRoot, { recursive: true, mode: 0o700 });
|
||||
for (const directory of persistenceProfile.directories) {
|
||||
const sourcePath = remotePersistencePath(directory);
|
||||
if (!sourcePath)
|
||||
@@ -16372,6 +16434,16 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
throw new Error("runner_harness_state_mismatch: checkpoint_directory_missing");
|
||||
}
|
||||
}
|
||||
return verified.providerSessionIdentity;
|
||||
};
|
||||
try {
|
||||
const providerSessionIdentity = await retryTransientRemoteCheckpointRead(
|
||||
captureSettledState,
|
||||
(attempt, attempts) => input.onLog?.(
|
||||
"stderr",
|
||||
`[paperclip-runner] remote checkpoint read interrupted (attempt ${attempt} of ${attempts}); reading the settled state again\n`,
|
||||
),
|
||||
);
|
||||
const manifest = buildNativeHarnessBackupManifest({
|
||||
backupRoot: pendingRoot,
|
||||
execution: input.execution,
|
||||
|
||||
Reference in new issue
Block a user