fix(native): bound per-turn harness backups by inactivity, not total time (#22382)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - After a remote native turn is accepted, the executor saves the
provider's harness state to durable storage, so another web server can
continue the same conversation.
> - That save runs under a fixed 300 s total deadline
(`AbortSignal.timeout(300_000)`). The deadline covers archive capture,
digest, encryption, the upload, any storage-outage wait and a full
verifying read-back.
> - On staging, five accepted runs failed this save 337–388 s after it
began. The run then reported `native_workspace_sync_out_failed` with
"The operation was aborted" or tar-stream's "already finalized or
destroyed", and workspace copy-back went to retry.
> - #19906 fixed the same fixed-deadline rule on the restore side. The
write side still uses it. Since #20772/#20871 the save also waits out
storage throttling and outages for up to five minutes, which cannot
finish inside a five-minute total.
> - This pull request bounds the upload and the read-back by inactivity
instead of total time. A slow transfer continues while the store makes
progress. A transfer that stops fails with a named code.
> - The benefit is that an accepted turn no longer loses its provider
backup only because the backup is large or the store is slow.

## Linked Issues or Issue Description

Refs #19906 (same idle-deadline rule for failover-backup restores),
#20772 and #20871 (file-source resends and storage-outage waits during
the save), #19108 (stops tar-stream from hiding the real abort), #18932
(durable harness backups).

**What happened?**

On staging, five native per-turn runs failed within one hour, all after
the provider turn was accepted. In each run,
`harness_state.backup.persist` failed after 359,588 / 365,422 / 387,471
/ 383,801 / 337,335 ms, and `session.checkpoint.persist` failed with it.
The only fixed limit on that path is `AbortSignal.timeout(300_000)`
around `persistNativeHarnessBackup` in `native-session-executor.ts`. The
durations sit just above 300 s, plus a variable sandbox sync-out step
before the save. Errors were reported as "The operation was aborted"
and, before #19108, as "already finalized or destroyed".

**Expected behavior**

An accepted turn saves its provider backup while the store keeps making
progress, even when the total time is more than five minutes. A write or
read-back that stalls fails with a named, diagnosable code. A failed
save publishes no backup.

**Steps to reproduce**

1. Persist a native harness backup through a store that takes upload
chunks slowly, so the full save takes longer than the deadline.
2. With the executor's fixed total deadline, the save fails with "The
operation was aborted" and publishes nothing.
3. A store that stops taking bytes, with no caller deadline, waits
indefinitely.

## What Changed

- `native-harness-backup-storage.ts`: generalize
`withDownloadProgressDeadline` into `withTransferProgressDeadline`. It
still wraps `getObject` exactly as before, and it now also wraps
`putFile` with a five-minute idle timer:
- A provider that reads the stream body (local disk) renews it with each
chunk it takes.
- A provider that sends the file source itself (S3 with `sourceFile`,
which never reads the stream) renews it with each request the store
answers, each multipart part it takes, and each resend or outage wait
that starts.
- After the last of them the timer stays armed until the store answers,
so a write that never completes is named too.
- `persistNativeHarnessBackup` gains `transferIdleTimeoutMs` and
`persistMaxMs` (default 30 minutes). The outer bound wraps the whole
save, including the storage-outage wait. With an idle limit, a stalled
upload throws `native_harness_storage_upload_stalled`, a stalled
read-back throws `native_harness_storage_download_stalled`, and the
whole save throws `native_harness_storage_persist_deadline_exceeded` at
the outer bound. Caller cancellation keeps its own reason, and a storage
outage that outlasts its own budget still surfaces as the storage error.
Callers that pass no idle limit (controller archive, offline migration,
preservation) are unchanged.
- `storage/request-stats.ts`: add `observeStorageProgress(onProgress,
work)`, an `AsyncLocalStorage` observer called from the existing
`recordStorageRequest`, `recordStoragePart` and `recordStorageBackoff`
hooks. It is scoped to the wrapped `putFile` call only, and a throwing
observer is ignored. Stats collection is unchanged.
- `native-session-executor.ts`: the per-turn backup passes
`NATIVE_HARNESS_UPLOAD_IDLE_TIMEOUT_MS` (five minutes) instead of
`AbortSignal.timeout(300_000)`.
- Restore behavior and error codes are unchanged:
`withDownloadProgressDeadline` now delegates to the shared helper with
the same defaults.
- `doc/database-backups.md`: document the write-side idle limit, what
counts as progress, and the new codes.

Previous-release story (`rules/upgrade-safe-run-state.mdc`): no
persisted shape changes. Backup rows, references, receipts and
checkpoints are unchanged. A deploy that lands during a save stops the
process, and the pending row is never published, as before.

## Verification

Run from `server/` (provider env vars unset):

- `pnpm exec vitest run
src/services/native-runtime/native-harness-backup-upload-deadline.integration.test.ts`:
6 tests passed. They use embedded PostgreSQL, local-disk storage and an
incompressible 1 MB acpx backup.
- "keeps saving a backup whose upload outlasts the idle window while the
store keeps taking bytes". Under the former fixed-total policy the same
slow upload fails with "The operation was aborted" and publishes
nothing. With the idle limit it completes after more than twice the
window and restores.
- "names an upload that stops taking bytes and publishes no backup",
then saves the same source once the store responds.
- "keeps a file-source upload going while the store answers its
requests, and names one that stops". The fake store never reads the
stream and records each request answer the way the S3 provider does.
- "names a read-back that stops sending bytes and publishes no backup".
  - "stops at the outer bound while an upload still progresses".
  - "lets Stop end a progressing upload with its own reason".
- RED, part 1: with `native-harness-backup-storage.ts` and
`request-stats.ts` reverted to `master`, 4 tests fail. Three stall tests
hang until the 10 s test timeout, and the outer-bound test resolves
instead of rejecting.
- RED, part 2: with only the progress observer disabled, the file-source
test fails with `native_harness_storage_upload_stalled` on an upload
that is still progressing.
- `native-harness-backup-*` suites,
`native-harness-online-preservation`, `src/storage/`,
`storage-request-stats` and `storage-s3-multipart-backoff`: 200 of 201
passed. The one failure, `native-harness-backup-cleanup.test.ts`
("removes retired backups containing immutable skills…", expected mode
256 to be 292), fails the same way on unmodified `master` in this
sandbox.
- `native-session-executor.test.ts`: 1541 of 1542 passed. The new
assertion checks that the per-turn backup passes `transferIdleTimeoutMs`
and no fixed signal, and flipping it fails the `opencode-new-durable*`
cases. The one failure, "compares export process compatibility without
changing conversation identity (changed: false)" (`db.transaction is not
a function`), fails the same way on unmodified `master`.
- `node scripts/typecheck-layers.mjs`: no errors in `server/`. The local
filtered install lacked some CLI/UI/demo packages, which reported
unrelated errors. CI covers the full typecheck.

Not verified: a live staging run with this build. The staging evidence
shows the save failing just after 300 s on several runs. It does not
show which step was slow (upload, read-back or local capture), so the
fix removes the fixed total for every step on that path.

## Risks

- A save that progresses very slowly can now take longer than five
minutes. Before, it failed. The settle step waits for it, up to the
30-minute outer bound.
- A local step that hangs (for example a stalled disk during capture) is
now cut off at 30 minutes instead of 5 minutes, with
`native_harness_storage_persist_deadline_exceeded`.
- For S3 file-source uploads, progress is measured per request, not per
byte. A single request that takes more than five minutes counts as
stalled. For a 16 MiB single PUT, that is a store slower than about 55
KB/s. Multipart parts are 8 MiB, eight in flight.
- If a successor takes over during a long save, the existing fence
checks before and after the upload reject it with `owner_superseded`,
and nothing is published. This is unchanged behavior.

## Model Used

- Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), with
tool use (shell, file edits) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-20 03:47:31 -08:00
1 parent 3f6c3d37dc
commit 62bc0519d9
6 files changed
+284 -18

No files matched your search

+11
View File
@@ -81,6 +81,17 @@ Environment overrides:
and publishes no partial backup. The whole restore, including local decrypt,
digest and extraction, has a 30-minute outer limit
(`native_harness_storage_restore_deadline_exceeded`). Stop ends the restore.
The per-turn backup that saves provider state after an accepted turn follows the
same rule. Its upload and its verifying read-back each have a five-minute idle
limit instead of a fixed five-minute total, so a large backup or a slow store
keeps saving while the store makes progress. For an upload, progress is a chunk
the store takes or, when the store sends the file itself (S3), a request it
answers, a part it takes or a resend that starts its wait. An upload with no
progress for five minutes fails with `native_harness_storage_upload_stalled`, a
stalled read-back with `native_harness_storage_download_stalled`, and neither
publishes a backup. The whole backup, including archive capture, encryption and
any storage-outage wait, has a 30-minute outer limit
(`native_harness_storage_persist_deadline_exceeded`).
- `PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS=<days>` sets how long native
session continuity-break archives and quarantined session state are kept (see
`doc/recoverable-disk-cleanup.md`). The default is `7` and the minimum is `1`. A
@@ -5,7 +5,7 @@ import { mkdtemp, readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import os from "node:os";
import { pipeline } from "node:stream/promises";
import { Transform } from "node:stream";
import { Readable, Transform } from "node:stream";
import { and, eq, inArray, isNotNull, or, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRuns, issues, nativeHarnessBackups, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { createLocalEncryptedCipher } from "../../secrets/local-encrypted-provider.js";
@@ -14,6 +14,7 @@ import { lockNativeRunnerCheckpointOwner, validateNativeRunnerCheckpointOwner, t
import { loadConfig } from "../../config.js";
import { getStorageService } from "../../storage/index.js";
import { storageOutageWaitFor, waitOutStorageOutages } from "../../storage/storage-unavailable.js";
import { observeStorageProgress } from "../../storage/request-stats.js";
import { readNativeHarnessBackupStamp, verifyNativeHarnessBackupStamp } from "./native-harness-backup-stamp.js";
import { captureNativeHarnessArchive, restoreNativeHarnessArchive, MAX_NATIVE_HARNESS_OBJECT_BYTES } from "./native-harness-backup-archive.js";
import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js";
@@ -201,8 +202,13 @@ type PersistNativeHarnessBackupInput = {
providerArtifacts?: NativeProviderArtifactSource;
};
export function persistNativeHarnessBackup(input: PersistNativeHarnessBackupInput): Promise<void> {
return waitOutStorageOutages(() => storageOutageWaitFor("saving the provider session", () => persistNativeHarnessBackupOnce(input)));
export function persistNativeHarnessBackup(input: PersistNativeHarnessBackupInput & {
transferIdleTimeoutMs?: number; persistMaxMs?: number;
}): Promise<void> {
return withTransferProgressDeadline({ storage: input.storage, signal: input.signal, idleTimeoutMs: input.transferIdleTimeoutMs,
maxMs: input.persistMaxMs ?? NATIVE_HARNESS_PERSIST_MAX_MS, deadlineCode: NATIVE_HARNESS_PERSIST_DEADLINE_EXCEEDED },
(storage, signal) => waitOutStorageOutages(() => storageOutageWaitFor("saving the provider session",
() => persistNativeHarnessBackupOnce({ ...input, storage, signal }))));
}
async function persistNativeHarnessBackupOnce(input: PersistNativeHarnessBackupInput): Promise<void> {
@@ -298,39 +304,57 @@ async function persistNativeHarnessBackupOnce(input: PersistNativeHarnessBackupI
/** How long a durable read may go without progress. */
export const NATIVE_HARNESS_DOWNLOAD_IDLE_TIMEOUT_MS = 300_000;
/** How long a durable write may go without progress. */
export const NATIVE_HARNESS_UPLOAD_IDLE_TIMEOUT_MS = 300_000;
/** Outer bound for one whole restore or verification, including local steps. */
export const NATIVE_HARNESS_RESTORE_MAX_MS = 30 * 60_000;
/** Outer bound for one whole backup, including capture, encryption and read-back. */
export const NATIVE_HARNESS_PERSIST_MAX_MS = 30 * 60_000;
export const NATIVE_HARNESS_DOWNLOAD_STALLED = "native_harness_storage_download_stalled";
export const NATIVE_HARNESS_UPLOAD_STALLED = "native_harness_storage_upload_stalled";
export const NATIVE_HARNESS_RESTORE_DEADLINE_EXCEEDED = "native_harness_storage_restore_deadline_exceeded";
export const NATIVE_HARNESS_PERSIST_DEADLINE_EXCEEDED = "native_harness_storage_persist_deadline_exceeded";
/**
* Bound each durable read by inactivity, not by a short total time. A large or
* slow backup keeps restoring while bytes flow through the read; a read that
* makes no progress for the idle window aborts the whole restore with a named
* error. A generous outer bound keeps a hung local step diagnosable. Caller
* cancellation (Stop) keeps its own reason.
*/
async function withDownloadProgressDeadline<T>(
input: { storage: StorageService; signal?: AbortSignal; downloadIdleTimeoutMs?: number; restoreMaxMs?: number },
run: (storage: StorageService, signal: AbortSignal | undefined) => Promise<T>,
): Promise<T> {
const idleMs = input.downloadIdleTimeoutMs;
return withTransferProgressDeadline({ storage: input.storage, signal: input.signal, idleTimeoutMs: input.downloadIdleTimeoutMs,
maxMs: input.restoreMaxMs ?? Math.max(input.downloadIdleTimeoutMs ?? 0, NATIVE_HARNESS_RESTORE_MAX_MS),
deadlineCode: NATIVE_HARNESS_RESTORE_DEADLINE_EXCEEDED }, run);
}
/**
* Bound each durable read and write by inactivity, not by a short total time.
* A large or slow backup keeps moving while the store makes progress; a
* transfer that makes none for the idle window aborts the whole operation with
* a named error. A generous outer bound keeps a hung local step diagnosable.
* Caller cancellation (Stop) keeps its own reason.
*/
async function withTransferProgressDeadline<T>(
input: { storage: StorageService; signal?: AbortSignal; idleTimeoutMs?: number; maxMs: number; deadlineCode: string },
run: (storage: StorageService, signal: AbortSignal | undefined) => Promise<T>,
): Promise<T> {
const idleMs = input.idleTimeoutMs;
if (idleMs === undefined) return run(input.storage, input.signal);
if (!Number.isSafeInteger(idleMs) || idleMs <= 0) throw new Error("native_harness_storage_idle_timeout_invalid");
const stall = new AbortController();
const outer = setTimeout(() => stall.abort(new Error(NATIVE_HARNESS_RESTORE_DEADLINE_EXCEEDED)),
input.restoreMaxMs ?? Math.max(idleMs, NATIVE_HARNESS_RESTORE_MAX_MS));
const outer = setTimeout(() => stall.abort(new Error(input.deadlineCode)), input.maxMs);
outer.unref?.();
const signal = input.signal ? AbortSignal.any([input.signal, stall.signal]) : stall.signal;
const base = input.storage;
const getObject: StorageService["getObject"] = async (companyId, objectKey, options) => {
const idleTimer = (code: string) => {
let timer: NodeJS.Timeout | undefined;
const disarm = () => { if (timer) clearTimeout(timer); timer = undefined; };
const arm = () => {
disarm();
timer = setTimeout(() => stall.abort(new Error(NATIVE_HARNESS_DOWNLOAD_STALLED)), idleMs);
timer = setTimeout(() => stall.abort(new Error(code)), idleMs);
timer.unref?.();
};
return { arm, disarm };
};
const getObject: StorageService["getObject"] = async (companyId, objectKey, options) => {
const { arm, disarm } = idleTimer(NATIVE_HARNESS_DOWNLOAD_STALLED);
arm();
let object: Awaited<ReturnType<StorageService["getObject"]>>;
try {
@@ -348,10 +372,30 @@ async function withDownloadProgressDeadline<T>(
pipeline(object.stream, observed, { signal }).catch(() => disarm());
return { ...object, stream: observed };
};
const putFile: StorageService["putFile"] = async request => {
const { arm, disarm } = idleTimer(NATIVE_HARNESS_UPLOAD_STALLED);
const withSignal = { signal: request.signal ? AbortSignal.any([request.signal, signal]) : signal };
arm();
try {
// A provider that sends the file source itself (S3) never reads the
// stream; each request the store answers, part it takes or resend it
// waits for renews the window instead. After the last of them the window
// stays armed until the store answers, so a write that never completes
// is named too.
return await observeStorageProgress(arm, async () => {
if (!(request.body instanceof Readable)) return base.putFile({ ...request, ...withSignal });
// A provider that reads the stream renews the window with each chunk it takes.
const observed = new Transform({ transform(bytes, _encoding, callback) { arm(); callback(null, bytes); } });
pipeline(request.body, observed, { signal }).catch(() => undefined);
return base.putFile({ ...request, body: observed, ...withSignal } as typeof request);
});
} finally { disarm(); }
};
// Forward every other member to the real service so its own state stays intact.
const storage = new Proxy(base, {
get(target, property) {
if (property === "getObject") return getObject;
if (property === "putFile") return putFile;
const value = Reflect.get(target, property, target) as unknown;
return typeof value === "function" ? (value as (...args: unknown[]) => unknown).bind(target) : value;
},
@@ -0,0 +1,188 @@
import { createHash, randomBytes, randomUUID } from "node:crypto";
import { mkdir, readFile, stat, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { Readable } from "node:stream";
import { and, eq } from "drizzle-orm";
import { nativeHarnessBackups, nativeRunFinalizations } from "@paperclipai/db";
import { afterEach, expect, it, vi } from "vitest";
import { describeEmbeddedPostgres, nativeRestartFixture } from "../../__tests__/helpers/native-restart.js";
import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js";
import { createStorageService } from "../../storage/service.js";
import { recordStorageRequest } from "../../storage/request-stats.js";
import type { StorageService } from "../../storage/types.js";
import { nativeHarnessBackupOwner, persistNativeHarnessBackup, restoreDurableNativeHarnessBackup } from "./native-harness-backup-storage.js";
import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js";
/** A store that takes one upload chunk every `delayMs`, optionally no more after `stallAfter` chunks. */
function throttledUploads(storage: StorageService, delayMs: number, stallAfter?: number) {
let taken = 0;
const writes: string[] = [];
const throttled = new Proxy(storage, { get(target, property) {
if (property !== "putFile") {
const value = Reflect.get(target, property, target) as unknown;
return typeof value === "function" ? (value as (...args: unknown[]) => unknown).bind(target) : value;
}
return async (input: Parameters<StorageService["putFile"]>[0]) => {
writes.push(input.objectKey ?? "");
if (!(input.body instanceof Readable)) return target.putFile(input);
const body = input.body, signal = input.signal;
async function* slow() {
for await (const part of body as AsyncIterable<Buffer>) {
if (stallAfter !== undefined && taken >= stallAfter) {
await new Promise<void>(resolve => signal?.addEventListener("abort", () => resolve(), { once: true }));
signal?.throwIfAborted();
}
await new Promise(resolve => setTimeout(resolve, delayMs));
signal?.throwIfAborted();
taken++;
yield part;
}
}
return target.putFile({ ...input, body: Readable.from(slow()) } as typeof input);
};
} });
return { storage: throttled, writes };
}
/**
* A store that, like S3 with a file source, never reads the stream body: it
* sends `requests` requests of `delayMs` each, recording each answer as the S3
* provider does, then stores the file. With `stallAfter`, a request never ends.
*/
function fileSourceUploads(storage: StorageService, requests: number, delayMs: number, stallAfter?: number) {
return new Proxy(storage, { get(target, property) {
if (property !== "putFile") {
const value = Reflect.get(target, property, target) as unknown;
return typeof value === "function" ? (value as (...args: unknown[]) => unknown).bind(target) : value;
}
return async (input: Parameters<StorageService["putFile"]>[0]) => {
if (!("sourceFile" in input) || !input.sourceFile) throw new Error("expected a file source");
if (input.body instanceof Readable) input.body.destroy();
for (let sent = 0; sent < requests; sent++) {
if (stallAfter !== undefined && sent >= stallAfter) {
await new Promise<void>(resolve => input.signal?.addEventListener("abort", () => resolve(), { once: true }));
}
await new Promise(resolve => setTimeout(resolve, delayMs));
input.signal?.throwIfAborted();
recordStorageRequest({ $metadata: { attempts: 1 } }, false);
}
const { body: _stream, byteSize: _size, sha256: _sha, sourceFile, ...rest } = input;
return target.putFile({ ...rest, body: await readFile(sourceFile) });
};
} });
}
describeEmbeddedPostgres("durable native backup upload deadlines", () => {
const environment = nativeRestartFixture();
const ownedHomes: string[] = [];
afterEach(async () => {
for (const home of ownedHomes.splice(0)) await removeNativeHarnessBackup(home);
vi.restoreAllMocks();
vi.unstubAllEnvs();
});
/** A backup large enough that its upload is many store-sized chunks. */
async function fixture() {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "52".repeat(32));
const f = await environment.seedRun(randomUUID());
const { companyId, runtimeRoot, successor } = environment.current;
await f.db.update(nativeRunFinalizations).set({ controllerBootId: successor.bootId, controllerGeneration: 1,
leaseOwner: "backup-writer", leaseExpiresAt: new Date(Date.now() + 120_000) }).where(eq(nativeRunFinalizations.runId, f.runId));
const owner = await nativeHarnessBackupOwner(f.db, { companyId, issueId: f.issueId, runId: f.runId, bootId: successor.bootId });
const home = join(runtimeRoot, randomUUID()), root = join(home, "pending");
ownedHomes.push(home);
await mkdir(join(root, "acpx", "sessions"), { recursive: true });
const identity = { normalizedSessionId: randomUUID(), providerSessionId: randomUUID(), provider: "acpx" };
// Incompressible bytes, so the encrypted object spans many file-stream chunks.
await writeFile(join(root, "acpx", "sessions", "history.json"),
JSON.stringify({ ...identity, turns: randomBytes(768 * 1024).toString("base64") }));
await writeFile(join(root, "manifest.json"),
JSON.stringify({ schema: "paperclip.native-harness-backup.v1", ...identity, directories: ["acpx"] }), { mode: 0o600 });
const scopeDigest = createHash("sha256").update(randomUUID()).digest("hex");
const storage = createStorageService(createLocalDiskStorageProvider(join(runtimeRoot, "durable-objects")));
const current = async () => f.db.select().from(nativeHarnessBackups).where(and(
eq(nativeHarnessBackups.scopeDigest, scopeDigest), eq(nativeHarnessBackups.status, "current")));
return { db: f.db, companyId, owner, root, scopeDigest, storage, identity, current };
}
function replacementRoot() {
const root = join(environment.current.runtimeRoot, randomUUID(), "failover-backups");
ownedHomes.push(root);
return root;
}
it("keeps saving a backup whose upload outlasts the idle window while the store keeps taking bytes", async () => {
const f = await fixture(), idle = 200, slow = throttledUploads(f.storage, 40);
// The former fixed total deadline aborted this same progressing upload with
// the generic "The operation was aborted" seen after accepted staging turns.
await expect(persistNativeHarnessBackup({ ...f, storage: slow.storage, signal: AbortSignal.timeout(idle) }))
.rejects.toThrow("The operation was aborted");
expect(await f.current()).toHaveLength(0);
const startedAt = Date.now();
slow.writes.splice(0);
await persistNativeHarnessBackup({ ...f, storage: slow.storage, transferIdleTimeoutMs: idle });
expect(Date.now() - startedAt).toBeGreaterThan(idle * 2);
expect(slow.writes.some(key => key.includes("/native-harness-backups/"))).toBe(true);
expect(await f.current()).toHaveLength(1);
const restored = replacementRoot();
expect(await restoreDurableNativeHarnessBackup({ ...f, backupRoot: restored })).toBe(true);
expect(JSON.parse(await readFile(join(restored, "current/acpx/sessions/history.json"), "utf8"))).toMatchObject(f.identity);
});
it("names an upload that stops taking bytes and publishes no backup", { timeout: 10_000 }, async () => {
const f = await fixture();
await expect(persistNativeHarnessBackup({ ...f, storage: throttledUploads(f.storage, 5, 2).storage,
transferIdleTimeoutMs: 250 })).rejects.toThrow("native_harness_storage_upload_stalled");
expect(await f.current()).toHaveLength(0);
// The local source is untouched and saves once the store responds again.
await persistNativeHarnessBackup({ ...f, transferIdleTimeoutMs: 250 });
expect(await f.current()).toHaveLength(1);
});
it("keeps a file-source upload going while the store answers its requests, and names one that stops", { timeout: 10_000 }, async () => {
const f = await fixture(), idle = 200;
// Ten requests of 60 ms each: longer than the idle window in total, but each answer renews it.
const startedAt = Date.now();
await persistNativeHarnessBackup({ ...f, storage: fileSourceUploads(f.storage, 10, 60), transferIdleTimeoutMs: idle });
expect(Date.now() - startedAt).toBeGreaterThan(idle * 2);
expect(await f.current()).toHaveLength(1);
await expect(persistNativeHarnessBackup({ ...f, storage: fileSourceUploads(f.storage, 10, 20, 3),
transferIdleTimeoutMs: idle })).rejects.toThrow("native_harness_storage_upload_stalled");
});
it("names a read-back that stops sending bytes and publishes no backup", { timeout: 10_000 }, async () => {
const f = await fixture();
const stalledReads = new Proxy(f.storage, { get(target, property) {
if (property === "getObject") return async (companyId: string, objectKey: string, options?: { signal?: AbortSignal }) => {
const object = await target.getObject(companyId, objectKey, options);
object.stream.destroy();
return { ...object, stream: Readable.from((async function* () {
await new Promise<void>(resolve => options?.signal?.addEventListener("abort", () => resolve(), { once: true }));
options?.signal?.throwIfAborted();
})()) };
};
const value = Reflect.get(target, property, target) as unknown;
return typeof value === "function" ? (value as (...args: unknown[]) => unknown).bind(target) : value;
} });
await expect(persistNativeHarnessBackup({ ...f, storage: stalledReads, transferIdleTimeoutMs: 250 }))
.rejects.toThrow("native_harness_storage_download_stalled");
expect(await f.current()).toHaveLength(0);
});
it("stops at the outer bound while an upload still progresses", async () => {
const f = await fixture();
await expect(persistNativeHarnessBackup({ ...f, storage: throttledUploads(f.storage, 40).storage,
transferIdleTimeoutMs: 5_000, persistMaxMs: 150 })).rejects.toThrow("native_harness_storage_persist_deadline_exceeded");
expect(await f.current()).toHaveLength(0);
});
it("lets Stop end a progressing upload with its own reason", async () => {
const f = await fixture(), stop = new AbortController();
setTimeout(() => stop.abort(new Error("caller_stopped")), 80);
const failure = await persistNativeHarnessBackup({ ...f, storage: throttledUploads(f.storage, 40).storage,
signal: stop.signal, transferIdleTimeoutMs: 5_000 }).then(() => null, (error: unknown) => error);
const reason = failure as Error & { cause?: unknown };
expect(failure).toBeInstanceOf(Error);
expect([reason.message, (reason.cause as Error | undefined)?.message]).toContain("caller_stopped");
expect(await f.current()).toHaveLength(0);
await expect(stat(f.root)).resolves.toBeTruthy();
});
});
@@ -3188,6 +3188,10 @@ describe("remote provider pack manifest", () => {
await driver.mock.calls.at(-1)![0].checkpoint!("settled");
expect(providerStateUnsaved.mark).not.toHaveBeenCalled();
expect(harnessStorage.persistNativeHarnessBackup).toHaveBeenCalledTimes(1);
// The per-turn backup is bounded by transfer inactivity, not a fixed total time.
const [persisted] = vi.mocked(harnessStorage.persistNativeHarnessBackup).mock.calls[0]!;
expect(persisted.transferIdleTimeoutMs).toBe(harnessStorage.NATIVE_HARNESS_UPLOAD_IDLE_TIMEOUT_MS);
expect(persisted.signal).toBeUndefined();
expect(checkpointReadResets).toBe(0);
expect(inspectionResets).toBe(0);
return;
@@ -23,8 +23,8 @@ import { readClaimedNativeStopRecovery } from "./native-cancellation-request.js"
import { publishNativeHarnessBackup } from "./native-harness-backup-publication.js";
import { restartRecoveryMeasurementFor, speedBudgetAdapter } from "./native-speed-budgets.js";
import { getStartupRecoveryState } from "../../startup-recovery-state.js";
import { assertNativeHarnessBackupOwner, durableNativeHarnessStorage, NATIVE_HARNESS_DOWNLOAD_IDLE_TIMEOUT_MS, nativeHarnessBackupOwner, persistNativeHarnessBackup,
restoreDurableNativeHarnessBackup } from "./native-harness-backup-storage.js";
import { assertNativeHarnessBackupOwner, durableNativeHarnessStorage, NATIVE_HARNESS_DOWNLOAD_IDLE_TIMEOUT_MS, NATIVE_HARNESS_UPLOAD_IDLE_TIMEOUT_MS,
nativeHarnessBackupOwner, persistNativeHarnessBackup, restoreDurableNativeHarnessBackup } from "./native-harness-backup-storage.js";
import { restoreNativeProviderArtifacts, retainNativeProviderArtifacts, retainExistingNativeProviderArtifacts,
retainReadyNativeProviderArtifacts, NativeProviderArtifactTransferTimeoutError, nativeProviderArtifactTransferBounds,
type NativeProviderArtifactReceipt } from "./native-provider-artifacts.js";
@@ -18395,7 +18395,7 @@ async function createRunnerdBackendWithinSessionClaim(
if (durableStorage && backupOwner) {
await persistNativeHarnessBackup({ db: input.db, storage: durableStorage, owner: backupOwner,
scopeDigest: createHash("sha256").update(sessionScopeId).digest("hex"), root: pendingRoot,
signal: AbortSignal.timeout(300_000) });
transferIdleTimeoutMs: NATIVE_HARNESS_UPLOAD_IDLE_TIMEOUT_MS });
}
// Without durable storage the local publication below is the
// save; its result is uncertain once it starts, so keep the
+19
View File
@@ -36,6 +36,22 @@ export function collectStorageRequestStats<T>(stats: StorageRequestStats, work:
return current.run(stats, work);
}
const progress = new AsyncLocalStorage<() => void>();
/**
* Runs `work` calling `onProgress` each time the store answers a request,
* takes a multipart part, or a resend starts its wait. A caller uses it to
* tell a slow transfer from a stalled one when the provider sends a file
* source itself and no stream chunk is visible.
*/
export function observeStorageProgress<T>(onProgress: () => void, work: () => Promise<T>): Promise<T> {
return progress.run(onProgress, work);
}
function noteProgress(): void {
try { progress.getStore()?.(); } catch { return; }
}
function metadataOf(value: unknown): { attempts?: unknown; totalRetryDelay?: unknown } | undefined {
if (!value || typeof value !== "object") return undefined;
const metadata = (value as { $metadata?: unknown }).$metadata;
@@ -47,6 +63,7 @@ const count = (value: unknown, fallback: number) =>
/** Records one S3 request's outcome from its SDK response or error metadata. */
export function recordStorageRequest(outcome: unknown, failed: boolean): void {
noteProgress();
const stats = current.getStore();
if (!stats) return;
const metadata = metadataOf(outcome);
@@ -57,6 +74,7 @@ export function recordStorageRequest(outcome: unknown, failed: boolean): void {
}
export function recordStoragePart(): void {
noteProgress();
const stats = current.getStore();
if (stats) stats.parts += 1;
}
@@ -69,6 +87,7 @@ export function recordStorageThrottle(): void {
/** Records a wait before resending a part. */
export function recordStorageBackoff(waitMs: number): void {
noteProgress();
const stats = current.getStore();
if (stats) stats.backoffMs += count(waitMs, 0);
}