fix(native): resend a throttled harness backup upload instead of losing it (#20772)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - After a native run in a per-turn remote sandbox, the server saves a
durable harness backup to object storage. The next run restores the
provider session from it.
> - On staging, two OpenCode runs had an accepted result, but
`harness_state.backup.persist` failed. The run log for both says `remote
checkpoint failed: Please reduce your request rate.` This is the S3
`SlowDown` (HTTP 503) answer.
> - The backup and the OpenCode provider-artifact bundle were uploaded
as one stream `PUT`. The S3 SDK cannot send a stream body again, and
this path had no retry for a request-rate limit. One `SlowDown` lost the
backup.
> - The sandbox was then released. The follow-up run had no state to
resume and failed with `PRP provider resume state is unavailable` (class
P, see #20757).
> - This pull request uploads both objects from their encrypted scratch
file, as workspace snapshots already do. The existing throttled-retry
path then sends them again.
> - The benefit is that a short request-rate limit no longer loses the
only copy of a provider session.

## Linked Issues or Issue Description

Refs #20757 (the failure this causes on the next run). Refs #19917
(added the file-source upload path that this change uses).

**What happened?**

1. An OpenCode run in a per-turn Daytona sandbox finished, and its
result was accepted.
2. The server uploaded the encrypted harness backup with
`storage.putFile` and a stream body.
3. Object storage answered `503 SlowDown` ("Please reduce your request
rate."). Nothing sent the request again.
4. `harness_state.backup.persist` failed, and
`native.post_result_failure` was recorded.
5. The next run for the same chat task started in a new sandbox and
could not resume the provider session.

**Expected behavior**

A request-rate limit slows the upload down. It does not lose the backup.

**Steps to reproduce**

1. Point the storage service at an S3 endpoint that answers the first
`PUT` of each object with `503 SlowDown`.
2. Call `persistNativeHarnessBackup`.
3. On `master` it throws `SlowDown: Please reduce your request rate.`

**Paperclip version or commit**

`master` at `9574065a28`.

## What Changed

- `server/src/services/native-runtime/native-harness-backup-storage.ts`:
pass the encrypted scratch file as `sourceFile` to `putFile`.
- `server/src/services/native-runtime/native-provider-artifacts.ts`: the
same for the provider-artifact bundle.
- `native-harness-backup-throttle.integration.test.ts` (new, focused
file): a real S3 client against a local endpoint that throttles the
first `PUT` of each object once. Cases for a Codex backup, and for an
OpenCode backup with provider artifacts. Each case also throttles the
first read-back of each object once. Each case checks that the backup is
current and restores the provider session bytes.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-harness-backup-throttle.integration.test.ts`:
both cases fail on `master` with `SlowDown: Please reduce your request
rate.` and pass with this change.
- `cd server && npx vitest run
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
68 pass (file unchanged).
- `cd server && npx tsc --noEmit -p .`: clean.
- `pnpm verify`: it selected the full server suite and failed 16 of
34,997 tests, all local-environment failures unrelated to object
storage: missing `.claude/skills/*/SKILL.md` files in this worktree,
`ENAMETOOLONG` under the macOS temp directory, and 30 s hook timeouts
under load. Required CI is the authority for that suite.

## Risks

- Low. A file source also sends a `Content-MD5` header, so the backend
rejects any byte it did not receive intact. Workspace snapshots have
used this path since #19917.
- An upload above 16 MiB now uses the file multipart path. That path has
the same verified-part retry.
- The read-back check (`getObject`) needs no change: a GET has no body,
so the S3 client resends it after `SlowDown`. The test now also
throttles the first read of each object once and proves the backup still
publishes.
- The Rust Runner is unchanged.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with
tool use (shell, file edit, test execution) and read-only staging API
reads.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>

server/src/services/native-runtime/native-harness-backup-storage.ts

@@ -242,7 +242,9 @@ export async function persistNativeHarnessBackup(input: {
     });
     await input.storage.putFile({ companyId: ref.companyId, namespace: "native-harness-backups", objectKey: ref.objectKey,
       originalFilename: null, contentType: "application/octet-stream", body: createReadStream(encrypted),
-      byteSize: ref.byteSize, sha256: ref.sha256, signal: input.signal });
+      // The file source lets a request-rate limit (503 SlowDown) or a dropped
+      // connection resend the upload; a stream body can be sent only once.
+      byteSize: ref.byteSize, sha256: ref.sha256, sourceFile: encrypted, signal: input.signal });
     await receive(input.storage, ref, undefined, input.signal);
     await input.db.transaction(async tx => {
       await fence(tx, input.owner, inactiveSource, input.signal);

server/src/services/native-runtime/native-harness-backup-throttle.integration.test.ts

@@ -0,0 +1,145 @@
+import { createHash, randomUUID } from "node:crypto";
+import { chmod, mkdir, readFile, writeFile } from "node:fs/promises";
+import { createServer } from "node:http";
+import type { AddressInfo } from "node:net";
+import { join } from "node:path";
+import { and, eq } from "drizzle-orm";
+import { heartbeatRuns, nativeHarnessBackups, nativeRunFinalizations } from "@paperclipai/db";
+import { afterEach, expect, it, vi } from "vitest";
+import { describeEmbeddedPostgres, nativeRestartFixture } from "../../__tests__/helpers/native-restart.js";
+import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js";
+import { createS3StorageProvider } from "../../storage/s3-provider.js";
+import { createStorageService } from "../../storage/service.js";
+import { nativeHarnessBackupOwner, persistNativeHarnessBackup, restoreDurableNativeHarnessBackup } from "./native-harness-backup-storage.js";
+import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js";
+import { canonicalProviderPackJson } from "./remote-provider-pack-verification.js";
+
+// Durable harness backups against an S3 endpoint that throttles requests.
+describeEmbeddedPostgres("durable native provider backups under an object-storage request-rate limit", () => {
+  const environment = nativeRestartFixture();
+  const ownedHomes: string[] = [];
+  afterEach(async () => {
+    for (const home of ownedHomes.splice(0)) await removeNativeHarnessBackup(home);
+    vi.unstubAllEnvs();
+  });
+  async function fixture(provider = "codex") {
+    vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "51".repeat(32));
+    const f = await environment.seedRun(randomUUID());
+    const { companyId, runtimeRoot, successor } = environment.current;
+    await f.db.update(nativeRunFinalizations).set({ controllerBootId: successor.bootId, controllerGeneration: 1,
+      leaseOwner: "backup-writer", leaseExpiresAt: new Date(Date.now() + 120_000) }).where(eq(nativeRunFinalizations.runId, f.runId));
+    const owner = await nativeHarnessBackupOwner(f.db, { companyId, issueId: f.issueId, runId: f.runId, bootId: successor.bootId });
+    const home = join(runtimeRoot, randomUUID()), root = join(home, "failover-backups", "current");
+    ownedHomes.push(home);
+    await mkdir(join(root, provider, "sessions"), { recursive: true });
+    await mkdir(join(root, "runner"));
+    const identity = { normalizedSessionId: randomUUID(), providerSessionId: randomUUID(), provider };
+    await writeFile(join(root, provider, "sessions", "history.json"), JSON.stringify(identity), { mode: 0o600 });
+    await writeFile(join(root, "runner", "runner-state.json"), JSON.stringify({ runId: f.runId, lifecycle: "suspended" }), { mode: 0o600 });
+    const manifest = JSON.stringify({ schema: "paperclip.native-harness-backup.v1", ...identity, directories: [provider, "runner"] });
+    await writeFile(join(root, "manifest.json"), manifest, { mode: 0o600 });
+    await chmod(join(root, provider, "sessions"), 0o500);
+    const sessionScopeId = randomUUID(), scopeDigest = createHash("sha256").update(sessionScopeId).digest("hex");
+    const storage = createStorageService(createLocalDiskStorageProvider(join(runtimeRoot, "durable-objects")));
+    return { ...f, home, root, companyId, owner, scopeDigest, sessionScopeId, storage, identity,
+      manifestSha256: `sha256:${createHash("sha256").update(manifest).digest("hex")}` };
+  }
+  async function artifactFixture(artifactVersion = randomUUID()) {
+    const f = await fixture("opencode");
+    const sourceRoot = join(f.home, "qualified-provider-pack");
+    await mkdir(sourceRoot);
+    const original = { nodeCommand: "qualified Node executable", opencodeExecutable: "qualified OpenCode executable",
+      opencodeProxy: `qualified original proxy bytes ${artifactVersion}` };
+    const artifacts = Object.fromEntries(await Promise.all(Object.entries(original).map(async ([name, bytes]) => {
+      await writeFile(join(sourceRoot, name), bytes, { mode: 0o500 });
+      return [name, { path: name, sha256: `sha256:${createHash("sha256").update(bytes).digest("hex")}` }];
+    })));
+    const payload = { runnerSourceRevision: "a".repeat(40), target: { platform: "linux", architecture: "x64" },
+      pins: { opencode: "1.18.34" }, artifacts: { ...artifacts, opencodeCommand: artifacts.opencodeExecutable } };
+    const manifest = { schema: "paperclip-runner/remote-provider-pack/v1", payload,
+      digest: `sha256:${createHash("sha256").update(canonicalProviderPackJson(payload)).digest("hex")}` };
+    const launchRoot = `/home/daytona/.paperclip-runtime/paperclip-runner/provider-artifacts/${manifest.digest.slice(7)}`;
+    await f.db.update(heartbeatRuns).set({ nativeSessionId: f.identity.normalizedSessionId, driverKind: "opencode_server",
+      runnerProfileJson: { ...f.preparedProfile, nativeProviderPackAdmissionState: {
+        schema: "paperclip.native-provider-pack-admission-state.v1", state: "verified", admissionEpoch: randomUUID() },
+      nativeProviderPackAdmission: { schema: "paperclip.native-provider-pack-admission.v1", stage: "session_admitted",
+        authority: "core_verified_selection", companyId: f.companyId, agentId: environment.current.agentId,
+        runId: f.runId, sessionId: f.identity.normalizedSessionId, runnerInstanceId: "artifact-fixture-runner",
+        runnerEnvironmentLeaseId: "artifact-fixture-lease", providerPackDigest: manifest.digest,
+        opencodeProxySha256: artifacts.opencodeProxy.sha256, sourceRevision: payload.runnerSourceRevision } },
+    }).where(eq(heartbeatRuns.id, f.runId));
+    return { ...f, original, manifest, launchRoot, providerArtifacts: { sourceRoot, manifest, launchRoot, accountUid: 1000 } };
+  }
+  // Staging: two OpenCode per-turn runs had an accepted result, but object
+  // storage answered the backup upload with one 503 SlowDown ("Please reduce
+  // your request rate."). The upload was a single stream PUT that nothing
+  // retried, so the only provider-session backup was lost when the sandbox was
+  // released, and the follow-up could not resume.
+  it.each(["Codex backup", "OpenCode backup with provider artifacts"])("backs off and publishes when object storage throttles each %s upload and read-back once", async label => {
+    const f = label.startsWith("OpenCode") ? await artifactFixture() : await fixture();
+    const objects = new Map<string, Buffer>(), throttled = new Set<string>(), throttledReads = new Set<string>(); let puts = 0, reads = 0;
+    const server = createServer((request, response) => {
+      const key = new URL(request.url!, "http://localhost").pathname;
+      const chunks: Buffer[] = [];
+      request.on("data", chunk => chunks.push(chunk));
+      request.on("end", () => {
+        if (request.method === "PUT") {
+          puts++;
+          if (!throttled.has(key)) {
+            throttled.add(key);
+            response.writeHead(503, { "Content-Type": "application/xml" }).end(
+              "<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><RequestId>throttled</RequestId></Error>");
+            return;
+          }
+          objects.set(key, Buffer.concat(chunks));
+          response.writeHead(200, { ETag: `"${createHash("md5").update(objects.get(key)!).digest("hex")}"` }).end();
+        } else if (request.method === "GET" || request.method === "HEAD") {
+          const body = objects.get(key);
+          if (request.method === "GET" && body) {
+            reads++;
+            if (!throttledReads.has(key)) {
+              throttledReads.add(key);
+              response.writeHead(503, { "Content-Type": "application/xml" }).end(
+                "<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><RequestId>throttled-read</RequestId></Error>");
+              return;
+            }
+          }
+          if (!body) { response.writeHead(404, { "Content-Type": "application/xml" }).end("<Error><Code>NoSuchKey</Code></Error>"); return; }
+          response.writeHead(200, { "Content-Length": body.length, ETag: `"${createHash("md5").update(body).digest("hex")}"`,
+            "Content-Type": "application/octet-stream" }).end(request.method === "GET" ? body : undefined);
+        } else if (request.method === "DELETE") {
+          objects.delete(key); response.writeHead(204).end();
+        } else response.writeHead(500).end();
+      });
+    });
+    await new Promise<void>((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); });
+    vi.stubEnv("AWS_ACCESS_KEY_ID", "fixture"); vi.stubEnv("AWS_SECRET_ACCESS_KEY", "fixture");
+    const random = vi.spyOn(Math, "random").mockReturnValue(0);
+    const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
+    const storage = createStorageService(createS3StorageProvider({ bucket: "bucket", region: "us-east-1", forcePathStyle: true,
+      endpoint: `http://127.0.0.1:${(server.address() as AddressInfo).port}` }));
+    try {
+      await persistNativeHarnessBackup({ ...f, storage });
+      // Every object was refused once and then stored: the harness archive, and
+      // for OpenCode the provider artifact bundle too.
+      expect(throttled.size).toBe(label.startsWith("OpenCode") ? 2 : 1);
+      expect(puts).toBe(2 * throttled.size);
+      expect(objects.size).toBe(throttled.size);
+      // The verification read after each upload was refused once and repeated
+      // (the S3 client already resends a bodiless GET after SlowDown).
+      expect(throttledReads.size).toBe(throttled.size);
+      expect(reads).toBe(2 * throttled.size);
+      const [current] = await f.db.select().from(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest),
+        eq(nativeHarnessBackups.status, "current")));
+      expect(current).toBeDefined();
+      const replacement = join(environment.current.runtimeRoot, randomUUID(), "restored-throttled"); ownedHomes.push(replacement);
+      expect(await restoreDurableNativeHarnessBackup({ ...f, storage, backupRoot: replacement })).toBe(true);
+      const providerDirectory = label.startsWith("OpenCode") ? "opencode" : "codex";
+      expect(JSON.parse(await readFile(join(replacement, "current", providerDirectory, "sessions", "history.json"), "utf8"))).toEqual(f.identity);
+    } finally {
+      random.mockRestore(); warn.mockRestore();
+      server.closeAllConnections();
+      await new Promise<void>(resolve => server.close(() => resolve()));
+    }
+  });
+});

server/src/services/native-runtime/native-provider-artifacts.ts

@@ -300,6 +300,8 @@ export async function retainNativeProviderArtifacts(input: {
         body: createReadStream(encrypted),
         byteSize: reference.byteSize,
         sha256: reference.sha256.slice(7),
+        // A file source can be resent after a request-rate limit; a stream cannot.
+        sourceFile: encrypted,
         signal: input.signal,
       });
       await receive(