From 1b39cf1794477a9b0d2af50445ef37315ea4e8e1 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 10:28:55 -0600 Subject: [PATCH] fix(native): retry a contended provider-pack admission write (#20850) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native OpenCode run on a remote sandbox must save a provider-pack admission receipt on its run row after the provider session opens. Artifact retention refuses to continue without it. > - The admission write locks the coordinator row (`native_run_finalizations`) and then the run row, each with a 1000 ms `lock_timeout`. The runner's durable checkpoint writes lock the same coordinator row for the whole checkpoint transaction. > - On a loaded database one checkpoint transaction can outlast one lock wait. The admission write then fails with `55P03`, the state stays `pending`, and artifact retention fails the run as `native_provider_artifacts_admission_unverified` before the first turn. The task rests blocked. > - #20682 fixed the same symptom for a run-event append on the run row. This is the coordinator-row half. > - This pull request retries a contended admission write a bounded number of times, each in a new transaction that re-proves the claim. It also names the database cause in the run log and adds a chaos option that makes every fresh sandbox take the pack upload and admission path. > - The benefit is that a healthy fresh sandbox is no longer failed and blocked by ordinary lock contention. ## Linked Issues or Issue Description Refs #20682 (same symptom, run-row lock; this is the coordinator-row lock). **What happened?** On a hosted staging instance, an OpenCode run on a fresh Daytona sandbox found no matching preinstalled provider pack (`provider_pack.select` reason `cached_packs_unverified`). It uploaded and verified the pack (61 s), started the runner, and the runner authenticated to the durable control plane. 3.4 s later the run log shows `native provider-pack admission receipt could not be saved`. `nativeProviderPackAdmissionState.state` stayed `pending`. The run failed with `native_provider_artifacts_admission_unverified` before any provider turn. The task rests `blocked` with no Retry. **Steps to reproduce** 1. Start a native run that waits for its claim (`waitForClaim: true`, as a fresh run with artifact storage does). 2. Hold its `native_run_finalizations` row `FOR UPDATE` in another transaction for 1.6 s, as a checkpoint write does. 3. Call `recordNativeProviderPackAdmission`. It fails with `canceling statement due to lock timeout` (`55P03`). **Expected behavior** The admission waits out the checkpoint write within a bound, re-proves the claim, and saves the receipt. ## What Changed - `recordNativeProviderPackAdmission` retries a write that waits for its claim when it fails with `55P03`, `40P01` or `40001`, up to 5 attempts with 100–400 ms back-off. Each attempt is a new transaction with the same checks (claim, run, epoch, receipt). A stale claim still returns `false`. Other errors still throw. A write that does not wait for its claim keeps its `NOWAIT` refusal. - New `nativeProviderPackAdmissionFailureCode` reads the SQLSTATE from the error or its cause. - The executor's run-log line now ends with `(cause=)`, so a lost admission is diagnosable. It carries no paths or secrets. - Restart chaos: `--daytona-pack-mismatch ` starts sandboxes from another image while the controller keeps `--daytona-image`'s runnerd and pack. Every fresh sandbox then takes the upload and admission path, which the matched-image check made unreachable. Config validation and README updated. - `doc/execution-semantics.md` describes the bounded retry. ## Verification - RED on `origin/master` (`1d67e309e0`): `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-provider-pack-admission.test.ts -t "checkpoint write holds"` fails with `canceling statement due to lock timeout` (`55P03`), the staging error. - GREEN: the same file, 39/39 pass, including the existing fence tests (`cannot publish while another transaction holds the coordinator claim`, revoked-claim recheck) and the new bound test (`gives up after the bounded retry window while the claim stays held`, under 15 s). - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/scheduler.test.ts`: 34/34 pass, including the new option test. - `pnpm --filter @paperclipai/server exec tsc --noEmit` and `tsc --noEmit -p tests/restart-chaos/tsconfig.json` pass. - Not run: a Daytona chaos wave with `--daytona-pack-mismatch`. It needs a runner image built at this revision. The staging cause is inferred from the timing and the code path; the original error was swallowed, which is why this PR adds the cause to the run log. ## Risks - Low risk. The normal path has no contention and makes one attempt. Under contention the write waits at most about 11 s before the first provider turn. Without this change that case failed the run. - A retry cannot widen authority: every attempt re-checks the coordinator claim, lease expiry, run status and admission epoch. - When admission still fails after the retries, the run fails as before. This PR does not change that recovery path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 8 +++- .../native-provider-pack-admission.test.ts | 44 +++++++++++++++++++ .../native-provider-pack-admission.ts | 42 +++++++++++++++++- .../native-runtime/native-session-executor.ts | 10 +++-- tests/restart-chaos/README.md | 13 ++++++ tests/restart-chaos/config.ts | 13 +++++- tests/restart-chaos/daytona-image.test.ts | 15 ++++++- tests/restart-chaos/daytona-image.ts | 13 ++++++ tests/restart-chaos/main.ts | 14 +++++- tests/restart-chaos/scheduler.test.ts | 8 ++++ tests/restart-chaos/types.ts | 5 +++ 11 files changed, 173 insertions(+), 12 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index e84dd070334..08154d6ddbd 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -4926,8 +4926,12 @@ and identify `inheritedFromRunId`. The first receipt is immutable history. `nativeProviderPackAdmissionState` separately describes current admission proof: `pending`, `verified` or `unknown`, with an opaque admission epoch. The mandatory coordinator claim atomically sets a new pending epoch before session admission. -Only the matching owner and epoch can publish post-attachment proof. Failed -receipt writes leave pending evidence; changed or unsupported admission leaves +Only the matching owner and epoch can publish post-attachment proof. A write +that waits for its claim retries a contended write (lock timeout, deadlock or +serialization abort) up to five times, because the runner's durable checkpoint +writes hold the same coordinator row. Each attempt re-proves the claim, run and +epoch in a new transaction. The run log names the database cause of a write +that still fails. Failed receipt writes leave pending evidence; changed or unsupported admission leaves unknown evidence and does not prevent authorized execution. A missing legacy receipt remains unknown; the current release cannot supply that missing history. Retained recovery needs positively verified prior provenance, captured before diff --git a/server/src/services/native-runtime/native-provider-pack-admission.test.ts b/server/src/services/native-runtime/native-provider-pack-admission.test.ts index 8ac60525f12..90a421485f9 100644 --- a/server/src/services/native-runtime/native-provider-pack-admission.test.ts +++ b/server/src/services/native-runtime/native-provider-pack-admission.test.ts @@ -257,6 +257,50 @@ describe("provider pack admission PostgreSQL fencing", () => { expect((await read(f.identity.runId)).runnerProfileJson?.nativeProviderPackAdmissionState) .toEqual({ ...f.profile.nativeProviderPackAdmissionState, state: "verified" }); }); + it("keeps the admission when a checkpoint write holds the claim longer than one lock wait", async () => { + // Staging run 3f317172 (OpenCode, fresh Daytona sandbox): the pack upload + // and runner start succeeded, the runner began writing durable checkpoints, + // and the admission write lost one 1000 ms lock wait on the coordinator row + // a checkpoint transaction held. The state stayed pending, artifact + // retention failed the run as native_provider_artifacts_admission_unverified + // and the task rested blocked before its first turn. + const f = await fixture(); + let release!: () => void, acquired!: () => void; + const holding = new Promise(resolve => { acquired = resolve; }); + const gate = new Promise(resolve => { release = resolve; }); + const checkpoint = db.transaction(async tx => { + await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.identity.runId)).for("update"); + acquired(); await gate; + }); + await holding; + const recording = recordNativeProviderPackAdmission(db, { ...f, waitForClaim: true } as Parameters[1]) + .then(value => ({ value }), error => ({ error })); + // Longer than one lock wait, shorter than the bounded retry window. + const held = setTimeout(release, 1_600); + try { + expect(await recording).toEqual({ value: true }); + } finally { clearTimeout(held); release(); await checkpoint; } + expect((await read(f.identity.runId)).runnerProfileJson?.nativeProviderPackAdmissionState) + .toEqual({ ...f.profile.nativeProviderPackAdmissionState, state: "verified" }); + }); + it("gives up after the bounded retry window while the claim stays held", async () => { + const f = await fixture(); + let release!: () => void, acquired!: () => void; + const holding = new Promise(resolve => { acquired = resolve; }); + const gate = new Promise(resolve => { release = resolve; }); + const checkpoint = db.transaction(async tx => { + await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.identity.runId)).for("update"); + acquired(); await gate; + }); + await holding; + const started = Date.now(); + try { + await expect(recordNativeProviderPackAdmission(db, { ...f, waitForClaim: true } as Parameters[1])) + .rejects.toMatchObject({ cause: { code: "55P03" } }); + expect(Date.now() - started).toBeLessThan(15_000); + } finally { release(); await checkpoint; } + expect((await read(f.identity.runId)).runnerProfileJson).toEqual(f.profile); + }, 30_000); it.each([false, true])("waits within the durable-capture bound and rechecks the claim after its lock (revoked: %s)", async revoked => { const f = await fixture(); let release!: () => void, acquired!: () => void; diff --git a/server/src/services/native-runtime/native-provider-pack-admission.ts b/server/src/services/native-runtime/native-provider-pack-admission.ts index d815e427f63..c5a31b4345f 100644 --- a/server/src/services/native-runtime/native-provider-pack-admission.ts +++ b/server/src/services/native-runtime/native-provider-pack-admission.ts @@ -123,13 +123,51 @@ export function admittedNativeProviderPack(session: NativeSession, identity: Nat ...(admittedRunId === actual.runId ? {} : { inheritedFromRunId: admittedRunId }) }; } -export async function recordNativeProviderPackAdmission(db: Db, input: { +type AdmissionInput = { session: NativeSession; identity: NativeRunIdentity; admissionEpoch: string; claim: { leaseOwner: string; attempt: number; controllerGeneration: number; controllerBootId: string }; waitForClaim?: boolean; -}): Promise { +}; + +/** Lock timeouts, deadlocks and serialization aborts that clear on their own. */ +const CONTENDED_ADMISSION_CODES = new Set(["55P03", "40P01", "40001"]); +/** Attempts of one admission write while its claim is contended. */ +export const NATIVE_PROVIDER_PACK_ADMISSION_ATTEMPTS = 5; + +export function nativeProviderPackAdmissionFailureCode(error: unknown): string | null { + for (let value = error, depth = 0; value && typeof value === "object" && depth < 4; depth += 1) { + const code = (value as { code?: unknown }).code; + if (typeof code === "string" && /^[0-9A-Z]{5}$/.test(code)) return code; + value = (value as { cause?: unknown }).cause; + } + return null; +} + +/** + * Saves the session's verified provider-pack admission on its run row. A run + * that waits for its claim retries a contended write: the runner's durable + * checkpoint writes hold the same coordinator row, and on a loaded database one + * can outlast a single lock wait. Each attempt is a new transaction that + * re-proves the claim, the run and the admission epoch, so a retry never + * admits anything the first attempt would have refused. Only contention is + * retried; a stale claim still returns false and other errors still throw. + */ +export async function recordNativeProviderPackAdmission(db: Db, input: AdmissionInput): Promise { + for (let attempt = 1; ; attempt += 1) { + try { + return await recordNativeProviderPackAdmissionOnce(db, input); + } catch (error) { + const code = nativeProviderPackAdmissionFailureCode(error); + if (!input.waitForClaim || attempt >= NATIVE_PROVIDER_PACK_ADMISSION_ATTEMPTS + || !code || !CONTENDED_ADMISSION_CODES.has(code)) throw error; + await new Promise(resolve => setTimeout(resolve, 100 * attempt)); + } + } +} + +async function recordNativeProviderPackAdmissionOnce(db: Db, input: AdmissionInput): Promise { const receipt = admittedNativeProviderPack(input.session, input.identity); return db.transaction(async tx => { // Bounds every lock wait below, including the run-row lock. A run-event diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 95019c2a012..cfa74418b02 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -44,7 +44,7 @@ import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export- import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; import { providerPackWorkspaceId, recordAdmittedRemoteProviderPack, resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js"; -import { nativeProviderPackAdmissionPending, rememberAdmittedNativeProviderPack, rememberRecoveredNativeProviderPack, recordNativeProviderPackAdmission, type NativeProviderPackAdmissionSource } from "./native-provider-pack-admission.js"; +import { nativeProviderPackAdmissionFailureCode, nativeProviderPackAdmissionPending, rememberAdmittedNativeProviderPack, rememberRecoveredNativeProviderPack, recordNativeProviderPackAdmission, type NativeProviderPackAdmissionSource } from "./native-provider-pack-admission.js"; import { type ProviderPackInstallation, verifyRemoteProviderPack as verifyRemoteProviderPackIdentity } from "./remote-provider-pack-verification.js"; import { writeEnvironmentCleanup } from "../environment-cleanup-notifications.js"; import { NATIVE_SESSION_EXECUTION_LEASE_TTL_MS, lockNativeExecutionLeaseForReacquire, nativeExecutionLeaseHeldCondition, releaseUnstartedNativeExecutionClaim, startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal } from "./native-execution-lease.js"; @@ -12075,8 +12075,12 @@ async function executePaperclipNativeSessionAttempt( claim: { leaseOwner, attempt, controllerGeneration, controllerBootId: controller.bootId }, waitForClaim: runnerdBackend?.providerAdmissionRequired?.() === true, }); - } catch { - try { await input.onLog?.("stderr", "[paperclip-runner] native provider-pack admission receipt could not be saved\n"); } catch {} + } catch (error) { + // Name the database cause (a SQLSTATE only), so a lost + // admission is diagnosable from the run log. + const cause = nativeProviderPackAdmissionFailureCode(error) + ?? (error instanceof Error && /^native_[a-z_]+$/.test(error.message) ? error.message : "unknown"); + try { await input.onLog?.("stderr", `[paperclip-runner] native provider-pack admission receipt could not be saved (cause=${cause})\n`); } catch {} } await runnerdBackend?.providerAdmissionPersisted?.(); if (session.resolveRuntimeRequest) await liveQuestions.attach(); diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index e1d815cf974..c9660eaea57 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -658,6 +658,19 @@ runs once per image. A copy that fails part-way, or stalls past its 20-minute bo (`RESTART_CHAOS_SKOPEO_ATTEMPT_TIMEOUT_MS`), retries from an empty directory. Run it alone with `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/daytona-image.ts `. +**Provider-pack mismatch.** Because the controller takes its pack from the +same image the sandboxes start from, the preinstalled pack always verifies and +the upload path (`provider_pack.select` with `stateSource: uploaded`, reason +`cached_packs_unverified`) and its session admission never run. Staging runs +there whenever a deploy is newer than the sandbox image. `--daytona-pack-mismatch +` starts every sandbox from that other image (an +older runner image) while the controller keeps `--daytona-image`'s runnerd and +pack. Every fresh sandbox then uploads and verifies the controller's pack, and +the session must record its admission before artifact retention. Pass: every +task finishes with no Retry and no run ends with +`native_provider_artifacts_admission_unverified`. The timeline records +`daytona_pack_mismatch`. + Before starting, the harness compares the image's `io.paperclip.runner.content-id` label with the content ID of this checkout (`pnpm test:e2e:runner:image-id`, which hashes every runner input) and refuses to run on a mismatch. To get an diff --git a/tests/restart-chaos/config.ts b/tests/restart-chaos/config.ts index f4b29526727..fb4540aba0c 100644 --- a/tests/restart-chaos/config.ts +++ b/tests/restart-chaos/config.ts @@ -103,6 +103,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC "poll-ms": { type: "string" }, replay: { type: "string" }, "daytona-image": { type: "string" }, + "daytona-pack-mismatch": { type: "string" }, "daytona-cpu": { type: "string" }, "daytona-reuse-lease": { type: "boolean" }, // The reusable warm Daytona fixture, or an environment that inherits the lifecycle (a user's default). @@ -142,7 +143,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000); const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000); if (values.replay !== undefined) { - for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "daytona-warm", "daytona-inherit", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on", "k15-hold-seconds", "tm1-hold-seconds"] as const) { + for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-pack-mismatch", "daytona-cpu", "daytona-reuse-lease", "daytona-warm", "daytona-inherit", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on", "k15-hold-seconds", "tm1-hold-seconds"] as const) { if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`); } return { mode: "replay", ref: parseReplayRef(values.replay), keep, @@ -157,6 +158,14 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC throw new Error("--env daytona needs an immutable runner image: --daytona-image or PAPERCLIP_E2E_DAYTONA_IMAGE"); } if (env !== "daytona" && values["daytona-image"] !== undefined) throw new Error("--daytona-image applies only to --env daytona"); + const daytonaSandboxImage = values["daytona-pack-mismatch"]; + if (daytonaSandboxImage !== undefined) { + if (env !== "daytona") throw new Error("--daytona-pack-mismatch applies only to --env daytona"); + if (!/^.+@sha256:[0-9a-f]{64}$/.test(daytonaSandboxImage)) { + throw new Error("--daytona-pack-mismatch needs an immutable image whose provider pack differs: "); + } + if (daytonaSandboxImage === daytonaImage) throw new Error("--daytona-pack-mismatch must name a different image than --daytona-image"); + } // B5 grades bench.sh progress read from the live workspace while it runs. A // Daytona run's live workspace is inside the sandbox, so B5 is local-only. const workloadChoices = env === "daytona" ? WORKLOAD_IDS.filter(id => id !== "B5") : WORKLOAD_IDS; @@ -198,7 +207,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC seed: integer(values.seed, "seed", now % 2_147_483_647, 0, 2_147_483_647), budgetCents, pollMs, - ...(env === "daytona" ? { daytonaImage, daytonaCpu: integer(values["daytona-cpu"], "daytona-cpu", DAYTONA_DEFAULT_CPU, 1, 4), + ...(env === "daytona" ? { daytonaImage, ...(daytonaSandboxImage ? { daytonaSandboxImage } : {}), daytonaCpu: integer(values["daytona-cpu"], "daytona-cpu", DAYTONA_DEFAULT_CPU, 1, 4), ...(values["daytona-reuse-lease"] ? { daytonaReuseLease: true } : {}), ...(values["daytona-warm"] ? { daytonaWarm: true } : {}), ...(values["daytona-inherit"] ? { daytonaInherit: true } : {}) } : {}), ...(values["rc06-masked"] ? { masks: ["rc06"] } : {}), diff --git a/tests/restart-chaos/daytona-image.test.ts b/tests/restart-chaos/daytona-image.test.ts index f43c003c132..05cbf50e6b6 100644 --- a/tests/restart-chaos/daytona-image.test.ts +++ b/tests/restart-chaos/daytona-image.test.ts @@ -3,7 +3,7 @@ import { chmod, mkdir, mkdtemp, readFile, readlink, rm, stat, writeFile } from " import os from "node:os"; import path from "node:path"; import { afterEach, describe, expect, it } from "vitest"; -import { assertControllerMatchesImage, copyImageToDir, extractFromLayers, prepareDaytonaImageArtifacts, normalizeLayerPath, walkTar, type LayerSource } from "./daytona-image.js"; +import { assertControllerMatchesImage, assertProviderPackMismatch, copyImageToDir, extractFromLayers, prepareDaytonaImageArtifacts, normalizeLayerPath, walkTar, type LayerSource } from "./daytona-image.js"; type Entry = | { path: string; type: "file"; content: string; mode?: number } @@ -193,6 +193,19 @@ describe("assertControllerMatchesImage", () => { }); }); +describe("assertProviderPackMismatch", () => { + const controller = { image: "ghcr.io/x/y@sha256:" + "a".repeat(64), providerPackDigest: `sha256:${"1".repeat(64)}` }; + it("accepts a sandbox image whose provider pack differs", () => { + expect(() => assertProviderPackMismatch(controller, + { image: "ghcr.io/x/y@sha256:" + "b".repeat(64), providerPackDigest: `sha256:${"2".repeat(64)}` })).not.toThrow(); + }); + it("refuses another image that carries the same provider pack, which would skip the upload path", () => { + expect(() => assertProviderPackMismatch(controller, + { image: "ghcr.io/x/y@sha256:" + "b".repeat(64), providerPackDigest: controller.providerPackDigest })) + .toThrow(/carries the controller's provider pack/); + }); +}); + describe("copyImageToDir", () => { async function fakeSkopeo(root: string, failures: number): Promise { const script = path.join(root, "skopeo"); diff --git a/tests/restart-chaos/daytona-image.ts b/tests/restart-chaos/daytona-image.ts index 64b186c4984..3fd49d73591 100644 --- a/tests/restart-chaos/daytona-image.ts +++ b/tests/restart-chaos/daytona-image.ts @@ -327,6 +327,19 @@ async function readReceipt(directory: string): Promise, + sandbox: Pick): void { + if (controller.providerPackDigest === sandbox.providerPackDigest) { + throw new Error(`--daytona-pack-mismatch image ${sandbox.image} carries the controller's provider pack ` + + `(${controller.providerPackDigest}); choose an image with a different pack`); + } +} + export function defaultDaytonaImageCacheRoot(): string { return path.resolve(process.env.RESTART_CHAOS_IMAGE_CACHE ?? path.join(import.meta.dirname, "results", "daytona-images")); } diff --git a/tests/restart-chaos/main.ts b/tests/restart-chaos/main.ts index 131eafc10dd..0a725569f79 100644 --- a/tests/restart-chaos/main.ts +++ b/tests/restart-chaos/main.ts @@ -25,7 +25,7 @@ import { describeProviderFault, installProviderFaultSeam, resolveRealCodex, with import { installAcpxSidecarSeam, withAcpxSidecarHook, type AcpxSidecarSeam } from "./acpx-provider-faults.js"; import { DiskFullFaultSeam, isDiskFullFault, receiptsForIssue, withDiskFullCheck } from "./disk-full-faults.js"; import { CREDENTIAL_FOR_ADAPTER, parseCommand, replayCommand, waveIdFor } from "./config.js"; -import { assertControllerMatchesImage, defaultDaytonaImageCacheRoot, prepareDaytonaImageArtifacts } from "./daytona-image.js"; +import { assertControllerMatchesImage, assertProviderPackMismatch, defaultDaytonaImageCacheRoot, prepareDaytonaImageArtifacts } from "./daytona-image.js"; import { PLUGIN_BUILD_STAMP, pluginBuildState } from "./plugin-build.js"; import { DaytonaWaveSweeper, leaksForTask, unattributedLeaks, type DaytonaSweepResult } from "./daytona-sweep.js"; import { knownSecretValues, redactKnownSecrets, writeEvidenceBundle } from "./evidence.js"; @@ -135,6 +135,14 @@ async function prepareDaytona(plan: WavePlan, timeline: Timeline): Promise console.info(`[daytona-image] ${line}`) }); + assertProviderPackMismatch(artifacts, sandbox); + timeline.add("daytona_pack_mismatch", { controllerImage: image, sandboxImage: sandbox.image, + controllerProviderPackDigest: artifacts.providerPackDigest, sandboxProviderPackDigest: sandbox.providerPackDigest }); + } timeline.add("daytona_vcpu_plan", { cpuPerSandbox: plan.options.daytonaCpu, concurrency: plan.options.concurrency, taskVcpu: plan.options.concurrency * (plan.options.daytonaCpu ?? 0) }); return { PAPERCLIP_RUNNER_REMOTE_BINARY_PATH: artifacts.runnerdPath, PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH: artifacts.providerPackPath }; @@ -349,7 +357,9 @@ async function main(): Promise { await supervisor.start(serverEnvironment); // TM2/TM3 age the database; the observer and grader read run times back at their real times. const api: ChaosApi = new TimeShiftedApi(supervisor.baseUrl, supervisor.agingShifts); - const lab = await seedChaosLab({ api, env: plan.options.env, waitForReadyAgents: true, cancelled: () => interrupted, daytonaImage: plan.options.daytonaImage, daytonaCpu: plan.options.daytonaCpu, + const lab = await seedChaosLab({ api, env: plan.options.env, waitForReadyAgents: true, cancelled: () => interrupted, + // The controller keeps daytonaImage's runnerd and pack; a pack-mismatch wave starts sandboxes from another image. + daytonaImage: plan.options.daytonaSandboxImage ?? plan.options.daytonaImage, daytonaCpu: plan.options.daytonaCpu, daytonaReuseLease: plan.options.daytonaReuseLease, daytonaWarm: plan.options.daytonaWarm, daytonaInherit: plan.options.daytonaInherit, adapters: [...new Set([...plan.tasks.map(task => task.adapter), diff --git a/tests/restart-chaos/scheduler.test.ts b/tests/restart-chaos/scheduler.test.ts index 5cdcb40f39b..32a98d20a55 100644 --- a/tests/restart-chaos/scheduler.test.ts +++ b/tests/restart-chaos/scheduler.test.ts @@ -261,6 +261,14 @@ describe("Daytona options", () => { expect(() => parseCommand(["--env", "local", "--daytona-cpu", "2"])).toThrow(/only to --env daytona/); expect(() => parseCommand(["--env", "local", "--daytona-reuse-lease"])).toThrow(/only to --env daytona/); }); + it("starts sandboxes from a pack-mismatched image while the controller keeps the matched pack", () => { + const older = `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"b".repeat(64)}`; + expect(parseCommand(["--env", "daytona", "--daytona-image", image, "--daytona-pack-mismatch", older])) + .toMatchObject({ mode: "wave", options: { daytonaImage: image, daytonaSandboxImage: older } }); + expect(() => parseCommand(["--env", "daytona", "--daytona-image", image, "--daytona-pack-mismatch", image])).toThrow(/different image/); + expect(() => parseCommand(["--env", "daytona", "--daytona-image", image, "--daytona-pack-mismatch", "ghcr.io/x/y:old"])).toThrow(/immutable image/); + expect(() => parseCommand(["--env", "local", "--daytona-pack-mismatch", older])).toThrow(/only to --env daytona/); + }); it("counts every task's sandbox against the budget when warm sandboxes outlive their task", () => { const wave = (...extra: string[]) => parseCommand(["--env", "daytona", "--daytona-image", image, "--tasks", "20", "--concurrency", "4", ...extra]); expect(() => wave("--daytona-warm")).toThrow(/--daytona-warm keeps each task's sandbox.*20 tasks/); diff --git a/tests/restart-chaos/types.ts b/tests/restart-chaos/types.ts index 79e95820a3f..734a70493a0 100644 --- a/tests/restart-chaos/types.ts +++ b/tests/restart-chaos/types.ts @@ -61,6 +61,11 @@ export interface WaveOptions { pollMs: number; /** `--env daytona`: the immutable runner image (registry/repository@sha256:…). */ daytonaImage?: string; + /** `--daytona-pack-mismatch `: sandboxes start from this + * image while the controller keeps `daytonaImage`'s runnerd and provider pack, + * so the sandbox's preinstalled pack does not match and every fresh sandbox + * takes the provider-pack upload and admission path (staging run 3f317172). */ + daytonaSandboxImage?: string; /** `--env daytona`: vCPU per sandbox (see seed.ts chaosDaytonaShape). */ daytonaCpu?: number; /** `--daytona-reuse-lease`: the chaos environment keeps its sandboxes between turns (a per-turn run stops and keeps its sandbox). */