mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
fix(native): retry a contended provider-pack admission write (#20850)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native OpenCode run on a remote sandbox must save a provider-pack admission receipt on its run row after the provider session opens. Artifact retention refuses to continue without it. > - The admission write locks the coordinator row (`native_run_finalizations`) and then the run row, each with a 1000 ms `lock_timeout`. The runner's durable checkpoint writes lock the same coordinator row for the whole checkpoint transaction. > - On a loaded database one checkpoint transaction can outlast one lock wait. The admission write then fails with `55P03`, the state stays `pending`, and artifact retention fails the run as `native_provider_artifacts_admission_unverified` before the first turn. The task rests blocked. > - #20682 fixed the same symptom for a run-event append on the run row. This is the coordinator-row half. > - This pull request retries a contended admission write a bounded number of times, each in a new transaction that re-proves the claim. It also names the database cause in the run log and adds a chaos option that makes every fresh sandbox take the pack upload and admission path. > - The benefit is that a healthy fresh sandbox is no longer failed and blocked by ordinary lock contention. ## Linked Issues or Issue Description Refs #20682 (same symptom, run-row lock; this is the coordinator-row lock). **What happened?** On a hosted staging instance, an OpenCode run on a fresh Daytona sandbox found no matching preinstalled provider pack (`provider_pack.select` reason `cached_packs_unverified`). It uploaded and verified the pack (61 s), started the runner, and the runner authenticated to the durable control plane. 3.4 s later the run log shows `native provider-pack admission receipt could not be saved`. `nativeProviderPackAdmissionState.state` stayed `pending`. The run failed with `native_provider_artifacts_admission_unverified` before any provider turn. The task rests `blocked` with no Retry. **Steps to reproduce** 1. Start a native run that waits for its claim (`waitForClaim: true`, as a fresh run with artifact storage does). 2. Hold its `native_run_finalizations` row `FOR UPDATE` in another transaction for 1.6 s, as a checkpoint write does. 3. Call `recordNativeProviderPackAdmission`. It fails with `canceling statement due to lock timeout` (`55P03`). **Expected behavior** The admission waits out the checkpoint write within a bound, re-proves the claim, and saves the receipt. ## What Changed - `recordNativeProviderPackAdmission` retries a write that waits for its claim when it fails with `55P03`, `40P01` or `40001`, up to 5 attempts with 100–400 ms back-off. Each attempt is a new transaction with the same checks (claim, run, epoch, receipt). A stale claim still returns `false`. Other errors still throw. A write that does not wait for its claim keeps its `NOWAIT` refusal. - New `nativeProviderPackAdmissionFailureCode` reads the SQLSTATE from the error or its cause. - The executor's run-log line now ends with `(cause=<SQLSTATE or native_* code>)`, so a lost admission is diagnosable. It carries no paths or secrets. - Restart chaos: `--daytona-pack-mismatch <image@sha256:…>` starts sandboxes from another image while the controller keeps `--daytona-image`'s runnerd and pack. Every fresh sandbox then takes the upload and admission path, which the matched-image check made unreachable. Config validation and README updated. - `doc/execution-semantics.md` describes the bounded retry. ## Verification - RED on `origin/master` (`1d67e309e0`): `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-provider-pack-admission.test.ts -t "checkpoint write holds"` fails with `canceling statement due to lock timeout` (`55P03`), the staging error. - GREEN: the same file, 39/39 pass, including the existing fence tests (`cannot publish while another transaction holds the coordinator claim`, revoked-claim recheck) and the new bound test (`gives up after the bounded retry window while the claim stays held`, under 15 s). - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/scheduler.test.ts`: 34/34 pass, including the new option test. - `pnpm --filter @paperclipai/server exec tsc --noEmit` and `tsc --noEmit -p tests/restart-chaos/tsconfig.json` pass. - Not run: a Daytona chaos wave with `--daytona-pack-mismatch`. It needs a runner image built at this revision. The staging cause is inferred from the timing and the code path; the original error was swallowed, which is why this PR adds the cause to the run log. ## Risks - Low risk. The normal path has no contention and makes one attempt. Under contention the write waits at most about 11 s before the first provider turn. Without this change that case failed the run. - A retry cannot widen authority: every attempt re-checks the coordinator claim, lease expiry, run status and admission epoch. - When admission still fails after the retries, the run fails as before. This PR does not change that recovery path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
381b862135
commit
1b39cf1794
11 files changed
+173
-12
No files matched your search
@@ -4926,8 +4926,12 @@ and identify `inheritedFromRunId`. The first receipt is immutable history.
|
||||
`nativeProviderPackAdmissionState` separately describes current admission proof:
|
||||
`pending`, `verified` or `unknown`, with an opaque admission epoch. The mandatory
|
||||
coordinator claim atomically sets a new pending epoch before session admission.
|
||||
Only the matching owner and epoch can publish post-attachment proof. Failed
|
||||
receipt writes leave pending evidence; changed or unsupported admission leaves
|
||||
Only the matching owner and epoch can publish post-attachment proof. A write
|
||||
that waits for its claim retries a contended write (lock timeout, deadlock or
|
||||
serialization abort) up to five times, because the runner's durable checkpoint
|
||||
writes hold the same coordinator row. Each attempt re-proves the claim, run and
|
||||
epoch in a new transaction. The run log names the database cause of a write
|
||||
that still fails. Failed receipt writes leave pending evidence; changed or unsupported admission leaves
|
||||
unknown evidence and does not prevent authorized execution. A missing legacy
|
||||
receipt remains unknown; the current release cannot supply that missing history.
|
||||
Retained recovery needs positively verified prior provenance, captured before
|
||||
|
||||
@@ -257,6 +257,50 @@ describe("provider pack admission PostgreSQL fencing", () => {
|
||||
expect((await read(f.identity.runId)).runnerProfileJson?.nativeProviderPackAdmissionState)
|
||||
.toEqual({ ...f.profile.nativeProviderPackAdmissionState, state: "verified" });
|
||||
});
|
||||
it("keeps the admission when a checkpoint write holds the claim longer than one lock wait", async () => {
|
||||
// Staging run 3f317172 (OpenCode, fresh Daytona sandbox): the pack upload
|
||||
// and runner start succeeded, the runner began writing durable checkpoints,
|
||||
// and the admission write lost one 1000 ms lock wait on the coordinator row
|
||||
// a checkpoint transaction held. The state stayed pending, artifact
|
||||
// retention failed the run as native_provider_artifacts_admission_unverified
|
||||
// and the task rested blocked before its first turn.
|
||||
const f = await fixture();
|
||||
let release!: () => void, acquired!: () => void;
|
||||
const holding = new Promise<void>(resolve => { acquired = resolve; });
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
const checkpoint = db.transaction(async tx => {
|
||||
await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.identity.runId)).for("update");
|
||||
acquired(); await gate;
|
||||
});
|
||||
await holding;
|
||||
const recording = recordNativeProviderPackAdmission(db, { ...f, waitForClaim: true } as Parameters<typeof recordNativeProviderPackAdmission>[1])
|
||||
.then(value => ({ value }), error => ({ error }));
|
||||
// Longer than one lock wait, shorter than the bounded retry window.
|
||||
const held = setTimeout(release, 1_600);
|
||||
try {
|
||||
expect(await recording).toEqual({ value: true });
|
||||
} finally { clearTimeout(held); release(); await checkpoint; }
|
||||
expect((await read(f.identity.runId)).runnerProfileJson?.nativeProviderPackAdmissionState)
|
||||
.toEqual({ ...f.profile.nativeProviderPackAdmissionState, state: "verified" });
|
||||
});
|
||||
it("gives up after the bounded retry window while the claim stays held", async () => {
|
||||
const f = await fixture();
|
||||
let release!: () => void, acquired!: () => void;
|
||||
const holding = new Promise<void>(resolve => { acquired = resolve; });
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
const checkpoint = db.transaction(async tx => {
|
||||
await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.identity.runId)).for("update");
|
||||
acquired(); await gate;
|
||||
});
|
||||
await holding;
|
||||
const started = Date.now();
|
||||
try {
|
||||
await expect(recordNativeProviderPackAdmission(db, { ...f, waitForClaim: true } as Parameters<typeof recordNativeProviderPackAdmission>[1]))
|
||||
.rejects.toMatchObject({ cause: { code: "55P03" } });
|
||||
expect(Date.now() - started).toBeLessThan(15_000);
|
||||
} finally { release(); await checkpoint; }
|
||||
expect((await read(f.identity.runId)).runnerProfileJson).toEqual(f.profile);
|
||||
}, 30_000);
|
||||
it.each([false, true])("waits within the durable-capture bound and rechecks the claim after its lock (revoked: %s)", async revoked => {
|
||||
const f = await fixture();
|
||||
let release!: () => void, acquired!: () => void;
|
||||
|
||||
@@ -123,13 +123,51 @@ export function admittedNativeProviderPack(session: NativeSession, identity: Nat
|
||||
...(admittedRunId === actual.runId ? {} : { inheritedFromRunId: admittedRunId }) };
|
||||
}
|
||||
|
||||
export async function recordNativeProviderPackAdmission(db: Db, input: {
|
||||
type AdmissionInput = {
|
||||
session: NativeSession;
|
||||
identity: NativeRunIdentity;
|
||||
admissionEpoch: string;
|
||||
claim: { leaseOwner: string; attempt: number; controllerGeneration: number; controllerBootId: string };
|
||||
waitForClaim?: boolean;
|
||||
}): Promise<boolean> {
|
||||
};
|
||||
|
||||
/** Lock timeouts, deadlocks and serialization aborts that clear on their own. */
|
||||
const CONTENDED_ADMISSION_CODES = new Set(["55P03", "40P01", "40001"]);
|
||||
/** Attempts of one admission write while its claim is contended. */
|
||||
export const NATIVE_PROVIDER_PACK_ADMISSION_ATTEMPTS = 5;
|
||||
|
||||
export function nativeProviderPackAdmissionFailureCode(error: unknown): string | null {
|
||||
for (let value = error, depth = 0; value && typeof value === "object" && depth < 4; depth += 1) {
|
||||
const code = (value as { code?: unknown }).code;
|
||||
if (typeof code === "string" && /^[0-9A-Z]{5}$/.test(code)) return code;
|
||||
value = (value as { cause?: unknown }).cause;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Saves the session's verified provider-pack admission on its run row. A run
|
||||
* that waits for its claim retries a contended write: the runner's durable
|
||||
* checkpoint writes hold the same coordinator row, and on a loaded database one
|
||||
* can outlast a single lock wait. Each attempt is a new transaction that
|
||||
* re-proves the claim, the run and the admission epoch, so a retry never
|
||||
* admits anything the first attempt would have refused. Only contention is
|
||||
* retried; a stale claim still returns false and other errors still throw.
|
||||
*/
|
||||
export async function recordNativeProviderPackAdmission(db: Db, input: AdmissionInput): Promise<boolean> {
|
||||
for (let attempt = 1; ; attempt += 1) {
|
||||
try {
|
||||
return await recordNativeProviderPackAdmissionOnce(db, input);
|
||||
} catch (error) {
|
||||
const code = nativeProviderPackAdmissionFailureCode(error);
|
||||
if (!input.waitForClaim || attempt >= NATIVE_PROVIDER_PACK_ADMISSION_ATTEMPTS
|
||||
|| !code || !CONTENDED_ADMISSION_CODES.has(code)) throw error;
|
||||
await new Promise(resolve => setTimeout(resolve, 100 * attempt));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function recordNativeProviderPackAdmissionOnce(db: Db, input: AdmissionInput): Promise<boolean> {
|
||||
const receipt = admittedNativeProviderPack(input.session, input.identity);
|
||||
return db.transaction(async tx => {
|
||||
// Bounds every lock wait below, including the run-row lock. A run-event
|
||||
|
||||
@@ -44,7 +44,7 @@ import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-
|
||||
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
|
||||
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
|
||||
import { providerPackWorkspaceId, recordAdmittedRemoteProviderPack, resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js";
|
||||
import { nativeProviderPackAdmissionPending, rememberAdmittedNativeProviderPack, rememberRecoveredNativeProviderPack, recordNativeProviderPackAdmission, type NativeProviderPackAdmissionSource } from "./native-provider-pack-admission.js";
|
||||
import { nativeProviderPackAdmissionFailureCode, nativeProviderPackAdmissionPending, rememberAdmittedNativeProviderPack, rememberRecoveredNativeProviderPack, recordNativeProviderPackAdmission, type NativeProviderPackAdmissionSource } from "./native-provider-pack-admission.js";
|
||||
import { type ProviderPackInstallation, verifyRemoteProviderPack as verifyRemoteProviderPackIdentity } from "./remote-provider-pack-verification.js";
|
||||
import { writeEnvironmentCleanup } from "../environment-cleanup-notifications.js";
|
||||
import { NATIVE_SESSION_EXECUTION_LEASE_TTL_MS, lockNativeExecutionLeaseForReacquire, nativeExecutionLeaseHeldCondition, releaseUnstartedNativeExecutionClaim, startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal } from "./native-execution-lease.js";
|
||||
@@ -12075,8 +12075,12 @@ async function executePaperclipNativeSessionAttempt(
|
||||
claim: { leaseOwner, attempt, controllerGeneration, controllerBootId: controller.bootId },
|
||||
waitForClaim: runnerdBackend?.providerAdmissionRequired?.() === true,
|
||||
});
|
||||
} catch {
|
||||
try { await input.onLog?.("stderr", "[paperclip-runner] native provider-pack admission receipt could not be saved\n"); } catch {}
|
||||
} catch (error) {
|
||||
// Name the database cause (a SQLSTATE only), so a lost
|
||||
// admission is diagnosable from the run log.
|
||||
const cause = nativeProviderPackAdmissionFailureCode(error)
|
||||
?? (error instanceof Error && /^native_[a-z_]+$/.test(error.message) ? error.message : "unknown");
|
||||
try { await input.onLog?.("stderr", `[paperclip-runner] native provider-pack admission receipt could not be saved (cause=${cause})\n`); } catch {}
|
||||
}
|
||||
await runnerdBackend?.providerAdmissionPersisted?.();
|
||||
if (session.resolveRuntimeRequest) await liveQuestions.attach();
|
||||
|
||||
@@ -658,6 +658,19 @@ runs once per image. A copy that fails part-way, or stalls past its 20-minute bo
|
||||
(`RESTART_CHAOS_SKOPEO_ATTEMPT_TIMEOUT_MS`), retries from an empty directory.
|
||||
Run it alone with `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/daytona-image.ts <image>`.
|
||||
|
||||
**Provider-pack mismatch.** Because the controller takes its pack from the
|
||||
same image the sandboxes start from, the preinstalled pack always verifies and
|
||||
the upload path (`provider_pack.select` with `stateSource: uploaded`, reason
|
||||
`cached_packs_unverified`) and its session admission never run. Staging runs
|
||||
there whenever a deploy is newer than the sandbox image. `--daytona-pack-mismatch
|
||||
<registry/repository@sha256:…>` starts every sandbox from that other image (an
|
||||
older runner image) while the controller keeps `--daytona-image`'s runnerd and
|
||||
pack. Every fresh sandbox then uploads and verifies the controller's pack, and
|
||||
the session must record its admission before artifact retention. Pass: every
|
||||
task finishes with no Retry and no run ends with
|
||||
`native_provider_artifacts_admission_unverified`. The timeline records
|
||||
`daytona_pack_mismatch`.
|
||||
|
||||
Before starting, the harness compares the image's `io.paperclip.runner.content-id`
|
||||
label with the content ID of this checkout (`pnpm test:e2e:runner:image-id`,
|
||||
which hashes every runner input) and refuses to run on a mismatch. To get an
|
||||
|
||||
@@ -103,6 +103,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
"poll-ms": { type: "string" },
|
||||
replay: { type: "string" },
|
||||
"daytona-image": { type: "string" },
|
||||
"daytona-pack-mismatch": { type: "string" },
|
||||
"daytona-cpu": { type: "string" },
|
||||
"daytona-reuse-lease": { type: "boolean" },
|
||||
// The reusable warm Daytona fixture, or an environment that inherits the lifecycle (a user's default).
|
||||
@@ -142,7 +143,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000);
|
||||
const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000);
|
||||
if (values.replay !== undefined) {
|
||||
for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "daytona-warm", "daytona-inherit", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on", "k15-hold-seconds", "tm1-hold-seconds"] as const) {
|
||||
for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-pack-mismatch", "daytona-cpu", "daytona-reuse-lease", "daytona-warm", "daytona-inherit", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on", "k15-hold-seconds", "tm1-hold-seconds"] as const) {
|
||||
if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`);
|
||||
}
|
||||
return { mode: "replay", ref: parseReplayRef(values.replay), keep,
|
||||
@@ -157,6 +158,14 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
throw new Error("--env daytona needs an immutable runner image: --daytona-image <registry/repository@sha256:…> or PAPERCLIP_E2E_DAYTONA_IMAGE");
|
||||
}
|
||||
if (env !== "daytona" && values["daytona-image"] !== undefined) throw new Error("--daytona-image applies only to --env daytona");
|
||||
const daytonaSandboxImage = values["daytona-pack-mismatch"];
|
||||
if (daytonaSandboxImage !== undefined) {
|
||||
if (env !== "daytona") throw new Error("--daytona-pack-mismatch applies only to --env daytona");
|
||||
if (!/^.+@sha256:[0-9a-f]{64}$/.test(daytonaSandboxImage)) {
|
||||
throw new Error("--daytona-pack-mismatch needs an immutable image whose provider pack differs: <registry/repository@sha256:…>");
|
||||
}
|
||||
if (daytonaSandboxImage === daytonaImage) throw new Error("--daytona-pack-mismatch must name a different image than --daytona-image");
|
||||
}
|
||||
// B5 grades bench.sh progress read from the live workspace while it runs. A
|
||||
// Daytona run's live workspace is inside the sandbox, so B5 is local-only.
|
||||
const workloadChoices = env === "daytona" ? WORKLOAD_IDS.filter(id => id !== "B5") : WORKLOAD_IDS;
|
||||
@@ -198,7 +207,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
seed: integer(values.seed, "seed", now % 2_147_483_647, 0, 2_147_483_647),
|
||||
budgetCents,
|
||||
pollMs,
|
||||
...(env === "daytona" ? { daytonaImage, daytonaCpu: integer(values["daytona-cpu"], "daytona-cpu", DAYTONA_DEFAULT_CPU, 1, 4),
|
||||
...(env === "daytona" ? { daytonaImage, ...(daytonaSandboxImage ? { daytonaSandboxImage } : {}), daytonaCpu: integer(values["daytona-cpu"], "daytona-cpu", DAYTONA_DEFAULT_CPU, 1, 4),
|
||||
...(values["daytona-reuse-lease"] ? { daytonaReuseLease: true } : {}),
|
||||
...(values["daytona-warm"] ? { daytonaWarm: true } : {}), ...(values["daytona-inherit"] ? { daytonaInherit: true } : {}) } : {}),
|
||||
...(values["rc06-masked"] ? { masks: ["rc06"] } : {}),
|
||||
|
||||
@@ -3,7 +3,7 @@ import { chmod, mkdir, mkdtemp, readFile, readlink, rm, stat, writeFile } from "
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { assertControllerMatchesImage, copyImageToDir, extractFromLayers, prepareDaytonaImageArtifacts, normalizeLayerPath, walkTar, type LayerSource } from "./daytona-image.js";
|
||||
import { assertControllerMatchesImage, assertProviderPackMismatch, copyImageToDir, extractFromLayers, prepareDaytonaImageArtifacts, normalizeLayerPath, walkTar, type LayerSource } from "./daytona-image.js";
|
||||
|
||||
type Entry =
|
||||
| { path: string; type: "file"; content: string; mode?: number }
|
||||
@@ -193,6 +193,19 @@ describe("assertControllerMatchesImage", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("assertProviderPackMismatch", () => {
|
||||
const controller = { image: "ghcr.io/x/y@sha256:" + "a".repeat(64), providerPackDigest: `sha256:${"1".repeat(64)}` };
|
||||
it("accepts a sandbox image whose provider pack differs", () => {
|
||||
expect(() => assertProviderPackMismatch(controller,
|
||||
{ image: "ghcr.io/x/y@sha256:" + "b".repeat(64), providerPackDigest: `sha256:${"2".repeat(64)}` })).not.toThrow();
|
||||
});
|
||||
it("refuses another image that carries the same provider pack, which would skip the upload path", () => {
|
||||
expect(() => assertProviderPackMismatch(controller,
|
||||
{ image: "ghcr.io/x/y@sha256:" + "b".repeat(64), providerPackDigest: controller.providerPackDigest }))
|
||||
.toThrow(/carries the controller's provider pack/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("copyImageToDir", () => {
|
||||
async function fakeSkopeo(root: string, failures: number): Promise<string> {
|
||||
const script = path.join(root, "skopeo");
|
||||
|
||||
@@ -327,6 +327,19 @@ async function readReceipt(directory: string): Promise<DaytonaImageArtifacts | n
|
||||
return receipt;
|
||||
}
|
||||
|
||||
/**
|
||||
* `--daytona-pack-mismatch`: the sandbox image must not carry the controller's
|
||||
* provider pack. With the same pack the runtime reuses the preinstalled copy and
|
||||
* the wave would never take the upload and admission path it exists to test.
|
||||
*/
|
||||
export function assertProviderPackMismatch(controller: Pick<DaytonaImageArtifacts, "image" | "providerPackDigest">,
|
||||
sandbox: Pick<DaytonaImageArtifacts, "image" | "providerPackDigest">): void {
|
||||
if (controller.providerPackDigest === sandbox.providerPackDigest) {
|
||||
throw new Error(`--daytona-pack-mismatch image ${sandbox.image} carries the controller's provider pack `
|
||||
+ `(${controller.providerPackDigest}); choose an image with a different pack`);
|
||||
}
|
||||
}
|
||||
|
||||
export function defaultDaytonaImageCacheRoot(): string {
|
||||
return path.resolve(process.env.RESTART_CHAOS_IMAGE_CACHE ?? path.join(import.meta.dirname, "results", "daytona-images"));
|
||||
}
|
||||
|
||||
@@ -25,7 +25,7 @@ import { describeProviderFault, installProviderFaultSeam, resolveRealCodex, with
|
||||
import { installAcpxSidecarSeam, withAcpxSidecarHook, type AcpxSidecarSeam } from "./acpx-provider-faults.js";
|
||||
import { DiskFullFaultSeam, isDiskFullFault, receiptsForIssue, withDiskFullCheck } from "./disk-full-faults.js";
|
||||
import { CREDENTIAL_FOR_ADAPTER, parseCommand, replayCommand, waveIdFor } from "./config.js";
|
||||
import { assertControllerMatchesImage, defaultDaytonaImageCacheRoot, prepareDaytonaImageArtifacts } from "./daytona-image.js";
|
||||
import { assertControllerMatchesImage, assertProviderPackMismatch, defaultDaytonaImageCacheRoot, prepareDaytonaImageArtifacts } from "./daytona-image.js";
|
||||
import { PLUGIN_BUILD_STAMP, pluginBuildState } from "./plugin-build.js";
|
||||
import { DaytonaWaveSweeper, leaksForTask, unattributedLeaks, type DaytonaSweepResult } from "./daytona-sweep.js";
|
||||
import { knownSecretValues, redactKnownSecrets, writeEvidenceBundle } from "./evidence.js";
|
||||
@@ -135,6 +135,14 @@ async function prepareDaytona(plan: WavePlan, timeline: Timeline): Promise<Recor
|
||||
timeline.add("daytona_image_matched", { image, contentId: artifacts.contentId, localContentId, sourceRevision: artifacts.sourceRevision,
|
||||
runnerdSha256: artifacts.runnerdSha256, providerPackDigest: artifacts.providerPackDigest, platformManifestDigest: artifacts.platformManifestDigest });
|
||||
ensureDaytonaPlugin(timeline);
|
||||
if (plan.options.daytonaSandboxImage) {
|
||||
// Prove the sandbox image's preinstalled pack differs, so every fresh sandbox must upload the controller's.
|
||||
const sandbox = await prepareDaytonaImageArtifacts({ image: plan.options.daytonaSandboxImage, cacheRoot: defaultDaytonaImageCacheRoot(),
|
||||
log: line => console.info(`[daytona-image] ${line}`) });
|
||||
assertProviderPackMismatch(artifacts, sandbox);
|
||||
timeline.add("daytona_pack_mismatch", { controllerImage: image, sandboxImage: sandbox.image,
|
||||
controllerProviderPackDigest: artifacts.providerPackDigest, sandboxProviderPackDigest: sandbox.providerPackDigest });
|
||||
}
|
||||
timeline.add("daytona_vcpu_plan", { cpuPerSandbox: plan.options.daytonaCpu, concurrency: plan.options.concurrency,
|
||||
taskVcpu: plan.options.concurrency * (plan.options.daytonaCpu ?? 0) });
|
||||
return { PAPERCLIP_RUNNER_REMOTE_BINARY_PATH: artifacts.runnerdPath, PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH: artifacts.providerPackPath };
|
||||
@@ -349,7 +357,9 @@ async function main(): Promise<number> {
|
||||
await supervisor.start(serverEnvironment);
|
||||
// TM2/TM3 age the database; the observer and grader read run times back at their real times.
|
||||
const api: ChaosApi = new TimeShiftedApi(supervisor.baseUrl, supervisor.agingShifts);
|
||||
const lab = await seedChaosLab({ api, env: plan.options.env, waitForReadyAgents: true, cancelled: () => interrupted, daytonaImage: plan.options.daytonaImage, daytonaCpu: plan.options.daytonaCpu,
|
||||
const lab = await seedChaosLab({ api, env: plan.options.env, waitForReadyAgents: true, cancelled: () => interrupted,
|
||||
// The controller keeps daytonaImage's runnerd and pack; a pack-mismatch wave starts sandboxes from another image.
|
||||
daytonaImage: plan.options.daytonaSandboxImage ?? plan.options.daytonaImage, daytonaCpu: plan.options.daytonaCpu,
|
||||
daytonaReuseLease: plan.options.daytonaReuseLease,
|
||||
daytonaWarm: plan.options.daytonaWarm, daytonaInherit: plan.options.daytonaInherit,
|
||||
adapters: [...new Set([...plan.tasks.map(task => task.adapter),
|
||||
|
||||
@@ -261,6 +261,14 @@ describe("Daytona options", () => {
|
||||
expect(() => parseCommand(["--env", "local", "--daytona-cpu", "2"])).toThrow(/only to --env daytona/);
|
||||
expect(() => parseCommand(["--env", "local", "--daytona-reuse-lease"])).toThrow(/only to --env daytona/);
|
||||
});
|
||||
it("starts sandboxes from a pack-mismatched image while the controller keeps the matched pack", () => {
|
||||
const older = `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"b".repeat(64)}`;
|
||||
expect(parseCommand(["--env", "daytona", "--daytona-image", image, "--daytona-pack-mismatch", older]))
|
||||
.toMatchObject({ mode: "wave", options: { daytonaImage: image, daytonaSandboxImage: older } });
|
||||
expect(() => parseCommand(["--env", "daytona", "--daytona-image", image, "--daytona-pack-mismatch", image])).toThrow(/different image/);
|
||||
expect(() => parseCommand(["--env", "daytona", "--daytona-image", image, "--daytona-pack-mismatch", "ghcr.io/x/y:old"])).toThrow(/immutable image/);
|
||||
expect(() => parseCommand(["--env", "local", "--daytona-pack-mismatch", older])).toThrow(/only to --env daytona/);
|
||||
});
|
||||
it("counts every task's sandbox against the budget when warm sandboxes outlive their task", () => {
|
||||
const wave = (...extra: string[]) => parseCommand(["--env", "daytona", "--daytona-image", image, "--tasks", "20", "--concurrency", "4", ...extra]);
|
||||
expect(() => wave("--daytona-warm")).toThrow(/--daytona-warm keeps each task's sandbox.*20 tasks/);
|
||||
|
||||
@@ -61,6 +61,11 @@ export interface WaveOptions {
|
||||
pollMs: number;
|
||||
/** `--env daytona`: the immutable runner image (registry/repository@sha256:…). */
|
||||
daytonaImage?: string;
|
||||
/** `--daytona-pack-mismatch <image@sha256:…>`: sandboxes start from this
|
||||
* image while the controller keeps `daytonaImage`'s runnerd and provider pack,
|
||||
* so the sandbox's preinstalled pack does not match and every fresh sandbox
|
||||
* takes the provider-pack upload and admission path (staging run 3f317172). */
|
||||
daytonaSandboxImage?: string;
|
||||
/** `--env daytona`: vCPU per sandbox (see seed.ts chaosDaytonaShape). */
|
||||
daytonaCpu?: number;
|
||||
/** `--daytona-reuse-lease`: the chaos environment keeps its sandboxes between turns (a per-turn run stops and keeps its sandbox). */
|
||||
|
||||
Reference in new issue
Block a user