From ffe05b1b7c15079a74468767bc98799db7ccfce0 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 12:05:54 -0600 Subject: [PATCH] fix(heartbeat): retry every pre-provider lease acquisition failure and rest it on Retry (#20857) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task run takes an environment lease (for example a Daytona sandbox) before the agent starts. No provider work happens before that lease exists. > - Only a provider HTTP status or a connection errno counted as a transient lease failure. #20828 (merged while this was open) added the Daytona acquisition deadline. A database error while the lease was recorded, a plugin RPC deadline, and an unfinished provisional creation still did not count. > - Those failures became a generic `setup_failed` with no bounded retry. And when the bounded retries of any `environment_provider_unavailable` failure were used up, recovery still put the task on a board-owned `stranded_assigned_issue` action. > - A hosted instance saw this for six runs in 45 minutes when concurrent runs rose from about 8 to about 20. Two tasks stayed blocked and waited for the board. > - This pull request classifies every lease acquisition failure that has structured transient evidence as `environment_provider_unavailable`. When the bounded retries are used up, the task rests on a system-owned action and a working Retry. > - It also adds the restart-chaos fault EP1, so the harness covers lease acquisition failures from now on. > - The benefit is that a busy or slow sandbox provider no longer strands tasks on the board. ## Linked Issues or Issue Description No public issue exists. **What happened?** Five runs failed with `Failed to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona lease acquisition timed out; allocation cleanup is pending` after 5 to 6 minutes in `preparing`. One run failed with `Database query failed (lease_acquire_failed)` within 10 seconds. Every run recorded `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`, but the error code was `setup_failed`. No run got a bounded retry. Two tasks were left `blocked` with a board-owned `stranded_assigned_issue` recovery action. **Expected behavior** A lease acquisition failure happens before any provider work. The run must retry with a bounded, visible backoff. When the bound is used up, the task must rest with a Retry that works, and no board-owned recovery action. **Steps to reproduce** 1. Make lease acquisition fail with the provider's acquisition deadline (the validated acquisition diagnostic that the environment runtime records), or with a database driver error. 2. Let the run start. 3. Before this change, the run fails with `setup_failed`, no retry is scheduled, and recovery creates a board-owned `stranded_assigned_issue` action. ## What Changed **How this PR and #20828 divide the problem.** #20828 (merged) makes the Daytona plugin end a stuck create early and classifies the acquisition-deadline envelope as `environment_provider_unavailable`. This PR does not change the Daytona plugin or that deadline path. It adds only what #20828 leaves out: - a lease-acquisition failure that is not a create timeout: the plugin RPC deadline (`-32003`) and an unfinished provisional creation recorded for this run; - a transient database error while the lease is recorded; - the exhausted state: a system-owned recovery action and Retry instead of a board-owned `stranded_assigned_issue`, for every `environment_provider_unavailable` cause (including #20828's); - the chaos fault EP1 (EP1, EP1N, EP1D). - `readTransientLeaseAcquisitionFailure` (`server/src/services/environment-provider-unavailable-error.ts`) classifies a lease acquisition failure as transient when it has structured evidence: the provider status or errno (as before), the run's own validated acquisition diagnostic (deadline or unfinished provisional creation), the plugin RPC deadline, or a transient database error (a postgres.js connection code, or SQLSTATE class 08 or 53, 57P01-57P03, 57014, 55P03, 40001, 40P01). A deterministic SQL error (syntax, schema, constraint) is not retried. A definite non-transient status (for example HTTP 401 or 404) is still `setup_failed`. Message text is not parsed. - `environment-run-orchestrator.ts` uses this classifier, so these failures get `environment_provider_unavailable` and the existing bounded bootstrap retry (30 s, then 60 s, with jitter). The task shows "retry scheduled" while it waits. - When the retries are used up, recovery uses the cause `environment_provider_unavailable` with a system-owned action and no board routing. The notice says "Retry when ready" and does not ask for a board decision. The task's Retry starts a new acquisition. - Restart-chaos fault EP1: a test-mode server seam (`environment-lease-fault-injection.ts`, read from `PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL`, ignored in production) throws the staging error shapes before any provider call. `tests/restart-chaos/env-provider-wave.ts` runs EP1 (deadline twice, then success), EP1N (deadline forever, then Retry after disarm) and EP1D (database error once). The oracle fails on any `setup_failed` run, any board-owned recovery action, a wrong retry count, or a task that does not complete. - Docs: `doc/execution-semantics.md` and `tests/restart-chaos/README.md`. ## Verification - RED on `origin/master` (`1d67e309e0`, before #20828): the three new tests in `heartbeat-process-recovery.test.ts` ("transient sandbox provider refusal during lease acquisition") fail with the staging result: `expected [ [ 'failed', 'setup_failed' ] ] to deeply equal [ [ 'failed', 'environment_provider_unavailable' ], [ 'scheduled_retry', null ] ]`. On current master the deadline now retries (#20828), but the exhausted case still rests on a board-owned `stranded_assigned_issue` action (the existing HTTP 502 exhaustion test showed `ownerType: "board"` before the recovery change), and the database error is still `setup_failed`. - GREEN: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "lease acquisition"` passes 5 of 5. This includes the deadline and the database error that retry once and then complete, and the deadline that never clears. That task rests `blocked` with a system-owned action, `canRetry: true`, no execution blocker, and Retry completes it. - Unit tests: `environment-provider-unavailable-error.test.ts` + `environment-lease-fault-injection.test.ts` (56), `tests/restart-chaos/env-provider-faults.test.ts` (6) pass. - Related suites pass: `environment-run-orchestrator`, `task-interruption-query`, `issue-recovery-actions`, `issue-recovery-deadlines`, `stranded-notice`, `heartbeat-native-resume-lease-failure`. - `server` and `tests/restart-chaos` typecheck pass. - Chaos waves (local, Codex `gpt-6.1-sol`, head `1bd387e98f`; the later head changes only database classification and the notice), all pass: - EP1 `w-20261111T171803-s71-ep`, 2 tasks: each had exactly 2 `environment_provider_unavailable` runs, then succeeded with no user action, no board-owned recovery action, brief oracle pass. - EP1D `w-20261111T172850-s73-ep`, 1 task: 1 failed run on the database error, then succeeded. - EP1N `w-20261111T173346-s74-ep` (and `w-20261111T162752-s72-ep` before the rebase): 3 failed runs, the task rested `blocked` with `canRetry: true` and no board-owned action; after disarm, Retry completed it. - No Daytona wave was run. The fault is injected in the server before the provider call, so a local instance reaches the same retry and recovery code. ## Risks - More failures now retry, all bounded. Only transient SQLSTATEs retry. The staging database error recorded no SQLSTATE. If it was a deterministic SQL error, it still fails `setup_failed` and goes to the board as before; that is intended, because a retry cannot fix it. - Each retry is a new creation attempt with its own creation record. A provisional allocation whose cleanup is pending stays with its record and the creation reconciler, as before. - The retry bound is the shared bootstrap bound (two retries). A provider outage longer than about 15 minutes still rests on Retry, but no longer on a board action. - The fault seam reads a file only when the environment variable is set, and never when `NODE_ENV=production`. - Saved state from the previous release: no persisted shape changes. A task already stranded on a board action keeps it (`preserveExistingOwner`). New failures use the new cause. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) through Claude Code, model ID `claude-opus-5-5`, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 --- doc/execution-semantics.md | 10 + .../heartbeat-process-recovery.test.ts | 93 +++++++- .../environment-lease-fault-injection.test.ts | 63 ++++++ .../environment-lease-fault-injection.ts | 76 +++++++ ...ronment-provider-unavailable-error.test.ts | 78 +++++++ .../environment-provider-unavailable-error.ts | 80 ++++++- .../services/environment-run-orchestrator.ts | 14 +- server/src/services/recovery/service.ts | 22 +- .../services/recovery/stranded-notice.test.ts | 14 ++ .../src/services/recovery/stranded-notice.ts | 4 +- tests/restart-chaos/README.md | 24 ++ .../restart-chaos/env-provider-faults.test.ts | 49 +++++ tests/restart-chaos/env-provider-faults.ts | 106 +++++++++ tests/restart-chaos/env-provider-wave.ts | 205 ++++++++++++++++++ 14 files changed, 825 insertions(+), 13 deletions(-) create mode 100644 server/src/services/environment-lease-fault-injection.test.ts create mode 100644 server/src/services/environment-lease-fault-injection.ts create mode 100644 tests/restart-chaos/env-provider-faults.test.ts create mode 100644 tests/restart-chaos/env-provider-faults.ts create mode 100644 tests/restart-chaos/env-provider-wave.ts diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 61125608984..6f1334faebd 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1458,6 +1458,16 @@ provider failure as the reason, retries automatically, and on exhaustion becomes `blocked` with a specific recovery notice and a Retry. Because no provider work started, no execution-reconciliation hold is created. +Lease acquisition precedes provider work by construction, so the same budget +covers every lease acquisition failure that carries structured transient evidence, +not only a provider status: the provider's acquisition deadline or an unfinished +provisional creation (the validated acquisition diagnostic the runtime recorded +for that run), the plugin RPC deadline, and a database driver error while the +lease is recorded. A definite non-transient status (for example HTTP 401 or 404) +still fails as `setup_failed`. On exhaustion the task's recovery action is +system-owned (cause `environment_provider_unavailable`), not a board escalation: +nothing needs board repair, and the task's Retry starts a new acquisition. + A saved workspace that cannot be restored into a new allocation before provider dispatch fails with `remote_workspace_restore_failed`. The run error names the server step (`snapshot_lookup`, `snapshot_download`, or `sandbox_restore`) and, diff --git a/server/src/__tests__/heartbeat-process-recovery.test.ts b/server/src/__tests__/heartbeat-process-recovery.test.ts index 559d9890075..d3ae160fe9a 100644 --- a/server/src/__tests__/heartbeat-process-recovery.test.ts +++ b/server/src/__tests__/heartbeat-process-recovery.test.ts @@ -28,6 +28,7 @@ import { recordLegacyWorkspaceRestoreFailure, terminalizeLegacyExecution } from import { issueService } from "../services/issues.js"; import { getExecutionBlocker } from "../services/execution-blocker.js"; import { JsonRpcCallError } from "@paperclipai/plugin-sdk"; +import { leaseFaultError } from "../services/environment-lease-fault-injection.js"; import { taskInterruptionPresentation } from "../services/task-interruption-presentation.js"; import { requestTaskRetry } from "../services/task-retry.js"; import { adapterExecutionControls, createAdapterExecutionControl } from "../services/adapter-execution-control.js"; @@ -5903,7 +5904,20 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { message: "Daytona sandbox creation failed (HTTP 502); cleanup could not be confirmed for paperclip-create-05f81d77", data: { schema: "paperclip/environment-acquire-error/v1", providerDiagnostic: { httpStatus: 502 } } }); - function failLeaseAcquisition(times: number) { + // EP1: Daytona's own acquisition deadline expired while the provisional + // sandbox was still being created, as staging recorded at 14:00-14:45Z when + // concurrent runs rose from about 8 to about 20. The environment runtime + // records the validated acquisition diagnostic for this run before rethrowing. + // The chaos fault EP1 injects these same shapes (environment-lease-fault-injection.ts). + type LeaseFault = (input: { companyId: string; environment: { id: string }; heartbeatRunId?: string | null }) => Error; + const daytonaAcquisitionDeadline: LeaseFault = input => leaseFaultError("timeout", + { companyId: input.companyId, environmentId: input.environment.id, heartbeatRunId: input.heartbeatRunId! }); + // EP1 database: a control-plane query failed while the lease was being + // recorded (staging: "Database query failed (lease_acquire_failed)"). + const leaseRecordQueryFailure: LeaseFault = input => leaseFaultError("database", + { companyId: input.companyId, environmentId: input.environment.id, heartbeatRunId: input.heartbeatRunId! }); + + function failLeaseAcquisition(times: number, fault: LeaseFault = daytonaCreate502) { let remaining = times; const acquisitions = { refused: 0, delegated: 0 }; const originalFactory = environmentOrchestrator.environmentRunOrchestrator; @@ -5913,7 +5927,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { if (remaining > 0) { remaining -= 1; acquisitions.refused += 1; - throw daytonaCreate502(); + throw fault(input); } acquisitions.delegated += 1; return runtime.acquireRunLease(input); @@ -6028,6 +6042,81 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { mockAdapterExecute.mockClear(); } }, 90_000); + it.each([ + ["a provider acquisition deadline", daytonaAcquisitionDeadline, "timeout"], + ["a database error while recording the lease", leaseRecordQueryFailure, "overloaded"], + ] as const)("retries %s before provider work with a bounded, visible backoff", async (_label, fault, category) => { + const { companyId, agentId, runId, issueId } = await seedQueuedIssueRunFixture(); + const lease = failLeaseAcquisition(1, fault); + try { + const heartbeat = heartbeatService(db); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + + const afterFailure = await runsFor(agentId); + expect(afterFailure.map(run => [run.status, run.errorCode])).toEqual([ + ["failed", "environment_provider_unavailable"], ["scheduled_retry", null]]); + expect(afterFailure[0]!.resultJson).toMatchObject({ providerFailure: { phase: "startup", category, retryability: "transient" }, + executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }); + expect(afterFailure[1]).toMatchObject({ retryOfRunId: runId, scheduledRetryAttempt: 1, scheduledRetryReason: "transient_failure" }); + const [waiting] = await db.select().from(issues).where(eq(issues.id, issueId)); + expect(waiting).toMatchObject({ status: "in_progress", executionRunId: afterFailure[1]!.id }); + expect(await taskInterruptionPresentation(db, waiting!)).toMatchObject({ state: "retry_scheduled", responsibility: "system" }); + expect(await getExecutionBlocker(db, companyId, issueId)).toBeNull(); + + await promoteLatestScheduledRetry(heartbeat, agentId); + const successors = (await runsFor(agentId)).filter(run => run.retryOfRunId === runId); + expect(successors).toHaveLength(1); + expect(successors[0]).toMatchObject({ status: "succeeded" }); + expect(lease.acquisitions.refused).toBe(1); + // The mock adapter's later disposition repair is unrelated; no setup failure reached recovery. + expect((await selectRecoveryActionsForIssue(issueId)).filter(action => action.kind === "stranded_assigned_issue")).toEqual([]); + } finally { + lease.restore(); + mockAdapterExecute.mockClear(); + } + }, 60_000); + + it("rests a provider deadline that never clears on a working Retry, never on a board recovery action", async () => { + const { companyId, agentId, issueId } = await seedQueuedIssueRunFixture(); + const lease = failLeaseAcquisition(BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS + 1, daytonaAcquisitionDeadline); + try { + const heartbeat = heartbeatService(db); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + for (let attempt = 0; attempt < BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS; attempt += 1) { + await promoteLatestScheduledRetry(heartbeat, agentId); + } + const exhausted = await runsFor(agentId); + expect(exhausted.map(run => run.errorCode)).toEqual( + Array(BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS + 1).fill("environment_provider_unavailable")); + expect(mockAdapterExecute).not.toHaveBeenCalled(); + + const blocked = await waitForValue(async () => { + const [row] = await db.select().from(issues).where(eq(issues.id, issueId)); + return row?.status === "blocked" ? row : null; + }); + const presentation = await taskInterruptionPresentation(db, blocked!); + expect(presentation).toMatchObject({ state: "failed", responsibility: "system", canRetry: true, + observedRunId: exhausted.at(-1)!.id, providerFailure: { category: "timeout", retryability: "transient" } }); + expect(await getExecutionBlocker(db, companyId, issueId)).toBeNull(); + // The task rests on a system-owned action and Retry; nothing waits on the board. + expect((await selectRecoveryActionsForIssue(issueId)).map(action => [action.ownerType, action.cause])) + .toEqual([["system", "environment_provider_unavailable"]]); + + const receipt = await requestTaskRetry({ db, companyId, issueId, actorId: "responsible-user", + idempotencyKey: randomUUID(), observedRunId: presentation!.observedRunId }); + expect(receipt.accepted).toBe(true); + await heartbeat.resumeTaskRetry(companyId, receipt.requestId!); + await heartbeat.drainActiveRunExecutions(); + expect(await waitForValue(async () => (await runsFor(agentId)).find(run => run.status === "succeeded") ?? null, 10_000)) + .not.toBeNull(); + expect(lease.acquisitions.delegated).toBe(1); + } finally { + lease.restore(); + mockAdapterExecute.mockClear(); + } + }, 90_000); }); it("classifies only the installed-but-not-ready sandbox provider plugin message as a configuration gap", () => { diff --git a/server/src/services/environment-lease-fault-injection.test.ts b/server/src/services/environment-lease-fault-injection.test.ts new file mode 100644 index 00000000000..8d163213d58 --- /dev/null +++ b/server/src/services/environment-lease-fault-injection.test.ts @@ -0,0 +1,63 @@ +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { parseLeaseFaultControl } from "./environment-lease-fault-injection.js"; + +describe("lease acquisition fault injection (chaos EP1)", () => { + let directory: string; + let control: string; + let seam: typeof import("./environment-lease-fault-injection.js"); + // Loaded after the reset, so it shares the seam's diagnostic receipts. + let readTransientLeaseAcquisitionFailure: typeof import("./environment-provider-unavailable-error.js").readTransientLeaseAcquisitionFailure; + const input = (issueId: string | null = "issue-1") => ({ companyId: "company-1", environmentId: "env-1", heartbeatRunId: "run-1", issueId }); + + beforeAll(async () => { + directory = await mkdtemp(path.join(os.tmpdir(), "lease-fault-")); + control = path.join(directory, "control.json"); + vi.stubEnv("PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL", control); + vi.resetModules(); + seam = await import("./environment-lease-fault-injection.js"); + ({ readTransientLeaseAcquisitionFailure } = await import("./environment-provider-unavailable-error.js")); + }); + afterAll(async () => { + vi.unstubAllEnvs(); + await rm(directory, { recursive: true, force: true }); + }); + + it("parses only a well-formed control", () => { + expect(parseLeaseFaultControl('{"token":"a","kind":"timeout","failures":2}')).toEqual({ token: "a", kind: "timeout", failures: 2 }); + expect(parseLeaseFaultControl('{"token":"a","kind":"database","failures":null}')).toEqual({ token: "a", kind: "database", failures: null }); + expect(parseLeaseFaultControl('{"token":"a","kind":"other","failures":1}')).toBeNull(); + expect(parseLeaseFaultControl('{"token":"a b","kind":"timeout","failures":1}')).toBeNull(); + expect(parseLeaseFaultControl("")).toBeNull(); + }); + + it("is disarmed without a control file", () => { + expect(() => seam.injectLeaseAcquisitionFault(input())).not.toThrow(); + }); + + it("fails each task's first N acquisitions with a retryable staging shape, then lets them through", async () => { + await writeFile(control, JSON.stringify({ token: "t1", kind: "timeout", failures: 2 })); + for (let attempt = 0; attempt < 2; attempt += 1) { + let thrown: unknown; + try { seam.injectLeaseAcquisitionFault(input()); } catch (error) { thrown = error; } + expect(readTransientLeaseAcquisitionFailure(thrown, { id: "run-1", companyId: "company-1" })).toEqual({ acquisitionTimedOut: true }); + } + expect(() => seam.injectLeaseAcquisitionFault(input())).not.toThrow(); + // Another task keeps its own count; a run with no task is never faulted. + expect(() => seam.injectLeaseAcquisitionFault(input("issue-2"))).toThrow(/lease acquisition timed out/); + expect(() => seam.injectLeaseAcquisitionFault(input(null))).not.toThrow(); + }); + + it("re-arms with a new token, fails forever with null, and disarms on an empty file", async () => { + await writeFile(control, JSON.stringify({ token: "t2", kind: "database", failures: null })); + for (let attempt = 0; attempt < 5; attempt += 1) { + let thrown: unknown; + try { seam.injectLeaseAcquisitionFault(input()); } catch (error) { thrown = error; } + expect(readTransientLeaseAcquisitionFailure(thrown, { id: "run-1", companyId: "company-1" })).toEqual({ reason: "database_error" }); + } + await writeFile(control, ""); + expect(() => seam.injectLeaseAcquisitionFault(input())).not.toThrow(); + }); +}); diff --git a/server/src/services/environment-lease-fault-injection.ts b/server/src/services/environment-lease-fault-injection.ts new file mode 100644 index 00000000000..ac4ce3f59c3 --- /dev/null +++ b/server/src/services/environment-lease-fault-injection.ts @@ -0,0 +1,76 @@ +/** + * Restart-chaos fault EP1 (tests/restart-chaos/env-provider-faults.ts): make a + * run's lease acquisition fail before any provider call, in the exact shapes + * staging recorded, so the bounded retry and Retry paths can be exercised on a + * local instance without a paid sandbox provider. + * + * Arming (test mode only; ignored when NODE_ENV=production): + * + * PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL= + * + * Each acquisition reads the file. Missing, empty or malformed means disarmed. + * Content: {"token":"","kind":"timeout"|"database","failures":|null}. + * The first acquisitions of each task under one token fail (null: every + * one), so writing a new token re-arms with fresh counts and an empty file + * disarms. Counts live in this process, so a restart starts them again. + * + * This is test instrumentation, not Telemetry or Observability + * (rules/data-paths.mdc): nothing is recorded or sent anywhere. + */ +import { readFileSync } from "node:fs"; +import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES, readEnvironmentAcquisitionDiagnostic } from "@paperclipai/plugin-sdk"; +import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js"; + +export const LEASE_FAULT_CONTROL_ENV = "PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL"; + +export type LeaseFaultControl = { token: string; kind: "timeout" | "database"; failures: number | null }; + +export function parseLeaseFaultControl(text: string): LeaseFaultControl | null { + try { + const value = JSON.parse(text) as Record; + if (typeof value.token !== "string" || !/^[A-Za-z0-9_-]{1,80}$/.test(value.token)) return null; + if (value.kind !== "timeout" && value.kind !== "database") return null; + const failures = value.failures; + if (failures !== null && !(Number.isSafeInteger(failures) && (failures as number) >= 0)) return null; + return { token: value.token, kind: value.kind, failures: failures as number | null }; + } catch { return null; } +} + +const controlPath = process.env.NODE_ENV === "production" ? null : process.env[LEASE_FAULT_CONTROL_ENV]?.trim() || null; +const counts = new Map(); + +/** The staging shapes: Daytona's acquisition deadline, and a failed control-plane query. */ +export function leaseFaultError(kind: LeaseFaultControl["kind"], input: { + companyId: string; environmentId: string; heartbeatRunId: string; +}): Error { + if (kind === "database") { + return new Error("Failed query: insert into \"environment_leases\" (\"id\") values ($1)\nparams: ", { + cause: Object.assign(new Error("terminating connection due to administrator command"), { name: "PostgresError", code: "57P01" }), + }); + } + const error = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR, + message: "Daytona lease acquisition timed out; allocation cleanup is pending", + data: { schema: "paperclip/environment-creation-cleanup/v1", + cleanup: { providerLeaseId: `chaos-ep1-${input.heartbeatRunId.slice(0, 8)}`, companyId: input.companyId, + environmentId: input.environmentId, runId: input.heartbeatRunId, attemptId: input.heartbeatRunId, + accountFingerprint: "0".repeat(64), labels: {} }, + acquisitionDiagnostic: { phase: "create", elapsedMs: 300_000, budgetMs: 300_000, providerState: "creating" } } }); + captureEnvironmentAcquisitionDiagnostic(error, { companyId: input.companyId, environmentId: input.environmentId, + runId: input.heartbeatRunId }, readEnvironmentAcquisitionDiagnostic(error)); + return error; +} + +/** Throws the armed fault for this acquisition; a single null check when disarmed. */ +export function injectLeaseAcquisitionFault(input: { + companyId: string; environmentId: string; heartbeatRunId: string; issueId: string | null; +}): void { + if (controlPath === null || !input.issueId) return; + let control: LeaseFaultControl | null; + try { control = parseLeaseFaultControl(readFileSync(controlPath, "utf8")); } catch { return; } + if (!control) return; + const key = `${control.token}:${input.issueId}`; + const seen = counts.get(key) ?? 0; + if (control.failures !== null && seen >= control.failures) return; + counts.set(key, seen + 1); + throw leaseFaultError(control.kind, input); +} diff --git a/server/src/services/environment-provider-unavailable-error.test.ts b/server/src/services/environment-provider-unavailable-error.test.ts index 319647827a3..7c568c5b18f 100644 --- a/server/src/services/environment-provider-unavailable-error.test.ts +++ b/server/src/services/environment-provider-unavailable-error.test.ts @@ -1,8 +1,11 @@ import { describe, expect, it } from "vitest"; +import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES, readEnvironmentAcquisitionDiagnostic } from "@paperclipai/plugin-sdk"; +import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js"; import { EnvironmentProviderUnavailableError, findEnvironmentProviderUnavailableError, readTransientEnvironmentProviderFailure, + readTransientLeaseAcquisitionFailure, } from "./environment-provider-unavailable-error.js"; function withStatus(status: number, key: "status" | "statusCode" = "status") { @@ -112,3 +115,78 @@ describe("EnvironmentProviderUnavailableError", () => { expect(findEnvironmentProviderUnavailableError(new Error("plain"))).toBeNull(); }); }); + +describe("readTransientLeaseAcquisitionFailure", () => { + const run = { id: "run-1", companyId: "company-1" }; + const creationFailure = (elapsedMs: number, runId = run.id) => { + const error = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR, + message: "Daytona lease acquisition timed out; allocation cleanup is pending", + data: { schema: "paperclip/environment-creation-cleanup/v1", + cleanup: { providerLeaseId: "paperclip-create-1", companyId: run.companyId, environmentId: "env-1", runId, + attemptId: "attempt-1", accountFingerprint: "a".repeat(64), labels: {} }, + acquisitionDiagnostic: { phase: "create", elapsedMs, budgetMs: 300_000 } } }); + captureEnvironmentAcquisitionDiagnostic(error, { companyId: run.companyId, environmentId: "env-1", runId }, + readEnvironmentAcquisitionDiagnostic(error)); + return error; + }; + + it("keeps the provider's transient status", () => { + expect(readTransientLeaseAcquisitionFailure(withStatus(503), run)).toEqual({ httpStatus: 503 }); + }); + + it("retries a creation that hit the acquisition deadline", () => { + expect(readTransientLeaseAcquisitionFailure(new Error("wrapped", { cause: creationFailure(300_000) }), run)) + .toEqual({ acquisitionTimedOut: true }); + }); + + it("retries a provisional creation that did not finish", () => { + expect(readTransientLeaseAcquisitionFailure(creationFailure(12_000), run)).toEqual({ reason: "acquisition_incomplete" }); + }); + + it("ignores an unfinished-creation diagnostic recorded for another run", () => { + expect(readTransientLeaseAcquisitionFailure(creationFailure(12_000, "run-2"), run)).toBeNull(); + }); + + it("retries a plugin RPC deadline", () => { + const timeout = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.TIMEOUT, message: "RPC call timed out" }); + expect(readTransientLeaseAcquisitionFailure(timeout, run)).toEqual({ acquisitionTimedOut: true }); + }); + + const pg = (code: string) => Object.assign(new Error("database error"), { name: "PostgresError", code }); + it.each([ + ["a terminated connection", pg("57P01")], + ["a connection failure", pg("08006")], + ["too many connections", pg("53300")], + ["a lock timeout", pg("55P03")], + ["a closed connection", Object.assign(new Error("closed"), { code: "CONNECTION_CLOSED" })], + ["a Drizzle wrapper around a statement timeout", new Error("Failed query: select 1\nparams: ", { cause: pg("57014") })], + ])("retries %s while recording the lease", (_label, cause) => { + expect(readTransientLeaseAcquisitionFailure(new Error("lease", { cause }), run)).toEqual({ reason: "database_error" }); + }); + + it.each([ + ["a unique violation", pg("23505")], + ["an undefined column", pg("42703")], + ["a Drizzle wrapper around a syntax error", new Error("Failed query: select\nparams: ", { cause: pg("42601") })], + ["a bare Drizzle wrapper", new Error("Failed query: select 1\nparams: ")], + ["an errno that looks like a SQLSTATE", Object.assign(new Error("x"), { code: "EPERM" })], + ])("does not retry %s", (_label, cause) => { + expect(readTransientLeaseAcquisitionFailure(new Error("lease", { cause }), run)).toBeNull(); + }); + + it("does not retry a definite non-transient status, even with other evidence", () => { + const refused = Object.assign(new Error("unauthorized", { cause: creationFailure(300_000) }), { status: 401 }); + expect(readTransientLeaseAcquisitionFailure(refused, run)).toBeNull(); + }); + + it("does not retry an error without structured evidence", () => { + expect(readTransientLeaseAcquisitionFailure(new Error("Daytona lease acquisition timed out"), run)).toBeNull(); + }); + + it("names the reason in the board-facing failure", () => { + const error = new EnvironmentProviderUnavailableError({ environmentName: "Sandbox", driver: "daytona", + evidence: { reason: "database_error" }, cause: null }); + expect(error.providerFailure).toMatchObject({ category: "overloaded", providerCode: "database_error" }); + expect(error.message).toContain("a database query failed while recording the lease"); + }); +}); diff --git a/server/src/services/environment-provider-unavailable-error.ts b/server/src/services/environment-provider-unavailable-error.ts index 488a57bda00..7f4d54adc9f 100644 --- a/server/src/services/environment-provider-unavailable-error.ts +++ b/server/src/services/environment-provider-unavailable-error.ts @@ -1,5 +1,6 @@ -import { readEnvironmentAcquireErrorDiagnostic, readEnvironmentAcquisitionDiagnostic } from "@paperclipai/plugin-sdk"; +import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES, readEnvironmentAcquireErrorDiagnostic, readEnvironmentAcquisitionDiagnostic } from "@paperclipai/plugin-sdk"; import type { ProviderFailure } from "@paperclipai/shared"; +import { getEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js"; /** * Setup failure code for a lease acquisition that the environment provider @@ -21,7 +22,13 @@ const TRANSIENT_ERROR_CODES = new Set([ "UND_ERR_CONNECT_TIMEOUT", "UND_ERR_SOCKET", ]); -type TransientProviderEvidence = { httpStatus?: number; errorCode?: string; acquisitionTimedOut?: true }; +/** + * `reason` names evidence that has neither a provider status nor a deadline: + * the provider reported a provisional creation that did not finish, or a + * control-plane database query failed while the lease was taken. + */ +type TransientLeaseFailureReason = "acquisition_incomplete" | "database_error"; +type TransientProviderEvidence = { httpStatus?: number; errorCode?: string; acquisitionTimedOut?: true; reason?: TransientLeaseFailureReason }; function own(value: unknown, key: string): unknown { try { @@ -54,6 +61,69 @@ export function readTransientEnvironmentProviderFailure(error: unknown): Transie return null; } +/** A definite non-transient HTTP status anywhere on the chain is authoritative: retrying cannot change it. */ +function definiteNonTransientStatus(error: unknown): boolean { + for (let cause = error, depth = 0; cause && depth < 5; cause = own(cause, "cause"), depth++) { + const status = readEnvironmentAcquireErrorDiagnostic(cause)?.httpStatus + ?? [own(cause, "status"), own(cause, "statusCode")].find(value => Number.isInteger(value)); + if (typeof status === "number" && status >= 400 && !TRANSIENT_HTTP_STATUSES.has(status)) return true; + } + return false; +} + +// postgres.js connection-level codes, and SQLSTATEs a repeated statement can +// pass: connection (08), insufficient resources (53), operator intervention +// (57P01-57P03, 57014 query cancelled or timed out), lock not available (55P03), +// serialization failure and deadlock (40001, 40P01). A deterministic SQL error +// (syntax, schema, constraint) is not retried. +const DATABASE_CONNECTION_CODES = new Set(["CONNECTION_CLOSED", "CONNECTION_ENDED", "CONNECTION_DESTROYED", "CONNECT_TIMEOUT"]); +const TRANSIENT_SQLSTATES = new Set(["57P01", "57P02", "57P03", "57014", "55P03", "40001", "40P01"]); + +function transientDatabaseFailure(error: unknown): boolean { + for (let cause = error, depth = 0; cause && depth < 5; cause = own(cause, "cause"), depth++) { + const code = own(cause, "code"); + if (typeof code !== "string") continue; + if (DATABASE_CONNECTION_CODES.has(code)) return true; + if (/^[0-9A-Z]{5}$/.test(code) && (code.startsWith("08") || code.startsWith("53") || TRANSIENT_SQLSTATES.has(code))) { + // A five-character code is a SQLSTATE only on a database driver error. + const name = own(cause, "name"); + if (name === "PostgresError" || name === "DatabaseError" || own(cause, "severity") !== undefined) return true; + } + } + return false; +} + +/** + * Classify a failed lease acquisition for the bounded bootstrap retry. Lease + * acquisition precedes any provider (agent) work by construction, and every + * retry is a new creation attempt with its own idempotent identity, so a + * failure is retried unless it carries a definite non-transient status. Only + * structured evidence is read: the provider's status or errno, the validated + * acquisition diagnostic the runtime recorded for this run, the plugin RPC + * deadline, or a transient database driver code. Message text is never parsed. + */ +export function readTransientLeaseAcquisitionFailure( + error: unknown, + run: { id: string; companyId: string }, +): TransientProviderEvidence | null { + const provider = readTransientEnvironmentProviderFailure(error); + if (provider) return provider; + if (definiteNonTransientStatus(error)) return null; + for (let cause = error, depth = 0; cause && depth < 5; cause = own(cause, "cause"), depth++) { + const acquisition = getEnvironmentAcquisitionDiagnostic(cause, run); + if (acquisition) { + return acquisition.elapsedMs >= acquisition.budgetMs ? { acquisitionTimedOut: true } : { reason: "acquisition_incomplete" }; + } + if (cause instanceof JsonRpcCallError && cause.code === PLUGIN_RPC_ERROR_CODES.TIMEOUT) return { acquisitionTimedOut: true }; + } + return transientDatabaseFailure(error) ? { reason: "database_error" } : null; +} + +const REASON_LABELS: Record = { + acquisition_incomplete: "sandbox creation did not finish", + database_error: "a database query failed while recording the lease", +}; + function category(evidence: TransientProviderEvidence): ProviderFailure["category"] { if (evidence.httpStatus === 429) return "rate_limit"; if (evidence.acquisitionTimedOut || evidence.httpStatus === 408 || evidence.httpStatus === 504 @@ -70,13 +140,15 @@ export class EnvironmentProviderUnavailableError extends Error { readonly providerFailure: ProviderFailure; constructor(input: { environmentName: string; driver: string; evidence: TransientProviderEvidence; cause: unknown }) { - const status = input.evidence.httpStatus !== undefined ? `HTTP ${input.evidence.httpStatus}` : input.evidence.errorCode; + const status = input.evidence.httpStatus !== undefined ? `HTTP ${input.evidence.httpStatus}` + : input.evidence.errorCode ?? (input.evidence.reason ? REASON_LABELS[input.evidence.reason] : undefined); super(status ? `Environment provider for "${input.environmentName}" was temporarily unavailable (${status}).` : `Environment provider for "${input.environmentName}" did not finish starting the sandbox in time.`, { cause: input.cause }); this.name = "EnvironmentProviderUnavailableError"; this.httpStatus = input.evidence.httpStatus ?? null; this.errorCode = input.evidence.errorCode ?? null; + const providerCode = input.evidence.errorCode ?? (input.evidence.httpStatus === undefined ? input.evidence.reason : undefined); this.providerFailure = { schema: "paperclip.provider-failure.v1", provider: /^[a-zA-Z0-9_.:/-]{1,80}$/.test(input.driver) ? input.driver : "environment", @@ -87,7 +159,7 @@ export class EnvironmentProviderUnavailableError extends Error { : `The sandbox for "${input.environmentName}" did not finish starting in time. `) + "No agent work had started. Paperclip retries automatically; if those retries are used up, use Retry once the provider is available.", ...(input.evidence.httpStatus !== undefined ? { httpStatus: input.evidence.httpStatus } : {}), - ...(input.evidence.errorCode !== undefined ? { providerCode: input.evidence.errorCode } : {}), + ...(providerCode !== undefined ? { providerCode } : {}), retryability: "transient", }; } diff --git a/server/src/services/environment-run-orchestrator.ts b/server/src/services/environment-run-orchestrator.ts index 8149b84004f..aa1c0273f62 100644 --- a/server/src/services/environment-run-orchestrator.ts +++ b/server/src/services/environment-run-orchestrator.ts @@ -22,7 +22,8 @@ import { environmentLeases, type Db } from "@paperclipai/db"; import { and, eq } from "drizzle-orm"; import { remoteExecutionHasStopped } from "./remote-execution-termination.js"; import { remoteRecoveryLeaseMismatch, type RemoteRecoveryLeaseMismatchField } from "./native-runtime/remote-recovery-lease-mismatch.js"; -import { EnvironmentProviderUnavailableError, readTransientEnvironmentProviderFailure } from "./environment-provider-unavailable-error.js"; +import { EnvironmentProviderUnavailableError, readTransientLeaseAcquisitionFailure } from "./environment-provider-unavailable-error.js"; +import { injectLeaseAcquisitionFault } from "./environment-lease-fault-injection.js"; import { findReusableSandboxResumeError } from "./reusable-sandbox-resume-error.js"; import { findPluginWorkerUnavailableError } from "./plugin-worker-unavailable-error.js"; import type { @@ -232,13 +233,18 @@ export function environmentRunOrchestrator( runnerReconnectGraceMs?: number; }): Promise { try { + // Restart-chaos EP1 only; disarmed (always in production) this is one null check. + injectLeaseAcquisitionFault({ companyId: input.companyId, environmentId: input.environment.id, + heartbeatRunId: input.heartbeatRunId, issueId: input.issueId }); return await environmentRuntime.acquireRunLease({ ...input, assertCompanyBinding: true }); } catch (err) { - // A transient provider refusal (HTTP 5xx/429/timeout) is retried by the - // setup path with its bounded bootstrap budget. Saved-sandbox resume and + // Lease acquisition precedes any provider work, so a failure without a + // definite non-transient status (HTTP 5xx/429, errno, acquisition + // deadline, unfinished creation, database error) is retried by the setup + // path with its bounded bootstrap budget. Saved-sandbox resume and // stopped-worker refusals keep their own, more specific classifications. const transient = findReusableSandboxResumeError(err) || findPluginWorkerUnavailableError(err) - ? null : readTransientEnvironmentProviderFailure(err); + ? null : readTransientLeaseAcquisitionFailure(err, { id: input.heartbeatRunId, companyId: input.companyId }); throw new EnvironmentRunError( "lease_acquire_failed", `Failed to acquire lease for environment "${input.environment.name}" (${input.environment.driver}): ${err instanceof Error ? err.message : String(err)}`, diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index 65292d727b1..b4700a75722 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -299,6 +299,7 @@ export type StrandedRecoveryCause = | "deliberate_wait_without_target" | "process_lost" | "provider_quota" + | typeof ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE | "codex_output_inactivity_monitor" | "workspace_validation_failed" | "configuration_incomplete" @@ -444,12 +445,24 @@ function isProviderQuotaRecovery(latestRun: LatestIssueRun) { ); } +/** + * Lease acquisition failed before any provider work, and its bounded retry + * budget is spent. Nothing needs inspection or repair by the board: the task + * rests on a system-owned action and the normal Retry, never on a board one. + */ +function isPreDispatchEnvironmentProviderFailure(latestRun: LatestIssueRun) { + if (latestRun?.errorCode !== ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE) return false; + const recovery = parseObject(parseObject(latestRun.resultJson).executionRecovery); + return recovery.kind === "bootstrap" && recovery.providerWorkStarted === false; +} + function resolveStrandedRecoveryCause( latestRun: LatestIssueRun, explicitCause?: StrandedRecoveryCause, ): StrandedRecoveryCause { if (explicitCause) return explicitCause; if (isProviderQuotaRecovery(latestRun)) return "provider_quota"; + if (isPreDispatchEnvironmentProviderFailure(latestRun)) return ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE; if (latestRun?.errorCode === "process_lost") return "process_lost"; if (latestRun?.errorCode === "codex_output_inactivity_monitor") { return "codex_output_inactivity_monitor"; @@ -2553,6 +2566,7 @@ export function recoveryService( latestRun: input.latestRun, }); const isProviderQuotaWait = recoveryCause === "provider_quota"; + const isEnvironmentProviderWait = recoveryCause === ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE; const now = new Date(); const actions = database === db ? recoveryActionsSvc : issueRecoveryActionService(database); const action = await actions.upsertSourceScoped({ @@ -2565,7 +2579,7 @@ export function recoveryService( supersedeOnIdentityChange: recoveryCause === "configuration_incomplete", preserveExistingOwner: true, kind: strandedRecoveryActionKind(recoveryCause), - ownerType: isProviderQuotaWait ? "system" : "board", + ownerType: isProviderQuotaWait || isEnvironmentProviderWait ? "system" : "board", ownerAgentId: null, ownerUserId: null, previousOwnerAgentId: input.issue.assigneeAgentId, @@ -2587,7 +2601,7 @@ export function recoveryService( failureSummary: summarizeRunFailureForIssueComment(input.latestRun)?.trim() ?? null, }, - evidenceOnCreate: isProviderQuotaWait + evidenceOnCreate: isProviderQuotaWait || isEnvironmentProviderWait ? {} : { routingPolicy: STRANDED_BOARD_ESCALATION_POLICY }, nextAction: @@ -2597,6 +2611,8 @@ export function recoveryService( ? "Board operator: inspect the retry history, then explicitly retry the original owner, reassign, or intentionally resolve the task." : recoveryCause === "provider_quota" ? "Wait for provider quota recovery, then retry the original assignee; do not wake a takeover owner." + : isEnvironmentProviderWait + ? "No agent work started. Retry the task once the environment provider is available." : recoveryCause === "codex_output_inactivity_monitor" ? "Board operator: inspect the inactivity evidence, then explicitly retry the original owner, reassign, or intentionally resolve the task." : recoveryCause === "workspace_validation_failed" @@ -2632,6 +2648,8 @@ export function recoveryService( type: "monitor_only", reason: recoveryCause, } + : isEnvironmentProviderWait + ? { type: "manual_retry", reason: recoveryCause, preservesSourceAssignee: true } : { type: "board_escalation", reason: recoveryCause, diff --git a/server/src/services/recovery/stranded-notice.test.ts b/server/src/services/recovery/stranded-notice.test.ts index 8f810f0e1bf..551187539f4 100644 --- a/server/src/services/recovery/stranded-notice.test.ts +++ b/server/src/services/recovery/stranded-notice.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it } from "vitest"; import { noticeMetadataReferencesRecoveryAction } from "./successful-run-handoff.js"; +import { ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE } from "../environment-provider-unavailable-error.js"; import { DEFAULT_STRANDED_RECOVERY_NOTICE_BODY, buildConfigurationIncompleteRecoveryNoticeSeed, @@ -147,6 +148,19 @@ describe("buildStrandedRecoveryEscalationNotice", () => { expect(notice.body).toContain("before the agent started"); expect(JSON.stringify(notice.metadata)).toContain("worker status"); }); + it("asks for Retry, not a board decision, when the environment provider stayed unavailable", () => { + const notice = buildStrandedRecoveryEscalationNotice({ + seed: buildImmediateExecutionPathRecoveryNoticeSeed({ status: "in_progress" }), + recoveryCause: ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE, + recoveryActionId: "provider-recovery", + recoveryOwner: null, + sourceRun: { id: "failed-run", status: "failed", errorCode: ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE }, + }); + const metadata = JSON.stringify(notice.metadata); + expect(notice.presentation.title).toBe("Environment provider was temporarily unavailable"); + expect(metadata).toContain("Retry when ready"); + expect(metadata).not.toContain("Board decision required"); + }); const actionId = "6a2f8e64-6f5e-4b58-b7fd-111111111111"; const owner = { id: "9b1c2d3e-4f50-4a61-8b72-222222222222", name: "CTO" }; const sourceRun = { diff --git a/server/src/services/recovery/stranded-notice.ts b/server/src/services/recovery/stranded-notice.ts index 694a6689aef..1367cd2e7e3 100644 --- a/server/src/services/recovery/stranded-notice.ts +++ b/server/src/services/recovery/stranded-notice.ts @@ -1,4 +1,5 @@ import type { IssueCommentMetadata, IssueCommentPresentation } from "@paperclipai/shared"; +import { ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE } from "../environment-provider-unavailable-error.js"; import { agentLinkRow, keyValueRow, @@ -281,7 +282,8 @@ export function buildStrandedRecoveryEscalationNotice(input: { ? agentLinkRow("Recovery owner", input.recoveryOwner) : keyValueRow( "Recovery owner", - "Board decision required", + // A pre-dispatch provider outage needs no board decision, only Retry. + input.recoveryCause === ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE ? "Retry when ready" : "Board decision required", ), keyValueRow( "Next action", diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index 6d08b5bd47f..bd69b2022d5 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -461,6 +461,30 @@ The observer clicks Retry in the board UI, as for every other cell. In waves `w- and `w-20261111T134316-s9413` the disk-full rest showed "Recover saved files", not a Retry button, while the API reported `canRetry: true`; R3 records this as `ui_retry_button_missing`. +## Environment-provider faults (EP1, `env-provider-wave.ts`) + +```sh +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/env-provider-wave.ts --fault EP1 --adapters codex --tasks 2 --seed 71 +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/env-provider-wave.ts --fault EP1N --tasks 1 --seed 72 +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/env-provider-wave.ts --fault EP1D --tasks 1 --seed 73 +``` + +Lease acquisition fails before any provider call, in the shapes staging recorded +on 2026-11-11: Daytona's acquisition deadline ("lease acquisition timed out; +allocation cleanup is pending") and a failed database query. The server seam +`server/src/services/environment-lease-fault-injection.ts` (test mode only) reads +`PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL` and fails each task's first N acquisitions. +It runs on a local instance; no sandbox provider is needed. One fault per wave. + +| Fault | Shape | Pass | +| --- | --- | --- | +| EP1 | Deadline, 2 times (the automatic bound), then succeeds | Exactly 2 `environment_provider_unavailable` runs, no `setup_failed`, completes with no user action | +| EP1N | Deadline on every attempt | 3 failed runs, the task rests offering Retry; the harness disarms, clicks Retry, and the task completes | +| EP1D | Database error once, then succeeds | 1 failed run, completes with no user action | + +Every fault also fails if a board-owned recovery action is ever active on the +task, and the A1 brief oracle must pass. Results go to `results/-ep/`. + ## Config faults (CF1–CF3, `config-wave.ts`) ```sh diff --git a/tests/restart-chaos/env-provider-faults.test.ts b/tests/restart-chaos/env-provider-faults.test.ts new file mode 100644 index 00000000000..a17301c1398 --- /dev/null +++ b/tests/restart-chaos/env-provider-faults.test.ts @@ -0,0 +1,49 @@ +import { describe, expect, it } from "vitest"; +import { BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS } from "../../server/src/services/heartbeat/retries.js"; +import { parseLeaseFaultControl } from "../../server/src/services/environment-lease-fault-injection.js"; +import { AUTOMATIC_RETRY_BOUND, ENV_PROVIDER_FAULTS, gradeEnvProviderTask, leaseFaultControl, type EpTaskRecord } from "./env-provider-faults.js"; + +const base = (overrides: Partial): EpTaskRecord => ({ + taskId: "t01", nonce: "n", fault: "EP1", issueId: "issue", harnessError: null, runs: [], activeRecoveryActions: [], + recoveryReads: 3, recoveryReadFailures: 0, rested: null, retryClicked: false, finalStatus: "done", briefOk: true, ...overrides, +}); +const failedRun = (id: string, errorCode = "environment_provider_unavailable") => ({ id, status: "failed", errorCode }); + +describe("EP1 environment-provider faults", () => { + it("tracks the server's automatic retry bound", () => { + expect(AUTOMATIC_RETRY_BOUND).toBe(BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS); + }); + + it("writes a control the server seam accepts", () => { + for (const fault of Object.values(ENV_PROVIDER_FAULTS)) { + expect(parseLeaseFaultControl(leaseFaultControl(fault, "wave-1"))).toEqual({ token: "wave-1", kind: fault.shape, failures: fault.failures }); + } + }); + + it("passes a task that retried within the bound and completed by itself", () => { + const grade = gradeEnvProviderTask(base({ runs: [failedRun("a"), failedRun("b"), { id: "c", status: "succeeded", errorCode: null }] })); + expect(grade.pass).toBe(true); + }); + + it("fails the staging shape: a generic setup failure stranded on a board action", () => { + const grade = gradeEnvProviderTask(base({ runs: [failedRun("a", "setup_failed")], finalStatus: "blocked", briefOk: false, + activeRecoveryActions: [{ ownerType: "board", kind: "stranded_assigned_issue", cause: "stranded_assigned_issue" }] })); + expect(grade.checks.filter(check => !check.pass).map(check => check.name)) + .toEqual(["bounded_retry", "no_board_recovery_action", "completes_without_user", "brief_oracle"]); + }); + + it("fails when recovery actions could not be read", () => { + const runs = [failedRun("a"), failedRun("b"), { id: "c", status: "succeeded", errorCode: null }]; + expect(gradeEnvProviderTask(base({ runs, recoveryReadFailures: 1 })).checks.find(check => check.name === "recovery_evidence_read")?.pass).toBe(false); + expect(gradeEnvProviderTask(base({ runs, recoveryReads: 0 })).pass).toBe(false); + }); + + it("requires a never-clearing fault to rest on Retry and finish after it", () => { + const runs = [failedRun("a"), failedRun("b"), failedRun("c"), { id: "d", status: "succeeded", errorCode: null }]; + expect(gradeEnvProviderTask(base({ fault: "EP1N", runs, rested: { issueStatus: "blocked", canRetry: true, observedRunId: "c" }, + retryClicked: true, activeRecoveryActions: [{ ownerType: "system", kind: "stranded_assigned_issue", cause: "environment_provider_unavailable" }] })).pass) + .toBe(true); + expect(gradeEnvProviderTask(base({ fault: "EP1N", runs, rested: { issueStatus: "blocked", canRetry: false, observedRunId: null } })).pass) + .toBe(false); + }); +}); diff --git a/tests/restart-chaos/env-provider-faults.ts b/tests/restart-chaos/env-provider-faults.ts new file mode 100644 index 00000000000..337e33fb0d7 --- /dev/null +++ b/tests/restart-chaos/env-provider-faults.ts @@ -0,0 +1,106 @@ +/** + * Environment-provider fault family (EP1): pure catalog, control format and oracle. + * + * Staging (2026-11-11, 14:00-14:45Z) had five Daytona lease acquisitions time + * out ("Daytona lease acquisition timed out; allocation cleanup is pending") + * and one fail on a database query, all before any provider work. None got a + * bounded retry, and two tasks were left blocked on a board-owned + * `stranded_assigned_issue` recovery action. Every other harness fault hits + * the control plane, storage or the provider stream; nothing failed lease + * acquisition itself. + * + * The server seam is `server/src/services/environment-lease-fault-injection.ts` + * (test mode only): it throws the staging error shapes for each task's first + * N acquisitions, before any provider call, so a local wave exercises the + * same retry and Retry paths a Daytona outage does. + * + * Oracle, per task: + * 1. bounded retry: exactly the expected number of runs fail with + * `environment_provider_unavailable`, never a generic `setup_failed`; + * 2. no board-owned recovery action is ever active on the task; + * 3. EP1/EP1D: the task completes with no user action; + * EP1N (never clears): the task rests offering Retry, and after the + * harness disarms the fault and clicks Retry the task completes; + * 4. the A1 brief oracle passes. + * + * I/O lives in env-provider-wave.ts; this file is credential-free and unit tested. + */ + +export const ENV_PROVIDER_FAULT_KINDS = ["EP1", "EP1N", "EP1D"] as const; +export type EnvProviderFaultKind = (typeof ENV_PROVIDER_FAULT_KINDS)[number]; + +/** BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS in server/src/services/heartbeat/retries.ts. */ +export const AUTOMATIC_RETRY_BOUND = 2; + +export interface EnvProviderFault { + kind: EnvProviderFaultKind; + label: string; + /** The server seam's error shape. */ + shape: "timeout" | "database"; + /** Failures per task; null never clears until the harness disarms it. */ + failures: number | null; +} + +export const ENV_PROVIDER_FAULTS: Record = { + EP1: { kind: "EP1", label: "Lease acquisition times out N times, then succeeds", shape: "timeout", failures: AUTOMATIC_RETRY_BOUND }, + EP1N: { kind: "EP1N", label: "Lease acquisition times out on every attempt", shape: "timeout", failures: null }, + EP1D: { kind: "EP1D", label: "A database query fails during lease acquisition once", shape: "database", failures: 1 }, +}; + +/** The control file the seam reads (PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL). An empty file disarms. */ +export function leaseFaultControl(fault: EnvProviderFault, token: string): string { + return JSON.stringify({ token, kind: fault.shape, failures: fault.failures }); +} + +/** Runs of the task in creation order, as `/api/issues/:id/runs` reports them. */ +export interface EpRunSample { id: string; status: string; errorCode: string | null } + +export interface EpTaskRecord { + taskId: string; + nonce: string; + fault: EnvProviderFaultKind; + issueId: string | null; + harnessError: string | null; + runs: EpRunSample[]; + /** Every recovery action seen active on the task at any poll. */ + activeRecoveryActions: Array<{ ownerType: string; kind: string; cause: string | null }>; + /** Polls that read the task's recovery actions, and polls where that read failed. */ + recoveryReads: number; + recoveryReadFailures: number; + /** EP1N: the resting state before Retry. */ + rested: { issueStatus: string; canRetry: boolean; observedRunId: string | null } | null; + retryClicked: boolean; + finalStatus: string | null; + briefOk: boolean | null; +} + +export interface EpCheck { name: string; pass: boolean; detail: string } +export interface EpGrade { pass: boolean; checks: EpCheck[] } + +export function gradeEnvProviderTask(record: EpTaskRecord): EpGrade { + const fault = ENV_PROVIDER_FAULTS[record.fault]; + const failed = record.runs.filter(run => run.errorCode === "environment_provider_unavailable").length; + const generic = record.runs.filter(run => run.errorCode === "setup_failed").length; + const expected = fault.failures ?? AUTOMATIC_RETRY_BOUND + 1; + const board = record.activeRecoveryActions.filter(action => action.ownerType === "board"); + const checks: EpCheck[] = [ + { name: "no_harness_error", pass: record.harnessError === null, detail: record.harnessError ?? "ok" }, + { name: "bounded_retry", pass: failed === expected && generic === 0, + detail: `${failed} environment_provider_unavailable run(s), expected ${expected}; ${generic} setup_failed` }, + { name: "recovery_evidence_read", pass: record.recoveryReads > 0 && record.recoveryReadFailures === 0, + detail: `${record.recoveryReads} read(s), ${record.recoveryReadFailures} failed` }, + { name: "no_board_recovery_action", pass: board.length === 0, + detail: board.length ? board.map(action => `${action.kind}/${action.cause ?? "-"}`).join(", ") : "none" }, + ]; + if (fault.failures === null) { + checks.push({ name: "rests_with_retry", pass: record.rested !== null && record.rested.canRetry, + detail: record.rested ? `${record.rested.issueStatus}, canRetry=${String(record.rested.canRetry)}` : "never rested" }); + checks.push({ name: "retry_completes", pass: record.retryClicked && record.finalStatus === "done", + detail: `retry ${record.retryClicked ? "clicked" : "not clicked"}, final ${record.finalStatus ?? "unknown"}` }); + } else { + checks.push({ name: "completes_without_user", pass: !record.retryClicked && record.finalStatus === "done", + detail: `final ${record.finalStatus ?? "unknown"}` }); + } + checks.push({ name: "brief_oracle", pass: record.briefOk === true, detail: record.briefOk === null ? "not read" : String(record.briefOk) }); + return { pass: checks.every(check => check.pass), checks }; +} diff --git a/tests/restart-chaos/env-provider-wave.ts b/tests/restart-chaos/env-provider-wave.ts new file mode 100644 index 00000000000..f08c27506f1 --- /dev/null +++ b/tests/restart-chaos/env-provider-wave.ts @@ -0,0 +1,205 @@ +/** + * Environment-provider fault waves (EP1). See env-provider-faults.ts for the + * oracle and README.md ("Environment-provider faults") for usage. + * + * node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/env-provider-wave.ts \ + * --fault EP1 --adapters codex --tasks 2 --seed 71 + * + * One fault per wave: the server seam reads one control file, and every task's + * own first acquisitions fail. The harness launches A1 brief tasks, watches + * their runs and recovery actions, and for EP1N disarms the fault once every + * task rests and then uses the task's Retry. + */ +import { spawnSync } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { existsSync } from "node:fs"; +import { mkdir, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { parseArgs } from "node:util"; +import { setTimeout as delay } from "node:timers/promises"; +import { releaseInstance } from "../runner-e2e/instance-retention.js"; +import { ChaosApi } from "./api.js"; +import { CREDENTIAL_FOR_ADAPTER, waveIdFor } from "./config.js"; +import { + ENV_PROVIDER_FAULT_KINDS, ENV_PROVIDER_FAULTS, gradeEnvProviderTask, leaseFaultControl, + type EnvProviderFaultKind, type EpGrade, type EpTaskRecord, +} from "./env-provider-faults.js"; +import { knownSecretValues, redactKnownSecrets } from "./evidence.js"; +import { seedChaosLab, type ChaosLab } from "./seed.js"; +import { ChaosSupervisor } from "./supervisor.js"; +import { Timeline } from "./timeline.js"; +import { ADAPTER_IDS, type AdapterId } from "./types.js"; +import { workloadDefinitions } from "./workloads.js"; + +const repositoryRoot = path.resolve(import.meta.dirname, "../.."); +const resultsRoot = path.resolve(process.env.RESTART_CHAOS_RESULTS_DIR ?? path.join(import.meta.dirname, "results")); +const POLL_MS = 4_000; +type Json = Record; + +interface WaveOptions { fault: EnvProviderFaultKind; adapters: AdapterId[]; tasks: number; seed: number; restBoundMs: number; doneBoundMs: number } + +export function parseEnvProviderWaveArgs(argv: readonly string[]): { options: WaveOptions; keep: boolean } { + const { values } = parseArgs({ args: argv[0] === "--" ? argv.slice(1) : [...argv], strict: true, options: { + fault: { type: "string" }, adapters: { type: "string" }, tasks: { type: "string" }, seed: { type: "string" }, + "rest-seconds": { type: "string" }, "done-seconds": { type: "string" }, keep: { type: "boolean" }, + } }); + const fault = (values.fault ?? "EP1") as EnvProviderFaultKind; + if (!ENV_PROVIDER_FAULT_KINDS.includes(fault)) throw new Error(`--fault must be one of ${ENV_PROVIDER_FAULT_KINDS.join(", ")}`); + const adapters = (values.adapters ?? "codex").split(",").map(item => item.trim()).filter(Boolean) as AdapterId[]; + for (const adapter of adapters) if (!ADAPTER_IDS.includes(adapter)) throw new Error(`Unknown adapter "${adapter}"`); + const int = (raw: string | undefined, name: string, fallback: number, min: number, max: number) => { + if (raw === undefined) return fallback; + const value = Number(raw); + if (!Number.isSafeInteger(value) || value < min || value > max) throw new Error(`--${name} must be an integer between ${min} and ${max}`); + return value; + }; + return { keep: values.keep === true, options: { fault, adapters, tasks: int(values.tasks, "tasks", 2, 1, 6), + seed: int(values.seed, "seed", Date.now() % 100_000, 0, 2 ** 31 - 1), + // Two automatic retries back off about 30 s and 60 s (with jitter) after each failure. + restBoundMs: int(values["rest-seconds"], "rest-seconds", 420, 120, 1_800) * 1_000, + doneBoundMs: int(values["done-seconds"], "done-seconds", 900, 120, 3_600) * 1_000 } }; +} + +async function sample(api: ChaosApi, record: EpTaskRecord): Promise<{ issue: Json; interruption: Json | null } | null> { + try { + const [issue, runs, recovery] = await Promise.all([ + api.get(`/api/issues/${record.issueId}`, 5_000), + api.get(`/api/issues/${record.issueId}/runs`, 5_000), + // A failed read is recorded, never taken as "no actions": the grade needs this evidence. + api.get(`/api/issues/${record.issueId}/recovery-actions`, 5_000).catch(() => null), + ]); + if (recovery && Array.isArray(recovery.actions)) record.recoveryReads += 1; + else record.recoveryReadFailures += 1; + const known = new Map(record.runs.map(run => [run.id, run])); + for (const run of runs ?? []) { + const id = String(run.runId ?? run.id); + known.set(id, { id, status: String(run.status), errorCode: run.errorCode ?? null }); + } + record.runs = [...known.values()]; + for (const action of (recovery?.actions ?? []) as Json[]) { + if (!["active", "escalated"].includes(String(action.status))) continue; + const seen = { ownerType: String(action.ownerType), kind: String(action.kind), cause: action.cause ?? null }; + if (!record.activeRecoveryActions.some(prior => prior.ownerType === seen.ownerType && prior.kind === seen.kind && prior.cause === seen.cause)) { + record.activeRecoveryActions.push(seen); + } + } + return { issue, interruption: (issue.interruption ?? null) as Json | null }; + } catch { + return null; + } +} + +async function waitFor(api: ChaosApi, record: EpTaskRecord, boundMs: number, done: (state: { issue: Json; interruption: Json | null }) => boolean) { + const deadline = Date.now() + boundMs; + while (Date.now() < deadline) { + const state = await sample(api, record); + if (state) { + record.finalStatus = String(state.issue.status); + if (done(state)) return state; + } + await delay(POLL_MS); + } + return null; +} + +async function runWave(options: WaveOptions, waveDir: string, timeline: Timeline, keep: boolean): Promise> { + const fault = ENV_PROVIDER_FAULTS[options.fault]; + const supervisor = await ChaosSupervisor.create({ repositoryRoot, waveDir, timeline }); + const control = path.join(supervisor.paths.temporaryRoot, "lease-fault-control.json"); + await writeFile(control, leaseFaultControl(fault, `wave${options.seed}`), { mode: 0o600 }); + let stopFailed = false; + const records: EpTaskRecord[] = []; + try { + await supervisor.start({ PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL: control }); + const api = new ChaosApi(supervisor.baseUrl); + const lab: ChaosLab = await seedChaosLab({ api, env: "local", waitForReadyAgents: true, adapters: [...new Set(options.adapters)], + concurrency: options.tasks, budgetCents: 3_000, fixtureRoot: path.join(supervisor.paths.temporaryRoot, "fixtures"), + waveId: path.basename(waveDir), timeline }); + const a1 = workloadDefinitions.A1; + for (let index = 0; index < options.tasks; index += 1) { + const adapter = options.adapters[index % options.adapters.length]!; + const nonce = randomUUID().slice(0, 8); + const record: EpTaskRecord = { taskId: `t${String(index + 1).padStart(2, "0")}`, nonce, fault: options.fault, issueId: null, harnessError: null, + runs: [], activeRecoveryActions: [], recoveryReads: 0, recoveryReadFailures: 0, rested: null, retryClicked: false, finalStatus: null, briefOk: null }; + records.push(record); + try { + const issue = await api.post(`/api/companies/${lab.companyId}/issues`, { title: `Chaos ${options.fault} ${nonce}`, + description: a1.description(nonce), status: "todo", priority: "medium", assigneeAgentId: lab.agents[adapter].id, projectId: lab.projectId }); + record.issueId = String(issue.id); + timeline.add("ep_launched", { taskId: record.taskId, issueId: record.issueId, adapter }); + } catch (error) { + record.harnessError = `launch: ${error instanceof Error ? error.message : String(error)}`.slice(0, 600); + } + } + const live = records.filter(record => record.issueId); + if (fault.failures === null) { + // Never clears: each task must rest offering Retry within the bound. + await Promise.all(live.map(async record => { + const rested = await waitFor(api, record, options.restBoundMs, state => state.interruption?.canRetry === true + && ["blocked", "todo", "in_progress"].includes(String(state.issue.status)) + && !record.runs.some(run => ["queued", "running", "scheduled_retry"].includes(run.status))); + record.rested = rested ? { issueStatus: String(rested.issue.status), canRetry: rested.interruption?.canRetry === true, + observedRunId: rested.interruption?.observedRunId ?? null } : null; + timeline.add("ep_rested", { taskId: record.taskId, rested: record.rested }); + })); + await writeFile(control, "", { mode: 0o600 }); + timeline.add("ep_disarmed", {}); + for (const record of live) { + if (!record.rested?.observedRunId) continue; + const response = await api.post(`/api/issues/${record.issueId}/retry`, { idempotencyKey: randomUUID(), observedRunId: record.rested.observedRunId }); + record.retryClicked = response?.accepted === true; + timeline.add("ep_retry_clicked", { taskId: record.taskId, accepted: response?.accepted ?? null }); + } + } + await Promise.all(live.map(record => waitFor(api, record, options.doneBoundMs, state => state.issue.status === "done"))); + for (const record of live) { + const brief = await api.get(`/api/issues/${record.issueId}/documents/brief`).then(row => typeof row?.body === "string" ? row.body : null).catch(() => null); + record.briefOk = a1.grade(record.nonce, { issueStatus: record.finalStatus, documents: { brief }, agentComments: [], files: {}, commands: {}, workspacePath: null }).verdict === "pass"; + } + } finally { + try { await supervisor.stop(); } catch { stopFailed = true; } + await releaseInstance(supervisor.paths.temporaryRoot, { keep, stopFailed, owner: path.basename(waveDir) }) + .catch(error => timeline.add("instance_cleanup_failed", { error: String(error) })); + } + return records.map(record => ({ record, grade: gradeEnvProviderTask(record) })); +} + +export function envProviderSummaryMarkdown(waveId: string, gitSha: string, fault: EnvProviderFaultKind, rows: Array<{ record: EpTaskRecord; grade: EpGrade }>): string { + const lines = [`# Environment-provider fault wave ${waveId}`, "", `Commit ${gitSha}`, "", `Fault ${fault}: ${ENV_PROVIDER_FAULTS[fault].label}`, "", + "| Task | Runs (errorCode) | Final | Pass |", "| --- | --- | --- | --- |"]; + for (const { record, grade } of rows) { + lines.push(`| ${record.taskId} | ${record.runs.map(run => `${run.status}${run.errorCode ? `/${run.errorCode}` : ""}`).join(", ")} | ${record.finalStatus ?? "?"} | ${grade.pass ? "pass" : "FAIL"} |`); + } + lines.push("", "## Checks", ""); + for (const { record, grade } of rows) { + lines.push(`### ${record.taskId}`, "", ...grade.checks.map(check => `- ${check.pass ? "pass" : "FAIL"} ${check.name}: ${check.detail}`), ""); + } + return lines.join("\n"); +} + +async function main(): Promise { + const { options, keep } = parseEnvProviderWaveArgs(process.argv.slice(2)); + const missing = [...new Set(options.adapters.map(adapter => CREDENTIAL_FOR_ADAPTER[adapter]))].filter(key => !process.env[key]); + if (missing.length) throw new Error(`Missing credentials ${missing.join(", ")}. Run: set -a; source ~/.secrets; set +a`); + const runner = process.env.PAPERCLIP_RUNNER_BINARY ?? path.join(repositoryRoot, "packages/paperclip-runner/runner/target/debug/paperclip-runnerd"); + if (!existsSync(runner)) throw new Error(`Runner binary missing at ${runner}. Run: node scripts/agent-preflight.mjs`); + const waveId = `${waveIdFor(options.seed)}-ep`; + const waveDir = path.join(resultsRoot, waveId); + await mkdir(waveDir, { recursive: true, mode: 0o700 }); + const timeline = new Timeline(path.join(waveDir, "timeline.jsonl")); + const gitSha = spawnSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8" }).stdout.trim(); + timeline.add("ep_wave_planned", { waveId, gitSha, options }); + const rows = await runWave(options, waveDir, timeline, keep); + for (const row of rows) { + await writeFile(path.join(waveDir, `${row.record.taskId}.ep-grade.json`), + `${redactKnownSecrets(JSON.stringify(row, null, 2), knownSecretValues())}\n`, { mode: 0o600 }); + } + const summary = envProviderSummaryMarkdown(waveId, gitSha, options.fault, rows); + await writeFile(path.join(waveDir, "SUMMARY.md"), summary, { mode: 0o600 }); + console.info(`\n${summary}\nWave ${waveId}: results ${waveDir}`); + return rows.length === options.tasks && rows.every(row => row.grade.pass) ? 0 : 1; +} + +if (process.argv[1] && path.resolve(process.argv[1]) === path.resolve(import.meta.filename)) { + process.exit(await main().catch(error => { console.error(error); return 2; })); +}