mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
fix(recovery): wait out a sandbox the provider could not start, with no Retry (#22194)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that runs in a sandbox environment first gets a lease from the environment provider (for example Daytona) > - Sometimes the provider creates the sandbox but cannot start it. Daytona then puts the sandbox in its `error` state > - The Daytona plugin deletes that failed sandbox and rethrows the provider error. The error has no HTTP status and no errno, so the host cannot tell that it is a provider-side failure > - The run fails with `setup_failed`, and the task rests on "Retry when ready". No agent work had started, and a fresh sandbox works, so a person must click Retry for something the system can do itself > - Recovery must just work, and the target for runs that need Retry is 0% > - This pull request makes the plugin report a removed sandbox that the provider could not start as structured evidence (the step and the provider state). The host classifies it as a transient provider failure. The task then uses the existing bounded retries and the provider-outage wait from #22139 > - The benefit is that a provider start failure delays the task by seconds instead of stranding it on Retry ## Linked Issues or Issue Description Refs #22139 (the provider-outage wait lane that this failure now reaches). Refs #22128 (setup failure after work; this failure happens before any work, so that path did not apply). **What happened?** In a restart-chaos wave with graceful release swaps on 2026-11-18, two tasks started a run at 14:51:15Z and 14:51:24Z. Both runs failed at 14:51:53Z with `setup_failed` and `failureCause: lease_acquire_failed`. The cause was `Sandbox <id> failed to start with status: error, error reason: null` from the Daytona SDK, for two different sandboxes at the same moment. Both tasks rested on Retry. A Retry click started each task again, and both tasks completed. **Expected behavior** A sandbox that the provider cannot start is a provider failure before any agent work. Paperclip deletes it and tries again with a new sandbox, with no Retry. If the provider keeps failing, the task waits for the provider without a Retry, and a person gets a working Retry only after the 30-minute ceiling. **Steps to reproduce** 1. Make the Daytona create call fail while the new sandbox is in the provider's `error` state. 2. Assign a task to an agent that uses that environment. 3. The run fails with `setup_failed`, and the task rests on Retry. ## What Changed - Plugin SDK (`environment-creation-cleanup.ts`, `environment-acquire-error.ts`, `index.ts`): new `PluginEnvironmentAcquisitionError`. It carries the acquisition diagnostic (step, elapsed time, budget, provider state) for a failed allocation that the provider already removed, with no cleanup envelope. The acquire RPC error data (`paperclip/environment-acquire-error/v1`) now carries this diagnostic. `readEnvironmentAcquisitionDiagnostic` reads it from that envelope. The existing cleanup envelope path is unchanged. No provider text crosses the RPC. - Daytona plugin (`plugin.ts`): `destroyFailedCreation` returns the state of the sandbox that it deleted. When a create fails without an HTTP status and the deleted sandbox was in the `error` state, the plugin throws `PluginEnvironmentAcquisitionError` with `providerState: "error"`. A `build_failed` sandbox (a deterministic snapshot build failure) and any other state keep the original error. - Server (`environment-runtime.ts`): the acquire path records the diagnostic for this run also when the error has no cleanup envelope. A cleanup envelope for another run, company or environment still records nothing. - Server (`environment-provider-unavailable-error.ts`): an acquisition diagnostic with `providerState: "error"` is the transient reason `provider_start_failed` ("the provider could not start the sandbox"). It becomes `EnvironmentProviderUnavailableError`, so the run uses the two bounded retries and then the provider-outage wait lane (one probe per environment, 30-minute ceiling). - `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for the changed SDK sources (generated by `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --write`). Module boundaries: the change to the shared SDK contract is additive. The Daytona plugin owns the provider state. The host reads only the closed diagnostic fields. A plugin that does not use the new class behaves as before. ## Verification - `packages/plugins/sandbox-providers/daytona`: `npx vitest run src/plugin.test.ts -t "failed sandbox creation cleanup"` (37 passed). New: "reports a sandbox that failed to start as a removed allocation with its observed step" (red before the plugin change, green after) and "keeps the create error when the removed sandbox was build_failed / started". - `packages/plugins/sdk`: `npx vitest run tests/environment-acquire-error.test.ts tests/environment-creation-cleanup.test.ts` (50 passed). New: the diagnostic crosses the real worker RPC without provider text, and forged or unstructured envelopes give no diagnostic. - `server`: `npx vitest run src/services/environment-provider-unavailable-error.test.ts src/__tests__/environment-acquisition-diagnostics.test.ts` (74 passed). New: the reason is `provider_start_failed` with a transient provider failure, and a diagnostic that was not recorded for this run is not retried. - `server`: `npx vitest run src/__tests__/environment-runtime.test.ts -t "provider could not start|retains uncertain plugin creation"` (12 passed). New: "records the step of a sandbox the provider could not start and removed" for a worker error (red before the runtime change, green after) and for a forged error (not recorded). - `server`: `npx vitest run src/services/__tests__/run-failure-diagnostics.test.ts src/__tests__/heartbeat-process-recovery.test.ts -t "provider|acquisition|diagnostic"` (167 passed). - Typecheck: `server` `tsc --noEmit`, `pnpm run typecheck` in `packages/plugins/sdk` and in the Daytona plugin all pass. - `pnpm verify` exits 2 (the SDK input registry is a global CI input). Required CI supplies the broad coverage. ## Risks - **Automatic retries for a deterministic start failure.** If a provider image or configuration made every sandbox go to `error`, the task now waits in the provider-outage lane instead of resting on Retry at once. The lane uses one probe per environment on a 1, 2, then 5 minute backoff, and it hands the task to a person with a working Retry after 30 minutes. The aggregate automatic-run brake is unchanged. `build_failed` is not retried by this change. - **Shared SDK contract.** The new field is additive. An older host ignores it, and the run fails as before. - No database migration. No UI change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a9dc0df9da
commit
f0bd99ba8a
11 files changed
+182
-26
No files matched your search
@@ -39,7 +39,7 @@ import plugin, {
|
||||
STUCK_STOPPING_FORCE_AFTER_MS,
|
||||
} from "./plugin.js";
|
||||
import type { PluginContext } from "@paperclipai/plugin-sdk";
|
||||
import { environmentAcquireErrorData, environmentResumeErrorData, readEnvironmentAcquireErrorDiagnostic, environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncErrorCapture, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError, readEnvironmentAcquisitionDiagnostic, PluginEnvironmentStopError, environmentStopErrorData, readEnvironmentStopFailure } from "@paperclipai/plugin-sdk";
|
||||
import { PluginEnvironmentAcquisitionError, environmentAcquireErrorData, environmentResumeErrorData, readEnvironmentAcquireErrorDiagnostic, environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncErrorCapture, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError, readEnvironmentAcquisitionDiagnostic, PluginEnvironmentStopError, environmentStopErrorData, readEnvironmentStopFailure } from "@paperclipai/plugin-sdk";
|
||||
import manifest from "./manifest.js";
|
||||
import { parseTarVerboseListingLine, splitLinkEntryOnce } from "./file-sync.js";
|
||||
import * as duplexCommands from "./duplex-command-stream.js";
|
||||
@@ -1729,6 +1729,33 @@ describe("Daytona sandbox provider plugin", () => {
|
||||
expect(orphan!.process.executeCommand).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("reports a sandbox that failed to start as a removed allocation with its observed step", async () => {
|
||||
const startError = new Error("Sandbox private-sandbox-id failed to start with status: error, error reason: null");
|
||||
mockCreate.mockRejectedValue(startError);
|
||||
let orphan: ReturnType<typeof ownedSandbox>;
|
||||
mockGet.mockImplementation(async () => {
|
||||
orphan = { ...ownedSandbox(), state: "error" };
|
||||
orphan.delete.mockImplementation(async () => { orphan.state = "destroyed"; });
|
||||
return orphan;
|
||||
});
|
||||
const failure = await plugin.definition.onEnvironmentAcquireLease!(params).catch(error => error);
|
||||
expect(orphan!.delete).toHaveBeenCalledWith(10, true);
|
||||
expect(failure).toBeInstanceOf(PluginEnvironmentAcquisitionError);
|
||||
expect(failure.message).not.toContain("private-sandbox-id");
|
||||
expect(failure.cause).toBe(startError);
|
||||
expect(readEnvironmentCreationCleanupError(failure)).toBeNull();
|
||||
expect(readEnvironmentAcquisitionDiagnostic(failure)).toMatchObject({ phase: "create", providerState: "error" });
|
||||
expect(environmentAcquireErrorData(failure)).toMatchObject({
|
||||
schema: "paperclip/environment-acquire-error/v1", acquisitionDiagnostic: { phase: "create", providerState: "error" } });
|
||||
});
|
||||
|
||||
it.each(["build_failed", "started"])("keeps the create error when the removed sandbox was %s", async (state) => {
|
||||
const startError = new Error("Sandbox failed to start");
|
||||
mockCreate.mockRejectedValue(startError);
|
||||
mockGet.mockImplementation(async () => ({ ...ownedSandbox(), state }));
|
||||
await expect(plugin.definition.onEnvironmentAcquireLease!(params)).rejects.toBe(startError);
|
||||
});
|
||||
|
||||
it.each(["paperclip-company-id", "paperclip-environment-id", "paperclip-run-id", "paperclip-create-attempt", "paperclip-provider"])("never deletes a sandbox with a mismatched %s", async (label) => {
|
||||
let orphan: ReturnType<typeof ownedSandbox>;
|
||||
mockGet.mockImplementation(async () => {
|
||||
|
||||
@@ -9,7 +9,7 @@ import type {
|
||||
Resources,
|
||||
Sandbox,
|
||||
} from "@daytonaio/sdk";
|
||||
import { decodeChannelBytes, definePlugin, NOOP_PLUGIN_TRACER, PluginEnvironmentCreationCleanupError, PluginEnvironmentRunnerIngressError, PluginEnvironmentStopError, readEnvironmentCreationCleanupError, withEnvironmentSyncTransferStep, preserveEnvironmentSyncErrorDiagnostic } from "@paperclipai/plugin-sdk";
|
||||
import { decodeChannelBytes, definePlugin, NOOP_PLUGIN_TRACER, PluginEnvironmentAcquisitionError, PluginEnvironmentCreationCleanupError, PluginEnvironmentRunnerIngressError, PluginEnvironmentStopError, readEnvironmentCreationCleanupError, withEnvironmentSyncTransferStep, preserveEnvironmentSyncErrorDiagnostic } from "@paperclipai/plugin-sdk";
|
||||
import type {
|
||||
PluginContext,
|
||||
PluginEnvironmentCreationCleanup,
|
||||
@@ -1301,16 +1301,21 @@ async function createSandbox(
|
||||
if (options.claimCleanup && !options.claimCleanup()) throw createError;
|
||||
// Observe the create failure before compensating deletion can stall or fail.
|
||||
const diagnostic = options.onCreateFailure?.();
|
||||
let removedState: PluginEnvironmentAcquisitionProviderState | undefined;
|
||||
try {
|
||||
// A not-found lookup after an uncertain create is not a deletion receipt:
|
||||
// the provider may still materialize the request. Keep the name in the
|
||||
// error so cleanup can be reconciled rather than declaring success.
|
||||
await destroyFailedCreation(config, cleanup, params.driverKey);
|
||||
removedState = await destroyFailedCreation(config, cleanup, params.driverKey);
|
||||
} catch (cleanupError) {
|
||||
const status = creationHttpStatus(createError);
|
||||
throw new PluginEnvironmentCreationCleanupError([createError, cleanupError],
|
||||
`Daytona sandbox creation failed${status === undefined ? "" : ` (HTTP ${status})`}; cleanup could not be confirmed for ${name}`, cleanup, diagnostic);
|
||||
}
|
||||
if (removedState === "error" && diagnostic && creationHttpStatus(createError) === undefined) {
|
||||
throw new PluginEnvironmentAcquisitionError("Daytona could not start the sandbox; it was removed.",
|
||||
{ ...diagnostic, providerState: "error" }, createError);
|
||||
}
|
||||
throw createError;
|
||||
}
|
||||
}
|
||||
@@ -1333,7 +1338,7 @@ async function destroyFailedCreation(
|
||||
config: DaytonaDriverConfig, cleanup: PluginEnvironmentCreationCleanup,
|
||||
driverKey: string,
|
||||
requireDurableObservation = false,
|
||||
): Promise<void> {
|
||||
): Promise<PluginEnvironmentAcquisitionProviderState | undefined> {
|
||||
if (cleanup.providerLeaseId !== `paperclip-create-${cleanup.attemptId}` ||
|
||||
cleanup.accountFingerprint !== sandboxAccountDiscriminator(config) ||
|
||||
cleanup.labels["paperclip-provider"] !== "daytona" ||
|
||||
@@ -1360,7 +1365,7 @@ async function destroyFailedCreation(
|
||||
// observed with exact ownership cannot materialize as a new allocation.
|
||||
if (cleanup.observedProviderLeaseId && error instanceof DaytonaNotFoundError) {
|
||||
idleActivity.confirm(observationProof, true);
|
||||
return;
|
||||
return undefined;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
@@ -1377,8 +1382,11 @@ async function destroyFailedCreation(
|
||||
}
|
||||
idleActivity.observe(allocation, providerActivityKey(identity, config, orphan.id));
|
||||
const deletionProof = idleActivity.proof(providerActivityKey(identity, config, orphan.id));
|
||||
const observedState = typeof orphan.state === "string" && REPORTED_CREATE_STATES.has(orphan.state)
|
||||
? orphan.state as PluginEnvironmentAcquisitionProviderState : undefined;
|
||||
await withLivenessTimeout("sandbox.failedCreateDelete", 15_000, () => orphan.delete(10, true));
|
||||
idleActivity.confirm(deletionProof);
|
||||
return observedState;
|
||||
}
|
||||
|
||||
// Daytona states reported as-is after a create deadline; any other is `other`.
|
||||
|
||||
@@ -1,4 +1,11 @@
|
||||
import { PluginEnvironmentCreationCleanupError, environmentCreationCleanupErrorData } from "./environment-creation-cleanup.js";
|
||||
import {
|
||||
ENVIRONMENT_ACQUIRE_ERROR_SCHEMA,
|
||||
ENVIRONMENT_CREATION_CLEANUP_SCHEMA,
|
||||
PluginEnvironmentAcquisitionError,
|
||||
PluginEnvironmentCreationCleanupError,
|
||||
environmentCreationCleanupErrorData,
|
||||
readEnvironmentAcquisitionDiagnostic,
|
||||
} from "./environment-creation-cleanup.js";
|
||||
import {
|
||||
providerErrorDiagnostic,
|
||||
sanitizeProviderErrorDiagnostic,
|
||||
@@ -15,8 +22,8 @@ import {
|
||||
*/
|
||||
export type PluginEnvironmentAcquireErrorDiagnostic = PluginEnvironmentResumeErrorDiagnostic;
|
||||
|
||||
const SCHEMA = "paperclip/environment-acquire-error/v1";
|
||||
const CLEANUP_SCHEMA = "paperclip/environment-creation-cleanup/v1";
|
||||
const SCHEMA = ENVIRONMENT_ACQUIRE_ERROR_SCHEMA;
|
||||
const CLEANUP_SCHEMA = ENVIRONMENT_CREATION_CLEANUP_SCHEMA;
|
||||
|
||||
function field(value: unknown, key: string): unknown {
|
||||
try { return value && typeof value === "object" ? (value as Record<string, unknown>)[key] : undefined; }
|
||||
@@ -35,7 +42,14 @@ export function environmentAcquireErrorData(error: unknown): unknown {
|
||||
if (cleanup && typeof cleanup === "object") {
|
||||
return providerDiagnostic ? { ...cleanup, providerDiagnostic } : cleanup;
|
||||
}
|
||||
return providerDiagnostic ? { schema: SCHEMA, providerDiagnostic } : undefined;
|
||||
const acquisitionDiagnostic = error instanceof PluginEnvironmentAcquisitionError
|
||||
? readEnvironmentAcquisitionDiagnostic(error) : null;
|
||||
if (!providerDiagnostic && !acquisitionDiagnostic) return undefined;
|
||||
return {
|
||||
schema: SCHEMA,
|
||||
...(providerDiagnostic ? { providerDiagnostic } : {}),
|
||||
...(acquisitionDiagnostic ? { acquisitionDiagnostic } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
/** Host side: read and re-validate the provider diagnostic from an RPC error. */
|
||||
|
||||
@@ -11,7 +11,8 @@ export interface PluginEnvironmentCreationCleanup {
|
||||
labels: Record<string, string>;
|
||||
}
|
||||
|
||||
const SCHEMA = "paperclip/environment-creation-cleanup/v1";
|
||||
export const ENVIRONMENT_CREATION_CLEANUP_SCHEMA = "paperclip/environment-creation-cleanup/v1";
|
||||
export const ENVIRONMENT_ACQUIRE_ERROR_SCHEMA = "paperclip/environment-acquire-error/v1";
|
||||
|
||||
/** The step observed at the first acquisition failure, not its upstream cause. */
|
||||
export interface PluginEnvironmentAcquisitionDiagnostic {
|
||||
@@ -32,12 +33,19 @@ export const ENVIRONMENT_ACQUISITION_PROVIDER_STATES = [
|
||||
] as const;
|
||||
export type PluginEnvironmentAcquisitionProviderState = typeof ENVIRONMENT_ACQUISITION_PROVIDER_STATES[number];
|
||||
|
||||
function acquisitionDiagnosticContainer(error: unknown): object | null {
|
||||
if (error instanceof PluginEnvironmentAcquisitionError) return error;
|
||||
if (readEnvironmentCreationCleanupError(error)) {
|
||||
return error instanceof PluginEnvironmentCreationCleanupError ? error : (error as { data?: object }).data ?? null;
|
||||
}
|
||||
const data = error && typeof error === "object" ? Object.getOwnPropertyDescriptor(error, "data")?.value : undefined;
|
||||
return data && typeof data === "object" && Object.getOwnPropertyDescriptor(data, "schema")?.value === ENVIRONMENT_ACQUIRE_ERROR_SCHEMA
|
||||
? data : null;
|
||||
}
|
||||
|
||||
export function readEnvironmentAcquisitionDiagnostic(error: unknown): PluginEnvironmentAcquisitionDiagnostic | null {
|
||||
try {
|
||||
// Diagnostics never make an invalid ownership envelope usable for cleanup.
|
||||
if (!readEnvironmentCreationCleanupError(error)) return null;
|
||||
const container = error instanceof PluginEnvironmentCreationCleanupError
|
||||
? error : (error as { data?: object }).data;
|
||||
const container = acquisitionDiagnosticContainer(error);
|
||||
const value = container && Object.getOwnPropertyDescriptor(container, "acquisitionDiagnostic")?.value;
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
|
||||
const fields = Object.getOwnPropertyDescriptors(value);
|
||||
@@ -72,18 +80,28 @@ export class PluginEnvironmentCreationCleanupError extends AggregateError {
|
||||
}
|
||||
}
|
||||
|
||||
export class PluginEnvironmentAcquisitionError extends Error {
|
||||
readonly acquisitionDiagnostic: PluginEnvironmentAcquisitionDiagnostic;
|
||||
|
||||
constructor(message: string, acquisitionDiagnostic: PluginEnvironmentAcquisitionDiagnostic, cause?: unknown) {
|
||||
super(message, { cause });
|
||||
this.name = "PluginEnvironmentAcquisitionError";
|
||||
this.acquisitionDiagnostic = acquisitionDiagnostic;
|
||||
}
|
||||
}
|
||||
|
||||
export function environmentCreationCleanupErrorData(error: unknown, includeAcquisitionDiagnostic = false): unknown {
|
||||
if (!(error instanceof PluginEnvironmentCreationCleanupError)) return undefined;
|
||||
const cleanup = readEnvironmentCreationCleanupError(error);
|
||||
const acquisitionDiagnostic = includeAcquisitionDiagnostic ? readEnvironmentAcquisitionDiagnostic(error) : null;
|
||||
return cleanup ? { schema: SCHEMA, cleanup, ...(acquisitionDiagnostic ? { acquisitionDiagnostic } : {}) } : undefined;
|
||||
return cleanup ? { schema: ENVIRONMENT_CREATION_CLEANUP_SCHEMA, cleanup, ...(acquisitionDiagnostic ? { acquisitionDiagnostic } : {}) } : undefined;
|
||||
}
|
||||
|
||||
export function readEnvironmentCreationCleanupError(error: unknown): PluginEnvironmentCreationCleanup | null {
|
||||
const data = error instanceof PluginEnvironmentCreationCleanupError
|
||||
? { schema: SCHEMA, cleanup: error.cleanup }
|
||||
? { schema: ENVIRONMENT_CREATION_CLEANUP_SCHEMA, cleanup: error.cleanup }
|
||||
: error && typeof error === "object" && "data" in error ? error.data : null;
|
||||
if (!data || typeof data !== "object" || !("schema" in data) || data.schema !== SCHEMA || !("cleanup" in data)) return null;
|
||||
if (!data || typeof data !== "object" || !("schema" in data) || data.schema !== ENVIRONMENT_CREATION_CLEANUP_SCHEMA || !("cleanup" in data)) return null;
|
||||
const value = data.cleanup;
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
|
||||
const row = value as Record<string, unknown>;
|
||||
|
||||
@@ -476,7 +476,7 @@ export {
|
||||
PRINCIPAL_TYPES,
|
||||
} from "@paperclipai/shared";
|
||||
|
||||
export { PluginEnvironmentCreationCleanupError, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError, readEnvironmentAcquisitionDiagnostic, ENVIRONMENT_ACQUISITION_PROVIDER_STATES } from "./environment-creation-cleanup.js";
|
||||
export { PluginEnvironmentAcquisitionError, PluginEnvironmentCreationCleanupError, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError, readEnvironmentAcquisitionDiagnostic, ENVIRONMENT_ACQUISITION_PROVIDER_STATES } from "./environment-creation-cleanup.js";
|
||||
export type { PluginEnvironmentCreationCleanup, PluginEnvironmentAcquisitionDiagnostic, PluginEnvironmentAcquisitionProviderState } from "./environment-creation-cleanup.js";
|
||||
export { preserveEnvironmentSyncErrorDiagnostic, environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncErrorCapture, withEnvironmentSyncTransferStep, recordEnvironmentSyncError } from "./environment-sync-error.js";
|
||||
export type { PluginEnvironmentSyncErrorDiagnostic } from "./environment-sync-error.js";
|
||||
|
||||
@@ -3,7 +3,12 @@ import { PassThrough } from "node:stream";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { definePlugin } from "../src/define-plugin.js";
|
||||
import { readEnvironmentAcquireErrorDiagnostic } from "../src/environment-acquire-error.js";
|
||||
import { PluginEnvironmentCreationCleanupError, readEnvironmentCreationCleanupError } from "../src/environment-creation-cleanup.js";
|
||||
import {
|
||||
PluginEnvironmentAcquisitionError,
|
||||
PluginEnvironmentCreationCleanupError,
|
||||
readEnvironmentAcquisitionDiagnostic,
|
||||
readEnvironmentCreationCleanupError,
|
||||
} from "../src/environment-creation-cleanup.js";
|
||||
import { createRequest, isJsonRpcResponse, JsonRpcCallError, parseMessage, serializeMessage, type JsonRpcResponse } from "../src/protocol.js";
|
||||
import { startWorkerRpcHost } from "../src/worker-rpc-host.js";
|
||||
|
||||
@@ -62,6 +67,25 @@ describe("environment acquisition refusal diagnostics", () => {
|
||||
expect(readEnvironmentAcquireErrorDiagnostic(plain)).toBeNull();
|
||||
});
|
||||
|
||||
it("carries the observed step of a removed failed allocation without provider text", async () => {
|
||||
const callError = await acquire(new PluginEnvironmentAcquisitionError("Daytona could not start the sandbox.",
|
||||
{ phase: "create", elapsedMs: 38_000, budgetMs: 90_000, providerState: "error" },
|
||||
new Error("Sandbox private-sandbox-id failed to start with status: error")));
|
||||
expect(readEnvironmentAcquisitionDiagnostic(callError)).toEqual({ phase: "create", elapsedMs: 38_000, budgetMs: 90_000, providerState: "error" });
|
||||
expect(readEnvironmentCreationCleanupError(callError)).toBeNull();
|
||||
expect(readEnvironmentAcquireErrorDiagnostic(callError)).toBeNull();
|
||||
expect(JSON.stringify(callError.data)).not.toContain("private-sandbox-id");
|
||||
});
|
||||
|
||||
it("reads no acquisition step from an unstructured or forged refusal", async () => {
|
||||
expect(readEnvironmentAcquisitionDiagnostic(await acquire(new Error("plain")))).toBeNull();
|
||||
const forged = (data: unknown) => Object.assign(new Error("x"), { data });
|
||||
expect(readEnvironmentAcquisitionDiagnostic(forged({ schema: "other",
|
||||
acquisitionDiagnostic: { phase: "create", elapsedMs: 1, budgetMs: 2 } }))).toBeNull();
|
||||
expect(readEnvironmentAcquisitionDiagnostic(forged({ schema: "paperclip/environment-acquire-error/v1",
|
||||
acquisitionDiagnostic: { phase: "anything", elapsedMs: 1, budgetMs: 2 } }))).toBeNull();
|
||||
});
|
||||
|
||||
it("rejects forged or out-of-range envelopes", () => {
|
||||
const forged = (data: unknown) => Object.assign(new Error("x"), { data });
|
||||
expect(readEnvironmentAcquireErrorDiagnostic(forged({ schema: "other", providerDiagnostic: { httpStatus: 502 } }))).toBeNull();
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { JsonRpcCallError, environmentResumeErrorData } from "@paperclipai/plugin-sdk";
|
||||
import { collectRunFailureDiagnostics } from "../services/run-failure-diagnostics.js";
|
||||
import { readTransientLeaseAcquisitionFailure } from "../services/environment-provider-unavailable-error.js";
|
||||
import { PluginWorkerUnavailableError } from "../services/plugin-worker-unavailable-error.js";
|
||||
import { createSandboxOrphanCleanupSpool } from "../services/sandbox-orphan-cleanup-spool.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
@@ -1496,6 +1497,37 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
.toEqual([expect.objectContaining({ state: "pending", originalEnvironmentId: seeded.environment.id, runId: seeded.runId })]);
|
||||
});
|
||||
|
||||
it.each(["worker", "forged error", "invalid worker diagnostic"])("records the step of a sandbox the provider could not start and removed: %s", async (mode) => {
|
||||
const seeded = await seedReusablePluginSandboxLease();
|
||||
await environmentService(db).releaseLease(seeded.reusableLease.id, "expired");
|
||||
const data = { schema: "paperclip/environment-acquire-error/v1",
|
||||
acquisitionDiagnostic: { phase: mode === "invalid worker diagnostic" ? "private-command" : "create",
|
||||
elapsedMs: 38_000, budgetMs: 90_000, providerState: "error" } };
|
||||
const failure = mode === "forged error"
|
||||
? Object.assign(new Error("Daytona could not start the sandbox; it was removed."), { data })
|
||||
: new JsonRpcCallError({ code: -32002, message: "Daytona could not start the sandbox; it was removed.", data });
|
||||
const call = vi.fn(async (_id: string, method: string) => {
|
||||
if (method === "environmentAcquireLease") throw failure;
|
||||
throw new Error(`Unexpected method ${method}`);
|
||||
});
|
||||
const worker = { isRunning: () => true, getWorker: () => ({ supportedMethods: ["environmentDestroyLease"] }), call } as unknown as PluginWorkerManager;
|
||||
const caught = await environmentRuntimeService(db, { pluginWorkerManager: worker }).acquireRunLease({
|
||||
companyId: seeded.companyId, environment: seeded.environment, issueId: null, heartbeatRunId: seeded.runId,
|
||||
persistedExecutionWorkspace: null }).catch(error => error);
|
||||
expect(caught).toBe(failure);
|
||||
const run = { id: seeded.runId, companyId: seeded.companyId };
|
||||
const execution = collectRunFailureDiagnostics({ ...run, errorCode: "setup_failed", resultJson: null,
|
||||
} as typeof heartbeatRuns.$inferSelect, { error: caught, phase: "setup" }).execution;
|
||||
if (mode === "worker") {
|
||||
expect(execution).toMatchObject({ environmentAcquisitionPhase: "create", environmentAcquisitionProviderState: "error" });
|
||||
expect(readTransientLeaseAcquisitionFailure(caught, run)).toEqual({ reason: "provider_start_failed" });
|
||||
} else {
|
||||
expect(execution).not.toHaveProperty("environmentAcquisitionPhase");
|
||||
expect(readTransientLeaseAcquisitionFailure(caught, run)).toBeNull();
|
||||
}
|
||||
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.status, "pending_cleanup"))).toHaveLength(0);
|
||||
});
|
||||
|
||||
it.each(["deleted receipt lost", "observation response lost", "journal write failed", "foreign observation"])(
|
||||
"journals observed creation identity before deletion and recovers: %s", async (fault) => {
|
||||
const seeded = await seedReusablePluginSandboxLease();
|
||||
|
||||
@@ -154,6 +154,36 @@ describe("readTransientLeaseAcquisitionFailure", () => {
|
||||
expect(readTransientLeaseAcquisitionFailure(creationFailure(12_000), run)).toEqual({ reason: "acquisition_incomplete" });
|
||||
});
|
||||
|
||||
it("retries a sandbox the provider could not start and already removed", () => {
|
||||
const error = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR,
|
||||
message: "Daytona could not start the sandbox; it was removed.",
|
||||
data: { schema: "paperclip/environment-acquire-error/v1",
|
||||
acquisitionDiagnostic: { phase: "create", elapsedMs: 38_000, budgetMs: 90_000, providerState: "error" } } });
|
||||
captureEnvironmentAcquisitionDiagnostic(error, { companyId: run.companyId, environmentId: "env-1", runId: run.id },
|
||||
readEnvironmentAcquisitionDiagnostic(error));
|
||||
expect(readTransientLeaseAcquisitionFailure(new Error("lease", { cause: error }), run)).toEqual({ reason: "provider_start_failed" });
|
||||
const unavailable = new EnvironmentProviderUnavailableError({ environmentName: "Daytona warm", driver: "sandbox",
|
||||
evidence: { reason: "provider_start_failed" }, cause: error });
|
||||
expect(unavailable.providerFailure).toMatchObject({ retryability: "transient", providerCode: "provider_start_failed" });
|
||||
expect(unavailable.message).toContain("the provider could not start the sandbox");
|
||||
});
|
||||
|
||||
it("names a later-step failure an unfinished acquisition, not a start failure", () => {
|
||||
const error = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR, message: "x",
|
||||
data: { schema: "paperclip/environment-acquire-error/v1",
|
||||
acquisitionDiagnostic: { phase: "shell", elapsedMs: 38_000, budgetMs: 90_000, providerState: "error" } } });
|
||||
captureEnvironmentAcquisitionDiagnostic(error, { companyId: run.companyId, environmentId: "env-1", runId: run.id },
|
||||
readEnvironmentAcquisitionDiagnostic(error));
|
||||
expect(readTransientLeaseAcquisitionFailure(error, run)).toEqual({ reason: "acquisition_incomplete" });
|
||||
});
|
||||
|
||||
it("does not retry a removed sandbox whose step was never recorded for this run", () => {
|
||||
const error = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR, message: "x",
|
||||
data: { schema: "paperclip/environment-acquire-error/v1",
|
||||
acquisitionDiagnostic: { phase: "create", elapsedMs: 38_000, budgetMs: 90_000, providerState: "error" } } });
|
||||
expect(readTransientLeaseAcquisitionFailure(error, run)).toBeNull();
|
||||
});
|
||||
|
||||
it("ignores an unfinished-creation diagnostic recorded for another run", () => {
|
||||
expect(readTransientLeaseAcquisitionFailure(creationFailure(12_000, "run-2"), run)).toBeNull();
|
||||
});
|
||||
|
||||
@@ -19,7 +19,7 @@ export const ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE = "environment_provider_unava
|
||||
* the provider reported a provisional creation that did not finish, or a
|
||||
* control-plane database query failed while the lease was taken.
|
||||
*/
|
||||
type TransientLeaseFailureReason = "acquisition_incomplete" | "database_error";
|
||||
type TransientLeaseFailureReason = "acquisition_incomplete" | "provider_start_failed" | "database_error";
|
||||
type TransientProviderEvidence = { httpStatus?: number; errorCode?: string; acquisitionTimedOut?: true; reason?: TransientLeaseFailureReason };
|
||||
|
||||
function own(value: unknown, key: string): unknown {
|
||||
@@ -104,7 +104,8 @@ export function readTransientLeaseAcquisitionFailure(
|
||||
for (let cause = error, depth = 0; cause && depth < 5; cause = own(cause, "cause"), depth++) {
|
||||
const acquisition = getEnvironmentAcquisitionDiagnostic(cause, run);
|
||||
if (acquisition) {
|
||||
return acquisition.elapsedMs >= acquisition.budgetMs ? { acquisitionTimedOut: true } : { reason: "acquisition_incomplete" };
|
||||
if (acquisition.elapsedMs >= acquisition.budgetMs) return { acquisitionTimedOut: true };
|
||||
return { reason: acquisition.phase === "create" && acquisition.providerState === "error" ? "provider_start_failed" : "acquisition_incomplete" };
|
||||
}
|
||||
if (cause instanceof JsonRpcCallError && cause.code === PLUGIN_RPC_ERROR_CODES.TIMEOUT) return { acquisitionTimedOut: true };
|
||||
}
|
||||
@@ -113,6 +114,7 @@ export function readTransientLeaseAcquisitionFailure(
|
||||
|
||||
const REASON_LABELS: Record<TransientLeaseFailureReason, string> = {
|
||||
acquisition_incomplete: "sandbox creation did not finish",
|
||||
provider_start_failed: "the provider could not start the sandbox",
|
||||
database_error: "a database query failed while recording the lease",
|
||||
};
|
||||
|
||||
|
||||
@@ -2047,8 +2047,9 @@ function createSandboxEnvironmentDriver(
|
||||
resolvePluginSandboxRpcTimeoutMs(workerConfig, pluginProvider.resolved.driver.defaultAcquireTimeoutMs));
|
||||
} catch (error) {
|
||||
const cleanup = readEnvironmentCreationCleanupError(error);
|
||||
const diagnostic = error instanceof JsonRpcCallError && cleanup?.companyId === input.companyId &&
|
||||
cleanup.environmentId === input.environment.id && cleanup.runId === identity.runId
|
||||
const ownsReceipt = cleanup === null || (cleanup.companyId === input.companyId &&
|
||||
cleanup.environmentId === input.environment.id && cleanup.runId === identity.runId);
|
||||
const diagnostic = error instanceof JsonRpcCallError && ownsReceipt
|
||||
? readEnvironmentAcquisitionDiagnostic(error) : null;
|
||||
captureEnvironmentAcquisitionDiagnostic(error, {
|
||||
companyId: input.companyId, environmentId: input.environment.id, runId: identity.runId,
|
||||
|
||||
@@ -6,8 +6,8 @@
|
||||
"packages/plugins/sdk/src/define-plugin.ts": "a99e07ce77ef72fcf1df1347346aacb8ba12e284",
|
||||
"packages/plugins/sdk/src/dev-cli.ts": "7c1b0e2b6e4b7fd82027cca5321066e473a03d20",
|
||||
"packages/plugins/sdk/src/dev-server.ts": "2eadff81b74ca0706aec53aaf8637797fe4b4093",
|
||||
"packages/plugins/sdk/src/environment-acquire-error.ts": "50a74c3d1f83150f4215d44a763aa8ce4d9469c2",
|
||||
"packages/plugins/sdk/src/environment-creation-cleanup.ts": "35f23cd19ab0dd4148731846419a1c9b9942786b",
|
||||
"packages/plugins/sdk/src/environment-acquire-error.ts": "4686b19d7918ae40e909028154a3614da4d17f35",
|
||||
"packages/plugins/sdk/src/environment-creation-cleanup.ts": "419ead5f876d752f156e3852653b6da548a559a8",
|
||||
"packages/plugins/sdk/src/environment-resume-error.ts": "51ddfcd00e4e8c8a8b6bb39152655926b10aacad",
|
||||
"packages/plugins/sdk/src/environment-runner-ingress-error.ts": "fdb47797c2a1d2df82390e42b9593d4fa4cf49e7",
|
||||
"packages/plugins/sdk/src/environment-stop-error.ts": "b0e03f8aa2b2170ba095da5c7138050fced3f3cb",
|
||||
@@ -17,7 +17,7 @@
|
||||
"packages/plugins/sdk/src/host-client-factory.ts": "e0301c390ea13f2340b669383f07af2455e48c8a",
|
||||
"packages/plugins/sdk/src/idle-drain.test.ts": "cc59f7881e7d46c89395a1d93d74f6c6b9b0a64a",
|
||||
"packages/plugins/sdk/src/idle-drain.ts": "ae6e4c6be4657a3d541396922b3bc551b8db557f",
|
||||
"packages/plugins/sdk/src/index.ts": "7a10d0e833956d6265250eedbb768c7971c36d09",
|
||||
"packages/plugins/sdk/src/index.ts": "de69769668c7d555d52576548d7fc1a35617627e",
|
||||
"packages/plugins/sdk/src/invocation-tracer.ts": "b28ac4aa4bc3e82658abac4a8081ed4508008696",
|
||||
"packages/plugins/sdk/src/protocol.duplex-channel.test.ts": "18255eb3d63038dc459240c111cbcb29fcba25cc",
|
||||
"packages/plugins/sdk/src/protocol.ts": "4861d78f1c57f1485a79e037932f39e8c91446b8",
|
||||
|
||||
Reference in new issue
Block a user