Files
PaperClipAI/packages/plugins/sdk/src/environment-creation-cleanup.ts
T
657e71584a fix(daytona): record provider state when lease creation hits its deadline (#15946)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents can run in Daytona sandboxes. The Daytona plugin acquires a
lease before each run.
> - Some acquisitions time out. The report then says
`environmentAcquisitionPhase=create` and that the whole 300 s budget was
used.
> - The Daytona SDK create timeout is equal to the acquisition budget.
The outer timer always fires first. Thus the report never shows the
state that Daytona left the sandbox in.
> - Without that state, we cannot choose a fix. A stuck snapshot pull, a
slow start, and a missing allocation need different fixes.
> - This pull request adds one bounded, read-only provider lookup at the
create deadline. It records a fixed-enum provider state in the existing
acquisition diagnostic.
> - The benefit is that the next timeout identifies where the provider
stopped. Retry, budget, and cleanup behavior do not change.

## Linked Issues or Issue Description

Refs #15736 (bounded Daytona acquisition failure observations). This
pull request extends those observations.

**What happened?**

A Daytona lease acquisition timed out with `Daytona lease acquisition
timed out; allocation cleanup is pending`. The run-failure report gave
`environmentAcquisitionPhase=create`,
`environmentAcquisitionElapsedMs=300004`, and
`environmentAcquisitionBudgetMs=300000`. The report did not show the
Daytona sandbox state. The SDK create timeout is equal to the outer
budget, so the outer timer always fires first. The SDK never reports the
stuck state.

**Expected behavior**

When creation is still pending at the deadline, the report should show
where the provider left the allocation. Examples are `pulling_snapshot`,
`starting`, `error`, or `absent`.

**Steps to reproduce**

1. Configure a Daytona environment.
2. Start a run when Daytona does not finish create and start within the
acquisition budget.
3. Inspect the setup-failure report. Only the phase, the elapsed time,
and the budget are present.

**Paperclip version or commit**

`master` at ba04968.

## What Changed

- `packages/plugins/sdk`: Add an optional `providerState` field to
`PluginEnvironmentAcquisitionDiagnostic`. The values are a fixed enum:
`absent`, `creating`, `pending_build`, `building_snapshot`,
`pulling_snapshot`, `starting`, `started`, `error`, `build_failed`,
`other`, `unavailable`. The reader omits an unknown value and keeps the
other fields. Old envelopes stay valid.
- `packages/plugins/sandbox-providers/daytona`: When the acquisition
deadline fires while `client.create` is still pending, make one
`client.get` of the provisional `paperclip-create-<attempt>` name. The
lookup has a 5 s bound, inside the host's 30 s RPC margin. Report the
state only when the name and all ownership labels match. Otherwise
report `unavailable`. A not-found error reports `absent`.
- The lookup does not delete. It does not set `observedProviderLeaseId`.
It does not change retries, budgets, or the cleanup envelope.
- After the deadline, only the timer settles the call. A late create
result cannot replace the timeout error before the lookup completes. A
test proves this guard.
- `server`: Project the value to
`run_execution.environmentAcquisitionProviderState`. It uses the same
private, run-scoped receipt as the other acquisition fields.
- `doc/observability.md`: Document the new field and its limits.

## Verification

- `cd packages/plugins/sandbox-providers/daytona && npx vitest run
--config vitest.config.ts`: 392 passed, 6 skipped.
- `npx vitest run
packages/plugins/sdk/tests/environment-creation-cleanup.test.ts
server/src/services/__tests__/run-failure-diagnostics.test.ts
server/src/__tests__/environment-acquisition-diagnostics.test.ts
server/src/__tests__/run-failure-sentry-real-sdk.test.ts
server/src/__tests__/environment-runtime.test.ts`: 302 passed, 11
skipped.
- `tsc --noEmit` passes for `packages/plugins/sdk` and
`packages/plugins/sandbox-providers/daytona`.
- New tests cover state mapping, `other` for unlisted states, `absent`,
`unavailable` for lookup failure and label mismatch, a hung lookup
bounded to 5 s, no deletion, an unchanged cleanup envelope, and a late
create that settles during the lookup.
- A mutation check removed the late-settlement guard. The late-create
test then failed.
- The local server `tsc` was killed by the sandbox memory limit. CI must
confirm the server typecheck. The server change is one line.

## Risks

- Low risk. The timeout rejection can come up to 5 s later, and only for
a create-phase deadline. The host RPC margin is 30 s.
- The lookup is one extra read-only Daytona API call for each
create-phase timeout.
- This is a diagnostic. It does not fix the upstream cause. The next
timeout report must show the state before a fix is chosen.
- This is a Sentry run-failure report change. It is not a Telemetry
change. The new value is a fixed enum. It does not contain provider IDs,
causes, URLs, or paths.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edit, test runs). Context window and
reasoning configuration are set by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Devin Foley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:22:39 -07:00

109 lines
5.9 KiB
TypeScript

/** Non-secret ownership evidence for an allocation whose creation failed. */
export interface PluginEnvironmentCreationCleanup {
providerLeaseId: string;
/** Provider ID observed with matching ownership, distinct from a provisional creation name. */
observedProviderLeaseId?: string;
companyId: string;
environmentId: string;
runId?: string;
attemptId: string;
accountFingerprint: string;
labels: Record<string, string>;
}
const SCHEMA = "paperclip/environment-creation-cleanup/v1";
/** The step observed at the first acquisition failure, not its upstream cause. */
export interface PluginEnvironmentAcquisitionDiagnostic {
phase: "create" | "workspace" | "shell" | "expiry" | "sentinel";
/** Milliseconds from acquisition start to the first failure, before cleanup. */
elapsedMs: number;
budgetMs: number;
/**
* Provider state of the provisional allocation, observed once after a create
* deadline. Only a fixed set of states is reported; it is not the cause.
*/
providerState?: PluginEnvironmentAcquisitionProviderState;
}
export const ENVIRONMENT_ACQUISITION_PROVIDER_STATES = [
"absent", "creating", "pending_build", "building_snapshot", "pulling_snapshot",
"starting", "started", "error", "build_failed", "other", "unavailable",
] as const;
export type PluginEnvironmentAcquisitionProviderState = typeof ENVIRONMENT_ACQUISITION_PROVIDER_STATES[number];
export function readEnvironmentAcquisitionDiagnostic(error: unknown): PluginEnvironmentAcquisitionDiagnostic | null {
try {
// Diagnostics never make an invalid ownership envelope usable for cleanup.
if (!readEnvironmentCreationCleanupError(error)) return null;
const container = error instanceof PluginEnvironmentCreationCleanupError
? error : (error as { data?: object }).data;
const value = container && Object.getOwnPropertyDescriptor(container, "acquisitionDiagnostic")?.value;
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
const fields = Object.getOwnPropertyDescriptors(value);
const phase = fields.phase?.value;
const elapsedMs = fields.elapsedMs?.value;
const budgetMs = fields.budgetMs?.value;
const providerState = fields.providerState?.value;
if (!["create", "workspace", "shell", "expiry", "sentinel"].includes(phase as string) ||
typeof elapsedMs !== "number" || !Number.isSafeInteger(elapsedMs) || elapsedMs < 0 || elapsedMs > 604_800_000 ||
typeof budgetMs !== "number" || !Number.isSafeInteger(budgetMs) || budgetMs < 1 || budgetMs > 86_400_000) return null;
// An unknown optional state is omitted; it never invalidates the observation.
const knownState = ENVIRONMENT_ACQUISITION_PROVIDER_STATES.find(value => value === providerState);
return { phase: phase as PluginEnvironmentAcquisitionDiagnostic["phase"], elapsedMs, budgetMs,
...(knownState ? { providerState: knownState } : {}) };
} catch { return null; }
}
/**
* Never serialize the provider exceptions: they may contain credentials.
* The host persists only this explicit ownership envelope, then retries teardown.
*/
export class PluginEnvironmentCreationCleanupError extends AggregateError {
readonly cleanup: PluginEnvironmentCreationCleanup;
readonly acquisitionDiagnostic?: PluginEnvironmentAcquisitionDiagnostic;
constructor(errors: unknown[], message: string, cleanup: PluginEnvironmentCreationCleanup,
acquisitionDiagnostic?: PluginEnvironmentAcquisitionDiagnostic) {
super(errors, message);
this.name = "PluginEnvironmentCreationCleanupError";
this.cleanup = cleanup;
this.acquisitionDiagnostic = acquisitionDiagnostic;
}
}
export function environmentCreationCleanupErrorData(error: unknown, includeAcquisitionDiagnostic = false): unknown {
if (!(error instanceof PluginEnvironmentCreationCleanupError)) return undefined;
const cleanup = readEnvironmentCreationCleanupError(error);
const acquisitionDiagnostic = includeAcquisitionDiagnostic ? readEnvironmentAcquisitionDiagnostic(error) : null;
return cleanup ? { schema: SCHEMA, cleanup, ...(acquisitionDiagnostic ? { acquisitionDiagnostic } : {}) } : undefined;
}
export function readEnvironmentCreationCleanupError(error: unknown): PluginEnvironmentCreationCleanup | null {
const data = error instanceof PluginEnvironmentCreationCleanupError
? { schema: SCHEMA, cleanup: error.cleanup }
: error && typeof error === "object" && "data" in error ? error.data : null;
if (!data || typeof data !== "object" || !("schema" in data) || data.schema !== SCHEMA || !("cleanup" in data)) return null;
const value = data.cleanup;
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
const row = value as Record<string, unknown>;
const identifier = (value: unknown): value is string => typeof value === "string" && /^[a-zA-Z0-9_-]{1,200}$/.test(value);
if (![row.providerLeaseId, row.companyId, row.environmentId, row.attemptId].every(identifier)) return null;
if (row.runId !== undefined && !identifier(row.runId)) return null;
if (row.observedProviderLeaseId !== undefined && !identifier(row.observedProviderLeaseId)) return null;
if (typeof row.accountFingerprint !== "string" || !/^[a-f0-9]{64}$/.test(row.accountFingerprint)) return null;
if (!row.labels || typeof row.labels !== "object" || Array.isArray(row.labels)) return null;
const labels = row.labels as Record<string, unknown>;
if (Object.keys(labels).length > 16 || Object.entries(labels).some(([key, value]) => !/^paperclip-[a-z-]+$/.test(key) || !identifier(value))) return null;
return {
providerLeaseId: row.providerLeaseId as string,
...(typeof row.observedProviderLeaseId === "string" ? { observedProviderLeaseId: row.observedProviderLeaseId } : {}),
companyId: row.companyId as string,
environmentId: row.environmentId as string,
...(typeof row.runId === "string" ? { runId: row.runId } : {}),
attemptId: row.attemptId as string,
accountFingerprint: row.accountFingerprint,
labels: { ...labels } as Record<string, string>,
};
}