From 42d4499489117d6ab7094ca68b90d4a561c8dbb0 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 10 Nov 2026 13:20:30 -0600 Subject: [PATCH] fix(daytona): force a stuck stop before starting a retained sandbox (#20609) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On Daytona, a follow-up task resumes the agent's retained sandbox before the agent starts > - The host stops an idle sandbox gracefully. A process that ignores SIGTERM can leave Daytona in `stopping` for many minutes > - Resume waited for that state change until its 300 s start deadline. It never escalated, so each attempt failed after five minutes, and the bounded retries did the same > - On staging, a user comment waited about 21 minutes before the agent started. Two resume attempts each used 299.7 s in the plugin dispatch > - This pull request forces a stuck stop once, then starts the sandbox within the same deadline. It also traces the wait with the observed state > - The benefit is that a stuck stop costs about 30 s instead of failing the run, and the next case is diagnosable from the trace ## Linked Issues or Issue Description **What happened?** A user commented on a task that used a retained Daytona sandbox. The first run and its first retry each failed after about 5 min 20 s with `reusable_sandbox_resume_failed (cause=provider_error)`. The Honeycomb trace shows `execution.plugin_dispatch` at 299.7 s in both. Inside it there is only `lease.lookup` (44 ms) and `lease.channel_cleanup`, and no `lease.start`. So the whole budget went to `waitOutSandboxTransition`, which polled a sandbox that stayed in a transition state. The later attempt, after a control-plane restart, found the sandbox `stopped`. Its `lease.start` took 1.3 s. A transition that ends in `stopped` is a stop. **Expected behavior** A sandbox that stays `stopping` for longer than the host's own graceful-stop budget is forced to stop, then starts. The run does not fail on the start deadline. **Steps to reproduce** 1. Retain a Daytona sandbox whose graceful stop does not finish (Daytona reports `stopping`). 2. Resume or realize the lease. 3. Before this change, the call polls until the 300 s deadline and fails with "is still stopping at its start deadline". ## What Changed - `packages/plugins/sandbox-providers/daytona/src/plugin.ts`: `waitOutSandboxTransition` tracks how long the sandbox stays `stopping`. After `STUCK_STOPPING_FORCE_AFTER_MS` (30 s) it calls `sandbox.stop(seconds, true)` once, then keeps polling. A refused force stop keeps the old wait within the same deadline. - The wait is a `lease.wait_transition` span. The force stop is a `lease.force_stop` child span. - `lease-tracing.ts`: the closed `paperclip.sandbox.startup.state` label also records `starting`, `stopping`, `restoring`, `archiving` and `resizing`. Before, these were all `other`. - `server/src/services/plugin-host-services.ts`: the host admits the two new span names and the new closed state values. - `doc/observability.md`: documents the new spans and states. ## Verification - Red/green: with the plugin change stashed, the three new stuck-`stopping` tests (realize, refused force stop, resume) fail by hanging to the deadline. With the change they pass. The "settles on its own" test confirms that no force stop happens for a normal stop. - `vitest run` in `packages/plugins/sandbox-providers/daytona`: 14 files, 618 tests passed (8 skipped). `tsc --noEmit` passed. - `server/src/__tests__/plugin-host-services-span.test.ts`: red before the allowlist change, 25/25 after. - `pnpm verify` exited 2 (the doc is a global CI input), so CI supplies the broad coverage. - Not verified on a live Daytona sandbox. Hosted acceptance needs the change deployed to staging. ## Risks - The force stop sends SIGKILL to processes in a sandbox that is already stopping and is about to start. The disk persists across a stop, so the retained workspace is kept. A graceful stop that is still running past 30 s is killed. The host's own graceful stop budget is 30 s, so the host had already given up on it. - Daytona can refuse a stop while a state change is in progress. Then the code waits as before, so the result is no worse than today. - The root cause is inferred from the trace (a transition that ends in `stopped`). The new state label confirms or refutes it the next time it happens. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, browser, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/observability.md | 12 +- .../daytona/src/lease-tracing.ts | 11 +- .../daytona/src/plugin.test.ts | 139 ++++++++++++++++++ .../sandbox-providers/daytona/src/plugin.ts | 92 +++++++++++- .../plugin-host-services-span.test.ts | 5 +- server/src/services/plugin-host-services.ts | 10 +- .../runner-e2e/source-sdk-public-inputs.json | 4 +- 7 files changed, 258 insertions(+), 15 deletions(-) diff --git a/doc/observability.md b/doc/observability.md index 69c9d899cdf..25c4ce9a769 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -1999,6 +1999,8 @@ execution.plugin_dispatch ├── sandbox.daytona.lease.lookup ├── sandbox.daytona.lease.refresh (missing-container confirmation) ├── sandbox.daytona.lease.channel_cleanup (before restart) + ├── sandbox.daytona.lease.wait_transition (only while the sandbox is changing state) + │ └── sandbox.daytona.lease.force_stop (only when `stopping` outlasts 30 s) ├── sandbox.daytona.lease.start (or lease.recover) └── sandbox.daytona.lease.verify ├── sandbox.daytona.lease.workspace @@ -2011,8 +2013,16 @@ execution.plugin_dispatch internal readiness wait. They also cover starts triggered by execution or sync. They do not separate SDK-internal requests or provider-side queue time. Their `paperclip.sandbox.startup.state` attribute records the state before the -call: `started`, `stopped`, `archived`, `error`, or `other`. Both the plugin and +call: `started`, `stopped`, `archived`, `error`, a transition (`starting`, +`stopping`, `restoring`, `archiving`, `resizing`), or `other`. Both the plugin and host restrict this value; no provider error text or resource identity is sent. +`lease.wait_transition` covers polling a sandbox that is changing state before a +start; its state attribute is the state it found. A start that finds the sandbox +`stopping` for more than 30 s forces the stop (`lease.force_stop`) and keeps +waiting within the same start deadline if Daytona refuses. The forced stop belongs +to the sandbox: parallel callers send one, and none of them starts the sandbox +until that request settles. A confirmed start ends that stuck stop, so a later +one is forced again. Fresh acquisition emits `lease.create`, `lease.workspace`, `lease.shell`, `lease.expiry`, and `lease.sentinel`. `lease.create` covers the existing creation diff --git a/packages/plugins/sandbox-providers/daytona/src/lease-tracing.ts b/packages/plugins/sandbox-providers/daytona/src/lease-tracing.ts index 8ad09499f64..99d7a4f7e58 100644 --- a/packages/plugins/sandbox-providers/daytona/src/lease-tracing.ts +++ b/packages/plugins/sandbox-providers/daytona/src/lease-tracing.ts @@ -7,6 +7,8 @@ export type LeaseSpanName = | "lease.channel_cleanup" | "lease.start" | "lease.recover" + | "lease.wait_transition" + | "lease.force_stop" | "lease.verify" | "lease.workspace" | "lease.sentinel" @@ -15,12 +17,17 @@ export type LeaseSpanName = | "lease.expiry" | "lease.cleanup"; +// A closed set: the settled states plus the transitions a start waits out. +const LABELLED_STATES: ReadonlySet = new Set([ + "started", "stopped", "archived", "error", + "starting", "stopping", "restoring", "archiving", "resizing", +]); + /** Keep provider state useful without exporting a new SDK value as free text. */ export function leaseStateAttributes(state: unknown): Record { return { "paperclip.sandbox.startup.state": - state === "started" || state === "stopped" || state === "archived" || state === "error" - ? state : "other", + typeof state === "string" && LABELLED_STATES.has(state) ? state : "other", }; } diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts index c1bf861a4d8..1ecd185ea8a 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts @@ -36,6 +36,7 @@ import plugin, { __getDaytonaWritableDirsForTest, __setDaytonaPluginContextForTest, WARM_IDLE_AUTO_STOP_MARGIN_MINUTES, + STUCK_STOPPING_FORCE_AFTER_MS, } from "./plugin.js"; import type { PluginContext } from "@paperclipai/plugin-sdk"; import { environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncErrorCapture, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError, readEnvironmentAcquisitionDiagnostic, PluginEnvironmentStopError, environmentStopErrorData, readEnvironmentStopFailure } from "@paperclipai/plugin-sdk"; @@ -330,6 +331,111 @@ describe("Daytona idle eligibility", () => { expect(sandbox.start.mock.calls[0]![0]).toBeLessThanOrEqual(2); }); + it("does not force a stop that settles on its own", async () => { + const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopping" }); + mockGet.mockResolvedValue(sandbox); + sandbox.refreshData.mockImplementation(async () => { sandbox.state = "stopped"; }); + sandbox.start.mockImplementation(async () => { sandbox.state = "started"; }); + await expect(realize()).resolves.toMatchObject({ cwd: "/home/daytona/paperclip-workspace" }); + expect(sandbox.stop).not.toHaveBeenCalled(); + }); + + it("force-stops a sandbox stuck stopping, then starts it", async () => { + // A graceful stop whose processes ignore SIGTERM can leave Daytona in + // `stopping` for many minutes. Only a forced stop completes it. + const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopping" }); + mockGet.mockResolvedValue(sandbox); + sandbox.refreshData.mockImplementation(async () => {}); + sandbox.stop.mockImplementation(async (_timeout?: number, force?: boolean) => { + if (!force) throw stateChangeInProgress(); + sandbox.state = "stopped"; + }); + sandbox.start.mockImplementation(async () => { sandbox.state = "started"; }); + vi.useFakeTimers(); + try { + const realized = realize(); + await vi.advanceTimersByTimeAsync(STUCK_STOPPING_FORCE_AFTER_MS + 5_000); + await expect(realized).resolves.toMatchObject({ cwd: "/home/daytona/paperclip-workspace" }); + } finally { + vi.useRealTimers(); + } + expect(sandbox.stop).toHaveBeenCalledTimes(1); + expect(sandbox.stop.mock.calls[0]![1]).toBe(true); + expect(sandbox.start).toHaveBeenCalledTimes(1); + }); + + it("sends one forced stop for parallel callers and starts nothing until it settles", async () => { + const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopping" }); + mockGet.mockResolvedValue(sandbox); + sandbox.refreshData.mockImplementation(async () => {}); + let finishStop!: () => void; + sandbox.stop.mockImplementation(() => new Promise(resolve => { + finishStop = () => { sandbox.state = "stopped"; resolve(); }; + })); + sandbox.start.mockImplementation(async () => { sandbox.state = "started"; }); + vi.useFakeTimers(); + try { + const first = realize(); + const second = realize(); + await vi.advanceTimersByTimeAsync(STUCK_STOPPING_FORCE_AFTER_MS + 5_000); + expect(sandbox.stop).toHaveBeenCalledTimes(1); + // The forced stop outlives its own liveness bound; no caller may start + // the sandbox while that request could still land. + await vi.advanceTimersByTimeAsync(60_000); + expect(sandbox.stop).toHaveBeenCalledTimes(1); + expect(sandbox.start).not.toHaveBeenCalled(); + finishStop(); + await vi.advanceTimersByTimeAsync(2_000); + await expect(first).resolves.toMatchObject({ cwd: "/home/daytona/paperclip-workspace" }); + await expect(second).resolves.toMatchObject({ cwd: "/home/daytona/paperclip-workspace" }); + } finally { + vi.useRealTimers(); + } + expect(sandbox.stop).toHaveBeenCalledTimes(1); + expect(sandbox.start.mock.invocationCallOrder[0]!).toBeGreaterThan(sandbox.stop.mock.invocationCallOrder[0]!); + }); + + it("forces a new stuck stop after the sandbox restarted, inside the cooldown", async () => { + const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopping" }); + mockGet.mockResolvedValue(sandbox); + sandbox.refreshData.mockImplementation(async () => {}); + sandbox.stop.mockImplementation(async (_timeout?: number, force?: boolean) => { + if (force) sandbox.state = "stopped"; + }); + sandbox.start.mockImplementation(async () => { sandbox.state = "started"; }); + vi.useFakeTimers(); + try { + for (let cycle = 0; cycle < 2; cycle++) { + sandbox.state = "stopping"; // The next graceful stop is stuck again. + const realized = realize(); + await vi.advanceTimersByTimeAsync(STUCK_STOPPING_FORCE_AFTER_MS + 5_000); + await expect(realized).resolves.toMatchObject({ cwd: "/home/daytona/paperclip-workspace" }); + } + } finally { + vi.useRealTimers(); + } + expect(sandbox.stop).toHaveBeenCalledTimes(2); + expect(sandbox.start).toHaveBeenCalledTimes(2); + }); + + it("keeps waiting to the deadline when the forced stop is refused", async () => { + const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopping" }); + mockGet.mockResolvedValue(sandbox); + sandbox.refreshData.mockImplementation(async () => {}); + sandbox.stop.mockRejectedValue(stateChangeInProgress()); + vi.useFakeTimers(); + try { + const realized = realize(STUCK_STOPPING_FORCE_AFTER_MS + 20_000); + const outcome = expect(realized).rejects.toThrow("is still stopping at its start deadline"); + await vi.advanceTimersByTimeAsync(STUCK_STOPPING_FORCE_AFTER_MS + 40_000); + await outcome; + } finally { + vi.useRealTimers(); + } + expect(sandbox.stop).toHaveBeenCalledTimes(1); + expect(sandbox.start).not.toHaveBeenCalled(); + }); + it("fails other start errors at once", async () => { const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopped" }); mockGet.mockResolvedValue(sandbox); @@ -2511,6 +2617,39 @@ describe("Daytona sandbox provider plugin", () => { expect(sandbox.delete).not.toHaveBeenCalled(); }); + it("resumes a retained sandbox stuck stopping by forcing the stop", async () => { + const sandbox = createMockSandbox({ id: "sandbox-retained", state: "stopping" }); + const allocationRoot = "/home/daytona/paperclip-workspace"; + const workspaceId = "6028a46b-b65a-421f-952d-66b9c002b8e2"; + const remoteCwd = `${allocationRoot}/.paperclip-workspaces/${workspaceId}`; + const sentinelPath = `${allocationRoot}/.paperclip-runtime/reusable-sandbox-lease.json`; + sandbox.process.executeCommand.mockImplementation(async command => ({ exitCode: 0, + result: command.startsWith("cat ") ? JSON.stringify({ token: "sentinel-token" }) : "bash" })); + sandbox.refreshData.mockImplementation(async () => {}); + sandbox.stop.mockImplementation(async (_timeout?: number, force?: boolean) => { + if (force) sandbox.state = "stopped"; + }); + sandbox.start.mockImplementation(async () => { sandbox.state = "started"; }); + mockGet.mockResolvedValue(sandbox); + vi.useFakeTimers(); + try { + const resuming = plugin.definition.onEnvironmentResumeLease!({ + driverKey: "daytona", companyId: "company-1", environmentId: "env-1", providerLeaseId: sandbox.id, + config: { apiKey: "fixture-key", reuseLease: true }, requestedCwd: remoteCwd, leaseMetadata: { remoteCwd, + remoteWorkspaceId: workspaceId, sandboxId: sandbox.id, reuseLease: true, workspaceSentinel: { path: sentinelPath, + token: "sentinel-token", runId: "fixture-run", providerLeaseId: sandbox.id, result: "written" } }, + }); + await vi.advanceTimersByTimeAsync(STUCK_STOPPING_FORCE_AFTER_MS + 5_000); + await expect(resuming).resolves.toMatchObject({ providerLeaseId: sandbox.id }); + } finally { + vi.useRealTimers(); + } + expect(sandbox.stop).toHaveBeenCalledWith(expect.any(Number), true); + expect(sandbox.start).toHaveBeenCalledTimes(1); + expect(mockCreate).not.toHaveBeenCalled(); + expect(sandbox.delete).not.toHaveBeenCalled(); + }); + it.each([null, 7, "", "relative/workspace", "/workspace/../other", "/workspace\0other"])( "rejects a malformed recorded cwd before touching the provider: %j", async remoteCwd => { await expect(plugin.definition.onEnvironmentResumeLease!({ driverKey: "daytona", companyId: "company-1", diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.ts index 15839c36ad0..b6df53f97c7 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.ts @@ -646,27 +646,104 @@ function isSandboxStateChangeInProgress(error: unknown): boolean { return /state change in progress/i.test(error instanceof Error ? error.message : String(error)); } -async function waitOutSandboxTransition(sandbox: Sandbox, deadline: number, readTimeoutMs: number): Promise { - while (SANDBOX_TRANSITION_STATES.has(sandbox.state ?? "") && Date.now() + SANDBOX_TRANSITION_POLL_MS < deadline) { - await new Promise(resolve => setTimeout(resolve, SANDBOX_TRANSITION_POLL_MS)); - await withLivenessTimeout("sandbox.refreshData", Math.min(readTimeoutMs, Math.max(1, deadline - Date.now())), - () => sandbox.refreshData()); +/** + * How long a sandbox may stay `stopping` before a start forces the stop. The + * host stops an idle sandbox gracefully with a 30 s budget, so a stop still + * pending after this is stuck, typically on a process that ignores SIGTERM. + * Daytona can leave such a sandbox `stopping` for many minutes. + */ +export const STUCK_STOPPING_FORCE_AFTER_MS = 30_000; +const STUCK_STOPPING_FORCE_STOP_SECONDS = 30; +/** + * One forced stop per stuck stop, whichever caller sends it. A confirmed start + * ends the episode early (see {@link endForcedStopEpisode}). + */ +export const FORCED_STOP_COOLDOWN_MS = 10 * 60_000; + +// Forced stops are owned by the sandbox, not by a caller. Several callers can +// wait on one stuck sandbox at once (execute, sync, realize, resume). Only one +// of them sends the forced stop, and no caller starts the sandbox while that +// request is unsettled: a late stop would otherwise kill a sandbox another +// caller has just restarted. The entry outlives its liveness timeout until +// the SDK request itself settles. +const forcedSandboxStops = new Map | null }>(); + +function forcedStopInFlight(key: string): Promise | null { + return forcedSandboxStops.get(key)?.pending ?? null; +} + +// A confirmed start ends the stuck stop. A later stuck stop is a new episode +// and may be forced at once. An unsettled request still blocks every start. +function endForcedStopEpisode(key: string): void { + if (forcedSandboxStops.get(key)?.pending === null) forcedSandboxStops.delete(key); +} + +function claimForcedStop(key: string, now: number): boolean { + const previous = forcedSandboxStops.get(key); + if (previous && (previous.pending || now - previous.issuedAt < FORCED_STOP_COOLDOWN_MS)) return false; + forcedSandboxStops.set(key, { issuedAt: now, pending: null }); + return true; +} + +async function sendForcedStop(sandbox: Sandbox, key: string, deadline: number): Promise { + const remainingSeconds = Math.max(1, Math.floor((deadline - Date.now()) / 1_000)); + const stopSeconds = Math.min(STUCK_STOPPING_FORCE_STOP_SECONDS, remainingSeconds); + const entry = forcedSandboxStops.get(key)!; + // `request` settles only when the SDK call does, even past the liveness bound. + const request = Promise.resolve().then(() => sandbox.stop(stopSeconds, true)); + entry.pending = request.then(() => undefined, () => undefined) + .finally(() => { if (forcedSandboxStops.get(key) === entry) entry.pending = null; }); + try { + await traceLeaseStep("lease.force_stop", () => withLivenessTimeout("sandbox.stop", + stopSeconds * 1_000 + LIVENESS_START_TIMEOUT_MARGIN_MS, () => request)); + } catch { + // Keep waiting within the same deadline; the next read reports the state. } } +async function waitOutSandboxTransition(sandbox: Sandbox, scope: SandboxScope, deadline: number, + readTimeoutMs: number): Promise { + const key = sandboxHandleCacheKey(scope); + const mustWait = () => SANDBOX_TRANSITION_STATES.has(sandbox.state ?? "") || forcedStopInFlight(key) !== null; + if (!mustWait()) return; + await traceLeaseStep("lease.wait_transition", async () => { + let stoppingSince: number | null = null; + while (mustWait() && Date.now() + SANDBOX_TRANSITION_POLL_MS < deadline) { + stoppingSince = sandbox.state === "stopping" ? stoppingSince ?? Date.now() : null; + // The sandbox is about to be started, so killing what holds the stop + // loses nothing a graceful stop would keep: the disk survives a stop. + // A refusal leaves the plain wait unchanged. + if (stoppingSince !== null && Date.now() - stoppingSince >= STUCK_STOPPING_FORCE_AFTER_MS + && claimForcedStop(key, Date.now())) { + await sendForcedStop(sandbox, key, deadline); + continue; + } + const pending = forcedStopInFlight(key); + await Promise.race([ + new Promise(resolve => setTimeout(resolve, SANDBOX_TRANSITION_POLL_MS)), + ...(pending ? [pending] : []), + ]); + await withLivenessTimeout("sandbox.refreshData", Math.min(readTimeoutMs, Math.max(1, deadline - Date.now())), + () => sandbox.refreshData()); + } + }, leaseStateAttributes(sandbox.state)); +} + async function ensureSandboxStarted(sandbox: Sandbox, timeoutSeconds: number, scope: SandboxScope): Promise { if (sandbox.state === "started") return; sandboxHandleSessionStore.clear(scope); const deadline = Date.now() + timeoutSeconds * 1_000; const readTimeoutMs = Math.min(scope.config.livenessTimeoutMs, 30_000); for (;;) { - await waitOutSandboxTransition(sandbox, deadline, readTimeoutMs); - if (SANDBOX_TRANSITION_STATES.has(sandbox.state ?? "") || Date.now() >= deadline) { + await waitOutSandboxTransition(sandbox, scope, deadline, readTimeoutMs); + if (SANDBOX_TRANSITION_STATES.has(sandbox.state ?? "") || forcedStopInFlight(sandboxHandleCacheKey(scope)) + || Date.now() >= deadline) { throw new Error(`Daytona sandbox ${sandbox.id} is still ${sandbox.state ?? "changing state"} at its start deadline`); } const remainingSeconds = Math.max(1, Math.ceil((deadline - Date.now()) / 1_000)); try { await startSettledSandbox(sandbox, remainingSeconds, scope); + endForcedStopEpisode(sandboxHandleCacheKey(scope)); return; } catch (error) { if (!isSandboxStateChangeInProgress(error) || Date.now() + SANDBOX_TRANSITION_POLL_MS >= deadline) throw error; @@ -1858,6 +1935,7 @@ export function __resetDaytonaSandboxHandleCacheForTest(): void { sandboxHandleActivityGates.reset(); sandboxHandleLeaseAdmissionStates.reset(); sandboxHandleWritableDirs.reset(); + forcedSandboxStops.clear(); sandboxHandleSessionStore.reset(); runnerIngressGenerationStore.reset(); retiredDaytonaDuplexChannelByRoute.clear(); diff --git a/server/src/__tests__/plugin-host-services-span.test.ts b/server/src/__tests__/plugin-host-services-span.test.ts index 81b03f40970..1c3011f1746 100644 --- a/server/src/__tests__/plugin-host-services-span.test.ts +++ b/server/src/__tests__/plugin-host-services-span.test.ts @@ -349,6 +349,9 @@ describe("nested worker timing receipts",()=>{ startTimeMs: now - 80, endTimeMs: now - 20, attributes: { "paperclip.sandbox.startup.state": "archived" } }); expect(JSON.stringify(mockRecordSpan.mock.calls)).not.toContain("private-token"); expect(clampProviderSpanAttributes({ "paperclip.sandbox.startup.state": "private-state" })).toEqual({}); + for (const state of ["starting", "stopping", "restoring", "archiving", "resizing"]) { + expect(clampProviderSpanAttributes({ "paperclip.sandbox.startup.state": state })).toEqual({ "paperclip.sandbox.startup.state": state }); + } }); it("admits a workspace failure span with only a closed failure reason", () => { @@ -366,7 +369,7 @@ describe("nested worker timing receipts",()=>{ it("admits every closed lease phase and rejects dynamic phase names", () => { mockRecordSpan.mockReset(); const names = ["lease.wait_for_teardown", "lease.lookup", "lease.refresh", "lease.channel_cleanup", - "lease.start", "lease.recover", "lease.verify", "lease.workspace", "lease.sentinel", "lease.shell", + "lease.start", "lease.recover", "lease.wait_transition", "lease.force_stop", "lease.verify", "lease.workspace", "lease.sentinel", "lease.shell", "lease.create", "lease.expiry", "lease.cleanup", "lease.private-resource-id"]; for (const name of names) recordWorkerProviderSpan({ name }, { traceparent: VALID_TRACEPARENT }); expect(mockRecordSpan.mock.calls.map(call => call[0].name)).toEqual([ diff --git a/server/src/services/plugin-host-services.ts b/server/src/services/plugin-host-services.ts index cbabcf674cf..ee41f2c1e33 100644 --- a/server/src/services/plugin-host-services.ts +++ b/server/src/services/plugin-host-services.ts @@ -541,7 +541,7 @@ const KNOWN_PROVIDER_SPAN_NAMES: ReadonlySet = new Set([ "session.open", "session.close", "lease.wait_for_teardown", "lease.lookup", "lease.refresh", "lease.channel_cleanup", - "lease.start", "lease.recover", "lease.verify", "lease.workspace", "lease.sentinel", + "lease.start", "lease.recover", "lease.wait_transition", "lease.force_stop", "lease.verify", "lease.workspace", "lease.sentinel", "lease.shell", "lease.create", "lease.expiry", "lease.cleanup", "plugin.worker", "plugin.host.identity.read", "plugin.host.config.read", "plugin.host.progress.read", "plugin.host.progress.write", "plugin.workspace.prepare", "plugin.workspace.failure", "plugin.storage.queue", "plugin.storage.execute", @@ -591,6 +591,12 @@ const KNOWN_SPAN_OUTCOMES: ReadonlySet = new Set(["ok", "skipped", "fail /** The closed value set for the `transfer.direction` attribute. */ const KNOWN_TRANSFER_DIRECTIONS: ReadonlySet = new Set(["inbound", "outbound"]); +// Settled states plus the transitions a start waits out; anything else is dropped. +const PROVIDER_STARTUP_STATES: ReadonlySet = new Set([ + "started", "stopped", "archived", "error", "other", + "starting", "stopping", "restoring", "archiving", "resizing", +]); + /** * Re-clamp the worker-sent attributes at the trust boundary. Drop every key that * is not on the allowlist. Re-map `provider` through `normalizeProviderFamily`, @@ -606,7 +612,7 @@ export function clampProviderSpanAttributes( for (const [key, value] of Object.entries(raw)) { if (!PROVIDER_SPAN_ATTR_ALLOWLIST.has(key)) continue; if (key === SPAN_ATTRS.state) { - if (typeof value === "string" && ["started", "stopped", "archived", "error", "other"].includes(value)) clamped[key] = value; + if (typeof value === "string" && PROVIDER_STARTUP_STATES.has(value)) clamped[key] = value; continue; } if (key === "plugin.operation") { diff --git a/tests/runner-e2e/source-sdk-public-inputs.json b/tests/runner-e2e/source-sdk-public-inputs.json index 51fdf687bc7..5211d62ac65 100644 --- a/tests/runner-e2e/source-sdk-public-inputs.json +++ b/tests/runner-e2e/source-sdk-public-inputs.json @@ -528,12 +528,12 @@ "packages/shared/src/worktree-seed-source.test.ts": "852856b1b5b8f4d0d3dada6501886fe50a2c2dde", "packages/shared/src/worktree-seed-source.ts": "3d1a09bed2d969feabef3400ff84fd499c1408f4", "packages/shared/tsconfig.json": "5a24989cd30dcf3749ca7e60d4e83d6ddbb14714", - "pnpm-lock.yaml": "851e65af7af027e3db99e759ece3d89e4a4700c2", + "pnpm-lock.yaml": "6d825f9785f931f3bb08b0a6d2a741781f28f58a", "scripts/ensure-plugin-build-deps.mjs": "356e0e0f80e50890067027da3003485cf5d2ccbe", "tsconfig.base.json": "2d0e41f723add0c678936eb9ae1c60eab710f8e0", "tsconfig.json": "994ee490fb97be7cc0954f4dfd88cf0723290e6c" }, - "lockSha256": "2360d5e93876ce3cb6e810257d92d669ac2b109027b0e01af36f2048861648c0", + "lockSha256": "b32083066ce89ba9a570a2927b3c4c8385294a5f04efcbae462db82b2e8c4455", "compilerPackageSha256": "3722b30210616a13a3213ded11575ba6b2dbab10c32a5ef67afca8513e27017e", "parserPackageSha256": "6d6a65b879cd7cd249a4367c0b57cce00e78c728de45056ac2a5ca1d1f528d1a", "parserEntrySha256": "6969920ae0610df927b6b3e675d1309372c268e36d391652af8e3e0183cbe8f8",