mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
fix(daytona): force a stuck stop before starting a retained sandbox (#20609)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On Daytona, a follow-up task resumes the agent's retained sandbox before the agent starts > - The host stops an idle sandbox gracefully. A process that ignores SIGTERM can leave Daytona in `stopping` for many minutes > - Resume waited for that state change until its 300 s start deadline. It never escalated, so each attempt failed after five minutes, and the bounded retries did the same > - On staging, a user comment waited about 21 minutes before the agent started. Two resume attempts each used 299.7 s in the plugin dispatch > - This pull request forces a stuck stop once, then starts the sandbox within the same deadline. It also traces the wait with the observed state > - The benefit is that a stuck stop costs about 30 s instead of failing the run, and the next case is diagnosable from the trace ## Linked Issues or Issue Description **What happened?** A user commented on a task that used a retained Daytona sandbox. The first run and its first retry each failed after about 5 min 20 s with `reusable_sandbox_resume_failed (cause=provider_error)`. The Honeycomb trace shows `execution.plugin_dispatch` at 299.7 s in both. Inside it there is only `lease.lookup` (44 ms) and `lease.channel_cleanup`, and no `lease.start`. So the whole budget went to `waitOutSandboxTransition`, which polled a sandbox that stayed in a transition state. The later attempt, after a control-plane restart, found the sandbox `stopped`. Its `lease.start` took 1.3 s. A transition that ends in `stopped` is a stop. **Expected behavior** A sandbox that stays `stopping` for longer than the host's own graceful-stop budget is forced to stop, then starts. The run does not fail on the start deadline. **Steps to reproduce** 1. Retain a Daytona sandbox whose graceful stop does not finish (Daytona reports `stopping`). 2. Resume or realize the lease. 3. Before this change, the call polls until the 300 s deadline and fails with "is still stopping at its start deadline". ## What Changed - `packages/plugins/sandbox-providers/daytona/src/plugin.ts`: `waitOutSandboxTransition` tracks how long the sandbox stays `stopping`. After `STUCK_STOPPING_FORCE_AFTER_MS` (30 s) it calls `sandbox.stop(seconds, true)` once, then keeps polling. A refused force stop keeps the old wait within the same deadline. - The wait is a `lease.wait_transition` span. The force stop is a `lease.force_stop` child span. - `lease-tracing.ts`: the closed `paperclip.sandbox.startup.state` label also records `starting`, `stopping`, `restoring`, `archiving` and `resizing`. Before, these were all `other`. - `server/src/services/plugin-host-services.ts`: the host admits the two new span names and the new closed state values. - `doc/observability.md`: documents the new spans and states. ## Verification - Red/green: with the plugin change stashed, the three new stuck-`stopping` tests (realize, refused force stop, resume) fail by hanging to the deadline. With the change they pass. The "settles on its own" test confirms that no force stop happens for a normal stop. - `vitest run` in `packages/plugins/sandbox-providers/daytona`: 14 files, 618 tests passed (8 skipped). `tsc --noEmit` passed. - `server/src/__tests__/plugin-host-services-span.test.ts`: red before the allowlist change, 25/25 after. - `pnpm verify` exited 2 (the doc is a global CI input), so CI supplies the broad coverage. - Not verified on a live Daytona sandbox. Hosted acceptance needs the change deployed to staging. ## Risks - The force stop sends SIGKILL to processes in a sandbox that is already stopping and is about to start. The disk persists across a stop, so the retained workspace is kept. A graceful stop that is still running past 30 s is killed. The host's own graceful stop budget is 30 s, so the host had already given up on it. - Daytona can refuse a stop while a state change is in progress. Then the code waits as before, so the result is no worse than today. - The root cause is inferred from the trace (a transition that ends in `stopped`). The new state label confirms or refutes it the next time it happens. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, browser, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
9b21dcbdda
commit
42d4499489
7 files changed
+258
-15
No files matched your search
+11
-1
@@ -1999,6 +1999,8 @@ execution.plugin_dispatch
|
||||
├── sandbox.daytona.lease.lookup
|
||||
├── sandbox.daytona.lease.refresh (missing-container confirmation)
|
||||
├── sandbox.daytona.lease.channel_cleanup (before restart)
|
||||
├── sandbox.daytona.lease.wait_transition (only while the sandbox is changing state)
|
||||
│ └── sandbox.daytona.lease.force_stop (only when `stopping` outlasts 30 s)
|
||||
├── sandbox.daytona.lease.start (or lease.recover)
|
||||
└── sandbox.daytona.lease.verify
|
||||
├── sandbox.daytona.lease.workspace
|
||||
@@ -2011,8 +2013,16 @@ execution.plugin_dispatch
|
||||
internal readiness wait. They also cover starts triggered by execution or sync.
|
||||
They do not separate SDK-internal requests or provider-side queue time.
|
||||
Their `paperclip.sandbox.startup.state` attribute records the state before the
|
||||
call: `started`, `stopped`, `archived`, `error`, or `other`. Both the plugin and
|
||||
call: `started`, `stopped`, `archived`, `error`, a transition (`starting`,
|
||||
`stopping`, `restoring`, `archiving`, `resizing`), or `other`. Both the plugin and
|
||||
host restrict this value; no provider error text or resource identity is sent.
|
||||
`lease.wait_transition` covers polling a sandbox that is changing state before a
|
||||
start; its state attribute is the state it found. A start that finds the sandbox
|
||||
`stopping` for more than 30 s forces the stop (`lease.force_stop`) and keeps
|
||||
waiting within the same start deadline if Daytona refuses. The forced stop belongs
|
||||
to the sandbox: parallel callers send one, and none of them starts the sandbox
|
||||
until that request settles. A confirmed start ends that stuck stop, so a later
|
||||
one is forced again.
|
||||
|
||||
Fresh acquisition emits `lease.create`, `lease.workspace`, `lease.shell`,
|
||||
`lease.expiry`, and `lease.sentinel`. `lease.create` covers the existing creation
|
||||
|
||||
@@ -7,6 +7,8 @@ export type LeaseSpanName =
|
||||
| "lease.channel_cleanup"
|
||||
| "lease.start"
|
||||
| "lease.recover"
|
||||
| "lease.wait_transition"
|
||||
| "lease.force_stop"
|
||||
| "lease.verify"
|
||||
| "lease.workspace"
|
||||
| "lease.sentinel"
|
||||
@@ -15,12 +17,17 @@ export type LeaseSpanName =
|
||||
| "lease.expiry"
|
||||
| "lease.cleanup";
|
||||
|
||||
// A closed set: the settled states plus the transitions a start waits out.
|
||||
const LABELLED_STATES: ReadonlySet<string> = new Set([
|
||||
"started", "stopped", "archived", "error",
|
||||
"starting", "stopping", "restoring", "archiving", "resizing",
|
||||
]);
|
||||
|
||||
/** Keep provider state useful without exporting a new SDK value as free text. */
|
||||
export function leaseStateAttributes(state: unknown): Record<string, string> {
|
||||
return {
|
||||
"paperclip.sandbox.startup.state":
|
||||
state === "started" || state === "stopped" || state === "archived" || state === "error"
|
||||
? state : "other",
|
||||
typeof state === "string" && LABELLED_STATES.has(state) ? state : "other",
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -36,6 +36,7 @@ import plugin, {
|
||||
__getDaytonaWritableDirsForTest,
|
||||
__setDaytonaPluginContextForTest,
|
||||
WARM_IDLE_AUTO_STOP_MARGIN_MINUTES,
|
||||
STUCK_STOPPING_FORCE_AFTER_MS,
|
||||
} from "./plugin.js";
|
||||
import type { PluginContext } from "@paperclipai/plugin-sdk";
|
||||
import { environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncErrorCapture, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError, readEnvironmentAcquisitionDiagnostic, PluginEnvironmentStopError, environmentStopErrorData, readEnvironmentStopFailure } from "@paperclipai/plugin-sdk";
|
||||
@@ -330,6 +331,111 @@ describe("Daytona idle eligibility", () => {
|
||||
expect(sandbox.start.mock.calls[0]![0]).toBeLessThanOrEqual(2);
|
||||
});
|
||||
|
||||
it("does not force a stop that settles on its own", async () => {
|
||||
const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopping" });
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
sandbox.refreshData.mockImplementation(async () => { sandbox.state = "stopped"; });
|
||||
sandbox.start.mockImplementation(async () => { sandbox.state = "started"; });
|
||||
await expect(realize()).resolves.toMatchObject({ cwd: "/home/daytona/paperclip-workspace" });
|
||||
expect(sandbox.stop).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("force-stops a sandbox stuck stopping, then starts it", async () => {
|
||||
// A graceful stop whose processes ignore SIGTERM can leave Daytona in
|
||||
// `stopping` for many minutes. Only a forced stop completes it.
|
||||
const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopping" });
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
sandbox.refreshData.mockImplementation(async () => {});
|
||||
sandbox.stop.mockImplementation(async (_timeout?: number, force?: boolean) => {
|
||||
if (!force) throw stateChangeInProgress();
|
||||
sandbox.state = "stopped";
|
||||
});
|
||||
sandbox.start.mockImplementation(async () => { sandbox.state = "started"; });
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
const realized = realize();
|
||||
await vi.advanceTimersByTimeAsync(STUCK_STOPPING_FORCE_AFTER_MS + 5_000);
|
||||
await expect(realized).resolves.toMatchObject({ cwd: "/home/daytona/paperclip-workspace" });
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
expect(sandbox.stop).toHaveBeenCalledTimes(1);
|
||||
expect(sandbox.stop.mock.calls[0]![1]).toBe(true);
|
||||
expect(sandbox.start).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("sends one forced stop for parallel callers and starts nothing until it settles", async () => {
|
||||
const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopping" });
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
sandbox.refreshData.mockImplementation(async () => {});
|
||||
let finishStop!: () => void;
|
||||
sandbox.stop.mockImplementation(() => new Promise<void>(resolve => {
|
||||
finishStop = () => { sandbox.state = "stopped"; resolve(); };
|
||||
}));
|
||||
sandbox.start.mockImplementation(async () => { sandbox.state = "started"; });
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
const first = realize();
|
||||
const second = realize();
|
||||
await vi.advanceTimersByTimeAsync(STUCK_STOPPING_FORCE_AFTER_MS + 5_000);
|
||||
expect(sandbox.stop).toHaveBeenCalledTimes(1);
|
||||
// The forced stop outlives its own liveness bound; no caller may start
|
||||
// the sandbox while that request could still land.
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
expect(sandbox.stop).toHaveBeenCalledTimes(1);
|
||||
expect(sandbox.start).not.toHaveBeenCalled();
|
||||
finishStop();
|
||||
await vi.advanceTimersByTimeAsync(2_000);
|
||||
await expect(first).resolves.toMatchObject({ cwd: "/home/daytona/paperclip-workspace" });
|
||||
await expect(second).resolves.toMatchObject({ cwd: "/home/daytona/paperclip-workspace" });
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
expect(sandbox.stop).toHaveBeenCalledTimes(1);
|
||||
expect(sandbox.start.mock.invocationCallOrder[0]!).toBeGreaterThan(sandbox.stop.mock.invocationCallOrder[0]!);
|
||||
});
|
||||
|
||||
it("forces a new stuck stop after the sandbox restarted, inside the cooldown", async () => {
|
||||
const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopping" });
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
sandbox.refreshData.mockImplementation(async () => {});
|
||||
sandbox.stop.mockImplementation(async (_timeout?: number, force?: boolean) => {
|
||||
if (force) sandbox.state = "stopped";
|
||||
});
|
||||
sandbox.start.mockImplementation(async () => { sandbox.state = "started"; });
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
for (let cycle = 0; cycle < 2; cycle++) {
|
||||
sandbox.state = "stopping"; // The next graceful stop is stuck again.
|
||||
const realized = realize();
|
||||
await vi.advanceTimersByTimeAsync(STUCK_STOPPING_FORCE_AFTER_MS + 5_000);
|
||||
await expect(realized).resolves.toMatchObject({ cwd: "/home/daytona/paperclip-workspace" });
|
||||
}
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
expect(sandbox.stop).toHaveBeenCalledTimes(2);
|
||||
expect(sandbox.start).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("keeps waiting to the deadline when the forced stop is refused", async () => {
|
||||
const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopping" });
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
sandbox.refreshData.mockImplementation(async () => {});
|
||||
sandbox.stop.mockRejectedValue(stateChangeInProgress());
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
const realized = realize(STUCK_STOPPING_FORCE_AFTER_MS + 20_000);
|
||||
const outcome = expect(realized).rejects.toThrow("is still stopping at its start deadline");
|
||||
await vi.advanceTimersByTimeAsync(STUCK_STOPPING_FORCE_AFTER_MS + 40_000);
|
||||
await outcome;
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
expect(sandbox.stop).toHaveBeenCalledTimes(1);
|
||||
expect(sandbox.start).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("fails other start errors at once", async () => {
|
||||
const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopped" });
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
@@ -2511,6 +2617,39 @@ describe("Daytona sandbox provider plugin", () => {
|
||||
expect(sandbox.delete).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("resumes a retained sandbox stuck stopping by forcing the stop", async () => {
|
||||
const sandbox = createMockSandbox({ id: "sandbox-retained", state: "stopping" });
|
||||
const allocationRoot = "/home/daytona/paperclip-workspace";
|
||||
const workspaceId = "6028a46b-b65a-421f-952d-66b9c002b8e2";
|
||||
const remoteCwd = `${allocationRoot}/.paperclip-workspaces/${workspaceId}`;
|
||||
const sentinelPath = `${allocationRoot}/.paperclip-runtime/reusable-sandbox-lease.json`;
|
||||
sandbox.process.executeCommand.mockImplementation(async command => ({ exitCode: 0,
|
||||
result: command.startsWith("cat ") ? JSON.stringify({ token: "sentinel-token" }) : "bash" }));
|
||||
sandbox.refreshData.mockImplementation(async () => {});
|
||||
sandbox.stop.mockImplementation(async (_timeout?: number, force?: boolean) => {
|
||||
if (force) sandbox.state = "stopped";
|
||||
});
|
||||
sandbox.start.mockImplementation(async () => { sandbox.state = "started"; });
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
const resuming = plugin.definition.onEnvironmentResumeLease!({
|
||||
driverKey: "daytona", companyId: "company-1", environmentId: "env-1", providerLeaseId: sandbox.id,
|
||||
config: { apiKey: "fixture-key", reuseLease: true }, requestedCwd: remoteCwd, leaseMetadata: { remoteCwd,
|
||||
remoteWorkspaceId: workspaceId, sandboxId: sandbox.id, reuseLease: true, workspaceSentinel: { path: sentinelPath,
|
||||
token: "sentinel-token", runId: "fixture-run", providerLeaseId: sandbox.id, result: "written" } },
|
||||
});
|
||||
await vi.advanceTimersByTimeAsync(STUCK_STOPPING_FORCE_AFTER_MS + 5_000);
|
||||
await expect(resuming).resolves.toMatchObject({ providerLeaseId: sandbox.id });
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
expect(sandbox.stop).toHaveBeenCalledWith(expect.any(Number), true);
|
||||
expect(sandbox.start).toHaveBeenCalledTimes(1);
|
||||
expect(mockCreate).not.toHaveBeenCalled();
|
||||
expect(sandbox.delete).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([null, 7, "", "relative/workspace", "/workspace/../other", "/workspace\0other"])(
|
||||
"rejects a malformed recorded cwd before touching the provider: %j", async remoteCwd => {
|
||||
await expect(plugin.definition.onEnvironmentResumeLease!({ driverKey: "daytona", companyId: "company-1",
|
||||
|
||||
@@ -646,27 +646,104 @@ function isSandboxStateChangeInProgress(error: unknown): boolean {
|
||||
return /state change in progress/i.test(error instanceof Error ? error.message : String(error));
|
||||
}
|
||||
|
||||
async function waitOutSandboxTransition(sandbox: Sandbox, deadline: number, readTimeoutMs: number): Promise<void> {
|
||||
while (SANDBOX_TRANSITION_STATES.has(sandbox.state ?? "") && Date.now() + SANDBOX_TRANSITION_POLL_MS < deadline) {
|
||||
await new Promise(resolve => setTimeout(resolve, SANDBOX_TRANSITION_POLL_MS));
|
||||
await withLivenessTimeout("sandbox.refreshData", Math.min(readTimeoutMs, Math.max(1, deadline - Date.now())),
|
||||
() => sandbox.refreshData());
|
||||
/**
|
||||
* How long a sandbox may stay `stopping` before a start forces the stop. The
|
||||
* host stops an idle sandbox gracefully with a 30 s budget, so a stop still
|
||||
* pending after this is stuck, typically on a process that ignores SIGTERM.
|
||||
* Daytona can leave such a sandbox `stopping` for many minutes.
|
||||
*/
|
||||
export const STUCK_STOPPING_FORCE_AFTER_MS = 30_000;
|
||||
const STUCK_STOPPING_FORCE_STOP_SECONDS = 30;
|
||||
/**
|
||||
* One forced stop per stuck stop, whichever caller sends it. A confirmed start
|
||||
* ends the episode early (see {@link endForcedStopEpisode}).
|
||||
*/
|
||||
export const FORCED_STOP_COOLDOWN_MS = 10 * 60_000;
|
||||
|
||||
// Forced stops are owned by the sandbox, not by a caller. Several callers can
|
||||
// wait on one stuck sandbox at once (execute, sync, realize, resume). Only one
|
||||
// of them sends the forced stop, and no caller starts the sandbox while that
|
||||
// request is unsettled: a late stop would otherwise kill a sandbox another
|
||||
// caller has just restarted. The entry outlives its liveness timeout until
|
||||
// the SDK request itself settles.
|
||||
const forcedSandboxStops = new Map<string, { issuedAt: number; pending: Promise<void> | null }>();
|
||||
|
||||
function forcedStopInFlight(key: string): Promise<void> | null {
|
||||
return forcedSandboxStops.get(key)?.pending ?? null;
|
||||
}
|
||||
|
||||
// A confirmed start ends the stuck stop. A later stuck stop is a new episode
|
||||
// and may be forced at once. An unsettled request still blocks every start.
|
||||
function endForcedStopEpisode(key: string): void {
|
||||
if (forcedSandboxStops.get(key)?.pending === null) forcedSandboxStops.delete(key);
|
||||
}
|
||||
|
||||
function claimForcedStop(key: string, now: number): boolean {
|
||||
const previous = forcedSandboxStops.get(key);
|
||||
if (previous && (previous.pending || now - previous.issuedAt < FORCED_STOP_COOLDOWN_MS)) return false;
|
||||
forcedSandboxStops.set(key, { issuedAt: now, pending: null });
|
||||
return true;
|
||||
}
|
||||
|
||||
async function sendForcedStop(sandbox: Sandbox, key: string, deadline: number): Promise<void> {
|
||||
const remainingSeconds = Math.max(1, Math.floor((deadline - Date.now()) / 1_000));
|
||||
const stopSeconds = Math.min(STUCK_STOPPING_FORCE_STOP_SECONDS, remainingSeconds);
|
||||
const entry = forcedSandboxStops.get(key)!;
|
||||
// `request` settles only when the SDK call does, even past the liveness bound.
|
||||
const request = Promise.resolve().then(() => sandbox.stop(stopSeconds, true));
|
||||
entry.pending = request.then(() => undefined, () => undefined)
|
||||
.finally(() => { if (forcedSandboxStops.get(key) === entry) entry.pending = null; });
|
||||
try {
|
||||
await traceLeaseStep("lease.force_stop", () => withLivenessTimeout("sandbox.stop",
|
||||
stopSeconds * 1_000 + LIVENESS_START_TIMEOUT_MARGIN_MS, () => request));
|
||||
} catch {
|
||||
// Keep waiting within the same deadline; the next read reports the state.
|
||||
}
|
||||
}
|
||||
|
||||
async function waitOutSandboxTransition(sandbox: Sandbox, scope: SandboxScope, deadline: number,
|
||||
readTimeoutMs: number): Promise<void> {
|
||||
const key = sandboxHandleCacheKey(scope);
|
||||
const mustWait = () => SANDBOX_TRANSITION_STATES.has(sandbox.state ?? "") || forcedStopInFlight(key) !== null;
|
||||
if (!mustWait()) return;
|
||||
await traceLeaseStep("lease.wait_transition", async () => {
|
||||
let stoppingSince: number | null = null;
|
||||
while (mustWait() && Date.now() + SANDBOX_TRANSITION_POLL_MS < deadline) {
|
||||
stoppingSince = sandbox.state === "stopping" ? stoppingSince ?? Date.now() : null;
|
||||
// The sandbox is about to be started, so killing what holds the stop
|
||||
// loses nothing a graceful stop would keep: the disk survives a stop.
|
||||
// A refusal leaves the plain wait unchanged.
|
||||
if (stoppingSince !== null && Date.now() - stoppingSince >= STUCK_STOPPING_FORCE_AFTER_MS
|
||||
&& claimForcedStop(key, Date.now())) {
|
||||
await sendForcedStop(sandbox, key, deadline);
|
||||
continue;
|
||||
}
|
||||
const pending = forcedStopInFlight(key);
|
||||
await Promise.race([
|
||||
new Promise(resolve => setTimeout(resolve, SANDBOX_TRANSITION_POLL_MS)),
|
||||
...(pending ? [pending] : []),
|
||||
]);
|
||||
await withLivenessTimeout("sandbox.refreshData", Math.min(readTimeoutMs, Math.max(1, deadline - Date.now())),
|
||||
() => sandbox.refreshData());
|
||||
}
|
||||
}, leaseStateAttributes(sandbox.state));
|
||||
}
|
||||
|
||||
async function ensureSandboxStarted(sandbox: Sandbox, timeoutSeconds: number, scope: SandboxScope): Promise<void> {
|
||||
if (sandbox.state === "started") return;
|
||||
sandboxHandleSessionStore.clear(scope);
|
||||
const deadline = Date.now() + timeoutSeconds * 1_000;
|
||||
const readTimeoutMs = Math.min(scope.config.livenessTimeoutMs, 30_000);
|
||||
for (;;) {
|
||||
await waitOutSandboxTransition(sandbox, deadline, readTimeoutMs);
|
||||
if (SANDBOX_TRANSITION_STATES.has(sandbox.state ?? "") || Date.now() >= deadline) {
|
||||
await waitOutSandboxTransition(sandbox, scope, deadline, readTimeoutMs);
|
||||
if (SANDBOX_TRANSITION_STATES.has(sandbox.state ?? "") || forcedStopInFlight(sandboxHandleCacheKey(scope))
|
||||
|| Date.now() >= deadline) {
|
||||
throw new Error(`Daytona sandbox ${sandbox.id} is still ${sandbox.state ?? "changing state"} at its start deadline`);
|
||||
}
|
||||
const remainingSeconds = Math.max(1, Math.ceil((deadline - Date.now()) / 1_000));
|
||||
try {
|
||||
await startSettledSandbox(sandbox, remainingSeconds, scope);
|
||||
endForcedStopEpisode(sandboxHandleCacheKey(scope));
|
||||
return;
|
||||
} catch (error) {
|
||||
if (!isSandboxStateChangeInProgress(error) || Date.now() + SANDBOX_TRANSITION_POLL_MS >= deadline) throw error;
|
||||
@@ -1858,6 +1935,7 @@ export function __resetDaytonaSandboxHandleCacheForTest(): void {
|
||||
sandboxHandleActivityGates.reset();
|
||||
sandboxHandleLeaseAdmissionStates.reset();
|
||||
sandboxHandleWritableDirs.reset();
|
||||
forcedSandboxStops.clear();
|
||||
sandboxHandleSessionStore.reset();
|
||||
runnerIngressGenerationStore.reset();
|
||||
retiredDaytonaDuplexChannelByRoute.clear();
|
||||
|
||||
@@ -349,6 +349,9 @@ describe("nested worker timing receipts",()=>{
|
||||
startTimeMs: now - 80, endTimeMs: now - 20, attributes: { "paperclip.sandbox.startup.state": "archived" } });
|
||||
expect(JSON.stringify(mockRecordSpan.mock.calls)).not.toContain("private-token");
|
||||
expect(clampProviderSpanAttributes({ "paperclip.sandbox.startup.state": "private-state" })).toEqual({});
|
||||
for (const state of ["starting", "stopping", "restoring", "archiving", "resizing"]) {
|
||||
expect(clampProviderSpanAttributes({ "paperclip.sandbox.startup.state": state })).toEqual({ "paperclip.sandbox.startup.state": state });
|
||||
}
|
||||
});
|
||||
|
||||
it("admits a workspace failure span with only a closed failure reason", () => {
|
||||
@@ -366,7 +369,7 @@ describe("nested worker timing receipts",()=>{
|
||||
it("admits every closed lease phase and rejects dynamic phase names", () => {
|
||||
mockRecordSpan.mockReset();
|
||||
const names = ["lease.wait_for_teardown", "lease.lookup", "lease.refresh", "lease.channel_cleanup",
|
||||
"lease.start", "lease.recover", "lease.verify", "lease.workspace", "lease.sentinel", "lease.shell",
|
||||
"lease.start", "lease.recover", "lease.wait_transition", "lease.force_stop", "lease.verify", "lease.workspace", "lease.sentinel", "lease.shell",
|
||||
"lease.create", "lease.expiry", "lease.cleanup", "lease.private-resource-id"];
|
||||
for (const name of names) recordWorkerProviderSpan({ name }, { traceparent: VALID_TRACEPARENT });
|
||||
expect(mockRecordSpan.mock.calls.map(call => call[0].name)).toEqual([
|
||||
|
||||
@@ -541,7 +541,7 @@ const KNOWN_PROVIDER_SPAN_NAMES: ReadonlySet<string> = new Set([
|
||||
"session.open",
|
||||
"session.close",
|
||||
"lease.wait_for_teardown", "lease.lookup", "lease.refresh", "lease.channel_cleanup",
|
||||
"lease.start", "lease.recover", "lease.verify", "lease.workspace", "lease.sentinel",
|
||||
"lease.start", "lease.recover", "lease.wait_transition", "lease.force_stop", "lease.verify", "lease.workspace", "lease.sentinel",
|
||||
"lease.shell", "lease.create", "lease.expiry", "lease.cleanup",
|
||||
"plugin.worker", "plugin.host.identity.read", "plugin.host.config.read", "plugin.host.progress.read", "plugin.host.progress.write",
|
||||
"plugin.workspace.prepare", "plugin.workspace.failure", "plugin.storage.queue", "plugin.storage.execute",
|
||||
@@ -591,6 +591,12 @@ const KNOWN_SPAN_OUTCOMES: ReadonlySet<string> = new Set(["ok", "skipped", "fail
|
||||
/** The closed value set for the `transfer.direction` attribute. */
|
||||
const KNOWN_TRANSFER_DIRECTIONS: ReadonlySet<string> = new Set(["inbound", "outbound"]);
|
||||
|
||||
// Settled states plus the transitions a start waits out; anything else is dropped.
|
||||
const PROVIDER_STARTUP_STATES: ReadonlySet<string> = new Set([
|
||||
"started", "stopped", "archived", "error", "other",
|
||||
"starting", "stopping", "restoring", "archiving", "resizing",
|
||||
]);
|
||||
|
||||
/**
|
||||
* Re-clamp the worker-sent attributes at the trust boundary. Drop every key that
|
||||
* is not on the allowlist. Re-map `provider` through `normalizeProviderFamily`,
|
||||
@@ -606,7 +612,7 @@ export function clampProviderSpanAttributes(
|
||||
for (const [key, value] of Object.entries(raw)) {
|
||||
if (!PROVIDER_SPAN_ATTR_ALLOWLIST.has(key)) continue;
|
||||
if (key === SPAN_ATTRS.state) {
|
||||
if (typeof value === "string" && ["started", "stopped", "archived", "error", "other"].includes(value)) clamped[key] = value;
|
||||
if (typeof value === "string" && PROVIDER_STARTUP_STATES.has(value)) clamped[key] = value;
|
||||
continue;
|
||||
}
|
||||
if (key === "plugin.operation") {
|
||||
|
||||
@@ -528,12 +528,12 @@
|
||||
"packages/shared/src/worktree-seed-source.test.ts": "852856b1b5b8f4d0d3dada6501886fe50a2c2dde",
|
||||
"packages/shared/src/worktree-seed-source.ts": "3d1a09bed2d969feabef3400ff84fd499c1408f4",
|
||||
"packages/shared/tsconfig.json": "5a24989cd30dcf3749ca7e60d4e83d6ddbb14714",
|
||||
"pnpm-lock.yaml": "851e65af7af027e3db99e759ece3d89e4a4700c2",
|
||||
"pnpm-lock.yaml": "6d825f9785f931f3bb08b0a6d2a741781f28f58a",
|
||||
"scripts/ensure-plugin-build-deps.mjs": "356e0e0f80e50890067027da3003485cf5d2ccbe",
|
||||
"tsconfig.base.json": "2d0e41f723add0c678936eb9ae1c60eab710f8e0",
|
||||
"tsconfig.json": "994ee490fb97be7cc0954f4dfd88cf0723290e6c"
|
||||
},
|
||||
"lockSha256": "2360d5e93876ce3cb6e810257d92d669ac2b109027b0e01af36f2048861648c0",
|
||||
"lockSha256": "b32083066ce89ba9a570a2927b3c4c8385294a5f04efcbae462db82b2e8c4455",
|
||||
"compilerPackageSha256": "3722b30210616a13a3213ded11575ba6b2dbab10c32a5ef67afca8513e27017e",
|
||||
"parserPackageSha256": "6d6a65b879cd7cd249a4367c0b57cce00e78c728de45056ac2a5ca1d1f528d1a",
|
||||
"parserEntrySha256": "6969920ae0610df927b6b3e675d1309372c268e36d391652af8e3e0183cbe8f8",
|
||||
|
||||
Reference in new issue
Block a user