mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On Daytona, a follow-up task resumes the agent's retained sandbox before the agent starts > - The host stops an idle sandbox gracefully. A process that ignores SIGTERM can leave Daytona in `stopping` for many minutes > - Resume waited for that state change until its 300 s start deadline. It never escalated, so each attempt failed after five minutes, and the bounded retries did the same > - On staging, a user comment waited about 21 minutes before the agent started. Two resume attempts each used 299.7 s in the plugin dispatch > - This pull request forces a stuck stop once, then starts the sandbox within the same deadline. It also traces the wait with the observed state > - The benefit is that a stuck stop costs about 30 s instead of failing the run, and the next case is diagnosable from the trace ## Linked Issues or Issue Description **What happened?** A user commented on a task that used a retained Daytona sandbox. The first run and its first retry each failed after about 5 min 20 s with `reusable_sandbox_resume_failed (cause=provider_error)`. The Honeycomb trace shows `execution.plugin_dispatch` at 299.7 s in both. Inside it there is only `lease.lookup` (44 ms) and `lease.channel_cleanup`, and no `lease.start`. So the whole budget went to `waitOutSandboxTransition`, which polled a sandbox that stayed in a transition state. The later attempt, after a control-plane restart, found the sandbox `stopped`. Its `lease.start` took 1.3 s. A transition that ends in `stopped` is a stop. **Expected behavior** A sandbox that stays `stopping` for longer than the host's own graceful-stop budget is forced to stop, then starts. The run does not fail on the start deadline. **Steps to reproduce** 1. Retain a Daytona sandbox whose graceful stop does not finish (Daytona reports `stopping`). 2. Resume or realize the lease. 3. Before this change, the call polls until the 300 s deadline and fails with "is still stopping at its start deadline". ## What Changed - `packages/plugins/sandbox-providers/daytona/src/plugin.ts`: `waitOutSandboxTransition` tracks how long the sandbox stays `stopping`. After `STUCK_STOPPING_FORCE_AFTER_MS` (30 s) it calls `sandbox.stop(seconds, true)` once, then keeps polling. A refused force stop keeps the old wait within the same deadline. - The wait is a `lease.wait_transition` span. The force stop is a `lease.force_stop` child span. - `lease-tracing.ts`: the closed `paperclip.sandbox.startup.state` label also records `starting`, `stopping`, `restoring`, `archiving` and `resizing`. Before, these were all `other`. - `server/src/services/plugin-host-services.ts`: the host admits the two new span names and the new closed state values. - `doc/observability.md`: documents the new spans and states. ## Verification - Red/green: with the plugin change stashed, the three new stuck-`stopping` tests (realize, refused force stop, resume) fail by hanging to the deadline. With the change they pass. The "settles on its own" test confirms that no force stop happens for a normal stop. - `vitest run` in `packages/plugins/sandbox-providers/daytona`: 14 files, 618 tests passed (8 skipped). `tsc --noEmit` passed. - `server/src/__tests__/plugin-host-services-span.test.ts`: red before the allowlist change, 25/25 after. - `pnpm verify` exited 2 (the doc is a global CI input), so CI supplies the broad coverage. - Not verified on a live Daytona sandbox. Hosted acceptance needs the change deployed to staging. ## Risks - The force stop sends SIGKILL to processes in a sandbox that is already stopping and is about to start. The disk persists across a stop, so the retained workspace is kept. A graceful stop that is still running past 30 s is killed. The host's own graceful stop budget is 30 s, so the host had already given up on it. - Daytona can refuse a stop while a state change is in progress. Then the code waits as before, so the result is no worse than today. - The root cause is inferred from the trace (a transition that ends in `stopped`). The new state label confirms or refutes it the next time it happens. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, browser, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
427 lines
18 KiB
TypeScript
427 lines
18 KiB
TypeScript
import { beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { createHostClientHandlers } from "../../../packages/plugins/sdk/src/host-client-factory.js";
|
|
import type { WorkerHostCallContext } from "../../../packages/plugins/sdk/src/protocol.js";
|
|
import { SANDBOX_STARTUP_SPAN_ATTRS as A } from "@paperclipai/adapter-utils/acpx-engine/startup-timing";
|
|
import {
|
|
buildHostServices,
|
|
clampProviderSpanAttributes,
|
|
parseTraceparent,
|
|
recordWorkerProviderSpan,
|
|
} from "../services/plugin-host-services.js";
|
|
|
|
// Capture every span the host trust boundary hands to the real tracer.
|
|
const mockRecordSpan = vi.hoisted(() => vi.fn());
|
|
|
|
vi.mock("../instrumentation.js", () => ({
|
|
recordProviderPluginSpan: mockRecordSpan,
|
|
traceparentFromContextToken: () => undefined,
|
|
}));
|
|
|
|
function createEventBusStub() {
|
|
return {
|
|
forPlugin() {
|
|
return { emit: vi.fn(), subscribe: vi.fn() };
|
|
},
|
|
} as never;
|
|
}
|
|
|
|
// A well-formed W3C traceparent (the host mints it; the handler validates it).
|
|
const VALID_TRACEPARENT = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01";
|
|
|
|
function servicesFor() {
|
|
return buildHostServices({} as never, "plugin-record-id", "daytona", createEventBusStub());
|
|
}
|
|
|
|
function handlersFor(capabilities: readonly string[]) {
|
|
return createHostClientHandlers({
|
|
pluginId: "daytona",
|
|
capabilities: capabilities as never,
|
|
services: servicesFor(),
|
|
});
|
|
}
|
|
|
|
describe("plugin provider span host handler", () => {
|
|
beforeEach(() => {
|
|
mockRecordSpan.mockReset();
|
|
});
|
|
|
|
it("records a span with the clamped name and the allowlisted attributes", async () => {
|
|
const services = servicesFor();
|
|
await services.tracer.record(
|
|
{
|
|
name: "pack",
|
|
attributes: {
|
|
[A.provider]: "daytona",
|
|
[A.packWallMs]: 12,
|
|
},
|
|
},
|
|
{ traceparent: VALID_TRACEPARENT } as WorkerHostCallContext,
|
|
);
|
|
|
|
expect(mockRecordSpan).toHaveBeenCalledTimes(1);
|
|
const call = mockRecordSpan.mock.calls[0]![0] as {
|
|
name: string;
|
|
parent: { traceId: string; spanId: string; traceFlags: number };
|
|
attributes: Record<string, unknown>;
|
|
};
|
|
expect(call.name).toBe("sandbox.daytona.pack");
|
|
expect(call.parent).toEqual({
|
|
traceId: "0af7651916cd43dd8448eb211c80319c",
|
|
spanId: "b7ad6b7169203331",
|
|
traceFlags: 1,
|
|
});
|
|
expect(call.attributes[A.provider]).toBe("daytona");
|
|
expect(call.attributes[A.packWallMs]).toBe(12);
|
|
});
|
|
|
|
it("drops every forbidden attribute before the span reaches the tracer", async () => {
|
|
const services = servicesFor();
|
|
await services.tracer.record(
|
|
{
|
|
name: "transfer",
|
|
attributes: {
|
|
[A.transferGuardCount]: 2,
|
|
// Forbidden fields that must never ride a span.
|
|
[A.execCommand]: "bash",
|
|
command: "rm -rf /",
|
|
args: "--force",
|
|
stdout: "secret output",
|
|
stderr: "secret error",
|
|
path: "/etc/passwd",
|
|
extra: "leak",
|
|
},
|
|
},
|
|
{ traceparent: VALID_TRACEPARENT } as WorkerHostCallContext,
|
|
);
|
|
|
|
expect(mockRecordSpan).toHaveBeenCalledTimes(1);
|
|
const attributes = (mockRecordSpan.mock.calls[0]![0] as { attributes: Record<string, unknown> })
|
|
.attributes;
|
|
// Only the allowlisted key survives; every forbidden key is dropped.
|
|
expect(attributes).toEqual({ [A.transferGuardCount]: 2 });
|
|
for (const forbidden of [A.execCommand, "command", "args", "stdout", "stderr", "path", "extra"]) {
|
|
expect(attributes).not.toHaveProperty(forbidden);
|
|
}
|
|
});
|
|
|
|
it("drops a status message (it could carry standard-stream text)", async () => {
|
|
const services = servicesFor();
|
|
await services.tracer.record(
|
|
{ name: "transfer", status: { code: 2, message: "secret error text" } },
|
|
{ traceparent: VALID_TRACEPARENT } as WorkerHostCallContext,
|
|
);
|
|
const call = mockRecordSpan.mock.calls[0]![0] as { status?: { code: number; message?: string } };
|
|
expect(call.status).toEqual({ code: 2 });
|
|
expect(call.status).not.toHaveProperty("message");
|
|
});
|
|
|
|
it("clamps an unknown span name to sandbox.daytona.other", async () => {
|
|
const services = servicesFor();
|
|
await services.tracer.record(
|
|
{ name: "rm -rf / --no-preserve-root" },
|
|
{ traceparent: VALID_TRACEPARENT } as WorkerHostCallContext,
|
|
);
|
|
expect((mockRecordSpan.mock.calls[0]![0] as { name: string }).name).toBe(
|
|
"sandbox.daytona.other",
|
|
);
|
|
});
|
|
|
|
it("admits each per-round-trip span name to sandbox.daytona.<name>", async () => {
|
|
const services = servicesFor();
|
|
for (const name of [
|
|
"ensureDirectory",
|
|
"checkSymlinkEscape",
|
|
"promote",
|
|
"extractTarball",
|
|
"postUploadCommand",
|
|
]) {
|
|
await services.tracer.record(
|
|
{ name },
|
|
{ traceparent: VALID_TRACEPARENT } as WorkerHostCallContext,
|
|
);
|
|
}
|
|
const recorded = mockRecordSpan.mock.calls.map((c) => (c[0] as { name: string }).name);
|
|
expect(recorded).toEqual([
|
|
"sandbox.daytona.ensureDirectory",
|
|
"sandbox.daytona.checkSymlinkEscape",
|
|
"sandbox.daytona.promote",
|
|
"sandbox.daytona.extractTarball",
|
|
"sandbox.daytona.postUploadCommand",
|
|
]);
|
|
});
|
|
|
|
it("admits the session open and close span names", async () => {
|
|
const services = servicesFor();
|
|
for (const name of ["session.open", "session.close"]) {
|
|
await services.tracer.record(
|
|
{ name },
|
|
{ traceparent: VALID_TRACEPARENT } as WorkerHostCallContext,
|
|
);
|
|
}
|
|
const recorded = mockRecordSpan.mock.calls.map((c) => (c[0] as { name: string }).name);
|
|
expect(recorded).toEqual([
|
|
"sandbox.daytona.session.open",
|
|
"sandbox.daytona.session.close",
|
|
]);
|
|
});
|
|
|
|
it("forwards a valid start-time and end-time pair to the recorder", async () => {
|
|
const services = servicesFor();
|
|
const startTimeMs = Date.now() - 4500;
|
|
const endTimeMs = startTimeMs + 4500;
|
|
await services.tracer.record(
|
|
{ name: "ensureDirectory", startTimeMs, endTimeMs },
|
|
{ traceparent: VALID_TRACEPARENT } as WorkerHostCallContext,
|
|
);
|
|
const call = mockRecordSpan.mock.calls[0]![0] as {
|
|
startTimeMs?: number;
|
|
endTimeMs?: number;
|
|
};
|
|
expect(call.startTimeMs).toBe(startTimeMs);
|
|
expect(call.endTimeMs).toBe(endTimeMs);
|
|
});
|
|
|
|
it("accepts a pair whose end is a small skew ahead of the host clock", async () => {
|
|
const services = servicesFor();
|
|
// The worker clock leads the host clock by a few seconds. This small skew
|
|
// is within the allowed bound, so the host keeps the native width.
|
|
const startTimeMs = Date.now() + 5000;
|
|
const endTimeMs = startTimeMs + 1000;
|
|
await services.tracer.record(
|
|
{ name: "ensureDirectory", startTimeMs, endTimeMs },
|
|
{ traceparent: VALID_TRACEPARENT } as WorkerHostCallContext,
|
|
);
|
|
const call = mockRecordSpan.mock.calls[0]![0] as {
|
|
startTimeMs?: number;
|
|
endTimeMs?: number;
|
|
};
|
|
expect(call.startTimeMs).toBe(startTimeMs);
|
|
expect(call.endTimeMs).toBe(endTimeMs);
|
|
});
|
|
|
|
it("drops an invalid timestamp pair so the synchronous path runs", async () => {
|
|
const services = servicesFor();
|
|
const now = Date.now();
|
|
const invalidPairs: Array<{ startTimeMs?: unknown; endTimeMs?: unknown; why: string }> = [
|
|
{ startTimeMs: now, endTimeMs: now - 1000, why: "reversed order" },
|
|
{ startTimeMs: Number.NaN, endTimeMs: now, why: "non-finite start" },
|
|
{ startTimeMs: now, endTimeMs: Number.POSITIVE_INFINITY, why: "non-finite end" },
|
|
{ startTimeMs: now, endTimeMs: now + 11 * 60 * 1000, why: "over-ceiling duration" },
|
|
{ startTimeMs: now - 2 * 60 * 60 * 1000, endTimeMs: now - 2 * 60 * 60 * 1000 + 10, why: "over-age start" },
|
|
{ startTimeMs: now, endTimeMs: now + 2 * 60 * 1000, why: "end far in the future" },
|
|
{ startTimeMs: now + 5 * 60 * 1000, endTimeMs: now + 5 * 60 * 1000 + 10, why: "start and end in the future" },
|
|
];
|
|
for (const pair of invalidPairs) {
|
|
mockRecordSpan.mockReset();
|
|
await services.tracer.record(
|
|
{ name: "ensureDirectory", startTimeMs: pair.startTimeMs, endTimeMs: pair.endTimeMs } as never,
|
|
{ traceparent: VALID_TRACEPARENT } as WorkerHostCallContext,
|
|
);
|
|
// The span still records (the synchronous path), but without a timestamp.
|
|
const call = mockRecordSpan.mock.calls[0]![0] as {
|
|
startTimeMs?: number;
|
|
endTimeMs?: number;
|
|
};
|
|
expect(call.startTimeMs, pair.why).toBeUndefined();
|
|
expect(call.endTimeMs, pair.why).toBeUndefined();
|
|
}
|
|
});
|
|
|
|
it("records the synchronous path when the timestamp pair is absent", async () => {
|
|
const services = servicesFor();
|
|
await services.tracer.record(
|
|
{ name: "pack" },
|
|
{ traceparent: VALID_TRACEPARENT } as WorkerHostCallContext,
|
|
);
|
|
const call = mockRecordSpan.mock.calls[0]![0] as {
|
|
startTimeMs?: number;
|
|
endTimeMs?: number;
|
|
};
|
|
expect(call.startTimeMs).toBeUndefined();
|
|
expect(call.endTimeMs).toBeUndefined();
|
|
});
|
|
|
|
it("rejects a malformed traceparent — no span is recorded", async () => {
|
|
const services = servicesFor();
|
|
for (const bad of [
|
|
undefined,
|
|
"not-a-traceparent",
|
|
"00-xyz-b7ad6b7169203331-01",
|
|
"00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331", // missing flags
|
|
"00-00000000000000000000000000000000-b7ad6b7169203331-01", // all-zero trace id
|
|
]) {
|
|
await services.tracer.record(
|
|
{ name: "pack" },
|
|
{ traceparent: bad } as WorkerHostCallContext,
|
|
);
|
|
}
|
|
expect(mockRecordSpan).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("rejects a span from a plugin that lacks the environment-driver capability", async () => {
|
|
const handlers = handlersFor([]);
|
|
await expect(
|
|
handlers["span.record"](
|
|
{ name: "pack" },
|
|
{ traceparent: VALID_TRACEPARENT } as WorkerHostCallContext,
|
|
),
|
|
).rejects.toThrow(/capabilit/i);
|
|
expect(mockRecordSpan).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("admits a span from a plugin that holds the environment-driver capability", async () => {
|
|
const handlers = handlersFor(["environment.drivers.register"]);
|
|
await handlers["span.record"](
|
|
{ name: "pack", attributes: { [A.provider]: "daytona" } },
|
|
{ traceparent: VALID_TRACEPARENT } as WorkerHostCallContext,
|
|
);
|
|
expect(mockRecordSpan).toHaveBeenCalledTimes(1);
|
|
});
|
|
});
|
|
|
|
describe("parseTraceparent", () => {
|
|
it("accepts a well-formed traceparent and returns the parts", () => {
|
|
expect(parseTraceparent(VALID_TRACEPARENT)).toEqual({
|
|
traceId: "0af7651916cd43dd8448eb211c80319c",
|
|
spanId: "b7ad6b7169203331",
|
|
traceFlags: 1,
|
|
});
|
|
});
|
|
|
|
it("rejects malformed, all-zero, and forbidden-version traceparents", () => {
|
|
expect(parseTraceparent(undefined)).toBeNull();
|
|
expect(parseTraceparent("garbage")).toBeNull();
|
|
expect(parseTraceparent("00-0af7651916cd43dd8448eb211c80319c-0000000000000000-01")).toBeNull();
|
|
expect(parseTraceparent("ff-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01")).toBeNull();
|
|
});
|
|
});
|
|
|
|
describe("clampProviderSpanAttributes", () => {
|
|
it("keeps only allowlisted keys and normalizes the provider family", () => {
|
|
expect(
|
|
clampProviderSpanAttributes({
|
|
[A.provider]: "some-operator-key",
|
|
[A.packWallMs]: 7,
|
|
[A.transferWallMs]: Number.NaN,
|
|
[A.execCommand]: "bash",
|
|
}),
|
|
).toEqual({
|
|
// An unknown provider key maps to `plugin`, never the raw key.
|
|
[A.provider]: "plugin",
|
|
[A.packWallMs]: 7,
|
|
// A non-finite number yields no attribute; `exec.command` is not allowed.
|
|
});
|
|
});
|
|
|
|
it("keeps the transfer direction for each closed-set value", () => {
|
|
expect(clampProviderSpanAttributes({ [A.transferDirection]: "inbound" })).toEqual({
|
|
[A.transferDirection]: "inbound",
|
|
});
|
|
expect(clampProviderSpanAttributes({ [A.transferDirection]: "outbound" })).toEqual({
|
|
[A.transferDirection]: "outbound",
|
|
});
|
|
});
|
|
|
|
it("drops a transfer direction outside the closed set or of the wrong type", () => {
|
|
// A free-form string, an empty string, and a non-string all yield no
|
|
// attribute, so the direction stays bounded and low-cardinality.
|
|
expect(clampProviderSpanAttributes({ [A.transferDirection]: "sideways" })).toEqual({});
|
|
expect(clampProviderSpanAttributes({ [A.transferDirection]: "" })).toEqual({});
|
|
expect(clampProviderSpanAttributes({ [A.transferDirection]: 1 })).toEqual({});
|
|
});
|
|
});
|
|
|
|
describe("nested worker timing receipts",()=>{
|
|
it("keeps lease verification beneath its worker and clamps provider state", () => {
|
|
mockRecordSpan.mockReset();
|
|
const rootParent = { traceId: "0af7651916cd43dd8448eb211c80319c", spanId: "1111111111111111", traceFlags: 1 };
|
|
const verifyParent = { ...rootParent, spanId: "2222222222222222" };
|
|
mockRecordSpan.mockReturnValueOnce(rootParent).mockReturnValueOnce(verifyParent);
|
|
const now = Date.now();
|
|
recordWorkerProviderSpan({ name: "plugin.worker", startTimeMs: now - 100, endTimeMs: now, children: [
|
|
{ name: "lease.verify", startTimeMs: now - 90, endTimeMs: now - 10, children: [
|
|
{ name: "lease.sentinel", startTimeMs: now - 80, endTimeMs: now - 20,
|
|
attributes: { "paperclip.sandbox.startup.state": "archived", token: "private-token" } },
|
|
] },
|
|
] }, { traceparent: VALID_TRACEPARENT });
|
|
expect(mockRecordSpan.mock.calls[1][0]).toMatchObject({ name: "sandbox.daytona.lease.verify", parent: rootParent });
|
|
expect(mockRecordSpan.mock.calls[2][0]).toMatchObject({ name: "sandbox.daytona.lease.sentinel", parent: verifyParent,
|
|
startTimeMs: now - 80, endTimeMs: now - 20, attributes: { "paperclip.sandbox.startup.state": "archived" } });
|
|
expect(JSON.stringify(mockRecordSpan.mock.calls)).not.toContain("private-token");
|
|
expect(clampProviderSpanAttributes({ "paperclip.sandbox.startup.state": "private-state" })).toEqual({});
|
|
for (const state of ["starting", "stopping", "restoring", "archiving", "resizing"]) {
|
|
expect(clampProviderSpanAttributes({ "paperclip.sandbox.startup.state": state })).toEqual({ "paperclip.sandbox.startup.state": state });
|
|
}
|
|
});
|
|
|
|
it("admits a workspace failure span with only a closed failure reason", () => {
|
|
mockRecordSpan.mockReset();
|
|
const now = Date.now();
|
|
recordWorkerProviderSpan({ name: "plugin.workspace.failure", startTimeMs: now - 1, endTimeMs: now, status: { code: 2 },
|
|
attributes: { "plugin.failure_reason": "storage_grant_unavailable", message: "Project mount grants unavailable" } }, { traceparent: VALID_TRACEPARENT });
|
|
expect(mockRecordSpan.mock.calls[0][0]).toMatchObject({ name: "plugin.workspace.failure", attributes: { "plugin.failure_reason": "storage_grant_unavailable" } });
|
|
expect(JSON.stringify(mockRecordSpan.mock.calls)).not.toContain("mount grants");
|
|
expect(clampProviderSpanAttributes({ "plugin.failure_reason": "unknown" })).toEqual({ "plugin.failure_reason": "unknown" });
|
|
expect(clampProviderSpanAttributes({ "plugin.failure_reason": "token=private" })).toEqual({});
|
|
expect(clampProviderSpanAttributes({ "plugin.failure_reason": 1 })).toEqual({});
|
|
});
|
|
|
|
it("admits every closed lease phase and rejects dynamic phase names", () => {
|
|
mockRecordSpan.mockReset();
|
|
const names = ["lease.wait_for_teardown", "lease.lookup", "lease.refresh", "lease.channel_cleanup",
|
|
"lease.start", "lease.recover", "lease.wait_transition", "lease.force_stop", "lease.verify", "lease.workspace", "lease.sentinel", "lease.shell",
|
|
"lease.create", "lease.expiry", "lease.cleanup", "lease.private-resource-id"];
|
|
for (const name of names) recordWorkerProviderSpan({ name }, { traceparent: VALID_TRACEPARENT });
|
|
expect(mockRecordSpan.mock.calls.map(call => call[0].name)).toEqual([
|
|
...names.slice(0, -1).map(name => `sandbox.daytona.${name}`), "sandbox.daytona.other",
|
|
]);
|
|
});
|
|
|
|
it("uses host-generated parents, clamps content and bounds child clocks",()=>{
|
|
mockRecordSpan.mockReset();
|
|
const childParent={traceId:"0af7651916cd43dd8448eb211c80319c",spanId:"1111111111111111",traceFlags:1};
|
|
mockRecordSpan.mockReturnValue(childParent);
|
|
const now=Date.now();
|
|
recordWorkerProviderSpan({name:"plugin.worker",startTimeMs:now-100,endTimeMs:now,children:[
|
|
{name:"plugin.remote.operation",startTimeMs:now-90,endTimeMs:now-10,attributes:{"plugin.operation":"script.settings",secret:"private"}},
|
|
{name:"plugin.runtime.verify",startTimeMs:now-200,endTimeMs:now},
|
|
]},{traceparent:VALID_TRACEPARENT} as WorkerHostCallContext);
|
|
expect(mockRecordSpan).toHaveBeenCalledTimes(2);
|
|
expect(mockRecordSpan.mock.calls[1][0]).toMatchObject({parent:childParent,attributes:{"plugin.operation":"script.settings"}});
|
|
expect(JSON.stringify(mockRecordSpan.mock.calls)).not.toContain("private");
|
|
});
|
|
});
|
|
|
|
it("allows a workspace provider timing receipt without environment-driver authority",async()=>{
|
|
mockRecordSpan.mockReset();
|
|
await handlersFor(["workspaces.manage"])["span.record"]({name:"plugin.worker"},{traceparent:VALID_TRACEPARENT} as WorkerHostCallContext);
|
|
expect(mockRecordSpan).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("retains short children of a multi-hour worker invocation and shares its aggregate budget",()=>{
|
|
mockRecordSpan.mockReset();mockRecordSpan.mockReturnValue({traceId:"0af7651916cd43dd8448eb211c80319c",spanId:"1111111111111111",traceFlags:1});
|
|
const now=Date.now(),context={traceparent:VALID_TRACEPARENT,providerSpanBudget:{remaining:3}};
|
|
recordWorkerProviderSpan({name:"plugin.worker",startTimeMs:now-7200000,endTimeMs:now,children:[
|
|
{name:"pack",startTimeMs:now-7199990,endTimeMs:now-7199980}
|
|
]},context);
|
|
recordWorkerProviderSpan({name:"pack"},context);recordWorkerProviderSpan({name:"pack"},context);
|
|
expect(mockRecordSpan).toHaveBeenCalledTimes(3);
|
|
expect(mockRecordSpan.mock.calls[1][0]).toMatchObject({name:"sandbox.daytona.pack",startTimeMs:now-7199990});
|
|
});
|
|
|
|
|
|
it("links bounded earlier remote lifecycles to their host-owned observer",()=>{
|
|
mockRecordSpan.mockReset();mockRecordSpan.mockReturnValue({traceId:"0af7651916cd43dd8448eb211c80319c",spanId:"1111111111111111",traceFlags:1});
|
|
const now=Date.now();
|
|
recordWorkerProviderSpan({name:"plugin.worker",startTimeMs:now-10,endTimeMs:now,children:[
|
|
{name:"plugin.remote.lifecycle",relationship:"linked",startTimeMs:now-1000,endTimeMs:now-100,children:[
|
|
{name:"plugin.remote.operation",startTimeMs:now-900,endTimeMs:now-200}
|
|
]},
|
|
{name:"plugin.remote.operation",relationship:"linked",startTimeMs:now-1000,endTimeMs:now-100},
|
|
{name:"plugin.remote.lifecycle",relationship:"linked",startTimeMs:now-700000,endTimeMs:now-100},
|
|
]},{traceparent:VALID_TRACEPARENT});
|
|
expect(mockRecordSpan).toHaveBeenCalledTimes(3);
|
|
expect(mockRecordSpan.mock.calls[1][0]).toMatchObject({name:"plugin.remote.lifecycle",linked:true,startTimeMs:now-1000});
|
|
expect(mockRecordSpan.mock.calls[2][0].linked).toBeUndefined();
|
|
});
|