Files
PaperClipAI/server/src/__tests__/environment-execution-target.test.ts
T
Nicky LeachandClaude Opus 5.5 11ac90e31f fix(build): typecheck server/src/__tests__ and fix every type error (#18739)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server has about 1,100 test and helper files in
`server/src/__tests__`
> - `server/tsconfig.json` excludes that directory, Vitest does not
check types, and no script runs the tsconfig in that directory
> - So a test can call a function with the wrong arguments and still
pass, and no tool reports it
> - This gap hid 2,168 real type errors, and some of them hid tests that
did not test what they claim
> - This pull request fixes all of these errors and adds the directory
to the server layers from paperclipai/paperclip#18737
> - The benefit is that a type error in a server test now fails the
typecheck, with no baseline and no backlog

## Linked Issues or Issue Description

Refs paperclipai/paperclip#18737, which is merged and added the server
layers. Refs paperclipai/paperclip#18738, which is merged and checks the
test projects outside `pnpm -r`, so it is not a duplicate. Refs
paperclipai/paperclip#18137, which records that this directory has no
check.

**What happened?**

No tool checked the types of 1,074 of the 1,101 files in
`server/src/__tests__`. I searched `package.json`, the workflows, and
all tsconfig files. 27 files are in the program of another tsconfig, and
these 27 have 0 errors. 11 of the 27 are in
`helpers/maintenance/tsconfig.json`. No script ran that project when I
measured this. paperclipai/paperclip#18738 now runs it.

**Expected behavior**

The server typecheck checks these files, and a type error fails the
check.

**Steps to reproduce**

1. On the base commit, run `tsc -p server/src/__tests__/tsconfig.json
--noEmit`. It reports 6,536 errors and needs 12.3 GiB.
2. Add a type error to any file in `server/src/__tests__`.
3. Run `pnpm --filter @paperclipai/server typecheck`. It passes.

**Paperclip version or commit**

I measured the error breakdown below on commit `8dfd632452`, with
TypeScript 7.0.2. The branch now includes master up to
paperclipai/paperclip#18865.

**Deployment mode**

Not applicable. This issue applies to development and CI. No production
source file in the server changes.

### Error breakdown

Each row adds one change to the row above.

| Step | Total errors | Change | Cause |
| --- | --- | --- | --- |
| Config as committed | 6,536 | | |
| Add `src/types/*.d.ts` | 4,447 | -2,089 | Config. The Express request
types were missing. |
| Set `rootDir` to the repository root | 2,436 | -2,011 | Config. |
| Set `allowImportingTsExtensions` | 2,181 | -255 | Config. Vitest
accepts `.ts` imports. |
| Add `ES2024.Promise` to `lib` | 2,138 | -43 | Config. Node 24 has
`Promise.withResolvers`. |
| Set `allowJs` for `.mjs` imports | 2,148 | +10 | Config. It removes 8
`TS7016` errors and shows new real errors. |
| Set `declaration` to false | 2,142 | -6 | Config. Tests emit nothing.
|
| Count only files in `server/src` | 2,058 | -84 | Other packages: 53
Daytona, 29 Runner test, 2 UI. |

So 4,478 of the 6,536 errors (69%) came from config or scope, and 2,058
are real type errors in 314 files. The 29 Runner test errors are also
real, which gives 2,087 errors. Tests that master added after this
measurement had 81 more errors, so this pull request fixes 2,168 errors.

This table classifies the first 2,058 errors.

| Kind of real error | Errors |
| --- | --- |
| Wrong argument or assignment type | 1,023 |
| Stale or partial fixture objects | 525 |
| Property that the type does not have | 317 |
| Missing narrowing of null or unknown | 101 |
| Implicit any | 60 |
| Missing name or module | 16 |
| Other | 16 |

### What the type errors hid

Most errors were stale fixtures and loose mocks. These errors hid real
test problems, and this pull request fixes them:

- **A test that never ran.** `instrumentation.test.ts` required
`sdk-trace-base`, which is not installed. So 3 span tests were skipped.
They now run and pass.
- **Assertions that could not fail.**
- `issue-thread-interactions-service` used a company ID as an issue ID.
- `status-card-evaluation-work` compared `[]` with `[]` in a rollback
check.
- `slack-conversation-lifecycle` read `item.issueId`, which does not
exist.
  - `email-channels` used a permission key that does not exist.
- `heartbeat-project-env` imported `LOW_TRUST_REVIEW_PRESET`, which is
not exported. The value was undefined.
- **Postgres processes that did not stop.** 11 test files called `stop`
on the test database helper. That method does not exist, and the helper
method is `cleanup`.
- **Arguments that the service ignored.**
- `methodKey` is now `connectionMethodKey` in the Daytona, Unikraft,
1Password, and Railway tests.
- 108 calls passed `"owner-user"` to a parameter that expects an options
object.
- 9 chat wizard fixtures passed `managementContext` as `source`, so they
ran `existing_app` and not `manifest`.
  - 2 tests passed a `{ userId }` actor that the function ignores.
- Inserts into `environments` passed a `companyId` column that the table
does not have.
- **Problems in tests that master added during this work.**
- `sandbox-creation-operations` read `database.connectionString` when an
external test database was set. In that mode `database` is undefined, so
the fixture would stop with a TypeError. It now uses the external URL.
- `idle-completed-history` set a Dot operation to `dispatched`. That is
a Muse status, and the Dot type does not have it. The fixture now uses
`admitted`, which is a real in-flight Dot status.

### Possible product problems, not changed

These may be real problems in production code or types. I did not change
them, because this pull request does not change production behavior:

- `decisions-service` `service.create` does not validate the lenient
`cancel_issue_tree` input.
- `createConnection` ignores `input.connectionPurpose`.
- The `createConnection` route schema default stores `{}` for
`transportConfig`.
- `prepareManagedAiRuntime` accepted the invalid value `mode:
"connection"`.
- `createBinding` accepts invalid `projectionClass` strings outside
class 3.
- `ensureManagedProjectWorkspace` forwards an untyped `repoRef` by
spread.
- `routineVariableSchema.label` is optional, but `RoutineVariable.label`
is required.
- `ToolRunContext.projectId` may be too strict.
- `withdrawInteraction` reads `issue.status`, which its type omits.
- `issueService.list` and `reconcileNativeFinalizations` omit fields
that they return at runtime.
- `createGitHubExternalObjectProvider` handles `tokenProvider: null`,
but its type forbids this value.

## What Changed

- Fixed the 2,168 type errors. The diff changes 322 test and helper
files.
- Corrected `server/src/__tests__/tsconfig.json`. Editors and the
typecheck now read the same options and report the same errors.
- Added three `test-tree` layers to
`server/scripts/typecheck-layers.mjs`. They read the server layers as
declarations, and any error fails them. A hash of the path selects the
shard, so a new file does not move other files.
- Added `server/scripts/typecheck-stubs/daytona-sdk.d.ts`. Server tests
reach Daytona plugin source, and the root install does not have
`@daytonaio/sdk`.
- Changed two imports in `ui/src/lib/issue-execution-policy.ts` to use
`.js` extensions, because server tests import this file.
- Made `declarationsConfig` and `--declarations-only` skip the
`test-tree` layers, because these layers emit nothing. Other projects
read only the source layers. Added a test for this.
- Updated `doc/DEVELOPING.md`.

## Verification

Run these commands:

```sh
pnpm --filter @paperclipai/server typecheck
pnpm exec vitest run server/scripts/typecheck-layers.test.mjs
```

Results on macOS (arm64), Node 24, 16 cores. Each check ran after the
merge of paperclipai/paperclip#18738 unless the row says otherwise:

| Check | Commit | Result |
| --- | --- | --- |
| Server typecheck, `PAPERCLIP_TYPECHECK_LOW_MEMORY=1`, all 7 layers |
`6e45a08573` | Exit 0, 0 errors, 62 seconds, peak 3.52 GiB |
| `node scripts/typecheck-test-projects.mjs --low-memory` | `a3306bd13e`
| Exit 0, 0 errors, 42 seconds, peak 3.25 GiB |
| The same script with `--reuse-declarations` | `1b878aabc0` | Exit 0, 0
errors |
| `typecheck-layers.test.mjs` and `live-recovery-work.test.ts` |
`01db5634e4` | 90 passed. These include real compiler runs for a pass, a
type error, and an invalid tsconfig. |
| `tsconfig.declarations.json` after a full server typecheck |
`0fb1e4fb0a` | It references only `below`, `cycle`, and `above`. No
generated config names a `tsconfig.layers.json` file. |
| Owned runtime test in `idle-completed-history` | `e8f2f448ac` |
Passed, with the fix that master later replaced |
| `idle-incoming-wake`, `plugin-idle-manager`, and the changed
`chat-channels` test | `6e45a08573` | 30 passed |
| Runner `native-execution.test.ts` | `eab4dcc1f4` | 114 passed |
| The 19 server test files that changed since `eab4dcc1f4`, and
`typecheck-layers.test.mjs` | `259737b700` | 4,275 of 4,277 passed. The
2 failures are in `chat-channels` and are in the list of 7 below. |

The peak is the largest single process, from `/usr/bin/time -l`. Peaks
for the server typecheck were between 3.42 GiB and 3.56 GiB across 8
runs. Peaks for the test projects were 3.25 GiB and 3.37 GiB across 2
runs. I did not measure a cgroup.

**Runtime check of the fixes.** Types do not prove that a test still
works. So I ran all 309 changed server test files, 11,929 tests, before
the rebase onto paperclipai/paperclip#18737:

| Result | Tests | Detail |
| --- | --- | --- |
| Passed | 11,913 | Includes the 2 tests and the 54 `dot-runner` tests
that passed in a later run, as described below. |
| Failed | 7 | The same 7 tests also fail on the commit before the
fixes. |
| Skipped | 9 | These tests have their own skip conditions: `sentry` 5,
and 1 each in `bundled-plugin-registration`, `customer-success-cloud`,
`onepassword-connection`, and `workspace-runtime-https-live-exercise`. |

The 7 tests that fail with and without this change, on my machine:

- 3 tests in `chat-channels` and `email-channels` stop with `ENOENT` on
a `SKILL.md` file. The path is in the parent checkout, above the git
worktree that I used, so I expect that the worktree is the cause. I did
not prove this.
- `workspace-instance-cleanup` compares `/private/var/...` with
`/var/...`. This is a macOS symlink difference.
- 2 tests in `git-reproducible-managed-checkouts` get
`unknown_base_git_files` and expect `unpushed_managed_checkout`. I did
not find the cause.
- 1 more test in `chat-channels`, `materializes direct external-chat
finals once`, gets `undefined`. I did not find the cause.

CI is the check for these 7 tests. `ci / verify` passed on `eab4dcc1f4`.

Failures during this work:

- My fix to `agent-runtime-connector-access` changed an invalid
permission key in the setup, but not in the assertion. The runtime check
found this. I corrected the assertion, and the file passes.
- `heartbeat-plugin-environment` failed one wait in the full run. It
passed three times alone, so I think that load caused the failure. It
can fail again under load.
- `dot-runner` skipped all 54 tests in the full run because `cargo` was
not on my `PATH`. It then passed 54 of 54 with the toolchain on the
`PATH`.
- My first `lib` value removed the default DOM types, and the error
count went up by 375. I added the DOM libraries.
- My Daytona stub with bare `any` left 48 errors, because generic
helpers then inferred `unknown`. I added shapes that return promises.
- One fix to `agent-secrets-routes` used `class_3_static_lease`, and the
test failed. That class allows only specific paths. I reverted to
`unclassified`.
- `ci / General tests (server (8/12))` failed on `259737b700` because of
the flaky port in `idle-completed-history`. paperclipai/paperclip#18853
fixed this test on master, so this pull request keeps that fix.

**Audit of the diff.** I counted these patterns in added and removed
lines, to find fixes that only hide an error:

| Pattern | Added | Removed |
| --- | --- | --- |
| `@ts-ignore` | 0 | 0 |
| `@ts-expect-error` | 0 | 1 |
| `.skip` | 0 | 0 |
| `as any` | 9 | 9 |
| `: any` | 11 | 2 |
| `as never` | 8 | 2 |
| `as unknown as` | 35 | 4 |
| `Promise<unknown>` | 16 | 5 |
| `expect(` | 128 | 113 |

No hunk removes more `expect(` calls than it adds. Three removals of
test content were intentional, and I checked each one:

- `muse-control` removed `signalToken`. Line 1012 already asserts that
it is absent.
- `merged-pr-confirmation-sweep` removed legacy aliases. Other tests
cover them.
- `heartbeat-workspace-branch-containment` removed a `finalize` branch
that no test row can reach.

Some partial fixtures stay unchecked through `as unknown as` and
`Promise<unknown>`, mostly in `issue-agent-mutation-ownership-routes`
and `issue-comment-reopen-routes`.

**Accuracy of the check.** One program over the tree and the three
shards reported the same errors at the same positions. Memory, as the
peak of the largest process on one thread, was 3.93 GiB for one program
and 2.43 GiB for the largest shard. I measured these two values before
the fixes.

Not run or not proven:

- I did not run the full `pnpm test:run`, or a root `pnpm build`. I
expect CI to run the full suites.
- I did not run the 4 GiB container check, or the full
`scripts/typecheck-low-memory.mjs`. The peaks above come from macOS, so
they are not a Linux cgroup measurement.
- I did not run the UI typecheck after the merges.
- `workspace-runtime-https-live-exercise` is skipped, so its `cleanup`
fix did not run.

## Risks

- **A large diff in test files.** 328 files change. A pull request that
edits the same tests can get merge conflicts. New tests on master can
also add type errors before this merges. Any such error fails this check
and needs a fix here.
- **A higher cost for changes to source types.** A change to a type can
add errors in test files that the author did not touch. The author must
then fix them. This is the purpose of the check, but it is new work.
- **Changed test behavior.** Some tests now run paths that they did not
run before, such as the 9 chat wizard fixtures and the 3 span tests.
They pass here, but they can be more sensitive to timing or to the
environment.
- **The Daytona stub can go stale.** The stub is written by hand. If the
plugin uses a new SDK member, the typecheck reports an error until
someone updates the stub.
- **The release path.** `release-verify.yml` and `scripts/release.sh`
run `pnpm -r typecheck`, so they now run this check.
- **The tsconfig change.** The test transform reads
`server/src/__tests__/tsconfig.json`. The options that I changed are for
type checks, and the runs above passed.
- **A gap that remains.** Tsconfigs in `packages/paperclip-runner` and
in most sandbox provider plugins exclude `*.test.ts`. This check does
not read most of those test files, so they can still drift. That work is
separate from this pull request.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with
reasoning, tool use, code execution, and subagents. The runtime does not
show the context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass (7 tests fail locally, with
and without this change. See Verification.)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 15:27:38 -08:00

1430 lines
51 KiB
TypeScript

import { beforeEach, describe, expect, it, vi } from "vitest";
const { mockResolveEnvironmentDriverConfigForRuntime } = vi.hoisted(() => ({
mockResolveEnvironmentDriverConfigForRuntime: vi.fn(),
}));
vi.mock("../services/environment-config.js", () => ({
resolveEnvironmentDriverConfigForRuntime: mockResolveEnvironmentDriverConfigForRuntime,
}));
import {
measureStartupStep,
runWithoutActiveStep,
SANDBOX_STARTUP_SPAN_ATTRS,
} from "@paperclipai/adapter-utils/acpx-engine/startup-timing";
import {
DEFAULT_SANDBOX_REMOTE_CWD,
resolveEnvironmentExecutionTarget,
} from "../services/environment-execution-target.js";
const A = SANDBOX_STARTUP_SPAN_ATTRS;
// A recording trace context that models the OTel parenting contract:
// `startSpan(name, options, context)` reads the parent from the explicit
// `context` token that `contextWithSpan` built. A test asserts the exact parent
// of each child span without an OTel package.
function createRecordingTrace() {
const spans: Array<{
name: string;
attributes: Record<string, unknown>;
parent: unknown;
ended: boolean;
setAttribute(key: string, value: unknown): void;
setStatus(status: { code: number; message?: string }): void;
end(): void;
}> = [];
const tracer = {
startSpan(name: string, _options?: unknown, context?: unknown) {
const parent =
context && typeof context === "object" && "span" in context
? (context as { span: unknown }).span
: null;
const span = {
name,
attributes: {} as Record<string, unknown>,
parent,
ended: false,
setAttribute(key: string, value: unknown) {
span.attributes[key] = value;
},
setStatus(_status: { code: number; message?: string }) {},
end() {
span.ended = true;
},
};
spans.push(span);
return span;
},
};
const contextWithSpan = (span: unknown) => ({ span });
return { tracer, contextWithSpan, spans };
}
// A fake tracer that records the third `startSpan` argument — the parent-context
// token — for each span, keyed by the span name. `contextWithSpan` wraps a span
// in a token and keeps that token, so a test asserts the exact token identity,
// not a rebuilt copy. The exec seam reads the parent-context token from the
// active step store and passes it as the third `startSpan` argument. This helper
// holds the parent assertion in one place for reuse.
function recordParentContext() {
const calls: Array<{ name: string; parentContext: unknown; span: unknown }> = [];
const tokens = new Map<unknown, unknown>();
const tracer = {
startSpan(name: string, _options?: unknown, parentContext?: unknown) {
const span = {
name,
setAttribute(_key: string, _value: unknown) {},
setStatus(_status: { code: number; message?: string }) {},
end() {},
};
calls.push({ name, parentContext, span });
return span;
},
};
const contextWithSpan = (span: unknown) => {
const token = { span };
tokens.set(span, token);
return token;
};
// The span object recorded for a given span name.
const spanNamed = (name: string) => calls.find((call) => call.name === name)?.span;
// The parent-context token that `startSpan` received for a given span name.
const parentContextFor = (name: string) =>
calls.find((call) => call.name === name)?.parentContext;
// The parent-context token that `contextWithSpan` returned for a given span.
const tokenForSpan = (span: unknown) => tokens.get(span);
return { tracer, contextWithSpan, calls, spanNamed, parentContextFor, tokenForSpan };
}
describe("resolveEnvironmentExecutionTarget", () => {
beforeEach(() => {
mockResolveEnvironmentDriverConfigForRuntime.mockReset();
delete process.env.PAPERCLIP_API_URL;
delete process.env.PAPERCLIP_RUNTIME_API_URL;
});
it("uses a bounded default cwd for sandbox targets when lease metadata omits remoteCwd", async () => {
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
driver: "sandbox",
config: {
provider: "fake-plugin",
reuseLease: false,
timeoutMs: 30_000,
},
});
const target = await resolveEnvironmentExecutionTarget({
db: {} as never,
companyId: "company-1",
adapterType: "codex_local",
environment: {
id: "env-1",
driver: "sandbox",
config: {
provider: "fake-plugin",
},
},
leaseId: "lease-1",
leaseMetadata: {},
lease: null,
environmentRuntime: null,
});
expect(target).toMatchObject({
kind: "remote",
transport: "sandbox",
providerKey: "fake-plugin",
remoteCwd: DEFAULT_SANDBOX_REMOTE_CWD,
leaseId: "lease-1",
environmentId: "env-1",
timeoutMs: 30_000,
});
});
it("keeps sandbox targets on bridge mode even when lease metadata includes a Paperclip API URL", async () => {
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
driver: "sandbox",
config: {
provider: "fake-plugin",
reuseLease: false,
timeoutMs: 30_000,
},
});
const target = await resolveEnvironmentExecutionTarget({
db: {} as never,
companyId: "company-1",
adapterType: "codex_local",
environment: {
id: "env-1",
driver: "sandbox",
config: {
provider: "fake-plugin",
},
},
leaseId: "lease-1",
leaseMetadata: {
paperclipApiUrl: "https://paperclip.example.test",
},
lease: null,
environmentRuntime: null,
});
expect(target).toMatchObject({
kind: "remote",
transport: "sandbox",
providerKey: "fake-plugin",
remoteCwd: DEFAULT_SANDBOX_REMOTE_CWD,
});
expect(target).not.toHaveProperty("paperclipApiUrl");
expect(target).not.toHaveProperty("paperclipTransport");
});
it("passes through a provider-declared sandbox shell command from lease metadata", async () => {
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
driver: "sandbox",
config: {
provider: "fake-plugin",
reuseLease: false,
timeoutMs: 30_000,
},
});
const target = await resolveEnvironmentExecutionTarget({
db: {} as never,
companyId: "company-1",
adapterType: "claude_local",
environment: {
id: "env-1",
driver: "sandbox",
config: {
provider: "fake-plugin",
},
},
leaseId: "lease-1",
leaseMetadata: {
shellCommand: "bash",
},
lease: null,
environmentRuntime: null,
});
expect(target).toMatchObject({
kind: "remote",
transport: "sandbox",
shellCommand: "bash",
});
});
it("keeps sandbox targets on callback bridge execution even when lease metadata advertises SSH access", async () => {
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
driver: "sandbox",
config: {
provider: "fake-plugin",
reuseLease: false,
timeoutMs: 30_000,
},
});
const target = await resolveEnvironmentExecutionTarget({
db: {} as never,
companyId: "company-1",
adapterType: "claude_local",
environment: {
id: "env-1",
driver: "sandbox",
config: {
provider: "fake-plugin",
},
},
leaseId: "lease-1",
leaseMetadata: {
remoteCwd: "/home/sandbox/paperclip-workspace",
sshAccess: {
type: "ssh",
host: "ssh.example.test",
port: 22,
username: "paperclip",
},
},
lease: null,
environmentRuntime: null,
});
expect(target).toMatchObject({
kind: "remote",
transport: "sandbox",
providerKey: "fake-plugin",
remoteCwd: "/home/sandbox/paperclip-workspace",
});
});
it("resolves sandbox targets for every remote-managed adapter, including grok_local and kimi_local", async () => {
for (const adapterType of [
"claude_local",
"codex_local",
"cursor",
"gemini_local",
"grok_local",
"kimi_local",
"opencode_local",
"pi_local",
]) {
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
driver: "sandbox",
config: {
provider: "fake-plugin",
reuseLease: false,
timeoutMs: 30_000,
},
});
const target = await resolveEnvironmentExecutionTarget({
db: {} as never,
companyId: "company-1",
adapterType,
environment: {
id: "env-1",
driver: "sandbox",
config: { provider: "fake-plugin" },
},
leaseId: "lease-1",
leaseMetadata: {},
lease: null,
environmentRuntime: null,
});
expect(target, `adapter ${adapterType}`).toMatchObject({
kind: "remote",
transport: "sandbox",
providerKey: "fake-plugin",
});
}
});
it("returns null for adapters without remote-managed environment support", async () => {
for (const driver of ["sandbox", "ssh"] as const) {
const target = await resolveEnvironmentExecutionTarget({
db: {} as never,
companyId: "company-1",
adapterType: "process",
environment: {
id: "env-1",
driver,
config: { provider: "fake-plugin" },
},
leaseId: "lease-1",
leaseMetadata: {},
lease: null,
environmentRuntime: null,
});
expect(target, `driver ${driver}`).toBeNull();
}
expect(mockResolveEnvironmentDriverConfigForRuntime).not.toHaveBeenCalled();
});
it("resolves SSH execution targets for grok_local", async () => {
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
driver: "ssh",
config: {
host: "ssh.example.test",
port: 22,
username: "paperclip",
remoteWorkspacePath: "/srv/paperclip",
privateKey: "PRIVATE KEY",
knownHosts: "[ssh.example.test]:22 ssh-ed25519 AAAA",
strictHostKeyChecking: true,
},
});
const target = await resolveEnvironmentExecutionTarget({
db: {} as never,
companyId: "company-1",
adapterType: "grok_local",
environment: {
id: "env-ssh-1",
driver: "ssh",
config: {},
},
leaseId: "lease-ssh-1",
leaseMetadata: {},
lease: null,
environmentRuntime: null,
});
expect(target).toMatchObject({
kind: "remote",
transport: "ssh",
remoteCwd: "/srv/paperclip",
});
});
it("resolves SSH execution targets for kimi_local", async () => {
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
driver: "ssh",
config: {
host: "ssh.example.test",
port: 22,
username: "paperclip",
remoteWorkspacePath: "/srv/paperclip",
privateKey: "PRIVATE KEY",
knownHosts: "[ssh.example.test]:22 ssh-ed25519 AAAA",
strictHostKeyChecking: true,
},
});
const target = await resolveEnvironmentExecutionTarget({
db: {} as never,
companyId: "company-1",
adapterType: "kimi_local",
environment: {
id: "env-ssh-1",
driver: "ssh",
config: {},
},
leaseId: "lease-ssh-1",
leaseMetadata: {},
lease: null,
environmentRuntime: null,
});
expect(target).toMatchObject({
kind: "remote",
transport: "ssh",
remoteCwd: "/srv/paperclip",
});
});
it("resolves SSH execution targets in bridge mode", async () => {
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
driver: "ssh",
config: {
host: "ssh.example.test",
port: 22,
username: "paperclip",
remoteWorkspacePath: "/srv/paperclip",
privateKey: "PRIVATE KEY",
knownHosts: "[ssh.example.test]:22 ssh-ed25519 AAAA",
strictHostKeyChecking: true,
},
});
const target = await resolveEnvironmentExecutionTarget({
db: {} as never,
companyId: "company-1",
adapterType: "codex_local",
environment: {
id: "env-ssh-1",
driver: "ssh",
config: {},
},
leaseId: "lease-ssh-1",
leaseMetadata: {},
lease: null,
environmentRuntime: null,
});
expect(target).toMatchObject({
kind: "remote",
transport: "ssh",
remoteCwd: "/srv/paperclip",
leaseId: "lease-ssh-1",
environmentId: "env-ssh-1",
spec: {
host: "ssh.example.test",
port: 22,
username: "paperclip",
remoteWorkspacePath: "/srv/paperclip",
remoteCwd: "/srv/paperclip",
},
});
expect(target).not.toHaveProperty("paperclipApiUrl");
});
it("exposes a sandbox runner with single-stream stdin upload disabled", async () => {
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
driver: "sandbox",
config: {
provider: "fake-plugin",
reuseLease: false,
timeoutMs: 30_000,
},
});
const environmentRuntime = {
execute: vi.fn().mockResolvedValue({
exitCode: 0,
signal: null,
timedOut: false,
stdout: "ok",
stderr: "",
metadata: { durationMs: 600, getDurationMs: 15 },
}),
supportsSync: vi.fn().mockReturnValue(false),
};
const target = await resolveEnvironmentExecutionTarget({
db: {} as never,
companyId: "company-1",
adapterType: "codex_local",
environment: { id: "env-1", driver: "sandbox", config: { provider: "fake-plugin" } },
leaseId: "lease-1",
leaseMetadata: { remoteCwd: "/workspace" },
lease: { id: "lease-1" } as never,
environmentRuntime: environmentRuntime as never,
});
const runner = (target as { runner?: {
supportsSingleStreamStdinProgress?: boolean;
execute(input: { command: string; args?: string[] }): Promise<unknown>;
} }).runner;
expect(runner).toBeTruthy();
// Provider-backed sandbox RPCs do not surface bounded mid-stream progress
// for a single stdin upload, so the runner leaves the capability disabled.
expect(runner!.supportsSingleStreamStdinProgress).toBe(false);
// The exec seam still runs each command; the run-log no longer carries the
// detailed per-step round-trip or provider-duration counts.
await runner!.execute({ command: "echo", args: ["a"] });
await runner!.execute({ command: "echo", args: ["b"] });
expect(environmentRuntime.execute).toHaveBeenCalledTimes(2);
});
it("forwards the session flags to the environment runtime execute", async () => {
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
driver: "sandbox",
config: {
provider: "fake-plugin",
reuseLease: false,
timeoutMs: 30_000,
},
});
const environmentRuntime = {
execute: vi.fn().mockResolvedValue({
exitCode: 0,
signal: null,
timedOut: false,
stdout: "ok",
stderr: "",
metadata: { durationMs: 600, getDurationMs: 15 },
}),
supportsSync: vi.fn().mockReturnValue(false),
};
const target = await resolveEnvironmentExecutionTarget({
db: {} as never,
companyId: "company-1",
adapterType: "codex_local",
environment: { id: "env-1", driver: "sandbox", config: { provider: "fake-plugin" } },
leaseId: "lease-1",
leaseMetadata: { remoteCwd: "/workspace" },
lease: { id: "lease-1" } as never,
environmentRuntime: environmentRuntime as never,
});
const runner = (target as { runner?: {
execute(input: {
command: string;
args?: string[];
useSession?: boolean;
bypassSession?: boolean;
}): Promise<unknown>;
} }).runner!;
// The agent command opts onto the persistent session with `useSession`,
// which the seam maps to `forceSession`. It never bypasses the session.
await runner.execute({ command: "node", args: ["script.js"], useSession: true });
// A bridge control-plane exec opts off the persistent session with
// `bypassSession`, which the seam forwards unchanged.
await runner.execute({ command: "sh", args: ["-c", "cat"], bypassSession: true });
const first = environmentRuntime.execute.mock.calls[0]![0] as {
forceSession?: boolean;
bypassSession?: boolean;
};
expect(first.forceSession).toBe(true);
expect(first.bypassSession).toBeUndefined();
const second = environmentRuntime.execute.mock.calls[1]![0] as {
forceSession?: boolean;
bypassSession?: boolean;
};
expect(second.forceSession).toBeUndefined();
expect(second.bypassSession).toBe(true);
});
// A recording tracer that captures each provider-exec span's name, attribute
// map, and end. It satisfies the structural tracer the seam calls.
function createRecordingExecTracer() {
const spans: Array<{
name: string;
attributes: Record<string, unknown>;
status: { code: number; message?: string } | null;
ended: boolean;
}> = [];
const tracer = {
startSpan(name: string) {
const span = {
name,
attributes: {} as Record<string, unknown>,
status: null as { code: number; message?: string } | null,
ended: false,
setAttribute(key: string, value: unknown) {
span.attributes[key] = value;
},
setStatus(status: { code: number; message?: string }) {
span.status = status;
},
end() {
span.ended = true;
},
};
spans.push(span);
return span;
},
};
return { tracer, spans };
}
// The value of `SpanStatusCode.ERROR` in `@opentelemetry/api`. A failed exec
// span must carry this native status, not only the `failed` outcome attribute.
const SPAN_STATUS_CODE_ERROR = 2;
// The closed span-attribute allowlist for a `sandbox.exec` span. A test
// asserts every recorded key is in this set, so a command, an argument, a
// path, an id, or an error-text key can never ride the span.
const ALLOWED_EXEC_SPAN_ATTRIBUTE_KEYS = new Set<string>([
A.provider,
A.execCommand,
A.execExitCode,
A.execWallMs,
A.execWaitBeforeMs,
A.execSandboxMs,
A.execNetworkMs,
A.execCriticalPath,
A.execCacheHit,
A.outcome,
]);
async function runnerFor(input: {
provider: string;
execResult: Record<string, unknown>;
tracer: unknown;
}) {
return runnerWithExecute({
provider: input.provider,
tracer: input.tracer,
execute: vi.fn().mockResolvedValue(input.execResult),
});
}
// Build the sandbox runner with a custom provider-exec implementation, so a
// test can drive a thrown execution or assert the span order around the await.
async function runnerWithExecute(input: {
provider: string;
tracer: unknown;
execute: (input: { onLog?: (s: string, c: string) => Promise<void> }) => Promise<unknown>;
}) {
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
driver: "sandbox",
config: { provider: input.provider, reuseLease: false, timeoutMs: 30_000 },
});
const environmentRuntime = {
execute: input.execute,
supportsSync: vi.fn().mockReturnValue(false),
};
const target = await resolveEnvironmentExecutionTarget({
db: {} as never,
companyId: "company-1",
adapterType: "codex_local",
environment: { id: "env-1", driver: "sandbox", config: { provider: input.provider } },
leaseId: "lease-1",
leaseMetadata: { remoteCwd: "/workspace" },
lease: { id: "lease-1" } as never,
environmentRuntime: environmentRuntime as never,
tracer: input.tracer as never,
});
return (target as { runner?: {
execute(input: { command: string; args?: string[]; env?: Record<string, string>; cwd?: string; stdin?: string }): Promise<unknown>;
} }).runner!;
}
// Run the sandbox runner's execute with an incremental log sink and collect
// the ordered deliveries. The runner's execute accepts an `onLog`, so this
// casts past the narrowed helper return type.
async function runExecuteCollectingLogs(
runner: { execute(input: unknown): Promise<unknown> },
): Promise<Array<[string, string]>> {
const delivered: Array<[string, string]> = [];
await runner.execute({
command: "echo",
onLog: async (stream: "stdout" | "stderr", chunk: string) => {
delivered.push([stream, chunk]);
},
});
return delivered;
}
it("delivers only the un-streamed suffix after a provider streams a prefix then polls the complete result", async () => {
// The provider streams a prefix through the incremental sink, then its
// stream fails and it polls the complete output as the final result. The
// reconciler must deliver the remaining tail once, so no output byte is lost
// or repeated.
const { tracer } = createRecordingExecTracer();
const runner = await runnerWithExecute({
provider: "daytona",
tracer,
execute: async (input: unknown) => {
const typed = input as { onLog?: (s: "stdout" | "stderr", c: string) => Promise<void> };
await typed.onLog?.("stdout", "hello ");
await typed.onLog?.("stderr", "warn:");
return { exitCode: 0, signal: null, timedOut: false, stdout: "hello world", stderr: "warn:done" };
},
});
const delivered = await runExecuteCollectingLogs(
runner as { execute(input: unknown): Promise<unknown> },
);
expect(delivered).toEqual([
["stdout", "hello "],
["stderr", "warn:"],
["stdout", "world"],
["stderr", "done"],
]);
});
it("does not repeat output when the provider already streamed the complete result", async () => {
// The provider streams the whole output through the incremental sink and
// returns the same complete result. The suffix is empty, so the reconciler
// never re-delivers the streamed bytes.
const { tracer } = createRecordingExecTracer();
const runner = await runnerWithExecute({
provider: "daytona",
tracer,
execute: async (input: unknown) => {
const typed = input as { onLog?: (s: "stdout" | "stderr", c: string) => Promise<void> };
await typed.onLog?.("stdout", "full");
return { exitCode: 0, signal: null, timedOut: false, stdout: "full", stderr: "" };
},
});
const delivered = await runExecuteCollectingLogs(
runner as { execute(input: unknown): Promise<unknown> },
);
expect(delivered).toEqual([["stdout", "full"]]);
});
it("does not replay a capped provider tail after streaming more than 4 MiB", async () => {
const receipt = JSON.stringify({ type: "turn_end", message: { usage: { input: 20, output: 10, cost: { total: 0.004 } } } }) + "\n";
const output = "x".repeat(5 * 1024 * 1024) + "\n" + receipt;
const { tracer } = createRecordingExecTracer();
const runner = await runnerWithExecute({ provider: "createos", tracer,
execute: async (input: unknown) => {
const typed = input as { onLog?: (s: "stdout" | "stderr", c: string) => Promise<void> };
for (let offset = 0; offset < output.length; offset += 65536) await typed.onLog?.("stdout", output.slice(offset, offset + 65536));
await typed.onLog?.("stderr", "diagnostic prefix\nlast warning\n");
return { exitCode: 0, signal: null, timedOut: false, stdout: output.slice(-4 * 1024 * 1024), stderr: "last warning\n" };
},
});
const delivered = await runExecuteCollectingLogs(runner as { execute(input: unknown): Promise<unknown> });
expect(delivered.filter(([stream]) => stream === "stdout").map(([, chunk]) => chunk).join("")).toBe(output);
expect(delivered.filter(([stream]) => stream === "stderr")).toEqual([["stderr", "diagnostic prefix\nlast warning\n"]]);
});
it("delivers the full captured output when the provider streams nothing incrementally", async () => {
// The provider streams no incremental chunk, so the whole final result is
// the suffix and reaches the sink once.
const { tracer } = createRecordingExecTracer();
const runner = await runnerWithExecute({
provider: "daytona",
tracer,
execute: async () => ({
exitCode: 0,
signal: null,
timedOut: false,
stdout: "batch-out",
stderr: "batch-err",
}),
});
const delivered = await runExecuteCollectingLogs(
runner as { execute(input: unknown): Promise<unknown> },
);
expect(delivered).toEqual([
["stdout", "batch-out"],
["stderr", "batch-err"],
]);
});
it("delivers the whole final output when the poll fallback buffer does not continue the streamed prefix", async () => {
// The provider streams a prefix, then its stream fails and it polls a
// buffer that does NOT start with that prefix. A length slice would drop
// the leading bytes of the poll buffer and corrupt the durable log, so the
// reconciler delivers the whole final output instead. The streamed prefix
// repeats, but no output byte is lost or truncated.
const { tracer } = createRecordingExecTracer();
const runner = await runnerWithExecute({
provider: "daytona",
tracer,
execute: async (input: unknown) => {
const typed = input as { onLog?: (s: "stdout" | "stderr", c: string) => Promise<void> };
await typed.onLog?.("stdout", "hello ");
await typed.onLog?.("stderr", "warn:");
// The poll buffer starts with different leading text on both streams.
return { exitCode: 0, signal: null, timedOut: false, stdout: "RESYNCED output", stderr: "RESET err" };
},
});
const delivered = await runExecuteCollectingLogs(
runner as { execute(input: unknown): Promise<unknown> },
);
expect(delivered).toEqual([
["stdout", "hello "],
["stderr", "warn:"],
["stdout", "RESYNCED output"],
["stderr", "RESET err"],
]);
});
it("sets the provider duration attributes from finite Daytona-shaped metadata", async () => {
const { tracer, spans } = createRecordingExecTracer();
const runner = await runnerFor({
provider: "daytona",
execResult: {
exitCode: 0,
signal: null,
timedOut: false,
stdout: "ok",
stderr: "",
metadata: { durationMs: 600, getDurationMs: 15 },
},
tracer,
});
await runner.execute({ command: "echo", args: ["a"] });
expect(spans).toHaveLength(1);
const span = spans[0]!;
expect(span.name).toBe("sandbox.exec");
expect(span.ended).toBe(true);
// `sandbox_ms` = provider in-sandbox run; `wait_before_ms` = handle-fetch.
expect(span.attributes[A.execSandboxMs]).toBe(600);
expect(span.attributes[A.execWaitBeforeMs]).toBe(15);
expect(span.attributes[A.provider]).toBe("daytona");
// `echo` is a known command basename, so it rides as a clamped label.
expect(span.attributes[A.execCommand]).toBe("echo");
expect(span.attributes[A.execExitCode]).toBe(0);
expect(span.attributes[A.outcome]).toBe("ok");
// A successful exec leaves the native span status unset (default OTel status).
expect(span.status).toBeNull();
expect(span.attributes[A.execCriticalPath]).toBe(true);
// The wall time is a real, finite, non-negative number.
expect(typeof span.attributes[A.execWallMs]).toBe("number");
expect(span.attributes[A.execWallMs] as number).toBeGreaterThanOrEqual(0);
});
it("carries the explicit exec cache_hit from result.metadata.cacheHit", async () => {
const { tracer, spans } = createRecordingExecTracer();
const runner = await runnerFor({
provider: "daytona",
execResult: {
exitCode: 0,
signal: null,
timedOut: false,
stdout: "ok",
stderr: "",
metadata: { durationMs: 600, getDurationMs: 0, cacheHit: true },
},
tracer,
});
await runner.execute({ command: "echo", args: ["a"] });
const span = spans[0]!;
// The flag comes from the metadata boolean, not from a zero handle-fetch.
expect(span.attributes[A.execCacheHit]).toBe(true);
});
it("omits exec cache_hit when the provider reports no cacheHit metadata", async () => {
const { tracer, spans } = createRecordingExecTracer();
const runner = await runnerFor({
provider: "daytona",
execResult: {
exitCode: 0,
signal: null,
timedOut: false,
stdout: "ok",
stderr: "",
metadata: { durationMs: 600, getDurationMs: 15 },
},
tracer,
});
await runner.execute({ command: "echo", args: ["a"] });
const span = spans[0]!;
// A provider that omits the boolean yields no attribute — never `false`.
expect(span.attributes).not.toHaveProperty(A.execCacheHit);
});
it("omits each duration attribute when a provider returns no timing (does not throw, keeps provider)", async () => {
const { tracer, spans } = createRecordingExecTracer();
const runner = await runnerFor({
provider: "kubernetes",
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
tracer,
});
await expect(runner.execute({ command: "echo" })).resolves.toBeTruthy();
expect(spans).toHaveLength(1);
const span = spans[0]!;
expect(A.execSandboxMs in span.attributes).toBe(false);
expect(A.execWaitBeforeMs in span.attributes).toBe(false);
// With no provider durations, the derived network time is also omitted.
expect(A.execNetworkMs in span.attributes).toBe(false);
// The provider attribute is always present so a trace shows which provider ran.
expect(span.attributes[A.provider]).toBe("kubernetes");
});
it("never emits a `0` duration attribute for a Daytona timeout that omits durationMs", async () => {
const { tracer, spans } = createRecordingExecTracer();
const runner = await runnerFor({
provider: "daytona",
execResult: {
exitCode: 124,
signal: null,
timedOut: true,
stdout: "",
stderr: "",
// The Daytona timeout branch may leave durationMs undefined.
metadata: { getDurationMs: 20 },
},
tracer,
});
await runner.execute({ command: "sleep", args: ["999"] });
const span = spans[0]!;
expect(A.execSandboxMs in span.attributes).toBe(false);
expect(span.attributes[A.execWaitBeforeMs]).toBe(20);
// A non-zero exit yields `failed`; the exit code rides as a number.
expect(span.attributes[A.outcome]).toBe("failed");
// A failed exec also sets the native span status to ERROR.
expect(span.status).toEqual({ code: SPAN_STATUS_CODE_ERROR });
expect(span.attributes[A.execExitCode]).toBe(124);
// `sleep` is not in the known-command allowlist, so it clamps to `other`.
expect(span.attributes[A.execCommand]).toBe("other");
});
it("never sets a command, arg, env, cwd, or stream text as a span attribute, even with secret-like input", async () => {
const { tracer, spans } = createRecordingExecTracer();
const runner = await runnerFor({
provider: "daytona",
execResult: {
exitCode: 0,
signal: null,
timedOut: false,
// Secret-like standard-stream text must never ride the span.
stdout: "AKIAIOSFODNN7EXAMPLE token=s3cr3t-stdout",
stderr: "error at /home/agent/.ssh/id_rsa: s3cr3t-stderr",
metadata: { durationMs: 5, getDurationMs: 1 },
},
tracer,
});
await runner.execute({
// A full command line, a secret-like argument, a secret-like env value, a
// stdin blob, and a path-like cwd — none may ride the span.
command: "bash -lc 'rm -rf /secret/path'",
args: ["--token", "s3cr3t-arg", "--password", "hunter2"],
env: { AWS_SECRET_ACCESS_KEY: "s3cr3t-env", HOME: "/home/agent" },
cwd: "/home/agent/secret-workspace/.git",
stdin: "s3cr3t-stdin-blob",
});
const span = spans[0]!;
for (const key of Object.keys(span.attributes)) {
expect(ALLOWED_EXEC_SPAN_ATTRIBUTE_KEYS.has(key), `non-allowlisted key "${key}"`).toBe(true);
}
// The command clamps to the bounded `other` fallback (not a known basename).
expect(span.attributes[A.execCommand]).toBe("other");
// No forbidden substring rides any attribute value.
const values = Object.values(span.attributes).map(String);
for (const forbidden of [
"rm -rf",
"s3cr3t",
"hunter2",
"AKIA",
"id_rsa",
"/secret/path",
"/home/agent",
"secret-workspace",
"stdin",
]) {
expect(
values.some((value) => value.includes(forbidden)),
`attribute value leaked "${forbidden}"`,
).toBe(false);
}
});
it("normalizes a plugin-backed provider key to `plugin` and keeps a built-in family as-is", async () => {
const plugin = createRecordingExecTracer();
const pluginRunner = await runnerFor({
provider: "acme-custom-sandbox",
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
tracer: plugin.tracer,
});
await pluginRunner.execute({ command: "echo" });
expect(plugin.spans[0]!.attributes[A.provider]).toBe("plugin");
const builtIn = createRecordingExecTracer();
const builtInRunner = await runnerFor({
provider: "e2b",
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
tracer: builtIn.tracer,
});
await builtInRunner.execute({ command: "echo" });
expect(builtIn.spans[0]!.attributes[A.provider]).toBe("e2b");
});
it("parents the exec span to the active step span when the exec runs inside a measured step", async () => {
const { tracer, contextWithSpan, spans } = createRecordingTrace();
const runner = await runnerFor({
provider: "daytona",
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
tracer,
});
// The exec seam reads the active step context through `getActiveStepContext`.
// Wrap the exec in one measured step and assert the exec span parents to the
// step span, not to nothing.
await measureStartupStep({}, () => 0, "stage.sync", () => runner.execute({ command: "echo" }), {
tracer,
contextWithSpan,
});
const stepSpan = spans.find((span) => span.name === "stage.sync");
const execSpan = spans.find((span) => span.name === "sandbox.exec");
expect(stepSpan).toBeTruthy();
expect(execSpan).toBeTruthy();
expect(execSpan!.parent).toBe(stepSpan);
});
it("opens an unparented exec span when the exec runs outside any measured step", async () => {
const { tracer, spans } = createRecordingTrace();
const runner = await runnerFor({
provider: "daytona",
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
tracer,
});
// No measured step wraps the exec, so the active step context is null and
// the exec span opens unparented (a no-op when tracing is off).
await runner.execute({ command: "echo" });
const execSpan = spans.find((span) => span.name === "sandbox.exec");
expect(execSpan).toBeTruthy();
expect(execSpan!.parent).toBeNull();
});
// The three baseline tests below record the third `startSpan` argument — the
// parent-context token — and assert its identity. They pin the current exec
// parenting so a later phase that re-points the exec parent has a fixed
// reference point.
it("test_exec_inside_measured_step_parents_to_step_context", async () => {
const rec = recordParentContext();
const runner = await runnerFor({
provider: "daytona",
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
tracer: rec.tracer,
});
// Run the seam `execute` inside one measured step. The step publishes its
// child context to the active step store, so the seam reads it and passes it
// as the third `startSpan` argument for the exec span.
await measureStartupStep({}, () => 0, "stage.sync", () => runner.execute({ command: "echo" }), {
tracer: rec.tracer,
contextWithSpan: rec.contextWithSpan,
});
const stepSpan = rec.spanNamed("stage.sync");
expect(stepSpan).toBeTruthy();
// The exec span parents to the step span. The third `startSpan` argument is
// the exact parent-context token that `contextWithSpan` built for the step
// span, not a rebuilt copy.
expect(rec.parentContextFor("sandbox.exec")).toBe(rec.tokenForSpan(stepSpan));
});
it("test_exec_in_root_region_is_unparented_today", async () => {
const rec = recordParentContext();
const runner = await runnerFor({
provider: "daytona",
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
tracer: rec.tracer,
});
// No measured step wraps the exec, so the active step store is empty. The
// seam reads no parent and passes `undefined` as the third `startSpan`
// argument. The exec span opens unparented today.
await runner.execute({ command: "echo" });
expect(rec.spanNamed("sandbox.exec")).toBeTruthy();
expect(rec.parentContextFor("sandbox.exec")).toBeUndefined();
});
it("test_exec_on_runWithoutActiveStep_is_unparented_today", async () => {
const rec = recordParentContext();
const runner = await runnerFor({
provider: "daytona",
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
tracer: rec.tracer,
});
// `runWithoutActiveStep` empties the active step store for the wrapped work.
// The seam reads no parent and passes `undefined` as the third `startSpan`
// argument. The exec span opens unparented today.
await runWithoutActiveStep(() => runner.execute({ command: "echo" }));
expect(rec.spanNamed("sandbox.exec")).toBeTruthy();
expect(rec.parentContextFor("sandbox.exec")).toBeUndefined();
});
it("opens the exec span before the provider await so the span wraps the execution", async () => {
const { tracer, spans } = createRecordingExecTracer();
// Assert the span is already open (started, not ended) while the provider
// runs. If the seam opened the span after the await, no open span would
// exist here and the native span duration would be near zero.
const runner = await runnerWithExecute({
provider: "daytona",
tracer,
execute: vi.fn().mockImplementation(async () => {
const open = spans.find((span) => span.name === "sandbox.exec");
expect(open, "the exec span must be open during the provider await").toBeTruthy();
expect(open!.ended).toBe(false);
return { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" };
}),
});
await runner.execute({ command: "echo" });
const span = spans.find((s) => s.name === "sandbox.exec");
expect(span!.ended).toBe(true);
expect(span!.attributes[A.outcome]).toBe("ok");
});
it("records a failed exec span and rethrows when the provider execution throws", async () => {
const { tracer, spans } = createRecordingExecTracer();
const runner = await runnerWithExecute({
provider: "daytona",
tracer,
execute: vi.fn().mockRejectedValue(new Error("provider transport failed")),
});
// The original error rides through unchanged; observability never swallows it.
await expect(runner.execute({ command: "echo" })).rejects.toThrow("provider transport failed");
// A thrown execution still produces one ended span with the `failed` outcome,
// instead of no span at all.
const span = spans.find((s) => s.name === "sandbox.exec");
expect(span).toBeTruthy();
expect(span!.ended).toBe(true);
expect(span!.attributes[A.outcome]).toBe("failed");
// A thrown execution also sets the native span status to ERROR.
expect(span!.status).toEqual({ code: SPAN_STATUS_CODE_ERROR });
expect(span!.attributes[A.provider]).toBe("daytona");
// `echo` is a known basename, so the clamped command rides the span.
expect(span!.attributes[A.execCommand]).toBe("echo");
// No exec result exists, so the exit code never rides the span.
expect(A.execExitCode in span!.attributes).toBe(false);
// The wall time is a real, finite, non-negative number.
expect(typeof span!.attributes[A.execWallMs]).toBe("number");
expect(span!.attributes[A.execWallMs] as number).toBeGreaterThanOrEqual(0);
// Only allowlisted keys ride the failed span.
for (const key of Object.keys(span!.attributes)) {
expect(ALLOWED_EXEC_SPAN_ATTRIBUTE_KEYS.has(key), `non-allowlisted key "${key}"`).toBe(true);
}
});
it("keeps the exec span outcome `ok` when the execution succeeds but a log callback rejects", async () => {
const { tracer, spans } = createRecordingExecTracer();
// The provider execution succeeds and returns stdout, so the seam invokes
// the log callback. The callback rejects, which models a downstream log-sink
// failure. The rejection must reach the caller, but it must never reclassify
// the successful execution as a failed span.
const runner = await runnerWithExecute({
provider: "daytona",
tracer,
execute: vi.fn().mockResolvedValue({
exitCode: 0,
signal: null,
timedOut: false,
stdout: "hello",
stderr: "",
}),
});
const onLog = vi.fn().mockRejectedValue(new Error("log sink rejected"));
// The rejection rides through unchanged; observability never swallows it.
await expect(
(runner as { execute(input: unknown): Promise<unknown> }).execute({ command: "echo", onLog }),
).rejects.toThrow("log sink rejected");
// The span ended with the successful outcome from the command result, not
// the failed outcome. A log failure never marks the execution failed.
const span = spans.find((s) => s.name === "sandbox.exec");
expect(span).toBeTruthy();
expect(span!.ended).toBe(true);
expect(span!.attributes[A.outcome]).toBe("ok");
// A log-callback rejection never marks the span as ERROR; the status stays unset.
expect(span!.status).toBeNull();
expect(span!.attributes[A.execExitCode]).toBe(0);
// The seam reached the log callback exactly once (the stdout delivery).
expect(onLog).toHaveBeenCalledTimes(1);
});
it("delivers incremental logs before the final result and does not duplicate them", async () => {
const { tracer } = createRecordingExecTracer();
const runner = await runnerWithExecute({
provider: "daytona",
tracer,
execute: vi.fn(async (input: { onLog?: (s: string, c: string) => Promise<void> }) => {
// The provider streams the output while the command runs.
await input.onLog?.("stdout", "chunk-1");
await input.onLog?.("stderr", "chunk-2");
await input.onLog?.("stdout", "chunk-3");
return {
exitCode: 0,
signal: null,
timedOut: false,
stdout: "chunk-1chunk-3",
stderr: "chunk-2",
};
}),
});
const onLog = vi.fn();
const result = await (runner as {
execute(input: unknown): Promise<{ stdout: string; stderr: string; exitCode: number }>;
}).execute({ command: "echo", onLog });
// The runner receives the incremental chunks in order, and NOT a repeated
// delivery of the final stdout/stderr.
expect(onLog.mock.calls).toEqual([
["stdout", "chunk-1"],
["stderr", "chunk-2"],
["stdout", "chunk-3"],
]);
// The final result stays available to the caller for parsing and fallback.
expect(result).toMatchObject({ exitCode: 0, stdout: "chunk-1chunk-3", stderr: "chunk-2" });
});
it("still delivers the final result to the runner when the provider does not stream", async () => {
const { tracer } = createRecordingExecTracer();
const runner = await runnerWithExecute({
provider: "daytona",
tracer,
// A provider that never calls onLog returns only the final result.
execute: vi.fn(async () => ({
exitCode: 0,
signal: null,
timedOut: false,
stdout: "final-out",
stderr: "final-err",
})),
});
const onLog = vi.fn();
const result = await (runner as {
execute(input: unknown): Promise<{ stdout: string; stderr: string }>;
}).execute({ command: "echo", onLog });
expect(onLog.mock.calls).toEqual([
["stdout", "final-out"],
["stderr", "final-err"],
]);
expect(result).toMatchObject({ stdout: "final-out", stderr: "final-err" });
});
it("creates exactly one sandbox.exec span for one streamed provider call", async () => {
const { tracer, spans } = createRecordingExecTracer();
const runner = await runnerWithExecute({
provider: "daytona",
tracer,
execute: vi.fn(async (input: { onLog?: (s: string, c: string) => Promise<void> }) => {
await input.onLog?.("stdout", "chunk-a");
await input.onLog?.("stdout", "chunk-b");
await input.onLog?.("stderr", "chunk-c");
return {
exitCode: 0,
signal: null,
timedOut: false,
stdout: "chunk-achunk-b",
stderr: "chunk-c",
};
}),
});
const onLog = vi.fn();
await (runner as { execute(input: unknown): Promise<unknown> }).execute({
command: "echo",
onLog,
});
// One long-lived provider call opens one span; each stream chunk opens none.
expect(spans).toHaveLength(1);
expect(spans[0]!.name).toBe("sandbox.exec");
expect(spans[0]!.ended).toBe(true);
});
it("keeps streamed log text and secret values out of span attributes", async () => {
const { tracer, spans } = createRecordingExecTracer();
const secret = "sk-super-secret-value";
const runner = await runnerWithExecute({
provider: "daytona",
tracer,
execute: vi.fn(async (input: { onLog?: (s: string, c: string) => Promise<void> }) => {
await input.onLog?.("stdout", `token=${secret}\n`);
return {
exitCode: 0,
signal: null,
timedOut: false,
stdout: `token=${secret}\n`,
stderr: "",
};
}),
});
const onLog = vi.fn();
// The secret rides the env and the streamed chunk, never the command label.
await (runner as { execute(input: unknown): Promise<unknown> }).execute({
command: "run-agent",
env: { API_KEY: secret },
onLog,
});
expect(spans).toHaveLength(1);
const span = spans[0]!;
// Attributes carry only the closed allowlist — never log text or secrets.
for (const key of Object.keys(span.attributes)) {
expect(ALLOWED_EXEC_SPAN_ATTRIBUTE_KEYS.has(key), `non-allowlisted key "${key}"`).toBe(true);
}
const serialized = JSON.stringify(span.attributes);
expect(serialized).not.toContain(secret);
expect(serialized).not.toContain("token=");
expect(serialized).not.toContain("chunk");
});
// Fire one run-time exec from a bridge continuation that runs after the step
// span ended. Each bridge step (`bridge.paperclip`, `bridge.process-session`)
// starts long-lived work with `criticalPath: false`. The bridge boundary wraps
// that long-lived work in `runWithoutActiveStep`, exactly as modeled here, so
// the continuation reads an empty active step. Return the recorded exec span.
async function runContinuationExec(step: string, options: { wrap: boolean }) {
const { tracer, contextWithSpan, spans } = createRecordingTrace();
const runner = await runnerFor({
provider: "daytona",
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
tracer,
});
let resolveExec!: () => void;
const execDone = new Promise<void>((resolve) => {
resolveExec = resolve;
});
// Schedule the exec from a timer inside the step body, so it fires after the
// step span ends. The `wrap` flag models the fix: when true, the boundary
// wraps the long-lived work in `runWithoutActiveStep`; when false, it models
// the pre-fix leak.
const scheduleContinuation = () => {
setTimeout(() => {
void runner.execute({ command: "echo" }).then(() => resolveExec());
}, 0);
};
await measureStartupStep(
{},
() => 0,
step,
async () => {
if (options.wrap) {
runWithoutActiveStep(scheduleContinuation);
} else {
scheduleContinuation();
}
return "started";
},
{ tracer, contextWithSpan, criticalPath: false },
);
await execDone;
return spans.find((span) => span.name === "sandbox.exec");
}
it("opens an unparented exec span for a process-session bridge continuation", async () => {
const execSpan = await runContinuationExec("bridge.process-session", { wrap: true });
expect(execSpan).toBeTruthy();
// The step span ended and the boundary emptied the store, so the continuation
// exec opens a root span, not one under the dead bridge step.
expect(execSpan!.parent).toBeNull();
});
it("opens an unparented exec span for a paperclip bridge continuation", async () => {
const execSpan = await runContinuationExec("bridge.paperclip", { wrap: true });
expect(execSpan).toBeTruthy();
expect(execSpan!.parent).toBeNull();
});
it("does not copy the stale criticalPath = false flag onto a continuation exec", async () => {
const execSpan = await runContinuationExec("bridge.process-session", { wrap: true });
expect(execSpan).toBeTruthy();
// The bridge step set `criticalPath: false`. The continuation reads an empty
// store, so the exec span records the default `true`, never the stale `false`.
expect(execSpan!.attributes[A.execCriticalPath]).toBe(true);
expect(execSpan!.attributes[A.execCriticalPath]).not.toBe(false);
});
it("leaks the ended step onto a continuation exec without the boundary wrap", async () => {
// The mechanism guard: an unwrapped continuation keeps the ended bridge step
// store, so the exec span parents to the dead step and copies its
// `criticalPath: false`. The boundary wrap in the two tests above removes both
// defects, so this suite fails if a future edit drops the wrap.
const { tracer, contextWithSpan, spans } = createRecordingTrace();
const runner = await runnerFor({
provider: "daytona",
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
tracer,
});
let resolveExec!: () => void;
const execDone = new Promise<void>((resolve) => {
resolveExec = resolve;
});
await measureStartupStep(
{},
() => 0,
"bridge.process-session",
async () => {
setTimeout(() => {
void runner.execute({ command: "echo" }).then(() => resolveExec());
}, 0);
return "started";
},
{ tracer, contextWithSpan, criticalPath: false },
);
await execDone;
const stepSpan = spans.find((span) => span.name === "bridge.process-session");
const execSpan = spans.find((span) => span.name === "sandbox.exec");
expect(stepSpan).toBeTruthy();
expect(execSpan).toBeTruthy();
// The unwrapped continuation parents the exec span to the ended step and
// copies the stale flag.
expect(execSpan!.parent).toBe(stepSpan);
expect(execSpan!.attributes[A.execCriticalPath]).toBe(false);
});
});