Files
PaperClipAI/tests/runner-e2e/harness-env.ts
T
Dotta 0c1e7504c0 fix(runner): persist warm Daytona workspaces across turns (#12904)
## Thinking Path

> - Daytona preserves a stopped sandbox filesystem, but deleting or
replacing a sandbox removes its only remote copy.
> - Warm reuse therefore improves latency but cannot be Paperclip's
durability boundary.
> - The host execution workspace must remain authoritative after every
successful turn, while same-run recovery must avoid overwriting
unexported remote work.
> - Result proposal, workspace export/merge, and terminal completion
need a durable, replayable ordering so a crash never starts a duplicate
provider turn.
> - A paid browser acceptance suite must exercise both legacy Codex and
Runner Codex for three real turns on one continuously warm Daytona
sandbox.

## Linked Issues or Issue Description

Refs #12901.

Runner Codex did not previously export successful Daytona workspace
changes back to the authoritative host workspace. That made warm reuse
depend on Daytona's remote filesystem and left deleted/replacement
sandboxes without a reliable reconstruction path. The existing paid
fixture also lacked a focused three-turn continuity case for both Codex
adapters.

## What Changed

- Persist versioned, atomic native workspace-sync descriptors and
durable seeds in `PAPERCLIP_HOME`, without credentials or a database
migration.
- Classify fresh, warm, replacement, and same-run-recovery workspace
preparation explicitly; ambiguous lease/root/digest evidence fails
closed.
- Finalize native workspace export/merge after semantic result proposal
and before run completion, with idempotent replay that never submits a
second provider turn.
- Surface legacy Codex workspace restoration failures instead of masking
them, while preserving an earlier provider error when both fail.
- Keep healthy reusable Daytona leases warm for legacy and native
adapters, stamp finalized workspace generations, and retain existing
cleanup behavior for per-turn or unhealthy leases.
- Preserve Runner Codex's provider process/session across warm turns,
including bounded post-terminal tail draining and exact authority
rotation.
- Add the exact paid `daytona-warm-continuity` matrix:
  - `legacy-codex × daytona × warm-three-turn`
  - `runner-codex × daytona × warm-three-turn`
- Drive all three turns through the browser, verify ordered file
continuity and stable lease/workspace/runtime identities, capture
per-turn timings, and delete the sandbox immediately after assertions.
- Document `pnpm test:e2e:runner -- --suite daytona-warm-continuity`; no
package script was added.

## Verification

- `pnpm typecheck` — passed, including migration safety (no migration
added)
- Focused server/runner Vitest coverage — 144 passed
- `pnpm test:e2e:runner:unit` — 114 passed
- `pnpm test:e2e:runner:typecheck` — passed
- `pnpm --filter @paperclipai/paperclip-runner test:codex` — 66 passed,
1 helper ignored
- `native-session-executor.test.ts` — 139 passed, including safe
fail-closed cleanup after remote runner identity capture failure
- Paid local browser acceptance, exact post-rebase Linux/amd64 runner
binary:
- Runner Codex — passed in 1.7m; 3 runs; lease outcomes `created,
resumed, resumed`; 10/10 matchers; cleanup passed
- Legacy Codex — passed in 2.7m; 3 runs; lease outcomes `created,
resumed, resumed`; 10/10 matchers; cleanup passed
- [Protected paid GitHub Actions
campaign](https://github.com/paperclipai/paperclip/actions/runs/34026735033)
against `7da42a91b95fa7fb2df126668ef7e37afb3b2b9d` — passed 2/2:
- Runner Codex — 3 runs; lease outcomes `created, resumed, resumed`;
evidence and cleanup passed
- Legacy Codex — 3 runs; lease outcomes `created, resumed, resumed`;
evidence and cleanup passed
  - Merge/enforcement, S3 history, and Pages publication jobs passed
- Paid result artifacts were scanned for both provider credentials;
neither secret was present.
- Current PR checks — 31 passed, 1 expected Storybook skip; Greptile
5/5; Superagent security scan passed
- `git diff --check origin/master...HEAD` — passed
- Confirmed no `package.json`, lockfile, migration, or SQL changes.

## Risks

- Workspace synchronization now sits on the terminal-success path, so a
remote export failure deliberately prevents false success. Retryable
state retains its lease/seed; loss of the only unexported remote copy
fails closed.
- Warm provider reuse has strict identity and quiescence checks.
Mismatched or ambiguous evidence blocks reuse rather than risking
concurrent provider work.
- The paid suite incurs Daytona and Codex cost only in the existing
protected scheduled/manual workflow and explicitly destroys its sandbox
after each cell.

## Model Used

OpenAI Codex with GPT-5 agentic reasoning, repository inspection, real
browser E2E execution, Rust/TypeScript test execution, and GitHub
Actions diagnostics.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked an existing issue or described the issue
in-PR
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name contains no internal ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have documented the dedicated suite invocation without adding a
package script
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green on the current revision
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
on the current revision
- [x] I will address all reviewer comments before requesting merge
2026-09-06 08:16:12 -05:00

191 lines
5.8 KiB
TypeScript

import path from "node:path";
import { CREDENTIAL_NAMES } from "./types.js";
import type { MatrixExecution } from "./types.js";
const DATABASE_KEYS = ["DATABASE_URL", "DATABASE_MIGRATION_URL"] as const;
const AMBIENT_PAPERCLIP_CREDENTIAL_KEYS = [
"PAPERCLIP_API_KEY",
"PAPERCLIP_AGENT_API_KEY",
"PAPERCLIP_TASK_BRIDGE_TOKEN",
"PAPERCLIP_SETUP_TOKEN",
"PAPERCLIP_SECRETS_MASTER_KEY",
"PAPERCLIP_SECRETS_MASTER_KEY_FILE",
] as const;
const GENERATED_SERVER_SECRET_KEYS = [
"PAPERCLIP_AGENT_JWT_SECRET",
"PAPERCLIP_DECISION_SIGNING_SECRET",
"PAPERCLIP_TOOL_ACTION_SIGNING_SECRET",
"BETTER_AUTH_SECRET",
] as const;
const AMBIENT_EXTERNAL_STATE_KEYS = [
"PAPERCLIP_STORAGE_S3_BUCKET",
"PAPERCLIP_STORAGE_S3_REGION",
"PAPERCLIP_STORAGE_S3_ENDPOINT",
"PAPERCLIP_STORAGE_S3_PREFIX",
"PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE",
] as const;
const PROVIDER_SECRET_KEY = /^(?:OPENAI|ANTHROPIC|OPENROUTER|DAYTONA)(?:_|$)/;
export function runnerE2EServerControlPaths(temporaryRoot: string) {
const controlDirectory = path.join(temporaryRoot, "control");
return {
controlDirectory,
restartRequestPath: path.join(
controlDirectory,
"server-restart.request.json",
),
restartAcknowledgementPath: path.join(
controlDirectory,
"server-restart.ack.json",
),
};
}
/**
* Native cells use the debug binary produced once by build:runner-binaries.
* Preserve an explicit override for release builds and developer workflows.
*/
export function resolvePaperclipRunnerBinaryForHarness(
executions: readonly MatrixExecution[],
repositoryRoot: string,
configuredPath = process.env.PAPERCLIP_RUNNER_BINARY,
platform: NodeJS.Platform = process.platform,
): string | undefined {
if (configuredPath?.trim()) return configuredPath;
if (
!executions.some((execution) => execution.profile.generation === "native")
) {
return undefined;
}
return path.join(
repositoryRoot,
"packages",
"paperclip-runner",
"runner",
"target",
"debug",
platform === "win32" ? "paperclip-runnerd.exe" : "paperclip-runnerd",
);
}
/**
* Remote native cells stage the same controller-owned binary whose digest is
* authorized by the PRP control plane. Local cells launch it directly.
*/
export function resolvePaperclipRemoteRunnerBinaryForHarness(
executions: readonly MatrixExecution[],
runnerBinary: string | undefined,
configuredPath = process.env.PAPERCLIP_RUNNER_REMOTE_BINARY_PATH,
platform: NodeJS.Platform = process.platform,
): string | undefined {
if (configuredPath?.trim()) return configuredPath;
if (!runnerBinary) return undefined;
// Daytona runs Linux. A default debug binary built by a macOS developer is
// Mach-O and cannot be staged into that sandbox. Leave the remote override
// unset so the pinned Daytona image's verified runnerd is discovered instead.
if (platform !== "linux") return undefined;
return executions.some(
(execution) =>
execution.profile.generation === "native" &&
execution.environment.expectedExecutionTarget.kind === "remote",
)
? runnerBinary
: undefined;
}
/**
* Keep fixture-only provider switches scoped to the one isolated harness that
* needs them. In particular, the pinned legacy OpenCode model is routed by the
* paid gateway and may not appear in OpenCode's public model catalog.
*/
export function buildRunnerE2EProcessEnvironment(
source: NodeJS.ProcessEnv,
executions: readonly MatrixExecution[],
): NodeJS.ProcessEnv {
const result = { ...source };
delete result.OPENCODE_ALLOW_ALL_MODELS;
if (
executions.length > 0 &&
executions.every(
(execution) =>
execution.profile.generation === "legacy" &&
execution.profile.provider === "opencode",
)
) {
result.OPENCODE_ALLOW_ALL_MODELS = "true";
}
return result;
}
/**
* Build the environment inherited by the Paperclip server. Paid credentials
* deliberately stay in the launcher/Playwright process and cross the server
* boundary only once, in the encrypted company-secrets API request.
*/
export function buildPaperclipServerEnvironment(
source: NodeJS.ProcessEnv,
overrides: NodeJS.ProcessEnv = {},
): NodeJS.ProcessEnv {
const result = { ...source };
for (const key of Object.keys(result)) {
if (PROVIDER_SECRET_KEY.test(key)) delete result[key];
}
for (const key of [
...CREDENTIAL_NAMES,
...DATABASE_KEYS,
...AMBIENT_PAPERCLIP_CREDENTIAL_KEYS,
...AMBIENT_EXTERNAL_STATE_KEYS,
]) {
delete result[key];
}
for (const key of GENERATED_SERVER_SECRET_KEYS) delete result[key];
Object.assign(result, overrides);
return result;
}
export function assertIsolatedServerEnvironment(
env: NodeJS.ProcessEnv,
expected: {
temporaryRoot: string;
paperclipHome: string;
configPath: string;
},
) {
const home = env.PAPERCLIP_HOME;
const config = env.PAPERCLIP_CONFIG;
if (home !== expected.paperclipHome || config !== expected.configPath) {
throw new Error(
"Paperclip server environment does not use the allocated home/config paths",
);
}
if (
!home.startsWith(`${expected.temporaryRoot}/`) ||
!config.startsWith(`${expected.temporaryRoot}/`)
) {
throw new Error(
"Paperclip server paths escape the isolated temporary root",
);
}
if (env.XDG_CACHE_HOME !== path.join(expected.temporaryRoot, "xdg-cache")) {
throw new Error(
"Paperclip server cache does not use the allocated temporary root",
);
}
for (const key of [
...CREDENTIAL_NAMES,
...DATABASE_KEYS,
...AMBIENT_PAPERCLIP_CREDENTIAL_KEYS,
...AMBIENT_EXTERNAL_STATE_KEYS,
]) {
if (env[key])
throw new Error(
`Paperclip server environment unexpectedly contains ${key}`,
);
}
for (const key of GENERATED_SERVER_SECRET_KEYS) {
if (!env[key])
throw new Error(`Paperclip server environment is missing ${key}`);
}
}