Files
PaperClipAI/tests/runner-e2e/harness-env.ts
T
cceeb0aa66 test(runner): add everyday workflow evaluation harness (#13474)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner must support project work, delegation, hiring, and
service access.
> - Browser tests exposed lost connection access, rejected helper
events, and stalled recovery.
> - Some eval failures also came from incorrect fixtures and decision
controls.
> - This pull request fixes those paths and adds eight everyday workflow
stories.
> - The tests retain observed failures and verify delivered files
independently.
> - The benefit is repeatable evidence for common user tasks and their
remaining gaps.

## Linked Issues or Issue Description

Related work: #13404 contains earlier workflow fixes. #13300 and #13470
changed the CI contracts used by the harness security tests. Merged
companion:
[paperclip-evals#22](https://github.com/paperclipai/paperclip-evals/pull/22).

**What happened?**

Native ACPX sessions did not receive the assigned connection gateway.
Codex helper events could arrive before their spawn receipt and fail
thread validation. A parent continuation could take a shared workspace
before its child retried. A failed native continuation could leave the
task status without a clear recovery blocker. The eval harness also
confused tool approvals with new connection requests and could reject a
valid delegated download.

**Expected behavior**

Keep assigned gateway access and its approval checks. Verify helper
lineage before accepting helper progress. Let a waiting child proceed
before automatic parent recovery. Preserve a failed task's recovery
ownership. Grade the actual requested workflow and its delivered files.

**Steps to reproduce**

Run the everyday workflow suite with the native Codex and Claude
profiles. Exercise service approval, connection refusal, delegated
project work, and teammate reuse. The commands and case requirements are
in `tests/runner-e2e/EVERYDAY-WORKFLOWS.md`. Use `pnpm
test:runner-recovery` for controlled crash and replacement cases.

## What Changed

- Pass the scoped connection gateway binding through the native ACPX
host and sidecar.
- Recognize Codex helper lineage from parent metadata and spawn
receipts. Verify early helper events with `thread/read`. Keep helper
events separate from root completion authority.
- Guide agents to use persistent hiring, child tasks, dependency
records, and a blocked handoff while waiting for a child.
- Defer automatic parent recovery while a child has an active execution
path in the same shared workspace. Allow parent recovery when the child
needs review.
- Record Blocked status and recovery evidence when a failed native
continuation needs reconciliation, including existing active or
escalated incidents. Preserve their owner and retry budget.
- Add eight browser-driven workflow cases. Use real decision controls,
explicit child feedback delivery, managed hiring credentials, and
independent ZIP checks inside a bounded Docker sandbox. Verify sandbox
availability before task creation. Record screenshot SHA-256 at capture.
- Keep runner crash probes in controlled recovery tests. Preserve the
original failure when cleanup also fails.
- Display missing accounting and replay revisions as unavailable. Align
harness security assertions with the approved CI changes.
- Make the channel-rejection browser fixture bind its file after the
send captures its payload. This prevents live refresh from removing the
file before the simulated race.

## Verification

- Full workspace `pnpm -r typecheck` passed after merging current
master.
- Runner E2E typecheck passed. Harness unit tests passed: 216/216.
- Wake-queue database tests passed: 55/55. The two added
existing-incident tests failed before the fix and pass after it.
- Docker artifact calibration passed: 12/12. Host-file and host-loopback
isolation tests failed before the fix and pass after it. Read-only
delivery and output limits are also verified.
- Full `pnpm build` passed. Targeted recovery tests passed: 83/83.
- The channel-rejection browser test passed five consecutive runs after
fixing the fixture race found in CI.
- Local general-server (12,351 tests), UI (6,250), CLI (485), and
workspace package groups passed. The monolithic run stopped at an
unchanged lock-heartbeat fixture race; the isolated workspace group
passed on rerun (shared: 747/747). A separate local serialized run
passed 97 files before two socket errors in the unchanged issue-list
route suite; that suite passed 15/15 on isolated rerun. These local full
commands did not finish uninterrupted; the complete CI matrix below
covers the remaining suites.
- Final head `0fb293733fe307be7e6667ae8f1364077d0c6455`: **34 successful
checks, 2 expected skips**, including every server/workspace shard,
browser shard, native runner verification, build, and typecheck. [Final
CI
run](https://github.com/paperclipai/paperclip/actions/runs/34989136700).
- Greptile reviewed this exact head at **5/5**; all review threads are
resolved. Both Superagent security checks are successful.
- ACPX credential-boundary tests passed: 118/118. Superagent accepted
the runner/sidecar versus provider-environment trace and cleared its
finding.
- The latest paid local campaign on source
`f6a2fdf7ac2af859826a2ae627ff4125a5478529` passed 22/24 cases: Sol 8/8,
Claude 7/8, Mini 7/8. These results predate the merge with current
master.
- The two remaining failures are in `hire-reuse`: Claude exceeded the
attempt deadline during final review; Mini made invalid deliverable tool
calls and remained Blocked.
- Six Daytona cases were not run because the matching immutable runner
image was unavailable. This PR does not claim new remote model results.

## Risks

The changes affect connection admission, helper identity, and recovery
scheduling. Assigned gateway grants and user approval still govern
service calls. The workspace admission gate still exists; the broader
folder-sync design is separate work. Provider behavior can still cause
the two recorded hiring failures. No database migration is required.
Paid cases are opt-in and have bounded attempt deadlines. Project
stories now require Docker and the documented pinned Python image on the
harness host.

## Model Used

OpenAI `gpt-6-astra` performed implementation, diagnosis, and
substantive review. OpenAI `gpt-5.6-luna` assisted with verification, PR
preparation, and review tracking. Both used repository tools and code
execution. Context-window sizes were not recorded.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused checks and
isolated reruns; full-run limitations are documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: OpenAI GPT-5.6 Luna <noreply@openai.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-15 11:04:16 -05:00

196 lines
6.1 KiB
TypeScript

import path from "node:path";
import { CREDENTIAL_NAMES } from "./types.js";
import type { MatrixExecution } from "./types.js";
const DATABASE_KEYS = ["DATABASE_URL", "DATABASE_MIGRATION_URL"] as const;
const AMBIENT_PAPERCLIP_CREDENTIAL_KEYS = [
"PAPERCLIP_API_KEY",
"PAPERCLIP_AGENT_API_KEY",
"PAPERCLIP_TASK_BRIDGE_TOKEN",
"PAPERCLIP_SETUP_TOKEN",
"PAPERCLIP_SECRETS_MASTER_KEY",
"PAPERCLIP_SECRETS_MASTER_KEY_FILE",
] as const;
const GENERATED_SERVER_SECRET_KEYS = [
"PAPERCLIP_AGENT_JWT_SECRET",
"PAPERCLIP_DECISION_SIGNING_SECRET",
"PAPERCLIP_TOOL_ACTION_SIGNING_SECRET",
"BETTER_AUTH_SECRET",
] as const;
const AMBIENT_EXTERNAL_STATE_KEYS = [
"PAPERCLIP_STORAGE_S3_BUCKET",
"PAPERCLIP_STORAGE_S3_REGION",
"PAPERCLIP_STORAGE_S3_ENDPOINT",
"PAPERCLIP_STORAGE_S3_PREFIX",
"PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE",
] as const;
const PROVIDER_SECRET_KEY = /^(?:OPENAI|ANTHROPIC|OPENROUTER|DAYTONA)(?:_|$)/;
export function runnerE2EServerControlPaths(temporaryRoot: string) {
const controlDirectory = path.join(temporaryRoot, "control");
return {
controlDirectory,
restartRequestPath: path.join(
controlDirectory,
"server-restart.request.json",
),
restartAcknowledgementPath: path.join(
controlDirectory,
"server-restart.ack.json",
),
};
}
/**
* Native cells use the debug binary produced once by build:runner-binaries.
* Preserve an explicit override for release builds and developer workflows.
*/
export function resolvePaperclipRunnerBinaryForHarness(
executions: readonly MatrixExecution[],
repositoryRoot: string,
configuredPath = process.env.PAPERCLIP_RUNNER_BINARY,
platform: NodeJS.Platform = process.platform,
): string | undefined {
if (configuredPath?.trim()) return configuredPath;
if (
!executions.some((execution) => execution.profile.generation === "native")
) {
return undefined;
}
return path.join(
repositoryRoot,
"packages",
"paperclip-runner",
"runner",
"target",
"debug",
platform === "win32" ? "paperclip-runnerd.exe" : "paperclip-runnerd",
);
}
/**
* Remote native cells stage the same controller-owned binary whose digest is
* authorized by the PRP control plane. Local cells launch it directly.
*/
export function resolvePaperclipRemoteRunnerBinaryForHarness(
executions: readonly MatrixExecution[],
runnerBinary: string | undefined,
configuredPath = process.env.PAPERCLIP_RUNNER_REMOTE_BINARY_PATH,
platform: NodeJS.Platform = process.platform,
): string | undefined {
if (configuredPath?.trim()) return configuredPath;
if (!runnerBinary) return undefined;
// Daytona runs Linux. A default debug binary built by a macOS developer is
// Mach-O and cannot be staged into that sandbox. Leave the remote override
// unset so the pinned Daytona image's verified runnerd is discovered instead.
if (platform !== "linux") return undefined;
return executions.some(
(execution) =>
execution.profile.generation === "native" &&
execution.environment.expectedExecutionTarget.kind === "remote",
)
? runnerBinary
: undefined;
}
/**
* Keep fixture-only provider switches scoped to the one isolated harness that
* needs them. In particular, the pinned legacy OpenCode model is routed by the
* paid gateway and may not appear in OpenCode's public model catalog.
*/
export function buildRunnerE2EProcessEnvironment(
source: NodeJS.ProcessEnv,
executions: readonly MatrixExecution[],
): NodeJS.ProcessEnv {
const result = { ...source };
delete result.OPENCODE_ALLOW_ALL_MODELS;
// Hiring needs the opt-in native API surface. Scope this to the explicit
// manual hiring story; production and other suites retain their defaults.
if (executions.some((e) => e.suite.id === "everyday-workflows" && e.task.id === "hire-reuse")) {
result.PAPERCLIP_RUNNER_API_TOOLS_ENABLED = "true";
}
if (
executions.length > 0 &&
executions.every(
(execution) =>
execution.profile.generation === "legacy" &&
execution.profile.provider === "opencode",
)
) {
result.OPENCODE_ALLOW_ALL_MODELS = "true";
}
return result;
}
/**
* Build the environment inherited by the Paperclip server. Paid credentials
* deliberately stay in the launcher/Playwright process and cross the server
* boundary only once, in the encrypted company-secrets API request.
*/
export function buildPaperclipServerEnvironment(
source: NodeJS.ProcessEnv,
overrides: NodeJS.ProcessEnv = {},
): NodeJS.ProcessEnv {
const result = { ...source };
for (const key of Object.keys(result)) {
if (PROVIDER_SECRET_KEY.test(key)) delete result[key];
}
for (const key of [
...CREDENTIAL_NAMES,
...DATABASE_KEYS,
...AMBIENT_PAPERCLIP_CREDENTIAL_KEYS,
...AMBIENT_EXTERNAL_STATE_KEYS,
]) {
delete result[key];
}
for (const key of GENERATED_SERVER_SECRET_KEYS) delete result[key];
Object.assign(result, overrides);
return result;
}
export function assertIsolatedServerEnvironment(
env: NodeJS.ProcessEnv,
expected: {
temporaryRoot: string;
paperclipHome: string;
configPath: string;
},
) {
const home = env.PAPERCLIP_HOME;
const config = env.PAPERCLIP_CONFIG;
if (home !== expected.paperclipHome || config !== expected.configPath) {
throw new Error(
"Paperclip server environment does not use the allocated home/config paths",
);
}
if (
!home.startsWith(`${expected.temporaryRoot}/`) ||
!config.startsWith(`${expected.temporaryRoot}/`)
) {
throw new Error(
"Paperclip server paths escape the isolated temporary root",
);
}
if (env.XDG_CACHE_HOME !== path.join(expected.temporaryRoot, "xdg-cache")) {
throw new Error(
"Paperclip server cache does not use the allocated temporary root",
);
}
for (const key of [
...CREDENTIAL_NAMES,
...DATABASE_KEYS,
...AMBIENT_PAPERCLIP_CREDENTIAL_KEYS,
...AMBIENT_EXTERNAL_STATE_KEYS,
]) {
if (env[key])
throw new Error(
`Paperclip server environment unexpectedly contains ${key}`,
);
}
for (const key of GENERATED_SERVER_SECRET_KEYS) {
if (!env[key])
throw new Error(`Paperclip server environment is missing ${key}`);
}
}