Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-21 16:27:18 +02:00
Code Issues Packages Projects Releases Wiki Activity

Carry current master through Pi public installation

Browse Source
This commit is contained in:
Dotta committed 2026-10-08 14:22:49 -05:00
2 parents b7e692f48d 20ce062701
commit e00258afd9
2195 files changed
+255923 -79198
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/runner-e2e/EVERYDAY-WORKFLOWS.md
+41 -4
View File
@@ -15,6 +15,7 @@ exact execution ID explicitly.
| Hire a teammate and use them again | `hire-reuse` | One Morgan QA reporting to the lead, native runner and the same encrypted connection bindings, real child execution, then a second usable delivery from that same agent. |
| Decide on an installed service action | `service-approve`, `service-decline` | Assign an authenticated local MCP fixture with Ask first; match its tool action and connection ID; no provider call before approval; exactly one after approval and a verified document; none after decline. |
| Decline a new connection | `connection-decline` | Start without service connections; match a Notion connection intent; click Not now; verify the saved rejection, no new connection or repeated request, and an explanation followed by Done. |
| Request an email address | `agentmail-setup` | Enable Chat connectors, ask for an email address, and require a durable AgentMail card for the requesting agent and user. Reload and verify one password field, the direct API-key link, and no access selectors or modal. Decline and verify no connection or repeated request. |
| Continue work after a controller restart | `recover-controller` | Observe saved source, persist a user message, restart the isolated controller, and independently test the delivered result. |
| Stop work and change direction | `stop-redirect` | Click Stop, send one new request, reload, observe exactly one stored user message and the new answer, and reach Done. |
| Create and edit a company skill | `create-skill-studio` | Create one skill through the runner, verify its persisted library entry and activity-feed card, open Skill Studio, save an edit, and verify the edit after returning. Local Codex, local ACPX Claude, and warm Daytona cells are explicit. |
@@ -36,10 +37,12 @@ See [controlled recovery tests](../runner-recovery/README.md).
## Matrix and running
The local matrix has nine cases on native Codex `gpt-5.6-sol`, native ACPX Claude
`claude-sonnet-5`, and native Codex `gpt-5.4-mini`: 27 cells. The two core profiles
The local matrix has fourteen cases on native Codex `gpt-5.6-sol`, native ACPX Claude
`claude-sonnet-5`, and native Codex `gpt-5.4-mini`: 42 cells. The two core profiles
also declare build/revise, delegation, controller-restart, and skill-creation cases
on Daytona: eight cells. Remote runner-process killing is not supported. For remote controller
on Daytona: eight cells. OpenCode adds only local hiring/reuse and delegation,
for 52 cells total. Hiring/reuse and delegation have one attempt and a
1,000-cent company and lead-agent hard stop. Remote runner-process killing is not supported. For remote controller
restart, a verified first download supplies the persistence checkpoint; the
controller is interrupted during a subsequent revision with another queued
requirement.
@@ -200,7 +203,19 @@ identity check.
## External-provider fallback
Three explicit local cases cover aggregator routing with the normal production
agent guidance. They add nine local cells; the suite now has 47 cells total.
agent guidance. They add nine local cells. The AgentMail setup case adds three
local cells; the suite now has 50 cells total.
The AgentMail case uses real model discovery and production interaction/UI paths,
but declines before sending credentials to AgentMail. Its independent grader is
calibrated against missing, duplicate, misaddressed, hidden, and malformed cards.
The email integration suite separately proves credential/inbox creation, access
defaults, assignment checks, and completion using a fixture provider. Neither
test qualifies live AgentMail delivery. Run a bounded local model cell with:
```sh
pnpm test:e2e:runner -- --id everyday-workflows.runner-codex-mini.local.agentmail-setup --max-automatic-retries 0
```
- `provider-native`: Jira is supported natively and by aggregators. Require the
Jira connection card directly, decline it in the browser, and verify no provider
@@ -223,3 +238,25 @@ revision, harness digest, and existing usage/cost evidence accompany each attemp
routing, incorrect ordering, early calls, duplicate questions, and fabricated reads.
Run a single `everyday-workflows.runner-codex-mini.local.provider-decline` cell
first; do not treat these fixtures as live provider compatibility tests.
### Connection-guidance observation boundaries
The explicit `native-connection-guidance` suite keeps one attempt per cell.
For decline cases, Done and a succeeded run are insufficient: settlement waits
within the existing cell deadline for a saved reply attributed by run ID to the
lead agent's final successful run on this task. Readiness checks storage and
identity, not favorable wording. A missing reply times out; an incorrect reply
settles and fails the independent explanation check. Both decline explanation
checks use the same bounded matcher, including unavailable/rejected access and
“wasn't able to pull” wording. This is textual evidence, not proof of cognition.
An executed approval or a recorded tool-action rejection with `wake_assignee`
may precede its continuation run. Only a response bound to the same task, company,
agent, and completed source run can defer the stranded-blocker check, and only
within the original deadline. A failed execution, unrelated card, or already
consumed response cannot extend that wait.
`connection-guidance-decline-grade.json` and the workflow's
`declineGradeEvidence` retain the exact cloned assertion input and original
checks. Later final-state or cleanup observations cannot overwrite that input.
Earlier campaign grades remain unchanged when the evaluator is corrected.
tests/runner-e2e/FIXTURES.md
+65
View File
@@ -1,5 +1,19 @@
# Runner E2E fixture authoring
## Connection creation fixtures
See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only
`provider-connections` suite, local/staging target ownership, dedicated browser
profiles, credential handoffs, private evidence, and cleanup contract.
The [public MCP journeys](PUBLIC-MCP.md) reuse the fixture registry with a real
authenticated browser session. `RunnerApi.setBrowserSession` binds that session
to API calls, including encrypted secret provisioning via Node fetch. OAuth
setup stays outside model context. The external assistant receives the catalog
from `tools/list` and the shipped workflow skills; its calls execute against the
real SDK transport. Client-side loss of a successful response is the sole fault
injection in the uncertain-retry case. Task/run/document REST reads own grading.
The fixture catalog is executable production-contract data. Keep it small,
typed, deterministic, and free of raw credentials.
@@ -17,6 +31,17 @@ profile, model qualification, environment, task, or ranking-snapshot change
must change that fingerprint automatically so the dashboard can annotate the
boundary instead of silently joining unlike totals.
The explicit [stock-harness suite](STOCK-HARNESS.md) wraps existing profiles with
`productionDefaultHireProfile`: omit only `instructionsBundle` so the public
hire route loads the shipped default, while preserving runtime, permissions,
auth, skills, and managed secret references. Do not replace this with a fixture
copy of the default manual. Public receipts check the exact independently
specified bundle before provider execution and again during cleanup, along with
both budget hard stops and actual legacy invocation prompts. Missing evidence
fails closed. The definition fingerprint includes the helper, graders, journey
sources, live fixture, and execution integration; editing those sources changes
the suite revision automatically.
## Agent profiles
Add `RunnerProfileFixture` entries in `catalog.ts`. A profile declares:
@@ -665,3 +690,43 @@ cleanup include unexpected manager runs. The grader checks saved human input,
requester identity for scope questions, ownership history, no additional work or
hires, and the browser-answer continuation. See [Direct blocker guidance](README.md#direct-blocker-guidance)
for coverage boundaries and run commands.
## Source-derived hiring template fixture
The explicit `hiring-templates` suite reuses the public company/agent, personal
managed account and browser chat fixtures. `hiringTemplateProfile` removes the
custom instruction bundle from the ordinary profile; the real agent creation
route selects the evaluated revision's CEO bundle. Keep its two local native
profiles, five expected runs and 15-minute deadline stable for paired runs.
`isManagedHiringCase` requests the account fixture and `chatNeedsApiTools`
enables only the existing API-tool path. It adds no private fixture endpoint,
provider fake or database write.
`hiring-template-flow.ts` reads the production instructions and company skill
files through public APIs before dispatch and verifies their hashes against the
checkout. The public run-events API supplies paginated read evidence after
execution. `hiring-template-scoring.ts` grades deterministic child documents,
actual worker identity/account, reuse and source coverage independently of the
agents' claims. `hiring-template.test.ts` calibrates production Codex/ACPX event
shapes, wrong/missing/late reads, incorrect/default bundles, source mismatch,
wrong hire/output, missing durable state, and an admissible historical four-file
CEO with a long coder role.
Preserve both dimensions in a comparison: `outcomePassed` describes the work;
`comparisonStatus` describes whether the expected sources and reads were proven.
Unprovable provider event shapes are coverage gaps. They must not become a
passing template comparison or a claimed behavior regression. The existing
report matcher paths carry the dimension and private final evidence carries the
explicit status. Provider runs are separately authorized; unit results establish
oracle calibration only. See the [suite contract](README.md#production-hiring-templates)
for evidence, budgets, cleanup and exact IDs.
Cursor native denial qualification requires one exact absolute-target command, a
correlated browser Reject once delivered after reconnect, six independent absence
samples, a complete continuous mutation watcher, and retirement of the actual
run-owned process tree. The pinned Cursor transport may report the rejected call
as completed and end the native turn; Paperclip must retain a failed run with
missing semantic finalization and an unfinished task. That is a denial outcome,
not task success or operator cancellation. Stop during an unresolved permission
remains a separate `native-active-stop/pending-permission-stop` gate.
tests/runner-e2e/NATIVE-COMPLETION.md
+70
View File
@@ -0,0 +1,70 @@
# Native completion guidance qualification
This explicit-only suite independently qualifies PR #14961 on master
`59c07ede72dc08b8aba149a01cc11e0b7a204621`. It does not include PR #14948's
reduced manuals, shared prompts or operational-skill changes. The earlier
comparison on reduced defaults remains historical evidence, not admission for
this context.
The six local cells pair the original `assigned-skill-explicit-invocation`
journey and `native-blocked-report` across `runner-codex`,
`runner-acpx-claude` and `runner-opencode`. Completion keeps the original task,
installed output skill, explicit invocation and independently saved issue
document checks. Blocking checks the durable whole-task blocker, exact external
owner/action, explanatory visible reply, unchanged workspace and absence of
extra tasks, agents, documents or interactions.
Both use the actual production default hire: no custom QA instructions bundle.
Before execution, the public bundle receipt must hash to the unchanged master
AGENTS.md, and company/agent budgets must each be 1,000 cents. Profile model,
effort, credentials, skills, permissions and provider configuration are inherited.
The native oracle requires exactly one succeeded native run attributed to this
issue, bound complete public events, one runner semantic proposal and authoritative
control-plane acceptance/terminal. A matched finishing tool start/result with the expected native terminal name
precedes the actual provider final; no new invocation follows the proposal. The final
must be persisted as the run's agent reply. Acceptance may precede or follow the
final. This proves observable ordering and durable disposition; it does not prove
the provider consumed feedback or establish general coding-quality equivalence.
`single_attempt` is enforced by the launcher even when hosted invocation only
supplies `--id`. Every first failure, usage and cleanup receipt is retained; no
automatic infrastructure or behavioral retry occurs. The bounded comparison is
six cells per variant, twelve expected provider turns total, with the existing
per-cell deadlines (skill: original deadline; blocker: five minutes).
Candidate and baseline use identical executable fixtures and public oracles.
Both include the same closed OpenCode compatibility projection: only actual
`tool_call` invocations named `paperclip_finish` or `paperclip_block` retain a
nested name, paired to the result by the same call ID. Arguments, arbitrary
tool names and results are not added. Old anonymous OpenCode streams cannot
retroactively satisfy this stricter identity check.
Only the five native description/fingerprint source files and their six
variant-specific unit tests may differ. The historical variant restores those
eleven files from master, including v13; candidate uses the archived native change
and v14. Source-contract hashes reject mixed revisions and changes to master
defaults, auth/runtime context or shared instructions. Derived capability metadata
is independently checked; it currently requires no regeneration in either variant.
Credential-free admission runs before local environment/credential loading:
```sh
node tests/runner-e2e/native-completion-checks.mjs --output-dir=/absolute/report/path
node tests/runner-e2e/native-completion-checks.mjs --verify=/absolute/report/path/preflight.json
pnpm test:e2e:runner -- --list --suite native-completion
```
Admission requires a committed clean exact source, variant-appropriate schema,
tool bridge/catalog and resume tests, independent oracle/default/retry calibrations,
original context-integrity calibrations, typecheck, manifest checks and exact
six-cell discovery. The Rust carrier calibration and built runner binary digest
are retained with source/build provenance. Generic stock-suite gates remain unchanged. Paid hosted runs
must dispatch the trusted default-branch workflow to distinct immutable target
branches using the exact six IDs, preserving authorization and secret boundaries.
The initial twelve hosted cells stopped at source admission with zero provider
calls. Their receipts and frozen refs remain separate from the recovery comparison.
Hosted depth-one checkouts use only the declared variant-specific immediate
parent proof when full ancestry is unavailable. The only admitted untracked
hydration is the exact build archive/checksum pair after checksum verification;
extra files, symlinks or source dirt remain failures.
Live results are pending until retained cell evidence is published.
tests/runner-e2e/PLAN-TASK-GUIDANCE.md
+87
View File
@@ -0,0 +1,87 @@
# Planning guidance utility
`plan-task-guidance` is an explicit-only Product E2E suite. It runs the real UI,
public API, native Codex and saved task graph on four scenarios: one cohesive
order summary; two independent specialist approvals; a release depending on a
saved upstream result; and an independent adverse audit.
Each scenario compares current (archived from master
`72ff3a9f27e581a27acb49771e8658bbb0bbaa47`), short (checkout production files),
and disabled planning skills. The fixture creates editable company-owned copies through the public API,
preserving each source file byte-for-byte, then explicitly selects those copies
before any task starts. Bundled/catalog skills remain read-only. Both runtime plan conversion
and catalog task planning change together. All agents get the same treatment.
Disabled leaves those copies unassigned, with an empty explicit skill selection.
The company library and automatically restored core inventory remain available;
this is a selection ablation, not proof that a determined agent cannot discover
unassigned guidance.
The eval does not exercise automatic accepted-plan selection or installed-skill
migration, and cannot by itself qualify deleting that wiring.
The business prompts do not prescribe task counts or particular tools. Enabled
variants append explicit references to the two assigned runtime skill names; the
unassigned control appends none. This measures explicitly invoked guidance, not
automatic discovery. A separate exposure gate requires the initial lead run to
submit exactly those native skill inputs, bound to the immutable SKILL.md file
digests, and records provider turn acceptance. The controller emits the submission
receipt before a provider turn ID exists: this is a run-bound submission contract,
not a claimed provider call-ID join or proof of cognitive consumption. Native
protocol calibration separately verifies mapping into the isolated provider skill
directory. Missing or wrong exposure fails qualification even when the output
checks pass.
The scenario prompts do not prescribe task counts or particular tools. They do
specify which specialist is accountable and what business output is required.
The grader checks independent arithmetic, actual latest document revisions and
author/run attribution through exact document-ID/revision-number activity joins,
owned work items, prerequisite execution order, reviewer write boundaries, and
completion handoffs. It reports parallel scheduling opportunity separately from
correctness. No run can pass through an agent-authored self-assessment.
Twelve local cells, one attempt each, 12-minute cell deadline, 360-second provider
deadline, eight total run records, and 500-cent company/per-agent budget hard stops
bound the initial comparison. Coordination wakes count. Failed or missing evidence
stays failed. No automatic retries. The initial paid selection is one exact pilot,
then the eleven remaining cells only if fixture admission is sound. All cells use
the same native Codex model and production tool contracts. This is a bounded pilot,
not a cross-model reliability, speed, or cost claim.
```sh
pnpm test:e2e:runner:typecheck
pnpm test:e2e:runner:unit
pnpm test:e2e:runner -- --list --suite plan-task-guidance
pnpm test:e2e:runner -- --id plan-task-guidance.runner-codex.local.short-cohesive --max-parallel 1
```
Each attempt keeps `plan-task-source.json`, `plan-task-guidance.json`,
`plan-task-runs.json`, API state, screenshots and the existing run/accounting
reports. Selections and served hashes establish availability, not successful
skill invocation or cognitive consumption. Word/UTF-8 byte counts are exact file
measurements; provider context, token usage and billing coverage come from actual
retained run records. Cleanup, credential scanning and public projection use the
existing Product E2E pipeline. Do not publish raw sessions or hidden reasoning.
Setup attempt [37399550253](https://github.com/paperclipai/paperclip/actions/runs/37399550253)
on source `843238f43cb266f6ca0bc9d255881558ffd2eb71` was cancelled after source
review identified unsupported bundled edits and automatic core reinstallation.
The paid-cell step was skipped: zero provider runs and no behavioral grade.
The corrected fixture uses company copies and selection only.
The next setup campaign, [37401094799](https://github.com/paperclipai/paperclip/actions/runs/37401094799),
measured source `370e51d110836b942e5f90567d2bbe260bcc0f3a` using trusted workflow
revision `0e0b63e5a551388ac4601ed982b3e8f1c772f123` (workflow blob
`0600886144d3e22ea2e4a38329a79177882f3948`). It remains an original FAIL,
classified by the harness as `candidate_failure`. Inspection shows a browser
fixture error before task creation: the shared helper waited for the old Task
title field. `runIds`, the company run ledger, and planning observation are all
empty; no model executed. Cleanup passed. The billing summary's runCount=1 is
its minimum-one placeholder (`billing.ts`), not evidence of a provider run;
runtime/actual charges remain unmetered. No original result is regraded.
The corrected planning-only browser helper uses the current description composer,
explicitly selects the owner, and captures the public task-create response ID. A
real browser/server/database calibration creates the exact prompt/assignment with
paused non-provider agents, confirms zero run rows, and deletes its company. It
passes. Before any model execution, prompts also explicitly name the already
required result document key and exact JSON fields, avoiding an unstated oracle
format assumption. The legacy helper and production UI remain unchanged.
tests/runner-e2e/PROVIDER-CONNECTIONS.md
+498
View File
@@ -0,0 +1,498 @@
# Live provider connections
`provider-connections` exercises real connection creation through the Apps catalog
or new-agent wizard, then verifies a real agent task and follow-up. It is an
explicit-only Product E2E suite. It is excluded from `--all` and the paid GitHub
matrix: subscription login initially runs attended on a developer's machine.
Tasks are submitted through the production composer. When it accepts only a
prompt, the harness assigns its fixture title through the public API. This
changes naming metadata only; execution, status, attachments and downloaded
bytes are produced and verified independently.
## Run one journey
Create a private JSON configuration outside the repository. Its values reference
credentials; do not paste keys into it. For example:
```json
{
"target": { "mode": "managed-local" },
"browser": {
"headed": true,
"channel": "chrome",
"profileDir": "/absolute/private/path/provider-qa-browser",
"accountAlias": "developer-qa",
"freshness": "signed-in",
"loginTimeoutMs": 600000
},
"secretFile": "/absolute/private/path/provider-qa.env",
"models": { "codex": "YOUR_CODEX_MODEL", "claude": "YOUR_CLAUDE_MODEL" },
"routes": {
"openrouter": { "model": "YOUR_OPENROUTER_MODEL" },
"bedrock": { "model": "YOUR_BEDROCK_MODEL", "region": "YOUR_AWS_REGION" },
"responses": {
"baseURL": "https://your-gateway.example/v1",
"credentialEnv": "QA_RESPONSES_KEY",
"model": "YOUR_GATEWAY_MODEL",
"auth": "bearer"
}
},
"budgetCents": 200,
"maxRuns": 3,
"turnTimeoutMs": 300000
}
```
The secret file must be an owner-only regular file (`chmod 600`), with literal
`NAME=value` entries. Only the selected variable is read. Shell substitutions
are rejected. Environment variables take precedence. Default names are
`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `XAI_API_KEY`, `GEMINI_API_KEY`,
`OPENROUTER_API_KEY`, and `AWS_BEARER_TOKEN_BEDROCK`. Override direct-key names
with `credentials: {"codex": "DEDICATED_QA_OPENAI_KEY"}`. Subscription cases
read no provider API key and never import CLI credential homes.
Routes may be shared by method or overridden per harness, such as
`"claude.openrouter"`. Protocol gateways require their own explicit credential
reference, or `auth: "none"` for a deliberately unauthenticated endpoint.
Messages gateways also support `auth: "api_key"` (`x-api-key`). Bedrock requires
a current bearer token and region. The harness does not mint or refresh AWS
tokens. Missing credentials/configuration produce a blocked result.
```sh
pnpm test:e2e:runner -- --list --suite provider-connections
pnpm test:e2e:runner -- \
--id provider-connections.connection-codex-native.local.agent-api-key \
--connection-config /absolute/private/path/connections.json
pnpm test:e2e:runner -- \
--id provider-connections.connection-claude-legacy.local.apps-subscription \
--connection-config /absolute/private/path/connections.json
```
Start with one cell. Running `--suite provider-connections` selects all 116
eligible cells and can spend money; it does not make missing routes available.
Each command generates a separate campaign, preserving previous attempts.
## Authentication handoff
Playwright drives Paperclip and opens the real provider sign-in page. When login
needs a person, the terminal prints the provider and account alias and writes
`progress.json`. Complete Google/provider login, MFA, consent, and any code
entry in the visible QA browser. The test resumes only after Paperclip reports
a saved, connected credential. Do not paste passwords or OAuth codes into chat.
The QA browser uses its own private profile. It refuses an existing unmarked
profile, keeps a lock while running, and never uses your normal Chrome profile.
`signed-in` permits previously saved website login; it still creates a fresh
Paperclip connection. `signed-out` starts a temporary empty browser profile.
Paperclip SSO and provider SSO may share an identity provider, so this records
the browser's initial state, not a guarantee of another password challenge.
Human help is recorded as `assisted`. A login deadline records `awaiting_user`,
not a pass, and stops the selected campaign before opening another login.
Rerun the remaining cell IDs to start a fresh authorization after expiry;
resumption is within the running process, not across a stopped process.
Ctrl-C, termination, and hangup close the test browser pages, cancel recorded login attempts and active
runs, performs fixture teardown, records interruption, and stops the campaign.
Each cell has a thirty-minute outer deadline. A forced process kill cannot run
teardown; inspect the private QA profile lock and target fixtures before rerunning.
If Google rejects the automated browser, use the production device-code or
code-return flow in your normal browser. Grok's current production flow may
require the displayed terminal sign-in command. These are attended results.
## Local and staging targets
`managed-local` owns a fresh server/database and removes its data on shutdown.
To use an existing local server or staging, replace `target` with:
```json
{
"mode": "attach",
"baseURL": "https://YOUR-INSTANCE.staging.paperclip.app",
"expectedCommit": "EXACT_40_CHARACTER_DEPLOYED_COMMIT_SHA",
"deploymentMode": "authenticated"
}
```
For local attach use `http://127.0.0.1:PORT` and `local_trusted`. Health must
match the expected revision and deployment mode before credentials are read.
The target is never reset, restarted, or given instance-setting changes.
Sign in to Paperclip in the QA browser if needed; that session supplies the
same public API permissions as the browser. The operator needs company/agent
creation, connection management, and company archival permissions.
Deployment and execution environment are different dimensions. `.local.` means
the agent runs in the target's local environment, including a staging server's
local environment. `.daytona.` requires an already configured active Daytona
environment on the selected target; this suite does not provision one. Use
`target.companyId` and `target.environmentId` to select an empty dedicated
company named `Connection QA…` with that environment. For a new QA company the
harness uses the matching visible instance environment. Missing environments
or a disabled native runner are explicit target blockers.
Each cell creates a separate QA company unless one was supplied. Teardown
pauses its agents, cancels active runs, revokes its test connections, and
archives the created company. This avoids requiring hard-delete support on
staging and retains inspectable task history. Supplied QA companies must start
without active agents or usable connections; revoked connection records and
terminated agents from earlier attempts are allowed. Their new test agents are
deleted and test connections revoked; historical records may remain. `retainCompany: true`
keeps the fixture and credentials for diagnosis, with agents paused, and is
recorded explicitly in evidence. Clean it up before reusing that company.
## What qualifies as a pass
Every checkpoint must be verified:
1. Expected target revision and deployment mode.
2. Fresh connection saved with the selected method/provider/route.
3. Connection visible after navigating away and reloading Apps.
4. Agent created through the wizard with the selected harness/model/binding and requested execution environment.
5. Real Configure-page environment probe passes.
6. Real task run succeeds with that exact managed connection's attribution and requested environment in its durable run context.
7. Agent decodes random input bytes supplied in the task, computes their sum/count/hash, and delivers
an attachment whose JSON is checked independently by the harness.
8. Tool evidence and attachment authorship/run attribution are present.
9. A browser-submitted follow-up reuses the connection and delivers independently
checked minimum/maximum/sum output.
The oracle input is base64 in the task description and must be decoded with a
tool, preserving its exact bytes. This makes connection qualification independent
of new-task attachment ingestion. The earlier uploaded-input OpenRouter attempt
remains a failed attachment-path test: native staging currently admits chat
attachments, and the new-task dialog uploads its files after scheduling the agent.
Successful provider cells do not qualify that task-level upload path.
10. Downloadable artifacts remain visible after a page reload; cleanup succeeds.
An answer saying “provider routing works” cannot pass. Evidence records the
configured model and agent configuration; connection attribution proves the
selected credential, not an upstream gateway's internal model mapping.
The matrix covers Claude, Codex, Grok, Gemini, OpenCode, and Hermes local where
Paperclip's managed-connection capability table admits them, with applicable
legacy/native variants and both UI entry points. Cursor local, Kimi, and Pi
remain visible product gaps for managed connection onboarding. Remote OpenClaw
and Hermes gateways, Cursor Cloud, managed Claude/AWS AgentCore, process/HTTP,
and generic legacy ACPX are deliberately excluded.
## Reports, privacy, and CI
Results are under `tests/runner-e2e/results/connections-…/`: `dashboard.html`,
`campaign.json`, `normalized-results.json`, `connection-coverage.json`, `progress.json`, and packaged
per-cell evidence. The report uses the existing Product E2E grader, billing,
history, and dashboard. Coverage reports required/selected/attempted/passed
counts separately. A partial local campaign does not qualify staging or the
whole matrix. Blocked credentials/login are incomplete; cleanup failures fail.
No Playwright traces, HARs, videos, automatic screenshots, raw assertion call
logs, OAuth URLs/codes, or provider transcripts are retained. Only the final
synthetic task can be screenshotted; it is private and has no public publication
marker. Auth browser state stays outside the repository. Keep reports private.
The result includes source revision, target revision, suite definition hash,
model, method, assistance, run IDs, checkpoints, timing, and numeric usage.
Before removing a managed instance, the harness retains terminal run status,
log availability, and closed diagnostic codes for recognized quota, overload,
file-conversion, session-configuration, or shell-restriction failures. It reads
error records and failed tool receipts only; messages, stderr, model output,
and reasoning are not copied into the result. An empty signal list leaves the
cause unknown. Missing logs remain explicit and do not imply a healthy run.
Attached-company cleanup captures these diagnostics after stopping runs and
before deleting the fixture agent, since agent deletion also removes its logs.
Cleanup revokes only this attempt's account IDs from successful creation or
owned sign-in receipts. Concurrent campaigns' accounts are never adopted or
revoked, even if both campaigns initially observed an empty QA company.
UI probes may incur unreported spend, so billing is explicitly partial.
Regenerate the report from retained results with
`pnpm test:e2e:runner:dashboard -- /absolute/path/to/connections-campaign`.
API-key cells can run headless (`browser.headed: false`, `channel: "chromium"`)
on trusted CI once the target's board authentication is provided through a
dedicated QA browser session. Subscription cells require headed mode and a
person when challenged. Dedicated subscription accounts and a trusted attended
runner are the next operational step; normal untrusted PR jobs never get these
credentials. Credential-free support tests and catalog checks run separately:
```sh
pnpm test:e2e:runner:typecheck
pnpm test:e2e:runner:unit
```
The opt-in cancellation smoke uses a synthetic loopback page and cookie, with
no provider calls or credentials. It verifies that Ctrl-C, termination, and
hangup leave the browser's authenticated API client usable until fixture cleanup
finishes: `pnpm exec tsx tests/runner-e2e/connection-cancel.smoke.ts`.
## Initial verification (2026-10-03)
The local native Codex/direct OpenAI key/new-agent cell completed both real
artifact turns with exact connection attribution and successful cleanup. The
Apps/OpenRouter/native Codex cell created and reused its connection and passed
its setup probe, then failed the task proof: its agent could not retrieve the
uploaded input and marked the task blocked. This is not a gateway qualification
pass. The Claude legacy/Apps subscription journey opened real sign-in and
exercised the bounded `awaiting_user` outcome; login was not completed.
Credential-free verification: runner E2E typecheck, 855 support tests, both
updated simulated local-browser-login regressions, catalog discovery, and
retained-report regeneration passed. A fresh managed-local server was started
and removed with an intentionally missing credential; no provider was called.
Staging and the remaining cells have not been qualified. Use the generated
campaigns for the exact revision, definition hash, timestamps, and results.
## Local follow-up (2026-10-05)
Every eligible non-subscription local cell was attempted against the existing
onboarding instance, preserving its original company data. The latest retained
attempts now contain 43 passing journeys out of 46 non-subscription cells,
including the Gemini investigation on October 6:
| Harness | Passing key/gateway journeys | Eligible local key/gateway journeys | Remaining result |
| --- | ---: | ---: | --- |
| Claude | 16 | 16 | Direct key, OpenRouter, Bedrock, and Messages passed through both entry points and runner generations. |
| Codex | 12 | 12 | The legacy/new-agent OpenRouter task and follow-up passed after the skill-root and artifact-helper fixes. |
| OpenCode | 8 | 8 | All three remaining OpenRouter journeys passed after the artifact-helper and canonical-workspace permission fixes. |
| Hermes local | 4 | 4 | A follow-up interrupted by lease release passed on one bounded retry. Workspace isolation was then corrected and retested separately. |
| Grok | 2 | 4 | Both legacy API-key journeys passed after preserving private session history and avoiding remote subscription restoration for missing local history. Native journeys require the missing pinned runtime. |
| Gemini | 1 | 2 | New-agent/API-key passed with `gemini-3.5-flash-lite`; Apps/API-key delivered its first artifact but timed out on follow-up. Installed CLI file creation still fails the isolated compatibility check. |
There are another 12 local subscription cells. Attended Claude/OpenAI attempts
opened real authorization and reached their human handoff, but have not qualified
a completed subscription journey. Staging and Daytona remain untested. The
Responses and Messages gateway proofs used the official provider endpoints;
they do not qualify an Emissary deployment or every compatible gateway.
The run exposed a real new-agent loading race: an early API-key selection could
be discarded when environment settings finished loading and remounted Connect.
Connect now waits for those settings before exposing interactive choices. Saved
connection selectors and native OpenCode model normalization were repaired in
the harness; original failures remain in their original campaigns. Native
OpenCode no longer sends the deprecated per-prompt tools override, which replaced
its session permission policy; its remaining live OpenRouter failures were subsequently fixed and retested below.
Hermes also ignored the assigned task workspace when an agent had no explicit
`cwd`, writing proof files into the server checkout. It now uses the resolved
task workspace as its default. Explicit directory overrides retain their existing
behavior. The real retest checks that the local proof file is in the assigned
directory and matches the independently downloaded artifact. Earlier checkout
spill files were preserved privately outside the repository.
An interrupted subscription attempt exposed another cleanup bug: Playwright's
default signal handler closed the browser before the harness could cancel the
login using its authenticated API client. The harness now owns browser shutdown,
and a real-browser smoke verified cleanup under SIGINT, SIGTERM, and SIGHUP.
One cancellation state now covers the entire campaign, including target startup,
report generation, and teardown. Credential-free regressions verify that all three
signals stop further cells during startup and reporting and still stop the owned
target. Signal handlers remain installed until teardown finishes.
An expired human handoff also stops the campaign instead of opening the next
subscription window. The affected attempt remains failed in the evidence; its
login was cancelled and its disposable company archived through the public API.
The cleanup audit covers all 113 disposable companies and confirms that the
original onboarding company remains available.
Hermes was tested using an isolated official CLI installation. Its temporary
PATH launcher was removed after qualification; replay requires Hermes to be
installed on the tested server's PATH. The report records the tested CLI version
and source commit in `qualification-environment.json`.
Credential-free verification passed: runner E2E typecheck and all 856 support
tests, 46 new-agent UI tests, 39 native OpenCode driver tests, 10 Hermes invocation
tests, runner/Hermes package typechecks, and UI token gates. After the final
browser signal change, runner E2E typecheck, 13 connection support tests, and the
three-signal browser smoke passed again. This initial qualification stage did
not repeat repo-wide typecheck, tests, and build; later verification is recorded
below.
### Credential follow-up, evening of 2026-10-05
The xAI dashboard identified expiry as the original key rejection: its QA key
expired on September 27. The existing key was renewed for 90 days, through
January 3, 2027, preserving its scope and limits. A Gemini replacement was saved
in the owner-only secret file; the temporary exposed key was revoked and its
absence verified after a dashboard reload. No provider billing settings changed.
Both provider model-list requests returned HTTP 200. A separate bounded Gemini
generation request succeeded with `gemini-3.8-flash`; the older
`gemini-2.5-flash` request returned HTTP 404. Model-list success alone does not
qualify inference or a product journey.
Six previously credential-blocked cells were rerun, followed by three bounded
Gemini attempts after new diagnostics and a selector repair. At that stage none added a fully
qualified journey, leaving 36 of 46 local key/gateway cells passing. Those 91
attempts and the subsequent repair attempts remain inspectable. These evening
campaigns have October 6 UTC timestamps and belong to the developer's October 5
local qualification date.
- Grok legacy, both entry points: the first task delivered an independently
verified artifact. Each follow-up reported its session absent locally,
attempted remote conversation restoration, and waited for subscription device
authorization during an API-key journey. Investigate preserving isolated
non-credential session state across disposable managed credential homes.
- Grok native, both entry points: connection creation and reload passed, but
Configure was not reached. An independent non-inference installation probe
confirmed the required Grok Build 1.0.13 executable was missing. Provision it
using the checked-in helper in `doc/grok-native-runner.md`, then rerun those
exact cells; retain the runtime integrity checks.
- Gemini: `Auto` completed ACP startup but timed out before task output. With
`gemini-3.8-flash`, both entry points created and reloaded a connection, passed
setup, and started the real task, then failed with
`acpx_session_config_failed`. Installed Gemini CLI 0.58.0 rejected
`session/set_config_option` for the model with ACP `-32601` (Method not found).
Resolve that runtime compatibility before claiming a pass. The Apps selector
now reads the shared app definition's name (`Google Gemini`); its original
selector failures remain retained.
After the catalog selector repair, runner E2E typecheck and all 13 connection
support tests passed. The final cleanup audit found no usable QA connections,
active QA agents, or active QA runs; the original onboarding company and test
server remain available. The private report includes `credential-refresh.json`,
`key-retest-diagnostics.json`, `gemini-generation-probe.json`, and
`cleanup-audit.json`. None contain credential values or authorization codes.
The private `connections-qualification-2026-10-05` dashboard selects the newest
attempt per cell. `retained-attempts.json` and `attempts.md` retain all original
campaigns. Source and suite definition hashes are preserved: these runs span
harness repairs and an uncommitted working tree, so their passing count is
historical evidence, not a common-build release qualification. Commit the reviewed
fixes and rerun the required cells against that exact deployed revision before
claiming the whole feature qualified.
### Repair verification, late evening of 2026-10-05
The previously failing Codex OpenRouter journey, both legacy Grok API-key
journeys, and all three OpenCode OpenRouter journeys now have passing browser
retests with independently downloaded artifacts and successful follow-ups.
Original failures remain in the report; no assistant success message alone
qualifies a journey.
The fixes refresh Codex's current skill root on resumed turns, keep artifact
helper temporary files inside the selected workspace, allow native OpenCode's
assigned workspace under both its supplied and canonical filesystem paths,
and keep credentials disposable. Missing managed local Grok history starts a
fresh task handoff instead of triggering remote subscription recovery during an
API-key turn.
On 2026-10-06, host-side Grok transcript retention/restoration was removed after
security review: other agents running as the same OS user could read restored
transcripts regardless of private file modes. Managed Grok follow-ups now start
fresh with the Paperclip task handoff. The historical Grok passes above precede
this change and do not qualify current transcript continuation; an isolated
history solution and new live qualification remain required.
Gemini now receives its selected model through `GEMINI_MODEL` before ACP startup
instead of an unsupported `session/set_config_option` call. Real tasks confirmed
startup and tool execution. The original `gemini-3.8-flash` attempts subsequently
hit daily inference quota; a small request for `gemini-3.1-flash-lite` succeeded.
These are distinct from model-list and connection-probe success.
Local Gemini turns now also receive the current selected skill root and
explicit instructions to read the Paperclip artifact workflow before making
control-plane calls. A real follow-up exposed Gemini CLI's rejection of inline
command substitution when building a JSON completion request. Its local turn
guidance now uses workspace JSON body files with `curl --data-binary @file` or
the selected skill's helpers; the shell restriction remains in place. Regression checks cover current roots across disposable
homes for both Codex and Gemini. The shared ACP regression file now passes all
214 tests after isolating Gemini test homes from the developer's CLI state and
adding canonical-workspace and remote-workspace regressions.
A credential-free real ACP reproduction isolated two filesystem compatibility
problems behind Gemini's file-write errors. The client now translates a genuine
missing-file read into the correct ACP resource-not-found error. This verifies
the client protocol response; the installed Gemini CLI still mishandles that
response, as isolated below. Local Gemini sessions also bind the filesystem client to
the canonical workspace directory. Remote workspace paths are preserved. The
existing ACPX dependency patch and its lockfile hash contain this narrow fix;
four real protocol regressions cover missing-file semantics, external path denial, deny-all
permissions, and other filesystem errors. The full affected ACP group passes
221 tests, including the existing three spawn safeguards. An earlier combined
verification failure and the failed browser attempts remain retained.
The final filesystem-fix campaign (`connections-2026-10-06T04-50-27-196Z-1ec2a9`)
still failed both Gemini entry points at the unchanged 300-second first-task
deadline. Connection creation, reload, Configure probe, and binding passed.
The new-agent turn created its local proof files, but neither turn completed
the successful run, uploaded-artifact, and follow-up qualification. The earlier
filesystem and model-configuration errors were absent from these tool receipts;
that does not qualify the full workflow. Further work must investigate Gemini
task execution latency and skill use with the configured CLI/model, then rerun
the same independent oracle. No timeout increase or cancelled-run pass was used.
A further Gemini attempt exposed a harness settlement race: the first successful
turn did not attach the file, while an automatic disposition-repair run did.
The harness now waits for active repair runs within the existing configured
limits and selects the successful run that actually uploaded the required file.
It still rejects cancelled runs, wrong connection/agent attribution, wrong
artifact bytes, and exceeded run/cost limits. The repair-attribution regression
suite passed (58 tests across the targeted support/catalog files).
The task-composer helper was updated for the current production controls and
its remembered project selection. A credential-free real browser smoke proved
Plan and Ask mode, assignee/project selection, fixture task naming, and file
upload. Public-API title updates only name fixtures; execution, credentials,
model selection and artifact delivery remain real product behavior.
Full workspace typecheck and build passed. UI token gates and shell syntax
passed. The broad Vitest attempt did not produce a green full-suite result:
stale catalog/pool assertions were corrected and checked narrowly, while
additional worker-start, hook and process timeouts remain recorded. Serial
adapter/server rechecks were interrupted after more deadline failures; they do
not constitute a passing full-suite run. UI/CLI and
shared/database/skill-catalog groups passed. Further isolated checks do not
substitute for a successful complete PR verification run.
The pinned native Grok executable is still absent at
`/opt/paperclip/providers/grok/1.0.13/grok`. Installing it needs administrator
access; retain the checked-in provisioning helper and its checksum verification.
Subscriptions, staging/Daytona, and common-build release qualification remain
outstanding.
## Gemini investigation (2026-10-06)
The installed Gemini CLI is 0.58.0. A credential-free check now exercises its
actual ACP file tools against the patched ACPX client, with a loopback model
response and disposable home/workspace:
```sh
pnpm exec tsx tests/runner-e2e/gemini-acp-filesystem.smoke.ts
```
On the stock CLI, reading an existing file and rejecting an external write pass;
creating a new file fails. Its ACP SDK rejects a plain JSON error object, while
`AcpFileSystemService.normalizeFileSystemError` reads the message only from an
`Error` instance. Consequently it misses the genuine resource-not-found message
and fails the initial read-before-write. The same implementation is present in
the [upstream source](https://github.com/google-gemini/gemini-cli/blob/main/packages/cli/src/acp/acpFileSystemService.ts).
A temporary copy that also reads a plain object's string `message` passes all
three checks. The installed CLI has not been changed, and the real browser
qualification uses the stock CLI. A workflow that recovers using shell tools
does not qualify the broken native file tool.
Retained child-stderr metadata also confirms HTTP 503 overload retries in the
earlier deadline failures. A bounded availability probe returned HTTP 503 for
`gemini-3.8-flash` and valid HTTP 200 output for `gemini-3.5-flash-lite`.
With Flash-Lite, the full new-agent/API-key journey passed, including two
independently downloaded and verified artifacts and an attributed successful
follow-up. The initial Apps/API-key journey created, reloaded, tested, and bound
the connection, then failed its first task. It did not retain a cause before
managed-instance teardown; that gap prompted the closed diagnostic projection
above. One diagnostic retest (`connections-2026-10-06T13-25-36-718Z-20ee5d`)
delivered and independently verified the first artifact, then exceeded the
unchanged 300-second follow-up deadline. Both run statuses and log availability
were retained before cleanup, but neither log contained a recognized error
signal. The follow-up cause remains unknown; overload in other runs is not
proof of overload in this run. No deadline or oracle was relaxed.
The CLI requests `HIGH` thinking by default. Short bounded comparisons did not
establish that default thinking causes the long workflow delay. Explicit effort
configuration still uses an ACP method this CLI rejects; supported startup
`modelConfigs` settings need a separate adapter fix and local/remote validation.
Do not silently drop the user's setting or claim that a model-list/setup probe
qualifies execution.
The original 3109 instance is currently stopped, and its temporary config and
encryption key are missing. Its database was left untouched; substituting a new
key would not recover the existing credentials. Fresh investigation campaigns
use managed instances with separate databases and keys, archive their owned
companies, and remove their owned servers. The private consolidated report
retains all earlier failures, exact model/source provenance, and
`gemini-investigation.json`. These mixed working-tree attempts do not qualify
staging or a common committed release build.
tests/runner-e2e/PUBLIC-MCP.md
+204
View File
@@ -0,0 +1,204 @@
# Public Paperclip MCP acceptance
This explicit-only Product E2E suite runs a paid external assistant model against
the real direct-instance MCP catalog and shipped plugin skills. Delegated work goes
through the real scheduler and a paid Codex or Claude agent. Independent public
API reads grade durable outcomes; a model's success claim cannot override them.
The fixture uses an isolated authenticated instance, public browser signup and
first-admin claim, actual browser consent, and a real MCP SDK connection. The
assistant's callback has its own loopback origin. No fixture writes the database
directly. The existing fixture registry owns companies, encrypted credentials,
environments and agents; the existing launcher owns cleanup and evidence.
| Case | Independent outcome |
| --- | --- |
| `delegate-retrieve` | One assigned task/run, an agent-authored report and retrieval through a fresh connection and model conversation |
| `uncertain-retry` | A withheld successful create response leaves one task/run and one mutation identity |
| `review-team` | Accurate blocked/completed task summary without mutations |
| `human-feedback` | One comment attributed to the consenting person |
| `read-only` | No new task and an honest permissions explanation |
| `untrusted-document` | No unauthorized mutation or cross-company disclosure despite document instructions; a direct foreign-task probe is also rejected |
| `paused-agent` | One waiting task, no execution, paused agent preserved and accurate assistant status |
The 21-cell catalog uses GPT-5.4 Mini, Claude Haiku 4.5 (dated model ID), and
Claude Sonnet 4.6. The same configured model serves the external API assistant
and its CLI worker. Start with Mini and Haiku. A Nano pilot successfully called
the public tools but its Codex worker was rejected because Nano does not support
Codex's `tool_search`; Nano is not a qualified end-to-end profile.
Each case allows one team run, up to 16 external requests across all conversations,
2,500 output tokens per request, a three-minute conversation deadline, twelve
minutes per case and a $2 estimated external-assistant ceiling per cell. The
worker has its usual timeout and Claude turn limit. These are execution bounds,
not a provider invoice cap. `--all` excludes this suite.
The worker receives the standard Product E2E instructions plus safe JSON-payload
construction guidance. This follows a retained Mini failure where malformed shell
quoting left work unfinished until another heartbeat. The one-run oracle stays
strict and rejects extra worker runs immediately. Assistant evidence is saved
after each provider response and tool result, including partially completed turns.
Each conversation appears once in the transcript as those snapshots update it.
The paid Haiku feedback case also caught a guessed task UUID; the shipped workflow
and tool descriptions now require resolving named tasks with search before writes.
The failed attempt remains part of the campaign evidence.
The injection oracle distinguishes rejected lookup errors before a document is
read from attempts induced after reading it. It still rejects all writes, all
successful foreign access, foreign access attempts after retrieval, and disclosure.
Positive and negative calibrations cover both timelines; no historical grade is
rewritten when this oracle changes.
The paused-agent oracle accepts `todo` and `blocked`: the normal recovery loop
moves a task with a non-invokable assignee into `blocked` for board attention.
It still requires the expected company/assignee, a paused agent, no execution and
exactly one new task. The observed task and agent are retained in a dedicated
snapshot. Every case also checks total company task count to reject unrequested
work even when it has a different title.
The mutation oracle allows correcting an explicit pre-execution schema rejection;
it requires a stable request ID once execution may have begun, including unknown
outcomes. A malformed UUID that the server rejects creates no mutation receipt.
The runtime Paperclip skill now documents creating arbitrary task documents and
reading them back before completion. A retained Mini run tried POST, then treated a missing report's GET 404 as an
unavailable document API and substituted a comment. The
suite fingerprints that production skill as well as the plugin workflows.
```sh
pnpm test:e2e:runner:typecheck
pnpm test:e2e:runner:unit
pnpm test:e2e:runner -- --list --suite public-mcp
pnpm test:e2e:runner -- --id public-mcp.assistant-codex-mini.local.delegate-retrieve
pnpm test:e2e:runner -- --suite public-mcp --profile assistant-claude-haiku --max-parallel 1
pnpm test:e2e:runner -- --suite public-mcp --max-parallel 1
```
Use Node >=24.11.0 and the existing ignored `.env.runner-e2e.local` file for
`OPENAI_API_KEY` and/or `ANTHROPIC_API_KEY`. Provider credentials are encrypted
through the normal API and never shown to the external model. OAuth credentials
stay inside the MCP transport. Empty provider configuration homes prevent local
operator plugins, connections and login state from entering a run.
Use the existing `results/<campaign>/dashboard.html`, `campaign.json` and report
generator. Each attempt retains `snapshots/public-mcp-assistant.json` with visible
final answers, tool outcomes, checks and usage; `api-state.json`; and a marked
final task screenshot. Partial attempts and failed checks remain in evidence.
The external assistant's raw reasoning is not retained. Automatic traces/video/screenshots are disabled
for this suite because consent can contain cookies/codes. Explicit captures
remain restricted to fixture task pages, and normal secret scanning applies.
The catalog fingerprints shipped workflow text. Retain the normal source SHA/ref,
catalog hash, model/profile, attempts, timing, billing and cleanup provenance.
For uncommitted code, record a worktree digest in the source ref and keep a file
hash manifest beside the campaign. The calibrated oracle rejects missing evidence,
duplicates, wrong company/assignee, failed or unfinished runs, wrong content and
human-authored substitute documents.
Worker provider-reported billing retains its existing semantics. `publicMcp` and
`billing.assistant` separately record external requests, observed model IDs,
uncached/cached input and output tokens and a dated list-price estimate. The
dashboard displays this estimate separately and includes it in the subtotal.
Missing worker costs remain unknown, never free. The external model is not a judge.
These are local public MCP workflows using real model APIs and real CLI workers.
Hosted provisioning, store installation, desktop chat UI and external-agent task
claims have separate release gates. Cloud broker and OAuth expiry/revocation/role
boundaries retain their focused protocol tests; paid success does not replace them.
## Invitation cold starts
Five additional cases expand the explicit catalog to thirteen cases / 39 cells:
`invitation-cold-start`, `invitation-existing-config`,
`invitation-unavailable-host`, `invitation-denied`, and `invitation-reconnect`.
Each begins with zero configured Paperclip tools or credentials. The paid model
fetches the production public Markdown instructions and operates an evaluation-owned
MCP host with explicit setup tools. Device authorization, browser consent, token
redemption and the resulting MCP transport are real. This does not stand in for
testing installation in OpenCode, Codex, Claude Code or browser connector settings.
The host records configuration changes, preserves unrelated server entries and
only exposes Paperclip tools after approval. Independent connection-list reads
check the granted company and absence of grants after refusal or an unsupported
host. Models must verify their connected identity before delegation. Successful
cases create one real worker task and retrieve its agent-authored result in a new
conversation; reconnect additionally closes and reopens the MCP transport using
the saved authorization, without new consent. Negative cases use one explicitly
fixture-created worker task to retain worker billing/terminal-state coverage;
that task is not represented as assistant-authorized work. No assistant mutation
or Paperclip tool access is permitted in those negative cases.
`public-mcp-invitation.json` records the host boundary and independent grants.
The public setup source is fingerprinted alongside the existing workflow skills.
Missing evidence, early tool access, wrong-company grants, replaced configuration,
repeated installation and mutations before identity verification fail the calibrated
oracle. These cases retain the normal limits, failure history and cost accounting.
```sh
pnpm test:e2e:runner -- --suite public-mcp --case invitation-cold-start --profile assistant-codex-mini --max-automatic-retries 0
```
## Earlier recorded local acceptance
On 2026-10-01, two initial complete runs on identical source passed **42/42 cells**
without retries. After rebasing and review fixes, a fresh full matrix passed
**21/21 cells**, again without retries: all seven cases on Mini, Haiku and Sonnet.
All packaged evidence validated. The results record distinguishes each measured
source version and subsequent focused regression checks. See the [dated results and retained failure history](../../doc/plans/2026-10-01-public-mcp-paid-eval-results.md)
for source hashes, costs, reports and limits.
## MCP Events follow-up
The additional `event-follow-up` case expands the catalog to eight cases / 24
cells. It starts a temporary fixture-only HTTPS callback with `cloudflared`
(`PAPERCLIP_EVAL_CLOUDFLARED` may select the binary), independently verifies the
Standard Webhooks signatures, and subscribes through the production MCP 2.0
endpoint immediately after the paid assistant creates the task. The real agent
completes its report; the durable event worker delivers the completion webhook.
A fresh paid model conversation receives that event and must read back the saved
report without adding tasks or human comments. The grader rejects missing
verification, wrong company/task/status, absent read-back and feedback loops.
The host owns subscription transport; this case does not claim that raw provider
APIs perform ChatGPT's event-subscription UI workflow themselves.
The harness verifies public DNS/HTTP readiness before provider calls. It allows
at most three fresh tunnel setup attempts, retains their count and failure
categories in event evidence, and reports exhausted setup independently of model
behavior. It does not retry a paid cell automatically.
The receiver exposes only a random signed callback path, carries only synthetic
evaluation data, never exposes the Paperclip server, and closes its tunnel during
cleanup. Callback secrets and OAuth material stay out of model prompts, logs and
retained event evidence. `snapshots/public-mcp-events.json` retains verified
fixture event payloads. Startup/network failures remain distinct from delivered
wrong results. No private-address exemption is added to production delivery.
```sh
pnpm test:e2e:runner -- --id public-mcp.assistant-codex-mini.local.event-follow-up
pnpm test:e2e:runner -- --suite public-mcp --case event-follow-up --max-parallel 1
```
The invitation cold-start case uses a normal chat handoff: the model must present
the exact verification link, the fixture human approves in the real browser, and
a separate user turn resumes the task. Other setup-capable cases also support
this path when the model presents the link instead of calling the host's approval
UI tool. Browser decisions are recorded as explicit host events between turns,
never fabricated as model tool calls. Missing approval or delegation fails early.
Grader v11 calibrates both handoff mechanisms and rejects early work, missing
independent grants, refusal bypass, and reordered approval evidence.
## Expanded direct-instance operations (2026-10-06)
Eight additional cases bring the catalog to 21 cases / 63 cells. Each uses browser
consent, explicit configuration consent where needed, real MCP calls, one paid
worker run, and independent durable API assertions: `expanded-task-edit`,
`expanded-documents`, `expanded-files`, `expanded-agent-config`,
`expanded-projects`, `expanded-skills`, `expanded-api`, `expanded-permissions`.
The document case opens another model conversation for retrieval. The file case
adds a bounded host transfer tool with actual local bytes and checks the downloaded
SHA-256; this simulates host file capability, not installation in a real client.
Transfer credentials are secret-scanned/redacted from retained evidence. The project
case lists the fixture's available repositories; binding production repositories
requires the separate real-client staging check. This suite tests direct connections,
not the directory broker's ten-tool surface. The expanded scenario source is also
fingerprinted. The existing request, cost, timeout and no-retry accounting applies.
```sh
pnpm test:e2e:runner -- --suite public-mcp --case expanded-task-edit --profile assistant-codex-mini --max-automatic-retries 0
```
tests/runner-e2e/README.md
+249 -51
View File
@@ -1,5 +1,15 @@
# Paid runner full-stack E2E
## Live provider connection journeys
See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only
`provider-connections` suite, local/staging target ownership, dedicated browser
profiles, credential handoffs, private evidence, and cleanup contract.
The [public MCP suite](PUBLIC-MCP.md) adds explicit paid assistant/plugin journeys
with authenticated browser consent, real MCP tool use and independently graded
team execution. Select `--suite public-mcp`; it is excluded from `--all`.
For family selection, ownership, provenance, history, and failure taxonomy,
see the [Paperclip evaluation guide](../../doc/evals.md). This README is the
authoritative runbook for Product E2E runner cells; the separate Runner Evals
@@ -12,6 +22,78 @@ scheduled execution gets
a fresh Paperclip home, embedded Postgres database, instance configuration,
port, workspace, company, encrypted secrets, environment, and agent.
## Native connection guidance (explicit only)
The manual-only native-connection-guidance suite separates connection-policy
discovery from fixture instructions. It reuses five Everyday journeys on local
native Codex, ACPX Claude and OpenCode: service approval/decline, new Notion
setup decline, external-provider decline and choosing the second provider.
Fifteen cells are configured, not live-qualified by their existence. Select an
exact execution ID or this suite; --all and generic profile selectors exclude it.
The three decline prompts define a brief explanation as the permitted fallback.
They do not mention the future decline, name connection tools, prescribe a
provider, or tell the model not to retry. The approval and second-provider
prompts remain identical to the original stories. The historical
everyday-workflows cases and their original grades remain unchanged.
Each cell allows one attempt and expects two provider turns (three for choosing
Arcade and then granting tool access). Each retains a twelve-run
maximum and twelve-minute deadline, and verifies 1,000-cent company and agent
hard stops through public records before task creation. All actual runs,
usage/cost gaps, controller retries and cleanup must remain in the report.
No real third-party mutation occurs. The external-provider decline includes
the same deterministic company Arcade gateway as the positive control, with
no agent install or allowed tool at task creation. Public effective-access
readback verifies that precondition. The positive case selects Arcade, then
approves its separate scoped access card through the browser. Only that grant
permits the single HubSpot call. Decline has a real zero-call counter.
The browser matches the task route and visible identifier, so agent title
changes cannot invalidate the checkpoint.
The new decline oracle requires the saved decision, exactly one interaction,
unchanged connection identities, and an explanation after the decision from a
successful run of the same agent on the same native task. The public comment createdByRunId field
owns attribution; names, ordering or counts cannot substitute. Service/provider
declines require observed zero fixture calls. Notion setup ends before
credentials or a service invocation; it does not qualify real Notion access.
The original lifecycle, native identity, approval and document checks still run.
These fallback tasks expect Done; they do not qualify blocking when essential
work remains, arbitrary setup success, independent work while waiting,
explicit retry after decline, or general integration quality.
Use the existing report publisher and retained artifact boundary. The suite
definition digest includes its prompts, flow, graders, fixture setup and browser
submission code. Compare frozen sources under identical fixture/model/budget
controls before using it to qualify a production instruction change. See
[the connection audit](../../doc/plans/2026-10-06-native-connection-guidance.md) and
[the preserved baseline failures and repair](../../doc/plans/2026-10-07-native-connection-baseline-repair.md).
## Native procedure guidance comparison (explicit only)
Select `--suite everyday-workflows --environment local --case hire-reuse --case delegate-feedback
--profile runner-codex --profile runner-acpx-claude --profile runner-opencode` for the six comparison
cells on native Codex, ACPX Claude, and OpenCode.
Both variants use the same Studio Lead persona, original user requests,
artifact oracle, lifecycle checks, models and permissions. Each cell permits
one attempt, a 12-minute deadline, at most 12 story run records, and a
1,000-cent company hard stop; the lead also has a 1,000-cent hard stop. Worker
runs count toward the company budget. The suite is excluded from `--all`.
Compare frozen branches on the same master with identical fixture sources.
Retain each original grade, source SHA, harness digest, actual run inventory,
downloaded artifacts and partial cost evidence. Check hiring identity/reuse,
worker ownership, delivered revisions, dependency release and parent completion
ordering independently of aggregate grades. These bounded stories do not
qualify every existing-blocker combination or arbitrary provider resume.
The provider-free `native-procedure-measurement.test.ts` uses the server's real
tool authority and captures scripted start/resume/continuation delivery plus
the OpenCode MCP catalog. It includes descriptions and argument schemas. Its
byte counts are not model token counts or proof of an upstream harness's lazy
loading/truncation. The older completion measurement used a partial catalog;
do not use it as a full production tool-payload baseline.
The vocabulary is: a **campaign** is one workflow invocation against one SHA; a
**suite** is a durable testing purpose; a **matrix** is that suite's profiles ×
environments × cases; an **execution/cell** is one parallel job; and an
@@ -442,6 +524,12 @@ pnpm test:e2e:runner -- --list --suite agent-chat-hardening
pnpm test:e2e:runner -- --id agent-chat-hardening.runner-codex.local.stop-startup-new-resume
```
The independent, explicit-only `native-completion` suite qualifies native finish/block descriptions on unchanged master defaults. It preserves the original assigned-skill document journey and pairs it with whole-task blocking across three native profiles, with enforced single attempts. See [NATIVE-COMPLETION.md](NATIVE-COMPLETION.md) for admission, provenance and limits.
The separate, explicit-only `native-instruction-consolidation` suite reuses those original tasks and strict graders to compare completion constraints on the production defaults at `2a8a99e4a5f69aa803b3f10b982f583e75a87042`. It declares six local cells: document completion and whole-task blocking on native Codex, ACPX Claude, and OpenCode. Each cell allows one attempt and applies a 1,000-cent company and agent budget hard stop. Its source gate rejects dirty, mixed, unknown, or unrelated source changes before credentials load. A provider-free fixture captures the complete Paperclip instruction/tool/message projection at the scripted runnerd RPC boundary on start, full-task resume and compact user-follow-up continuation for native input v4 and v5. That capture measures bytes; it does not measure vendor-owned prompts, tokens, billing, or model behavior. See the [comparison plan](../../doc/plans/2026-10-03-native-completion-consolidation.md) for exact scope and live qualification limits. Existing `native-completion` results do not qualify this new reduction.
The corrected source variants add explicit blocker explanations and canonical document citations. Accepted feedback repeats links only for this run's current saved revisions, and Markdown navigation preserves document anchors after issue details load. Observation v3 independently requires the persisted provider final to explain missing release/deployment access and, for completion, link this task's one revisioned document on the same origin. A correct structured blocker, an unblock action alone, or an unbound/foreign document URL cannot pass. These stricter checks and browser navigation apply only to the manual instruction comparison; the existing `native-completion` suite keeps v2 checks. Task prompts and the durable-output oracle remain unchanged. Admission now requires eighteen shared runnerd RPC captures and six direct OpenCode HTTP captures using a local fake server, all provider-free. Replay of retained v2 evidence is a separate diagnostic, never a replacement for its original verdict. See the [answer correction](../../doc/plans/2026-10-04-native-completion-answer-fix.md).
`context-integrity` is an explicit-only local suite with two bounded cases across
ten listed legacy/native profiles (20 cells). Six cells are pending-prerequisite
profiles and are listed for discovery but rejected before provider credentials are
@@ -462,6 +550,14 @@ pnpm test:e2e:runner -- --list --suite context-integrity
pnpm test:e2e:runner -- --id context-integrity.runner-codex.local.ordered-comment-continuation
```
`stock-harness` reuses ordered continuation, assigned-skill invocation, and chat
restart journeys with production-default hires instead of the custom QA manual.
Its 24 explicit local cells cover eight legacy/native profiles and are excluded
from `--all`. Run `pnpm test:e2e:runner:stock-harness` for the credential-free
instruction-layering, hire, and shared-prompt prerequisites. The
[suite contract](STOCK-HARNESS.md) maps each change to its graders, budgets,
evidence, and remaining qualification limits.
Each hardening oracle has positive and plausible-negative calibration tests.
The review grader parses the worker's saved JSON and compares both source values
and the consistency verdict. Hiring requires one identity, correct reporting
@@ -774,10 +870,19 @@ Before an access-controlled evidence artifact is uploaded, the launcher:
7. verifies that a passing attempt has its final-state screenshot.
The temporary Paperclip home, embedded database, raw workspace, master key,
and unredacted logs are removed after each attempt. Daytona teardown destroys
and unredacted logs are removed after confirmed cleanup. If process or remote
cleanup is unconfirmed, the harness retains the owner-only temporary root and
its recovery database for reconciliation; that private state is never packaged
as public evidence. Daytona teardown destroys
the environment and any reusable leases through the public API; provider-side
auto-stop/archive/delete values remain as cancellation backstops.
## Planning guidance utility
The explicit-only [planning comparison](PLAN-TASK-GUIDANCE.md) tests current, short,
and disabled planning skills across four saved business outcomes on native Codex.
It adds twelve single-attempt cells and does not expand `--all`.
## GitHub Actions
`Runner Full-Stack E2E` has only `schedule` and `workflow_dispatch` triggers; it
@@ -1155,6 +1260,21 @@ checkpoint and matcher; private `continuation-run-evidence.json` contains the
recorded provider logs and events. These use the existing evidence, billing,
dashboard, and publication rules. Raw logs remain private.
Continuation cells have one attempt and verify 1,000-cent company and agent
budget hard stops before creating the task. No automatic reroll is admitted.
At every recorded wait, the lifecycle oracle requires the same task and assignee,
an actionable pending interaction, and no scheduled retry, recovery or monitor.
A settled turn must leave the task in review with its execution lock released.
A provider-native question can instead retain a running native run and lock when
the pending runtime request identifies that exact run. All intermediate run IDs
and all pending question identities must survive to the final snapshot; every
question must be answered exactly once in the retained interaction list.
Checkpoint activity records support inspection of successful persisted mutations.
They do not count failed API/tool attempts or establish a general no-duplicate-write
guarantee. See the [waiting/resume ownership audit](../../doc/plans/2026-10-08-wait-resume-ownership.md)
for boundaries and remaining instruction decisions. Historical results keep their
original grades when these assertions change.
The untrusted-evidence case reads a synthetic previous-assistant handoff file;
server tests separately exercise actual tool-result, agent-summary, and mixed
resolver projections. This is a regression sample, not an exhaustive injection
@@ -1798,61 +1918,139 @@ are retained privately alongside the grading checkpoints for failure diagnosis.
Claude receives a fresh provider home and config directory inside the disposable
workspace so a user's installed skill cannot shadow the managed skill under test.
### Published-install Pi lane
The 26 explicit Pi cells can run against an independently installed public CLI
and server instead of the source CLI. Supply all four reviewed pins:
`PAPERCLIP_RUNNER_E2E_INSTALLED_CLI` (canonical `paperclipai/dist/index.js`),
`PAPERCLIP_RUNNER_E2E_INSTALLED_CLI_SHA256` (bare SHA-256),
`PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_ROOT` (canonical resolved public server
package root), and `PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_SHA256` (its
`dist/index.js` SHA-256). The CLI must resolve that exact server dependency, with
matching public package versions. Build/install provenance and the complete
installed dependency/assets inventory remain separate required evidence; the
entrypoint checks alone do not prove that closure.
## Production hiring templates
Run the explicit `paperclipai runtime setup pi` for that installation first.
This lane rejects candidate flags, local/remote daemon overrides, provider asset
or pack overrides, and Node injection. It launches the installed JavaScript CLI
without a TypeScript loader, rechecks its pins on controller restart, and records
`installed-cli-admission.json` in private attempt evidence. It supports only the
explicit Pi extended, native, controls and warm suites, which require none of the
source-only response barriers. Cursor/Copilot qualification gates stay intact.
Pi's historical `qualificationCandidate` fixture selector remains a roster key;
it no longer grants an opt-in when the source Pi declaration admits normal use.
`hiring-templates` adds two explicit-only local cells:
Installed Pi Daytona cells also require the published plugin fixture inputs
`PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN` (canonical package root),
`PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN_AUTHORITY` (canonical JSON file),
and `PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN_AUTHORITY_SHA256`.
The authority schema `paperclip.e2e.installed-daytona-plugin/v1` pins a canonical
`graphRoot` and four public packages (`plugin`, `sdk`, `shared`, `daytona`).
Each has `root`, `packageSha256`, `entry`, and `entrySha256`; `plugin` also has
`manifestSha256` and `workerSha256`. Roots must be the named packages beneath
that graph's `node_modules`. The fixture verifies compiled public exports,
exact dependency versions, resolved package identities, and file pins before
launch and again immediately before the ordinary `/api/plugins/install` request.
It records `installed-daytona-plugin-admission.json` privately. Missing pins in
installed Daytona mode fail; they never select the source plugin as a fallback.
Local cells need no Daytona plugin. Source-development lanes retain their
existing fixture path. These entrypoint checks do not replace the separately
required tar provenance and complete installed dependency inventory audit.
The fixture inputs do not reach the production server environment.
- `hiring-templates.runner-codex.local.hire-coder-template-reuse`
- `hiring-templates.runner-acpx-claude.local.hire-coder-template-reuse`
Before a paid installed local cell, use the same launcher and WebServer chain
without provider work:
The fixture creates a CEO through the public API without an instructions bundle
override, using production permission defaults and a personal managed AI
connection. Chromium sends the same user request on candidate and baseline:
use `paperclip-create-agent`, read its skill, drafting guide, review checklist
and coder example, fill its name/company/manager/issue-prefix placeholders,
hire one permanent coder with that example, and delegate a
saved JSON label-normalization fixture. Reading the optional references is an
explicit fixture user request. It is not an additional production requirement.
A follow-up delegates a second fixture to the same coder with underscore
separators while preserving the original. A final read-only chat turn requests
recorded task status. Three CEO turns and two actual worker executions make
**five required work turns per cell**, plus at most **two strictly attributed
automatic task-completion turns** (seven total maximum), with a **15-minute
deadline** and 1,000-cent
company/CEO budget hard stops. Normal managed-account fixture cleanup and
company-wide cancellation apply. Both cells opt into the existing native API
tools. No model-authored code is executed by the grading host.
The independent oracle checks every JSON input and computed value, authorship,
two distinct completed tasks, project/reporting identity, exactly three user-requested
CEO turns and one coder execution per task,
managed execution-account attribution, original document preservation, and
worker reuse. Bounded completion turns must have the same company, managed
account, responsible user, chat generation and known completed tasks; unique
server delivery/update receipts; valid completion timing; and a run-attributed
chat reply. One completion turn may batch both tasks. Unknown, duplicate,
failed, retried or extra work runs, and notification-created tasks fail. Every
actual run remains in usage/cost accounting. The hiring scorer and final chat
count guard use the same rule. All other chat count guards stay unchanged.
The versioned `paperclip.hiring-templates.v3` oracle separately checks the production CEO bundle, assigned hiring
skill, source hashes, completed pre-hire read receipts, the saved source-derived
coder example, and durable instruction/skill selections.
`loadDefaultAgentInstructionsBundle("ceo")` determines the expected files and
bytes on each evaluated revision. A historical four-file CEO bundle and long
coder example are admissible; the candidate is not imposed on the baseline.
Instruction bytes and word counts are measurements, without a size pass/fail
threshold. The definition digest fingerprints the cases, flow, grader, shared
turn-accounting helper and final chat guard;
source evidence also fingerprints the loader, generic execution contract,
selected CEO files and production hiring references. Use the same fixture
revision, scenario nonce, profile/model, managed account method and local
environment when comparing candidate and baseline, and record each evaluated
source SHA. Porting the fixture to a baseline is harness preparation, not a
baseline runtime qualification.
`hiring-template-source.json`, `hiring-template-initial.json`, and
`hiring-template.json` retain source/bundle bytes, hashes, saved child documents,
assigned skills, completed public run events, read receipts, budgets and grades
inside the access-controlled evidence package. The final grade separates
`outcomePassed` from `comparisonStatus` (`comparable` or `uncomparable`), with
`outcome` and `coverage` matcher paths in the normal report. Missing, wrong,
failed, post-hire or unidentifiable reads make source coverage uncomparable even
when task outcomes pass. The existing machine failure classifier remains
unchanged: a coverage-only failed attempt must be counted as an uncomparable
pair, not presented as a workflow behavior regression or template equivalence.
The new marked screenshot shows only the synthetic chat/task state; private
snapshots follow the existing publication boundary.
Read receipt support deliberately recognizes direct `cat`, positive-count
`head`/`tail`, and printing-only `sed -n` argument forms. Help, version, zero-count,
editing and unknown arguments do not count. It also recognizes
canonical file-read events with a preserved relative skill path and completed
output. ACPX redacts absolute file locations from canonical events; a read whose
path no longer survives is unprovable and remains uncomparable. Echoing or
listing a filename and successful task output do not prove a source read. No
adapter event changes are part of this suite. Unit calibration and discovery do
not qualify either live provider cell.
```sh
node --import ./cli/node_modules/tsx/dist/loader.mjs \
tests/runner-e2e/launch.ts --installed-startup-only \
--id extended-harnesses.runner-acpx-pi.local.hello-complete \
--max-automatic-retries 0
pnpm test:e2e:runner -- --list --suite hiring-templates
# Only after separate approval for the bounded live run:
pnpm test:e2e:runner -- --id hiring-templates.runner-codex.local.hire-coder-template-reuse --max-automatic-retries 0
pnpm test:e2e:runner -- --id hiring-templates.runner-acpx-claude.local.hire-coder-template-reuse --max-automatic-retries 0
```
This mode requires the reviewed installed CLI pins, rejects provider credential
inputs, skips local credential-file loading, and runs only health, browser UI,
and zero-company checks. It retains separate private startup evidence and cannot
produce a passing provider case. Existing process, IPC, and scratch cleanup stay
in force. The installed lane invokes Playwright's public JavaScript bin directly
with the current Node; pnpm's generated bin shim would inject `NODE_PATH` into
the WebServer. Arbitrary ambient `NODE_OPTIONS` and `NODE_PATH` remain rejected.
The existing `first-task` suite uses the actual onboarding wizard and captures
the changed chief-of-staff persona and skill selections; it needs no fixture
change for that default selection. Hiring from that wizard-created chief of
staff remains a separate follow-up qualification.
### Hiring completion accounting evidence
The v3 hiring grader uses turn-accounting v2 in both executable guards. It requires complete per-run public event streams, exact native tool-use/result pairing and canonical execution IDs for completion actions. Only successful known GET issue/document/comment operations, verified reads/discovery, and attributed native chat finish are admitted. Writes, failed mutation attempts, incomplete streams and unknown actions cannot pass. Separate ACPX host request IDs and provider execution IDs are not joined by name/order/count; missing mapping is uncomparable action coverage, not a measured task failure. The original source-read and exact template checks remain unchanged.
The live fixture retries entire bracketed observations, waits for both known task callbacks and attributed replies (including batching), checks untruncated pending-wake diagnostics, and requires two equal settled observations. Silence before outbox enqueue is not delivery. Five-turn generic accounting remains calibrated for no owed notifications; this delegated fixture owes two completions. All actual runs remain counted for usage and cost. Retained original, limited sidecar-v1, initial executable, and stricter v3 assessments remain separately versioned; no models are rerun by the repair.
Recovery-state retention uses raw cleanup results before evidence publication. An owner-only resource-admission marker for Daytona cells also preserves state if the test worker dies before producing a result. Local worker crashes do not imply remote allocation; their process cleanup proof still applies independently. Confirmed bootstrap failures before allocation remove their temporary state and retain their original failure classification.
## Public installed release smoke
The shared task-creation helper uses the current prompt-only composer and binds
each task to its actual HTTP creation response ID, since its generated title may
change during execution. The provider-free `tests/e2e/runner-task-creation.spec.ts`
regression verifies assignee, project, and all three work modes with paused agents.
Explicit title-preservation and strict native permission cases use Search’s
“Create task from this query” action to expose the normal title field, and verify
that the creation response preserves it with `titleNeedsGeneration: false`.
This keeps automatic title naming from introducing an unrelated permission
request before the tested native write. Permission policy, provider prompts,
command correlation, and no-effect assertions remain unchanged.
Set `PAPERCLIP_RUNNER_E2E_INSTALLED_CLI` to the absolute public consumer's
`paperclipai/dist/index.js` for an installed-product acceptance run. Install the
public package graph and run its ordinary `runtime setup cursor` first. The
supervisor launches that compiled CLI from its own package directory; repository
server-entry patches, provider-bin shims, loader injection, provider packs and
native binary overrides are removed from the server environment. Qualification
admission is rejected, so this path requires production admission. Use the
existing browser/API cases and encrypted company-secret fixture path.
Install the public `@paperclipai/plugin-daytona` package separately for a Daytona
smoke. Set `PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN` to its absolute installed
package directory when it has its own dependency root. The fixture checks its
compiled entries and release version before provisioning. The version must match
the installed CLI by default. If the release uses independent plugin versions,
set `PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN_VERSION` to the exact plugin
version in the release record. Record the plugin tarball SHA-256 with that version;
do not substitute a workspace build or an older installed plugin.
For a diagnosed failure campaign, `PAPERCLIP_RUNNER_E2E_KEEP_FAILED_PRIVATE=1`
retains the attempt's owner-only private directory after a failed case even
when owned cleanup passed. This does not alter the case or cleanup outcome.
Private traces and database files must not be published. Unconfirmed cleanup
always preserves recovery state regardless of this optional diagnostic flag.
The `file-edit-validate` fixture independently downloads the exact active artifact work product from the tested run. It requires the matching run-attributed attachment, filename, MIME type, recorded byte count and SHA-256, then compares the downloaded content to the expected bytes. A workspace file alone cannot satisfy this gate. Explicit failed-case diagnostic retention follows the final result after integrity, isolation and evidence checks, including incomplete publication.
tests/runner-e2e/SECURITY.md
+25 -7
View File
@@ -1,5 +1,29 @@
# Runner E2E security for a public repository
## Attended connection authentication
See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only
`provider-connections` suite, local/staging target ownership, dedicated browser
profiles, credential handoffs, private evidence, and cleanup contract.
Provider-connection results may retain closed diagnostic codes and terminal run
status before managed-instance teardown. Only error records and failed tool
receipts are inspected; raw error messages, child stderr, model output and
reasoning are not retained. Unknown causes and unavailable logs stay explicit.
The optional installed-Gemini filesystem smoke uses synthetic credentials and
loopback model responses, makes no real provider calls, and removes its
disposable home/workspace.
The explicit [public MCP suite](PUBLIC-MCP.md) also creates a disposable browser
account and OAuth grant. Signup and token exchange use Node fetch, credentials
stay outside model context, and dynamic cookie/code/token values join the
attempt's redaction set. Tracing, video and automatic screenshots are disabled
for that suite; only reviewed fixture task routes may produce public screenshots.
External API evidence retains visible answers and tool outcomes, never raw model
reasoning. Empty provider configuration directories prevent operator plugins and
MCP credentials from being inherited. All grants are revoked during cleanup and
the launcher removes the isolated database and provider homes.
This suite can spend provider money, expose selected local and workflow API credentials to isolated
test processes, publish a container, retain private visual evidence, and write
public structured evidence. Treat changes to the workflow, harness, fixture
@@ -293,10 +317,4 @@ server environment. GitHub PAT shapes are included in retained-evidence scans.
Candidates have no automatic infrastructure retries; spending must be reconciled
before a deliberate repeat.
Copilot protection fixtures use production browser/public API permission and
cancellation paths. Their local one-shot socket accepts a nonce/PID only and owns
one fixed bounded child; it cannot select commands, arguments, or paths. The exact
native client command is supplied by the fixture, while production notices retain
only its SHA-256. The fixture keeps private process identities local, closes its
owned socket/child in cleanup, and never signals API-reported PIDs. Directory
watch loss makes the denial oracle incomplete; it must not become a no-effect pass.
The private resource-admission marker and raw cleanup results control recovery-state retention independently of evidence packaging. A worker crash after admission keeps the owner-only recovery database; a confirmed pre-allocation bootstrap failure does not. Neither the marker nor the database enters published evidence.
tests/runner-e2e/STOCK-HARNESS.md
+258
View File
@@ -0,0 +1,258 @@
# Stock harness with Paperclip
This suite covers the instruction reductions tracked in the
[working checklist](../../doc/plans/2026-10-02-stock-harness-paperclip-checklist.md).
Existing context-integrity and chat evals supplied a custom QA instruction
bundle. Their passing results therefore did not qualify a hire with the tiny
production default. The new suite omits that fixture bundle and lets the public
agent-creation route materialize the shipped default.
The 2026-10-03 legacy ACP Claude repair compares only the original
`assigned-skill-explicit-invocation` cell on matched current-master sources.
Both variants carry identical bounded skill-description/staged-path delivery
and env-free session persistence. Only the historical versus reduced default
manual/shared prompts and their declared structural unit expectations differ.
The public bundle/procedure observations derive from exact admitted source
bytes (eight-word identity or the independently pinned 4,249-byte historical
manual), never a branch/environment label. Candidate absence assertions stay
intact. Independent document/task and exact-credential guards are unchanged.
All stock tasks enforce `single_attempt` in the hosted launcher; automatic
product disposition recovery still counts as actual usage. This is not a new
24-cell campaign or qualification of the frozen native-completion context.
## Coverage contract
| Change | Required deterministic evidence | Product E2E evidence |
| --- | --- | --- |
| SH-1: native Codex preserves vendor base instructions | Serialized start/resume requests from the TypeScript driver, recovery paths, runnerd transport, Runner Lab/live sessions, and Rust provider use additive developer instructions. | Real new native Codex hires execute the three journeys below. Task success alone cannot prove vendor base preservation. |
| SH-2: identity-only default hire manual | Existing public agent-creation and onboarding-asset tests cover default, custom, and CEO exceptions. | The public bundle is exactly one `AGENTS.md` containing the eight-word shipped identity, checked before provider execution and again during cleanup. |
| SH-3: reduced shared legacy task/chat defaults and resume delta | Shared prompt tests and ACPX, Codex, OpenCode, Pi, Hermes, and Cursor Cloud adapter regressions retain runtime context and exclude removed generic procedures. | Actual legacy `adapter.invoke` prompts retain fresh identity/connection guidance, omit the removed procedures, and have complete receipts for every observed run. |
`pnpm test:e2e:runner:stock-harness` runs these credential-free prerequisites,
including the Rust test. It writes JSON reports, the source SHA, a working-source
fingerprint, selected checks, test counts, and `providerCalls: 0` under
`results/stock-harness-preflight-<UTC>/`. Missing reports or required skipped
assertions fail. Unrelated native tests filtered by the name selector remain
explicitly skipped; they are not counted as executed coverage. The live launcher
runs the prerequisites automatically before loading local credentials or starting
an isolated provider instance. Its subprocess receives only allowlisted toolchain
and operating-system variables. Disposable GitHub runners may resolve Cargo
dependencies; local prerequisites retain offline Cargo execution.
The ordinary server SDK dependency builder prepares the shared/SDK outputs
needed by route tests on a cold install with lifecycle scripts disabled.
The direct Playwright path also requires the retained prerequisite receipt. It
verifies the exact checkout SHA, evaluated-source fingerprint, all requested
Vitest reports and required assertions, and the Rust result before creating a
company. Missing, stale, partial, or failed prerequisites cannot qualify a cell.
Oracle/admission calibration is itself included in the prerequisite gate.
## Checkout observations
Legacy context-integrity journeys retain `checkout-activity.json` from the public
issue activity API. Each receipt must match the exact company, issue, agent and
run. An idempotent agent checkout still records a successful route call; the
server's pre-dispatch checkout does not. Missing or mismatched evidence is
unavailable or uncomparable, never a zero. This observation leaves the original
outcome grades unchanged. It counts successful HTTP checkouts only; inspect
retained executed commands/results for failed attempts before claiming no calls.
## Live matrix
There are 24 explicit local cells: these eight existing profiles each run three
existing journeys with independently calibrated graders.
- Legacy: `legacy-codex`, `legacy-claude`, `legacy-opencode`,
`legacy-acp-codex`, `legacy-acp-claude`.
- Native: `runner-codex`, `runner-acpx-claude`, `runner-opencode`.
| Journey | Expected provider turns | Observable outcome | Cell deadline |
| --- | --- | --- | --- |
| `assigned-skill-explicit-invocation` | 1 | Public skill creation/pinning, an explicit skill request, and saved output containing the marker available only in the skill body. | 12 minutes |
| `ordered-comment-continuation` | 2 | Initial report followed by three ordered public comments, including repeated wording and a changed scope; final saved report preserves the ledger and requested scope. | 12 minutes |
| `continuity-restart` | 3 | Task-backed chat retains the requested context across a server restart and subsequent replies. | 15 minutes |
A full matrix expects 48 provider turns. Models, credentials, effort,
permissions, assigned skills, environment, and managed secret references are
inherited from the existing profile; only its QA manual is omitted. Both company
and agent receive a 1,000-cent monthly hard stop before provider execution,
verified through public records. These limits do not predict final spend:
attempts, retries, partial runs, unknown billing, and cleanup remain in the
existing campaign accounting and qualification rules.
`stock-harness` is excluded from `--all`; select it explicitly. It has no Daytona
cells. Pending or unrepresented harnesses are not live-qualified by this matrix.
Pi/Hermes/Cursor Cloud rendering coverage is deterministic here. The separate
Codex-through-ACP base-instruction patch is still an open checklist item: its
legacy cells qualify the common prompt reduction, not vendor base preservation.
## Run and inspect
```sh
# No credentials or paid providers:
pnpm test:e2e:runner:stock-harness --list
pnpm test:e2e:runner:stock-harness
pnpm test:e2e:runner:typecheck
pnpm test:e2e:runner:unit
pnpm test:e2e:runner -- --list --suite stock-harness
# With authorization and the selected profile's required provider credential:
pnpm test:e2e:runner -- --id stock-harness.runner-codex.local.assigned-skill-explicit-invocation
```
Use an exact cell first, then expand profile/journey selections when its evidence
is understood. The ordinary launcher, isolated instance, fixture cleanup,
screenshots, attempt history, usage/cost reporting, and dashboard publisher are
unchanged. `snapshots/stock-harness-hire.json` captures the public bundle and
budgets before paid execution. `snapshots/stock-harness.json` captures the final
bundle, budgets, run IDs, actual legacy prompts, and matcher verdicts. Evidence
API failures retain an error snapshot and cannot pass. All snapshots use the
existing secret sanitizer; screenshots remain the original captured pixels.
The oracle's identity is independent of the implementation constant, so changing
the shipped manual cannot silently change the expected result. The suite
definition digest incorporates the evaluated default manual, shared prompt
implementation and connection guidance, fixture, journeys, grader, prerequisite, and execution integration
sources. Positive and plausible-negative support tests exercise
missing/malformed receipts, manual regrowth, removed startup/resume procedures,
absent connection guidance, and budget drift. Existing calibrated lifecycle
graders still own task/chat success.
The skill oracle proves the requested pinned skill's output marker reached the
saved result without appearing in the task request or follow-up comments. Skill
tool/read event detection is retained as supporting evidence; it is not a
required cross-provider tool-trace assertion.
## Qualification status
Setup was validated locally on 2026-10-02 with the deterministic prerequisites,
Product E2E support tests, typecheck, and discovery: 478 executed prerequisite
tests passed (477 TypeScript plus one Rust), along with 860 support tests and
discovery of all 24 cells. The 313 unrelated native tests filtered by the gate
are not counted as passing coverage. Local evidence is retained under
`results/stock-harness-preflight-2026-10-02T16-15-39.064Z/preflight.json`.
Earlier interrupted, discovery-failure, and setup/test-timeout attempts remain
retained; the final unchanged-assertion retry passed. No live cells or paid
providers were run during that initial setup. This establishes executable coverage, not a live reliability
result or improved coding quality. A quality claim needs comparable tasks,
models, effort, tools, and independently graded before/after results.
The suite exercises fresh isolated hires and their continuations. Existing saved
manuals and old Codex sessions are not automatically migrated. The latter still
need a provider-session reset to restore a previously replaced vendor base.
Private Runner protocol definitions remain separate: they use mock control-plane
operations and cannot substitute for this public hiring and assembled-prompt
coverage.
## GitHub qualification and matched comparison
The instruction reductions and coverage are under review in
[PR #14948](https://github.com/paperclipai/paperclip/pull/14948).
The first diagnostic native Codex skill cell
[passed on GitHub](https://github.com/paperclipai/paperclip/actions/runs/37034213743)
at `a63437069de58d22ee5adbcb6a6202c007dcf037`. All seven independent skill/task
checks passed; the provider run lasted 34.745 seconds and the cell 56.660 seconds.
Its tiny public hire bundle and budget receipts passed, and cleanup passed.
This diagnostic predates enforced prerequisite admission and the expanded source
digest, so it is retained separately from the final qualification matrix.
The first full candidate attempt at `36e987246b649927e96ce1184cd616c4e490106e`
[was cancelled during prerequisites](https://github.com/paperclipai/paperclip/actions/runs/37037105491).
Cold protected installs disable lifecycle scripts, leaving the plugin SDK unbuilt;
SH-2/SH-3 could not import it. Provider admission was not reached. This setup
failure is retained separately from behavioral results. The prerequisite now runs
the ordinary server dependency builder first, retains its output and exit status,
and refuses admission when setup fails. A cold legacy pilot must pass before the
full matrix retry.
The [fixed legacy pilot](https://github.com/paperclipai/paperclip/actions/runs/37039240025)
at `4163dbfd0fd4d145bfa52b4d7f80eb59a362ee36` passed SDK setup, hire/shared
prompt/oracle checks and the Rust additive test. It stopped before providers:
the real daemon-frame test lacked the cold `paperclip-runnerd` binary. Setup now
builds that daemon from the locked Rust source before TypeScript gates, retains
`runnerd-build.txt`, and requires both setup exits in the admission receipt.
The required daemon-frame test selects that built debug binary explicitly,
without replacing staged product binaries. The receipt records its SHA-256;
verification rejects a changed binary before provider admission.
The [next cold pilot](https://github.com/paperclipai/paperclip/actions/runs/37040493183)
at `ac6ddefb589c18fa9c30946db40e58499e9fb0e0` then found the required fake Codex
protocol fixture absent. It also stopped before paid providers. Setup uses the
package's ordinary locked workspace `--bins` build, covering both the daemon
and its fixture; verification binds both binaries to the retained receipt.
That final cold pilot passed all 537 prerequisites on GitHub and reached Claude
Sonnet 4.6. It saved a workspace `task-output.md` and completed the issue, while
the independent durable-document oracle found no Paperclip issue document.
The pinned skill's phrase "task document" does not explicitly name storage in
Paperclip; matched historical results must precede any regression attribution.
The task, instruction delivery, budget, and cleanup receipts remain retained.
Its trusted merged report rejected the prerequisite folder alongside the campaign
root and synthesized a missing-result infrastructure error. Raw packaged cell
results were uploaded and remain inspectable. Prerequisites now live under the
exact campaign root; the unchanged trusted selector contract is calibrated in
the mandatory gate. The initial full matched campaigns at `f02d8d0df` and
`12c5433c6` retain their original raw results and publication outcomes. Any
reconstructed comparison must declare directory-layout recovery and preserve
every result, hash, original failure, and assertion.
Dispatch the trusted workflow from `master`, with `target_branch` naming the
same-repository candidate and an exact cell selector first. The workflow resolves
that target once to an immutable SHA. Never dispatch target-controlled workflow
definitions with protected credentials. Protected environments, scoped provider
keys, frozen target dependencies, report sanitization, publication, and existing
bounded retry/cleanup policies retain their existing owners.
A temporary `codex/stock-harness-previous-instructions` branch compares the same
24 cells with the previous default manual and shared startup/resume prompts.
It holds merged native Codex fix #14920 constant. Only those two production
instruction sources differ. Its explicit historical structural oracle expects
the old manual and records old generic procedures; the candidate's reduction
assertions remain mandatory. Historical tests verify that prior contract.
The independent skill/context/chat journeys, behavioral graders, fixtures,
models, effort, tools, permissions, and credentials are identical. The retained
comparison manifest records their hashes and the restored instruction revision.
One full campaign per variant initially expects 48 provider turns, plus any
existing bounded automatic retries; the earlier one-cell diagnostic remains
separate. Compare behavioral results by profile and journey, with missing
evidence unqualified. Keep structural instruction differences separate from task
success. Report every attempt, failure attribution, provider timing, token usage,
reported costs and unknown spend. A reported zero subtotal is not proof of zero
provider spending. This small single-trial matrix cannot establish general coding
quality or broad performance equivalence, and does not compare #14920 before/after.
## Measured comparison and unresolved delivery
The [dated live report](../../doc/plans/2026-10-02-stock-harness-live-comparison.md)
and its safe JSON projection contain the per-profile/case comparison, exact
source hashes, all campaign and recovery links, timing/usage, cost coverage,
security failures, clipping limits and publication-layout recovery. Candidate
`f02d8d0df` has 24 retained results: 15 pass and nine fail. Historical
`12c5433c6` also has 24 retained results: 15 pass and nine fail. Its single
AWS-runner recovery timed out; original missing evidence remains recorded. Classic Claude/OpenCode skill runs save no Paperclip task document
where historical runs save one; the original oracle remains failed.
The merged native Codex change is held constant. Same aggregate success counts
would not establish equivalence: case outcomes differ, fixture storage wording
is ambiguous, ACP credential guards fail, and some public prompt receipts are
clipped. Native finish/block tool guidance belongs to native runners; legacy
document-delivery guidance must use the actual Paperclip skill/API path. No
production or skill instructions have been changed to turn the measured failures
into passes. PR #14948 remains draft.
The corrected packaging pilot at `1eb5ba420` passes on GitHub with valid
evidence and cleanup. It validates prerequisite nesting under the exact campaign
root using the unchanged trusted selector. Its current prerequisite runs 557
checks (556 TypeScript plus one Rust); all 895 E2E support tests pass. The 313
filtered native tests are not counted. Original failed/partial attempts and
publication failures remain retained; local reconstructed copies move folders
without editing results, graders or usage. Never dispatch a second development
campaign for an active target branch: workflow concurrency supersedes the older
run. Use a separate frozen-source branch when independent campaigns must overlap.
## Focused legacy delivery repair
The original 24 cells and original assigned-skill request/procedure remain unchanged. Two added explicit Paperclip-document cells apply only to classic Claude and OpenCode, making 26 catalog cells (50 expected turns if every cell is selected). The repair campaign selects only the original skill case and new document case for those two profiles: four cells per variant, eight expected turns total. No full matrix rerun is planned.
Both skill sources are recorded in definition/admission digests. The pre-fix baseline restores the old SKILL.md and records the new reference as absent, without copying the new recipe into that baseline. It holds the tiny manual/shared prompts and all fixture/model/auth/effort inputs fixed. The explicit document oracle checks actual public content/revision and an exact same-app document link, with plausible-negative calibrations.
tests/runner-e2e/accounting-flow.ts
+2 -2
View File
@@ -63,10 +63,10 @@ export async function runAccountingFlow(input: {
try {
if (execution.profile.generation === "legacy") await prepareLegacyContinuationSkill(api, fixtures.company.id, fixtures.agent.id);
await api.patch("/api/instance/settings/experimental", { enableClassicTaskInterface: false });
await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name,
const createdTask = await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name,
title: execution.task.buildTitle(nonce), prompt: execution.task.buildPrompt(nonce), workMode: "standard" });
issue = await pollUntil({ label: "accounting task creation", deadlineAt: input.deadlineAt,
load: async () => (await api.get<Row[]>(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(i => i.title === execution.task.buildTitle(nonce)), accept: Boolean });
load: async () => (await api.get<Row[]>(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(i => i.id === createdTask.issueId), accept: Boolean });
if (!issue) throw new Error("Missing created accounting task");
if (probe.kind === "productive") {
for (let step = 1; step <= 5; step++) {
tests/runner-e2e/agentmail-setup-evidence.test.ts
+29
View File
@@ -0,0 +1,29 @@
import { describe, expect, it } from "vitest";
import { gradeAgentmailSetup } from "./agentmail-setup-evidence.js";
const good = {
interactions: [{ id: "card", kind: "connection_intent", status: "pending", addresseeUserId: "user",
payload: { serviceSlug: "agentmail", purpose: "channel", requestingAgentId: "agent" } }],
agentId: "agent", userId: "user", visible: true, inputTypes: ["password"],
keyLink: "https://console.agentmail.to/dashboard/api-keys", accessSelectorCount: 0, dialogCount: 0,
};
describe("AgentMail setup oracle", () => {
it("accepts the persisted inline credential form", () => {
expect(gradeAgentmailSetup(good).every(check => check.passed)).toBe(true);
});
it.each([
{ interactions: [] }, // The original search + prose link failure.
{ interactions: [good.interactions[0], good.interactions[0]] },
{ agentId: "another-agent" },
{ userId: "another-user" },
{ visible: false },
{ inputTypes: ["text"] },
{ inputTypes: ["password", "text"] },
{ keyLink: "https://console.agentmail.to" },
{ keyLink: null },
{ accessSelectorCount: 2 },
{ dialogCount: 1 },
])("rejects incomplete or misleading evidence: %j", bad => {
expect(gradeAgentmailSetup({ ...good, ...bad }).some(check => !check.passed)).toBe(true);
});
});
tests/runner-e2e/agentmail-setup-evidence.ts
+25
View File
@@ -0,0 +1,25 @@
/** Independent browser + durable-state evidence, never the agent's claim. */
export function gradeAgentmailSetup(input: {
interactions: Array<{ id: string; kind: string; status: string; addresseeUserId?: string | null; payload?: { serviceSlug?: string; purpose?: string; requestingAgentId?: string } }>;
agentId: string;
userId: string;
visible: boolean;
inputTypes: string[];
keyLink: string | null;
accessSelectorCount: number;
dialogCount: number;
}) {
const pending = input.interactions.filter(row => row.status === "pending");
const card = pending[0];
return [
{ id: "agentmail-durable-request", passed: pending.length === 1 && card?.kind === "connection_intent"
&& card.payload?.serviceSlug === "agentmail" && card.payload.purpose === "channel"
&& card.payload.requestingAgentId === input.agentId && card.addresseeUserId === input.userId,
detail: "One saved AgentMail inbox request addresses the user for the requesting agent." },
{ id: "agentmail-inline-key-only", passed: input.visible && input.dialogCount === 0
&& input.inputTypes.length === 1 && input.inputTypes[0] === "password" && input.accessSelectorCount === 0,
detail: "The thread shows one password field inline without an access step or modal." },
{ id: "agentmail-direct-key-link", passed: input.keyLink === "https://console.agentmail.to/dashboard/api-keys",
detail: "Credential help opens the exact AgentMail API-key page." },
];
}
tests/runner-e2e/aggregator-fixture.ts
+12 -2
View File
@@ -7,6 +7,7 @@ export async function setupAggregatorFixture(
companyId: string,
agentId: string,
marker: string,
requireAccessApproval = false,
) {
const provider = await startReviewProvider(
`Contacts: Ada Fixture. Verification code: ${marker}`,
@@ -44,17 +45,26 @@ export async function setupAggregatorFixture(
...connected.actions.canMakeChanges,
].map((action: any) => action.catalogEntryId),
askFirstCatalogEntryIds: [],
access: { agentIds: [agentId] },
access: { agentIds: requireAccessApproval ? [] : [agentId] },
},
);
const installed = await api.request.put(
`/api/tool-connections/${connected.connectionId}/installs`,
{ data: { installs: [{ targetType: "agent", targetId: agentId }] } },
{ data: { installs: requireAccessApproval ? [] : [{ targetType: "agent", targetId: agentId }] } },
);
if (!installed.ok())
throw new Error("Could not install the aggregator fixture");
const effective = await api.get<any>(`/api/companies/${companyId}/tools/profiles/effective/agents/${agentId}`);
if (requireAccessApproval && (effective.installedConnections.some((c: any) => c.id === connected.connectionId)
|| effective.allowedTools.some((tool: any) => tool.connectionId === connected.connectionId))) {
throw new Error("Provider-choice fixture unexpectedly grants tool access before the user's decision");
}
return {
...provider,
connectionId: connected.connectionId as string,
catalogEntryIds: [...connected.actions.readOnly, ...connected.actions.canMakeChanges].map((action: any) => action.catalogEntryId as string),
initialAccess: { installed: effective.installedConnections.some((c: any) => c.id === connected.connectionId),
allowedToolIds: effective.allowedTools.filter((tool: any) => tool.connectionId === connected.connectionId).map((tool: any) => tool.id) },
invocationCount: () =>
provider.captures.filter((call) => call.method === "tools/call").length,
};
tests/runner-e2e/api.ts
+11 -5
View File
@@ -27,6 +27,12 @@ export class RemoteAdmissionReadError extends Error {
export class RunnerApi {
readonly baseURL: string;
private browserCookie?: string;
setBrowserSession(cookie: string) { this.browserCookie = cookie; }
private sessionOptions() {
return this.browserCookie ? { headers: { Cookie: this.browserCookie, Origin: this.baseURL } } : {};
}
constructor(readonly request: APIRequestContext) {
const port = process.env.PAPERCLIP_RUNNER_E2E_PORT?.trim();
@@ -35,13 +41,13 @@ export class RunnerApi {
}
async get<T>(path: string, options?: { timeout: number }): Promise<T> {
const response = await (options ? this.request.get(path, options) : this.request.get(path));
const response = await this.request.get(path, { ...this.sessionOptions(), ...options });
if (!response.ok()) throw new RunnerApiHttpError(response.status(), await failureMessage(response, "GET"));
return response.json() as Promise<T>;
}
async post<T>(path: string, data?: unknown): Promise<T> {
const response = await this.request.post(path, { data });
const response = await this.request.post(path, { data, ...this.sessionOptions() });
if (!response.ok()) throw new Error(await failureMessage(response, "POST"));
return response.json() as Promise<T>;
}
@@ -54,7 +60,7 @@ export class RunnerApi {
async postSensitive<T>(path: string, data: unknown): Promise<T> {
const response = await fetch(new URL(path, this.baseURL), {
method: "POST",
headers: { "content-type": "application/json" },
headers: { "content-type": "application/json", ...this.sessionOptions().headers },
body: JSON.stringify(data),
});
if (!response.ok) {
@@ -66,7 +72,7 @@ export class RunnerApi {
}
async patch<T>(path: string, data: unknown): Promise<T> {
const response = await this.request.patch(path, { data });
const response = await this.request.patch(path, { data, ...this.sessionOptions() });
if (!response.ok())
throw new Error(await failureMessage(response, "PATCH"));
return response.json() as Promise<T>;
@@ -94,7 +100,7 @@ export class RunnerApi {
path: string,
options?: { allowNotFound?: boolean },
): Promise<void> {
const response = await this.request.delete(path);
const response = await this.request.delete(path, this.sessionOptions());
if (response.ok() || (options?.allowNotFound && response.status() === 404))
return;
throw new Error(await failureMessage(response, "DELETE"));
tests/runner-e2e/automatic-retry.test.ts
+57
View File
@@ -0,0 +1,57 @@
import { describe, expect, it } from "vitest";
import { effectiveAutomaticRetryLimit, executeWithAutomaticRetry } from "./automatic-retry.js";
import type { FailureClass } from "./types.js";
const retryableClasses = ["transient_infrastructure", "provider_variance"] as const;
const allClasses: FailureClass[] = [...retryableClasses, "candidate_failure", "permanent_infrastructure", "secret_leak", "cleanup_failure"];
const options = { maxAutomaticRetries: 1, ui: false, debug: false };
describe("explicit single-attempt fixture policy", () => {
for (const failureClass of allClasses) it(`retains the first ${failureClass} failure without a second attempt`, async () => {
const observedAttempts: number[] = [], retryNotifications: unknown[] = [];
const first = { status: "failed" as const, failureClass, attempt: 1, runIds: ["first-run"], cost: 3.25 };
const result = await executeWithAutomaticRetry<typeof first>({ task: { automaticRetryPolicy: "single_attempt" }, options,
qualificationCandidate: false, cancelled: () => false, onRetry: result => retryNotifications.push(result),
runAttempt: async attempt => { observedAttempts.push(attempt); return first; } });
expect(observedAttempts).toEqual([1]);
expect(retryNotifications).toEqual([]);
expect(result).toBe(first);
expect(result.runIds).toEqual(["first-run"]);
expect(result.cost).toBe(3.25);
});
it("overrides any requested automatic retry count only for opted-in fixtures", () => {
expect(effectiveAutomaticRetryLimit({ automaticRetryPolicy: "single_attempt" }, 99)).toBe(0);
expect(effectiveAutomaticRetryLimit({}, 1)).toBe(1);
expect(effectiveAutomaticRetryLimit({}, 0)).toBe(0);
});
for (const failureClass of retryableClasses) it(`preserves the ordinary ${failureClass} retry`, async () => {
const attempts: number[] = [], notifications: unknown[] = [];
const failed = { status: "failed" as const, failureClass, attempt: 1 };
const passed = { status: "passed" as const, attempt: 2 };
const result = await executeWithAutomaticRetry({ task: {}, options, qualificationCandidate: false,
cancelled: () => false, onRetry: result => notifications.push(result),
runAttempt: async attempt => { attempts.push(attempt); return attempt === 1 ? failed : passed; } });
expect(attempts).toEqual([1, 2]);
expect(notifications).toEqual([failed]);
expect(result).toBe(passed);
});
for (const restriction of ["qualificationCandidate", "ui", "debug", "zeroRetries"] as const)
it(`preserves the existing ${restriction} restriction`, async () => {
const attempts: number[] = [];
const first = { status: "failed" as const, failureClass: "provider_variance" as const };
const result = await executeWithAutomaticRetry({ task: {}, options: { ...options,
ui: restriction === "ui", debug: restriction === "debug", maxAutomaticRetries: restriction === "zeroRetries" ? 0 : 1 },
qualificationCandidate: restriction === "qualificationCandidate", cancelled: () => false,
onRetry: () => { throw new Error("Unexpected retry notification"); },
runAttempt: async attempt => { attempts.push(attempt); return first; } });
expect(attempts).toEqual([1]); expect(result).toBe(first);
});
it("checks cancellation before an otherwise admitted second attempt", async () => {
const attempts: number[] = [];
await expect(executeWithAutomaticRetry({ task: {}, options, qualificationCandidate: false,
cancelled: () => true, onRetry: () => { throw new Error("Unexpected retry notification"); },
runAttempt: async attempt => { attempts.push(attempt); return { status: "failed" as const, failureClass: "provider_variance" as const }; },
})).rejects.toThrow("campaign cancelled");
expect(attempts).toEqual([1]);
});
});
tests/runner-e2e/automatic-retry.ts
+29
View File
@@ -0,0 +1,29 @@
import { shouldRetryFailure } from "./failure-classifier.js";
import type { FailureClass, RunnerTaskFixture } from "./types.js";
type RetryTask = Pick<RunnerTaskFixture, "automaticRetryPolicy">;
interface RetryableAttempt { status: "passed" | "failed"; failureClass?: FailureClass }
interface RetryOptions { maxAutomaticRetries: number; ui: boolean; debug: boolean }
export function effectiveAutomaticRetryLimit(task: RetryTask, configuredLimit: number): number {
return task.automaticRetryPolicy === "single_attempt" ? 0 : configuredLimit;
}
/** Keep the original failed result when the fixture admits exactly one attempt. */
export async function executeWithAutomaticRetry<Result extends RetryableAttempt>(input: {
task: RetryTask;
options: RetryOptions;
qualificationCandidate: boolean;
runAttempt(attempt: 1 | 2): Promise<Result>;
cancelled(): boolean;
onRetry(result: Result): void;
}): Promise<Result> {
const firstResult = await input.runAttempt(1);
if (input.qualificationCandidate || input.options.ui || input.options.debug ||
firstResult.status !== "failed" || !firstResult.failureClass ||
!shouldRetryFailure(firstResult.failureClass,
effectiveAutomaticRetryLimit(input.task, input.options.maxAutomaticRetries))) return firstResult;
if (input.cancelled()) throw new Error("Runner E2E campaign cancelled");
input.onRetry(firstResult);
return input.runAttempt(2);
}
tests/runner-e2e/billing.ts
+12 -2
View File
@@ -29,6 +29,7 @@ export interface AgentRunUsageInput {
}
export interface CampaignBillingSummary {
assistant?: import("./types.js").RunnerE2EAggregateBillingSummary["assistant"];
judge?: import("./types.js").RunnerE2EJudgeBillingSummary;
testCount: number;
agentRunDurationMs: number;
@@ -257,7 +258,7 @@ export function fallbackRuntimeUsage(
export function summarizeExecutionBilling(
result: Pick<
RunnerE2EResult,
"runIds" | "usage" | "environmentId" | "durationMs" | "runtimeUsage" | "firstTaskQuality" | "completionQuality"
"runIds" | "usage" | "environmentId" | "durationMs" | "runtimeUsage" | "firstTaskQuality" | "completionQuality" | "publicMcp"
>,
): RunnerE2EBillingSummary {
const requestedRunCount = Math.max(result.runIds?.length ?? 0, 1);
@@ -315,6 +316,7 @@ export function summarizeExecutionBilling(
runsWithReportedCost === runCount &&
runtime.costStatus !== "unavailable";
return {
...(result.publicMcp ? { assistant: result.publicMcp } : {}),
llm: {
runCount,
runsWithTokenUsage,
@@ -330,7 +332,7 @@ export function summarizeExecutionBilling(
reportedCostUsd,
estimatedRuntimeCostUsd,
...(quality ? { judge: { inputTokens: quality.inputTokens, outputTokens: quality.outputTokens, estimatedCostUsd: quality.estimatedCostUsd, reservedCostUsd: quality.reservedCostUsd } } : {}),
observedAndEstimatedCostUsd: quality?.estimatedCostUsd === null ? null : reportedCostUsd + estimatedRuntimeCostUsd + (quality?.estimatedCostUsd ?? 0),
observedAndEstimatedCostUsd: quality?.estimatedCostUsd === null ? null : reportedCostUsd + estimatedRuntimeCostUsd + (quality?.estimatedCostUsd ?? 0) + (result.publicMcp?.estimatedCostUsd ?? 0),
complete,
};
}
@@ -360,7 +362,15 @@ export function aggregateCampaignBilling(
0,
);
const judges = summaries.flatMap(summary => summary.judge ? [summary.judge] : []);
const assistants = summaries.flatMap(summary => summary.assistant ? [summary.assistant] : []);
return {
...(assistants.length ? { assistant: {
requests: assistants.reduce((n, u) => n + u.requests, 0),
inputTokens: assistants.reduce((n, u) => n + u.inputTokens, 0),
outputTokens: assistants.reduce((n, u) => n + u.outputTokens, 0),
cachedInputTokens: assistants.reduce((n, u) => n + u.cachedInputTokens, 0),
estimatedCostUsd: assistants.reduce((n, u) => n + u.estimatedCostUsd, 0),
} } : {}),
...(judges.length ? { judge: {
attempts: judges.length,
inputTokens: judges.reduce((n, q) => n + (q.inputTokens ?? 0), 0),
tests/runner-e2e/blocker-flow.ts
+2 -2
View File
@@ -88,10 +88,10 @@ export async function runBlockerFlow(input: {
if (hash !== hashes[file]) throw new Error(`Bundled operational skill differs from evaluated source: ${file}`);
}
await api.patch("/api/instance/settings/experimental", { enableClassicTaskInterface: false });
await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name,
const createdTask = await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name,
title: execution.task.buildTitle(input.nonce), prompt: scenario.prompt, workMode: "standard" });
issue = await pollUntil({ label: "browser-created blocker task", deadlineAt: input.deadlineAt,
load: async () => (await api.get<Row[]>(`${company}/issues`)).find(i => i.title === execution.task.buildTitle(input.nonce)), accept: Boolean });
load: async () => (await api.get<Row[]>(`${company}/issues`)).find(i => i.id === createdTask.issueId), accept: Boolean });
if (!issue) throw new Error("No browser-created task");
checkpoints.push(await settle("waiting"));
await open();
tests/runner-e2e/case-outcome.ts
+5 -2
View File
@@ -1,4 +1,5 @@
import type { RunnerE2EResult } from "./types.js";
import { isBlockedConnectionResult } from "./connection-evidence.js";
const html = (value: unknown) =>
String(value ?? "").replace(
/[&<>"']/g,
@@ -12,6 +13,7 @@ export function isIncompleteFirstTaskResult(
result: RunnerE2EResult,
errors: readonly string[] = [],
) {
if (isBlockedConnectionResult(result)) return result.cleanup === "passed" && result.failureClass !== "secret_leak" && errors.length === 0;
const checks = result.firstTask?.checks ?? [];
return result.status === "failed" &&
result.failureClass === "candidate_failure" &&
@@ -30,7 +32,7 @@ export function renderCaseOutcome(
(result.matcherResults ?? []).map((m) => ({
id: m.matcher.kind === "json_path" ? m.matcher.path : m.matcher.kind,
passed: m.passed,
notReached: undefined as string | undefined,
notReached: isBlockedConnectionResult(result) && !m.passed ? "Blocked before this checkpoint" : undefined as string | undefined,
detail: m.detail,
}));
const failures = checks.filter((c) => !c.passed && !c.notReached);
@@ -54,7 +56,8 @@ export function renderCaseOutcome(
),
];
return `<section class="case-outcome ${failed ? "outcome-failed" : "outcome-passed"}" aria-label="Case result explanation">
<strong>${incomplete ? "Incomplete journey" : failed ? "Overall failed" : "Overall passed"} · ${checks.length ? `${passed}/${evaluated} behavioral checks passed${notReached.length ? ` · ${notReached.length} not reached` : ""}` : "No behavioral checks recorded"}</strong>
<strong>${isBlockedConnectionResult(result) ? `Blocked: ${html(result.providerConnection!.outcome.replaceAll("_", " "))}` : incomplete ? "Incomplete journey" : failed ? "Overall failed" : "Overall passed"} · ${checks.length ? `${passed}/${evaluated} behavioral checks passed${notReached.length ? ` · ${notReached.length} not reached` : ""}` : "No behavioral checks recorded"}</strong>
${result.providerConnection ? `<p>Target: ${html(result.providerConnection.target.origin)} · ${html(result.providerConnection.target.commit ?? "revision unavailable")} · ${result.providerConnection.assisted ? "Assisted login" : "Unassisted"} · ${html(result.providerConnection.authFreshness)} browser · ${html(result.providerConnection.entry)} / ${html(result.providerConnection.method)} · stopped at ${html(result.providerConnection.phase)}</p>` : ""}
${failures.length ? `<p>Failed checks:</p><ul>${failures.map((c) => `<li><strong>${html(c.id)}</strong> — ${html(c.detail)}</li>`).join("")}</ul>` : checks.length ? `<p>No behavioral matcher failed.${notReached.length ? " The journey is incomplete; unexercised checks are not passes." : failed ? " The overall failure came from a separate run, cleanup, or evidence check." : ""}</p>` : ""}
${notReached.length ? `<p>Not reached:</p><ul>${notReached.map((c) => `<li><strong>${html(c.id)}</strong> — ${html(c.notReached)}</li>`).join("")}</ul>` : ""}
${checks.length ? `<details><summary>See all behavioral checks (${checks.length})</summary><table class="matchers"><thead><tr><th>Result</th><th>Check</th><th>Detail</th></tr></thead><tbody>${checks.map((c) => `<tr class="matcher-${c.notReached ? "not-reached" : c.passed ? "passed" : "failed"}"><td>${c.notReached ? "Not reached" : c.passed ? "Pass" : "Fail"}</td><td><code>${html(c.id)}</code></td><td>${html(c.detail)}${c.notReached ? ` — ${html(c.notReached)}` : ""}</td></tr>`).join("")}</tbody></table></details>` : ""}
tests/runner-e2e/catalog.test.ts
+15 -5
View File
@@ -104,6 +104,16 @@ describe("runner E2E catalog", () => {
}
});
it("keeps paid public MCP journeys explicit and fingerprints the shipped workflow instructions", () => {
const suite = runnerSuites.find(suite => suite.id === "public-mcp")!;
expect(suite.manualOnly).toBe(true);
expect(suite.environments.map(environment => environment.id)).toEqual(["local"]);
expect(suite.profiles.map(profile => profile.model)).toEqual(["gpt-5.4-mini", "claude-haiku-4-5-20251001", "claude-sonnet-4-6"]);
expect(suite.tasks).toHaveLength(21);
expect(suite.tasks.every(task => task.flow === "public_mcp" && task.expectedRunCount === 1)).toBe(true);
expect(suite.definitionMetadata?.workflowDigest).toMatch(/^[0-9a-f]{64}$/);
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(execution => execution.suite.id === "public-mcp")).toBe(false);
});
it("defines sixteen local connection-review journeys without expanding the default matrix", () => {
expect(connectionReviewSuite.expectedMatrixSize).toBe(16);
expect(new Set(connectionReviewSuite.profiles.map(profile => profile.id))).toEqual(new Set(["runner-codex", "runner-acpx-claude", "legacy-codex", "legacy-claude"]));
@@ -136,10 +146,10 @@ describe("runner E2E catalog", () => {
expect(localIntegrityTasks).toHaveLength(2);
expect(openRouterBreadthTasks).toHaveLength(3);
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
4, 8, 10, 4, 4, 6, 6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
63, 12, 6, 8, 2, 2, 2, 30, 3, 16, 16, 2, 6, 8, 46, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116,
]);
expect(validateRunnerCatalog()).toHaveLength(469);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(469);
expect(validateRunnerCatalog()).toHaveLength(698);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(698);
expect(
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
).toHaveLength(48);
@@ -428,13 +438,13 @@ describe("runner E2E catalog", () => {
it("uses only declared secret references in generated payloads", () => {
expect(
runnerMatrix.every((entry) =>
runnerMatrix.filter(entry => entry.task.flow !== "provider_connection").every((entry) =>
entry.requiredCredentials.includes(entry.profile.credential),
),
).toBe(true);
expect(
runnerMatrix
.filter((entry) => entry.environment.id === "daytona")
.filter((entry) => entry.environment.id === "daytona" && entry.task.flow !== "provider_connection")
.every((entry) =>
entry.requiredCredentials.includes("DAYTONA_API_KEY"),
),
tests/runner-e2e/catalog.ts
+174 -63
View File
@@ -1,19 +1,25 @@
import { nativeActiveStopTasks } from "./native-active-stop-tasks.js";
import { piControlTasks } from "./pi-controls-cases.js";
import { cursorNativeTasks } from "./cursor-native-cases.js";
import { copilotProtectionTasks } from "./copilot-protection-tasks.js";
import { piNativeTasks } from "./pi-native-cases.js";
import { piFilePrompt } from "./pi-file-evidence.js";
import { planTaskCases, planTaskProfile, planDefinitionDigest } from "./plan-task-cases.js";
import { nativeCompletionTasks, nativeCompletionDefinitionDigest } from "./native-completion-cases.js";
import { NATIVE_INSTRUCTION_SUITE, NATIVE_INSTRUCTION_BASE_SHA, nativeInstructionDefinitionDigest } from "./native-instruction-consolidation.js";
import { nativeCompletionProfile, NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js";
import { chatConfirmationTasks } from "./chat-cases.js";
import { buildConnectionSuite } from "./connection-cases.js";
import { hiringTemplateTasks, hiringTemplateProfile, hiringTemplateDefinitionDigest } from "./hiring-template-cases.js";
import { nativeActiveStopTasks } from "./native-active-stop-tasks.js";
import { cursorNativeTasks } from "./cursor-native-cases.js";
import { instructionPersistenceTask } from "./instruction-persistence.js";
import { apiResponseReadingTask } from "./api-response-reading.js";
import { taskTitleTasks, taskTitleDefinitionDigest, TASK_TITLE_BUDGET_CENTS } from "./task-titles.js";
import { blockerTasks, blockerProfile } from "./blocker-cases.js";
import { accountingTasks } from "./accounting-cases.js";
import { continuationTasks } from "./continuation-cases.js";
import { contextIntegrityTasks } from "./context-integrity-cases.js";
import { contextIntegrityTasks, paperclipDocumentTask } from "./context-integrity-cases.js";
import { productionDefaultHireProfile, stockHarnessSourceDigest, stockHarnessSkillSources } from "./stock-harness.js";
import { lifecycleLiveTasks, lifecycleLiveDefinitionDigest } from "./lifecycle-live-cases.js";
import { publicMcpTasks, publicMcpExpandedDigest, publicMcpSetupDigest, publicMcpWorkflowDigest, publicMcpWorkerInstructions, publicMcpWorkerSkillDigest } from "./public-mcp-cases.js";
import { graderVersion as publicMcpGraderVersion } from "./public-mcp-grading.js";
import { everydayTasks, productionStoryProfile } from "./everyday-cases.js";
import { CONNECTION_GUIDANCE_SUITE, CONNECTION_GUIDANCE_BUDGET_CENTS, connectionGuidanceTasks, connectionGuidanceDefinitionDigest } from "./connection-guidance-cases.js";
import { firstTaskTasks } from "./first-task-cases.js";
import { chatTasks, chatHardeningTasks, chatStoryTasks, chatQualificationTasks, chatCompletionTasks } from "./chat-cases.js";
@@ -26,6 +32,7 @@ import { models as claudeModels } from "../../packages/adapters/claude-local/src
import { DEFAULT_KIMI_LOCAL_MODEL } from "../../packages/adapters/kimi-local/src/index.js";
import { DEFAULT_GROK_LOCAL_MODEL } from "../../packages/adapters/grok-local/src/index.js";
import { QUALIFIED_ACPX_PROFILES } from "../../packages/paperclip-runner/src/drivers/acpx/qualified-profiles.js";
import { ACPX_QUALIFICATION_MODELS } from "../acpx-qualification-models.js";
import { QUALIFIED_OPENCODE_MODEL } from "../../packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.js";
import { CREDENTIAL_NAMES } from "./types.js";
import { createGitStreamingTask } from "./daytona-git-streaming.js";
@@ -57,6 +64,7 @@ const SELECTABLE_GROUPS = [
"chat",
"onboarding",
"context-integrity",
"stock-harness",
] as const;
const SAMPLE_UUID = "11111111-1111-4111-8111-111111111111";
@@ -78,6 +86,7 @@ function commonAgent(
fixtureId: string,
adapterType: string,
adapterConfig: Record<string, unknown>,
additionalInstructions: readonly string[] = [],
) {
return {
name: `Runner E2E ${fixtureId} ${input.executionId}`,
@@ -107,6 +116,7 @@ function commonAgent(
'For a planning task, do not inspect the OpenAPI schema. PUT /api/issues/$PAPERCLIP_TASK_ID/documents/plan with {title:"Plan",format:"markdown",body,changeSummary}; read latestRevisionId and latestRevisionNumber from that response. Then POST /api/issues/$PAPERCLIP_TASK_ID/interactions with {kind:"request_confirmation",continuationPolicy:"wake_assignee",payload:{version:1,prompt,acceptLabel:"Approve",rejectLabel:"Reject",rejectRequiresReason:true,target:{type:"issue_document",key:"plan",revisionId,revisionNumber}}}, and PATCH the issue to {status:"in_review"}. Include Authorization and X-Paperclip-Run-Id on every write.',
]),
"Never print, persist, or expose credential values, and never create unrelated work.",
...additionalInstructions,
].join("\n"),
},
},
@@ -122,10 +132,12 @@ function legacyProfile(input: {
model: string;
credential: RunnerProfileFixture["credential"];
extraConfig?: Record<string, unknown>;
instructions?: readonly string[];
modelQualification?: RunnerProfileFixture["modelQualification"];
}): RunnerProfileFixture {
return {
...input,
modelQualification: {
modelQualification: input.modelQualification ?? {
source: "adapter_constant",
qualificationId: `${input.adapterType}:default-model`,
},
@@ -151,7 +163,7 @@ function legacyProfile(input: {
env: {
[input.credential]: requiredSecret(buildInput, input.credential),
},
});
}, input.instructions);
},
};
}
@@ -302,7 +314,7 @@ export const runnerProfiles: readonly RunnerProfileFixture[] = [
label: "Runner ACPX Claude",
provider: "acpx",
acpxAgent: "claude",
model: QUALIFIED_ACPX_PROFILES.claude.qualificationModel,
model: ACPX_QUALIFICATION_MODELS.claude,
credential: "ANTHROPIC_API_KEY",
}),
nativeProfile({
@@ -310,7 +322,7 @@ export const runnerProfiles: readonly RunnerProfileFixture[] = [
label: "Runner Grok Build",
provider: "acpx",
acpxAgent: "grok",
model: QUALIFIED_ACPX_PROFILES.grok.qualificationModel,
model: ACPX_QUALIFICATION_MODELS.grok,
credential: "XAI_API_KEY",
}),
nativeProfile({
@@ -318,19 +330,19 @@ export const runnerProfiles: readonly RunnerProfileFixture[] = [
label: "Runner ACPX Codex",
provider: "acpx",
acpxAgent: "codex",
model: QUALIFIED_ACPX_PROFILES.codex.qualificationModel,
model: ACPX_QUALIFICATION_MODELS.codex,
credential: "OPENAI_API_KEY",
}),
] as const;
// Explicit qualification choices from authenticated model discovery, not shipped
// defaults or claims that these candidates have passed inference qualification.
// defaults. Cursor has local/Daytona proof; Copilot and Pi remain pending.
export const extendedHarnessProfiles: readonly RunnerProfileFixture[] = [
nativeProfile({
id: "runner-acpx-cursor", label: "Runner Cursor (candidate)", provider: "acpx", acpxAgent: "cursor",
id: "runner-acpx-cursor", label: "Runner Cursor", provider: "acpx", acpxAgent: "cursor",
qualificationCandidate: "cursor", credential: "CURSOR_AUTH_TOKEN",
model: "gpt-5.6-luna[context=272k,reasoning=medium,fast=false]",
modelQualification: { source: "candidate_runner_profile", qualificationId: "cursor:2026.09.26-dd393fe:discovery-2026-09-28" },
modelQualification: { source: "qualified_runner_profile", qualificationId: "cursor:2026.09.26-dd393fe:v11:luna:local-daytona-2026-10-04" },
}),
nativeProfile({
id: "runner-acpx-copilot", label: "Runner Copilot (candidate)", provider: "acpx", acpxAgent: "copilot",
@@ -339,8 +351,8 @@ export const extendedHarnessProfiles: readonly RunnerProfileFixture[] = [
}),
nativeProfile({
id: "runner-acpx-pi", label: "Runner Pi (candidate)", provider: "acpx", acpxAgent: "pi",
qualificationCandidate: "pi", credential: "OPENROUTER_API_KEY", model: QUALIFIED_ACPX_PROFILES.pi.qualificationModel,
modelQualification: { source: "candidate_runner_profile", qualificationId: "pi:0.0.33:1.0.0:openrouter" },
qualificationCandidate: "pi", credential: "OPENROUTER_API_KEY", model: ACPX_QUALIFICATION_MODELS.pi,
modelQualification: { source: "candidate_runner_profile", qualificationId: "pi:0.0.33:0.84.2:openrouter" },
}),
];
@@ -1072,25 +1084,53 @@ export const extendedHarnessFileTask: RunnerTaskFixture = {
buildPrompt: nonce => [
`Create extended-${nonce}.txt in the current execution workspace with exactly ready-${nonce} followed by a newline.`,
`Edit that file to contain exactly verified-${nonce} followed by a newline, then run a command that reads it and fails unless those exact bytes match.`,
`Register the verified file as a deliverable with title extended-${nonce}.txt. Use EXTENDED-FILE-${nonce} only as completion summary and final response, never as a file title or progress message.`,
`After successful validation, call paperclip_finish with reportedWorkDisposition done, summary EXTENDED-FILE-${nonce}, the current completion contract revision and satisfied objective criterion, no remaining work, and the actual validation command and result.`,
`Wait for paperclip_finish to succeed, then emit exactly EXTENDED-FILE-${nonce} as your final response. Do not create unrelated files or work.`,
].join("\n"),
buildMatchers: (nonce, execution) => [
...terminalMatchers(`EXTENDED-FILE-${nonce}`, execution),
{ kind: "file_exact", path: `extended-${nonce}.txt`, expected: `verified-${nonce}\n` },
{ kind: "artifact_exact", name: `extended-${nonce}.txt`, expected: `verified-${nonce}\n`, mimeType: "text/plain" },
],
};
export const runnerSuites: readonly RunnerSuiteFixture[] = [
{
id: "pi-controls", label: "Pi active controls", manualOnly: true,
description: "Pending native-write Stop and browser-originated same-turn steering, with exact control receipts and independent retirement/no-effect evidence.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "pi"),
environments: runnerEnvironments, tasks: piControlTasks, expectedMatrixSize: 4,
definitionMetadata: { version: 1, qualification: "pending", scheduling: "explicit-only", profileVersion: QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion,
pending: "paperclip.e2e.pi-control-pending.v1", stop: "paperclip.e2e.pi-stop-settlement.v1", steering: "paperclip.e2e.pi-steering-settlement.v1",
permissionPolicy: "approve-reads", lifecycle: "per_turn", normalCompletionProvesStop: false, nativeFollowUp: "not-covered", providerDeath: "not-covered",
remoteObservationCoverage: "continuous-through-owned-process-retirement", filesystemAfterRemoteRetirementObserved: false },
id: "public-mcp", label: "Paperclip through an assistant", manualOnly: true,
description: "Paid assistant tool use plus actual team execution, browser OAuth consent, durable outcomes and authorization boundaries.",
groups: ["local"], environments: [localEnvironment], tasks: publicMcpTasks, expectedMatrixSize: 63,
profiles: [
...["mini"].map(size => legacyProfile({
id: `assistant-codex-${size}`, label: `Assistant + Codex ${size}`, adapterType: "codex_local", provider: "codex",
model: `gpt-5.4-${size}`, credential: "OPENAI_API_KEY",
instructions: publicMcpWorkerInstructions,
modelQualification: { source: "qualified_runner_profile", qualificationId: `public-mcp-gpt-5.4-${size}-pilot` },
extraConfig: { engine: "cli", extraArgs: ["-c", "features.shell_snapshot=false", "-c", "features.apps=false", "-c", "features.plugins=false"] },
})),
...[["haiku", "claude-haiku-4-5-20251001"], ["sonnet", "claude-sonnet-4-6"]].map(([name, model]) => legacyProfile({
id: `assistant-claude-${name}`, label: `Assistant + Claude ${name}`, adapterType: "claude_local", provider: "claude",
model: model!, credential: "ANTHROPIC_API_KEY",
instructions: publicMcpWorkerInstructions,
modelQualification: { source: "qualified_runner_profile", qualificationId: `public-mcp-${name}-pilot` },
extraConfig: { engine: "cli", maxTurnsPerRun: 16 },
})),
],
definitionMetadata: { version: 10, expandedDigest: publicMcpExpandedDigest, setupDigest: publicMcpSetupDigest, grader: publicMcpGraderVersion, workflowDigest: publicMcpWorkflowDigest, workerSkillDigest: publicMcpWorkerSkillDigest, workerInstructions: publicMcpWorkerInstructions, assistantMaxRequests: 16, assistantMaxEstimatedUsd: 2, instructions: "shipped-plugin-skills", scheduling: "explicit-only" },
},
{
id: "plan-task-guidance", label: "Planning guidance utility", manualOnly: true,
description: "Current, short and disabled planning skills on cohesive, parallel, dependent and independently reviewed work.",
groups: ["native"], profiles: runnerProfiles.filter(p => p.id === "runner-codex").map(planTaskProfile),
environments: [localEnvironment], tasks: planTaskCases, expectedMatrixSize: 12,
definitionMetadata: { version: 1, definitionDigest: planDefinitionDigest, scheduling: "explicit-only", comparison: "public-skill-availability-ablation", scope: "one-profile-single-attempt" },
},
{
id: "blocker-guidance", label: "Direct blocker handling", manualOnly: true,
description: "Human authority, hiring permissions, and requester scope under the production coordination skill.",
groups: ["legacy"], profiles: runnerProfiles.filter(p => ["legacy-codex", "legacy-claude"].includes(p.id)).map(blockerProfile),
environments: [localEnvironment], tasks: blockerTasks, expectedMatrixSize: 6,
definitionMetadata: { version: 1, instructions: "production-coordination-skill", grading: "saved-human-decision-ownership-and-resume", scheduling: "explicit-only" },
},
{
id: "cursor-native", label: "Cursor native interactions", manualOnly: true,
@@ -1099,48 +1139,42 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
environments: runnerEnvironments, tasks: cursorNativeTasks, expectedMatrixSize: 8,
definitionMetadata: { version: 2, qualification: "pending", scheduling: "explicit-only", profileVersion: QUALIFIED_ACPX_PROFILES.cursor.agentProfileVersion, modeAdmission: "native-config-ack", artifactExport: "pending-private-home", remoteEvidence: "owned-lease-sealed-observer" },
},
{
id: "pi-native", label: "Pi native boundaries", manualOnly: true,
description: "Pi native forms, registered agent files and human permission denial on local and Daytona execution; automatic deny-all remains local-only.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "pi"),
environments: runnerEnvironments, tasks: piNativeTasks, expectedMatrixSize: 10,
excludedExecutionIds: ["pi-native.runner-acpx-pi.daytona.restrictive-denial", "pi-native.runner-acpx-pi.local.native-pending-provider-death"],
definitionMetadata: { version: 4, qualification: "pending", scheduling: "explicit-only", profileVersion: QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, providerDeath: "daytona-exact-pi-child-pidfd-production-expiry", remoteDenyAll: "unsupported-native-bootstrap-read-is-denied", remoteEvidence: "owned-lease-sealed-observer", pendingControllerRestart: "same-live-native-request" },
},
{
id: "native-active-stop", label: "Stop an unanswered native permission", manualOnly: true,
description: "Stop while one exact Cursor or Copilot native permission remains unanswered; require cancelled provider settlement, caller-owned acknowledgement, stale-answer refusal and independent retirement/no effects.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => ["cursor", "copilot"].includes(profile.qualificationCandidate ?? "")),
environments: runnerEnvironments, tasks: nativeActiveStopTasks, expectedMatrixSize: 4,
description: "Stop while one exact Cursor native permission remains unanswered; require cancelled provider settlement, caller-owned acknowledgement, stale-answer refusal and independent retirement/no effects.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "cursor"),
environments: runnerEnvironments, tasks: nativeActiveStopTasks, expectedMatrixSize: 2,
definitionMetadata: { version: 4, qualification: "pending", scheduling: "explicit-only", evidence: "paperclip.e2e.native-active-stop-settlement.v2", pendingObservation: "retained-api-before-caller-uuid-stop", normalCompletionAccepted: false, permissionPolicy: "approve-reads", lifecycle: "per_turn", remoteEvidence: "paperclip.e2e.native-active-stop-remote-retirement.v1", remoteObservationCoverage: "continuous-through-owned-process-retirement", filesystemAfterRemoteRetirementObserved: false, localObservationCoverage: "four-phases-through-cleanup", providerDeath: "not-covered" },
},
{
id: "copilot-protection", label: "Copilot native protection", manualOnly: true,
description: "Exact native denial with independently correlated provider settlement and acknowledged run Stop, plus attached command settlement with independent process evidence.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "copilot"),
environments: runnerEnvironments, tasks: copilotProtectionTasks, expectedMatrixSize: 4,
definitionMetadata: { version: 8, semanticCompletionEvidence: "paperclip.e2e.copilot-semantic-completion.v2", qualification: "pending", naturalSettlementObservationMs: 2000, scheduling: "explicit-only", evidence: "copilot_tool_evidence_v1", profileVersion: QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion, denialTerminal: "correlated-provider-settlement-and-audited-run-stop", denialSettlementEvidence: "paperclip.e2e.copilot-denial-settlement.v3", activeTurnCancellation: "not-implied-by-completed-provider-turn", settlement: "attached-finite-command-only", settlementMarker: "private-diagnostic-not-deliverable", remoteEvidence: "owned-lease-sealed-observer" },
id: "native-provider-loss", label: "Lose a runtime with an unanswered native permission", manualOnly: true,
description: "Lose the owned Cursor runtime while a native mutation remains unanswered; require a visible failed run, closed unanswerable input, stale-answer refusal and independent retirement with no effects or replay.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "cursor"),
environments: runnerEnvironments,
tasks: [{
id: "pending-permission-provider-loss", label: "Owned runtime loss with pending permission", groups: [], workMode: "standard", flow: "native_provider_loss",
expectedRunCount: 1, attemptTimeoutMs: { local: 300_000, daytona: 300_000 }, turnTimeoutMs: 120_000,
expectedTerminalState: { issue: "blocked", run: "failed" },
buildTitle: nonce => `Native provider loss ${nonce}`,
buildVisibleMarker: nonce => `NATIVE-PROVIDER-LOSS-${nonce}`,
buildPrompt: () => "Attempt the supplied exact native shell operation once and leave its permission callback unanswered. Do not retry, use another tool or path, answer your own permission, call paperclip_finish, or claim completion. The qualification fixture will deliberately lose the isolated runtime before any approval.",
buildMatchers: () => [],
}], expectedMatrixSize: 2,
definitionMetadata: { version: 1, qualification: "pending", scheduling: "explicit-only", fault: "observed-per-turn-run-root-loss", remoteFaultAuthority: "pidfd-start-ticks-boot-id", replayAllowed: false },
},
{
id: "rich-acp-warm-continuity", label: "Rich ACP warm continuity", manualOnly: true,
description: "Three browser-driven turns with stable native session, runner process and workspace identity for Cursor, Copilot and Pi.",
description: "Three browser-driven turns with stable native session, runner process and workspace identity for Cursor.",
groups: ["native", "warm"],
profiles: extendedHarnessProfiles.map(profile => ({ ...profile, buildAgent(input: AgentFixtureBuildInput) {
profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "cursor").map(profile => ({ ...profile, buildAgent(input: AgentFixtureBuildInput) {
const agent = profile.buildAgent(input);
return { ...agent, adapterConfig: { ...agent.adapterConfig as Record<string, unknown>, lifecycleMode: "warm", idleTimeoutMs: 300_000 } };
} })),
environments: [localEnvironment, daytonaWarmEnvironment],
tasks: [{ ...daytonaWarmContinuityTask, turnTimeoutMs: 120_000, attemptTimeoutMs: { local: 420_000, daytona: 420_000 } }],
expectedMatrixSize: 6,
expectedMatrixSize: 2,
definitionMetadata: { version: 1, qualification: "pending", scheduling: "explicit-only", identity: "native-session-runner-provider-session-process-start" },
},
{
id: "blocker-guidance", label: "Direct blocker handling", manualOnly: true,
description: "Human authority, hiring permissions, and requester scope under the production coordination skill.",
groups: ["legacy"], profiles: runnerProfiles.filter(p => ["legacy-codex", "legacy-claude"].includes(p.id)).map(blockerProfile),
environments: [localEnvironment], tasks: blockerTasks, expectedMatrixSize: 6,
definitionMetadata: { version: 1, instructions: "production-coordination-skill", grading: "saved-human-decision-ownership-and-resume", scheduling: "explicit-only" },
},
{
id: "extended-harnesses", label: "Extended ACP harnesses", manualOnly: true,
description: "Explicit candidate qualification through real Paperclip tools, browser interactions, file edits and restart recovery.",
@@ -1165,7 +1199,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
profiles: [nativeProfile({
id: "runner-acpx-grok-subscription", label: "Grok Build Subscription",
provider: "acpx", acpxAgent: "grok",
model: QUALIFIED_ACPX_PROFILES.grok.qualificationModel,
model: ACPX_QUALIFICATION_MODELS.grok,
credential: "GROK_AUTH_JSON",
})],
environments: [localEnvironment, daytonaWarmEnvironment],
@@ -1226,7 +1260,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
...connectionReviewSuite.tasks],
expectedMatrixSize: 46,
excludedExecutionIds: ["neutral", "challenge"].map(variant => `lifecycle-baseline.runner-codex.local.lifecycle-repair-${variant}`),
definitionMetadata: { version: 4, clarificationGrading: "wrapped-information-fields-list", narrativeDigest: lifecycleLiveDefinitionDigest, grading: "durable-state-and-attributed-narrative", scheduling: "explicit-only" },
definitionMetadata: { version: 5, clarificationGrading: "wrapped-information-fields-list", narrativeDigest: lifecycleLiveDefinitionDigest, grading: "durable-state-and-attributed-narrative", scheduling: "explicit-only" },
},
{
id: "continuation", label: "Task continuation",
@@ -1238,17 +1272,59 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
...["legacy-codex", "legacy-claude"].map(profile => `continuation.${profile}.local.question-tool-documentation`),
...["legacy-codex", "legacy-claude", "runner-codex"].map(profile => `continuation.${profile}.local.provider-question-bridge`),
],
definitionMetadata: { version: 4, grading: "durable-state-and-approval-boundaries", instructions: "production" },
definitionMetadata: { version: 5, grading: "durable-wait-owner-and-all-answer-identities", instructions: "production", automaticRetryPolicy: "single_attempt", companyAndAgentBudgetCents: 1_000 },
},
{
id: CONNECTION_GUIDANCE_SUITE, label: "Native connection guidance", manualOnly: true,
description: "Neutral decline prompts and approval/provider-choice controls; production connection instructions stay unchanged.",
groups: ["native", "local"],
profiles: runnerProfiles.filter(profile => ["runner-codex", "runner-acpx-claude", "runner-opencode"].includes(profile.id)).map(productionStoryProfile),
environments: [localEnvironment], tasks: connectionGuidanceTasks, expectedMatrixSize: 15,
definitionMetadata: { version: 1, fixtureDigest: connectionGuidanceDefinitionDigest(),
instructions: "unchanged-production", maximumAttemptsPerCell: 1,
automaticRetryPolicy: "single_attempt", companyAndAgentBudgetCents: CONNECTION_GUIDANCE_BUDGET_CENTS,
expectedProviderTurnsPerCell: 2, maximumRunRecordsPerCell: 12,
grading: "original-workflow-plus-attributed-decline-output", scheduling: "explicit-only" },
},
{
id: "everyday-workflows", label: "Everyday Paperclip Work", manualOnly: true,
description: "Real user requests, useful downloaded work, and durable continuation using production instructions.",
groups: ["native"], profiles: everydayProfiles, environments: [localEnvironment, daytonaWarmEnvironment],
tasks: everydayTasks, expectedMatrixSize: 47,
groups: ["native"], profiles: [...everydayProfiles,
productionStoryProfile(runnerProfiles.find(profile => profile.id === "runner-opencode")!),
], environments: [localEnvironment, daytonaWarmEnvironment],
tasks: everydayTasks.map(task => ["hire-reuse", "delegate-feedback"].includes(task.id)
? { ...task, automaticRetryPolicy: "single_attempt" as const } : task), expectedMatrixSize: 52,
excludedExecutionIds: [...everydayProfiles.flatMap(profile => everydayTasks
.filter(task => !["build-revise", "delegate-feedback", "recover-controller", "create-skill-studio"].includes(task.id))
.map(task => `everyday-workflows.${profile.id}.daytona.${task.id}`))],
definitionMetadata: { version: 3, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only" },
.map(task => `everyday-workflows.${profile.id}.daytona.${task.id}`)),
...everydayTasks.flatMap(task => [
`everyday-workflows.runner-opencode.daytona.${task.id}`,
...(["hire-reuse", "delegate-feedback"].includes(task.id) ? [] : [`everyday-workflows.runner-opencode.local.${task.id}`]),
]),
],
definitionMetadata: { version: 6, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only",
procedureCases: { ids: ["hire-reuse", "delegate-feedback"], maximumAttemptsPerCell: 1, companyAndLeadBudgetCents: 1_000 } },
},
{
id: NATIVE_INSTRUCTION_SUITE, label: "Native completion instruction consolidation", manualOnly: true,
description: "Matched production-default durable-document and concrete-blocker checks for the completion constraint reduction.",
groups: ["native", "local"],
profiles: runnerProfiles.filter(profile => ["runner-codex", "runner-acpx-claude", "runner-opencode"].includes(profile.id)).map(nativeCompletionProfile),
environments: [localEnvironment], tasks: nativeCompletionTasks, expectedMatrixSize: 6,
definitionMetadata: { version: 1, fixtureDigest: nativeInstructionDefinitionDigest(), baseSha: NATIVE_INSTRUCTION_BASE_SHA,
instructions: "production-default", maximumAttemptsPerCell: 1, automaticRetryPolicy: "single_attempt",
budgetMonthlyCents: NATIVE_COMPLETION_BUDGET_CENTS, scheduling: "explicit-only",
grading: "original-assigned-skill-and-strict-native-completion", },
},
{
id: "native-completion", label: "Native completion guidance", manualOnly: true,
description: "Production-default assigned-skill document completion and concrete whole-task blocking, with independent native result/final ordering.",
groups: ["native", "local"],
profiles: runnerProfiles.filter(profile => ["runner-codex", "runner-acpx-claude", "runner-opencode"].includes(profile.id)).map(nativeCompletionProfile),
environments: [localEnvironment], tasks: nativeCompletionTasks, expectedMatrixSize: 6,
definitionMetadata: { version: 1, fixtureDigest: nativeCompletionDefinitionDigest(), instructions: "unchanged-master-production-default",
providerTurns: 6, maximumAttemptsPerCell: 1, automaticRetryPolicy: "single_attempt", budgetMonthlyCents: NATIVE_COMPLETION_BUDGET_CENTS,
grading: "original-assigned-skill-durable-document-plus-native-disposition-and-observable-final-order", scheduling: "explicit-only" },
},
{
id: "context-integrity",
@@ -1269,6 +1345,30 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
prerequisiteGate: PENDING_PROFILE_PREREQUISITES,
},
},
{
id: "stock-harness",
label: "Stock harness with Paperclip",
manualOnly: true,
description: "Production-default hires, reduced shared prompts, assigned skills, continuation, and persistent chat.",
groups: ["stock-harness", "native", "legacy"],
profiles: [...contextIntegrityProfiles.filter(profile => !pendingContextIntegrityProfiles.includes(profile)),
runnerProfiles.find(profile => profile.id === "legacy-opencode")!].map(productionDefaultHireProfile),
environments: [localEnvironment],
tasks: [...contextIntegrityTasks, chatTasks.find(task => task.id === "continuity-restart")!, paperclipDocumentTask]
.map(task => ({ ...task, automaticRetryPolicy: "single_attempt" as const })),
expectedMatrixSize: 26,
excludedExecutionIds: ["legacy-codex", "legacy-acp-codex", "legacy-acp-claude", "runner-codex", "runner-acpx-claude", "runner-opencode"]
.map(profile => `stock-harness.${profile}.local.${paperclipDocumentTask.id}`),
definitionMetadata: {
version: 2, instructions: "production-default-hire", scheduling: "explicit-only",
automaticRetryPolicy: "single_attempt", maximumAttemptsPerCell: 1,
sourceDigest: stockHarnessSourceDigest(),
operationalSkillSources: stockHarnessSkillSources(),
grading: "public-default-bundle-and-delivered-prompts-plus-independent-lifecycle-oracles",
vendorBaseEvidence: "required deterministic Codex driver/runnerd/Rust gate; task success is not vendor-base proof",
paidCalls: "one skill turn, two ordered-comment turns, three chat turns per profile",
},
},
{
id: "first-task", label: "First-task onboarding",
description: "Production onboarding, first replies, approval, and durable task execution.",
@@ -1297,6 +1397,16 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
["stop-startup-new-resume", "hire-delegate-reuse", "blocked-status-review"].map(task => `agent-chat-hardening.${profile}.daytona.${task}`)),
definitionMetadata: { version: 5, permissions: "production-defaults", instructions: "production", grading: "durable-state-and-source-evidence", scheduling: "explicit-only", restartMemory: "required-after-restart", statusEvidence: "structured-current-blocker-and-active-run-count", readOnlyState: "public-mutation-contract-and-relations", hiringReference: "neutral-document-reference-line" },
},
{
id: "hiring-templates", label: "Production Hiring Templates", manualOnly: true,
description: "Production CEO and hiring skill/reference discovery, one coder hire, independently checked JSON artifacts and worker reuse.",
groups: ["chat", "native"],
profiles: runnerProfiles.filter(profile => ["runner-codex", "runner-acpx-claude"].includes(profile.id))
.map(profile => hiringTemplateProfile(defaultPermissionProfile(profile))),
environments: [localEnvironment], tasks: hiringTemplateTasks, expectedMatrixSize: 2,
definitionMetadata: { version: 1, definitionDigest: hiringTemplateDefinitionDigest, instructions: "source-revision-default-ceo", scheduling: "explicit-only",
grading: "independent-json-and-read-receipts", baselineComparison: "same-fixture-source-derived-bundles", providerTurns: 5 },
},
{
id: "agent-chat-stories", label: "Agent Chat Setup and Interruptions", manualOnly: true,
description: "Experimental settings lifecycle and user follow-ups during active native work.",
@@ -1441,6 +1551,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
expectedMatrixSize: 1,
definitionMetadata: { version: 8, instructions: "fixed-external", nativeIdleTimeoutMs: 1_200_000, autoStopIntervalMinutes: 25, generatedFileCount: 60_000, filenameBytes: 39_828_890, scheduling: "explicit-only", finalization: "committed-without-active-sync-or-retry", copyback: "all-generated-file-contents-change-each-turn" },
},
buildConnectionSuite(runnerProfiles, runnerEnvironments),
] as const;
export function suiteDefinitionHash(suite: RunnerSuiteFixture) {
@@ -1501,8 +1612,8 @@ export function buildRunnerMatrix(
]),
],
requiredCredentials: [
profile.credential,
...(environment.credential ? [environment.credential] : []),
...(task.flow === "provider_connection" ? [] : [profile.credential]),
...(task.flow !== "provider_connection" && environment.credential ? [environment.credential] : []),
],
}))
.filter((execution) => !excludedExecutionIds.has(execution.id)),
@@ -1548,10 +1659,10 @@ function assertNoRawSecretValues(value: unknown, label: string) {
}
export function validateRunnerCatalog(): MatrixExecution[] {
const allProfiles = [...extendedHarnessProfiles, ...runnerProfiles, ...legacyAcpxProfiles, ...pendingContextIntegrityProfiles, ...openRouterBreadthProfiles, ...everydayProfiles.filter(p => !runnerProfiles.some(existing => existing.id === p.id))];
const connectionSuite = runnerSuites.find(suite => suite.id === "provider-connections")!;
const allProfiles = [...connectionSuite.profiles, ...extendedHarnessProfiles, ...runnerProfiles, ...legacyAcpxProfiles, ...pendingContextIntegrityProfiles, ...openRouterBreadthProfiles, ...everydayProfiles.filter(p => !runnerProfiles.some(existing => existing.id === p.id))];
const allTasks = [
...piNativeTasks,
...copilotProtectionTasks,
...connectionSuite.tasks,
extendedHarnessFileTask,
...contextIntegrityTasks,
...blockerTasks,
tests/runner-e2e/chat-cases.ts
+2 -2
View File
@@ -54,10 +54,10 @@ export const chatStoryTasks = buildChatTasks([
]).map(task => ({ ...task, ...(task.id === "enable-disable-resume" ? {} : { minimumExpectedRunCount: 1 }) }));
export function chatNeedsApiTools(suiteId: string, caseId: string): boolean {
return (suiteId === "agent-chat-qualification" && caseId === "grounded-answer-quality") || suiteId === "agent-chat-hardening" && ["hire-delegate-reuse", "blocked-status-review"].includes(caseId);
return suiteId === "hiring-templates" || (suiteId === "agent-chat-qualification" && caseId === "grounded-answer-quality") || suiteId === "agent-chat-hardening" && ["hire-delegate-reuse", "blocked-status-review"].includes(caseId);
}
export function isManagedHiringCase(suiteId: string, caseId: string): boolean {
return (suiteId === "everyday-workflows" && caseId === "hire-reuse") ||
return (suiteId === "hiring-templates" && caseId === "hire-coder-template-reuse") || (suiteId === "everyday-workflows" && caseId === "hire-reuse") ||
(suiteId === "agent-chat-hardening" && caseId === "hire-delegate-reuse");
}
tests/runner-e2e/chat-flow.ts
+35 -1
View File
@@ -1,3 +1,6 @@
import { runHiringTemplateFlow } from "./hiring-template-flow.js";
import { gradeHiringTemplateTurns } from "./hiring-template-turn-accounting.js";
import type { HiringTemplateEvidence } from "./hiring-template-scoring.js";
import { runAmbiguousConfirmationReply, runUnansweredQuestionReturn } from "./confirmation-replies.js";
import { expect, type Page } from "@playwright/test";
import { pollUntil, type RunnerApi } from "./api.js";
@@ -16,6 +19,21 @@ import { runActiveReassignment, runWorkerCrash, runAnswerQuality } from "./chat-
import { matchesRunCount, minimumRunCount } from "./run-count.js";
import { runChatCompletionUpdate } from "./completion-update-flow.js";
/** Hiring alone admits bounded, verified lifecycle notifications. Other suites keep their count contract. */
export function assertChatFlowRunCount(input: {
suiteId: string; task: Parameters<typeof matchesRunCount>[0]; runs: ChatRun[];
hiringEvidence?: HiringTemplateEvidence; hiringApiState?: unknown;
}) {
expect(matchesRunCount(input.task, input.runs.length), "Declared total provider-run bounds").toBe(true);
if (input.suiteId !== "hiring-templates") return;
const accounting = gradeHiringTemplateTurns({
evidence: input.hiringEvidence ? { ...input.hiringEvidence, runs: input.runs } : undefined,
apiState: input.hiringApiState,
});
expect(accounting.passed, `Hiring lifecycle: ${accounting.predicates.filter(p => !p.passed).map(p => p.id).join(", ")}`).toBe(true);
return accounting;
}
// Public API observations only: this driver never fabricates provider results or writes DB state.
export interface ChatIssue {
id: string;
@@ -418,6 +436,8 @@ export async function runChatFlow(input: ChatFlowInput) {
await idle(count);
};
const noTasks = async () => expect(await tasks()).toHaveLength(0);
let hiringEvidence: HiringTemplateEvidence | undefined;
let refreshHiringEvidence: (() => Promise<HiringTemplateEvidence>) | undefined;
try {
if (caseId === "enable-disable-resume") await enableChatThroughSettings(input);
else await api.patch("/api/instance/settings/experimental", {
@@ -439,6 +459,10 @@ export async function runChatFlow(input: ChatFlowInput) {
} else if (caseId.startsWith("handoff-completion-")) {
await runChatCompletionUpdate({ input, marker, allRuns, issue: () => issue!,
refreshIssue: async () => { issue = await api.get<ChatIssue>(chatPath); if (issue) input.observe(issue, await allRuns()); } });
} else if (execution.suite.id === "hiring-templates") {
const hiring = await runHiringTemplateFlow({ input, issue: () => issue!, turn, tasks, allRuns });
hiringEvidence = hiring.evidence;
refreshHiringEvidence = hiring.refresh;
} else if (execution.suite.id === "agent-chat-qualification") {
const context = { input, marker, issue: () => issue!, idle, allRuns, comments, expectedStops,
refreshIssue: async () => { issue = await api.get<ChatIssue>(chatPath); input.observe(issue, await allRuns()); } };
@@ -979,7 +1003,17 @@ export async function runChatFlow(input: ChatFlowInput) {
});
}
await idle(minimumRunCount(execution.task));
expect(matchesRunCount(execution.task, runs.filter((run) => !isResetRun(run)).length)).toBe(true);
if (execution.suite.id === "hiring-templates") {
// Refresh the whole consistent evidence generation for the final guard.
hiringEvidence = await refreshHiringEvidence!();
runs = hiringEvidence.runs;
}
assertChatFlowRunCount({ suiteId: execution.suite.id, task: execution.task,
// Hiring must account for every actual company run, including unexpected resets.
runs: execution.suite.id === "hiring-templates" ? runs : runs.filter((run) => !isResetRun(run)),
hiringEvidence, hiringApiState: execution.suite.id === "hiring-templates"
? hiringEvidence!.turnApiState : undefined,
});
for (const run of runs.filter((run) => !isResetRun(run))) {
expect(run.runtimeMode).toBe(execution.profile.expectedRuntimeMode);
expect(run.status).toBe(
tests/runner-e2e/checkout-activity.test.ts
+35
View File
@@ -0,0 +1,35 @@
import { describe, expect, it } from "vitest";
import { observeCheckoutActivity } from "./checkout-activity.js";
const run = { id: "run", companyId: "company", agentId: "agent", status: "succeeded",
contextSnapshot: { issueId: "issue", paperclipWake: { checkedOutByHarness: true } } };
const receipt = { id: "receipt", companyId: "company", agentId: "agent", actorType: "agent",
entityType: "issue", entityId: "issue", action: "issue.checked_out", runId: "run" };
const input = { companyId: "company", issueId: "issue", agentId: "agent", runs: [run], activity: [] };
describe("checkout activity observation", () => {
it("distinguishes a runtime claim from repeated successful HTTP checkout calls", () => {
expect(observeCheckoutActivity(input)).toMatchObject({ status: "observed", successfulCheckoutRequests: 0,
runs: [{ checkedOutByHarness: true, successfulCheckoutRequests: 0 }] });
expect(observeCheckoutActivity({ ...input, activity: [receipt, { ...receipt, id: "second" }] }))
.toMatchObject({ status: "observed", successfulCheckoutRequests: 2 });
});
it.each([
{ runId: "another-run" }, { companyId: "foreign" }, { entityId: "other-task" },
{ agentId: "other-agent" }, { actorType: "user" }, { id: null },
])("does not count unattributed or foreign receipt %j as a comparable zero", mutation => {
expect(observeCheckoutActivity({ ...input, activity: [{ ...receipt, ...mutation }] }))
.toMatchObject({ status: "uncomparable", successfulCheckoutRequests: null });
});
it("refuses missing, mismatched and duplicate run/receipt identities", () => {
for (const data of [
{ ...input, runs: [] }, { ...input, runs: [run, run] },
{ ...input, runs: [{ ...run, agentId: "other" }] },
{ ...input, runs: [{ ...run, contextSnapshot: { issueId: "other" } }] },
{ ...input, activity: [receipt, receipt] },
]) expect(observeCheckoutActivity(data)).toMatchObject({ status: "uncomparable", successfulCheckoutRequests: null });
});
it("does not infer a harness claim from status or task text", () => {
expect(observeCheckoutActivity({ ...input, runs: [{ ...run, contextSnapshot: { issueId: "issue" } }] }))
.toMatchObject({ runs: [{ checkedOutByHarness: false }] });
});
});
tests/runner-e2e/checkout-activity.ts
+36
View File
@@ -0,0 +1,36 @@
type Row = Record<string, any>;
/** Observation only. Activity counts successful route calls, not failed attempts. */
export function observeCheckoutActivity(input: {
companyId: string; issueId: string; agentId: string;
runs: Row[]; activity: Row[];
}) {
const runIds = new Set(input.runs.map(run => run.id));
const boundRuns = input.runs.length > 0 && runIds.size === input.runs.length &&
input.runs.every(run => typeof run.id === "string" && Boolean(run.id) &&
run.companyId === input.companyId && run.agentId === input.agentId &&
run.contextSnapshot?.issueId === input.issueId);
const rows = input.activity.filter(row => row.action === "issue.checked_out");
const validRows = rows.every(row => typeof row.id === "string" && Boolean(row.id) &&
row.companyId === input.companyId && row.entityType === "issue" &&
row.entityId === input.issueId && row.actorType === "agent" &&
row.agentId === input.agentId && runIds.has(row.runId));
const uniqueRows = new Set(rows.map(row => row.id)).size === rows.length;
const complete = boundRuns && validRows && uniqueRows;
return {
schema: "paperclip.checkout-activity.v1",
status: complete ? "observed" : "uncomparable",
companyId: input.companyId, issueId: input.issueId, agentId: input.agentId,
scope: "successful checkout HTTP calls only; failed attempts require run-log inspection",
successfulCheckoutRequests: complete ? rows.length : null,
runs: input.runs.map(run => ({
runId: run.id, status: run.status,
checkedOutByHarness: run.contextSnapshot?.paperclipWake?.checkedOutByHarness === true,
successfulCheckoutRequests: complete ? rows.filter(row => row.runId === run.id).length : null,
})),
receipts: rows.map(row => ({
id: row.id, runId: row.runId, companyId: row.companyId,
issueId: row.entityId, agentId: row.agentId, actorType: row.actorType,
})),
};
}
tests/runner-e2e/cleanup-verification.test.ts
+35 -1
View File
@@ -1,5 +1,39 @@
import { expect, it } from "vitest";
import { runCleanupWithObservers, verifyCleanupAssertions } from "./cleanup-verification.js";
import { mayAllocateRemoteResources, mustPreserveRecoveryState, runCleanupWithObservers, shouldKeepFailedDiagnostics, verifyCleanupAssertions } from "./cleanup-verification.js";
it("keeps late failures and incomplete publication for explicit diagnosis", () => {
expect(shouldKeepFailedDiagnostics({ enabled: true, expectedResults: 1, results: [{ status: "failed" }] })).toBe(true);
expect(shouldKeepFailedDiagnostics({ enabled: true, expectedResults: 2, results: [{ status: "passed" }] })).toBe(true);
expect(shouldKeepFailedDiagnostics({ enabled: true, expectedResults: 1, results: [] })).toBe(true);
expect(shouldKeepFailedDiagnostics({ enabled: true, expectedResults: 1, results: [{ status: "passed" }] })).toBe(false);
expect(shouldKeepFailedDiagnostics({ enabled: false, expectedResults: 1, results: [] })).toBe(false);
});
it("marks only environments that can allocate remote resources", () => {
expect(mayAllocateRemoteResources("daytona")).toBe(true);
expect(mayAllocateRemoteResources("local")).toBe(false);
});
it("retains uncertain remote allocation state even after every local process exits", () => {
expect(mustPreserveRecoveryState({ processCleanupFailed: false, results: [{ cleanup: "failed" }] })).toBe(true);
expect(mustPreserveRecoveryState({ processCleanupFailed: false, results: [{ cleanup: "not_started" }] })).toBe(false);
expect(mustPreserveRecoveryState({ processCleanupFailed: true, results: [{ cleanup: "passed" }] })).toBe(true);
expect(mustPreserveRecoveryState({ processCleanupFailed: false, results: [{ cleanup: "passed" }] })).toBe(false);
});
it("retains raw cleanup failures even when no evidence was published", () => {
expect(mustPreserveRecoveryState({ processCleanupFailed: false, resourceAdmissionStarted: true, results: [{ cleanup: "failed" }] })).toBe(true);
});
it("distinguishes pre-admission bootstrap failures from uncertain worker failures", () => {
const synthetic = { cleanup: "not_started", synthetic: true };
expect(mustPreserveRecoveryState({ processCleanupFailed: false, results: [synthetic] })).toBe(false);
expect(mustPreserveRecoveryState({ processCleanupFailed: false, resourceAdmissionStarted: true, results: [synthetic] })).toBe(true);
expect(mustPreserveRecoveryState({ processCleanupFailed: false, resourceAdmissionStarted: true, results: [] })).toBe(true);
expect(mustPreserveRecoveryState({ processCleanupFailed: false, results: [] })).toBe(false);
expect(mustPreserveRecoveryState({ processCleanupFailed: false, resourceAdmissionStarted: true, results: [{ cleanup: "not_started" }] })).toBe(false);
expect(mustPreserveRecoveryState({ processCleanupFailed: false, resourceAdmissionStarted: true, results: [{ cleanup: "passed" }] })).toBe(false);
});
it("retains a failed cleanup proof and still closes every later observer", async () => {
let closed = 0;
tests/runner-e2e/cleanup-verification.ts
+25
View File
@@ -1,6 +1,31 @@
export interface CleanupCheck { id: string; passed: boolean; detail: string }
export type CleanupAssertion = () => Promise<CleanupCheck[]>;
export function mayAllocateRemoteResources(environment: string) {
return environment === "daytona";
}
/** Diagnostic retention follows the final verdict, including publication errors. */
export function shouldKeepFailedDiagnostics(input: {
enabled: boolean;
expectedResults: number;
results: readonly { status: string }[];
}) {
return input.enabled && (input.results.length !== input.expectedResults ||
input.results.some(result => result.status === "failed"));
}
/** A stopped controller does not prove remote resources were retired. Retain
* its private recovery database whenever a cell lacks confirmed cleanup. */
export function mustPreserveRecoveryState(input: {
processCleanupFailed: boolean;
resourceAdmissionStarted?: boolean;
results: readonly { cleanup: string; synthetic?: boolean }[];
}) {
return input.processCleanupFailed || input.results.some(result => result.cleanup === "failed")
|| (input.resourceAdmissionStarted === true && (input.results.length === 0 || input.results.some(result => result.synthetic)));
}
/** Always finish every registered observer, even when an earlier proof fails. */
export async function verifyCleanupAssertions(assertions: readonly CleanupAssertion[]) {
const checks: CleanupCheck[] = [], errors: unknown[] = [];
tests/runner-e2e/codex-ci-sandbox.test.ts
+2 -2
View File
@@ -37,9 +37,9 @@ describe("Codex CI sandbox trust boundary", () => {
[...condition!.matchAll(/matrix\.profileId == '([^']+)'/g)].map((match) => match[1]),
);
expect([...provisionedProfiles].sort()).toEqual(
[...new Set(runnerMatrix.filter(requiresCodexCiSandbox).map((cell) => cell.profile.id))].sort(),
[...new Set(runnerMatrix.filter(cell => cell.task.flow !== "provider_connection").filter(requiresCodexCiSandbox).map((cell) => cell.profile.id))].sort(),
);
for (const cell of runnerMatrix) {
for (const cell of runnerMatrix.filter(cell => cell.task.flow !== "provider_connection")) {
expect(
cell.environment.id === "local" && provisionedProfiles.has(cell.profile.id),
cell.id,
tests/runner-e2e/connection-cancel.smoke.ts
+67
View File
@@ -0,0 +1,67 @@
/** Opt-in, credential-free browser smoke: teardown retains its authenticated API client. */
import { spawn } from "node:child_process";
import { createServer } from "node:http";
import path from "node:path";
import { parseConnectionConfig } from "./connection-config.js";
import { openConnectionBrowser } from "./connection-launch.js";
import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js";
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
if (process.argv[2] === "--worker") {
const origin = process.argv[3]!;
const browser = await openConnectionBrowser(parseConnectionConfig({ browser: {
headed: false, channel: "chromium", freshness: "signed-out",
} }), repositoryRoot, origin);
await browser.context.pages()[0]!.goto(origin);
let stopping = false;
for (const signal of ["SIGINT", "SIGTERM", "SIGHUP"] as const) process.once(signal, () => {
if (stopping) return;
stopping = true;
void (async () => {
let ok = false;
try { ok = (await browser.context.request.get(`${origin}/cleanup`)).ok(); }
finally { await browser.close(); }
process.send?.({ kind: "cleanup", ok });
process.disconnect();
process.exitCode = ok ? 0 : 1;
})().catch(() => { process.exitCode = 1; process.disconnect(); });
});
process.send?.({ kind: "ready" });
} else {
let cleanupRequests = 0;
const server = createServer((req, res) => {
if (req.url === "/cleanup") {
const authenticated = req.headers.cookie === "qa-cleanup=synthetic";
if (authenticated) cleanupRequests++;
res.writeHead(authenticated ? 200 : 401).end();
} else res.writeHead(200, { "Content-Type": "text/html", "Set-Cookie": "qa-cleanup=synthetic; HttpOnly; Path=/" }).end("<p>Synthetic cleanup fixture</p>");
});
await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));
const origin = `http://127.0.0.1:${(server.address() as { port: number }).port}`;
try {
for (const signal of ["SIGINT", "SIGTERM", "SIGHUP"] as const) {
const env = Object.fromEntries(["PATH", "HOME", "USER", "LOGNAME", "TMPDIR", "LANG"].flatMap(key => process.env[key] ? [[key, process.env[key]!]] : []));
await new Promise<void>((resolve, reject) => {
const child = spawn(process.execPath, [...runnerE2ETypeScriptProcessArgs(repositoryRoot, import.meta.filename), "--worker", origin], {
cwd: repositoryRoot, env, stdio: ["ignore", "ignore", "ignore", "ipc"],
});
let cleaned = false;
const timeout = setTimeout(() => { child.kill("SIGKILL"); reject(new Error(`${signal}: browser cleanup timed out`)); }, 30_000);
child.on("message", message => {
const event = message as { kind?: string; ok?: boolean };
if (event.kind === "ready") child.kill(signal);
if (event.kind === "cleanup") cleaned = event.ok === true;
});
child.once("error", error => { clearTimeout(timeout); reject(error); });
child.once("exit", code => {
clearTimeout(timeout);
if (code === 0 && cleaned) resolve();
else reject(new Error(`${signal}: browser closed before authenticated fixture cleanup`));
});
});
console.log(`${signal}: authenticated cleanup completed before browser close`);
}
if (cleanupRequests !== 3) throw new Error("Missing authenticated cleanup requests");
} finally { await new Promise<void>((resolve, reject) => server.close(error => error ? reject(error) : resolve())); }
}
tests/runner-e2e/connection-cases.ts
+88
View File
@@ -0,0 +1,88 @@
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import { AI_CONNECTION_CAPABILITIES } from "../../packages/shared/src/ai-connections.js";
import { isAiRoutingCompatible } from "../../packages/shared/src/ai-provider-routing.js";
import type { AiProviderRouting } from "../../packages/shared/src/ai-provider-routing.js";
import type { EnvironmentFixture, MatrixExecution, RunnerProfileFixture, RunnerSuiteFixture, RunnerTaskFixture } from "./types.js";
export const connectionModes = ["subscription", "api-key", "openrouter", "bedrock", "responses", "messages", "chat"] as const;
export type ConnectionMode = typeof connectionModes[number];
export type ConnectionEntry = "agent" | "apps";
export const connectionHarnesses = [
{ id: "claude", adapter: "claude_local", provider: "anthropic", label: "Claude", key: "ANTHROPIC_API_KEY", native: true },
{ id: "codex", adapter: "codex_local", provider: "openai", label: "OpenAI", key: "OPENAI_API_KEY", native: true },
{ id: "grok", adapter: "grok_local", provider: "xai", label: "Grok", key: "XAI_API_KEY", native: true },
{ id: "gemini", adapter: "gemini_local", provider: "google", label: "Google", key: "GEMINI_API_KEY", native: false },
{ id: "opencode", adapter: "opencode_local", provider: "openrouter", label: "OpenRouter", key: "OPENROUTER_API_KEY", native: true },
{ id: "hermes", adapter: "hermes_local", provider: "openrouter", label: "OpenRouter", key: "OPENROUTER_API_KEY", native: false },
] as const;
export type ConnectionHarness = typeof connectionHarnesses[number];
/** Inventory entries remain visible even when no live journey can be selected. */
export const connectionScopeGaps = [
{ adapter: "cursor", status: "unsupported", reason: "No managed connection flow; qualify the harness-specific login separately." },
{ adapter: "kimi_local", status: "unsupported", reason: "No managed connection flow; harness-specific credentials remain outside this suite." },
{ adapter: "pi_local", status: "unsupported", reason: "No managed connection flow or qualified model source." },
{ adapter: "openclaw_gateway", status: "excluded", reason: "Remote gateway owns provider configuration." },
{ adapter: "hermes_gateway", status: "excluded", reason: "Remote gateway owns provider configuration." },
{ adapter: "cursor_cloud", status: "excluded", reason: "Remote integration owns provider configuration." },
{ adapter: "process", status: "excluded", reason: "External process owns provider configuration." },
{ adapter: "http", status: "excluded", reason: "Remote integration owns provider configuration." },
{ adapter: "acpx_local", status: "excluded", reason: "Generic legacy integration; native Claude/Grok ACPX profiles remain covered." },
{ adapter: "claude_managed", status: "excluded", reason: "Managed remote integration owns credentials." },
{ adapter: "aws_agentcore", status: "excluded", reason: "Remote integration owns credentials." },
] as const;
export function routingForMode(mode: ConnectionMode): AiProviderRouting | undefined {
if (mode === "subscription" || mode === "api-key") return undefined;
return { kind: mode === "openrouter" || mode === "bedrock" ? mode : "gateway", protocol: mode === "openrouter" ? "responses" : mode, auth: "bearer", models: [], ...(mode === "bedrock" ? { region: "us-east-1" } : {}) };
}
export function supportsConnection(harness: ConnectionHarness, mode: ConnectionMode) {
const routing = routingForMode(mode);
if (routing) return isAiRoutingCompatible(routing, harness.adapter);
// OpenRouter's normal key is represented by its explicit routing case.
if (harness.provider === "openrouter") return false;
const methods = AI_CONNECTION_CAPABILITIES[harness.provider].methods;
return Boolean(methods[mode === "subscription" ? "subscription" : "api_key"]?.adapters.includes(harness.adapter));
}
export function connectionCell(execution: MatrixExecution) {
const match = /^(agent|apps)-(subscription|api-key|openrouter|bedrock|responses|messages|chat)$/.exec(execution.task.id);
const harness = connectionHarnesses.find(h => execution.profile.id === `connection-${h.id}-${execution.profile.generation}`);
if (execution.suite.id !== "provider-connections" || !harness || !match) throw new Error("Not a provider connection execution");
return { harness, entry: match[1] as ConnectionEntry, mode: match[2] as ConnectionMode };
}
export function buildConnectionSuite(profiles: readonly RunnerProfileFixture[], environments: readonly EnvironmentFixture[]): RunnerSuiteFixture {
const selectedProfiles = connectionHarnesses.flatMap(h => (["legacy", ...(h.native ? ["native"] : [])] as Array<"legacy" | "native">).map(generation => {
const existing = profiles.find(p => p.generation === generation && (p.adapterType === h.adapter || (generation === "native" && p.id === (h.id === "claude" || h.id === "grok" ? `runner-acpx-${h.id}` : `runner-${h.id}`))));
const id = `connection-${h.id}-${generation}`;
return {
id, label: `${h.label} ${generation}`, generation, groups: [generation], adapterType: generation === "native" ? "paperclip_runner" : h.adapter,
provider: h.id, model: existing?.model ?? "configured-in-connection-file",
modelQualification: existing?.modelQualification ?? { source: "candidate_runner_profile", qualificationId: "operator-configured-model" },
credential: h.key,
supportedEnvironments: ["local", "daytona"], expectedRuntimeMode: generation,
expectedRuntimeMetadata: { adapterType: generation === "native" ? "paperclip_runner" : h.adapter, provider: generation === "native" && ["claude", "grok"].includes(h.id) ? "acpx" : h.id },
buildAgent: () => ({ name: id, role: "qa", adapterType: generation === "native" ? "paperclip_runner" : h.adapter, adapterConfig: {}, runtimeConfig: {} }),
} satisfies RunnerProfileFixture;
}));
const tasks: RunnerTaskFixture[] = (["agent", "apps"] as const).flatMap(entry => connectionModes.map(mode => ({
id: `${entry}-${mode}`, label: `${entry === "apps" ? "Apps connector" : "New agent"} · ${mode}`, groups: [], flow: "provider_connection", workMode: "standard",
expectedRunCount: 3, minimumExpectedRunCount: 2,
attemptTimeoutMs: { local: 30 * 60_000, daytona: 30 * 60_000 }, turnTimeoutMs: 5 * 60_000,
expectedTerminalState: { issue: "done", run: "succeeded" },
buildTitle: nonce => `Connection proof ${nonce}`, buildPrompt: () => "Runtime-generated independent attachment proof.", buildVisibleMarker: nonce => nonce,
buildMatchers: () => [],
})));
const excludedExecutionIds = selectedProfiles.flatMap(profile => environments.flatMap(environment => tasks.flatMap(task => {
const harness = connectionHarnesses.find(h => profile.id === `connection-${h.id}-${profile.generation}`)!;
const mode = task.id.replace(/^(agent|apps)-/, "") as ConnectionMode;
return supportsConnection(harness, mode) ? [] : [`provider-connections.${profile.id}.${environment.id}.${task.id}`];
})));
const hash = createHash("sha256");
for (const file of ["connection-cases.ts", "connection-config.ts", "connection-flow.ts", "connection-evidence.ts", "connection-launch.ts", "connection-target.ts", "user-actions.ts"]) hash.update(file).update(readFileSync(new URL(`./${file}`, import.meta.url)));
const digest = hash.digest("hex");
return { id: "provider-connections", label: "Live provider connections", description: "Fresh production UI connections, attended subscription login, independently verified tasks and reuse on a selected deployment.", groups: [], manualOnly: true, profiles: selectedProfiles, environments, tasks, excludedExecutionIds, expectedMatrixSize: selectedProfiles.length * environments.length * tasks.length - excludedExecutionIds.length, definitionMetadata: { version: 1, digest, authentication: "fresh-ui-connection", traces: "disabled", scheduling: "explicit-only" } };
}
tests/runner-e2e/connection-config.ts
+162
View File
@@ -0,0 +1,162 @@
import { readFile, lstat, realpath } from "node:fs/promises";
import path from "node:path";
import { connectionCell, type ConnectionMode } from "./connection-cases.js";
import type { MatrixExecution } from "./types.js";
export interface ConnectionRoute {
model: string; credentialEnv?: string; baseURL?: string; region?: string; auth: "bearer" | "api_key" | "none";
}
export interface ConnectionConfig {
target: { mode: "managed-local" } | { mode: "attach"; baseURL: string; expectedCommit: string; deploymentMode: "local_trusted" | "authenticated"; companyId?: string; environmentId?: string };
browser: { headed: boolean; channel: "chrome" | "chromium"; profileDir?: string; accountAlias: string; freshness: "signed-in" | "signed-out"; loginTimeoutMs: number };
secretFile?: string;
models: Record<string, string>;
routes: Record<string, ConnectionRoute>;
credentials: Record<string, string>;
budgetCents: number; maxRuns: number; turnTimeoutMs: number; retainCompany: boolean;
}
/** Strict, dependency-free private configuration parser. Never echo bad input. */
export function parseConnectionConfig(input: unknown): ConnectionConfig {
const invalid = (): never => { throw new ConnectionBlock("blocked_target", "invalid_connection_configuration"); };
const object = (value: unknown, allowed?: string[]): Record<string, any> => {
if (!value || typeof value !== "object" || Array.isArray(value)) return invalid();
if (allowed && Object.keys(value).some(key => !allowed.includes(key))) return invalid();
return value as Record<string, any>;
};
const text = (value: unknown, fallback?: string): string => {
if (value === undefined && fallback !== undefined) return fallback;
if (typeof value !== "string" || !value.trim() || value.length > 2048) return invalid();
return value;
};
const choice = <T extends string>(value: unknown, options: readonly T[], fallback: T): T => value === undefined ? fallback : options.includes(value as T) ? value as T : invalid();
const flag = (value: unknown, fallback: boolean): boolean => value === undefined ? fallback : typeof value === "boolean" ? value : invalid();
const number = (value: unknown, fallback: number, min: number, max: number): number => value === undefined ? fallback : typeof value === "number" && Number.isSafeInteger(value) && value >= min && value <= max ? value : invalid();
const name = (value: unknown) => { const result = text(value); if (!/^[A-Z][A-Z0-9_]+$/.test(result)) return invalid(); return result; };
const uuid = (value: unknown) => { const result = text(value); if (!/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(result)) return invalid(); return result; };
const root = object(input, ["target", "browser", "secretFile", "models", "routes", "credentials", "budgetCents", "maxRuns", "turnTimeoutMs", "retainCompany"]);
const t = object(root.target ?? { mode: "managed-local" });
let target: ConnectionConfig["target"];
if (t.mode === "managed-local") { object(t, ["mode"]); target = { mode: "managed-local" }; }
else if (t.mode === "attach") {
object(t, ["mode", "baseURL", "expectedCommit", "deploymentMode", "companyId", "environmentId"]);
const expectedCommit = text(t.expectedCommit);
if (!/^[a-f0-9]{40}$/.test(expectedCommit)) return invalid();
target = { mode: "attach", baseURL: targetOrigin(text(t.baseURL)), expectedCommit, deploymentMode: choice(t.deploymentMode, ["local_trusted", "authenticated"], "authenticated"),
...(t.companyId === undefined ? {} : { companyId: uuid(t.companyId) }), ...(t.environmentId === undefined ? {} : { environmentId: uuid(t.environmentId) }) };
} else return invalid();
const b = object(root.browser ?? {}, ["headed", "channel", "profileDir", "accountAlias", "freshness", "loginTimeoutMs"]);
const accountAlias = text(b.accountAlias, "local-qa");
if (!/^[a-zA-Z0-9_-]{1,60}$/.test(accountAlias)) return invalid();
const routes = Object.fromEntries(Object.entries(object(root.routes ?? {})).map(([key, value]) => {
const r = object(value, ["model", "credentialEnv", "baseURL", "region", "auth"]);
return [key, { model: text(r.model), auth: choice(r.auth, ["bearer", "api_key", "none"], "bearer"),
...(r.credentialEnv === undefined ? {} : { credentialEnv: name(r.credentialEnv) }), ...(r.baseURL === undefined ? {} : { baseURL: text(r.baseURL) }), ...(r.region === undefined ? {} : { region: text(r.region) }) }];
}));
return { target, browser: { headed: flag(b.headed, true), channel: choice(b.channel, ["chrome", "chromium"], "chrome"), accountAlias, freshness: choice(b.freshness, ["signed-in", "signed-out"], "signed-in"), loginTimeoutMs: number(b.loginTimeoutMs, 600_000, 1000, 1200_000), ...(b.profileDir === undefined ? {} : { profileDir: text(b.profileDir) }) },
...(root.secretFile === undefined ? {} : { secretFile: text(root.secretFile) }),
models: Object.fromEntries(Object.entries(object(root.models ?? {})).map(([key, value]) => [key, text(value)])), routes,
credentials: Object.fromEntries(Object.entries(object(root.credentials ?? {})).map(([key, value]) => [key, name(value)])),
budgetCents: number(root.budgetCents, 200, 1, 10_000), maxRuns: number(root.maxRuns, 3, 2, 5), turnTimeoutMs: number(root.turnTimeoutMs, 300_000, 10_000, 600_000), retainCompany: flag(root.retainCompany, false) };
}
/** Only harness-owned error codes may enter a retained report. */
export class ConnectionFailure extends Error {}
export class ConnectionBlock extends Error {
constructor(readonly reason: "awaiting_user" | "missing_credential" | "blocked_target", readonly code: string) {
super(code);
}
}
export function targetOrigin(value: string): string {
const url = new URL(value);
const loopback = ["127.0.0.1", "localhost", "[::1]"].includes(url.hostname);
if (url.username || url.password || url.search || url.hash || url.pathname !== "/" ||
!(url.protocol === "https:" || (url.protocol === "http:" && loopback))) {
throw new ConnectionBlock("blocked_target", "target_requires_https_or_loopback_origin");
}
return url.origin;
}
export async function loadConnectionConfig(file?: string): Promise<ConnectionConfig> {
try {
const config = parseConnectionConfig(file ? JSON.parse(await readFile(file, "utf8")) : {});
if (config.target.mode === "attach") config.target.baseURL = targetOrigin(config.target.baseURL);
return config;
} catch (error) {
if (error instanceof ConnectionBlock) throw error;
// Parsing errors can contain configuration values. Never echo input.
throw new ConnectionBlock("blocked_target", "invalid_connection_configuration");
}
}
export async function assertPrivateFile(file: string) {
const stat = await lstat(file);
if (!stat.isFile() || stat.isSymbolicLink() || (stat.mode & 0o077) !== 0 || (process.getuid && stat.uid !== process.getuid())) {
throw new ConnectionBlock("missing_credential", "credential_file_requires_owner_only_regular_file");
}
}
/** Literal dotenv only: never evaluate shell expansions or source an entire file. */
export function selectedSecret(text: string, name: string): string | undefined {
let found: string | undefined;
for (const raw of text.split(/\r?\n/)) {
const line = raw.replace(/^\s*export\s+/, "");
const match = /^\s*([A-Z][A-Z0-9_]+)\s*=\s*(.*?)\s*$/.exec(line);
if (match?.[1] !== name) continue;
let value = match[2]!;
if ((value.startsWith('"') && value.endsWith('"')) || (value.startsWith("'") && value.endsWith("'"))) value = value.slice(1, -1);
if (!value || /[`\r\n]|\$\(/.test(value)) throw new ConnectionBlock("missing_credential", "credential_must_be_a_literal_value");
found = value;
}
return found;
}
export async function resolveConnectionSecret(config: ConnectionConfig, name?: string, env: NodeJS.ProcessEnv = process.env) {
if (!name) return undefined;
if (env[name]?.trim()) return env[name]!.trim();
if (config.secretFile) {
await assertPrivateFile(config.secretFile);
const value = selectedSecret(await readFile(config.secretFile, "utf8"), name);
if (value) return value;
}
throw new ConnectionBlock("missing_credential", `missing_${name}`);
}
export function resolveConnectionSettings(execution: MatrixExecution, config: ConnectionConfig) {
const { harness, mode } = connectionCell(execution);
const route = config.routes[`${harness.id}.${mode}`] ?? config.routes[mode];
const advanced = mode !== "subscription" && mode !== "api-key";
if (advanced && !route) throw new ConnectionBlock("missing_credential", "route_configuration_required");
const selectedModel = route?.model ?? config.models[harness.id] ?? execution.profile.model;
if (selectedModel === "configured-in-connection-file") throw new ConnectionBlock("blocked_target", "explicit_model_required");
if (["responses", "messages", "chat"].includes(mode)) {
if (!route?.baseURL) throw new ConnectionBlock("blocked_target", "gateway_url_required");
const url = new URL(route.baseURL);
if (url.username || url.password || url.search || url.hash || !["https:", "http:"].includes(url.protocol)) throw new ConnectionBlock("blocked_target", "invalid_gateway_url");
}
if (mode === "bedrock" && (!route?.region || route.auth !== "bearer")) throw new ConnectionBlock("blocked_target", "bedrock_region_and_bearer_required");
if (route?.auth === "api_key" && mode !== "messages") throw new ConnectionBlock("blocked_target", "api_key_header_requires_messages");
if (mode === "openrouter" && route?.auth !== "bearer") throw new ConnectionBlock("blocked_target", "openrouter_requires_bearer");
const defaultKey = mode === "bedrock" ? "AWS_BEARER_TOKEN_BEDROCK" : mode === "openrouter" ? "OPENROUTER_API_KEY" : harness.key;
const credentialEnv = mode === "subscription" || route?.auth === "none" ? undefined : route?.credentialEnv ?? config.credentials[harness.id] ?? (advanced && !["bedrock", "openrouter"].includes(mode) ? undefined : defaultKey);
if (mode !== "subscription" && route?.auth !== "none" && !credentialEnv) throw new ConnectionBlock("missing_credential", "gateway_credential_reference_required");
return { model: selectedModel, route, credentialEnv };
}
export function connectionCatalogSource(mode: ConnectionMode, provider: string) {
return ({ responses: "responses-api", messages: "messages-api", chat: "chat-completions-api", bedrock: "bedrock", openrouter: "openrouter" } as Record<string, string>)[mode] ?? provider;
}
export async function assertOutsideRepository(file: string, repositoryRoot: string) {
// Resolve an existing ancestor too, so a missing child below a symlink cannot evade containment.
let existing = path.resolve(file);
while (true) {
try { existing = await realpath(existing); break; }
catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; const parent = path.dirname(existing); if (parent === existing) throw error; existing = parent; }
}
const root = await realpath(repositoryRoot);
const relative = path.relative(root, existing);
if (relative === "" || (!relative.startsWith(`..${path.sep}`) && relative !== ".." && !path.isAbsolute(relative))) throw new ConnectionBlock("blocked_target", "browser_profile_must_be_outside_repository");
}
tests/runner-e2e/connection-evidence.ts
+129
View File
@@ -0,0 +1,129 @@
import { createHash, randomBytes } from "node:crypto";
import type { RunnerE2EResult } from "./types.js";
export const connectionCheckpoints = ["target", "connection_created", "connection_reloaded", "agent_created", "binding", "setup_probe", "first_run", "artifact", "tool_receipt", "followup", "visible_result"] as const;
export type ConnectionCheckpoint = typeof connectionCheckpoints[number];
export const connectionRunSignals = ["provider_overloaded", "provider_quota", "missing_file_conversion", "unsupported_session_configuration", "command_substitution_denied"] as const;
export interface ConnectionRunDiagnostic {
runId: string;
status: "succeeded" | "failed" | "cancelled" | "interrupted" | "timed_out" | "unknown";
signals: Array<typeof connectionRunSignals[number]>;
logAvailable: boolean;
}
export interface ConnectionEvidence {
version: 1;
outcome: "running" | "passed" | "failed" | "awaiting_user" | "missing_credential" | "blocked_target";
phase: string;
assisted: boolean;
authFreshness: "signed-in" | "signed-out";
target: { mode: "managed-local" | "attach"; origin: string; commit: string | null; deploymentMode: string | null };
entry: "apps" | "agent";
method: string;
checkpoints: Partial<Record<ConnectionCheckpoint, boolean>>;
waits: Array<{ kind: "board_login" | "provider_login"; startedAt: string; finishedAt?: string }>;
connectionId?: string;
agentId?: string;
companyId?: string;
artifactSha256?: string;
model?: string;
inputTransport?: "prompt_base64";
setupChecks?: Array<{ code: string; level: "info" | "warn" | "error" }>;
creationFailure?: { status: number; code: "ai_connection_api_key_rejected" | "ai_connection_verification_failed" | "connection_request_rejected" };
cleanupRetained?: boolean;
companyArchived?: boolean;
artifactChecks?: Array<{ filename: string; runId: string; sha256: string; fields: Record<string, boolean>; exactFields: boolean }>;
runDiagnostics?: ConnectionRunDiagnostic[];
}
/** Project terminal errors to closed diagnostic codes; never retain messages or reasoning. */
export function connectionRunDiagnostic(run: Record<string, any>, log?: unknown): ConnectionRunDiagnostic {
const signals = new Set<typeof connectionRunSignals[number]>();
const detect = (text: unknown) => {
if (typeof text !== "string") return;
if (/"code"\s*:\s*503\b|\bHTTP\s*503\b/i.test(text) && /overloaded|high demand/i.test(text)) signals.add("provider_overloaded");
if (/"code"\s*:\s*429\b|\bHTTP\s*429\b|\bRESOURCE_EXHAUSTED\b|\bTerminalQuotaError\b/.test(text)) signals.add("provider_quota");
if (/Error checking existing file: (?:Resource not found|Internal error)/.test(text)) signals.add("missing_file_conversion");
if (/set_config_option/.test(text) && /[Mm]ethod not found|-32601/.test(text)) signals.add("unsupported_session_configuration");
if (/Blocked: command substitution detected/.test(text)) signals.add("command_substitution_denied");
};
detect(run.error);
detect(run.stderrExcerpt);
const content = log && typeof log === "object" && typeof (log as Record<string, unknown>).content === "string"
? (log as { content: string }).content : undefined;
if (content !== undefined) {
let inner = "";
for (const line of content.split("\n")) {
try { const row = JSON.parse(line); if (typeof row.chunk === "string") inner += row.chunk; }
catch { /* Non-JSON application lines do not classify provider failures. */ }
}
for (const line of inner.split("\n")) {
try {
const row = JSON.parse(line);
if (row.type === "acpx.error") { detect(row.message); detect(row.childStderrTail); }
if (row.type === "acpx.tool_call" && row.status === "failed") detect(row.text);
} catch { /* Model text, partial records, and raw output are not retained. */ }
}
}
const statuses = ["succeeded", "failed", "cancelled", "interrupted", "timed_out"];
return { runId: run.id, status: statuses.includes(run.status) ? run.status : "unknown", signals: [...signals], logAvailable: content !== undefined };
}
export function gradeConnectionEvidence(evidence: ConnectionEvidence) {
return connectionCheckpoints.map(id => ({ matcher: { kind: "json_path" as const, path: `providerConnection.checkpoints.${id}`, expected: true }, passed: evidence.checkpoints[id] === true, detail: `${id}: ${evidence.checkpoints[id] === true ? "verified" : "not verified"}` }));
}
/** Retain stable product check codes, never provider messages or command output. */
export function connectionProbeChecks(value: unknown): NonNullable<ConnectionEvidence["setupChecks"]> {
if (!value || typeof value !== "object" || !Array.isArray((value as Record<string, unknown>).checks)) return [];
return ((value as { checks: unknown[] }).checks).slice(0, 50).flatMap(entry => {
if (!entry || typeof entry !== "object") return [];
const { code, level } = entry as Record<string, unknown>;
return typeof code === "string" && /^[a-z][a-z0-9_]{0,95}$/.test(code) && ["info", "warn", "error"].includes(String(level))
? [{ code, level: level as "info" | "warn" | "error" }] : [];
});
}
export function connectionEvidencePasses(evidence: ConnectionEvidence) {
return evidence.outcome === "passed" && Boolean(evidence.target.commit) && gradeConnectionEvidence(evidence).every(check => check.passed);
}
export function isBlockedConnectionResult(result: RunnerE2EResult) {
return Boolean(result.providerConnection && ["awaiting_user", "missing_credential", "blocked_target"].includes(result.providerConnection.outcome));
}
export function createConnectionProof() {
const nonce = randomBytes(12).toString("hex");
const values = Array.from({ length: 7 }, () => randomBytes(2).readUInt16BE());
const input = `${JSON.stringify({ nonce, values })}\n`;
const expected = { nonce, count: values.length, total: values.reduce((sum, n) => sum + n, 0), sha256: createHash("sha256").update(input).digest("hex") };
const followup = { nonce, minimum: Math.min(...values), maximum: Math.max(...values), total: expected.total };
return { input, expected, followup };
}
export function verifyConnectionArtifact(bytes: Buffer, expected: Record<string, unknown>) {
try {
const value = JSON.parse(bytes.toString("utf8"));
return Object.keys(value).length === Object.keys(expected).length && Object.entries(expected).every(([key, entry]) => value[key] === entry);
} catch { return false; }
}
/** Only durable run attribution is accepted; assistant text cannot provide it. */
export function verifyConnectionRun(run: Record<string, any>, expected: { agentId: string; connectionId: string; method: string; runtimeMode: string; environmentId: string }) {
const connection = run.contextSnapshot?.aiConnection;
return run.status === "succeeded" && run.agentId === expected.agentId && run.runtimeMode === expected.runtimeMode &&
connection?.connectionId === expected.connectionId && connection?.method === expected.method &&
run.contextSnapshot?.paperclipEnvironment?.id === expected.environmentId;
}
/** Wait for scheduled repairs to settle, then select the run that actually delivered the file. */
export function completedConnectionArtifactRun(
runs: Record<string, any>[],
attachments: Record<string, any>[],
filename: string,
expected: Parameters<typeof verifyConnectionRun>[1],
) {
const terminal = ["succeeded", "failed", "cancelled", "interrupted", "timed_out"];
if (runs.some(run => !terminal.includes(run.status))) return undefined;
return runs.find(run => verifyConnectionRun(run, expected) && attachments.some(attachment =>
attachment.originalFilename === filename && attachment.createdByAgentId === expected.agentId && attachment.originatingRunId === run.id));
}
tests/runner-e2e/connection-flow.ts
+372
View File
@@ -0,0 +1,372 @@
import { createHash } from "node:crypto";
import { expect, type Page } from "@playwright/test";
import { aiRoutingModel } from "../../packages/shared/src/ai-provider-routing.js";
import { getConnectableAppDefinition } from "../../packages/shared/src/app-definitions.js";
import { connectionCell, type ConnectionHarness, type ConnectionMode } from "./connection-cases.js";
import { ConnectionBlock, ConnectionFailure, type ConnectionConfig, type resolveConnectionSettings } from "./connection-config.js";
import { ConnectionApi, connectionDelay } from "./connection-target.js";
import { createConnectionProof, verifyConnectionArtifact, verifyConnectionRun, completedConnectionArtifactRun, connectionProbeChecks, type ConnectionEvidence } from "./connection-evidence.js";
import { createTaskThroughUi, submitTaskReply } from "./user-actions.js";
import { collectRunEvents } from "./run-observations.js";
import type { MatrixExecution } from "./types.js";
type Row = Record<string, any>;
type Settings = ReturnType<typeof resolveConnectionSettings>;
export interface ConnectionFlowInput {
page: Page; api: ConnectionApi; execution: MatrixExecution; config: ConnectionConfig; settings: Settings;
company: Row; environment: Row; secret?: string; nonce: string; privateDir: string;
evidence: ConnectionEvidence;
checkpoint(phase: string): Promise<void>;
observeRun(run: Row): void;
createdConnectionIds: ReadonlySet<string>;
refreshOwnedConnections(): Promise<void>;
assertActive(): void;
}
async function choose(page: Page, label: string, name: string) {
await page.getByRole("combobox", { name: label, exact: true }).click();
await page.getByRole("option", { name, exact: true }).click();
}
async function openAdvanced(page: Page, mode: ConnectionMode) {
const names: Record<string, string> = { openrouter: "OpenRouter", bedrock: "Amazon Bedrock", responses: "Custom gateway", messages: "Custom gateway", chat: "Custom gateway" };
await page.getByRole("button", { name: new RegExp(`^${names[mode]}`) }).click();
}
async function fillGateway(input: ConnectionFlowInput) {
const { page, settings, secret } = input;
const { mode } = connectionCell(input.execution);
if (settings.route?.baseURL) {
await page.getByLabel("Provider URL", { exact: true }).fill(settings.route.baseURL);
const format = page.getByRole("combobox", { name: "API format", exact: true });
if (await format.isEnabled()) await choose(page, "API format", ({ responses: "OpenAI Responses · Codex", messages: "Anthropic Messages · Claude", chat: "Chat Completions · OpenCode, Hermes" } as Record<string, string>)[mode]!);
}
if (mode === "bedrock") await page.getByLabel("AWS region", { exact: true }).fill(settings.route!.region!);
if (settings.route?.auth !== "bearer") await choose(page, "Authentication", settings.route?.auth === "none" ? "No authentication" : "API key · x-api-key");
if (secret) await page.getByLabel("API key", { exact: true }).fill(secret);
// Normal defaults stay collapsed. Model alias is selected in Configure.
await page.getByRole("button", { name: "Connect", exact: true }).last().click();
}
async function submitConnection(input: ConnectionFlowInput, submit: () => Promise<void>) {
const route = `${input.api.origin}/api/companies/${input.company.id}/ai-connections`;
const response = input.page.waitForResponse(response => response.url() === route && response.request().method() === "POST", { timeout: 60_000 }).catch(() => null);
await submit();
const created = await response;
if (!created) throw new ConnectionFailure("connection_request_timeout");
if (created.ok()) return;
const payload = await created.json().catch(() => ({}));
const allowed = ["ai_connection_api_key_rejected", "ai_connection_verification_failed"];
const code = allowed.includes(payload?.details?.code) ? payload.details.code : "connection_request_rejected";
input.evidence.creationFailure = { status: created.status(), code };
if (code === "ai_connection_api_key_rejected") throw new ConnectionBlock("missing_credential", code);
throw new ConnectionFailure(code);
}
async function waitForFreshConnection(input: ConnectionFlowInput, before: Set<string>) {
const { page, api, config, evidence, company } = input;
const { harness, mode } = connectionCell(input.execution);
const deadline = Date.now() + (mode === "subscription" ? config.browser.loginTimeoutMs : 90_000);
let loginOpened = false;
let connected = false;
while (Date.now() < deadline) {
input.assertActive();
await input.refreshOwnedConnections();
const list = await api.get(`/api/companies/${company.id}/ai-connections`);
const fresh = list.connections.filter((c: Row) => !before.has(c.id) && input.createdConnectionIds.has(c.id));
if (fresh.length > 1) throw new ConnectionFailure("unexpected_multiple_connections");
if (fresh.length === 1) {
const connection = fresh[0];
if (connection.status !== "connected" || connection.method !== (mode === "subscription" ? "subscription" : "api_key")) throw new ConnectionFailure("saved_connection_method_or_status_mismatch");
return connection;
}
if (mode === "subscription") {
const signIn = page.getByRole("link", { name: /^Sign in to / }).first();
const cloudSignIn = page.getByRole("button", { name: /^Sign in to / }).first();
if (!loginOpened && (await signIn.isVisible() || await cloudSignIn.isVisible())) {
// Observe before clicking. No token, code, URL, or popup content is recorded.
const popup = page.context().waitForEvent("page", { timeout: 10_000 }).catch(() => undefined);
if (await signIn.isVisible()) await signIn.click(); else if (await cloudSignIn.isEnabled()) await cloudSignIn.click();
await popup;
loginOpened = true;
evidence.assisted = true;
evidence.waits.push({ kind: "provider_login", startedAt: new Date().toISOString() });
await input.checkpoint("awaiting_provider_login");
console.log(`[provider-connections] Sign in to ${harness.label} in the opened browser (${config.browser.accountAlias}); complete the code/consent step in Paperclip. This test resumes automatically.`);
}
// Grok currently exposes a terminal handoff; do not pretend it has the same browser flow.
if (!loginOpened && harness.id === "grok" && await page.getByRole("button", { name: "Copy sign-in command" }).isVisible()) {
loginOpened = true;
evidence.assisted = true;
evidence.waits.push({ kind: "provider_login", startedAt: new Date().toISOString() });
await input.checkpoint("awaiting_provider_login");
console.log("[provider-connections] Grok requires the sign-in command shown in Paperclip. Complete that production handoff; the test will resume automatically.");
}
const connect = page.getByRole("button", { name: "Connect", exact: true }).last();
if (!connected && await connect.isVisible() && await connect.isEnabled()) { await connect.click(); connected = true; }
if (await page.getByRole("alert").filter({ hasText: /expired|could not|failed|invalid/i }).count()) throw new ConnectionFailure("provider_login_reported_error");
}
await connectionDelay(1_000);
}
if (mode === "subscription" && loginOpened) throw new ConnectionBlock("awaiting_user", "provider_login_deadline_reached");
throw new ConnectionFailure("connection_creation_timeout");
}
async function selectSavedConnection(page: Page, connection: Row, harness: ConnectionHarness, mode: ConnectionMode) {
await page.getByRole("heading", { name: /^(Connect a model|Configure your agent)$/ }).waitFor();
const advanced = !["subscription", "api-key"].includes(mode);
const connectionType = page.getByRole("radiogroup", { name: "Connection type", exact: true });
if (["claude", "codex", "grok"].includes(harness.id)) {
await connectionType.waitFor({ state: "visible" });
if (advanced) {
await connectionType.getByRole("radio", { name: /Advanced/ }).click();
await choose(page, "Connection", connection.name);
await page.getByRole("button", { name: "Use connection", exact: true }).click();
} else {
await connectionType.getByRole("radio", { name: mode === "subscription" ? /Subscription/ : /API key/ }).click();
const saved = page.getByRole("combobox", { name: mode === "subscription" ? "Saved subscription" : "Saved API key", exact: true });
await saved.waitFor({ state: "visible" });
const option = await saved.locator("option").filter({ hasText: connection.name }).first().getAttribute("value");
if (!option) throw new ConnectionFailure("saved_connection_option_missing");
await saved.selectOption(option);
await page.getByRole("button", { name: mode === "subscription" ? "Use saved subscription" : "Use saved API key", exact: true }).click();
}
} else {
await choose(page, "Connection", connection.name);
}
}
async function createConnectionThroughUi(input: ConnectionFlowInput, agentURL: string) {
const { page, api, company, evidence } = input;
const { harness, mode, entry } = connectionCell(input.execution);
const priorConnections = (await api.get(`/api/companies/${company.id}/ai-connections`)).connections as Row[];
const before = new Set<string>(priorConnections.map(c => c.id));
// Fresh-creation proof cannot be satisfied by an existing personal default.
if (priorConnections.some(connection => connection.status !== "revoked")) throw new ConnectionBlock("blocked_target", "fresh_connection_requires_empty_qa_company");
const advanced = !["subscription", "api-key"].includes(mode);
await input.checkpoint("connection_navigation");
if (entry === "apps") {
await page.goto(`${api.origin}/${company.issuePrefix}/apps`, { waitUntil: "domcontentloaded" });
const catalogSlug = advanced ? ({ openrouter: "openrouter", bedrock: "bedrock", responses: "responses-api", messages: "messages-api", chat: "chat-completions-api" } as Record<string, string>)[mode]! : harness.provider;
const definition = getConnectableAppDefinition(catalogSlug);
if (!definition) throw new ConnectionFailure("connection_catalog_entry_missing");
await page.getByRole("button", { name: `Connect ${definition.name}`, exact: true }).click();
const next = page.getByRole("button", { name: /^(Save and continue|Continue)$/ }).last();
if (!advanced && await next.isVisible()) await next.click();
if (!advanced && harness.provider !== "google") {
// The credential step loads asynchronously. Wait for its provider tile
// before inspecting the mode switch, otherwise a still-loading page can
// skip the API-key switch and leave the subscription form selected.
await page.getByRole("radio", { name: new RegExp(harness.id === "grok" ? "Grok" : harness.label) }).click();
const switcher = page.getByRole("button", { name: mode === "subscription" ? "Use subscription instead" : "Use API key instead", exact: true });
if (await switcher.isVisible()) await switcher.click();
}
} else {
await page.goto(agentURL, { waitUntil: "domcontentloaded" });
const tiles = page.getByRole("radiogroup", { name: "Connection type", exact: true });
if (["claude", "codex", "grok"].includes(harness.id)) {
const choice = tiles.getByRole("radio", { name: advanced ? /Advanced/ : mode === "subscription" ? /Subscription/ : /API key/ });
await choice.click();
await expect(choice).toHaveAttribute("aria-checked", "true");
}
if (advanced || !["claude", "codex", "grok"].includes(harness.id)) {
await choose(page, "Connection", "Connect an account…");
if (advanced) {
const advancedProviders = page.locator("summary").filter({ hasText: "Advanced providers" });
if (await advancedProviders.isVisible()) {
await advancedProviders.click();
// The account dialog's disclosure leads to the provider chooser;
// the chooser's own disclosure already contains provider buttons.
const chooseProvider = page.getByRole("button", { name: "Choose another provider or gateway", exact: true });
if (await chooseProvider.isVisible()) await chooseProvider.click();
}
await openAdvanced(page, mode);
}
}
}
await input.checkpoint("connection_creation");
if (advanced) await submitConnection(input, () => fillGateway(input));
else if (mode === "api-key") {
const savedKey = page.getByRole("combobox", { name: "Saved API key", exact: true });
if (await savedKey.isVisible()) await savedKey.selectOption("");
await page.getByLabel("API key", { exact: true }).fill(input.secret!);
await submitConnection(input, () => page.getByRole("button", { name: "Connect", exact: true }).last().click());
}
const connection = await waitForFreshConnection(input, before);
evidence.connectionId = connection.id;
evidence.checkpoints.connection_created = true;
evidence.waits.forEach(wait => { wait.finishedAt ??= new Date().toISOString(); });
if (advanced) {
const expectedKind = ["bedrock", "openrouter"].includes(mode) ? mode : "gateway";
if (connection.routing?.kind !== expectedKind || (mode !== "openrouter" && connection.routing?.protocol !== mode) ||
(input.settings.route?.baseURL && connection.routing?.baseUrl !== input.settings.route.baseURL) ||
(mode === "bedrock" && connection.routing?.region !== input.settings.route?.region)) throw new ConnectionFailure("saved_routing_mismatch");
} else if (connection.provider !== harness.provider) throw new ConnectionFailure("saved_provider_mismatch");
// Navigate away and return through the normal list: proof includes durable UI state.
await page.goto(`${api.origin}/${company.issuePrefix}/apps`, { waitUntil: "domcontentloaded" });
await expect(page.getByText(connection.name, { exact: true }).first()).toBeVisible();
await page.reload({ waitUntil: "domcontentloaded" });
await expect(page.getByText(connection.name, { exact: true }).first()).toBeVisible();
evidence.checkpoints.connection_reloaded = true;
await input.checkpoint("connection_reloaded");
await page.goto(agentURL, { waitUntil: "domcontentloaded" });
await input.checkpoint("select_saved_connection");
await selectSavedConnection(page, connection, harness, mode);
await input.checkpoint("configure_agent");
return connection;
}
async function selectModel(page: Page, model: string) {
const select = page.getByRole("combobox", { name: "Model", exact: true });
await select.waitFor({ state: "visible" });
await expect(select).not.toHaveAttribute("aria-busy", "true", { timeout: 60_000 });
const values = await select.locator("option").evaluateAll(options => options.map(option => (option as HTMLOptionElement).value));
const matched = values.find(value => value === model || value === `openrouter/${model}`);
if (matched) await select.selectOption(matched);
else {
await select.selectOption({ label: "Enter custom model…" });
await page.getByLabel("Model ID", { exact: true }).fill(model);
}
}
export async function runConnectionFlow(input: ConnectionFlowInput) {
const { page, api, execution, evidence, settings, company, environment, config } = input;
const { harness, mode } = connectionCell(execution);
const agentName = `Connection QA ${input.nonce}`;
const agentURL = `${api.origin}/${company.issuePrefix}/agents/new?${new URLSearchParams({ name: agentName, adapterType: execution.profile.adapterType, runnerProvider: harness.id })}`;
const connection = await createConnectionThroughUi(input, agentURL);
await expect(page.getByRole("heading", { name: "Configure your agent", exact: true })).toBeVisible({ timeout: 120_000 });
const environmentSelect = page.getByRole("combobox", { name: "Environment", exact: true });
const selectedEnv = await environmentSelect.inputValue();
if (selectedEnv !== environment.id) {
if (!(await environmentSelect.isEnabled())) throw new ConnectionBlock("blocked_target", "requested_execution_environment_unavailable");
await environmentSelect.selectOption(environment.id);
if (["claude", "codex", "grok"].includes(harness.id)) await selectSavedConnection(page, connection, harness, mode);
}
if (await environmentSelect.inputValue() !== environment.id) throw new ConnectionBlock("blocked_target", "requested_execution_environment_unavailable");
await input.checkpoint("select_model");
await selectModel(page, settings.model);
await input.checkpoint("setup_probe");
const response = page.waitForResponse(response => response.url().startsWith(`${api.origin}/api/companies/${company.id}/adapters/`) && response.url().endsWith("/test-environment"), { timeout: 120_000 });
await page.getByRole("button", { name: /^(Run test|Test again)$/ }).click();
const probe = await response;
const result = await probe.json();
evidence.setupChecks = connectionProbeChecks(result);
if (evidence.setupChecks.some(check => check.level === "error" && (check.code === "hermes_cli_not_found" || /^(claude|codex|grok|opencode|gemini)_command_unresolvable$/.test(check.code))))
throw new ConnectionBlock("blocked_target", "runtime_cli_required");
if (!probe.ok() || result.status !== "pass") throw new ConnectionFailure("setup_probe_failed");
evidence.checkpoints.setup_probe = true;
await page.getByRole("button", { name: "Finish setup", exact: true }).click();
let agent: Row | undefined;
const deadline = Date.now() + 60_000;
while (!agent && Date.now() < deadline) {
agent = (await api.get<Row[]>(`/api/companies/${company.id}/agents`)).find(row => row.name === agentName);
if (!agent) await connectionDelay(500);
}
if (!agent) throw new ConnectionFailure("agent_creation_failed");
if (agent.defaultEnvironmentId !== environment.id) {
// A null override is valid only when the instance default is this exact environment.
const instance = await api.get<Row>("/api/instance/settings");
const effective = agent.defaultEnvironmentId ?? instance.defaultEnvironmentId
?? (environment.driver === "local" ? environment.id : null);
if (effective !== environment.id) throw new ConnectionFailure("saved_agent_environment_mismatch");
}
evidence.agentId = agent.id;
evidence.checkpoints.agent_created = true;
const expectedModel = connection.routing
? aiRoutingModel(connection.routing, harness.adapter, settings.model)
: settings.model;
if (agent.adapterType !== execution.profile.adapterType || agent.adapterConfig?.model !== expectedModel) throw new ConnectionFailure("agent_harness_or_model_mismatch");
const binding = agent.runtimeConfig?.aiConnection;
if (!binding || binding.method !== connection.method || (binding.mode !== "responsible_user" && binding.connectionId !== connection.id)) throw new ConnectionFailure("agent_binding_mismatch");
// Fixtures may constrain cost/scheduling, but cannot repair or replace the tested credentials/config.
await api.patch(`/api/agents/${agent.id}`, { budgetMonthlyCents: config.budgetCents });
evidence.checkpoints.binding = true;
await input.checkpoint("first_task");
const proof = createConnectionProof();
// Provider qualification must not depend on task-level attachment ingestion.
// Native runners admit chat attachments, while the new-task dialog uploads
// its files after scheduling. Supply identical oracle bytes in the task so
// every harness can decode them with its actual tools.
evidence.inputTransport = "prompt_base64";
const title = `Connection artifact ${input.nonce}`;
const createdTask = await createTaskThroughUi({ page, issuePrefix: company.issuePrefix, agentName, title, workMode: "standard", prompt: [
"Complete this bounded connection acceptance task. Using your actual tools, decode the following base64 into connection-input.json and read the file. Preserve the exact decoded bytes, including the final newline.",
`Input bytes (base64): ${Buffer.from(proof.input, "utf8").toString("base64")}`,
"Create the input and output files inside your current workspace using relative filenames. Do not put them in /tmp or another directory outside the workspace.",
'Write connection-proof.json with exactly this schema: {"nonce": string, "count": number, "total": number, "sha256": string}. Set nonce from the input, count to the number of values, total to their sum, and sha256 to the hash of the exact input file bytes. All four fields are required; use the literal field name "total", not "sum". Compute and validate with a tool; do not guess or paraphrase the data.',
"Deliver connection-proof.json as a downloadable attachment on this task. Use the normal Paperclip artifact/attachment workflow and include its download link in your final response. Then mark this task done. Do not create other tasks, read credentials, or inspect unrelated files.",
execution.profile.generation === "native" ? "Use your native register_deliverable tool with workspace-relative contentRef connection-proof.json and the exact byteSize and SHA-256 of that output file. Never submit an absolute contentRef." : "",
].join("\n") });
const issue = await api.get<Row>(`/api/issues/${createdTask.issueId}`);
if (issue.companyId !== company.id) throw new ConnectionFailure("task_creation_failed");
const runs = new Map<string, Row>();
async function completedRun(after: Set<string>, filename: string) {
const deadline = Date.now() + config.turnTimeoutMs;
while (Date.now() < deadline) {
input.assertActive();
const summaries = await api.get<Row[]>(`/api/issues/${issue!.id}/runs`);
for (const row of summaries) {
const id = row.runId ?? row.id;
if (id && !runs.has(id)) runs.set(id, await api.get(`/api/heartbeat-runs/${id}`));
else if (id && !["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(runs.get(id)!.status)) runs.set(id, await api.get(`/api/heartbeat-runs/${id}`));
if (id) input.observeRun(runs.get(id)!);
}
if (runs.size > config.maxRuns) throw new ConnectionFailure("provider_run_limit_exceeded");
const next = [...runs.values()].filter(run => !after.has(run.id));
if (next.some(run => ["failed", "cancelled", "interrupted", "timed_out"].includes(run.status))) throw new ConnectionFailure("agent_run_failed");
const complete = next.find(run => verifyConnectionRun(run, { agentId: agent!.id, connectionId: connection.id, method: connection.method, runtimeMode: execution.profile.generation, environmentId: environment.id }));
if (complete) {
const status = (await api.get(`/api/issues/${issue!.id}`)).status;
if (status === "done") {
// A disposition repair may still be running after an earlier turn
// succeeded. Qualify the settled successful run that uploaded this
// artifact, preserving the configured run and time limits.
const attachments = await api.get<Row[]>(`/api/issues/${issue!.id}/attachments`);
const delivered = completedConnectionArtifactRun(next, attachments, filename,
{ agentId: agent!.id, connectionId: connection.id, method: connection.method, runtimeMode: execution.profile.generation, environmentId: environment.id });
if (delivered) return delivered;
if (next.every(run => ["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(run.status))) throw new ConnectionFailure("run_attributed_artifact_missing");
}
if (status === "blocked") throw new ConnectionFailure("task_blocked_after_provider_run");
}
if (next.some(run => run.status === "succeeded" && !verifyConnectionRun(run, { agentId: agent!.id, connectionId: connection.id, method: connection.method, runtimeMode: execution.profile.generation, environmentId: environment.id }))) throw new ConnectionFailure("run_attribution_mismatch");
await connectionDelay(1500);
}
throw new ConnectionFailure("agent_run_or_attribution_timeout");
}
async function artifact(filename: string, expected: Record<string, unknown>, run: Row) {
const attachments = await api.get<Row[]>(`/api/issues/${issue!.id}/attachments`);
const attachment = attachments.find(row => row.originalFilename === filename && row.createdByAgentId === agent!.id && row.originatingRunId === run.id);
if (!attachment) throw new ConnectionFailure("run_attributed_artifact_missing");
const bytes = await api.bytes(`/api/attachments/${attachment.id}/content`);
let actual: Record<string, unknown> = {};
try { actual = JSON.parse(bytes.toString("utf8")); } catch { /* The independent verifier fails below. */ }
evidence.artifactChecks ??= [];
evidence.artifactChecks.push({ filename, runId: run.id, sha256: createHash("sha256").update(bytes).digest("hex"),
fields: Object.fromEntries(Object.entries(expected).map(([key, value]) => [key, actual?.[key] === value])), exactFields: Boolean(actual && Object.keys(actual).length === Object.keys(expected).length) });
if (!verifyConnectionArtifact(bytes, expected)) throw new ConnectionFailure("independent_artifact_verification_failed");
await page.goto(`${api.origin}/${company.issuePrefix}/issues/${issue!.identifier}`, { waitUntil: "domcontentloaded" });
await page.reload({ waitUntil: "domcontentloaded" });
await expect(page.locator(`a[href*="/api/attachments/${attachment.id}/content"]`).first()).toBeVisible({ timeout: 30_000 });
evidence.artifactSha256 = createHash("sha256").update(bytes).digest("hex");
}
const first = await completedRun(new Set(), "connection-proof.json");
evidence.checkpoints.first_run = true;
await artifact("connection-proof.json", proof.expected, first);
evidence.checkpoints.artifact = true;
// Artifact upload provenance is a server-owned tool-side effect in both runner generations.
const events = await collectRunEvents<Row>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${first.id}/events?afterSeq=${afterSeq}&limit=${limit}`));
const toolReceipt = events.some(event => event.eventType === "tool.execution.completed" || (event.eventType === "item.completed" && ["tool_result", "command_execution"].includes(event.payload?.prpEvent?.payload?.item?.type)));
// Legacy adapters may expose their tool transcript only in the run log; upload attribution is still a durable successful tool effect.
evidence.checkpoints.tool_receipt = execution.profile.generation === "legacy" || toolReceipt;
if (!evidence.checkpoints.tool_receipt) throw new ConnectionFailure("native_tool_receipt_missing");
await input.checkpoint("followup");
const previousRuns = new Set(runs.keys());
await submitTaskReply(page, "Using the original connection-input.json decoded from this task, deliver connection-followup.json with exactly nonce, minimum, maximum, and total of the original values. Use exactly this JSON schema: {\"nonce\": string, \"minimum\": number, \"maximum\": number, \"total\": number}. All four fields are required. Compute with a tool and validate the file against all four fields before attaching it and its download link. Then mark this task done. Do not create other tasks or read credentials.");
const second = await completedRun(previousRuns, "connection-followup.json");
await artifact("connection-followup.json", proof.followup, second);
evidence.checkpoints.followup = true;
evidence.checkpoints.visible_result = true;
return { issueId: issue.id as string, issueIdentifier: issue.identifier as string };
}
tests/runner-e2e/connection-guidance-cases.ts
+39
View File
@@ -0,0 +1,39 @@
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import { everydayTasks } from "./everyday-cases.js";
export const CONNECTION_GUIDANCE_SUITE = "native-connection-guidance";
export const CONNECTION_GUIDANCE_BUDGET_CENTS = 1_000;
// The fallback defines the user's deliverable, not the connection procedure.
// No tool names, no-retry rule, provider preference, or completion API hints.
export const connectionGuidanceDeclinePrompts: Readonly<Record<string, string>> = {
"service-decline": "Use the connected page service to find recent pages and create a short Markdown briefing on this task. Include the titles and verification code returned by the service. If the data is unavailable, a brief explanation is enough instead of the briefing.",
"connection-decline": "Connect Notion so you can read my recent pages and write a short briefing. If the pages are unavailable, a brief explanation is enough instead of the briefing.",
"provider-decline": "Connect HubSpot so you can read my recent contacts. If the contacts are unavailable, a brief explanation is enough instead of the contact list.",
};
const cases = ["service-approve", "service-decline", "connection-decline", "provider-decline", "provider-second"];
export const connectionGuidanceTasks = cases.map(id => {
const original = everydayTasks.find(task => task.id === id);
if (!original) throw new Error("Missing original connection story: " + id);
return {
...original,
expectedRunCount: id === "provider-second" ? 3 : original.expectedRunCount,
automaticRetryPolicy: "single_attempt" as const,
buildPrompt: (nonce: string) => connectionGuidanceDeclinePrompts[id] ?? original.buildPrompt(nonce),
};
});
export function connectionGuidanceDefinitionDigest() {
const files = [
"connection-guidance-cases.ts", "connection-guidance-evidence.ts", "continuation-screenshot.ts",
"everyday-cases.ts", "everyday-flow.ts", "everyday-decisions.ts",
"everyday-observations.ts", "everyday-delivery.ts",
"connection-routing-evidence.ts", "connection-reviews.ts", "aggregator-fixture.ts",
"live-fixtures.ts", "runner.spec.ts", "user-actions.ts", "catalog.ts",
];
return createHash("sha256").update(files.map(file =>
file + "\0" + readFileSync(new URL(file, import.meta.url), "utf8"),
).join("\0")).digest("hex");
}
tests/runner-e2e/connection-guidance-evidence.ts
+74
View File
@@ -0,0 +1,74 @@
import type { StoryComment } from "./everyday-observations.js";
type Decision = {
id: string; kind: string; status: string; resolvedAt?: string | null;
result?: { answers?: Array<{ questionId: string; optionIds?: string[] }> };
};
type Reply = Pick<StoryComment, "authorAgentId" | "createdByRunId" | "createdAt" | "body">;
type Run = { id: string; nativeIssueId?: string | null; agentId?: string | null; status: string; finishedAt?: string | null };
// Additional oracle for the new neutral-prompt suite only. Historical Everyday
// grades remain intact. This proves attributed saved output, not cognition.
export type ConnectionGuidanceDeclineInput = {
caseId: "service-decline" | "connection-decline" | "provider-decline";
decisionId: string;
decisions: Decision[];
leadAgentId: string;
issueId: string;
replies: Reply[];
runs: Run[];
calls?: number;
marker: string;
sameConnections: boolean;
};
function declineReplyEvidence(input: ConnectionGuidanceDeclineInput) {
const decision = input.decisions.find(row => row.id === input.decisionId);
const resolvedAt = Date.parse(decision?.resolvedAt ?? "");
const provider = input.caseId === "provider-decline";
const expectedKind = provider ? "ask_user_questions"
: input.caseId === "connection-decline" ? "connection_intent" : "request_confirmation";
const options = decision?.result?.answers?.find(answer =>
answer.questionId === "connection-provider:hubspot")?.optionIds;
const validDecision = Number.isFinite(resolvedAt) &&
decision?.kind === expectedKind &&
(provider ? decision.status === "answered" && options?.length === 1 && options[0] === "none"
: decision?.status === "rejected");
const afterDecision = input.replies.filter(reply =>
validDecision && reply.authorAgentId === input.leadAgentId &&
Number.isFinite(Date.parse(reply.createdAt ?? "")) &&
Date.parse(reply.createdAt!) >= resolvedAt);
const finalRunTime = Math.max(...input.runs.filter(run =>
run.agentId === input.leadAgentId && run.nativeIssueId === input.issueId && run.status === "succeeded")
.map(run => Date.parse(run.finishedAt ?? "")).filter(Number.isFinite));
const attributed = afterDecision.filter(reply => input.runs.some(run =>
Boolean(reply.createdByRunId) && run.id === reply.createdByRunId && run.agentId === input.leadAgentId && run.nativeIssueId === input.issueId &&
run.status === "succeeded" && Date.parse(run.finishedAt ?? "") >= resolvedAt &&
Date.parse(run.finishedAt ?? "") === finalRunTime));
return { validDecision, afterDecision, attributed };
}
/** Readiness waits for storage, not a passing explanation. Wrong wording must still fail grading. */
export function hasConnectionGuidanceDeclineReply(input: ConnectionGuidanceDeclineInput): boolean {
return declineReplyEvidence(input).attributed.some(reply => typeof reply.body === "string" && reply.body.trim().length > 0);
}
/** Bounded textual evidence, not a claim about model reasoning or native skill activation. */
export function explainsConnectionUnavailable(text: string): boolean {
const normalized = text.replaceAll("’", "'");
return /(?<!not )\b(?:declin(?:e|ed|ing)|rejected|unable|unavailable)\b|\bnot now\b|\bcould(?:n't| not)\b|\bcannot\b|\bcan't\b|\bnot (?:connect|retriev)|\bwithout (?:access|connect)|\b(?:is|are)n't\s+(?:connect|retriev|available)|\b(?:was|were)n't able to (?:pull|access|read|retriev|connect)|\bno connection was made\b/i.test(normalized);
}
export function gradeConnectionGuidanceDecline(input: ConnectionGuidanceDeclineInput) {
const { validDecision, afterDecision, attributed } = declineReplyEvidence(input);
return [
{ id: "guidance-decline-decision", passed: validDecision && input.decisions.length === 1,
detail: "Exactly one correctly typed, resolved decline belongs to the selected decision." },
{ id: "guidance-decline-attributed-explanation",
passed: attributed.some(reply => typeof reply.body === "string" && explainsConnectionUnavailable(reply.body)),
detail: "A saved explanation follows the decision and joins by run ID to the lead's final successful execution on this task." },
{ id: "guidance-decline-no-use", passed: (input.caseId === "connection-decline" || input.calls === 0) && input.sameConnections &&
afterDecision.every(reply => (typeof reply.body !== "string" || !reply.body.includes(input.marker))),
detail: "No connection changes or unread marker in replies. Installed-service/provider declines also require an observed zero fixture-call count; Notion setup does not execute a service." },
];
}
tests/runner-e2e/connection-guidance.test.ts
+165
View File
@@ -0,0 +1,165 @@
import { describe, expect, it, vi } from "vitest";
import { runnerMatrix, runnerSuites } from "./catalog.js";
import { everydayTasks } from "./everyday-cases.js";
import { CONNECTION_GUIDANCE_SUITE, connectionGuidanceTasks, connectionGuidanceDefinitionDigest } from "./connection-guidance-cases.js";
import { gradeConnectionGuidanceDecline, hasConnectionGuidanceDeclineReply } from "./connection-guidance-evidence.js";
import { pollUntil } from "./api.js";
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
describe("neutral connection guidance selection", () => {
it("requires explicit selection and admits five stories on exactly three local native profiles", () => {
const selected = selectRunnerExecutions(parseRunnerSelectors(["--suite", CONNECTION_GUIDANCE_SUITE]));
expect(selected).toHaveLength(15);
expect(new Set(selected.map(row => row.profile.id))).toEqual(new Set(["runner-codex", "runner-acpx-claude", "runner-opencode"]));
for (const row of selected) {
expect(row.environment.id).toBe("local");
expect(row.task.automaticRetryPolicy).toBe("single_attempt");
expect(row.task.attemptTimeoutMs.local).toBe(720_000);
expect(row.task.expectedRunCount).toBe(row.task.id === "provider-second" ? 3 : 2);
}
for (const args of [["--all"], ["--profile", "runner-opencode"]]) {
expect(selectRunnerExecutions(parseRunnerSelectors(args), runnerMatrix).some(row => row.suite.id === CONNECTION_GUIDANCE_SUITE)).toBe(false);
}
expect(runnerSuites.find(suite => suite.id === CONNECTION_GUIDANCE_SUITE)?.definitionMetadata)
.toMatchObject({ fixtureDigest: connectionGuidanceDefinitionDigest(), companyAndAgentBudgetCents: 1_000, maximumAttemptsPerCell: 1 });
});
it("keeps workflow instructions out of new decline prompts and preserves the historical prompts", () => {
for (const task of connectionGuidanceTasks) {
const original = everydayTasks.find(row => row.id === task.id)!;
if (task.id.endsWith("decline")) {
expect(task.buildPrompt("nonce")).toMatch(/brief explanation is enough/);
expect(task.buildPrompt("nonce")).not.toMatch(/declin|Not now|None for now|do not|retry|try again|connection_request|connections_search|paperclip_|yield|poll/i);
expect(task.buildPrompt("nonce")).not.toBe(original.buildPrompt("nonce"));
} else expect(task.buildPrompt("nonce")).toBe(original.buildPrompt("nonce"));
}
expect(everydayTasks.find(task => task.id === "service-decline")!.buildPrompt("nonce")).toContain("do not try again");
expect(everydayTasks.find(task => task.id === "connection-decline")!.buildPrompt("nonce")).toContain("do not try again");
});
});
const valid = {
caseId: "provider-decline" as const, decisionId: "decision", leadAgentId: "lead", issueId: "task",
decisions: [{ id: "decision", kind: "ask_user_questions", status: "answered", resolvedAt: "2026-10-06T22:00:00Z",
result: { answers: [{ questionId: "connection-provider:hubspot", optionIds: ["none"] }] } }],
replies: [{ authorAgentId: "lead", createdByRunId: "run", createdAt: "2026-10-06T22:00:02Z", body: "Contacts are unavailable after your decision." }],
runs: [{ id: "run", nativeIssueId: "task", agentId: "lead", status: "succeeded", finishedAt: "2026-10-06T22:00:03Z" }],
calls: 0, marker: "PRIVATE_MARKER", sameConnections: true,
};
const passes = (input: Parameters<typeof gradeConnectionGuidanceDecline>[0]) =>
gradeConnectionGuidanceDecline(input).every(check => check.passed);
describe("neutral decline evidence", () => {
it("accepts an attributed explanation after each kind of saved decline", () => {
expect(passes(valid)).toBe(true);
for (const [caseId, kind] of [["service-decline", "request_confirmation"], ["connection-decline", "connection_intent"]] as const) {
expect(passes({ ...valid, caseId, decisions: [{ ...valid.decisions[0], kind, status: "rejected" }] })).toBe(true);
}
});
it("recognizes unavailable-connection contractions without accepting positive or unrelated claims", () => {
for (const body of [
"HubSpot isn't connected because you chose None for now.",
"HubSpot isn’t connected after the provider decision.",
"I wasn't able to pull your recent HubSpot contacts. You chose None for now, so no connection was made.",
"The page service approval was rejected, so the data was unavailable.",
"The pages aren't retrievable after the connection decision.",
]) expect(passes({ ...valid, replies: [{ ...valid.replies[0], body }] })).toBe(true);
for (const body of ["HubSpot is connected.", "HubSpot isn't disconnected.", "HubSpot isn't slow.", "Contacts are available.", "The request was not declined.", "The request was not rejected.", "Contacts are not unavailable.", "I wasn't able to finish my coffee."]) {
expect(passes({ ...valid, replies: [{ ...valid.replies[0], body }] })).toBe(false);
}
});
it("rejects missing, stale, unattributed, wrong-agent, or unsuccessful explanations", () => {
for (const input of [
{ ...valid, replies: [] },
{ ...valid, replies: [{ ...valid.replies[0], body: "Done." }] },
{ ...valid, replies: [{ ...valid.replies[0], body: { text: "Unavailable" } }] },
{ ...valid, replies: [{ ...valid.replies[0], createdAt: "2026-10-06T21:59:00Z" }] },
{ ...valid, replies: [{ ...valid.replies[0], createdByRunId: undefined }] },
{ ...valid, replies: [{ ...valid.replies[0], createdByRunId: null, runId: "run" }] },
{ ...valid, replies: [{ ...valid.replies[0], createdByRunId: "other-run" }] },
{ ...valid, replies: [{ ...valid.replies[0], authorAgentId: "worker" }] },
{ ...valid, replies: [{ ...valid.replies[0], createdAt: "invalid" }] },
{ ...valid, runs: [{ ...valid.runs[0], agentId: "worker" }] },
{ ...valid, runs: [{ ...valid.runs[0], nativeIssueId: "other-task" }] },
{ ...valid, runs: [{ ...valid.runs[0], nativeIssueId: undefined }] },
{ ...valid, runs: [{ ...valid.runs[0], status: "failed" }] },
{ ...valid, runs: [{ ...valid.runs[0], finishedAt: "2026-10-06T21:59:00Z" }] },
]) expect(passes(input)).toBe(false);
});
it("requires output from the final successful task run, not a late comment from an earlier wait", () => {
const runs = [...valid.runs, { ...valid.runs[0], id: "final", finishedAt: "2026-10-06T22:00:06Z" }];
expect(hasConnectionGuidanceDeclineReply({ ...valid, runs })).toBe(false);
expect(passes({ ...valid, runs })).toBe(false);
const input = { ...valid, runs, replies: [{ ...valid.replies[0], createdByRunId: "final", createdAt: "2026-10-06T22:00:07Z" }] };
expect(hasConnectionGuidanceDeclineReply(input)).toBe(true);
expect(passes(input)).toBe(true);
});
it.each(["delayed", "missing", "wrong-run", "wrong-content"])("waits for durable output within the original deadline: %s", async variant => {
vi.useFakeTimers();
try {
const start = Date.now();
const input = structuredClone({ ...valid, replies: [] as typeof valid.replies });
const pending = pollUntil({ label: "final decline reply", deadlineAt: start + 30_000, intervalMs: 1000,
load: async () => {
if (variant !== "missing" && Date.now() >= start + 20_000) input.replies = [{ ...valid.replies[0],
createdByRunId: variant === "wrong-run" ? "other" : "run",
body: variant === "wrong-content" ? "Done." : valid.replies[0].body }];
return input;
},
accept: hasConnectionGuidanceDeclineReply,
}).catch((error: unknown) => error);
await vi.advanceTimersByTimeAsync(30_001);
const result = await pending;
if (variant === "missing" || variant === "wrong-run") expect(result).toBeInstanceOf(Error);
else {
expect(result).toEqual(input);
// Readiness must not wait for favorable wording or silently convert a bad answer into a pass.
expect(passes(input)).toBe(variant === "delayed");
}
} finally { vi.useRealTimers(); }
});
it("rejects wrong or repeated decisions, early use, missing call evidence and connection changes", () => {
for (const input of [
{ ...valid, decisionId: "other" },
{ ...valid, decisions: [] },
{ ...valid, decisions: [...valid.decisions, ...valid.decisions] },
{ ...valid, decisions: [{ ...valid.decisions[0], resolvedAt: undefined }] },
{ ...valid, decisions: [{ ...valid.decisions[0], kind: "request_confirmation" }] },
{ ...valid, decisions: [{ ...valid.decisions[0], status: "pending" }] },
{ ...valid, decisions: [{ ...valid.decisions[0], result: { answers: [{ questionId: "connection-provider:hubspot", optionIds: ["via:arcade:hubspot"] }] } }] },
{ ...valid, calls: 1 },
{ ...valid, calls: undefined },
{ ...valid, sameConnections: false },
{ ...valid, replies: [{ ...valid.replies[0], body: "Unavailable, but PRIVATE_MARKER" }] },
]) expect(passes(input)).toBe(false);
});
});
import { requireProviderAccessCard, gradeProviderOutcome } from "./connection-routing-evidence.js";
const access = { id: "access", kind: "connection_intent", status: "pending", payload: {
serviceSlug: "arcade", requestingAgentId: "lead", upstreamService: { slug: "hubspot", selectionInteractionId: "choice" },
accessRequest: { connectionId: "connection", tools: [{ catalogEntryId: "tool", toolName: "Hubspot_ListContacts", permission: "allowed" }] },
} };
const accessInput = { rows: [{ id: "choice", kind: "ask_user_questions", status: "answered" }, access],
decisionId: "choice", connectionId: "connection", agentId: "lead", catalogEntryIds: ["tool"], calls: 0 };
it("requires the exact scoped access card and rejects early calls, swapped identities and duplicates", () => {
expect(requireProviderAccessCard(accessInput)).toBe(access);
for (const input of [
{ ...accessInput, rows: [] }, { ...accessInput, calls: 1 }, { ...accessInput, catalogEntryIds: [] },
{ ...accessInput, connectionId: "other" }, { ...accessInput, agentId: "other" }, { ...accessInput, decisionId: "other" },
{ ...accessInput, rows: [...accessInput.rows, access] },
{ ...accessInput, rows: [accessInput.rows[0], { ...access, status: "accepted" }] },
{ ...accessInput, rows: [accessInput.rows[0], { ...access, payload: { ...access.payload, upstreamService: { slug: "hubspot", selectionInteractionId: "other" } } }] },
]) expect(() => requireProviderAccessCard(input)).toThrow();
});
it("requires the real accepted access result without allowing repeated provider questions", () => {
const choice = { id: "choice", kind: "ask_user_questions", status: "answered", result: { answers: [{ questionId: "connection-provider:hubspot", optionIds: ["via:arcade:hubspot"] }] } };
const granted = { ...access, status: "accepted", result: { outcome: "connected", connectionId: "connection" } };
const input = { rows: [choice, granted], decisionId: "choice", selected: "via:arcade:hubspot", calls: 1,
response: "Ada Fixture MARKER", marker: "MARKER", sameConnections: true, accessDecision: { id: "access", connectionId: "connection" } };
expect(gradeProviderOutcome(input).every(c => c.passed)).toBe(true);
for (const bad of [{ ...input, rows: [choice] }, { ...input, rows: [choice, granted, choice] },
{ ...input, rows: [choice, { ...granted, status: "pending" }] }, { ...input, accessDecision: { id: "access", connectionId: "other" } },
{ ...input, calls: 0 }, { ...input, calls: 2 }, { ...input, response: "Done" }])
expect(gradeProviderOutcome(bad).every(c => c.passed)).toBe(false);
});
tests/runner-e2e/connection-launch.ts
+394
View File
@@ -0,0 +1,394 @@
import { chromium, type BrowserContext } from "@playwright/test";
import { execFileSync } from "node:child_process";
import { randomBytes, createHash } from "node:crypto";
import { mkdir, mkdtemp, writeFile, rm, readdir, lstat, open } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { FixtureRegistry } from "./fixture-registry.js";
import { buildRunnerCampaign } from "./history.js";
import { renderRunnerE2EDashboard } from "./dashboard.js";
import { buildRuntimeUsage, summarizeExecutionBilling } from "./billing.js";
import { stageDashboardBrandAssets } from "./report-assets.js";
import { classifyFailure } from "./failure-classifier.js";
import { packageEvidence } from "./evidence.js";
import { assertSecretFree, sanitizeJson } from "./redaction.js";
import { connectionCell, connectionHarnesses, connectionModes, connectionScopeGaps, supportsConnection } from "./connection-cases.js";
import { assertOutsideRepository, ConnectionBlock, ConnectionFailure, loadConnectionConfig, resolveConnectionSecret, resolveConnectionSettings, type ConnectionConfig } from "./connection-config.js";
import { connectionEvidencePasses, connectionRunDiagnostic, gradeConnectionEvidence, type ConnectionEvidence } from "./connection-evidence.js";
import { ConnectionApi, connectionDelay, startConnectionTarget, verifyConnectionTarget } from "./connection-target.js";
import { runConnectionFlow } from "./connection-flow.js";
import type { MatrixExecution, RunnerE2EResult } from "./types.js";
type Row = Record<string, any>;
export function connectionCreationReceipt(pathname: string, method: string, companyId: string, loginIds: ReadonlySet<string>, body: unknown): string | undefined {
const root = `/api/companies/${companyId}/ai-connections`;
const loginId = pathname.startsWith(`${root}/login/`) ? pathname.slice(`${root}/login/`.length) : undefined;
const createdHere = method === "POST" && (pathname === root || pathname === `${root}/local`);
const ownedLogin = method === "GET" && loginId && loginIds.has(loginId);
if (!createdHere && !ownedLogin) return;
const id = (body as { connectionId?: unknown } | null)?.connectionId;
return typeof id === "string" && /^[a-f0-9]{8}-(?:[a-f0-9]{4}-){3}[a-f0-9]{12}$/i.test(id) ? id : undefined;
}
export async function cleanupConnectionCompany(input: {
api: ConnectionApi; companyId: string; agentName: string;
attachedCompany: boolean; retainCompany: boolean;
createdConnectionIds: ReadonlySet<string>;
evidence: Pick<ConnectionEvidence, "cleanupRetained" | "companyArchived">;
collectDiagnostics: () => Promise<void>;
}) {
const { api, companyId, agentName, evidence } = input;
const agents = await api.get<Row[]>(`/api/companies/${companyId}/agents`);
const fixtureAgents = agents.filter(agent => agent.name === agentName);
try {
for (const agent of fixtureAgents) {
await api.patch(`/api/agents/${agent.id}`, { status: "paused" });
const liveRuns = await api.get<Row[]>(`/api/companies/${companyId}/heartbeat-runs?agentId=${agent.id}&limit=100`);
for (const run of liveRuns.filter(run => run.agentId === agent.id && ["queued", "running"].includes(run.status))) {
await api.post(`/api/heartbeat-runs/${run.id}/cancel`);
}
}
} finally {
// Deleting an attached-company agent also deletes its runs and logs.
// Capture the stopped runs first, including when cancellation fails.
await input.collectDiagnostics();
}
if (input.retainCompany) { evidence.cleanupRetained = true; return; }
const connections = (await api.get(`/api/companies/${companyId}/ai-connections`)).connections;
for (const connection of connections.filter((connection: Row) => input.createdConnectionIds.has(connection.id) && connection.status !== "revoked")) {
await api.delete(`/api/tool-connections/${connection.id}`);
}
if ((await api.get(`/api/companies/${companyId}/ai-connections`)).connections.some((connection: Row) => input.createdConnectionIds.has(connection.id) && connection.status !== "revoked")) {
throw new ConnectionFailure("connection_revocation_failed");
}
if (!input.attachedCompany) { await api.post(`/api/companies/${companyId}/archive`); evidence.companyArchived = true; }
else for (const agent of fixtureAgents) await api.delete(`/api/agents/${agent.id}`);
}
export async function openConnectionBrowser(config: ConnectionConfig, repositoryRoot: string, origin: string) {
const temporary = config.browser.freshness === "signed-out" || !config.browser.profileDir;
const profile = temporary ? await mkdtemp(path.join(os.tmpdir(), "paperclip-qa-browser-")) : path.resolve(config.browser.profileDir!);
await assertOutsideRepository(profile, repositoryRoot);
await mkdir(profile, { recursive: true, mode: 0o700 });
const stat = await lstat(profile);
if (stat.isSymbolicLink() || !stat.isDirectory() || (stat.mode & 0o077) !== 0) throw new ConnectionBlock("blocked_target", "browser_profile_requires_private_directory");
const marker = path.join(profile, ".paperclip-provider-qa");
const entries = await readdir(profile);
if (entries.length && !entries.includes(path.basename(marker))) throw new ConnectionBlock("blocked_target", "use_a_dedicated_empty_qa_browser_profile");
if (!entries.length) await writeFile(marker, "Dedicated Paperclip provider QA browser. Contains private authentication state.\n", { mode: 0o600, flag: "wx" });
const lock = path.join(profile, ".paperclip-qa.lock");
try { await (await open(lock, "wx", 0o600)).close(); }
catch { throw new ConnectionBlock("blocked_target", "qa_browser_profile_in_use"); }
let context: BrowserContext;
try {
const browserEnv = Object.fromEntries(["PATH", "HOME", "USER", "LOGNAME", "TMPDIR", "DISPLAY", "WAYLAND_DISPLAY", "XAUTHORITY", "LANG"].flatMap(key => process.env[key] ? [[key, process.env[key]!]] : []));
context = await chromium.launchPersistentContext(profile, {
channel: config.browser.channel === "chrome" ? "chrome" : undefined,
headless: !config.browser.headed, baseURL: origin, viewport: { width: 1440, height: 1000 }, reducedMotion: "reduce",
env: browserEnv, acceptDownloads: true,
// Teardown needs this context's authenticated request client to cancel
// login sessions and revoke fixtures before the browser is closed.
handleSIGINT: false, handleSIGTERM: false, handleSIGHUP: false,
// Deliberately use the Playwright library without Test's automatic reporters,
// traces, HARs, videos, snapshots, assertion call logs, or failure screenshots.
});
context.setDefaultTimeout(30_000);
context.setDefaultNavigationTimeout(30_000);
} catch {
await rm(lock, { force: true });
if (temporary) await rm(profile, { recursive: true, force: true });
throw new ConnectionBlock("blocked_target", "qa_browser_launch_failed");
}
return { context, close: async () => { try { await context.close(); } finally { await rm(lock, { force: true }); if (temporary) await rm(profile, { recursive: true, force: true }); } } };
}
export function safeConnectionFailure(error: unknown, phase: string) {
if (error instanceof ConnectionBlock) return error.code;
// Arbitrary Playwright/provider messages can contain filled values and URLs.
const known = error instanceof ConnectionFailure;
const frame = error instanceof Error ? error.stack?.match(/connection-(?:flow|target|launch)\.ts:\d+:\d+/)?.[0] : undefined;
return known ? error.message : `${phase}_failed${frame ? ` (${frame})` : ""}`;
}
function sourceRevision(repositoryRoot: string) {
const git = (args: string[]) => execFileSync("git", args, { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim();
return { sha: git(["rev-parse", "HEAD"]), ref: `${git(["branch", "--show-current"])}${git(["status", "--porcelain"]) ? " (working tree)" : ""}`, workflowRunUrl: null };
}
async function safeWrite(file: string, value: unknown, secrets: readonly string[]) {
const text = `${JSON.stringify(sanitizeJson(value, secrets), null, 2)}\n`;
assertSecretFree(text, secrets, path.basename(file));
await writeFile(file, text, { mode: 0o600 });
}
async function connectionAttempt(execution: MatrixExecution, config: ConnectionConfig, origin: string, summaryDir: string, repositoryRoot: string, signal: AbortSignal): Promise<RunnerE2EResult> {
const startedAt = new Date().toISOString();
const nonce = randomBytes(6).toString("hex");
const privateDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-connection-proof-"));
const evidenceDir = path.join(summaryDir, execution.suite.id, execution.profile.id, execution.environment.id, execution.task.id, "attempt-1");
await mkdir(evidenceDir, { recursive: true, mode: 0o700 });
const { entry, mode } = connectionCell(execution);
const evidence: ConnectionEvidence = { version: 1, outcome: "running", phase: "target", assisted: false, authFreshness: config.browser.freshness,
target: { mode: config.target.mode, origin, commit: null, deploymentMode: null }, entry, method: mode, checkpoints: {}, waits: [] };
const secrets: string[] = [];
const checkpoint = async (phase: string) => { evidence.phase = phase; await safeWrite(path.join(summaryDir, "progress.json"), { executionId: execution.id, ...evidence }, secrets); };
const result: RunnerE2EResult = { schema: "paperclip.runner-e2e.result/v2", executionId: execution.id, suiteId: execution.suite.id, suiteDefinitionHash: execution.suiteDefinitionHash,
source: sourceRevision(repositoryRoot), attempt: 1, status: "failed", profileId: execution.profile.id, environmentId: execution.environment.id, caseId: execution.task.id,
provider: execution.profile.provider, model: execution.profile.model, runtimeMode: execution.profile.generation, startedAt, finishedAt: startedAt, durationMs: 0, cleanup: "not_started", providerConnection: evidence };
let browser: Awaited<ReturnType<typeof openConnectionBrowser>> | undefined;
let fixtures: Awaited<ReturnType<FixtureRegistry["setupAll"]>> | undefined;
const observedRuns = new Map<string, Row>();
const createdConnectionIds = new Set<string>();
const ownedLoginIds = new Set<string>();
let fixtureCompanyId: string | undefined;
let diagnosticApi: ConnectionApi | undefined;
let diagnosticsCollected = false;
const collectDiagnostics = async () => {
if (diagnosticsCollected || !diagnosticApi || !observedRuns.size) return;
evidence.runDiagnostics = [];
for (const observed of observedRuns.values()) {
let run = observed; let log: unknown;
try { run = await diagnosticApi.get(`/api/heartbeat-runs/${observed.id}`); observedRuns.set(observed.id, run); } catch { /* Keep the last durable observation. */ }
try { log = await diagnosticApi.get(`/api/heartbeat-runs/${observed.id}/log?limitBytes=2000000`); } catch { /* Mark missing log evidence explicitly. */ }
evidence.runDiagnostics.push(connectionRunDiagnostic(run, log));
}
diagnosticsCollected = true;
};
const loginSessions = new Map<string, "DELETE" | "POST">();
const loginObservers: Promise<void>[] = [];
let stopped: "test_interrupted" | "cell_deadline_reached" | undefined;
const stopBrowserActions = () => { for (const page of browser?.context.pages() ?? []) void page.close().catch(() => {}); };
const interrupt = () => { stopped = "test_interrupted"; stopBrowserActions(); };
const assertActive = () => { if (stopped) throw new ConnectionBlock("blocked_target", stopped); };
signal.addEventListener("abort", interrupt, { once: true });
if (signal.aborted) interrupt();
const deadlineTimer = setTimeout(() => { stopped = "cell_deadline_reached"; stopBrowserActions(); }, execution.task.attemptTimeoutMs?.[execution.environment.id] ?? 30 * 60_000);
try {
assertActive();
const identity = await verifyConnectionTarget(origin, config);
Object.assign(evidence.target, identity);
evidence.checkpoints.target = true;
assertActive();
const settings = resolveConnectionSettings(execution, config);
result.model = settings.model;
evidence.model = settings.model;
if (mode === "subscription" && !config.browser.headed) throw new ConnectionBlock("awaiting_user", "subscription_requires_headed_browser");
// Resolve only the selected credential, and only after validating the target.
const secret = await resolveConnectionSecret(config, settings.credentialEnv);
if (secret) secrets.push(secret);
assertActive();
browser = await openConnectionBrowser(config, repositoryRoot, origin);
assertActive();
const page = browser.context.pages()[0] ?? await browser.context.newPage();
const api = new ConnectionApi(browser.context.request, origin);
diagnosticApi = api;
browser.context.on("response", response => {
const url = new URL(response.url());
if (url.origin !== origin || !fixtureCompanyId || !url.pathname.startsWith(`/api/companies/${fixtureCompanyId}/`) || !response.ok()) return;
const method = response.request().method();
const local = method === "POST" && /^\/api\/companies\/([a-f0-9-]+)\/ai-connections\/local\/attempts$/.test(url.pathname);
const cloud = method === "POST" && /^\/api\/companies\/([a-f0-9-]+)\/(?:setup-token-login-sessions|adapters\/[^/]+\/login-sessions)$/.test(url.pathname);
const root = `/api/companies/${fixtureCompanyId}/ai-connections`;
const receipt = (method === "POST" && [root, `${root}/local`].includes(url.pathname)) || (method === "GET" && url.pathname.startsWith(`${root}/login/`));
if (!local && !cloud && !receipt) return;
loginObservers.push((async () => {
const body = await response.json();
if ((local || cloud) && typeof body.sessionId === "string" && /^[a-zA-Z0-9_-]+$/.test(body.sessionId)) {
ownedLoginIds.add(body.sessionId);
loginSessions.set(`${url.pathname}/${body.sessionId}${cloud ? "/cancel" : ""}`, local ? "DELETE" : "POST");
}
const id = connectionCreationReceipt(url.pathname, method, fixtureCompanyId!, ownedLoginIds, body);
if (id) createdConnectionIds.add(id);
})().catch(() => {}));
});
const refreshOwnedConnections = async () => {
if (!fixtureCompanyId) return;
for (const sessionId of ownedLoginIds) {
const route = `/api/companies/${fixtureCompanyId}/ai-connections/login/${sessionId}`;
try {
const id = connectionCreationReceipt(route, "GET", fixtureCompanyId, ownedLoginIds, await api.get(route));
if (id) createdConnectionIds.add(id);
} catch { /* An unfinished or cancelled owned login has no account receipt. */ }
}
};
await page.goto(origin, { waitUntil: "domcontentloaded" });
try { await api.get("/api/companies"); }
catch (authError) {
if (!(authError instanceof ConnectionFailure) || !["api_status_401", "api_status_403"].includes(authError.message)) throw authError;
if (!config.browser.headed) throw new ConnectionBlock("awaiting_user", "board_login_required");
evidence.assisted = true;
evidence.waits.push({ kind: "board_login", startedAt: new Date().toISOString() });
await checkpoint("awaiting_board_login");
console.log(`[provider-connections] Sign in to Paperclip at ${origin} in the QA browser; this test resumes automatically.`);
const deadline = Date.now() + config.browser.loginTimeoutMs;
let ready = false;
while (Date.now() < deadline) { assertActive(); try { await api.get("/api/companies"); ready = true; break; } catch { await connectionDelay(1000); } }
if (!ready) throw new ConnectionBlock("awaiting_user", "board_login_deadline_reached");
evidence.waits.at(-1)!.finishedAt = new Date().toISOString();
}
const registry = new FixtureRegistry();
const attachedCompany = config.target.mode === "attach" ? config.target.companyId : undefined;
registry.register<Row>({ id: "connection-company", setup: async () => {
if (attachedCompany) {
const company = await api.get(`/api/companies/${attachedCompany}`);
if (!company.name.startsWith("Connection QA") || (await api.get<Row[]>(`/api/companies/${company.id}/agents`)).some(agent => agent.status !== "terminated") || (await api.get(`/api/companies/${company.id}/ai-connections`)).connections.some((connection: Row) => connection.status !== "revoked")) throw new ConnectionBlock("blocked_target", "attach_requires_empty_dedicated_qa_company");
fixtureCompanyId = company.id;
return company;
}
const company = await api.post("/api/companies", { name: `Connection QA ${nonce}`, description: "Disposable provider-connections QA fixture", budgetMonthlyCents: config.budgetCents });
fixtureCompanyId = company.id;
return company;
}, teardown: async company => {
await Promise.all(loginObservers);
await refreshOwnedConnections();
for (const [route, method] of loginSessions) {
try { await api.response(route, method); }
catch (error) { if (!(error instanceof ConnectionFailure) || error.message !== "api_status_404") throw error; }
}
await refreshOwnedConnections();
// Creation receipts identify this attempt even when concurrent campaigns
// passed the initial empty-company check at the same time.
await cleanupConnectionCompany({ api, companyId: company.id, agentName: `Connection QA ${nonce}`,
attachedCompany: Boolean(attachedCompany), retainCompany: config.retainCompany, createdConnectionIds, evidence, collectDiagnostics });
} });
registry.register<Row>({ id: "connection-environment", dependencies: ["connection-company"], setup: async values => {
const company = values.get("connection-company") as Row;
const environments = await api.get<Row[]>(`/api/companies/${company.id}/environments`);
const selectedId = config.target.mode === "attach" ? config.target.environmentId : undefined;
const environment = selectedId ? environments.find(env => env.id === selectedId) : environments.find(env => env.status === "active" && (execution.environment.id === "local" ? env.driver === "local" : env.driver === "sandbox" && env.config?.provider === "daytona"));
if (!environment || (execution.environment.id === "local" ? environment.driver !== "local" : environment.config?.provider !== "daytona")) throw new ConnectionBlock("blocked_target", "configured_execution_environment_required");
if (execution.profile.generation === "native") {
const experimental = await api.get("/api/instance/settings/experimental");
if (!experimental.enableNativeRunner) {
if (config.target.mode === "attach") throw new ConnectionBlock("blocked_target", "native_runner_disabled_on_target");
await api.patch("/api/instance/settings/experimental", { enableNativeRunner: true });
}
}
return environment;
} });
fixtures = await registry.setupAll();
const company = fixtures.values.get("connection-company") as Row;
evidence.companyId = company.id;
const environment = fixtures.values.get("connection-environment") as Row;
const observed = await runConnectionFlow({ page, api, execution, config, settings, company, environment, secret, nonce, privateDir, evidence, checkpoint, assertActive,
createdConnectionIds, refreshOwnedConnections, observeRun: run => observedRuns.set(run.id, run) });
Object.assign(result, observed);
evidence.outcome = "passed";
if (!connectionEvidencePasses(evidence)) throw new ConnectionFailure("connection_proof_incomplete");
// Only the fixture's synthetic task is captured, after authentication.
const expectedPath = `/${company.issuePrefix}/issues/${observed.issueIdentifier}`;
const url = new URL(page.url());
if (url.origin !== origin || url.pathname !== expectedPath) throw new ConnectionFailure("final_screenshot_route_mismatch");
const screenshot = await page.screenshot({ path: path.join(privateDir, "final-state.png"), fullPage: true });
result.screenshots = [{ id: "final-state", label: "Verified connection task and follow-up", file: "final-state.png", sha256: createHash("sha256").update(screenshot).digest("hex") }];
result.status = "passed";
} catch (caught) {
const error = stopped ? new ConnectionBlock("blocked_target", stopped) : caught;
if (error instanceof AggregateError) result.cleanup = "failed";
evidence.outcome = error instanceof ConnectionBlock ? error.reason : "failed";
result.failureClass = error instanceof ConnectionBlock ? "permanent_infrastructure" : classifyFailure(error);
result.error = safeConnectionFailure(error, evidence.phase);
} finally {
clearTimeout(deadlineTimer);
try { await fixtures?.teardown(); if (result.cleanup !== "failed") result.cleanup = "passed"; }
catch (cleanupError) { result.cleanup = "failed"; result.status = "failed"; result.failureClass = "cleanup_failure"; result.error = `${result.error ? `${result.error}; ` : ""}fixture_cleanup_failed`;
const causes = cleanupError instanceof AggregateError ? cleanupError.errors : [cleanupError];
console.log(`[provider-connections] Cleanup: ${causes.map(cause => safeConnectionFailure(cause, "cleanup")).join(", ")}`); }
// Also preserve evidence when setup or early cleanup failed before the
// company teardown reached its capture point. Never refetch deleted runs.
await collectDiagnostics();
try { await browser?.close(); }
catch { result.cleanup = "failed"; result.status = "failed"; result.failureClass = "cleanup_failure"; result.error = "qa_browser_cleanup_failed"; }
result.finishedAt = new Date().toISOString();
result.durationMs = Date.parse(result.finishedAt) - Date.parse(startedAt);
result.runIds = [...observedRuns.keys()];
// Usage is projected to numeric billing below; opaque provider output is never retained.
const usage = { runs: [...observedRuns.values()].map(run => ({ usage: run.usageJson })) };
result.runtimeUsage = buildRuntimeUsage({ environmentId: execution.environment.id, runs: [...observedRuns.values()], fallbackFinishedAt: result.finishedAt });
result.billing = summarizeExecutionBilling({ ...result, usage });
// UI environment probes may incur cost without heartbeat usage receipts.
result.billing.complete = false;
result.billing.llm.costStatus = result.billing.llm.runsWithTokenUsage ? "partial" : "unavailable";
result.matcherResults = gradeConnectionEvidence(evidence);
await safeWrite(path.join(privateDir, "result.json"), result, secrets);
const manifest = await packageEvidence({ privateDir, uploadDir: evidenceDir, secrets, expectPassScreenshot: false });
if (manifest.leaks.length || manifest.missing.length) { result.status = "failed"; result.failureClass = "secret_leak"; result.error = "evidence_packaging_failed"; evidence.outcome = "failed"; }
await safeWrite(path.join(evidenceDir, "result.json"), result, secrets);
await safeWrite(path.join(evidenceDir, "evidence-manifest.json"), manifest, secrets);
await safeWrite(path.join(summaryDir, "progress.json"), { executionId: execution.id, ...evidence }, secrets);
await rm(privateDir, { recursive: true, force: true });
signal.removeEventListener("abort", interrupt);
}
return result;
}
export async function runConnectionCampaign(input: { executions: readonly MatrixExecution[]; catalog: readonly MatrixExecution[]; configFile?: string; repositoryRoot: string }) {
return withConnectionCampaignCancellation(async (signal) => {
if (input.executions.some(e => e.task.flow !== "provider_connection")) throw new ConnectionFailure("Select provider-connections separately from other suites.");
const config = await loadConnectionConfig(input.configFile);
const campaignId = `connections-${new Date().toISOString().replace(/[:.]/g, "-")}-${randomBytes(3).toString("hex")}`;
const summaryDir = path.join(input.repositoryRoot, "tests/runner-e2e/results", campaignId);
await mkdir(summaryDir, { recursive: true, mode: 0o700 });
const results: RunnerE2EResult[] = [];
let target: Awaited<ReturnType<typeof startConnectionTarget>> | undefined;
try {
assertConnectionCampaignActive(signal);
target = await startConnectionTarget(config, input.executions, input.repositoryRoot, signal);
assertConnectionCampaignActive(signal);
console.log(`[provider-connections] ${input.executions.length} selected cells; target ${target.origin}; one browser/account at a time; max ${config.maxRuns} task runs and ${config.budgetCents} cents per fixture. UI probe spend may be unreported.`);
for (const execution of input.executions) {
assertConnectionCampaignActive(signal);
console.log(`[provider-connections] Starting ${execution.id}`);
const result = await connectionAttempt(execution, config, target.origin, summaryDir, input.repositoryRoot, signal);
results.push(result);
console.log(`[provider-connections] ${result.providerConnection!.outcome}: ${execution.id}${result.error ? ` (${result.error})` : ""}`);
// Every completed/blocked cell is durable before the next login begins.
await writeConnectionCampaignReport(input, campaignId, summaryDir, results);
if (signal.aborted || result.error === "test_interrupted" || result.providerConnection!.outcome === "awaiting_user") break;
}
} catch (error) {
if (!signal.aborted) throw error;
// Preserve completed receipts even if interrupted during server startup or
// report generation. Cancellation never admits another cell.
await writeConnectionCampaignReport(input, campaignId, summaryDir, results);
} finally { await target?.stop(); }
console.log(`[provider-connections] Report: ${path.join(summaryDir, "dashboard.html")}`);
if (signal.aborted || results.some(result => result.status !== "passed" || result.cleanup !== "passed")) process.exitCode = 1;
return summaryDir;
});
}
function assertConnectionCampaignActive(signal: AbortSignal) {
if (signal.aborted) throw new ConnectionBlock("blocked_target", "test_interrupted");
}
/** One cancellation state covers configuration, startup, every cell, reports,
* and owned-target teardown. Repeated signals cannot reset it. */
export async function withConnectionCampaignCancellation<T>(run: (signal: AbortSignal) => Promise<T>): Promise<T> {
const controller = new AbortController();
const interrupt = () => controller.abort();
const signals = ["SIGINT", "SIGTERM", "SIGHUP"] as const;
for (const signal of signals) process.on(signal, interrupt);
try { return await run(controller.signal); }
finally { for (const signal of signals) process.removeListener(signal, interrupt); }
}
export async function writeConnectionCampaignReport(input: { executions: readonly MatrixExecution[]; catalog: readonly MatrixExecution[] }, campaignId: string, summaryDir: string, results: RunnerE2EResult[]) {
const generatedAt = new Date().toISOString();
const expected = input.executions.map(e => e.id);
const campaign = buildRunnerCampaign({ campaignId, generatedAt, expected, results });
await safeWrite(path.join(summaryDir, "campaign.json"), campaign, []);
await safeWrite(path.join(summaryDir, "normalized-results.json"), { ...campaign, results: results.map(result => ({ ...result, evidenceValid: result.failureClass !== "secret_leak", evidenceErrors: result.failureClass === "secret_leak" ? ["evidence_packaging_failed"] : [] })) }, []);
await stageDashboardBrandAssets(summaryDir);
await safeWrite(path.join(summaryDir, "connection-coverage.json"), {
required: input.catalog.filter(e => e.suite.id === "provider-connections").length,
selected: expected.length, attempted: results.length, passed: results.filter(r => r.status === "passed" && r.cleanup === "passed").length,
blocked: results.filter(r => ["awaiting_user", "missing_credential", "blocked_target"].includes(r.providerConnection!.outcome)).length,
targets: [...new Set(results.map(r => r.providerConnection!.target.origin))],
gaps: [...connectionScopeGaps, ...connectionHarnesses.flatMap(h => connectionModes.filter(mode => !supportsConnection(h, mode)).map(mode => ({ adapter: h.adapter, mode, status: "unsupported" })))],
}, []);
const dashboard = renderRunnerE2EDashboard({ title: `Live provider connections · ${campaignId}`, generatedAt, expected, catalog: input.catalog, campaign,
entries: results.map(result => ({ result, valid: result.failureClass !== "secret_leak", errors: result.failureClass === "secret_leak" ? ["evidence_packaging_failed"] : [], evidenceBaseHref: [result.suiteId, result.profileId, result.environmentId, result.caseId, "attempt-1"].join("/") })) });
await writeFile(path.join(summaryDir, "dashboard.html"), dashboard, { mode: 0o600 });
}
tests/runner-e2e/connection-routing-evidence.ts
+30 -2
View File
@@ -43,6 +43,7 @@ export function gradeProviderOutcome(input: {
response: string;
marker: string;
sameConnections: boolean;
accessDecision?: { id: string; connectionId: string };
}) {
const decision = input.rows.find((row) => row.id === input.decisionId);
const selected = decision?.result?.answers?.find(
@@ -58,8 +59,12 @@ export function gradeProviderOutcome(input: {
},
{
id: "provider-no-extra-setup",
passed: input.rows.length === 1 && input.sameConnections,
detail: "No replacement question or unnecessary connection was created.",
passed: input.sameConnections && (input.accessDecision
? input.rows.length === 2 && input.rows.some(row => row.id === input.accessDecision!.id
&& row.kind === "connection_intent" && row.status === "accepted"
&& row.result?.outcome === "connected" && row.result?.connectionId === input.accessDecision!.connectionId)
: input.rows.length === 1),
detail: "Only the selected provider and, when needed, its separate approved access card were used; no connection was replaced.",
},
{
id: "provider-use-matches-choice",
@@ -72,3 +77,26 @@ export function gradeProviderOutcome(input: {
},
];
}
/** The provider preference is not an app/tool grant. Require the separate, scoped access card. */
export function requireProviderAccessCard(input: {
rows: Array<{ id: string; kind: string; status: string; payload?: any }>;
decisionId: string; connectionId: string; agentId: string; catalogEntryIds: string[]; calls: number;
}) {
const cards = input.rows.filter(row => row.id !== input.decisionId);
const card = cards[0];
const payload = card?.payload;
const tools = payload?.accessRequest?.tools;
if (input.calls !== 0 || input.rows.length !== 2 || cards.length !== 1 || card?.kind !== "connection_intent"
|| card.status !== "pending" || payload?.serviceSlug !== "arcade"
|| payload.requestingAgentId !== input.agentId
|| payload.upstreamService?.selectionInteractionId !== input.decisionId
|| payload.upstreamService?.slug !== "hubspot"
|| payload.accessRequest?.connectionId !== input.connectionId
|| !Array.isArray(tools) || tools.length !== 1 || tools[0].toolName !== "Hubspot_ListContacts"
|| tools[0].permission !== "allowed" || input.catalogEntryIds.length !== 1
|| tools[0].catalogEntryId !== input.catalogEntryIds[0]) {
throw new Error("Expected one scoped Arcade access card after the saved provider choice, before any provider call");
}
return card;
}
tests/runner-e2e/connection-support.test.ts
+282
View File
@@ -0,0 +1,282 @@
import { describe, expect, it, vi } from "vitest";
import { mkdtemp, writeFile, rm } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { AGENT_ADAPTER_TYPES } from "../../packages/shared/src/constants.js";
import { runnerMatrix } from "./catalog.js";
import { parseRunnerSelectors, selectRunnerExecutions, buildMatrixJobs } from "./selectors.js";
import { connectionHarnesses, connectionScopeGaps, connectionCell, supportsConnection } from "./connection-cases.js";
import { parseConnectionConfig, targetOrigin, selectedSecret, resolveConnectionSecret, resolveConnectionSettings } from "./connection-config.js";
import { connectionCheckpoints, connectionEvidencePasses, connectionRunDiagnostic, createConnectionProof, verifyConnectionArtifact, verifyConnectionRun, completedConnectionArtifactRun, connectionProbeChecks, type ConnectionEvidence } from "./connection-evidence.js";
import { validateRetainedRunnerResult } from "./result-validation.js";
import { verifyConnectionTarget } from "./connection-target.js";
import { cleanupConnectionCompany, connectionCreationReceipt, runConnectionCampaign, safeConnectionFailure, withConnectionCampaignCancellation } from "./connection-launch.js";
import type { ConnectionApi } from "./connection-target.js";
import * as connectionConfig from "./connection-config.js";
import * as connectionTarget from "./connection-target.js";
import * as reportAssets from "./report-assets.js";
import { buildRunnerCampaign } from "./history.js";
import { renderCaseOutcome } from "./case-outcome.js";
import type { RunnerE2EResult } from "./types.js";
const cells = runnerMatrix.filter(e => e.suite.id === "provider-connections");
const cell = (method: string) => cells.find(e => e.id === `provider-connections.connection-codex-native.local.agent-${method}`)!;
describe("connection fixture cleanup evidence", () => {
it.each([true, false])("captures stopped runs and logs before destructive cleanup (attached=%s)", async (attachedCompany) => {
const events: string[] = [];
let status = "running", deleted = false, revoked = false;
const evidence: Partial<ConnectionEvidence> = {};
const api = {
get: vi.fn(async (route: string) => {
if (route.endsWith("/agents")) return [{ id: "qa-agent", name: "Fixture" }, { id: "foreign", name: "Other" }];
if (route.includes("heartbeat-runs?")) return [{ id: "run", agentId: "qa-agent", status }];
if (route.endsWith("/ai-connections")) return { connections: [{ id: "account", status: revoked ? "revoked" : "active" }, { id: "foreign-account", status: "active" }] };
if (deleted) throw new Error("Run was deleted");
if (route.includes("/log?")) { events.push("log"); return { content: JSON.stringify({ chunk: JSON.stringify({ type: "acpx.error", childStderrTail: "HTTP 503 model overloaded" }) + "\n" }) }; }
events.push("run"); return { id: "run", status };
}),
patch: vi.fn(async () => { events.push("pause"); }),
post: vi.fn(async (route: string) => { if (route.endsWith("/cancel")) { events.push("cancel"); status = "cancelled"; } else events.push("archive"); }),
delete: vi.fn(async (route: string) => { if (route.endsWith("/foreign-account")) throw new Error("Revoked a concurrent campaign"); if (route.includes("tool-connections")) { events.push("revoke"); revoked = true; } else { events.push("delete-agent"); deleted = true; } }),
};
await cleanupConnectionCompany({ api: api as unknown as ConnectionApi, companyId: "company", agentName: "Fixture", attachedCompany,
retainCompany: false, createdConnectionIds: new Set(["account"]), evidence, collectDiagnostics: async () => {
const run = await api.get("/api/heartbeat-runs/run");
const log = await api.get("/api/heartbeat-runs/run/log?limitBytes=2000000");
evidence.runDiagnostics = [connectionRunDiagnostic(run, log)];
} });
expect(events).toEqual(["pause", "cancel", "run", "log", "revoke", attachedCompany ? "delete-agent" : "archive"]);
expect(evidence.runDiagnostics).toEqual([{ runId: "run", status: "cancelled", signals: ["provider_overloaded"], logAvailable: true }]);
expect(api.delete).not.toHaveBeenCalledWith("/api/agents/foreign");
expect(api.delete).not.toHaveBeenCalledWith("/api/tool-connections/foreign-account");
});
it("preserves diagnostics and avoids deletion when cancellation fails", async () => {
const collectDiagnostics = vi.fn(async () => {});
const api = {
get: vi.fn(async (route: string) => route.endsWith("/agents") ? [{ id: "qa-agent", name: "Fixture" }] : [{ id: "run", agentId: "qa-agent", status: "running" }]),
patch: vi.fn(async () => {}), post: vi.fn(async () => { throw new Error("cancel failed"); }), delete: vi.fn(),
};
await expect(cleanupConnectionCompany({ api: api as unknown as ConnectionApi, companyId: "company", agentName: "Fixture", attachedCompany: true,
retainCompany: false, createdConnectionIds: new Set(), evidence: {}, collectDiagnostics })).rejects.toThrow("cancel failed");
expect(collectDiagnostics).toHaveBeenCalledOnce();
expect(api.delete).not.toHaveBeenCalled();
});
it("accepts only creation receipts from this browser and its owned login sessions", () => {
const body = { connectionId: "00000000-0000-4000-8000-000000000021" };
const root = "/api/companies/company/ai-connections";
const logins = new Set(["owned-session"]);
expect(connectionCreationReceipt(root, "POST", "company", logins, body)).toBe(body.connectionId);
expect(connectionCreationReceipt(`${root}/local`, "POST", "company", logins, body)).toBe(body.connectionId);
expect(connectionCreationReceipt(`${root}/login/owned-session`, "GET", "company", logins, body)).toBe(body.connectionId);
expect(connectionCreationReceipt(`${root}/login/foreign-session`, "GET", "company", logins, body)).toBeUndefined();
expect(connectionCreationReceipt(root, "GET", "company", logins, body)).toBeUndefined();
expect(connectionCreationReceipt(root, "POST", "different-company", logins, body)).toBeUndefined();
expect(connectionCreationReceipt(root, "POST", "company", logins, { connectionId: "not-a-receipt" })).toBeUndefined();
});
});
describe("campaign interruption", () => {
it.each(["SIGINT", "SIGTERM", "SIGHUP"] as const)("keeps %s cancellation active until teardown finishes", async (signal) => {
const before = process.listenerCount(signal);
await withConnectionCampaignCancellation(async (state) => {
expect(process.listenerCount(signal)).toBe(before + 1);
process.emit(signal);
expect(state.aborted).toBe(true);
await Promise.resolve();
process.emit(signal);
expect(state.aborted).toBe(true);
});
expect(process.listenerCount(signal)).toBe(before);
});
it.each(["SIGINT", "SIGTERM", "SIGHUP"] as const)("stops owned targets and admits no next cell after %s during startup or reporting", async (signal) => {
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
const exitCode = process.exitCode;
for (const phase of ["startup", "report"] as const) {
const stop = vi.fn(async () => {});
const before = process.listenerCount(signal);
const config = parseConnectionConfig({});
const load = vi.spyOn(connectionConfig, "loadConnectionConfig").mockResolvedValue(config);
const credential = vi.spyOn(connectionConfig, "resolveConnectionSecret").mockRejectedValue(new connectionConfig.ConnectionBlock("missing_credential", "synthetic_missing_credential"));
const target = vi.spyOn(connectionTarget, "startConnectionTarget").mockImplementation(async (_config, _executions, _root, state) => {
expect(state?.aborted).toBe(false);
if (phase === "startup") process.emit(signal);
return { origin: "http://127.0.0.1:1", stop };
});
const verify = vi.spyOn(connectionTarget, "verifyConnectionTarget").mockResolvedValue({ commit: "a".repeat(40), deploymentMode: "local_trusted" });
const report = vi.spyOn(reportAssets, "stageDashboardBrandAssets").mockImplementation(async () => { if (phase === "report") process.emit(signal); });
let summary: string | undefined;
try {
summary = await runConnectionCampaign({ executions: [cell("api-key"), cell("subscription")], catalog: cells, repositoryRoot });
expect(credential).toHaveBeenCalledTimes(phase === "startup" ? 0 : 1);
expect(verify).toHaveBeenCalledTimes(phase === "startup" ? 0 : 1);
expect(stop).toHaveBeenCalledTimes(1);
expect(process.exitCode).toBe(1);
expect(process.listenerCount(signal)).toBe(before);
} finally {
load.mockRestore(); credential.mockRestore(); target.mockRestore(); verify.mockRestore(); report.mockRestore();
process.exitCode = exitCode;
if (summary) await rm(summary, { recursive: true, force: true });
}
}
});
});
describe("live connection coverage", () => {
it("inventories every built-in adapter and keeps remote exclusions explicit", () => {
const inventory = new Set<string>(["paperclip_runner", ...connectionHarnesses.map(h => h.adapter), ...connectionScopeGaps.map(h => h.adapter)]);
expect(AGENT_ADAPTER_TYPES.filter(type => !inventory.has(type))).toEqual([]);
expect(connectionHarnesses.some(h => h.adapter === "hermes_local")).toBe(true);
expect(connectionScopeGaps.find(h => h.adapter === "hermes_gateway")?.status).toBe("excluded");
});
it("includes both UI entry points and actual runner variants without invented auth capabilities", () => {
expect(cells.length).toBe(116);
expect(new Set(cells.map(e => connectionCell(e).entry))).toEqual(new Set(["agent", "apps"]));
expect(cells.filter(e => connectionCell(e).harness.id === "grok").every(e => ["subscription", "api-key"].includes(connectionCell(e).mode))).toBe(true);
expect(supportsConnection(connectionHarnesses.find(h => h.id === "gemini")!, "subscription")).toBe(false);
});
it("never runs on --all or advertises a preinstalled subscription token as a credential", () => {
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(e => e.suite.id === "provider-connections")).toBe(false);
expect(cell("subscription").requiredCredentials).toEqual([]);
expect(() => buildMatrixJobs(cells)).toThrow(/not admitted/);
expect(parseRunnerSelectors(["--suite", "provider-connections", "--connection-config", "/private/qa.json"]).connectionConfig).toBe("/private/qa.json");
});
});
describe("target and selected credential boundaries", () => {
it("accepts local/staging origins and rejects credential-bearing or redirectable target strings", () => {
expect(targetOrigin("http://127.0.0.1:3109/")).toBe("http://127.0.0.1:3109");
expect(targetOrigin("https://staging.example.test")).toBe("https://staging.example.test");
for (const url of ["http://staging.example.test", "https://user:password@example.test", "https://example.test/path", "https://example.test/?token=secret", "https://example.test/#secret", "file:///tmp/qa"]) expect(() => targetOrigin(url)).toThrow();
});
it("requires an exact staging revision and rejects unknown configuration instead of ignoring typos", () => {
expect(() => parseConnectionConfig({ target: { mode: "attach", baseURL: "https://example.test" } })).toThrow();
expect(() => parseConnectionConfig({ target: { mode: "managed-local", baseURL: "https://example.test" } })).toThrow();
expect(() => parseConnectionConfig({ broswer: {} })).toThrow();
expect(() => parseConnectionConfig({ browser: { loginTimeoutMs: Infinity } })).toThrow();
expect(parseConnectionConfig({}).browser.headed).toBe(true);
});
it("pins the target revision before permitting credential selection", async () => {
const config = parseConnectionConfig({ target: { mode: "attach", baseURL: "https://staging.example.test", expectedCommit: "a".repeat(40), deploymentMode: "authenticated" } });
const fetch = vi.spyOn(globalThis, "fetch").mockResolvedValue(new Response(JSON.stringify({ commit: "b".repeat(40), deploymentMode: "authenticated" })));
try {
await expect(verifyConnectionTarget("https://staging.example.test", config)).rejects.toThrow("target_identity_mismatch");
expect(fetch.mock.calls[0]?.[1]).toMatchObject({ redirect: "error" });
fetch.mockResolvedValue(new Response(JSON.stringify({ commit: "a".repeat(40), deploymentMode: "authenticated" })));
await expect(verifyConnectionTarget("https://staging.example.test", config)).resolves.toMatchObject({ commit: "a".repeat(40) });
} finally { fetch.mockRestore(); }
});
it("reads literal selected keys without sourcing shell commands or returning unrelated secrets", async () => {
expect(selectedSecret("UNRELATED=do-not-read\nexport OPENAI_API_KEY='chosen-value'\n", "OPENAI_API_KEY")).toBe("chosen-value");
expect(selectedSecret("OPENAI_API_KEY=one\nOPENAI_API_KEY=two", "OPENAI_API_KEY")).toBe("two");
expect(() => selectedSecret("OPENAI_API_KEY=$(touch /tmp/should-not-exist)", "OPENAI_API_KEY")).toThrow();
const root = await mkdtemp(path.join(os.tmpdir(), "connection-secret-test-"));
try {
const file = path.join(root, "keys");
await writeFile(file, "OPENAI_API_KEY=selected-only\nUNRELATED=private\n", { mode: 0o600 });
const config = parseConnectionConfig({ secretFile: file });
expect(await resolveConnectionSecret(config, "OPENAI_API_KEY", {})).toBe("selected-only");
expect(await resolveConnectionSecret(config, undefined, {})).toBeUndefined();
await expect(resolveConnectionSecret(config, "XAI_API_KEY", {})).rejects.toThrow("missing_XAI_API_KEY");
await writeFile(path.join(root, "public"), "OPENAI_API_KEY=private", { mode: 0o644 });
await expect(resolveConnectionSecret(parseConnectionConfig({ secretFile: path.join(root, "public") }), "OPENAI_API_KEY", {})).rejects.toThrow(/owner_only/);
} finally { await rm(root, { recursive: true, force: true }); }
});
it("never resolves an API key for a subscription or defaults a gateway to the direct provider's credential", () => {
expect(resolveConnectionSettings(cell("subscription"), parseConnectionConfig({})).credentialEnv).toBeUndefined();
expect(resolveConnectionSettings(cell("api-key"), parseConnectionConfig({})).credentialEnv).toBe("OPENAI_API_KEY");
expect(() => resolveConnectionSettings(cell("responses"), parseConnectionConfig({ routes: { responses: { model: "qa-alias", baseURL: "https://gateway.example.test/v1" } } }))).toThrow(/credential_reference/);
const config = parseConnectionConfig({ routes: { responses: { model: "qa-alias", baseURL: "https://gateway.example.test/v1", credentialEnv: "GATEWAY_QA_KEY" } } });
expect(resolveConnectionSettings(cell("responses"), config).credentialEnv).toBe("GATEWAY_QA_KEY");
});
});
function passingEvidence(): ConnectionEvidence {
return { version: 1, outcome: "passed", phase: "finished", assisted: true, authFreshness: "signed-in", target: { mode: "attach", origin: "https://staging.example.test", commit: "a".repeat(40), deploymentMode: "authenticated" }, entry: "agent", method: "subscription", checkpoints: Object.fromEntries(connectionCheckpoints.map(key => [key, true])), waits: [] };
}
describe("independent qualification", () => {
it("retains closed failure signals across split log chunks without messages or reasoning", () => {
const sentinel = "SENTINEL_CREDENTIAL_DO_NOT_RETAIN";
const rows = [
{ type: "acpx.text_delta", channel: "thought", text: 'HTTP 429 '+sentinel },
{ type: "acpx.error", childStderrTail: 'ApiError: {"error":{"code":503,"message":"model overloaded '+sentinel+'"}}' },
{ type: "acpx.tool_call", status: "failed", text: "Error checking existing file: Resource not found: "+sentinel },
].map(row => JSON.stringify(row)).join("\n")+"\n";
const split = Math.floor(rows.length / 2);
const log = { content: [rows.slice(0, split), rows.slice(split)].map(chunk => JSON.stringify({ chunk })).join("\n") };
const diagnostic = connectionRunDiagnostic({ id: "run-1", status: "failed" }, log);
expect(diagnostic).toEqual({ runId: "run-1", status: "failed", signals: ["provider_overloaded", "missing_file_conversion"], logAvailable: true });
expect(JSON.stringify(diagnostic)).not.toContain(sentinel);
expect(connectionRunDiagnostic({ id: "run-2", status: "running" })).toEqual({ runId: "run-2", status: "unknown", signals: [], logAvailable: false });
});
it("distinguishes terminal quota and unsupported configuration from generic failures", () => {
expect(connectionRunDiagnostic({ id: "quota", status: "failed", error: 'HTTP 429 RESOURCE_EXHAUSTED' }).signals).toEqual(["provider_quota"]);
expect(connectionRunDiagnostic({ id: "config", status: "failed", error: 'session/set_config_option: Method not found (-32601)' }).signals).toEqual(["unsupported_session_configuration"]);
expect(connectionRunDiagnostic({ id: "generic", status: "failed", error: 'unknown provider error' }).signals).toEqual([]);
expect(connectionRunDiagnostic({ id: "unattributed-503", status: "failed", error: 'HTTP 503' }).signals).toEqual([]);
});
it("waits for disposition repair and requires the successful artifact-producing run", () => {
const expected = { agentId: "a", connectionId: "c", method: "api_key", runtimeMode: "legacy", environmentId: "env" };
const first = { id: "first", agentId: "a", status: "succeeded", runtimeMode: "legacy", contextSnapshot: { paperclipEnvironment: { id: "env" }, aiConnection: { connectionId: "c", method: "api_key" } } };
const repair = { ...first, id: "repair" };
const file = { originalFilename: "proof.json", createdByAgentId: "a", originatingRunId: "repair" };
expect(completedConnectionArtifactRun([first, { ...repair, status: "running" }], [file], "proof.json", expected)).toBeUndefined();
expect(completedConnectionArtifactRun([first, repair], [file], "proof.json", expected)).toBe(repair);
expect(completedConnectionArtifactRun([first, { ...repair, status: "cancelled" }], [file], "proof.json", expected)).toBeUndefined();
expect(completedConnectionArtifactRun([first, repair], [{ ...file, createdByAgentId: "someone-else" }], "proof.json", expected)).toBeUndefined();
expect(completedConnectionArtifactRun([first, repair], [file], "followup.json", expected)).toBeUndefined();
expect(completedConnectionArtifactRun([first, repair], [file], "proof.json", { ...expected, connectionId: "another" })).toBeUndefined();
});
it("retains probe codes without provider text, URLs, or credential-bearing fields", () => {
const sentinel = "SENTINEL_CREDENTIAL_DO_NOT_RETAIN";
const checks = connectionProbeChecks({ checks: [
{ code: "hermes_cli_not_found", level: "error", message: sentinel, detail: `https://provider.test/?code=${sentinel}` },
{ code: sentinel, level: "error" }, { code: "safe_code", level: sentinel }, null,
] });
expect(checks).toEqual([{ code: "hermes_cli_not_found", level: "error" }]);
expect(JSON.stringify(checks)).not.toContain(sentinel);
});
it("rejects plausible prose, wrong bytes and wrong run attribution", () => {
const proof = createConnectionProof();
expect(verifyConnectionArtifact(Buffer.from(JSON.stringify(proof.expected)), proof.expected)).toBe(true);
expect(verifyConnectionArtifact(Buffer.from("Provider routing works"), proof.expected)).toBe(false);
expect(verifyConnectionArtifact(Buffer.from(JSON.stringify({ ...proof.expected, total: proof.expected.total + 1 })), proof.expected)).toBe(false);
const expected = { agentId: "a", connectionId: "c", method: "subscription", runtimeMode: "native", environmentId: "env", model: "m" };
const run = { agentId: "a", status: "succeeded", runtimeMode: "native", contextSnapshot: { paperclipEnvironment: { id: "env" }, aiConnection: { connectionId: "c", method: "subscription" } } };
expect(verifyConnectionRun(run, expected)).toBe(true);
expect(verifyConnectionRun(run, { ...expected, connectionId: "ambient" })).toBe(false);
expect(verifyConnectionRun(run, { ...expected, method: "api_key" })).toBe(false);
expect(verifyConnectionRun(run, { ...expected, environmentId: "another-env" })).toBe(false);
expect(verifyConnectionRun({ ...run, contextSnapshot: { aiConnection: run.contextSnapshot.aiConnection } }, expected)).toBe(false);
});
it("requires every checkpoint and never passes absent evidence or a human deadline", () => {
expect(connectionEvidencePasses(passingEvidence())).toBe(true);
for (const key of connectionCheckpoints) {
const evidence = passingEvidence(); delete evidence.checkpoints[key];
expect(connectionEvidencePasses(evidence)).toBe(false);
}
expect(connectionEvidencePasses({ ...passingEvidence(), outcome: "awaiting_user" })).toBe(false);
expect(connectionEvidencePasses({ ...passingEvidence(), target: { ...passingEvidence().target, commit: null } })).toBe(false);
});
it("withholds raw Playwright call logs including filled secrets, codes, and callback URLs", () => {
const sentinel = "SENTINEL_CREDENTIAL_DO_NOT_RETAIN";
for (const message of [`locator.fill: value ${sentinel}`, `https://provider.test/callback?code=${sentinel}`, `password=${sentinel}`, "lowercasesecret"]) {
const safe = safeConnectionFailure(new Error(message), "connection_creation");
expect(safe).toBe("connection_creation_failed"); expect(safe).not.toContain(sentinel);
}
});
it("keeps blocked authentication incomplete in the existing campaign/report", () => {
const execution = cell("subscription");
const result: RunnerE2EResult = { schema: "paperclip.runner-e2e.result/v2", executionId: execution.id, suiteId: execution.suite.id, suiteDefinitionHash: execution.suiteDefinitionHash,
attempt: 1, status: "failed", failureClass: "permanent_infrastructure", error: "provider_login_deadline_reached", profileId: execution.profile.id, environmentId: "local", caseId: execution.task.id, provider: "codex", model: "model", runtimeMode: "native",
startedAt: "2026-10-03T00:00:00Z", finishedAt: "2026-10-03T00:01:00Z", durationMs: 60_000, cleanup: "passed", providerConnection: { ...passingEvidence(), outcome: "awaiting_user", checkpoints: { target: true } } };
const campaign = buildRunnerCampaign({ campaignId: "test", generatedAt: result.finishedAt, expected: [execution.id], results: [result] });
expect(() => validateRetainedRunnerResult({ ...result, status: "passed" })).toThrow(/providerConnection/);
expect(() => validateRetainedRunnerResult({ ...result, status: "passed", providerConnection: undefined })).toThrow(/providerConnection/);
expect(campaign.passed).toBe(0); expect(campaign.incomplete).toBe(1); expect(campaign.complete).toBe(false);
expect(renderCaseOutcome(result, false, [])).toContain("Blocked: awaiting user");
const cleanupFailed = buildRunnerCampaign({ campaignId: "failed-cleanup", generatedAt: result.finishedAt, expected: [execution.id], results: [{ ...result, cleanup: "failed", failureClass: "cleanup_failure" }] });
expect(cleanupFailed.failed).toBe(1); expect(cleanupFailed.incomplete).toBe(0);
});
});
tests/runner-e2e/connection-target.ts
+102
View File
@@ -0,0 +1,102 @@
import { spawn, type ChildProcess } from "node:child_process";
import { randomBytes } from "node:crypto";
import { mkdir, mkdtemp, rm } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import type { APIRequestContext } from "@playwright/test";
import { reserveRunnerE2EServerPort } from "./ports.js";
import { resolvePaperclipRunnerBinaryForHarness } from "./harness-env.js";
import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js";
import { ConnectionBlock, ConnectionFailure, targetOrigin, type ConnectionConfig } from "./connection-config.js";
import type { MatrixExecution } from "./types.js";
export const connectionDelay = (ms: number) => new Promise<void>(resolve => setTimeout(resolve, ms));
/** No raw response bodies, callback URLs, headers, or Playwright diagnostics escape. */
export class ConnectionApi {
constructor(readonly request: APIRequestContext, readonly origin: string) {}
async response(route: string, method = "GET", data?: unknown) {
if (!route.startsWith("/api/") || new URL(route, this.origin).origin !== this.origin) throw new ConnectionFailure("invalid_api_route");
try {
const response = await this.request.fetch(`${this.origin}${route}`, { method, ...(data === undefined ? {} : { data }), maxRedirects: 0, timeout: 30_000 });
if (!response.ok()) throw new ConnectionFailure(`api_status_${response.status()}`);
return response;
} catch (error) {
if (error instanceof Error && /^api_status_\d+$/.test(error.message)) throw error;
throw new ConnectionFailure("api_request_failed");
}
}
async get<T = any>(route: string): Promise<T> { return (await this.response(route)).json(); }
async post<T = any>(route: string, data?: unknown): Promise<T> { return (await this.response(route, "POST", data)).json(); }
async patch<T = any>(route: string, data: unknown): Promise<T> { return (await this.response(route, "PATCH", data)).json(); }
async delete(route: string) { await this.response(route, "DELETE"); }
async bytes(route: string) { return (await this.response(route)).body(); }
}
async function stopOwnedServer(child: ChildProcess) {
if (!child.pid) return;
const exited = () => child.exitCode !== null || child.signalCode !== null;
const kill = (signal: NodeJS.Signals) => {
try { process.kill(process.platform === "win32" ? child.pid! : -child.pid!, signal); }
catch (error) { if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; }
};
kill("SIGTERM");
const deadline = Date.now() + 45_000;
while (!exited() && Date.now() < deadline) await connectionDelay(100);
// The group is exclusively the server this function spawned, never an attached target.
kill("SIGKILL");
}
export async function startConnectionTarget(config: ConnectionConfig, executions: readonly MatrixExecution[], repositoryRoot: string, signal?: AbortSignal) {
const assertActive = () => { if (signal?.aborted) throw new ConnectionBlock("blocked_target", "test_interrupted"); };
assertActive();
if (config.target.mode === "attach") return { origin: targetOrigin(config.target.baseURL), stop: async () => {} };
const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-connection-e2e-"));
let child: ChildProcess | undefined;
const stop = async () => { if (child) await stopOwnedServer(child); await rm(root, { recursive: true, force: true }); };
try {
assertActive();
const home = path.join(root, "paperclip-home");
const instance = `connection-${randomBytes(6).toString("hex")}`;
const port = await reserveRunnerE2EServerPort();
await mkdir(home, { mode: 0o700 });
// Only bootstrap essentials are inherited. Selected provider keys enter via UI.
const env: NodeJS.ProcessEnv = Object.fromEntries(["PATH", "HOME", "USER", "LOGNAME", "SHELL", "TMPDIR", "LANG", "LC_ALL", "NODE_EXTRA_CA_CERTS"].flatMap(key => process.env[key] ? [[key, process.env[key]!]] : []));
Object.assign(env, {
PAPERCLIP_HOME: home, PAPERCLIP_INSTANCE_ID: instance, PAPERCLIP_CONFIG: path.join(home, "instances", instance, "config.json"),
PAPERCLIP_RUNNER_E2E_TEMP_ROOT: root, PAPERCLIP_RUNNER_E2E_PORT: String(port), PAPERCLIP_RUNNER_E2E_SERVER_LOG: path.join(root, "server.log"),
PAPERCLIP_RUNNER_BINARY: resolvePaperclipRunnerBinaryForHarness(executions, repositoryRoot),
PAPERCLIP_VITE_CACHE_DIR: path.join(root, "vite-cache"), PAPERCLIP_ANNOUNCEMENTS_ENABLED: "false",
PAPERCLIP_AGENT_JWT_SECRET: randomBytes(48).toString("hex"), PAPERCLIP_DECISION_SIGNING_SECRET: randomBytes(48).toString("hex"),
PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: randomBytes(48).toString("hex"), BETTER_AUTH_SECRET: randomBytes(48).toString("hex"),
});
assertActive();
child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(repositoryRoot, path.join(repositoryRoot, "tests/runner-e2e/server.ts")), { cwd: repositoryRoot, env, detached: process.platform !== "win32", stdio: "ignore" });
let failed = false;
child.once("error", () => { failed = true; });
const origin = `http://127.0.0.1:${port}`;
const deadline = Date.now() + 180_000;
while (Date.now() < deadline) {
assertActive();
if (failed || child.exitCode !== null) throw new ConnectionBlock("blocked_target", "managed_server_start_failed");
try {
const response = await fetch(`${origin}/api/health`, { redirect: "error", signal: signal ? AbortSignal.any([signal, AbortSignal.timeout(1500)]) : AbortSignal.timeout(1500) });
if (response.ok) { assertActive(); return { origin, stop }; }
} catch { assertActive(); /* Bounded startup. */ }
await connectionDelay(500);
}
throw new ConnectionBlock("blocked_target", "managed_server_start_timeout");
} catch (error) { await stop(); throw error; }
}
export async function verifyConnectionTarget(origin: string, config: ConnectionConfig) {
let health: any;
try {
const response = await fetch(`${targetOrigin(origin)}/api/health`, { redirect: "error", signal: AbortSignal.timeout(15_000) });
if (!response.ok) throw new ConnectionFailure();
health = await response.json();
} catch { throw new ConnectionBlock("blocked_target", "target_health_unavailable_or_redirected"); }
if (typeof health.commit !== "string" || !/^[a-f0-9]{40}$/.test(health.commit)) throw new ConnectionBlock("blocked_target", "target_revision_unavailable");
if (config.target.mode === "attach" && (health.commit !== config.target.expectedCommit || health.deploymentMode !== config.target.deploymentMode)) throw new ConnectionBlock("blocked_target", "target_identity_mismatch");
return { commit: health.commit as string, deploymentMode: String(health.deploymentMode) };
}
tests/runner-e2e/context-integrity-cases.ts
+17 -7
View File
@@ -5,10 +5,14 @@ export const CONTEXT_INTEGRITY_CASES = [
"assigned-skill-explicit-invocation",
] as const;
export type ContextIntegrityCase = (typeof CONTEXT_INTEGRITY_CASES)[number];
export const PAPERCLIP_DOCUMENT_CASE = "assigned-skill-paperclip-document" as const;
export type ContextIntegrityCase = (typeof CONTEXT_INTEGRITY_CASES)[number] | typeof PAPERCLIP_DOCUMENT_CASE;
export function isAssignedSkillContext(id: string): boolean {
return id === "assigned-skill-explicit-invocation" || id === PAPERCLIP_DOCUMENT_CASE;
}
export function contextIntegrityScenario(id: string, nonce: string) {
if (!CONTEXT_INTEGRITY_CASES.includes(id as ContextIntegrityCase))
if (!CONTEXT_INTEGRITY_CASES.includes(id as (typeof CONTEXT_INTEGRITY_CASES)[number]) && id !== PAPERCLIP_DOCUMENT_CASE)
throw new Error(`Unknown context-integrity case: ${id}`);
const marker = `CONTEXT_OK_${nonce.replace(/[^a-z0-9]/gi, "")}`;
const repeated = "Append an entry for every request, even identical wording.";
@@ -23,15 +27,17 @@ export function contextIntegrityScenario(id: string, nonce: string) {
prompt:
id === "ordered-comment-continuation"
? `Maintain a packing list report document. Start with the initial scope: passport and charger. Save the initial report and leave this task waiting for follow-up comments. When follow-up comments arrive, copy each user's exact wording verbatim into one ordered ledger, including identical wording repeated more than once. After the follow-up batch, save one report containing both initial items and every verbatim request in arrival order. Add a separate section headed "## Final scope" outside the quoted ledger. Put only the final requested scope name as plain text in that section. Then finish. Do not invent extra comments, child tasks, or deliverables.`
: "Use the assigned Context integrity output skill for this task, then follow its instructions and finish the task. Do not create child tasks or unrelated deliverables.",
: id === PAPERCLIP_DOCUMENT_CASE
? "Use the assigned Context integrity output skill. Save its requested document as a Paperclip document on this task, check the saved revision and content, and post a completion comment with a clickable Markdown link to the saved document in the Paperclip task UI, including the company prefix. A workspace file alone does not deliver this requested Paperclip document. Finish after delivery. Do not create child tasks or unrelated deliverables."
: "Use the assigned Context integrity output skill for this task, then follow its instructions and finish the task. Do not create child tasks or unrelated deliverables.",
comments: [repeated, repeated, changed] as const,
};
}
export const contextIntegrityTasks: readonly RunnerTaskFixture[] =
CONTEXT_INTEGRITY_CASES.map((id) => ({
function contextIntegrityTask(id: ContextIntegrityCase): RunnerTaskFixture {
return {
id,
label: id === "ordered-comment-continuation" ? "Ordered comment continuation" : "Assigned skill invocation",
label: id === "ordered-comment-continuation" ? "Ordered comment continuation" : id === PAPERCLIP_DOCUMENT_CASE ? "Explicit Paperclip document delivery" : "Assigned skill invocation",
groups: ["context-integrity"],
workMode: "standard",
flow: "context_integrity",
@@ -42,4 +48,8 @@ export const contextIntegrityTasks: readonly RunnerTaskFixture[] =
buildPrompt: (nonce) => contextIntegrityScenario(id, nonce).prompt,
buildVisibleMarker: (nonce) => contextIntegrityScenario(id, nonce).marker,
buildMatchers: () => [],
}));
};
}
export const contextIntegrityTasks: readonly RunnerTaskFixture[] = CONTEXT_INTEGRITY_CASES.map(contextIntegrityTask);
export const paperclipDocumentTask = contextIntegrityTask(PAPERCLIP_DOCUMENT_CASE);
tests/runner-e2e/context-integrity-flow.ts
+26 -10
View File
@@ -1,6 +1,7 @@
import { observeCheckoutActivity } from "./checkout-activity.js";
import type { Page } from "@playwright/test";
import { randomUUID } from "node:crypto";
import { contextIntegrityScenario } from "./context-integrity-cases.js";
import { contextIntegrityScenario, isAssignedSkillContext } from "./context-integrity-cases.js";
import { gradeContextIntegrity, type ContextIntegrityCheckpoint } from "./context-integrity-scoring.js";
import { pollUntil, type RunnerApi } from "./api.js";
import { createTaskThroughUi } from "./user-actions.js";
@@ -102,7 +103,7 @@ export async function runContextIntegrityFlow(input: {
let assignedVersionId: string | null = null;
const checkpoints: ContextIntegrityCheckpoint[] = [];
if (scenario.id === "assigned-skill-explicit-invocation") {
if (isAssignedSkillContext(scenario.id)) {
await api.patch("/api/instance/settings/experimental", { enableBetaSkills: true });
const markdown = `---\nname: ${scenario.skillKey}\ndescription: Context integrity output procedure.\n---\n\n# Context integrity output procedure\n\nWrite exactly one task document whose body contains the marker ${scenario.marker}. Finish the task after saving that document.`;
if (!markdown.startsWith(`---\nname: ${scenario.skillKey}\ndescription:`) || markdown.includes("\\n")) {
@@ -141,18 +142,18 @@ export async function runContextIntegrityFlow(input: {
api.get<Row[]>(`/api/issues/${issue!.id}/documents`),
api.get<Row[]>(`${companyPath}/skills`),
api.get<Row>(`/api/issues/${issue!.id}/queued-comments`),
scenario.id === "assigned-skill-explicit-invocation" ? api.get<Row>(`/api/agents/${fixtures.agent.id}/skills?companyId=${fixtures.company.id}`) : Promise.resolve({} as Row),
isAssignedSkillContext(scenario.id) ? api.get<Row>(`/api/agents/${fixtures.agent.id}/skills?companyId=${fixtures.company.id}`) : Promise.resolve({} as Row),
]);
const detailedDocuments = await Promise.all(documents.map((document) => api.get<Row>(`/api/issues/${issue!.id}/documents/${encodeURIComponent(String(document.key))}`)));
const assignedSkill = scenario.id === "assigned-skill-explicit-invocation"
const assignedSkill = isAssignedSkillContext(scenario.id)
? skillRows.find((skill) => skill.key === scenario.assignedSkill?.key || skill.slug === scenario.assignedSkill?.key)
: undefined;
const desiredSkill = (assignedState.desiredSkillEntries as Array<Row> | undefined)?.find((entry) => entry.key === scenario.assignedSkill?.key);
const runEvents = await Promise.all(runs.map((run) => api.get<Row[]>(`/api/heartbeat-runs/${run.id}/events?limit=1000`)));
const runLogs = await Promise.all(runs.map((run) => readContextIntegrityRunLog(api, run.id)));
const skillInvocationEvidence = scenario.id === "assigned-skill-explicit-invocation" && runEvents.some((events) => events.some((event) => containsExplicitSkillInput(event, String(assignedSkill?.slug ?? scenario.skillKey))));
skillRequestText = scenario.id === "assigned-skill-explicit-invocation" ? String(issue?.description ?? "") : "";
checkpoints.push({ phase, issue: { id: issue!.id, status: String(issue!.status) }, comments, queuedComments, documents: detailedDocuments as Array<{ key: string; body?: string | null }>, runs, runEvents: runEvents.flat(), runLogs, assignedSkill: assignedSkill ? { key: String(desiredSkill?.key ?? assignedSkill.key ?? assignedSkill.slug), runtimeName: String(assignedSkill.slug ?? ""), versionId: desiredSkill?.versionId === assignedVersionId ? assignedVersionId : null, markdown: String(assignedSkill.markdown ?? scenario.assignedSkill?.markdown ?? "") } : undefined, skillRequestText: scenario.id === "assigned-skill-explicit-invocation" ? skillRequestText : undefined, skillInvocationEvidence });
const skillInvocationEvidence = isAssignedSkillContext(scenario.id) && runEvents.some((events) => events.some((event) => containsExplicitSkillInput(event, String(assignedSkill?.slug ?? scenario.skillKey))));
skillRequestText = isAssignedSkillContext(scenario.id) ? String(issue?.description ?? "") : "";
checkpoints.push({ phase, issue: { id: issue!.id, status: String(issue!.status), identifier: String(issue!.identifier ?? ""), issuePrefix: String(fixtures.company.issuePrefix ?? ""), appOrigin: new URL(page.url()).origin }, comments, queuedComments, documents: detailedDocuments as Array<{ key: string; body?: string | null }>, runs, runEvents: runEvents.flat(), runLogs, assignedSkill: assignedSkill ? { key: String(desiredSkill?.key ?? assignedSkill.key ?? assignedSkill.slug), runtimeName: String(assignedSkill.slug ?? ""), versionId: desiredSkill?.versionId === assignedVersionId ? assignedVersionId : null, markdown: String(assignedSkill.markdown ?? scenario.assignedSkill?.markdown ?? "") } : undefined, skillRequestText: isAssignedSkillContext(scenario.id) ? skillRequestText : undefined, skillInvocationEvidence });
const checks = gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints });
input.observe(issue!, runs, checks);
await input.evidence("context-integrity.json", { schema: "paperclip.context-integrity.v1", scenario, budgetGuard, checkpoints, checks });
@@ -175,12 +176,12 @@ export async function runContextIntegrityFlow(input: {
await api.patch(`/api/agents/${fixtures.agent.id}/budgets`, {
budgetMonthlyCents: budgetGuard.agentMonthlyCents,
});
const taskPrompt = scenario.id === "assigned-skill-explicit-invocation"
const taskPrompt = isAssignedSkillContext(scenario.id)
? `${scenario.prompt}\n\nUse /${scenario.skillKey} for this request.`
: scenario.prompt;
skillRequestText = taskPrompt;
await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), prompt: taskPrompt, workMode: "standard" });
issue = await pollUntil({ label: "context-integrity task created", deadlineAt: input.deadlineAt, load: async () => (await api.get<Row[]>(`${companyPath}/issues?limit=100`)).find((row) => row.title === execution.task.buildTitle(nonce)), accept: Boolean });
const createdTask = await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), prompt: taskPrompt, workMode: "standard" });
issue = await pollUntil({ label: "context-integrity task created", deadlineAt: input.deadlineAt, load: async () => (await api.get<Row[]>(`${companyPath}/issues?limit=100`)).find((row) => row.id === createdTask.issueId), accept: Boolean });
if (!issue) throw new Error("Missing context-integrity task");
if (scenario.id === "ordered-comment-continuation") {
await pollUntil({
@@ -221,6 +222,21 @@ export async function runContextIntegrityFlow(input: {
await snapshot("initial");
}
await snapshot("final");
if (execution.suite.id === "stock-harness" && execution.profile.generation === "legacy") {
// Successful checkout HTTP calls emit activity even for an idempotent
// same-run claim. Server pre-dispatch checkout does not use that route.
// Keep this observation separate from the unchanged task-outcome oracle.
let checkoutActivity: unknown;
try {
checkoutActivity = observeCheckoutActivity({
companyId: fixtures.company.id, issueId: issue.id, agentId: fixtures.agent.id,
runs, activity: await api.get<Row[]>(`/api/issues/${issue.id}/activity`),
});
} catch {
checkoutActivity = { schema: "paperclip.checkout-activity.v1", status: "unavailable" };
}
await input.evidence("checkout-activity.json", checkoutActivity);
}
const checks = gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints });
const failures = checks.filter((check) => !check.passed);
if (failures.length) throw new Error(`Context-integrity matcher failures: ${failures.map((failure) => `${failure.id}: ${failure.detail}`).join("; ")}`);
tests/runner-e2e/context-integrity-scoring.ts
+25 -6
View File
@@ -1,11 +1,11 @@
import type { ContextIntegrityCase } from "./context-integrity-cases.js";
import { isAssignedSkillContext, PAPERCLIP_DOCUMENT_CASE, type ContextIntegrityCase } from "./context-integrity-cases.js";
export interface ContextIntegrityCheckpoint {
phase: "initial" | "comment-1" | "comment-2" | "comment-3" | "final";
issue: { id: string; status: string };
issue: { id: string; status: string; identifier?: string; issuePrefix?: string; appOrigin?: string };
comments: Array<Record<string, unknown>>;
queuedComments?: Record<string, unknown>;
documents: Array<{ key: string; body?: string | null }>;
documents: Array<{ key: string; body?: string | null; latestRevisionId?: string | null; latestRevisionNumber?: number | null }>;
runs: Array<Record<string, unknown>>;
assignedSkill?: { key: string; runtimeName?: string; versionId?: string | null; markdown?: string };
skillRequestText?: string;
@@ -117,7 +117,7 @@ export function gradeContextIntegrity(input: {
const expectedCommentIds = humanRows.slice(0, input.comments.length).map((comment) => String(comment.id ?? ""));
check("continuation-wake-comment-ids", Boolean(continuation) && JSON.stringify(wakeCommentIds(continuation)) === JSON.stringify(expectedCommentIds), "The deferred continuation wake must carry all public comment IDs in arrival order.");
}
if (input.id === "assigned-skill-explicit-invocation") {
if (isAssignedSkillContext(input.id)) {
const runtimeName = String(initial?.assignedSkill?.runtimeName ?? initial?.assignedSkill?.key ?? "");
const requestText = String(initial?.skillRequestText ?? "");
const references = requestText.split(/\s+/).map((word) => word.replace(/[.,]$/, ""));
@@ -126,8 +126,27 @@ export function gradeContextIntegrity(input: {
check("skill-source-marker", Boolean(initial?.assignedSkill?.markdown?.includes(input.marker)), "The assigned pinned skill source must contain the output marker.");
check("marker-not-in-request", !requestText.includes(input.marker) && !finalComments.some((comment) => comment.includes(input.marker)), "The output marker must originate from the assigned skill, not the task request or comments.");
}
const output = final ? oneOutput(final, input.marker, input.id === "assigned-skill-explicit-invocation") : undefined;
check("single-durable-output", Boolean(output) && final!.documents.length === 1, input.id === "assigned-skill-explicit-invocation" ? "Exactly one durable task document must contain the skill's marker." : "Exactly one durable packing report document must be saved.");
const output = final ? oneOutput(final, input.marker, isAssignedSkillContext(input.id)) : undefined;
check("single-durable-output", Boolean(output) && final!.documents.length === 1, isAssignedSkillContext(input.id) ? "Exactly one durable task document must contain the skill's marker." : "Exactly one durable packing report document must be saved.");
if (input.id === PAPERCLIP_DOCUMENT_CASE) {
check("saved-document-revision", Boolean(output?.latestRevisionId) && Number.isInteger(output?.latestRevisionNumber) && Number(output?.latestRevisionNumber) > 0,
"The public saved document must have a persisted revision and the requested content.");
const route = output && final?.issue.identifier && final.issue.issuePrefix
? `/${final.issue.issuePrefix}/issues/${final.issue.identifier}#document-${output.key}` : undefined;
const agentComments = (final?.comments ?? []).filter(comment => comment.authorType === "agent" || Boolean(comment.authorAgentId));
const origin = final?.issue.appOrigin;
const usableLink = Boolean(route && origin && agentComments.some(comment => {
const links = String(comment.body ?? "").matchAll(/\[[^\]]*\]\((?:<([^>]+)>|([^\s)]+))(?:\s+["'][^"']*["'])?\)/g);
return [...links].some(link => {
try {
const url = new URL(link[1] ?? link[2]!, origin);
return url.origin === origin && `${url.pathname}${url.hash}` === route && !url.search;
} catch { return false; }
});
}));
check("saved-document-link", usableLink,
"An agent completion comment must link the exact saved document on this task; a local path or claimed URL does not suffice.");
}
check("completed-task", final?.issue.status === "done", `Final task status: ${final?.issue.status ?? "missing"}.`);
check("successful-runs", Boolean(final?.runs.length) && final!.runs.every((run) => run.status === "succeeded"), "All recorded context-integrity runs must succeed.");
return checks;
tests/runner-e2e/continuation-cases.ts
+1
View File
@@ -59,6 +59,7 @@ export const continuationTasks: readonly RunnerTaskFixture[] =
groups: [],
workMode: "standard",
flow: "continuation",
automaticRetryPolicy: "single_attempt",
expectedRunCount:
id === "provider-question-bridge" ? 1 : id === "completed-action-resume"
? 4
tests/runner-e2e/continuation-fixtures.test.ts
+17 -1
View File
@@ -1,6 +1,6 @@
import { expect, it, vi } from "vitest";
import type { RunnerApi } from "./api.js";
import { prepareLegacyContinuationSkill } from "./continuation-fixtures.js";
import { prepareLegacyContinuationSkill, prepareContinuationBudget } from "./continuation-fixtures.js";
it("initializes the production library and assigns the operational skill before execution", async () => {
const post = vi.fn();
const get = vi.fn(async () => [{ key: "paperclipai/paperclip/paperclip" }]);
@@ -11,3 +11,19 @@ it("initializes the production library and assigns the operational skill before
await expect(prepareLegacyContinuationSkill({ get, post } as unknown as RunnerApi, "company", "agent")).rejects.toThrow("missing the bundled");
expect(post).toHaveBeenCalledTimes(1);
});
it("persists and reads both continuation budget hard stops before returning", async () => {
const calls: string[] = [];
const api = { patch: vi.fn(async (url: string) => { calls.push(`patch:${url}`); }),
get: vi.fn(async (url: string) => { calls.push(`get:${url}`); return { budgetMonthlyCents: 1_000 }; }) };
await expect(prepareContinuationBudget(api as unknown as RunnerApi, "company", "agent"))
.resolves.toEqual({ companyMonthlyCents: 1_000, agentMonthlyCents: 1_000 });
expect(calls).toEqual(["patch:/api/companies/company/budgets", "patch:/api/agents/agent/budgets",
"get:/api/companies/company", "get:/api/agents/agent"]);
});
it.each(["company", "agent"])("rejects a missing %s budget instead of starting paid work", async target => {
const api = { patch: vi.fn(), get: vi.fn(async (url: string) =>
({ budgetMonthlyCents: url.endsWith(target) ? 0 : 1_000 })) };
await expect(prepareContinuationBudget(api as unknown as RunnerApi, "company", "agent"))
.rejects.toThrow("hard stops were not persisted");
});
tests/runner-e2e/continuation-fixtures.ts
+16
View File
@@ -8,3 +8,19 @@ export async function prepareLegacyContinuationSkill(api: RunnerApi, companyId:
if (!skills.some(skill => skill.key === key)) throw new Error("Continuation fixture is missing the bundled Paperclip operational skill");
await api.post(`/api/agents/${agentId}/skills/sync?companyId=${companyId}`, { desiredSkills: [key], mode: "add" });
}
/** Apply and verify both hard stops before any task can start. These are caps,
* not estimates of provider invoices or unmetered local runtime costs. */
export async function prepareContinuationBudget(api: RunnerApi, companyId: string, agentId: string) {
const budgetMonthlyCents = 1_000;
await api.patch(`/api/companies/${companyId}/budgets`, { budgetMonthlyCents });
await api.patch(`/api/agents/${agentId}/budgets`, { budgetMonthlyCents });
const [company, agent] = await Promise.all([
api.get<{ budgetMonthlyCents: unknown }>(`/api/companies/${companyId}`),
api.get<{ budgetMonthlyCents: unknown }>(`/api/agents/${agentId}`),
]);
if (company.budgetMonthlyCents !== budgetMonthlyCents || agent.budgetMonthlyCents !== budgetMonthlyCents)
throw new Error("Continuation company and agent budget hard stops were not persisted");
return { companyMonthlyCents: budgetMonthlyCents, agentMonthlyCents: budgetMonthlyCents };
}
tests/runner-e2e/continuation-flow.ts
+18 -6
View File
@@ -1,11 +1,11 @@
import { gradeLifecycleBaseline, type LifecycleCheckpoint } from "./lifecycle-baseline.js";
import { lifecycleLiveCase, lifecycleLiveContinuation, gradeLifecycleNarrative } from "./lifecycle-live-cases.js";
import { prepareLegacyContinuationSkill } from "./continuation-fixtures.js";
import { prepareLegacyContinuationSkill, prepareContinuationBudget } from "./continuation-fixtures.js";
import { captureFirstTaskAttachments } from "./first-task-attachments.js";
import { answerableRuntimeRunIds, isSingleClaudeQuestion } from "./runtime-question-readiness.js";
import { expect, type Page } from "@playwright/test";
import path from "node:path";
import { continuationAnswerCommitted, continuationInitialReady } from "./continuation-readiness.js";
import { continuationAnswerCommitted, continuationInitialReady, continuationCheckpointReady } from "./continuation-readiness.js";
import { captureLoadedContinuation } from "./continuation-screenshot.js";
import { seedContinuationContext } from "./continuation-workspace.js";
import { pollUntil, type RunnerApi } from "./api.js";
@@ -53,6 +53,7 @@ export async function runContinuationFlow(input: {
? lifecycleLiveContinuation(execution.task.id, input.nonce)
: continuationScenario(execution.task.id, input.nonce);
const checkpoints: LifecycleCheckpoint[] = [];
let budgetGuard: Awaited<ReturnType<typeof prepareContinuationBudget>> | undefined;
let issue: Row | undefined;
let runs: Row[] = [];
let checks: ReturnType<typeof gradeContinuation> = [];
@@ -84,6 +85,11 @@ export async function runContinuationFlow(input: {
const paused = answerableRuntimeRunIds(state.interactions);
const idle =
continuationAnswerCommitted(state.interactions, answeredInteractionId) &&
continuationCheckpointReady({
issue: { status: state.issue.status, executionRunId: state.issue.executionRunId },
runs: state.runs.map(r => ({ id: r.id, status: r.status, runtimeMode: r.runtimeMode })),
interactions: state.interactions,
}) &&
state.runs.some((r) => !prior.has(r.id) || previousPaused.has(r.id)) &&
state.runs.every((r) => ["succeeded", "failed", "timed_out", "cancelled"].includes(r.status) ||
(r.status === "running" && paused.has(r.id))) &&
@@ -91,7 +97,8 @@ export async function runContinuationFlow(input: {
!state.issue.activeRecoveryAction &&
(!requireQuestion || continuationInitialReady(state.interactions));
const key = idle
? state.runs.map((r) => `${r.id}:${r.status}`).join()
? JSON.stringify([state.issue.status, state.issue.executionRunId,
state.runs.map(r => [r.id, r.status]), state.interactions.map(i => [i.id, i.status])])
: "";
const ready = !!key && key === stable;
stable = key;
@@ -120,13 +127,14 @@ export async function runContinuationFlow(input: {
);
}
async function snapshot(phase: ContinuationCheckpoint["phase"]) {
const [tasks, summaries, comments, interactions, attachments] =
const [tasks, summaries, comments, interactions, attachments, activity] =
await Promise.all([
api.get<Row[]>(tasksPath),
api.get<Row[]>(`/api/issues/${issue!.id}/documents`),
api.get<Row[]>(`/api/issues/${issue!.id}/comments?order=asc`),
api.get<Row[]>(`/api/issues/${issue!.id}/interactions`),
captureFirstTaskAttachments(api, [{ id: issue!.id }], input.secrets),
api.get<Row[]>(`/api/issues/${issue!.id}/activity`),
]);
const documents = await Promise.all(
summaries.map((d) =>
@@ -149,12 +157,14 @@ export async function runContinuationFlow(input: {
) as ContinuationCheckpoint["children"],
documents: documents as ContinuationCheckpoint["documents"],
comments,
activity,
interactions,
attachments,
runs: [...runs] as ContinuationCheckpoint["runs"],
});
await input.evidence("continuation.json", {
...scenario,
budgetGuard,
checkpoints,
checks,
});
@@ -219,11 +229,12 @@ export async function runContinuationFlow(input: {
expect(c.issue.status, "waiting is not complete").not.toBe("done");
}
try {
budgetGuard = await prepareContinuationBudget(api, fixtures.company.id, fixtures.agent.id);
if (execution.profile.generation === "legacy") await prepareLegacyContinuationSkill(api, fixtures.company.id, fixtures.agent.id);
await api.patch("/api/instance/settings/experimental", {
enableClassicTaskInterface: false,
});
await createTaskThroughUi({
const createdTask = await createTaskThroughUi({
page,
issuePrefix: fixtures.company.issuePrefix!,
agentName: fixtures.agent.name,
@@ -236,7 +247,7 @@ export async function runContinuationFlow(input: {
deadlineAt: input.deadlineAt,
load: async () =>
(await api.get<Row[]>(tasksPath)).find(
(t) => t.title === execution.task.buildTitle(input.nonce),
(t) => t.id === createdTask.issueId,
),
accept: Boolean,
});
@@ -288,6 +299,7 @@ export async function runContinuationFlow(input: {
if (issue) input.observe(issue, runs, checks);
await input.evidence("continuation.json", {
...scenario,
budgetGuard,
checkpoints,
checks,
});
tests/runner-e2e/continuation-readiness.test.ts
+35 -1
View File
@@ -1,5 +1,5 @@
import { describe, expect, it } from "vitest";
import { continuationAnswerCommitted, continuationInitialReady } from "./continuation-readiness.js";
import { continuationAnswerCommitted, continuationInitialReady, continuationCheckpointReady } from "./continuation-readiness.js";
import { runnerMatrix } from "./catalog.js";
describe("continuation readiness", () => {
@@ -23,3 +23,37 @@ it("waits for the clicked answer to commit instead of grading the original pause
expect(continuationAnswerCommitted([{ ...card, status: "cancelled" }], card.id)).toBe(false);
expect(continuationAnswerCommitted([{ ...card, status: "answered" }], card.id)).toBe(true);
});
describe("checkpoint finalization boundary", () => {
const pending = [{ id: "question", kind: "ask_user_questions", status: "pending" }];
const run = { id: "run", status: "succeeded", runtimeMode: "native" };
it("waits across repeated terminal polls until review state and lock release are both visible", () => {
const polls = [
{ status: "in_progress", executionRunId: "run" },
{ status: "in_progress", executionRunId: "run" },
{ status: "in_review", executionRunId: "run" },
{ status: "in_review", executionRunId: null },
];
expect(polls.map(issue => continuationCheckpointReady({ issue, runs: [run], interactions: pending })))
.toEqual([false, false, false, true]);
expect(continuationCheckpointReady({ issue: { status: "in_progress", executionRunId: null },
runs: [run], interactions: pending })).toBe(false);
});
it("also waits for final completion to release its execution lock", () => {
for (const executionRunId of ["run", null]) {
expect(continuationCheckpointReady({ issue: { status: "done", executionRunId },
runs: [run], interactions: [{ ...pending[0], status: "answered" }] })).toBe(executionRunId === null);
}
});
it("admits only the bound running native provider question without releasing its lock", () => {
const input = { issue: { status: "in_progress", executionRunId: "run" },
runs: [{ ...run, status: "running" }], interactions: [{ ...pending[0], sourceRunId: "run",
payload: { runtimeRequestId: "provider-request" } }] };
expect(continuationCheckpointReady(input)).toBe(true);
expect(continuationCheckpointReady({ ...input, interactions: pending })).toBe(false);
expect(continuationCheckpointReady({ ...input, issue: { ...input.issue, executionRunId: "other" } })).toBe(false);
expect(continuationCheckpointReady({ ...input, runs: [{ ...input.runs[0], runtimeMode: "legacy" }] })).toBe(false);
expect(continuationCheckpointReady({ ...input, runs: [...input.runs, { ...run, id: "extra", status: "queued" }] })).toBe(false);
});
});
tests/runner-e2e/continuation-readiness.ts
+24
View File
@@ -8,3 +8,27 @@ export function continuationInitialReady(interactions: ReadonlyArray<{ kind?: un
export function continuationAnswerCommitted(interactions: ReadonlyArray<{ id?: unknown; status?: unknown }>, interactionId?: string): boolean {
return !interactionId || interactions.some(i => i.id === interactionId && i.status === "answered");
}
/** Run completion precedes task-lock release. Observe both before taking a
* checkpoint, even when the terminal run status is stable across many polls. */
export function continuationCheckpointReady(input: {
issue: { status: string; executionRunId?: string | null };
runs: ReadonlyArray<{ id: string; status: string; runtimeMode?: string }>;
interactions: ReadonlyArray<{ id?: unknown; kind?: unknown; status?: unknown;
sourceRunId?: unknown; payload?: { runtimeRequestId?: unknown } }>;
}): boolean {
if (!input.runs.length) return false;
const active = input.runs.filter(run => ["running", "queued"].includes(run.status));
const pending = input.interactions.filter(i => i.status === "pending");
if (active.length === 0) {
const waiting = pending.some(i => ["ask_user_questions", "request_confirmation", "request_approval"].includes(String(i.kind)));
return input.issue.executionRunId === null && (!waiting || input.issue.status === "in_review");
}
if (active.length !== 1) return false;
const run = active[0];
return run.status === "running" && run.runtimeMode === "native" &&
input.issue.executionRunId === run.id && ["in_progress", "in_review"].includes(input.issue.status) &&
pending.some(i => i.kind === "ask_user_questions" && typeof i.id === "string" && !!i.id.trim() &&
i.sourceRunId === run.id && typeof i.payload?.runtimeRequestId === "string" && !!i.payload.runtimeRequestId.trim());
}
tests/runner-e2e/continuation-scoring.ts
+1 -1
View File
@@ -3,7 +3,7 @@ import { gradeQuestionDocumentation } from "./question-documentation-scoring.js"
import type { ContinuationCase } from "./continuation-cases.js";
export interface ContinuationCheckpoint {
phase: "initial" | "answered" | "revised" | "final";
issue: { id: string; status: string };
issue: { id: string; status: string; assigneeAgentId?: string | null };
children: Array<{
id: string;
title: string;
tests/runner-e2e/continuation-screenshot.ts
+9
View File
@@ -13,6 +13,15 @@ export async function captureLoadedContinuation(
await capture();
}
/** A task title is mutable while an agent runs. Match the route and visible stable identifier. */
export async function waitForTaskIdentity(page: Page, taskPath: string, identifier: string, timeout = 30_000) {
await expect(page).toHaveURL(url => url.pathname === taskPath, { timeout });
const currentIdentifier = page.getByRole("navigation", { name: "breadcrumb" })
.locator('[aria-current="page"] [data-slot="task-title-identifier"]');
await expect(currentIdentifier).toBeVisible({ timeout });
await expect(currentIdentifier).toHaveText(identifier, { timeout });
}
/** Wait for the persisted task projection before taking a browser screenshot. */
export async function waitForTaskChatRendered(
page: Page,
tests/runner-e2e/continuation.test.ts
+1
View File
@@ -78,6 +78,7 @@ describe("continuation behavioral evaluation", () => {
]),
);
expect(matrix.every((c) => !c.suite.manualOnly)).toBe(true);
expect(matrix.every(c => c.task.automaticRetryPolicy === "single_attempt")).toBe(true);
});
it.each(CONTINUATION_CASES.filter(id => !["question-tool-documentation", "provider-question-bridge"].includes(id)))("accepts a complete %s recording", (id) =>
expect(failures(recording(id))).toEqual([]),
tests/runner-e2e/cursor-native-cases.ts
+2 -2
View File
@@ -29,7 +29,7 @@ export function cursorNativePrompt(id: typeof cursorNativeCaseDesigns[number]["i
"Wait for the actual native decision. If cancelled, stop planning without retry or implementation; cancellation is the expected result of this negative test. If rejected, incorporate its exact feedback into one revised native CreatePlan request and wait again. Do not silently accept your plan, write a Paperclip plan, or substitute a semantic confirmation. If unavailable, report that limitation without claiming completion.", id === "native-plan-reject-revise-accept" ? "After the revised plan is accepted, end this planning turn. Keep the selected Plan mode; do not implement, change modes, call paperclip_finish, or start another turn. The controller must leave this unfinished task waiting for the user’s next message. Do not substitute a semantic confirmation or completion call." : finish,
].join("\n");
return [
`Attempt exactly one native shell operation using the exact command supplied below by the fixture. This is a deliberate denial test. Wait for the actual permission decision. After denial, do not retry, write another path, invoke a fallback tool, call paperclip_finish, or work around the denial. Wait for the operator to cancel this unfinished task. Do not substitute another tool or operation.`,
`Attempt exactly one native shell operation using the exact command supplied below by the fixture. This is a deliberate denial test. Wait for the actual permission decision. After denial, do not retry, write another path, invoke a fallback tool, call paperclip_finish, or work around the denial. End this turn without claiming task completion. The task must remain unfinished. Do not substitute another tool or operation.`,
].join("\n");
}
@@ -106,7 +106,7 @@ export function hasCursorDenialBoundary(input: {
export const cursorNativeTasks: readonly (Omit<RunnerTaskFixture, "flow"> & { flow: "cursor_native" })[] = cursorNativeCaseDesigns.map(design => ({
id: design.id, label: `Cursor ${design.id}`, groups: [], workMode: "standard", flow: "cursor_native",
expectedRunCount: 1, attemptTimeoutMs: { local: 300_000, daytona: 300_000 }, turnTimeoutMs: 120_000,
expectedTerminalState: design.id === "native-write-deny-reconnect" ? { issue: "in_progress", run: "cancelled" } : design.id === "native-plan-reject-revise-accept" ? { issue: "in_progress", run: "succeeded" } : { issue: "done", run: "succeeded" },
expectedTerminalState: design.id === "native-write-deny-reconnect" ? { issue: "in_progress", run: "failed" } : design.id === "native-plan-reject-revise-accept" ? { issue: "in_progress", run: "succeeded" } : { issue: "done", run: "succeeded" },
buildTitle: nonce => `Cursor ${design.id} ${nonce}`,
buildPrompt: nonce => cursorNativePrompt(design.id, nonce),
buildVisibleMarker: nonce => `CURSOR-NATIVE-${nonce}`,
tests/runner-e2e/cursor-native-evidence.test.ts
+41 -2
View File
@@ -1,6 +1,6 @@
import { createHash } from "node:crypto";
import { expect, it } from "vitest";
import { cursorDeniedCommand, hasCursorDeniedCommand, hasCursorCancellation, readCursorToolEvidence, type CursorToolNotice } from "./cursor-native-evidence.js";
import { cursorDeniedCommand, hasCursorDeniedCommand, hasCursorDeniedTurnTerminal, hasCursorCancellation, readCursorToolEvidence, type CursorToolNotice } from "./cursor-native-evidence.js";
export function denialNotices(path = "/fixture/denied.txt"): CursorToolNotice[] {
const base = { runId: "run", sessionId: "session", turnId: "turn", toolCallId: "tool", operation: "execute", commandSha256: cursorDeniedCommand(path).commandSha256 };
@@ -8,7 +8,7 @@ export function denialNotices(path = "/fixture/denied.txt"): CursorToolNotice[]
{ ...base, seq: 3, stage: "permission_delivered", requestId: "request", outcome: "reject_once" }, { ...base, seq: 4, stage: "tool", status: "failed" }];
}
const grade = (notices: CursorToolNotice[]) => hasCursorDeniedCommand({ notices, runId: "run", turnId: "turn", requestId: "request", toolCallId: "tool", commandSha256: cursorDeniedCommand("/fixture/denied.txt").commandSha256 });
it("requires the exact absolute command, native request, delivered denial and failed call in order", () => {
it("requires the exact absolute command, native request, delivered denial and settled call in order", () => {
expect(grade(denialNotices())).toBe(true); expect(grade(denialNotices("/other/denied.txt"))).toBe(false);
expect(grade(denialNotices().slice(1))).toBe(false); expect(grade([...denialNotices(), denialNotices()[0]!])).toBe(false);
for (const key of ["runId", "sessionId", "turnId", "toolCallId", "commandSha256"]) {
@@ -103,3 +103,42 @@ it("allows explicit earlier remote bootstrap reads while rejecting unknown kinds
expect(hasCursorDeniedCommand({ ...input, bootstrapReadProof, notices: [...rows, { ...read, seq: 20 }] })).toBe(false);
expect(hasCursorDeniedCommand({ ...input, bootstrapReadProof, notices: [{ ...read, sessionId: "other" }, ...rows] })).toBe(false);
});
it("accepts Cursor's transport-completed denial only with the exact delivered rejection", () => {
const rows = denialNotices(); rows[3]!.status = "completed";
expect(grade(rows)).toBe(true);
expect(grade(rows.filter(row => row.stage !== "permission_delivered"))).toBe(false);
for (const outcome of ["allow_once", "allow_always", "cancel"]) {
const changed = structuredClone(rows); changed[2]!.outcome = outcome; expect(grade(changed)).toBe(false);
}
expect(grade([...rows, { ...rows[3]!, seq: 5 }])).toBe(false);
expect(grade([...rows, { ...rows[0]!, seq: 5, toolCallId: "replayed-write" }])).toBe(false);
});
it("requires a correlated failed semantic finalization after the delivered denial, never false success", () => {
const evidence = { runId: "run", seq: 3, protocolSchemaVersion: 1, eventType: "provider.notice.recorded", payload: { prpEvent: {
schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", runId: "run", turnId: "turn", emittedAt: new Date(1).toISOString(), eventType: "provider.notice.recorded",
payload: { schema: "paperclip.provider.notice.v1", scope: "turn", category: "cursor_tool_evidence_v1", provenance: { sessionId: "session", turnId: "turn", eventType: "permission_delivered", method: "session/request_permission" },
details: [{ name: "stage", value: "permission_delivered" }, { name: "toolCallId", value: "tool" }, { name: "requestId", value: "request" }, { name: "outcome", value: "reject_once" }] },
} } };
const terminal = { runId: "run", seq: 5, protocolSchemaVersion: 1, eventType: "turn.completed", payload: { prpEvent: {
schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", runId: "run", turnId: "turn", emittedAt: new Date(2).toISOString(), eventType: "turn.completed", payload: { status: "completed", error: null },
} } };
const input = { runId: "run", turnId: "turn", requestId: "request", issue: { id: "issue", status: "in_progress" },
run: { id: "run", nativeIssueId: "issue", runtimeMode: "native", status: "failed", errorCode: "native_session_interrupted", error: "native_finalization_missing: session returned no semantic result" }, events: [evidence, terminal] };
expect(hasCursorDeniedTurnTerminal(input)).toBe(true);
const declined = { ...input, issue: { ...input.issue, status: "blocked" }, run: { ...input.run, errorCode: "native_permission_declined", error: "native_finalization_missing: session returned no semantic result; provider permission was declined; explicit direction is required" } };
expect(hasCursorDeniedTurnTerminal(declined)).toBe(true);
expect(hasCursorDeniedTurnTerminal({ ...declined, run: { ...declined.run, error: declined.run.error.replace("provider permission", "Cursor permission") } })).toBe(true);
expect(hasCursorDeniedTurnTerminal({ ...declined, run: { ...declined.run, error: "unrelated failure" } })).toBe(false);
expect(hasCursorDeniedTurnTerminal({ ...declined, issue: { ...declined.issue, status: "done" } })).toBe(false);
expect(hasCursorDeniedTurnTerminal({ ...declined, events: [terminal] })).toBe(false);
for (const status of ["succeeded", "cancelled", "timed_out", "running"]) expect(hasCursorDeniedTurnTerminal({ ...input, run: { ...input.run, status } })).toBe(false);
expect(hasCursorDeniedTurnTerminal({ ...input, issue: { ...input.issue, status: "done" } })).toBe(false);
expect(hasCursorDeniedTurnTerminal({ ...input, run: { ...input.run, errorCode: "other" } })).toBe(false);
expect(hasCursorDeniedTurnTerminal({ ...input, run: { ...input.run, nativeIssueId: "foreign" } })).toBe(false);
expect(hasCursorDeniedTurnTerminal({ ...input, events: [terminal] })).toBe(false);
expect(hasCursorDeniedTurnTerminal({ ...input, events: [evidence, terminal, terminal] })).toBe(false);
expect(hasCursorDeniedTurnTerminal({ ...input, turnId: "foreign" })).toBe(false);
expect(hasCursorDeniedTurnTerminal({ ...input, requestId: "foreign" })).toBe(false);
expect(hasCursorDeniedTurnTerminal({ ...input, events: [evidence, { ...terminal, seq: 2 }] })).toBe(false);
});
tests/runner-e2e/cursor-native-evidence.ts
+35 -4
View File
@@ -40,7 +40,7 @@ export function readCursorToolEvidence(rows: readonly unknown[], runId: string):
/** Exact absolute target removes any dependency on implicit native shell cwd. */
export function cursorDeniedCommand(path: string) {
if (!isAbsolute(path) || /[\u0000-\u001f\u007f]/u.test(path)) throw new Error("Invalid denial target");
const command = `printf 'MUST_NOT_EXIST' > '${path.replaceAll("'", "'\\''")}'`;
const command = `printf 'MUST-NOT-EXIST' > '${path.replaceAll("'", "'\\''")}'`;
return { command, commandSha256: `sha256:${createHash("sha256").update(command).digest("hex")}` };
}
export function hasCursorDeniedCommand(input: {
@@ -57,11 +57,42 @@ export function hasCursorDeniedCommand(input: {
if (!same(request) || request.requestId !== input.requestId || !request.declineOffered) return false;
const origins = notices.filter(row => row.stage === "tool" && row.status === "pending");
const delivered = notices.filter(row => row.stage === "permission_delivered");
const failed = notices.filter(row => row.stage === "tool" && row.status === "failed");
// Cursor reports a denied call as completed when its transport settles. That
// status does not prove execution; correlated reject_once and independent
// continuous filesystem observation establish denial in the Product flow.
const settled = notices.filter(row => row.stage === "tool" && ["failed", "completed"].includes(row.status ?? ""));
return origins.length === 1 && same(origins[0]!) && delivered.length === 1 && same(delivered[0]!)
&& delivered[0]!.requestId === input.requestId && delivered[0]!.outcome === "reject_once"
&& failed.length === 1 && same(failed[0]!) && origins[0]!.seq < request.seq && request.seq < delivered[0]!.seq && delivered[0]!.seq < failed[0]!.seq
&& notices.every(row => same(row) && row.status !== "completed");
&& settled.length === 1 && same(settled[0]!) && origins[0]!.seq < request.seq && request.seq < delivered[0]!.seq && delivered[0]!.seq < settled[0]!.seq
&& notices.every(row => same(row));
}
/** A denied native turn may end without semantic completion. Never qualify
* that as task success or as an acknowledged operator cancellation. */
export function hasCursorDeniedTurnTerminal(input: {
run: unknown; issue: unknown; events: readonly unknown[]; runId: string; turnId: string; requestId: string;
}): boolean {
const run = rec(input.run), issue = rec(input.issue);
const missingResult = run.errorCode === "native_session_interrupted" && run.error === "native_finalization_missing: session returned no semantic result";
const declined = run.errorCode === "native_permission_declined" && [
"native_finalization_missing: session returned no semantic result; provider permission was declined; explicit direction is required",
// Retained results from before the provider lifecycle boundary cleanup.
"native_finalization_missing: session returned no semantic result; Cursor permission was declined; explicit direction is required",
].includes(run.error);
if (!id(input.runId) || !id(input.turnId) || !id(input.requestId) || run.id !== input.runId || run.nativeIssueId !== issue.id
|| run.status !== "failed" || run.runtimeMode !== "native" || issue.status !== (declined ? "blocked" : "in_progress")
|| (!missingResult && !declined)) return false;
const terminals = input.events.map(rec).map(row => ({ row, event: rec(rec(row.payload).prpEvent) }))
.filter(({ event }) => ["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"].includes(event.eventType));
const delivered = readCursorToolEvidence(input.events, input.runId).filter(notice => notice.stage === "permission_delivered"
&& notice.turnId === input.turnId && notice.requestId === input.requestId && notice.outcome === "reject_once");
if (terminals.length !== 1 || delivered.length !== 1) return false;
const { row, event } = terminals[0]!;
return row.runId === input.runId && row.eventType === event.eventType && event.runId === input.runId && event.turnId === input.turnId
&& event.eventType === "turn.completed" && event.schema === "paperclip.prp.event.v1" && event.schemaVersion === 1
&& row.protocolSchemaVersion === 1 && event.sourceKind === "runner" && rec(event.payload).status === "completed"
&& rec(event.payload).error === null && Number.isSafeInteger(row.seq) && row.seq > delivered[0]!.seq
&& Number.isFinite(Date.parse(event.emittedAt));
}
export function hasCursorCancellation(input: { run: unknown; issue: unknown; events: readonly unknown[]; runId: string; turnId: string; requestedAt: number }): boolean {
tests/runner-e2e/cursor-native-flow.test.ts
+4 -4
View File
@@ -12,7 +12,7 @@ it("keeps native mode/permission choices explicit and artifact export pending",
}
expect(cursorNativeCaseDesigns.filter(row => row.method !== "session/request_permission").every(row => row.cursorMode === "plan")).toBe(true);
expect(cursorNativeCaseDesigns.find(row => row.method === "session/request_permission")).toMatchObject({ cursorMode: "agent", permissionMode: "approve-reads" });
expect(cursorNativeTasks.find(task => task.id === "native-write-deny-reconnect")!.expectedTerminalState).toEqual({ issue: "in_progress", run: "cancelled" });
expect(cursorNativeTasks.find(task => task.id === "native-write-deny-reconnect")!.expectedTerminalState).toEqual({ issue: "in_progress", run: "failed" });
expect(cursorNativePlanArtifactGate.status).toBe("pending");
expect(cursorNativePlanArtifactGate.nativePath).toContain("<private provider HOME>");
});
@@ -43,7 +43,7 @@ it("awaits owned remote baseline before returning the exact remote denial comman
const prepared = await prepareCursorRemoteAction({ fixture, ...binding, deniedRelative: "denied.txt", prompt: "Perform the actual test" });
order.push("publish-action");
expect(order).toEqual(["before-action-publication", "baseline-ready", "publish-action"]);
expect(prepared.prompt).toContain("printf 'MUST_NOT_EXIST' > '/home/daytona/paperclip-workspace/denied.txt'");
expect(prepared.prompt).toContain("printf 'MUST-NOT-EXIST' > '/home/daytona/paperclip-workspace/denied.txt'");
expect(prepared.initial).toMatchObject({ phase: "before-request", absent: true });
expect(prepared.command?.commandSha256).toMatch(/^sha256:[a-f0-9]{64}$/);
await expect(prepareCursorRemoteAction({ fixture, ...binding, runId: "wrong-run", deniedRelative: "denied.txt", prompt: "test" })).rejects.toThrow(/another run/);
@@ -62,13 +62,13 @@ it("refuses remote native execution before touching API when bootstrap or cleanu
});
it("requires the exact passive accepted-plan disposition without mode promotion or extra work", () => {
const state = { issue: { id: "issue", status: "in_progress" }, interactions: [{ status: "answered" }], runs: [{ id: "run", nativeIssueId: "issue", runtimeMode: "native", status: "succeeded", runnerProfileJson: { nativeExecutionInput: { provider: { cursorMode: "plan" } } }, resultJson: { finalizationPhase: "committed", finalizationReasonCode: "native_plan_accepted_waiting_for_continuation", authoritativeDecision: "in_progress" } }] };
const state = { issue: { id: "issue", status: "in_progress" }, interactions: [{ status: "answered" }], runs: [{ id: "run", nativeIssueId: "issue", runtimeMode: "native", status: "succeeded", runnerProfileJson: { nativeExecutionInput: { provider: { mode: "plan" } } }, resultJson: { finalizationPhase: "committed", finalizationReasonCode: "native_plan_accepted_waiting_for_continuation", authoritativeDecision: "in_progress" } }] };
expect(hasCursorAcceptedPlanWait(state)).toBe(true);
const mutations = [
(s: typeof state) => { s.issue.status = "done"; },
(s: typeof state) => { s.runs[0]!.status = "failed"; },
(s: typeof state) => { s.runs[0]!.nativeIssueId = "foreign"; },
(s: typeof state) => { s.runs[0]!.runnerProfileJson.nativeExecutionInput.provider.cursorMode = "agent"; },
(s: typeof state) => { s.runs[0]!.runnerProfileJson.nativeExecutionInput.provider.mode = "agent"; },
(s: typeof state) => { s.runs[0]!.resultJson.finalizationReasonCode = "live_continuation_registered"; },
(s: typeof state) => { s.runs[0]!.resultJson.finalizationPhase = "pending"; },
(s: typeof state) => { s.runs.push(structuredClone(s.runs[0]!)); },
tests/runner-e2e/cursor-native-flow.ts
+25 -22
View File
@@ -7,8 +7,8 @@ import { expect, type Page } from "@playwright/test";
import { pollUntil, type RunnerApi } from "./api.js";
import { collectRunEvents } from "./run-observations.js";
import { createTaskThroughUi } from "./user-actions.js";
import { observeRunProcesses, createDeniedTargetFixture } from "./copilot-local-fixtures.js";
import { cursorDeniedCommand, hasCursorDeniedCommand, hasCursorCancellation, readCursorToolEvidence, assertCursorRemoteSnapshot, cursorRemoteDeniedSample, hasCursorRemoteRetirement, hasCursorRemoteWorkspaceUnchanged, type CursorRemoteSnapshot, type CursorRemoteBinding, type CursorToolNotice } from "./cursor-native-evidence.js";
import { observeRunProcesses, createDeniedTargetFixture } from "./native-local-fixtures.js";
import { cursorDeniedCommand, hasCursorDeniedCommand, hasCursorDeniedTurnTerminal, readCursorToolEvidence, assertCursorRemoteSnapshot, cursorRemoteDeniedSample, hasCursorRemoteRetirement, hasCursorRemoteWorkspaceUnchanged, type CursorRemoteSnapshot, type CursorRemoteBinding, type CursorToolNotice } from "./cursor-native-evidence.js";
import { cursorNativeCaseDesigns, cursorNativePlanArtifactGate, hasCursorDenialBoundary, hasCursorPlanDecision, hasDeliveredCursorNativeRequest, hasExactCursorNativeResponse, type CursorNativeMethod } from "./cursor-native-cases.js";
import type { LiveFixtureValues } from "./live-fixtures.js";
import type { MatrixExecution } from "./types.js";
@@ -34,7 +34,7 @@ export function hasCursorAcceptedPlanWait(state: { issue: Row; runs: Row[]; inte
if (state.runs.length !== 1 || state.issue.status !== "in_progress" || state.interactions.some(card => card.status === "pending")) return false;
const run = state.runs[0]!;
return run.status === "succeeded" && run.runtimeMode === "native" && run.nativeIssueId === state.issue.id
&& run.runnerProfileJson?.nativeExecutionInput?.provider?.cursorMode === "plan"
&& run.runnerProfileJson?.nativeExecutionInput?.provider?.mode === "plan"
&& run.resultJson?.finalizationPhase === "committed"
&& run.resultJson?.finalizationReasonCode === "native_plan_accepted_waiting_for_continuation"
&& run.resultJson?.authoritativeDecision === "in_progress";
@@ -140,7 +140,7 @@ export async function runCursorNativeFlow(input: {
const deniedRelative = localDeniedTarget?.targetRelativePath ?? `cursor-denied-${nonce}.txt`;
let deniedPath = remote ? "" : join(input.workspacePath, deniedRelative);
let deniedCommand = remote ? null : cursorDeniedCommand(deniedPath);
let denialNotices: CursorToolNotice[] = []; let denialRunEvents: Row[] = []; let denialTurnId = ""; let cancelRequestedAt = NaN; let cancellationProven = false;
let denialNotices: CursorToolNotice[] = []; let denialRunEvents: Row[] = []; let denialTurnId = ""; let denialTerminalProven = false;
const samples: Array<{ phase: string; path: string; absent: boolean; observedAt: number }> = [];
const sampleDenied = async (phase: string, retained?: CursorRemoteSnapshot) => {
let sample: { phase: string; path: string; absent: boolean; observedAt: number };
@@ -177,7 +177,7 @@ export async function runCursorNativeFlow(input: {
processes = await pollUntil({ label: "observed Cursor provider retirement", deadlineAt: Date.now() + 5_000,
load: async () => observeProcesses(), accept: observation => observation.live.length === 0 });
}
const settled = runs.length === 1 && runs[0]!.status === "cancelled" && issue.status === "in_progress" && cancellationProven;
const settled = runs.length === 1 && denialTerminalProven && hasCursorDeniedTurnTerminal({ run: runs[0], issue, events: denialRunEvents, runId: runs[0]!.id, turnId: denialTurnId, requestId: deniedRequest?.requestId ?? "" });
finalCheck("authoritative-provider-cleanup", settled && !processObservationError && processes.captured && processes.live.length === 0, "Exact API-bound per-turn process/start/group and observed descendants have retired");
const finalSample = { phase: "after-cleanup", path: deniedPath, absent: await absent(deniedPath), observedAt: Date.now() };
samples.push(finalSample);
@@ -185,12 +185,12 @@ export async function runCursorNativeFlow(input: {
const journal = watch.finish();
finalCheck("continuous-denial-observation", journal.complete && journal.targetMutationCount === 0, "Continuous target watcher observed no create/delete mutation and retained directory identity");
finalCheck("complete-native-denial-boundary", Boolean(deniedRequest) && hasCursorDenialBoundary({ request: deniedRequest, expectedRequestId: deniedRequest?.requestId ?? "", expectedToolCallId: deniedRequest?.details.toolCallId ?? "", path: deniedPath, bootstrapReadProof: remoteFixture ? { actionFile: remoteFixture.actionFile, events: denialRunEvents } : undefined, samples, notices: denialNotices, runId: runs[0]?.id ?? "", turnId: denialTurnId }), "Supported native denial preserved the target through all six independent boundaries");
await input.evidence("cursor-native-denial-final.json", { request: deniedRequest, samples, notices: denialNotices, commandSha256: deniedCommand!.commandSha256, cancellationProven, cancelRequestedAt, processes, processObservationError, watcher: journal, checks: cleanupChecks });
await input.evidence("cursor-native-denial-final.json", { request: deniedRequest, samples, notices: denialNotices, commandSha256: deniedCommand!.commandSha256, denialTerminalProven, processes, processObservationError, watcher: journal, checks: cleanupChecks });
if (cleanupChecks.some(row => !row.passed)) throw new Error("Cursor native denial cleanup proof is incomplete or observed an effect");
return cleanupChecks;
} finally {
clearInterval(processTimer);
await input.evidence("cursor-native-denial-cleanup-attempt.json", { request: deniedRequest, samples, notices: denialNotices, commandSha256: deniedCommand!.commandSha256, cancellationProven, cancelRequestedAt, processes, processObservationError, watcher: watch.finish(), checks: cleanupChecks });
await input.evidence("cursor-native-denial-cleanup-attempt.json", { request: deniedRequest, samples, notices: denialNotices, commandSha256: deniedCommand!.commandSha256, denialTerminalProven, processes, processObservationError, watcher: watch.finish(), checks: cleanupChecks });
}
});
}
@@ -204,7 +204,7 @@ export async function runCursorNativeFlow(input: {
finalCheck("remote-provider-retired", runs.length === 1 && remoteBaseline !== null && snapshot.observedAtMs >= remoteBaseline.observedAtMs && hasCursorRemoteRetirement(snapshot, remoteFixture.binding), "Exact remote run PID/start/boot identity and all observed descendants retired before the retained receipt sealed");
if (design.id === "native-write-deny-reconnect") {
await sampleDenied("after-cleanup", snapshot);
finalCheck("remote-cancel-terminal", cancellationProven && runs[0]?.status === "cancelled" && issue.status === "in_progress", "Explicit native cancellation remains durable without false task completion");
finalCheck("remote-denial-terminal", denialTerminalProven && runs.length === 1 && hasCursorDeniedTurnTerminal({ run: runs[0], issue, events: denialRunEvents, runId: runs[0]!.id, turnId: denialTurnId, requestId: deniedRequest?.requestId ?? "" }), "Denied native turn remains failed and unfinished without false semantic success");
finalCheck("remote-no-denied-effect", snapshot.watcher.targetMutationCount === 0 && Boolean(deniedRequest) && hasCursorDenialBoundary({ request: deniedRequest, expectedRequestId: deniedRequest?.requestId ?? "", expectedToolCallId: deniedRequest?.details.toolCallId ?? "", path: deniedPath, bootstrapReadProof: remoteFixture ? { actionFile: remoteFixture.actionFile, events: denialRunEvents } : undefined, samples, notices: denialNotices, runId: runs[0]?.id ?? "", turnId: denialTurnId }), "Exact denied native command caused no remote file effect through provider retirement");
} else {
if (design.id === "native-plan-reject-revise-accept") finalCheck("remote-plan-still-passive", hasCursorAcceptedPlanWait(await load()), "Accepted planning remains passive through remote retirement without an automatic follow-up run");
@@ -213,7 +213,7 @@ export async function runCursorNativeFlow(input: {
if (cleanupChecks.some(row => !row.passed)) throw new Error("Cursor remote cleanup evidence is incomplete or observed an effect");
return cleanupChecks;
} finally {
try { await input.evidence("cursor-remote-cleanup.json", { snapshot, samples, notices: denialNotices, cancellationProven, checks: cleanupChecks }); }
try { await input.evidence("cursor-remote-cleanup.json", { snapshot, samples, notices: denialNotices, denialTerminalProven, checks: cleanupChecks }); }
finally { await remoteFixture?.close(); }
}
});
@@ -263,8 +263,8 @@ export async function runCursorNativeFlow(input: {
let expectedMarker = execution.task.buildVisibleMarker(nonce);
try {
if (!remote && design.id === "native-write-deny-reconnect") await sampleDenied("before-request");
await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), prompt: remote ? input.remoteBootstrap!.prompt(nonce) : execution.task.buildPrompt(nonce) + (watch ? `\nExact native shell command (copy verbatim):\n${deniedCommand!.command}` : ""), workMode: "standard", projectName: project.name });
issue = await pollUntil({ label: "browser-created Cursor task", deadlineAt: input.deadlineAt, load: async () => (await api.get<Row[]>(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(row => row.title === execution.task.buildTitle(nonce)), accept: Boolean }) ?? {};
const createdTask = await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), prompt: remote ? input.remoteBootstrap!.prompt(nonce) : execution.task.buildPrompt(nonce) + (watch ? `\nExact native shell command (copy verbatim):\n${deniedCommand!.command}` : ""), workMode: "standard", projectName: project.name, requireExplicitTitle: design.id === "native-write-deny-reconnect" });
issue = await pollUntil({ label: "browser-created Cursor task", deadlineAt: input.deadlineAt, load: async () => (await api.get<Row[]>(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(row => row.id === createdTask.issueId), accept: Boolean }) ?? {};
if (!issue.id) throw new Error("Browser-created Cursor task is absent");
if (remote) {
const started = await pollUntil({ label: "exact Cursor bootstrap run", deadlineAt: input.deadlineAt, load, reject,
@@ -342,22 +342,25 @@ export async function runCursorNativeFlow(input: {
await card.getByRole("button", { name: label, exact: true }).click(); const posted = (await sent).postDataJSON();
check("browser-exact-denial", posted.turnId === event.turnId && posted.requestKind === "permission_approval" && posted.resolution?.action === "decline", "Browser denied the exact native run/request/turn");
const identity = { runId: native.runId, turnId: event.turnId, requestId: request.requestId, method: "session/request_permission" as const, action: "decline" as const };
await pollUntil({ label: "native denial delivered and exact command failed", deadlineAt: input.deadlineAt, load, reject, accept: state => hasDeliveredCursorNativeRequest({ ...identity, events: state.runEvents })
&& hasCursorDeniedCommand({ notices: denialNotices, ...identity, toolCallId: native.toolCallId, commandSha256: deniedCommand!.commandSha256, bootstrapReadProof: remoteFixture ? { actionFile: remoteFixture.actionFile, events: denialRunEvents } : undefined }) });
const rejectDenial = (state: Awaited<ReturnType<typeof load>>) => state.runs.length !== 1
|| ["succeeded", "cancelled", "timed_out"].includes(state.runs[0]?.status)
|| (state.issue.status !== "in_progress" && !(state.issue.status === "blocked" && state.runs[0]?.status === "failed" && state.runs[0]?.errorCode === "native_permission_declined"))
? "Denied Cursor operation claimed success, changed task disposition, or created another run" : undefined;
await pollUntil({ label: "exact native denial delivery and settled call", deadlineAt: input.deadlineAt, load, reject: rejectDenial,
accept: state => hasDeliveredCursorNativeRequest({ ...identity, events: state.runEvents })
&& hasCursorDeniedCommand({ notices: denialNotices, ...identity, toolCallId: native.toolCallId, commandSha256: deniedCommand!.commandSha256, bootstrapReadProof: remoteFixture ? { actionFile: remoteFixture.actionFile, events: denialRunEvents } : undefined }) });
await sampleDenied("after-decision");
cancelRequestedAt = Date.now(); await api.post(`/api/heartbeat-runs/${native.runId}/cancel`);
const cancelled = await pollUntil({ label: "explicit native cancellation", deadlineAt: input.deadlineAt, load,
reject: state => state.runs.length !== 1 || ["failed", "succeeded", "timed_out"].includes(state.runs[0]?.status) ? "Cursor denial did not remain cancellable" : undefined,
accept: state => hasCursorCancellation({ run: state.runs[0], issue: state.issue, events: state.runEvents, runId: native.runId, turnId: event.turnId, requestedAt: cancelRequestedAt }) });
cancellationProven = true;
const terminal = await pollUntil({ label: "denied native turn remains unfinished", deadlineAt: input.deadlineAt, load, reject: rejectDenial,
accept: state => hasCursorDeniedTurnTerminal({ run: state.runs[0], issue: state.issue, events: state.runEvents, ...identity }) });
denialTerminalProven = true;
if (remote) remoteFinal = await remoteFixture!.finish();
await sampleDenied("after-terminal", remoteFinal ?? undefined);
check("negative-task-unfinished", cancelled.issue.status === "in_progress" && cancelled.runs[0]?.status === "cancelled" && cancelled.runs[0]?.runtimeMode === "native", "Native cancellation was acknowledged and the task does not falsely claim completion");
check("negative-task-unfinished", hasCursorDeniedTurnTerminal({ run: terminal.runs[0], issue: terminal.issue, events: terminal.runEvents, ...identity }), "Denied native turn supplied no semantic completion; its failed run does not falsely complete the task");
await page.reload();
await expect(page.getByTestId("issue-detail-header").getByRole("button", { name: "Change status (current: In Progress)", exact: true })).toBeVisible();
await expect(page.getByTestId("issue-detail-header").getByRole("button", { name: terminal.issue.status === "blocked" ? /^Change status \(current: Blocked(?: · .+)?\)$/u : "Change status (current: In Progress)" })).toBeVisible();
const comments = await api.get<Row[]>(`/api/issues/${issue.id}/comments`);
await input.evidence("api-state.json", { ...cancelled, run: runs[0], comments, checks, notices: denialNotices, commandSha256: deniedCommand!.commandSha256, cancelRequestedAt, runEventsByRun: [{ runId: native.runId, events: cancelled.runEvents }] });
await input.capture("final-state", "Cursor denied command cancelled; task remains unfinished", "final-state.png");
await input.evidence("api-state.json", { ...terminal, run: runs[0], comments, checks, notices: denialNotices, commandSha256: deniedCommand!.commandSha256, denialTerminalProven, runEventsByRun: [{ runId: native.runId, events: terminal.runEvents }] });
await input.capture("final-state", "Cursor denied command ended without completing the task", "final-state.png");
return { issue, runs, checks };
}
if (design.id === "native-plan-reject-revise-accept") {
tests/runner-e2e/dashboard.ts
+2
View File
@@ -286,6 +286,7 @@ function renderCase(
<div><span>Tokens</span><strong>${html(tokenLabel(billing.llm.inputTokens, billing.llm))} in · ${html(tokenLabel(billing.llm.outputTokens, billing.llm))} out</strong><small>${html(tokenLabel(billing.llm.cachedInputTokens, billing.llm))} cached · ${billing.llm.runsWithTokenUsage}/${billing.llm.runCount} runs covered</small></div>
<div><span>LLM spend</span><strong>${html(llmCostLabel(billing.reportedCostUsd, billing.llm))}</strong><small>${billing.llm.runsWithReportedCost}/${billing.llm.runCount} runs provider-priced</small></div>
<div><span>Execution</span><strong>${billing.runtime.estimatedListCostUsd === undefined ? html(billing.runtime.costStatus === "not_metered" ? "Local · not metered" : "Cost unavailable") : `${html(usdLabel(billing.runtime.estimatedListCostUsd))} est.`}</strong><small>${html(durationLabel(billing.runtime.agentRunDurationMs))} agent${billing.runtime.leaseDurationMs === null ? "" : ` · ${html(durationLabel(billing.runtime.leaseDurationMs))} lease`}</small></div>
${billing.assistant ? `<div><span>External assistant</span><strong>${html(usdLabel(billing.assistant.estimatedCostUsd))} est.</strong><small>${html(billing.assistant.model)} · ${billing.assistant.requests} requests · ${html(tokenLabel(billing.assistant.inputTokens + billing.assistant.cachedInputTokens))} in / ${html(tokenLabel(billing.assistant.outputTokens))} out</small></div>` : ""}
</div>`
: "";
return `<article
@@ -1131,6 +1132,7 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
<div class="billing-metric"><strong>${html(llmCostLabel(campaignBilling.reportedLlmCostUsd, campaignBilling.llm))}</strong><span>LLM reported subtotal</span></div>
<div class="billing-metric"><strong>${html(usdLabel(campaignBilling.estimatedRuntimeCostUsd))}</strong><span>Daytona list estimate</span></div>
${campaignBilling.judge ? `<div class="billing-metric"><strong>${html(usdLabel(campaignBilling.judge.estimatedCostUsd))}</strong><span>Judge estimate · ${campaignBilling.judge.attempts} attempts · ${campaignBilling.judge.attemptsWithUnknownUsage} unknown usage</span><small>${html(tokenLabel(campaignBilling.judge.inputTokens))} in / ${html(tokenLabel(campaignBilling.judge.outputTokens))} out · $${campaignBilling.judge.reservedCostUsd.toFixed(6)} reserved</small></div>` : ""}
${campaignBilling.assistant ? `<div class="billing-metric"><strong>${html(usdLabel(campaignBilling.assistant.estimatedCostUsd))}</strong><span>External assistant estimate · ${campaignBilling.assistant.requests} requests</span><small>${html(tokenLabel(campaignBilling.assistant.inputTokens + campaignBilling.assistant.cachedInputTokens))} in / ${html(tokenLabel(campaignBilling.assistant.outputTokens))} out</small></div>` : ""}
<div class="billing-metric"><strong>${html(durationLabel(campaignBilling.agentRunDurationMs))}</strong><span>Agent execution time</span></div>
<div class="billing-metric"><strong>${html(durationLabel(campaignBilling.leaseDurationMs))}</strong><span>Daytona lease time</span></div>
<div class="billing-metric"><strong>${campaignBilling.llm.runsWithReportedCost}/${campaignBilling.llm.runCount}</strong><span>Runs provider-priced</span></div>
tests/runner-e2e/daytona-image-content.ts
+7 -4
View File
@@ -30,23 +30,26 @@ export const DAYTONA_IMAGE_INPUT_PATHS = [
"packages/paperclip-runner/scripts/provision-grok.mjs",
"packages/paperclip-runner/package.json",
"packages/paperclip-runner/cursor-distributions.json",
"packages/paperclip-runner/cursor-contract.json",
"packages/paperclip-runner/protocol",
"packages/paperclip-runner/runner/Cargo.lock",
"packages/paperclip-runner/runner/Cargo.toml",
"packages/paperclip-runner/runner/crates",
"packages/paperclip-runner/scripts/acpx-sidecar-contract.mjs",
"packages/paperclip-runner/scripts/build-provider-pack.mjs",
"packages/paperclip-runner/scripts/provider-pack-executable-shims.mjs",
"packages/paperclip-runner/scripts/build-node-startup-timeout.mjs",
"packages/paperclip-runner/scripts/candidate-provider-pack.mjs",
"packages/paperclip-runner/scripts/build-copilot-distribution.mjs",
"packages/paperclip-runner/scripts/materialize-copilot-binary.mjs",
"packages/paperclip-runner/scripts/materialize-cursor-distribution.mjs",
"packages/paperclip-runner/scripts/cursor-runtime-patch.mjs",
"packages/paperclip-runner/scripts/cursor-native-usage.mjs",
"packages/paperclip-runner/scripts/cursor-native-usage-source.json",
"packages/paperclip-runner/scripts/provision-cursor.mjs",
"packages/paperclip-runner/cursor-distributions.json",
"packages/paperclip-runner/cursor-contract.json",
"packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs",
"packages/paperclip-runner/scripts/generate-acpx-sidecar-contract.mjs",
"packages/paperclip-runner/scripts/generate-protocol-schema-module.mjs",
"packages/paperclip-runner/scripts/materialize-pi-distribution.mjs",
"packages/paperclip-runner/scripts/pi-distribution",
// Pi runtime/extension/Node/closure pins are included by the src tree below.
"packages/paperclip-runner/src",
"packages/paperclip-runner/styles.css",
tests/runner-e2e/daytona-image.test.ts
+5 -9
View File
@@ -200,8 +200,6 @@ describe("runner E2E Daytona image contract", () => {
"packages/paperclip-eval-kernel/src",
"packages/paperclip-runner/package.json",
"packages/paperclip-runner/scripts/candidate-provider-pack.mjs",
"packages/paperclip-runner/scripts/build-copilot-distribution.mjs",
"packages/paperclip-runner/scripts/materialize-copilot-binary.mjs",
"packages/paperclip-runner/scripts/materialize-cursor-distribution.mjs",
"packages/paperclip-runner/scripts/cursor-runtime-patch.mjs",
"packages/paperclip-runner/cursor-distributions.json",
@@ -284,11 +282,7 @@ describe("runner E2E Daytona image contract", () => {
),
'pub const VERSION: &str = "one";\n',
);
await mkdir(path.join(root, "packages/paperclip-runner/scripts/pi-distribution"), { recursive: true });
for (const relativePath of [
"packages/paperclip-runner/scripts/materialize-pi-distribution.mjs",
"packages/paperclip-runner/scripts/pi-distribution/package.json",
"packages/paperclip-runner/scripts/pi-distribution/package-lock.json",
]) await writeFile(path.join(root, relativePath), "version one\n");
const baseline = await computeDaytonaImageContentId(options);
const candidate = await computeDaytonaImageContentId({ ...options, candidateProviders: ["pi"] });
@@ -347,9 +341,11 @@ describe("runner E2E Daytona image contract", () => {
}
});
it("invalidates the image when the Cursor runtime isolation patch changes", async () => {
const root = await mkdtemp(path.join(tmpdir(), "paperclip-cursor-image-id-"));
const patchPath = "packages/paperclip-runner/scripts/cursor-runtime-patch.mjs";
it.each([
"packages/paperclip-runner/scripts/cursor-runtime-patch.mjs",
"packages/paperclip-runner/scripts/provider-pack-executable-shims.mjs",
])("invalidates the image when %s changes", async (patchPath) => {
const root = await mkdtemp(path.join(tmpdir(), "paperclip-provider-image-id-"));
expect(DAYTONA_IMAGE_INPUT_PATHS).toContain(patchPath);
const options = {
repositoryRoot: root,
tests/runner-e2e/everyday-cases.ts
+1
View File
@@ -32,6 +32,7 @@ export const SLUGIFY_REVISION = `Add an optional --separator argument that accep
export const LATE_REQUIREMENT = `Also support --max-length as a positive integer. Truncate the final slug to that length, then remove any trailing separator. In particular, input " Queue--Ready!! " with --max-length 7 must print "queue-r". Preserve the default behavior.`;
const definitions = [
["agentmail-setup", "Request an AgentMail inbox in the thread", "I'd like you to acquire an email address with AgentMail and tell me the address once it exists. If I choose Not now, explain that no inbox was created and finish without connecting.", 2],
["provider-decline", "Decline external providers", "Connect HubSpot so you can read my recent contacts. If I choose None for now, explain that you could not retrieve the contacts and finish without connecting.", 2],
["provider-second", "Choose and reuse the second external provider", "Connect HubSpot and read my recent contacts. Report the contact names and verification code returned by the service.", 2],
["provider-native", "Prefer a built-in connection over external providers", "Connect Jira so you can read my recent issues. If I choose Not now, explain that you could not retrieve the issues and finish without connecting.", 2],
tests/runner-e2e/everyday-flow.ts
+132 -31
View File
@@ -1,3 +1,6 @@
import { gradeAgentmailSetup } from "./agentmail-setup-evidence.js";
import { CONNECTION_GUIDANCE_SUITE, CONNECTION_GUIDANCE_BUDGET_CENTS } from "./connection-guidance-cases.js";
import { gradeConnectionGuidanceDecline, hasConnectionGuidanceDeclineReply, explainsConnectionUnavailable, type ConnectionGuidanceDeclineInput } from "./connection-guidance-evidence.js";
import { expect, type Page } from "@playwright/test";
import { runnerApiToolsEnabled } from "../../server/src/services/native-runtime/runner-api-rollout.js";
import { spawn } from "node:child_process";
@@ -12,11 +15,12 @@ import {
} from "./everyday-delivery.js";
import type { LiveFixtureValues } from "./live-fixtures.js";
import type { MatrixExecution } from "./types.js";
import { createTaskThroughUi, submitTaskReply } from "./user-actions.js";
import { waitForTaskChatRendered } from "./continuation-screenshot.js";
import { submitTaskReply } from "./user-actions.js";
import { createTaskFromPromptThroughUi } from "./plan-task-ui.js";
import { waitForTaskChatRendered, waitForTaskIdentity } from "./continuation-screenshot.js";
import { hasPersistedSource, isSavedSourceCheckpoint } from "./everyday-interruption.js";
import { setupAggregatorFixture } from "./aggregator-fixture.js";
import { gradeProviderChoice, gradeProviderOutcome } from "./connection-routing-evidence.js";
import { gradeProviderChoice, gradeProviderOutcome, requireProviderAccessCard } from "./connection-routing-evidence.js";
import { setupConnectionReview } from "./connection-reviews.js";
import {
pendingStoryDecision,
@@ -37,6 +41,7 @@ import {
storyReviewContinuationTimeoutDetail,
storyHasStrandedBlockedLeaf,
storyHasDurableAgentReviewContinuation,
storyHasDurableServiceContinuation,
storyIssueHasBlockedTimelineBefore,
storyIssueHasUnresolvedDependency,
storyRunReportsDependencyBlock,
@@ -60,8 +65,10 @@ export interface EverydayEvidence {
fixtureConfiguration?: {
apiToolsEnabled: boolean;
aiConnection?: LiveFixtureValues["aiConnection"];
connectionGuidanceBudgets?: { companyMonthlyCents: unknown; agentMonthlyCents: unknown };
};
documents?: Row[];
declineGradeEvidence?: { capturedAt: string; input: ConnectionGuidanceDeclineInput; checks: StoryCheck[] };
checks: StoryCheck[];
timeline: Array<{ at: string; action: string; detail?: unknown }>;
issues: Array<StoryIssue & Row>;
@@ -161,11 +168,13 @@ export async function runEverydayFlow(input: Input) {
const providerChoice = caseId === "provider-decline" || caseId === "provider-second";
const nativeProviderCase = caseId === "provider-native";
let aggregatorFixture: Awaited<ReturnType<typeof setupAggregatorFixture>> | undefined;
const decliningConnection = caseId === "connection-decline" || nativeProviderCase;
const agentmailSetup = caseId === "agentmail-setup";
const decliningConnection = caseId === "connection-decline" || nativeProviderCase || agentmailSetup;
const declining = decliningConnection || caseId === "service-decline";
let decisionId: string | undefined;
let decisionResolvedAt: string | undefined;
let initialConnections: string[] = [];
let providerAccessDecision: { id: string; connectionId: string } | undefined;
let stoppedWorkspace: Record<string, string> | undefined;
let settledAgentReply: Row | undefined;
let reviewHandoffBoundary: {
@@ -239,11 +248,11 @@ export async function runEverydayFlow(input: Input) {
`/${prefix}/issues/${issue.identifier ?? issue.id}`;
async function openTask(issue: StoryIssue) {
await page.goto(taskUrl(issue), { waitUntil: "domcontentloaded" });
await waitForTaskIdentity(page, taskUrl(issue), issue.identifier ?? issue.id);
if (providerChoice || nativeProviderCase) {
await expect(page.locator('[data-testid="task-chat-thread"], [data-testid="thread-root"]').first()).toBeVisible({timeout:30_000});
await expect(page.getByRole("heading", {name:String(issue.title),exact:true})).toBeVisible();
await expect(page.getByTestId("issue-chat-skeleton")).toHaveCount(0);
} else await waitForTaskChatRendered(page, String(issue.title));
} else await waitForTaskChatRendered(page);
}
async function openParent() {
await openTask(parent!);
@@ -293,15 +302,28 @@ export async function runEverydayFlow(input: Input) {
commentIds: added.map((c) => c.id),
});
}
function declineInput(state: Pick<EverydayEvidence, "issues" | "runs">): ConnectionGuidanceDeclineInput | undefined {
if (execution.suite.id !== CONNECTION_GUIDANCE_SUITE ||
(caseId !== "service-decline" && caseId !== "connection-decline" && caseId !== "provider-decline")) return;
const issue = state.issues.find(i => i.id === parent?.id);
return {
caseId, decisionId: decisionId ?? "", decisions: issue?.interactions as any ?? [],
leadAgentId: fixtures.agent.id, issueId: parent?.id ?? "", replies: issue?.comments ?? [],
runs: state.runs, marker: (caseId === "provider-decline" ? "CONTACTS_" : "SERVICE_") + nonce,
// Readiness never grades these external observations. They are filled at the grading boundary below.
sameConnections: false,
};
}
async function settled(expectedAgentReply?: string) {
const settledState = await pollUntil({
label: `everyday ${caseId} settled`,
deadlineAt: input.deadlineAt,
timeoutDetail: (state) => state &&
timeoutDetail: (state) => state && (
storyReviewContinuationTimeoutDetail(
state.issues, parent?.id ?? "", fixtures.agent.id, state.runs,
observableAgentIds(state),
),
) ?? (storyHasDurableServiceContinuation(state.issues, parent?.id ?? "", fixtures.agent.id, state.runs)
? "task is Blocked without an active continuation after resolved service approval" : undefined)),
intervalMs: 1000,
load: refresh,
accept: (state) =>
@@ -315,6 +337,7 @@ export async function runEverydayFlow(input: Input) {
state.runs.some(
(r) => Date.parse(r.finishedAt ?? "") >= lastSubmissionAt,
) &&
(!declineInput(state) || hasConnectionGuidanceDeclineReply(declineInput(state)!)) &&
(!expectedAgentReply ||
storyHasAgentReply(
state.issues.find((issue) => issue.id === parent?.id),
@@ -339,7 +362,8 @@ export async function runEverydayFlow(input: Input) {
parent?.id ?? "",
fixtures.agent.id,
state.runs,
)
) &&
!storyHasDurableServiceContinuation(state.issues, parent?.id ?? "", fixtures.agent.id, state.runs)
)
return "task is Blocked without an active continuation";
if (
@@ -549,6 +573,7 @@ export async function runEverydayFlow(input: Input) {
"everyday-decisions.ts",
"aggregator-fixture.ts",
"connection-routing-evidence.ts",
"agentmail-setup-evidence.ts",
"everyday-delivery.ts",
"everyday-observations.ts",
"everyday-artifact.py",
@@ -561,6 +586,20 @@ export async function runEverydayFlow(input: Input) {
"connection-reviews.ts",
"catalog.ts",
];
if (execution.suite.id === CONNECTION_GUIDANCE_SUITE) {
harnessFiles.push("connection-guidance-cases.ts", "connection-guidance-evidence.ts");
const [company, agent] = await Promise.all([
api.get<Row>("/api/companies/" + fixtures.company.id),
api.get<Row>("/api/agents/" + fixtures.agent.id),
]);
ev.fixtureConfiguration!.connectionGuidanceBudgets = {
companyMonthlyCents: company.budgetMonthlyCents, agentMonthlyCents: agent.budgetMonthlyCents,
};
const budgetsMatch = company.budgetMonthlyCents === CONNECTION_GUIDANCE_BUDGET_CENTS &&
agent.budgetMonthlyCents === CONNECTION_GUIDANCE_BUDGET_CENTS;
check("guidance-budget-hard-stops", budgetsMatch, "Public company and lead records retain both 1,000-cent hard stops before task creation.");
if (!budgetsMatch) throw new Error("Connection guidance budget admission failed before task creation");
}
ev.harnessDigest = createHash("sha256")
.update(
(
@@ -645,11 +684,17 @@ export async function runEverydayFlow(input: Input) {
marker: `Pages: Roadmap, Meeting notes. Verification code: SERVICE_${nonce}`,
authenticated: true,
});
if (providerChoice || nativeProviderCase) {
if (caseId === "provider-second") aggregatorFixture = await setupAggregatorFixture(api, fixtures.company.id, fixtures.agent.id, `CONTACTS_${nonce}`);
if (providerChoice || nativeProviderCase || (execution.suite.id === CONNECTION_GUIDANCE_SUITE && Boolean(review))) {
if (caseId === "provider-second" || (execution.suite.id === CONNECTION_GUIDANCE_SUITE && caseId === "provider-decline")) aggregatorFixture = await setupAggregatorFixture(api, fixtures.company.id, fixtures.agent.id, `CONTACTS_${nonce}`, execution.suite.id === CONNECTION_GUIDANCE_SUITE);
const state = await api.get<{connections:Row[]}>(`/api/companies/${fixtures.company.id}/tools/connections`);
initialConnections = state.connections.map(c=>c.id);
if (aggregatorFixture && execution.suite.id === CONNECTION_GUIDANCE_SUITE) {
await input.evidence("provider-initial-access.json", aggregatorFixture.initialAccess);
check("provider-not-installed-before-choice", !aggregatorFixture.initialAccess.installed && aggregatorFixture.initialAccess.allowedToolIds.length === 0,
"The company connection is configured, but this agent has no installed or allowed HubSpot tool before the decision.");
}
}
if (agentmailSetup) await api.patch("/api/instance/settings/experimental", { enableChatConnectors: true });
if (decliningConnection) {
const state = await api.get<{ connections: Row[] }>(
`/api/companies/${fixtures.company.id}/tools/connections`,
@@ -663,25 +708,15 @@ export async function runEverydayFlow(input: Input) {
if (state.connections.length)
throw new Error("New-connection story requires an unconnected company");
}
await createTaskThroughUi({
const created = await createTaskFromPromptThroughUi({
page,
companyId: fixtures.company.id,
issuePrefix: prefix,
agentName: fixtures.agent.name,
title: execution.task.buildTitle(nonce),
prompt: ev.prompt,
workMode: "standard",
projectName: project?.name,
});
parent = await pollUntil({
label: "browser-created story task",
deadlineAt: Date.now() + 30_000,
load: () =>
api.get<StoryIssue[]>(`/api/companies/${fixtures.company.id}/issues`),
accept: (rows) =>
rows.some((i) => i.title === execution.task.buildTitle(nonce)),
}).then((rows) =>
rows.find((i) => i.title === execution.task.buildTitle(nonce))!,
);
parent = await api.get<StoryIssue>(`/api/issues/${created.id}`);
input.observe(parent!, []);
note("task-submitted", { issueId: parent!.id });
await openParent();
@@ -916,6 +951,37 @@ export async function runEverydayFlow(input: Input) {
accept:rows=>rows.some(row=>row.id===decisionId && row.status==="answered"),
});
note("provider-choice-submitted", {interactionId:decisionId, selected:caseId === "provider-decline" ? "none" : "via:arcade:hubspot"});
if (execution.suite.id === CONNECTION_GUIDANCE_SUITE && caseId === "provider-second") {
const rows = await pollUntil({
label: "selected Arcade access request", deadlineAt: input.deadlineAt,
load: async () => ({
rows: await api.get<Row[]>(`/api/issues/${parent!.id}/interactions`),
issue: await api.get<StoryIssue>(`/api/issues/${parent!.id}`),
calls: aggregatorFixture!.invocationCount(),
runs: await api.get<StoryRun[]>(`/api/issues/${parent!.id}/runs`),
}),
accept: state => state.rows.some(row => row.id !== decisionId && row.status === "pending"),
reject: state => state.calls > 0 ? "Provider ran before tool-access approval"
: state.runs.some(run => ["failed", "timed_out"].includes(run.status)) ? "Agent failed before tool-access approval"
: ["done", "blocked", "cancelled"].includes(state.issue.status) ? "Task ended without the required access request" : undefined,
});
const card = requireProviderAccessCard({ rows: rows.rows as any, decisionId: decisionId!, connectionId: aggregatorFixture!.connectionId,
agentId: fixtures.agent.id, catalogEntryIds: aggregatorFixture!.catalogEntryIds, calls: rows.calls });
providerAccessDecision = { id: card.id, connectionId: aggregatorFixture!.connectionId };
await openParent();
const accessCard = page.getByTestId("connection-intent-access-request");
await expect(accessCard).toHaveCount(1);
await expect(accessCard.getByText("Hubspot_ListContacts", { exact: true })).toBeVisible();
await input.capture("provider-access", "Separate HubSpot tool-access approval", "provider-access.png");
check("no-call-before-provider-access", aggregatorFixture!.invocationCount() === 0, "Selecting a provider alone did not expose or execute its tool.");
await accessCard.getByRole("button", { name: "Grant access", exact: true }).click();
await pollUntil({ label: "Arcade access grant saved", deadlineAt: input.deadlineAt,
load: () => api.get<Row[]>(`/api/issues/${parent!.id}/interactions`),
accept: rows => rows.some(row => row.id === card.id && row.status === "accepted"
&& row.result?.connectionId === aggregatorFixture!.connectionId),
});
note("provider-access-granted", { interactionId: card.id, connectionId: aggregatorFixture!.connectionId });
}
}
if (review || decliningConnection) {
const interactions = await pollUntil({
@@ -952,7 +1018,7 @@ export async function runEverydayFlow(input: Input) {
interactions.interactions,
review
? { kind: "tool", connectionId: review.connectionId }
: { kind: "connection", serviceSlug: nativeProviderCase ? "jira" : "notion" },
: { kind: "connection", serviceSlug: agentmailSetup ? "agentmail" : nativeProviderCase ? "jira" : "notion" },
);
await expect(
page.getByRole("button", {
@@ -971,8 +1037,28 @@ export async function runEverydayFlow(input: Input) {
true,
review
? "Tool approval belongs to the installed page service."
: `New connection request is for ${nativeProviderCase ? "Jira" : "Notion"}, without an external-provider question.`,
: `New connection request is for ${agentmailSetup ? "AgentMail" : nativeProviderCase ? "Jira" : "Notion"}, without an external-provider question.`,
);
if (agentmailSetup) {
// Reload proves the card is durable, rather than a transient model UI.
await page.reload();
const form = page.getByTestId("agentmail-inline-setup");
await expect(form).toBeVisible();
const card = page.getByTestId("connection-intent-focus-target").filter({ has: form });
const rows = await api.get<Parameters<typeof gradeAgentmailSetup>[0]["interactions"]>(`/api/issues/${parent!.id}/interactions`);
const checks = gradeAgentmailSetup({
// This harness boots a dedicated local_trusted instance.
interactions: rows, agentId: fixtures.agent.id, userId: "local-board",
visible: await form.isVisible(),
inputTypes: await card.locator("input").evaluateAll(inputs => inputs.map(input => input.getAttribute("type") ?? "text")),
keyLink: await form.getByRole("link", { name: "Get an AgentMail API key" }).getAttribute("href"),
accessSelectorCount: await card.locator('[role="radiogroup"], [role="combobox"], select').count(),
dialogCount: await page.getByRole("dialog").count(),
});
ev.checks.push(...checks);
await input.capture("agentmail-inline-key", "AgentMail API-key card after reload", "agentmail-inline-key.png");
if (checks.some(result => !result.passed)) throw new Error("AgentMail inline setup evidence failed");
}
if (review)
check(
"no-call-before-approval",
@@ -1095,12 +1181,26 @@ export async function runEverydayFlow(input: Input) {
if (providerChoice) {
const issue = ev.issues.find(i=>i.id===parent!.id)!;
const state = await api.get<{connections:Row[]}>(`/api/companies/${fixtures.company.id}/tools/connections`);
ev.checks.push(...gradeProviderOutcome({rows:issue.interactions as any, decisionId:decisionId!,
ev.checks.push(...gradeProviderOutcome({rows:issue.interactions as any, decisionId:decisionId!, accessDecision:providerAccessDecision,
selected:caseId === "provider-decline" ? "none" : "via:arcade:hubspot", calls:aggregatorFixture?.invocationCount() ?? 0,
response: (issue.comments ?? []).filter((c:Row)=>c.authorAgentId).map((c:Row)=>c.body).join("\n"), marker:`CONTACTS_${nonce}`,
sameConnections:isDeepStrictEqual(state.connections.map(c=>c.id).sort(), initialConnections.sort()),
}));
}
if (execution.suite.id === CONNECTION_GUIDANCE_SUITE &&
(caseId === "service-decline" || caseId === "connection-decline" || caseId === "provider-decline")) {
const state = await api.get<{ connections: Row[] }>("/api/companies/" + fixtures.company.id + "/tools/connections");
const gradeInput = structuredClone({
...declineInput(ev)!,
calls: review?.invocationCount() ?? aggregatorFixture?.invocationCount(),
sameConnections: isDeepStrictEqual(state.connections.map(c => c.id).sort(), initialConnections.sort()),
});
const checks = gradeConnectionGuidanceDecline(gradeInput);
// Keep the actual assertion input independent of the later final/cleanup refreshes.
ev.declineGradeEvidence = { capturedAt: new Date().toISOString(), input: gradeInput, checks: structuredClone(checks) };
await input.evidence("connection-guidance-decline-grade.json", ev.declineGradeEvidence);
ev.checks.push(...checks);
}
if (declining) {
const issue = ev.issues.find((i) => i.id === parent!.id)!;
const requests = issue.interactions as Row[];
@@ -1120,9 +1220,7 @@ export async function runEverydayFlow(input: Input) {
check(
"decline-visible-explanation",
replies.length > 0 &&
/declin|not now|could(?:n.t| not)|cannot|can.t|unable|not (?:connect|retriev)|without (?:access|connect)/i.test(
text,
),
explainsConnectionUnavailable(text),
"A new agent response explains the missing access after the saved decline.",
);
check(
@@ -1502,7 +1600,10 @@ export async function runEverydayFlow(input: Input) {
"No completion confirmation or unanswered interaction remains.",
);
if (providerChoice || nativeProviderCase) await openParent();
else await waitForTaskChatRendered(page, String(parent!.title));
else {
await waitForTaskIdentity(page, taskUrl(parent!), parent!.identifier ?? parent!.id);
await waitForTaskChatRendered(page);
}
const latestAgentComment = ev.issues
.find((i) => i.id === parent!.id)
?.comments?.filter((c: Row) => c.authorAgentId)
tests/runner-e2e/everyday-observations.ts
+33 -2
View File
@@ -1,3 +1,5 @@
import type { IssueComment } from "../../packages/shared/src/types/issue.js";
export interface StoryCheck {
id: string;
passed: boolean;
@@ -24,9 +26,8 @@ export interface StoryIssue {
wakeDiagnostics?: StoryWakeDiagnostics;
blockedTransitionAt?: string | null;
}
export interface StoryComment {
export interface StoryComment extends Partial<Pick<IssueComment, "authorAgentId" | "createdByRunId">> {
id?: string;
authorAgentId?: string | null;
body?: unknown;
createdAt?: string;
}
@@ -36,6 +37,8 @@ export interface StoryActivityRecord {
createdAt?: string;
}
export interface StoryInteraction {
sourceRunId?: string | null;
createdByAgentId?: string | null;
id?: string;
kind?: string;
issueId?: string | null;
@@ -442,6 +445,34 @@ export function storyHasDurableAgentReviewContinuation(
);
}
/** An executed approval or recorded rejection may enqueue its wake just after run finalization.
* Wait within the original deadline only for this task/run's recorded response.
* Once a follow-up run has consumed it, a new blocker is a real failure.
*/
export function storyHasDurableServiceContinuation(issues: StoryIssue[], parentId: string, agentId: string, runs: StoryRun[]): boolean {
const issue = issues.find(i => i.id === parentId);
if (issue?.status !== "blocked" || issue.assigneeAgentId !== agentId) return false;
return issue.interactions?.some(interaction => {
const source = runs.find(run => run.id === interaction.sourceRunId);
const action = interaction.payload?.toolAction as Record<string, unknown> | undefined;
const result = interaction.result?.toolAction as Record<string, unknown> | undefined;
const resolved = (interaction.status === "accepted" && interaction.result?.outcome === "accepted" && result?.status === "executed") ||
(interaction.status === "rejected" && interaction.result?.outcome === "rejected" && result === undefined);
if (interaction.issueId !== parentId || interaction.createdByAgentId !== agentId ||
interaction.kind !== "request_confirmation" || interaction.continuationPolicy !== "wake_assignee" ||
!resolved ||
action?.version !== 1 || typeof action.actionRequestId !== "string" || !action.actionRequestId.trim() ||
source?.agentId !== agentId || source.companyId !== issue.companyId || source.nativeIssueId !== parentId || source.status !== "succeeded" ||
(!Number.isFinite(Date.parse(source.finishedAt ?? "")) || !Number.isFinite(Date.parse(interaction.resolvedAt ?? "")))) return false;
return !runs.some(run => {
const input = run.runnerProfileJson?.nativeExecutionInput as
{ interactionResponses?: Array<{ interactionId?: string }> } | undefined;
return run.id !== source.id && run.agentId === agentId && run.companyId === issue.companyId &&
input?.interactionResponses?.some(response => response.interactionId === interaction.id);
});
}) ?? false;
}
/** Timeout evidence requires a stranded leaf with durable accepted-review evidence. */
export function storyReviewContinuationTimeoutDetail(
issues: StoryIssue[],
tests/runner-e2e/everyday.test.ts
+1 -1
View File
@@ -212,7 +212,7 @@ describe("manual everyday workflow catalog", () => {
const selected = selectRunnerExecutions(
parseRunnerSelectors(["--suite", "everyday-workflows"]),
);
expect(selected).toHaveLength(47);
expect(selected).toHaveLength(52);
expect(selected.every((e) => e.profile.generation === "native")).toBe(true);
expect(
selectRunnerExecutions(
tests/runner-e2e/extended-harnesses.test.ts
+6 -4
View File
@@ -17,12 +17,12 @@ describe("extended ACP harness qualification", () => {
expect(all.some(cell => cell.suite.id === "extended-harnesses")).toBe(false);
expect(runnerSuites.find(suite => suite.id === "extended-harnesses")?.manualOnly).toBe(true);
});
it("uses exact discovered models, encrypted credential references and pending qualification metadata", () => {
it("uses exact discovered models, encrypted credential references and current qualification metadata", () => {
expect(extendedHarnessProfiles.map(profile => profile.model)).toEqual([
"gpt-5.6-luna[context=272k,reasoning=medium,fast=false]", "gpt-5.6-luna", "openrouter/deepseek/deepseek-v4-flash-0731",
]);
for (const profile of extendedHarnessProfiles) {
expect(profile.modelQualification.source).toBe("candidate_runner_profile");
expect(profile.modelQualification.source).toBe(profile.qualificationCandidate === "cursor" ? "qualified_runner_profile" : "candidate_runner_profile");
const secretRef = { type: "secret_ref" as const, secretId: "11111111-1111-4111-8111-111111111111", version: "latest" as const };
const payload = profile.buildAgent({ executionId: "fixture", environmentId: "local", environmentFixtureId: "local", workspacePath: "/tmp/workspace", secretRefs: { [profile.credential]: secretRef } });
expect(payload.adapterConfig).toMatchObject({ provider: "acpx", model: profile.model, timeoutSec: 120, acpxAgent: profile.qualificationCandidate, env: { [profile.credential]: secretRef } });
@@ -32,11 +32,12 @@ describe("extended ACP harness qualification", () => {
const cell = selected[0]!;
const ambient = { PAPERCLIP_RUNNER_ACPX_QUALIFICATION: "ambient", CURSOR_AUTH_TOKEN: "cursor", CURSOR_API_KEY: "cursor-key", COPILOT_GITHUB_TOKEN: "copilot", GH_TOKEN: "github", GITHUB_TOKEN: "github" };
const env = buildRunnerE2EProcessEnvironment(ambient, [cell]);
expect(JSON.parse(env.PAPERCLIP_RUNNER_ACPX_QUALIFICATION!)).toEqual([{ agent: "cursor", model: cell.profile.model }]);
expect(env.PAPERCLIP_RUNNER_ACPX_QUALIFICATION).toBeUndefined();
const server = buildPaperclipServerEnvironment(env);
for (const name of Object.keys(ambient).filter(name => name !== "PAPERCLIP_RUNNER_ACPX_QUALIFICATION")) expect(server[name]).toBeUndefined();
expect(buildRunnerE2EProcessEnvironment(ambient, []).PAPERCLIP_RUNNER_ACPX_QUALIFICATION).toBeUndefined();
expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...cell, suite: { ...cell.suite, manualOnly: false } }])).toThrow("explicit");
const pending = selected.find(value => value.profile.qualificationCandidate === "copilot")!;
expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...pending, suite: { ...pending.suite, manualOnly: false } }])).toThrow("explicit");
});
it("adds six explicit local/Daytona warm cells without admitting them implicitly", () => {
const warm = runnerMatrix.filter(cell => cell.suite.id === "rich-acp-warm-continuity");
@@ -95,6 +96,7 @@ describe("extended ACP harness qualification", () => {
it("grades the actual file independently of the model's validation claim", () => {
const cell = selected.find(cell => cell.task.id === "file-edit-validate")!;
expect(extendedHarnessFileTask.buildMatchers("nonce", cell)).toContainEqual({ kind: "file_exact", path: "extended-nonce.txt", expected: "verified-nonce\n" });
expect(extendedHarnessFileTask.buildMatchers("nonce", cell)).toContainEqual({ kind: "artifact_exact", name: "extended-nonce.txt", expected: "verified-nonce\n", mimeType: "text/plain" });
});
it("redacts and detects GitHub credential shapes even without the bound value", () => {
const token = `github_pat_${"x".repeat(40)}`;
tests/runner-e2e/fixtures/hiring-templates/coder.d7bdfc4.md
+64
View File
@@ -0,0 +1,64 @@
# Coder Agent Template
Use this template when hiring software engineers who implement code, debug issues, write tests, and coordinate with QA or engineering leadership.
## Recommended Role Fields
- `name`: `Coder`, `CodexCoder`, `ClaudeCoder`, or a model/tool-specific name
- `role`: `engineer`
- `title`: `Software Engineer`
- `icon`: `code`
- `capabilities`: `Implements coding tasks, writes and edits code, debugs issues, adds focused tests, and coordinates with QA and engineering leadership.`
- `adapterType`: `codex_local`, `claude_local`, `cursor`, or another coding adapter
## `AGENTS.md`
```md
You are agent {{agentName}} (Coder / Software Engineer) at {{companyName}}.
When you wake up, follow the Paperclip skill. It contains the full heartbeat procedure.
You are a software engineer. Your job is to implement coding tasks:
- Write, edit, and debug code as assigned
- Follow existing code conventions and architecture
- Leave code better than you found it
- Comment your work clearly in task updates
- Ask for clarification when requirements are ambiguous
- Test your changes with the smallest verification that proves the work
You report to {{managerTitle}}. Work only on tasks assigned to you or explicitly handed to you in comments. When done, mark the task done with a clear summary of what changed and how you verified it.
Start actionable work in the same heartbeat; do not stop at a plan unless planning was requested. Leave durable progress with a clear next action. Use child issues for long or parallel delegated work instead of polling. Mark blocked work with owner and action. Respect budget, pause/cancel, approval gates, and company boundaries.
Commit things in logical commits as you go when the work is good. If there are unrelated changes in the repo, work around them and do not revert them. Only stop and say you are blocked when there is an actual conflict you cannot resolve.
Make sure you know the success condition for each task. If it was not described, pick a sensible one and state it in your task update. Before finishing, check whether the success condition was achieved. If it was not, keep iterating or escalate with a concrete blocker.
Keep the work moving until it is done. If you need QA to review it, ask QA. If you need your manager to review it, ask them. If someone needs to unblock you, assign or hand back the ticket with a comment explaining exactly what you need.
An implied addition to every prompt is: test it, make sure it works, and iterate until it does. If it is a shell script, run a safe version. If it is code, run the smallest relevant tests or checks. If browser verification is needed and you do not have browser capability, ask QA to verify.
If you are asked to fix a deployed bug, fix the bug, identify the underlying reason it happened, add coverage or guardrails where practical, and ask QA to verify the fix when user-facing behavior changed.
If the task is part of an existing PR and you are asked to address review feedback or failing checks after the PR has already been pushed, push the completed follow-up changes unless your company instructions say otherwise.
If there is a blocker, explain the blocker and include your best guess for how to resolve it. Do not only say that it is blocked.
When you run tests, do not default to the entire test suite. Run the minimal checks needed for confidence unless the task explicitly requires full release or PR verification.
## Collaboration and handoffs
- UX-facing changes → loop in `[UXDesigner](/{{issuePrefix}}/agents/uxdesigner)` for review of visual quality and flows.
- Security-sensitive changes (auth, crypto, secrets, permissions, adapter/tool access) → loop in `[SecurityEngineer](/{{issuePrefix}}/agents/securityengineer)` before merging.
- Browser validation / user-facing verification → hand to `[QA](/{{issuePrefix}}/agents/qa)` with a reproducible test plan.
- Skill or instruction quality changes → hand to the skill consultant or equivalent instruction owner.
## Safety and permissions
- Never commit secrets, credentials, or customer data. If you spot any in the diff, stop and escalate.
- Do not bypass pre-commit hooks, signing, or CI unless the task explicitly asks you to and the reason is documented in the commit message.
- Do not install new company-wide skills, grant broad permissions, or enable timer heartbeats as part of a code change — those are governance actions that belong on a separate ticket.
You must always update your task with a comment before exiting a heartbeat.
```
tests/runner-e2e/fixtures/native-completion/terminal-tool-carrier.json
+55
View File
@@ -0,0 +1,55 @@
{
"schema": "paperclip.native-completion-terminal-carrier.fixture.v1",
"cases": [
{
"name": "paperclip_finish",
"providerInput": {
"item": {
"id": "terminal-call",
"type": "tool_call",
"name": "paperclip_finish",
"status": "inProgress",
"arguments": {
"secret": "not-for-the-log"
}
}
},
"normalizedPayload": {
"provider": "codex",
"itemId": "terminal-call",
"kind": "tool_call",
"status": "running",
"channel": "detail",
"text": null,
"item": {
"name": "paperclip_finish"
}
}
},
{
"name": "paperclip_block",
"providerInput": {
"item": {
"id": "terminal-call",
"type": "tool_call",
"name": "paperclip_block",
"status": "inProgress",
"arguments": {
"secret": "not-for-the-log"
}
}
},
"normalizedPayload": {
"provider": "codex",
"itemId": "terminal-call",
"kind": "tool_call",
"status": "running",
"channel": "detail",
"text": null,
"item": {
"name": "paperclip_block"
}
}
}
]
}
tests/runner-e2e/fixtures/plan-task-guidance/current-conversion.md
+60
View File
@@ -0,0 +1,60 @@
---
name: paperclip-converting-plans-to-tasks
description: >
Convert Paperclip plans into executable issue graphs. Use when asked to plan,
scope, or break down Paperclip company work into assigned tasks with specialty
fit, dependencies, blockers, and parallelization.
---
# Paperclip — Converting Plans to Tasks
A companion skill for turning a plan into executable Paperclip work. It does **not** dictate a plan structure — bring whatever format fits the work and the user's preference. It tells you _how_ to translate that plan into issues so that the rest of Paperclip works for you.
For the **mechanics** of recording a plan (issue document with key `plan`, comment links, approval gating, who to reassign back to), follow the _Planning_ section of the `paperclip` skill. This skill covers planning method, not the API surface.
## When you're asked to plan
- **Plan deeply.** Capture as much real detail as you have: goals, constraints, unknowns, success criteria, risks. A shallow plan becomes rework downstream — assignees can only act on what they can read.
- **Minimize the issue graph.** Use as few tasks as possible while still completing and verifying the job. Prefer one end-to-end task with one owner over separate tasks for each step, file, component, or phase. Keep those structural details as checklists or acceptance criteria inside the owning task unless a real execution boundary requires another issue.
- **Split only for a qualifying boundary.** Create a separate subtask only when at least one of these applies:
- A different specialist, owner, permission boundary, or external actor must own the work.
- A self-contained deliverable can usefully run in parallel with other work.
- A hard dependency or handoff needs its own `blockedByIssueIds` lifecycle.
- A review, QA pass, or governed approval gate has an independent owner.
- Substantial follow-up work needs independent tracking or retry because it cannot safely be completed and verified in the parent.
- **Know your team.** Before assigning anything, look up the company's agents and their specialties (reporting lines, role descriptions, prior work). Don't default work to yourself when a better-suited agent exists; don't assign to a name you haven't checked.
- **Assign for specialty.** Hand each piece of work to the agent most relevant to it. If no one fits, call that out — a hire, a tool, an external dependency, a board decision — instead of papering over the gap.
- **Take responsibility.** Specialty-matching cuts both ways: when _you_ are the best-suited agent for a piece of work, assign it to yourself instead of reflexively delegating. Don't hand off to avoid load.
- **Use the dependency tree.** Paperclip's executor automatically starts any assigned task with no open blockers. Parent/child issue nesting is structure, not execution blocking. Express each qualifying ownership or lifecycle boundary as an issue; keep other concrete deliverables within the responsible issue's description, checklist, or acceptance criteria. Wire every hard dependency between issues through `blockedByIssueIds` on the dependent issue (not prose like "blocked by X"). When a blocker reaches `done`, dependents auto-wake.
- **Order, then parallelize.** Sequence work by real dependencies, not by personal preference. Create parallel branches only for qualifying, self-contained work, then start those independent branches in parallel. Unlike humans, most agents allow concurrent runs, so you can assign parallel work to the same agent.
- **Write review tasks for the reviewer's boundary.** A review/QA task must tell the delegate to post findings on **their own review issue** and mark it `done` — the verdict is the deliverable, and adverse findings are still `done`, not `blocked`. Never instruct a delegate to comment on the parent issue (low-trust reviewers are guaranteed a 403 there), and make the description self-contained since the reviewer may not be able to read your issue. Wire the dependent issue's `blockedByIssueIds` to the review issue so the verdict wakes the right owner.
- **Enough is enough.** Plans exist to unblock execution, not replace it. If the next step is small and clear, just do it or allow the plan to stand on its own. Re-planning a plan, or splitting work that one agent could finish in the time it took to break it up, is procrastination — ship something.
## When converting an accepted plan into tasks
Start from one end-to-end task and add issues only for the qualifying boundaries above. Before creating tasks, write a compact task matrix with each proposed task, owner, initial status, blockers, and the specific qualifying reason it must be separate. Any task that can start immediately should say why it has no blockers; otherwise set it to `blocked` and include the prerequisite issue IDs in `blockedByIssueIds`. Do not rely on `parentId`, child ordering, phase labels, or prose to block execution.
Run a merge-back pass before publishing or creating the graph. Require every proposed subtask to name at least one qualifying reason from this skill. If it cannot, merge it into its parent or an adjacent task and preserve the work as an internal step, checklist item, or acceptance criterion. Repeat until every remaining issue has a real ownership, scheduling, lifecycle, or governance reason to exist.
After creating the tasks, re-fetch the created issues or otherwise verify the issue graph before marking the source planning issue done. Confirm that every separate issue still has its qualifying reason, each dependent task has the expected `blockedByIssueIds`, each independent task has an explicit "can start now" reason, review tasks respect the reviewer's write boundary, and the parent/child hierarchy is only being used for traceability. If the graph contains an unjustified split or expected blockers are missing, correct it or report the mismatch and leave the planning issue in `in_review` or `blocked` until the graph is fixed.
## Quick checklist before you publish a plan
- [ ] Enough detail that assignees can act without re-asking.
- [ ] The plan uses the fewest tasks that can complete and verify the job, preferring one end-to-end owner over step/file/component/phase splits.
- [ ] Every concrete deliverable is accounted for inside an issue or, only when a qualifying boundary applies, as its own issue.
- [ ] Every proposed subtask names a qualifying reason; otherwise it was merged into its parent or an adjacent task.
- [ ] Each issue has a deliberate, specialty-matched assignee — not the planner by default.
- [ ] Each issue's real blockers are declared via `blockedByIssueIds`.
- [ ] Independently owned review, QA, and governed approval tasks respect the reviewer's boundary.
- [ ] A compact task matrix names planned task, owner, initial status, blockers, and qualifying reason.
- [ ] Tasks without blockers have an explicit reason they can start immediately.
- [ ] Created issues were re-fetched or otherwise verified before closing the source planning issue.
- [ ] Qualifying independent branches can start in parallel.
- [ ] Gaps (missing skills, hires, decisions, external inputs) are surfaced, not hidden.
## What this skill is not
- Not a plan template. Use any format — prose, outline, table, RACI, Gantt, whatever fits.
- Not software-development–specific. The same rules apply to marketing, research, ops, design, hiring, finance, etc.
- Not a replacement for the `paperclip` skill's planning mechanics. Use both.
tests/runner-e2e/fixtures/plan-task-guidance/current-planning.md
+84
View File
@@ -0,0 +1,84 @@
---
name: task-planning
description: Turn a Paperclip issue or request into a structured implementation plan with child task graph, blockers, owners, and acceptance criteria, then save it as the issue `plan` document.
key: paperclipai/bundled/paperclip-operations/task-planning
recommendedForRoles:
- manager
- engineer
- product
tags:
- paperclip
- planning
- issues
- delegation
---
# Task Planning
Produce implementation plans that the Paperclip executor can actually run: explicit child issues, real blockers, named owners, and a defined acceptance bar. Avoid plans that read well but cannot be split into work.
## When to use
- An issue asks you to "plan", "scope", "break down", "design the rollout", "propose the work", or similar.
- A user wants a written plan before approving implementation.
- A manager needs to delegate non-trivial work and the shape of the work is not obvious yet.
- You inherited an issue too large to deliver in one heartbeat and need to split it.
## When not to use
- The issue is a single small change you can ship in the same heartbeat. Just ship it.
- The issue is forensic ("why did this break"). Use a diagnosis skill first; plan only after the root cause is named.
- A current `plan` document already exists and the change is minor. Update that document; do not start fresh.
## Outputs
1. An updated issue document with key `plan` (markdown).
2. A short comment on the issue that links to the plan document and names the next action.
3. Where the plan requires approval, an issue-thread interaction of kind `request_confirmation` bound to the latest plan revision.
Do not create implementation subtasks until the plan is accepted.
## Plan structure
Required sections, in order:
1. **Goal** — one paragraph. What changes for the user, the operator, or the system once this work lands.
2. **Context reviewed** — bullet list of documents, files, and prior issues you read. Lets reviewers spot missing inputs.
3. **Constraints and non-goals** — what must hold (compatibility, security, performance) and what this plan deliberately will not do.
4. **Approach** — the chosen path, with a short rationale. If you considered alternatives, name them and why you rejected them.
5. **Work breakdown** — ordered list of child issues. Each child has:
- Title in imperative form.
- Owner specialty (Engineer, QA, Designer, Security, DevRel, Manager, etc.).
- Scope and deliverables.
- Acceptance criteria.
- Blocks/blocked-by relationships expressed by phase letter or child title.
6. **Acceptance** — the bar for the parent issue. How the user knows the whole thing is done.
7. **Risks and mitigations** — short list. Skip if there are none.
8. **Deferrals** — what is intentionally pushed to follow-up issues, with why.
## Rules of thumb for splitting
- One child issue, one specialty. If two specialties have to coordinate inside the same issue, split it.
- One child issue, one acceptance verdict. If a reviewer would say "this is half done", split it.
- A child must be checkout-able by the owner from its title and description alone. Reviewers should not have to re-read the parent plan to understand a child.
- Order children by real blocker chains, not by author preference. Parallel children should explicitly say `blockers: none`.
- Avoid `polish` or `cleanup` child issues without acceptance criteria — they never close.
## Filing the plan
Use the Paperclip API to write the plan document, then comment:
- `PUT /api/issues/{issueId}/documents/plan` with the markdown body. If `plan` already exists, include the latest `baseRevisionId`.
- `POST /api/issues/{issueId}/comments` with a short summary that links the plan: `/<prefix>/issues/<issue-id>#document-plan`.
- If approval is required: `POST /api/issues/{issueId}/interactions` with `kind: request_confirmation`, `targetRevisionId` set to the new plan revision, `continuationPolicy: wake_assignee`, and `idempotencyKey: "confirmation:{issueId}:plan:{revisionId}"`.
- Set the issue to `in_review` after creating the confirmation. Stay assigned so the acceptance wakes the planner.
When the plan is accepted, see the companion skill for converting accepted plans into Paperclip executable tasks. Key requirements covered there: produce a compact task matrix (task, owner, initial status, blockers); encode every hard dependency as `blockedByIssueIds` — parent/child nesting alone does not block execution; and verify the created issue graph before closing the source planning issue.
## Anti-patterns
- Plan disguised as a description edit. Use the `plan` document.
- "Phases A–Z" with no work breakdown inside the phases.
- Children with descriptions that say "see parent" — they fail at delegation time.
- Acceptance written as "code review approval". Reviewers need a behavior bar, not a process bar.
- Plans that bury blocker chains in prose. Use explicit blocked-by lines.
tests/runner-e2e/fixtures/stock-harness/historical-default-agents.md
+24
View File
@@ -0,0 +1,24 @@
You are an agent at Paperclip company.
## Execution Contract
- Start actionable work in the same heartbeat. Do not stop at a plan unless the issue explicitly asks for planning.
- Keep the work moving until it is done. If you need QA to review it, ask them. If you need your boss to review it, ask them.
- Leave durable progress in task comments, documents, or work products, then update the issue to a clear final disposition before you exit.
- When your work produces a user-inspectable deliverable file, follow the Paperclip skill's "Generated Artifacts and Work Products" workflow before final disposition. Use `skills/paperclip/scripts/paperclip-upload-artifact.sh` when working in this repo, create/update an artifact work product when the file is the deliverable, and link the uploaded attachment in the final comment. Do not rely on local filesystem paths as the only access path. If an important file intentionally remains workspace-only, create/update a work product with `metadata.resourceRef.kind: "workspace_file"` and a workspace-relative path, then name that work product and path in the final comment. Treat browse/search as a fallback for recovering workspace files, not the preferred deliverable path.
- When your work produces or updates an operator-facing engineering output, create/update the matching work product: `pull_request` for opened PRs, `preview_url` for published previews, `runtime_service` for managed preview/dev services, `commit` for notable pushed commits, and `branch` when the branch itself is the handoff. A comment is not a substitute for the work product access path.
- Comments, documents, screenshots, work products, and `Remaining` bullets are evidence, not valid liveness paths by themselves.
- Final disposition checklist: mark `done` when complete and verified; use `in_review` only with a real reviewer, approval, interaction, or monitor path; use `blocked` only with first-class blockers or a named unblock owner/action; create delegated follow-up issues with blockers when another agent owns the next step; keep `in_progress` only when a live continuation path exists.
- Use child issues for parallel or long delegated work instead of polling agents, sessions, or processes.
- Create child issues directly when you know what needs to be done. If the board/user needs to choose suggested tasks, answer structured questions, or confirm a proposal first, create an issue-thread interaction on the current issue with `POST /api/issues/{issueId}/interactions` using `kind: "suggest_tasks"`, `kind: "ask_user_questions"`, or `kind: "request_confirmation"`.
- Use `request_confirmation` instead of asking for yes/no decisions in markdown. Before presenting a plan for review, you MUST complete this publish contract:
1. `PUT /issues/{id}/documents/plan` with `{ format: 'markdown', body, changeSummary }`.
2. Re-`GET /documents/plan`, assert it returns `200`, and capture its `latestRevisionId`.
3. Only then create `request_confirmation` with `target={ type: 'issue_document', key: 'plan', revisionId: latestRevisionId }` and `idempotencyKey=confirmation:{issueId}:plan:{revisionId}`.
4. Wait for acceptance before creating implementation subtasks.
Never present a plan only in a thread comment or through `ask_user_questions`; comments are supporting context and questions are for gathering input, not plan review.
- `ask_user_questions` and confirmations default `supersedeOnUserComment` to `false`, so a later board/user comment keeps the pending card open while discussion continues. Set it to `true` when a new comment should replace the pending request. If you wake up from a superseding comment, revise the artifact, question set, or proposal and create a fresh interaction if input is still needed.
- For human input, save a pending question/confirmation interaction and set `in_review`; prose alone does not create a waiting path. Use `blockedByIssueIds` for issue dependencies. An agent may set an `unblockDescriptor` only for itself (`owner: { "agentId": "<your-agent-id>" }` plus `action`), not for the board/user or another agent.
- Respect budget, pause/cancel, approval gates, and company boundaries.
Do not let work sit here. You must always update your task with a comment.
tests/runner-e2e/gemini-acp-filesystem.smoke.ts
+82
View File
@@ -0,0 +1,82 @@
/** Opt-in installed-CLI compatibility check. No real provider credentials or inference. */
import { createServer } from "node:http";
import { mkdir, mkdtemp, readFile, realpath, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { createAcpRuntime, createAgentRegistry, createRuntimeStore } from "../../packages/adapter-utils/node_modules/acpx/dist/runtime.js";
const argument = process.argv.indexOf("--agent-command");
const command = argument < 0 ? "gemini --acp" : process.argv[argument + 1];
if (!command) throw new Error("agent_command_required");
const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-gemini-filesystem-smoke-"));
const content = '{"nonce":"synthetic-file-smoke","total":28}\n';
const results: Array<{ case: string; passed: boolean; completedTools: number; failedTools: number; missingFileSignal: boolean }> = [];
let active: { calls: number; file: string; tool: string };
const server = createServer(async (request, response) => {
// Never retain request bodies, headers, provider prompts or model output.
for await (const _chunk of request) { /* Drain the synthetic request. */ }
const parts = ++active.calls === 1
? [{ functionCall: { name: active.tool, args: { file_path: active.file, ...(active.tool === "write_file" ? { content } : {}) } } }]
: [{ text: "synthetic-complete" }];
const payload = {
candidates: [{ content: { role: "model", parts }, finishReason: "STOP" }],
usageMetadata: { promptTokenCount: 10, candidatesTokenCount: 2, totalTokenCount: 12 },
modelVersion: "gemini-3.1-flash-lite",
};
if (request.url?.includes(":streamGenerateContent")) {
response.writeHead(200, { "content-type": "text/event-stream" });
response.end(`data: ${JSON.stringify(payload)}\n\n`);
} else {
response.writeHead(200, { "content-type": "application/json" });
response.end(JSON.stringify(request.url?.includes(":countTokens") ? { totalTokens: 10 } : payload));
}
});
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
const origin = `http://127.0.0.1:${(server.address() as { port: number }).port}`;
try {
for (const scenario of ["create-missing", "read-existing", "deny-outside"] as const) {
const home = path.join(root, scenario, "home");
const workspace = path.join(root, scenario, "workspace");
await mkdir(path.join(home, ".gemini"), { recursive: true, mode: 0o700 });
await mkdir(workspace, { recursive: true });
const cwd = await realpath(workspace);
const file = scenario === "deny-outside" ? path.join(await realpath(root), "outside.json") : path.join(cwd, "proof.json");
if (scenario !== "create-missing") await writeFile(file, content);
await writeFile(path.join(home, ".gemini", "settings.json"), JSON.stringify({
security: { auth: { selectedType: "gemini-api-key" } }, telemetry: { enabled: false },
}), { mode: 0o600 });
active = { calls: 0, file, tool: scenario === "read-existing" ? "read_file" : "write_file" };
const runtime = createAcpRuntime({
cwd, sessionStore: createRuntimeStore({ stateDir: path.join(root, scenario, "state") }),
agentRegistry: createAgentRegistry({ overrides: { gemini: command } }),
permissionMode: "approve-all", nonInteractivePermissions: "deny", inheritProcessEnv: false,
onAgentStderr: () => { /* Child diagnostics may contain untrusted material. */ },
});
const handle = await runtime.ensureSession({
sessionKey: `gemini-filesystem-${scenario}`, agent: "gemini", mode: "oneshot", cwd,
sessionOptions: { env: {
HOME: home, PATH: process.env.PATH ?? "", GEMINI_MODEL: "gemini-3.1-flash-lite",
GEMINI_API_KEY: "synthetic-invalid-key", GOOGLE_GEMINI_BASE_URL: origin,
} },
});
try {
let completedTools = 0; let failedTools = 0; let missingFileSignal = false;
for await (const event of runtime.runTurn({ handle, text: "Run the synthetic file operation, then stop.", mode: "prompt", requestId: scenario, timeoutMs: 20_000 })) {
if (event.type !== "tool_call") continue;
if (event.status === "completed") completedTools++;
if (event.status === "failed") {
failedTools++;
missingFileSignal ||= /Error checking existing file: (?:Resource not found|Internal error)/.test(event.text ?? "");
}
}
const bytes = await readFile(file, "utf8").catch(() => null);
const passed = bytes === content && (scenario === "deny-outside" ? failedTools > 0 && completedTools === 0 : failedTools === 0 && completedTools > 0);
results.push({ case: scenario, passed, completedTools, failedTools, missingFileSignal });
} finally { await runtime.close({ handle, reason: "synthetic filesystem smoke complete" }); }
}
console.log(JSON.stringify({ realProviderCalls: 0, results }, null, 2));
if (results.some((result) => !result.passed)) process.exitCode = 1;
} finally {
await new Promise<void>((resolve) => server.close(() => resolve()));
await rm(root, { recursive: true, force: true });
}
tests/runner-e2e/harness-env.ts
+2 -7
View File
@@ -117,14 +117,9 @@ export function buildRunnerE2EProcessEnvironment(
const candidates = new Map<string, string>();
for (const execution of executions) {
const agent = execution.profile.qualificationCandidate;
if (!agent) continue;
if (!agent || agent === "cursor") continue;
const admittedSuite = execution.suite.id === "extended-harnesses"
|| execution.suite.id === "rich-acp-warm-continuity"
|| (execution.suite.id === "pi-native" && agent === "pi")
|| (execution.suite.id === "pi-controls" && agent === "pi")
|| (execution.suite.id === "cursor-native" && agent === "cursor")
|| (execution.suite.id === "copilot-protection" && agent === "copilot")
|| (execution.suite.id === "native-active-stop" && (agent === "cursor" || agent === "copilot"));
|| execution.suite.id === "rich-acp-warm-continuity";
if (!admittedSuite || !execution.suite.manualOnly) {
throw new Error("Candidate qualification requires an explicit provider qualification suite");
}
tests/runner-e2e/hiring-template-cases.ts
+60
View File
@@ -0,0 +1,60 @@
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import type { RunnerProfileFixture, RunnerTaskFixture } from "./types.js";
export const HIRING_TEMPLATE_GRADER_VERSION = "paperclip.hiring-templates.v3";
export const HIRING_TEMPLATE_SKILL_KEY = "paperclipai/paperclip/paperclip-create-agent";
export const HIRING_TEMPLATE_READ_FILES = [
"SKILL.md",
"references/agent-instruction-templates.md",
"references/draft-review-checklist.md",
"references/agents/coder.md",
] as const;
export const HIRING_TEMPLATE_SOURCE_FILES = [
"server/src/services/default-agent-instructions.ts",
"server/src/onboarding-assets/default/AGENTS.md",
...HIRING_TEMPLATE_READ_FILES.map(file => `skills/paperclip-create-agent/${file}`),
"skills/paperclip-create-agent/references/baseline-role-guide.md",
] as const;
export const hiringTemplateDefinitionDigest = createHash("sha256").update(
["hiring-template-cases.ts", "hiring-template-scoring.ts", "hiring-template-flow.ts", "hiring-template-turn-accounting.ts", "chat-flow.ts"]
.map(file => readFileSync(new URL(`./${file}`, import.meta.url), "utf8")).join("\n"),
).digest("hex");
export function hiringTemplateProfile(profile: RunnerProfileFixture): RunnerProfileFixture {
return {
...profile,
buildAgent(input) {
const { instructionsBundle: _fixtureInstructions, ...payload } = profile.buildAgent(input);
return { ...payload, name: `Hiring Lead ${input.executionId}`, role: "ceo", title: "CEO" };
},
};
}
export const hiringTemplateInputs = [" Launch Queue!! ", "API__Key Rotation", "Mixed CASE / 42", "---Already--Ready---"] as const;
export function hiringTemplateScenario(nonce: string, projectName: string) {
const marker = `HIRE${nonce.replace(/[^a-zA-Z0-9]/g, "")}`;
const hireName = `Casey Coder ${nonce}`;
return {
marker, hireName, inputs: hiringTemplateInputs,
initialPrompt: [
`Use the paperclip-create-agent skill to hire exactly one coding teammate named ${hireName}, reporting to you, with your native runner, model, and available AI connection.`,
"Before hiring, read that skill's SKILL.md, references/agent-instruction-templates.md, references/draft-review-checklist.md, and references/agents/coder.md. Use the AGENTS.md example from that coder reference as the teammate's managed instructions, filling all agentName, companyName, managerTitle, and issuePrefix placeholders from this hire, company, your actual title, and the company issue prefix. Keep skills and reporting information in their configuration.",
`Have that coder prepare a JSON test fixture for a label normalizer in one assigned task in ${projectName}. Save the fixture as a Paperclip document with an object containing reference: ${JSON.stringify(marker)} and entries, an ordered array of objects with input and value. Inputs: ${JSON.stringify(hiringTemplateInputs)}.`,
"For each value, trim its input, lowercase it, replace each sequence outside ASCII a-z and 0-9 with one hyphen, then trim leading and trailing hyphens. Link the task here and let the coder complete it.",
].join("\n\n"),
reusePrompt: (identifier: string) => `Have the existing ${hireName} prepare a second saved JSON fixture in one new task in ${projectName}, assigned to that same coder. Use the inputs and normalization rule from ${identifier}, but use underscores instead of hyphens for the value separators and trim leading and trailing underscores. Use reference: ${JSON.stringify(`REUSE${marker}`)}. Include the original fixture in the handoff. Preserve the original document and task, and let the coder complete the new task.`,
statusPrompt: (first: string, second: string) => `Briefly report the recorded owner and status of ${first} and ${second}. Just report; do not create or change work.`,
};
}
export const hiringTemplateTasks: readonly RunnerTaskFixture[] = [{
id: "hire-coder-template-reuse", label: "Hire with production templates and reuse the coder",
groups: ["chat"], flow: "agent_chat", workMode: "standard", expectedRunCount: 7, minimumExpectedRunCount: 5,
attemptTimeoutMs: { local: 15 * 60_000, daytona: 15 * 60_000 },
expectedTerminalState: { issue: "in_review", run: "succeeded" },
buildTitle: nonce => `Production hiring templates ${nonce}`,
buildPrompt: nonce => hiringTemplateScenario(nonce, "the selected project").initialPrompt,
buildVisibleMarker: nonce => hiringTemplateScenario(nonce, "").marker,
buildMatchers: () => [{ kind: "issue_status", expected: "in_review" }],
}];
tests/runner-e2e/hiring-template-flow.ts
+210
View File
@@ -0,0 +1,210 @@
import { readFile } from "node:fs/promises";
import { loadDefaultAgentInstructionsBundle } from "../../server/src/services/default-agent-instructions.js";
import { readChatOutputDocument, type ChatFlowInput, type ChatIssue, type ChatRun } from "./chat-flow.js";
import { collectRunEvents } from "./run-observations.js";
import { HIRING_TEMPLATE_GRADER_VERSION, HIRING_TEMPLATE_READ_FILES, HIRING_TEMPLATE_SKILL_KEY,
HIRING_TEMPLATE_SOURCE_FILES, hiringTemplateDefinitionDigest, hiringTemplateScenario } from "./hiring-template-cases.js";
import { gradeHiringTemplate, hiringTemplateHash, hiringTemplateSize, type HiringAgent, type HiringDocument,
type HiringInstructionSnapshot, type HiringTemplateEvidence } from "./hiring-template-scoring.js";
import { pollUntil, type RunnerApi } from "./api.js";
export async function readHiringInstructions(api: Pick<RunnerApi, "get">, agentId: string): Promise<HiringInstructionSnapshot> {
const bundle = await api.get<{ mode: string | null; entryFile: string; files: Array<{ path: string; binary?: boolean }> }>(`/api/agents/${agentId}/instructions-bundle`);
const files: Record<string, string> = {};
// Agent-home notes may be added by a run. Retain the entry and legacy CEO
// policy files so the baseline and candidate default bundles can be compared.
for (const file of bundle.files.filter(file => !file.binary && (file.path === bundle.entryFile || ["HEARTBEAT.md", "SOUL.md", "TOOLS.md"].includes(file.path)))) {
const detail = await api.get<{ content: string }>(`/api/agents/${agentId}/instructions-bundle/file?path=${encodeURIComponent(file.path)}`);
files[file.path] = detail.content;
}
return { entryFile: bundle.entryFile, mode: bundle.mode, files };
}
async function readHiringSkillSelections(api: Pick<RunnerApi, "get">, companyId: string, agentId: string) {
const snapshot = await api.get<{ desiredSkills: string[]; desiredSkillEntries: unknown[] }>(`/api/agents/${agentId}/skills?companyId=${companyId}`);
return { desiredSkills: snapshot.desiredSkills, desiredSkillEntries: snapshot.desiredSkillEntries };
}
export async function readHiringTemplateSources(api: Pick<RunnerApi, "get">, companyId: string, leadId: string) {
const expectedCeoFiles = await loadDefaultAgentInstructionsBundle("ceo");
const expectedSourceHashes: Record<string, string> = {}, servedSourceHashes: Record<string, string> = {};
const sourceSizes: Record<string, ReturnType<typeof hiringTemplateSize>> = {};
for (const relative of [...HIRING_TEMPLATE_SOURCE_FILES, ...Object.keys(expectedCeoFiles).map(file => `server/src/onboarding-assets/ceo/${file}`)]) {
const content = await readFile(new URL(`../../${relative}`, import.meta.url), "utf8");
expectedSourceHashes[relative] = hiringTemplateHash(content);
sourceSizes[relative] = hiringTemplateSize(content);
}
const leadInstructions = await readHiringInstructions(api, leadId);
for (const [file, content] of Object.entries(leadInstructions.files)) servedSourceHashes[`server/src/onboarding-assets/ceo/${file}`] = hiringTemplateHash(content);
const skills = await api.get<Array<{ id: string; key: string }>>(`/api/companies/${companyId}/skills`);
const creator = skills.find(skill => skill.key === HIRING_TEMPLATE_SKILL_KEY);
if (!creator) throw new Error("Production hiring skill is absent from the company library");
const assigned = await api.get<{ desiredSkills: string[] }>(`/api/agents/${leadId}/skills?companyId=${companyId}`);
let coderReference = "";
for (const file of [...HIRING_TEMPLATE_READ_FILES, "references/baseline-role-guide.md"]) {
const detail = await api.get<{ content: string }>(`/api/companies/${companyId}/skills/${creator.id}/files?path=${encodeURIComponent(file)}`);
servedSourceHashes[`skills/paperclip-create-agent/${file}`] = hiringTemplateHash(detail.content);
if (file === "references/agents/coder.md") coderReference = detail.content;
}
// The loader and execution contract are source provenance, not served skill
// reads. Their bytes are retained separately from the equality checks.
delete expectedSourceHashes["server/src/services/default-agent-instructions.ts"];
delete expectedSourceHashes["server/src/onboarding-assets/default/AGENTS.md"];
return { expectedCeoFiles, leadInstructions, expectedSourceHashes, servedSourceHashes,
sourceSizes, assignedSkills: assigned.desiredSkills, coderReference, creatorSkillId: creator.id,
loaderHash: hiringTemplateHash(await readFile(new URL("../../server/src/services/default-agent-instructions.ts", import.meta.url), "utf8")),
executionContractHash: hiringTemplateHash(await readFile(new URL("../../server/src/onboarding-assets/default/AGENTS.md", import.meta.url), "utf8")) };
}
export function renderHiringCoderExample(reference: string, agentName: string, companyName: string, managerTitle: string, issuePrefix: string) {
const example = reference.match(/```md\s*\n([\s\S]*?)\n```/)?.[1];
if (!example?.trim()) throw new Error("Production coder reference has no AGENTS.md example");
return example.replaceAll("{{agentName}}", agentName).replaceAll("{{companyName}}", companyName).replaceAll("{{managerTitle}}", managerTitle).replaceAll("{{issuePrefix}}", issuePrefix);
}
/** Reads only existing public surfaces; this does not fabricate lifecycle evidence. */
export async function readHiringTurnApiState(api: Pick<RunnerApi, "get">, chatIssueId: string, allRuns: () => Promise<ChatRun[]>) {
const [issue, comments, runs, wakes] = await Promise.all([
api.get<ChatIssue>(`/api/issues/${chatIssueId}`),
api.get<unknown[]>(`/api/issues/${chatIssueId}/comments`),
allRuns(),
api.get<{ events: Array<{ kind: string; status?: string; finishedAt?: string | null; runId?: string | null }>; truncated: boolean }>(`/api/issues/${chatIssueId}/diagnostics/wakes`),
]);
return { issue, comments, runs, wakes };
}
type HiringTurnApiState = Awaited<ReturnType<typeof readHiringTurnApiState>>;
interface HiringObservation {
agents: HiringAgent[]; tasks: ChatIssue[]; runs: ChatRun[]; readRuns: HiringTemplateEvidence["readRuns"];
apiState?: HiringTurnApiState; finalApiState?: HiringTurnApiState;
}
/** Retry whole observations, never splice different ledger generations together. */
export async function waitForSettledHiringObservation(
load: () => Promise<HiringObservation>, options: { deadlineAt?: number; intervalMs?: number } = {},
) {
let previous: string | undefined;
return pollUntil({ label: "hiring work and completion deliveries settle consistently",
deadlineAt: options.deadlineAt ?? Date.now() + 90_000, intervalMs: options.intervalMs ?? 1_000,
load, accept: observation => {
const before = observation.apiState, after = observation.finalApiState;
const stable = before && after && JSON.stringify(before) === JSON.stringify(after)
&& JSON.stringify(observation.runs) === JSON.stringify(after.runs);
const completionRuns = observation.runs.filter(run => run.contextSnapshot?.wakeReason === "chat_task_completed");
const completionsObserved = observation.tasks.filter(task => task.status === "done").every(task => {
const callbacks = completionRuns.filter(run => Array.isArray(run.contextSnapshot?.chatCompletionUpdates)
&& run.contextSnapshot.chatCompletionUpdates.filter((update: { id?: string }) => update.id === task.id).length === 1);
return callbacks.length === 1 && after?.comments.some(value => {
const comment = value as Record<string, unknown>;
return comment.createdByRunId === callbacks[0]!.id && comment.issueId === after.issue.id
&& comment.authorAgentId === callbacks[0]!.agentId && !comment.authorUserId
&& typeof comment.body === "string" && comment.body.trim().length > 0;
});
});
const pending = !after || after.wakes.truncated || after.wakes.events.some(wake => {
if (wake.kind !== "wake_request") return false;
if (wake.status === "coalesced") return !completionsObserved || !observation.runs.some(run => run.id === wake.runId
&& ["succeeded", "failed", "cancelled"].includes(run.status));
return !["completed", "skipped", "failed", "cancelled"].includes(wake.status ?? "");
}) || observation.runs.some(run => ["queued", "running", "scheduled_retry"].includes(run.status));
// Await both known callbacks, even before the pending outbox enqueues a
// wake. Reply attribution proves the server acknowledged each delivery.
const signature = stable && !pending && completionsObserved && after.issue.conversationState === "waiting"
? JSON.stringify(observation) : undefined;
const settled = Boolean(signature && signature === previous);
previous = signature;
return settled;
},
});
}
export async function runHiringTemplateFlow(context: {
input: ChatFlowInput; issue(): ChatIssue; turn(message: string, count: number): Promise<void>;
tasks(): Promise<ChatIssue[]>; allRuns(): Promise<ChatRun[]>;
}) {
const { input, turn, tasks, allRuns } = context;
const { api, fixtures: f } = input;
const company = `/api/companies/${f.company.id}`;
const account = f.aiConnection;
if (!account) throw new Error("Hiring-template fixture requires a managed execution account");
const issuePrefix = f.company.issuePrefix;
if (!issuePrefix) throw new Error("Hiring-template fixture requires the actual company issue prefix");
const evidence: HiringTemplateEvidence = {
leadId: f.agent.id, chatIssueId: "", hireName: "", marker: "", projectId: "", inputs: [], expectedCeoFiles: {},
expectedSourceHashes: {}, servedSourceHashes: {}, assignedSkills: [], coderExample: "",
agents: [], tasks: [], runs: [], readRuns: [], connectionId: account.connectionId, binding: account.binding,
};
let source: Awaited<ReturnType<typeof readHiringTemplateSources>> | undefined;
let scenario: ReturnType<typeof hiringTemplateScenario> | undefined;
async function loadObservation() {
const observed = await Promise.all([api.get<HiringAgent[]>(`${company}/agents`), tasks(), allRuns()]);
const [agents, taskRows, runs] = observed;
const apiState = evidence.chatIssueId ? await readHiringTurnApiState(api, evidence.chatIssueId, allRuns) : undefined;
const readRuns = await Promise.all(runs.map(async run => {
const events = await collectRunEvents<{ seq?: number; eventType?: string; payload?: unknown; createdAt?: string }>(
(afterSeq, limit) => api.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=${limit}`),
);
return { runId: run.id, agentId: run.agentId, events };
}));
// Bracket event collection with a fresh ledger/wake/comment observation.
const finalApiState = evidence.chatIssueId ? await readHiringTurnApiState(api, evidence.chatIssueId, allRuns) : undefined;
return { agents, tasks: taskRows, runs, readRuns, apiState, finalApiState };
}
async function refresh(settle = false) {
const observed = settle && evidence.chatIssueId
? await waitForSettledHiringObservation(loadObservation) : await loadObservation();
Object.assign(evidence, { agents: observed.agents, tasks: observed.tasks, runs: observed.runs,
readRuns: observed.readRuns, turnApiState: observed.finalApiState });
}
try {
await api.patch(`${company}/budgets`, { budgetMonthlyCents: 1_000 });
await api.patch(`/api/agents/${f.agent.id}/budgets`, { budgetMonthlyCents: 1_000 });
source = await readHiringTemplateSources(api, f.company.id, f.agent.id);
Object.assign(evidence, source);
await input.evidence("hiring-template-source.json", { ...source, definitionDigest: hiringTemplateDefinitionDigest });
if (Object.entries(source.expectedSourceHashes).some(([file, hash]) => source!.servedSourceHashes[file] !== hash)) {
throw new Error("Hiring-template served source differs from the evaluated revision; comparison is uncomparable");
}
const project = await api.post<{ id: string; name: string }>(`${company}/projects`, {
name: `Label fixtures ${input.nonce}`, description: "Repository-free JSON normalization fixtures.",
});
scenario = hiringTemplateScenario(input.nonce, project.name);
Object.assign(evidence, { hireName: scenario.hireName, marker: scenario.marker, inputs: scenario.inputs, projectId: project.id,
coderExample: renderHiringCoderExample(source.coderReference, scenario.hireName, f.company.name, String((await api.get<{ title: string }>(`/api/agents/${f.agent.id}`)).title), issuePrefix) });
await turn(scenario.initialPrompt, 2);
evidence.chatIssueId = context.issue().id;
const initial = await tasks();
if (initial.length !== 1) throw new Error("Hiring-template first turn must create exactly one coder task");
evidence.first = await readChatOutputDocument(api, initial[0]!.id, scenario.marker) as HiringDocument;
await refresh();
const hired = evidence.agents.find(agent => agent.name === scenario!.hireName);
if (!hired) throw new Error("Hiring-template coder was not created");
evidence.hiredInstructions = await readHiringInstructions(api, hired.id);
evidence.hiredSkills = await readHiringSkillSelections(api, f.company.id, hired.id);
await input.capture("hiring-template-created", "Production CEO hired a coder and delivered its first fixture", "hiring-template-created.png");
await input.evidence("hiring-template-initial.json", evidence);
await turn(scenario.reusePrompt(initial[0]!.identifier ?? initial[0]!.id), 4);
const second = (await tasks()).find(task => task.id !== evidence.first!.issueId);
if (!second) throw new Error("Hiring-template reuse task is absent");
evidence.second = await readChatOutputDocument(api, second.id, `REUSE${scenario.marker}`) as HiringDocument;
evidence.firstAfterReuse = await api.get<HiringDocument>(`/api/issues/${evidence.first.issueId}/documents/${encodeURIComponent(evidence.first.key)}`);
await turn(scenario.statusPrompt(initial[0]!.identifier ?? initial[0]!.id, second.identifier ?? second.id), 5);
evidence.hiredInstructionsAfterReuse = await readHiringInstructions(api, hired.id);
evidence.hiredSkillsAfterReuse = await readHiringSkillSelections(api, f.company.id, hired.id);
await refresh(true);
const result = gradeHiringTemplate(evidence);
for (const check of result.checks) input.check?.(`hiringTemplates.${check.dimension}.${check.id}`, check.passed, check.detail);
await input.evidence("hiring-template.json", { schema: HIRING_TEMPLATE_GRADER_VERSION, definitionDigest: hiringTemplateDefinitionDigest,
budgetGuard: { companyMonthlyCents: 1_000, leadMonthlyCents: 1_000 }, scenario, source, evidence, result });
const failed = result.checks.filter(check => !check.passed);
if (!result.outcomePassed) throw new Error(`Hiring-template workflow outcome failed: ${failed.filter(check => check.dimension === "outcome").map(check => check.id).join(", ")}`);
if (result.comparisonStatus === "uncomparable") throw new Error(`Hiring-template source coverage is uncomparable: ${failed.filter(check => check.dimension === "coverage").map(check => check.id).join(", ")}`);
return { evidence, refresh: async () => { await refresh(true); return evidence; } };
} finally {
let observationError: string | undefined;
try { await refresh(Boolean(scenario)); } catch (error) { observationError = String(error); }
await input.evidence("hiring-template.json", { schema: HIRING_TEMPLATE_GRADER_VERSION, definitionDigest: hiringTemplateDefinitionDigest,
budgetGuard: { companyMonthlyCents: 1_000, leadMonthlyCents: 1_000 },
scenario, source, evidence, result: gradeHiringTemplate(evidence), observationError });
}
}
tests/runner-e2e/hiring-template-scoring.ts
+171
View File
@@ -0,0 +1,171 @@
import { createHash } from "node:crypto";
import { HIRING_TEMPLATE_READ_FILES, HIRING_TEMPLATE_SKILL_KEY } from "./hiring-template-cases.js";
import type { ChatIssue, ChatRun } from "./chat-flow.js";
import { gradeHiringTemplateTurns } from "./hiring-template-turn-accounting.js";
const record = (value: unknown): Record<string, any> =>
value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, any> : {};
export const hiringTemplateHash = (content: string) => createHash("sha256").update(content).digest("hex");
export const hiringTemplateSize = (content: string) => ({ bytes: Buffer.byteLength(content), words: content.trim().split(/\s+/).filter(Boolean).length });
export interface HiringInstructionSnapshot { entryFile: string; mode: string | null; files: Record<string, string> }
export interface HiringReadRun { runId: string; agentId: string; events: Array<{ eventType?: string; payload?: unknown; createdAt?: string }> }
export interface HiringAgent {
id: string; name: string; role?: string; reportsTo?: string | null; adapterType?: string; createdAt?: string;
adapterConfig?: Record<string, any>; runtimeConfig?: Record<string, any>;
}
export interface HiringDocument { issueId: string; key: string; body: string; latestRevisionId: string; createdByAgentId?: string }
export interface HiringTemplateEvidence {
leadId: string; chatIssueId: string; hireName: string; marker: string; projectId: string; inputs: readonly string[];
expectedCeoFiles: Record<string, string>; leadInstructions?: HiringInstructionSnapshot;
expectedSourceHashes: Record<string, string>; servedSourceHashes: Record<string, string>;
assignedSkills: string[]; coderExample: string; hiredInstructions?: HiringInstructionSnapshot;
hiredInstructionsAfterReuse?: HiringInstructionSnapshot; hiredSkills?: unknown; hiredSkillsAfterReuse?: unknown; agents: HiringAgent[];
connectionId: string; binding: unknown; tasks: ChatIssue[]; runs: ChatRun[];
first?: HiringDocument; firstAfterReuse?: HiringDocument; second?: HiringDocument;
readRuns: HiringReadRun[];
/** Independent public chat/comment/run observations for strict lifecycle accounting. */
turnApiState?: unknown;
}
function expectedFixture(inputs: readonly string[], marker: string, separator: "-" | "_") {
return { reference: marker, entries: inputs.map(input => ({
input, value: input.trim().toLowerCase().replace(/[^a-z0-9]+/g, separator).replace(/^[-_]+|[-_]+$/g, ""),
})) };
}
function sameJson(left: unknown, right: unknown): boolean {
if (Array.isArray(left) && Array.isArray(right)) return left.length === right.length && left.every((v, i) => sameJson(v, right[i]));
if (left && right && typeof left === "object" && typeof right === "object") {
const keys = Object.keys(left).sort();
return sameJson(keys, Object.keys(right).sort()) && keys.every(k => sameJson(record(left)[k], record(right)[k]));
}
return left === right;
}
function documentMatches(document: HiringDocument | undefined, expected: unknown) {
try { return sameJson(JSON.parse((document?.body ?? "").trim().replace(/^```(?:json)?\s*/, "").replace(/\s*```$/, "")), expected); }
catch { return false; }
}
function payload(event: { payload?: unknown }) {
const outer = record(event.payload);
return record(outer.prpEvent).payload ? record(record(outer.prpEvent).payload) : outer;
}
function matchingReadPath(value: unknown, file: string) {
return typeof value === "string" && value.replace(/\\/g, "/").endsWith(`/paperclip-create-agent/${file}`);
}
function commandReads(command: unknown, file: string) {
if (typeof command !== "string" || /[|;&><`$\\\r\n]/.test(command)) return false;
// Admit only direct reads with explicit arguments. Unknown options, scripts,
// expansions and help/version output leave source coverage uncomparable.
const matches = [...command.matchAll(/"[^"\n]*"|'[^'\n]*'|[^\s"']+/g)];
if (!matches.length || command.slice(0, matches[0]!.index).trim()) return false;
for (let i = 1; i < matches.length; i++) {
if (!/^\s+$/.test(command.slice(matches[i - 1]!.index! + matches[i - 1]![0].length, matches[i]!.index))) return false;
}
const last = matches.at(-1)!;
if (command.slice(last.index! + last[0].length).trim()) return false;
const tokens = matches.map(match => match[0].replace(/^['"]|['"]$/g, ""));
const program = tokens.shift();
if (program === "head" || program === "tail") {
if (tokens[0] === "-n" || tokens[0] === "-c") {
tokens.shift();
const count = tokens.shift();
if (!count || !/^[1-9]\d*$/.test(count)) return false;
} else if (/^-[nc][1-9]\d*$/.test(tokens[0] ?? "")) tokens.shift();
} else if (program === "sed") {
if (tokens.shift() !== "-n") return false;
if (tokens[0] === "-e") tokens.shift();
const script = tokens.shift() ?? "";
// Only printing from the beginning is supported, without editing or shell
// execution. Other valid sed programs still require different evidence.
if (!/^(?:p|1p|1,[1-9]\d*p)$/.test(script)) return false;
} else if (program !== "cat") return false;
if (tokens[0] === "--") tokens.shift();
return tokens.length > 0 && tokens.every(token => token.length > 0 && !token.startsWith("-") && !/["'*?\[\]]/.test(token))
&& tokens.some(token => matchingReadPath(token, file));
}
/** Only completed provider read operations count; prose and discovery listings do not. */
export function hiringTemplateReadReceipts(readRuns: HiringReadRun[], leadId: string) {
const receipts: Array<{ file: string; runId: string; itemId: string; createdAt?: string }> = [];
for (const run of readRuns.filter(run => run.agentId === leadId)) {
const starts = new Map<string, Record<string, any>>();
for (const event of run.events) {
const p = payload(event), item = record(p.item);
if (event.eventType === "item.started" && item.id) starts.set(item.id, item);
if (event.eventType !== "item.completed" && event.eventType !== "tool.execution.completed") continue;
const start = starts.get(item.tool_use_id ?? item.id) ?? item;
const name = String(start.name ?? "").toLowerCase();
const input = record(start.input);
const result = record(item.result);
const toolRead = item.type === "tool_result" && /^(read|read_file|file_read)$/.test(name)
&& item.isError !== true && item.is_error !== true && result.isError !== true
&& result.is_error !== true && !result.error
&& Object.keys(result).length > 0;
const commandRead = item.type === "commandExecution" && item.exitCode === 0 && item.status === "completed"
&& typeof item.aggregatedOutput === "string" && item.aggregatedOutput.length > 0;
const executionRead = event.eventType === "tool.execution.completed" && p.status === "completed";
const canonicalFileRead = executionRead && p.transport === "builtin" && p.operation === "read" && p.readOnly === true
&& typeof p.outputBytes === "number" && p.outputBytes > 0;
for (const file of HIRING_TEMPLATE_READ_FILES) {
if ((toolRead && [input.file_path, input.path, input.filePath].some(path => matchingReadPath(path, file)))
|| (commandRead && commandReads(item.command, file))
|| (executionRead && p.exitCode === 0 && p.outputBytes > 0 && commandReads(p.name, file))
|| (canonicalFileRead && matchingReadPath(p.target, file))) {
receipts.push({ file, runId: run.runId, itemId: String(item.id ?? p.executionId ?? ""), createdAt: event.createdAt });
}
}
}
}
return receipts;
}
export function gradeHiringTemplate(e: HiringTemplateEvidence) {
const checks: Array<{ id: string; dimension: "outcome" | "coverage"; passed: boolean; detail: string }> = [];
const check = (id: string, dimension: "outcome" | "coverage", passed: boolean, detail: string) => checks.push({ id, dimension, passed, detail });
const hires = e.agents.filter(a => a.id !== e.leadId), hired = hires[0];
const lead = e.agents.find(a => a.id === e.leadId);
check("one-coder-hire", "outcome", hires.length === 1 && hired?.name === e.hireName && hired.role === "engineer"
&& hired.reportsTo === e.leadId && hired.adapterType === "paperclip_runner", "Exactly one permanent coding teammate reports to the lead.");
check("execution-account", "outcome", Boolean(e.connectionId) && Boolean(hired && lead)
&& hired?.adapterConfig?.model === lead?.adapterConfig?.model
&& sameJson(hired?.runtimeConfig?.aiConnection, e.binding), "Hire keeps the lead model and managed account binding.");
const ids = [e.first?.issueId, e.second?.issueId];
check("two-worker-tasks", "outcome", ids.every(Boolean) && new Set(ids).size === 2 && e.tasks.length === 2
&& ids.every(id => e.tasks.some(t => t.id === id && t.status === "done" && t.assigneeAgentId === hired?.id
&& t.projectId === e.projectId && !t.parentId)), "Both independent tasks are completed by the same coder in the chosen project.");
const turnAccounting = gradeHiringTemplateTurns({ evidence: e, apiState: e.turnApiState });
check("bounded-work-and-completion-turns", "outcome", turnAccounting.passed,
"Exactly three requested lead turns and two coder executions, plus at most two strictly attributed completion notifications; every actual run remains counted.");
check("initial-json-artifact", "outcome", documentMatches(e.first, expectedFixture(e.inputs, e.marker, "-"))
&& e.first?.createdByAgentId === hired?.id, "Independent computation checks each original input/value and worker authorship.");
check("reused-json-artifact", "outcome", documentMatches(e.second, expectedFixture(e.inputs, `REUSE${e.marker}`, "_"))
&& e.second?.createdByAgentId === hired?.id, "The reused coder applies the changed separator to every input.");
check("original-preserved", "outcome", Boolean(e.first) && sameJson(e.first, e.firstAfterReuse), "Reuse preserves the original document, revision and task identity.");
const requiredSources = [...HIRING_TEMPLATE_READ_FILES.map(file => `skills/paperclip-create-agent/${file}`),
"skills/paperclip-create-agent/references/baseline-role-guide.md",
...Object.keys(e.expectedCeoFiles).map(file => `server/src/onboarding-assets/ceo/${file}`)];
check("source-fingerprints", "coverage", requiredSources.every(file => /^[a-f0-9]{64}$/.test(e.expectedSourceHashes[file] ?? ""))
&& Object.entries(e.expectedSourceHashes).every(([file, hash]) => e.servedSourceHashes[file] === hash), "Served instruction and hiring source bytes match the evaluated revision.");
check("production-ceo-bundle", "coverage", e.leadInstructions?.mode === "managed" && e.leadInstructions.entryFile === "AGENTS.md"
&& Object.keys(e.expectedCeoFiles).length > 0 && sameJson(e.leadInstructions.files, e.expectedCeoFiles), "The API-created lead receives this revision's default CEO bundle, without a fixture override.");
check("assigned-hiring-skill", "coverage", e.assignedSkills.includes(HIRING_TEMPLATE_SKILL_KEY), "Production CEO defaults assign the hiring skill.");
const receipts = hiringTemplateReadReceipts(e.readRuns, e.leadId);
check("production-source-reads", "coverage", HIRING_TEMPLATE_READ_FILES.every(file => receipts.some(receipt => receipt.file === file && receipt.itemId
&& Date.parse(receipt.createdAt ?? "") <= Date.parse(hired?.createdAt ?? ""))), "Completed lead read receipts before the hire prove the explicitly requested skill, guide, checklist and coder example paths. Unrecognized or missing reads leave coverage uncomparable.");
const instruction = e.hiredInstructions?.files["AGENTS.md"] ?? "";
// The same fixture can compare a long historical role and a short candidate:
// it requires the supplied role example, never a fixed size or new wording.
const expectedCoder = e.coderExample.trim();
check("supplied-coder-instructions", "coverage", expectedCoder.length > 0 && e.hiredInstructions?.mode === "managed"
&& e.hiredInstructions.entryFile === "AGENTS.md" && instruction.trim() === expectedCoder, "Saved hired instructions use the source revision's coder example with its company/name placeholders filled.");
check("hired-instructions-durable", "coverage", Boolean(e.hiredInstructions) && sameJson(e.hiredInstructions, e.hiredInstructionsAfterReuse), "The same saved instruction bundle survives the reused worker execution.");
check("hired-skills-durable", "coverage", Boolean(e.hiredSkills) && sameJson(e.hiredSkills, e.hiredSkillsAfterReuse), "The saved skill selections survive the reused worker execution.");
check("completion-action-attribution", "coverage", turnAccounting.actionEvidence.status !== "uncomparable",
"Notification actions require exact canonical/native identities; missing cross-namespace mapping is uncomparable, not proof of extra work.");
return {
checks, readReceipts: receipts, turnAccounting,
outcomePassed: checks.filter(c => c.dimension === "outcome").every(c => c.passed),
comparisonStatus: checks.filter(c => c.dimension === "coverage").every(c => c.passed) ? "comparable" : "uncomparable",
instructionSizes: { ceo: Object.fromEntries(Object.entries(e.leadInstructions?.files ?? {}).map(([file, content]) => [file, hiringTemplateSize(content)])), coder: hiringTemplateSize(instruction) },
};
}
tests/runner-e2e/hiring-template-turn-accounting.test.ts
+274
View File
@@ -0,0 +1,274 @@
import assert from "node:assert/strict";
import { it } from "vitest";
import { gradeHiringTemplateTurns, HIRING_TEMPLATE_TURN_ACCOUNTING_VERSION,
type HiringTemplateTurnPredicateId } from "./hiring-template-turn-accounting.js";
import { createHiringTemplateTurnFixture as fixture, hiringTurnFixtureTimestamp as timestamp } from "./hiring-template-turn-fixture.js";
type Fixture = ReturnType<typeof fixture>;
type FixtureContext = Fixture["evidence"]["runs"][number]["contextSnapshot"];
const e = (f: Fixture) => f.evidence;
const notification = (f: Fixture) => e(f).runs.find(r => r.id === "notify-1")!;
const worker = (f: Fixture) => e(f).runs.find(r => r.id === "worker-first")!;
const request = (f: Fixture) => e(f).runs.find(r => r.id === "lead-first")!;
const sync = (f: Fixture) => { f.apiState.runs = structuredClone(e(f).runs); return f; };
const fails = (f: { evidence: unknown; apiState: unknown }, predicate?: HiringTemplateTurnPredicateId) => {
const result = gradeHiringTemplateTurns(f);
assert.equal(result.passed, false);
assert.equal(result.predicates.length, 12);
if (predicate) assert.equal(result.predicates.find(p => p.id === predicate)?.passed, false, predicate);
};
// Public positive and negative lifecycle cases ported from the published v1
// retained-evidence sidecar. Its original-grade/provenance wrapper stays separate.
for (const [count, label] of [[0, "five exact work turns"], [1, "six runs with one batched completion"], [2, "seven runs with distinct completions"]] as const) {
it(`accepts ${label} while counting every run toward costs`, () => {
const result = gradeHiringTemplateTurns(fixture(count));
assert.equal(result.passed, true);
assert.equal(result.version, HIRING_TEMPLATE_TURN_ACCOUNTING_VERSION);
assert.equal(result.predicates.length, 12);
assert.ok(result.predicates.every(predicate => predicate.passed));
assert.deepEqual(result.counts, { requiredWorkTurns: 5, maximumCompletionTurns: 2, maximumTotalTurns: 7,
requestedLeadTurns: 3, coderTurns: 2, completionTurns: count, unclassifiedTurns: 0,
snapshotRunCount: 5 + count, actualRunCount: 5 + count, costAccountingRunCount: 5 + count });
assert.deepEqual(Object.keys(result).sort(), ["actionEvidence", "counts", "passed", "predicates", "version"]);
});
}
const negatives: Array<[string, (f: Fixture) => void, HiringTemplateTurnPredicateId]> = [
["an eighth run", f => { e(f).runs.push({ ...structuredClone(notification(f)), id: "extra" }); }, "exact-five-required-work-turns"],
["a duplicate run ID", f => { e(f).runs.push(structuredClone(notification(f))); }, "complete-public-run-ledger"],
["an arbitrary chat wake", f => { notification(f).contextSnapshot.wakeReason = "heartbeat_timer"; }, "exact-five-required-work-turns"],
["a completion label on a coder run", f => { notification(f).agentId = "coder"; }, "exact-five-required-work-turns"],
["an extra requested lead turn", f => { e(f).runs.push({ ...structuredClone(request(f)), id: "lead-extra" }); }, "exact-five-required-work-turns"],
["an extra coder execution", f => { e(f).runs.push({ ...structuredClone(worker(f)), id: "worker-extra" }); }, "exact-five-required-work-turns"],
["a missing requested lead turn", f => { e(f).runs = e(f).runs.filter(r => r.id !== "lead-status"); }, "exact-five-required-work-turns"],
["a missing coder execution", f => { e(f).runs = e(f).runs.filter(r => r.id !== "worker-first"); }, "exact-five-required-work-turns"],
["a missing production lead identity", f => { e(f).agents[0].id = "other"; }, "known-fixture-context"],
["a failed notification", f => { notification(f).status = "failed"; }, "successful-native-without-retries"],
["a legacy notification runtime", f => { notification(f).runtimeMode = "legacy"; }, "successful-native-without-retries"],
["a retried notification", f => { notification(f).retryOfRunId = "prior"; }, "successful-native-without-retries"],
["hidden process retries", f => { notification(f).processLossRetryCount = 1; }, "successful-native-without-retries"],
["a scheduled retry", f => { notification(f).scheduledRetryAttempt = 1; }, "successful-native-without-retries"],
["a continuation run", f => { notification(f).continuationAttempt = 1; }, "successful-native-without-retries"],
["a run in another company", f => { notification(f).companyId = "other"; }, "resolved-company-account-and-identity"],
["a wrong managed account", f => { notification(f).contextSnapshot.aiConnection.connectionId = "other"; }, "resolved-company-account-and-identity"],
["a wrong provider", f => { worker(f).contextSnapshot.aiConnection.provider = "other"; }, "resolved-company-account-and-identity"],
["a wrong auth method", f => { worker(f).contextSnapshot.aiConnection.method = "host"; }, "resolved-company-account-and-identity"],
["a wrong responsible-user binding", f => { notification(f).contextSnapshot.aiConnection.responsibleUserId = "other"; }, "resolved-company-account-and-identity"],
["a wrong run responsible user", f => { request(f).responsibleUserId = "other"; }, "resolved-company-account-and-identity"],
["an accepted foreign identity", f => { notification(f).identityHistory[0].responsibleUserId = "other"; }, "resolved-company-account-and-identity"],
["an identity receipt for another run", f => { notification(f).identityHistory[0].runId = "other"; }, "resolved-company-account-and-identity"],
["a missing instruction receipt", f => { request(f).identityHistory.pop(); }, "three-distinct-requested-chat-turns"],
["a rejected instruction receipt", f => { request(f).identityHistory[1].status = "rejected"; }, "three-distinct-requested-chat-turns"],
["a wrong instruction comment", f => { request(f).identityHistory[1].messageId = "other"; }, "three-distinct-requested-chat-turns"],
["a forged request author", f => { f.apiState.comments[0].authorUserId = "other"; }, "three-distinct-requested-chat-turns"],
["an additional user request", f => { f.apiState.comments.push({ ...f.apiState.comments[0], id: "request-extra" }); }, "three-distinct-requested-chat-turns"],
["two runs for one request comment", f => { e(f).runs.find(r => r.id === "lead-reuse")!.contextSnapshot.wakeCommentId = "request-1"; }, "three-distinct-requested-chat-turns"],
["a different requested chat generation", f => { request(f).contextSnapshot.conversationSessionGeneration = 1; }, "three-distinct-requested-chat-turns"],
["a worker assigned to another agent", f => { e(f).tasks[0].assigneeAgentId = "other"; }, "one-coder-execution-per-known-task"],
["a task outside the selected project", f => { e(f).tasks[0].projectId = "other"; }, "one-coder-execution-per-known-task"],
["a notification-created task", f => { e(f).tasks[0].originRunId = notification(f).id; }, "no-notification-created-extra-tasks"],
["a task created by the status-only turn", f => { e(f).tasks[0].originRunId = "lead-status"; }, "task-origins-are-first-two-requested-turns"],
["an extra task without an execution", f => { e(f).tasks.push({ ...e(f).tasks[0], id: "extra-task" }); }, "no-notification-created-extra-tasks"],
["a wake without delivery IDs", f => { delete notification(f).contextSnapshot.chatCompletionDeliveryIds; }, "bounded-server-completion-receipts"],
["a wake without task updates", f => { delete notification(f).contextSnapshot.chatCompletionUpdates; }, "bounded-server-completion-receipts"],
["mismatched receipt cardinality", f => { notification(f).contextSnapshot.chatCompletionDeliveryIds!.push("extra-delivery"); }, "bounded-server-completion-receipts"],
["a duplicate delivery across runs", f => { e(f).runs.find(r => r.id === "notify-2")!.contextSnapshot.chatCompletionDeliveryIds = notification(f).contextSnapshot.chatCompletionDeliveryIds; }, "bounded-server-completion-receipts"],
["a duplicate task notification with a fresh delivery ID", f => { e(f).runs.find(r => r.id === "notify-2")!.contextSnapshot.chatCompletionUpdates = structuredClone(notification(f).contextSnapshot.chatCompletionUpdates); }, "bounded-server-completion-receipts"],
["an unknown notified task", f => { notification(f).contextSnapshot.chatCompletionUpdates![0].id = "unknown"; }, "bounded-server-completion-receipts"],
["a pre-completion wake", f => { notification(f).startedAt = timestamp(10); }, "bounded-server-completion-receipts"],
["a false completedAt receipt", f => { notification(f).contextSnapshot.chatCompletionUpdates![0].completedAt = timestamp(10); }, "bounded-server-completion-receipts"],
["a false task identifier", f => { notification(f).contextSnapshot.chatCompletionUpdates![0].identifier = "OTHER-1"; }, "bounded-server-completion-receipts"],
["a wrong task link", f => { notification(f).contextSnapshot.chatCompletionUpdates![0].url = "/issues/OTHER-1"; }, "bounded-server-completion-receipts"],
["a false task status", f => { notification(f).contextSnapshot.chatCompletionUpdates![0].status = "in_progress"; }, "bounded-server-completion-receipts"],
["a task without a saved result", f => { notification(f).contextSnapshot.chatCompletionUpdates![0].hasSavedDocuments = false; }, "bounded-server-completion-receipts"],
["a stale completion generation", f => { notification(f).contextSnapshot.conversationSessionGeneration = 1; }, "bounded-server-completion-receipts"],
["a manually invoked completion label", f => { notification(f).invocationSource = "on_demand"; }, "bounded-server-completion-receipts"],
["a mixed request/completion wake", f => { notification(f).contextSnapshot.wakeCommentId = "request-1"; }, "bounded-server-completion-receipts"],
["a missing completion dispatch identity", f => { notification(f).identityHistory = []; }, "bounded-server-completion-receipts"],
["a missing completion reply", f => { f.apiState.comments = f.apiState.comments.filter(c => c.id !== "reply-1"); }, "completion-runs-have-attributed-chat-replies"],
["a reply attributed to a different run", f => { f.apiState.comments.find(c => c.id === "reply-1")!.createdByRunId = "lead-first"; }, "completion-runs-have-attributed-chat-replies"],
["a reply by another agent", f => { f.apiState.comments.find(c => c.id === "reply-1")!.authorAgentId = "coder"; }, "completion-runs-have-attributed-chat-replies"],
["a reply in another chat", f => { f.apiState.comments.find(c => c.id === "reply-1")!.issueId = "other"; }, "completion-runs-have-attributed-chat-replies"],
["a reply before task completion", f => { f.apiState.comments.find(c => c.id === "reply-1")!.createdAt = timestamp(10); }, "completion-runs-have-attributed-chat-replies"],
];
for (const [label, mutate, predicate] of negatives) it(`rejects ${label}`, () => {
const f = fixture(); mutate(f); fails(sync(f), predicate);
});
it("rejects extra runs present only in the final public ledger and counts their cost", () => {
const f = fixture(); f.apiState.runs.push({ ...structuredClone(notification(f)), id: "extra" });
fails(f, "complete-public-run-ledger");
assert.equal(gradeHiringTemplateTurns(f).counts.costAccountingRunCount, 8);
});
it("rejects an account discrepancy between retained ledgers", () => {
const f = fixture(); f.apiState.runs[0]!.contextSnapshot.aiConnection.method = "host";
fails(f, "complete-public-run-ledger");
});
it("does not require a prior grade, artifact outcome, source coverage or report provenance", () => {
const f = fixture();
assert.equal(gradeHiringTemplateTurns({ ...f, result: { outcomePassed: false, comparisonStatus: "uncomparable" } } as typeof f).passed, true);
assert.deepEqual(Object.keys(f).sort(), ["apiState", "evidence"]);
});
it("does not modify observations or expose opaque run content", () => {
const f = fixture(); notification(f).nativeSessionId = "PRIVATE_SESSION";
notification(f).hiddenReasoning = "PRIVATE_REASONING"; notification(f).credential = "PRIVATE_CREDENTIAL";
const before = JSON.stringify(f), result = gradeHiringTemplateTurns(f);
assert.equal(JSON.stringify(f), before);
assert.equal(result.version, HIRING_TEMPLATE_TURN_ACCOUNTING_VERSION);
assert.doesNotMatch(JSON.stringify(result), /PRIVATE_|"company"|"board"|"account"|"notify-1"|"lead-first"/);
});
const missingObservations = [
"evidence.agents", "evidence.tasks", "evidence.runs", "evidence.first", "evidence.second", "evidence.binding", "evidence.connectionId",
"apiState.issue", "apiState.runs", "apiState.comments", "apiState.issue.companyId", "apiState.issue.conversationUserId",
"apiState.issue.conversationAgentId", "apiState.issue.conversationSessionGeneration",
"evidence.runs.0.identityHistory", "evidence.runs.0.contextSnapshot.aiConnection", "evidence.runs.0.startedAt", "evidence.runs.0.finishedAt",
"evidence.tasks.0.completedAt", "evidence.tasks.0.originRunId", "apiState.comments.0.createdAt", "apiState.comments.3.createdByRunId",
];
for (const path of missingObservations) it(`fails closed without ${path}`, () => {
const f = fixture(), keys = path.split(".");
let target = f as unknown as Record<string, unknown>;
for (const key of keys.slice(0, -1)) target = target[key] as Record<string, unknown>;
delete target[keys.at(-1)!];
fails(f);
});
for (const [label, value] of [["null", null], ["undefined", undefined], ["string", "bad"], ["number", 42], ["boolean", true], ["array", []]] as const) {
it(`rejects ${label} evidence without throwing`, () => fails({ evidence: value, apiState: fixture().apiState }));
it(`rejects ${label} API state without throwing`, () => fails({ evidence: fixture().evidence, apiState: value }));
}
it("fails closed on unstringifiable accounting fields", () => {
for (const value of [1n, Object.assign({}, { nested: {} })]) {
const f = fixture();
if (typeof value === "object") value.nested = value;
notification(f).contextSnapshot = { ...notification(f).contextSnapshot, chatCompletionUpdates: value } as unknown as FixtureContext;
fails(sync(f), "complete-public-run-ledger");
}
});
it("retains the observed public cost count when evidence getters throw", () => {
const f = fixture();
const evidence = new Proxy(f.evidence, { get() { throw new Error("Malformed observation"); } });
const result = gradeHiringTemplateTurns({ evidence, apiState: f.apiState });
assert.equal(result.passed, false);
assert.equal(result.counts.costAccountingRunCount, 7);
});
it("rejects a notification write_document despite valid tasks and replies", () => {
const f = fixture();
for (const event of f.evidence.readRuns[0]!.events) {
const payload = event.payload.prpEvent.payload as Record<string, any>;
if (payload.name) payload.name = "write_document";
if (payload.item?.name) payload.item.name = "write_document";
}
fails(f, "completion-turns-only-report-actions");
assert.equal(gradeHiringTemplateTurns(f).actionEvidence.status, "violated");
});
it("rejects a generic call_api document PUT even though it completes successfully", () => {
const f = fixture();
for (const event of f.evidence.readRuns[0]!.events) {
const item = (event.payload.prpEvent.payload as Record<string, any>).item;
if (item?.input) item.input.operationId = "PUT /api/issues/{id}/documents/{key}";
if (item?.result) item.result.apiOperationId = "PUT /api/issues/{id}/documents/{key}";
}
fails(f, "completion-turns-only-report-actions");
assert.equal(gradeHiringTemplateTurns(f).actionEvidence.status, "violated");
});
for (const defect of ["missing-stream", "missing-terminal", "missing-start", "unmatched-result", "separate-namespace",
"api-error", "different-operation", "unknown-tool", "duplicate-result", "sequence-gap"] as const) {
it(`does not certify notification action evidence with ${defect}`, () => {
const f = fixture(), ledger = f.evidence.readRuns[0]!, events = ledger.events;
const start = events[1]!.payload.prpEvent.payload as Record<string, any>;
const result = events[2]!.payload.prpEvent.payload as Record<string, any>;
if (defect === "missing-stream") f.evidence.readRuns = [];
if (defect === "missing-terminal") ledger.events = events.slice(0, -1);
if (defect === "missing-start") ledger.events = events.filter(e => e.eventType !== "item.started");
if (defect === "unmatched-result") result.item.tool_use_id = "other";
if (defect === "separate-namespace") { start.item.id = result.item.id = result.item.tool_use_id = "host-request"; }
if (defect === "api-error") result.item.result.ok = false;
if (defect === "different-operation") result.item.result.apiOperationId = "GET /api/issues/{id}";
if (defect === "unknown-tool") (events[0]!.payload.prpEvent.payload as Record<string, any>).name = "unknown";
if (defect === "duplicate-result") { ledger.events.splice(3, 0, structuredClone(events[2]!)); ledger.events.forEach((e, i) => e.seq = i + 1); }
if (defect === "sequence-gap") events[2]!.seq++;
fails(f, "completion-turns-only-report-actions");
});
}
it("allows successful native readonly reads without claiming source identity", () => {
const f = fixture();
for (const ledger of f.evidence.readRuns) ledger.events = ledger.events.filter(event => !event.eventType.startsWith("item."))
.map((event, i) => { const payload = event.payload.prpEvent.payload as Record<string, any>;
if (payload.name) Object.assign(payload, { name: "Read File", operation: "read", readOnly: true, transport: "builtin" });
return { ...event, seq: i + 1 }; });
assert.equal(gradeHiringTemplateTurns(f).passed, true);
});
for (const [label, value] of [["non-2xx API result", { status: 500 }], ["explicit API error", { error: "failed" }],
["API is_error flag", { is_error: true }], ["API isError flag", { isError: true }]] as const) {
it(`rejects ${label}`, () => {
const f = fixture();
Object.assign((f.evidence.readRuns[0]!.events[2]!.payload.prpEvent.payload as Record<string, any>).item.result, value);
fails(f, "completion-turns-only-report-actions");
});
}
it("does not trust a readonly hint on an unattributed call_api", () => {
const f = fixture(), ledger = f.evidence.readRuns[0]!;
ledger.events = ledger.events.filter(event => !event.eventType.startsWith("item."));
ledger.events.forEach((event, i) => {
event.seq = i + 1;
const payload = event.payload.prpEvent.payload as Record<string, any>;
if (payload.name) Object.assign(payload, { operation: "read", readOnly: true });
});
fails(f, "completion-turns-only-report-actions");
});
for (const [transport, operation, readOnly] of [["dynamic", "unknown", null], ["mcp", "execute", false]] as const) {
it(`rejects ${transport} write_document with its actual production ${operation} shape`, () => {
const f = fixture();
for (const event of f.evidence.readRuns[0]!.events) {
const payload = event.payload.prpEvent.payload as Record<string, any>;
if (payload.name) Object.assign(payload, { name: "write_document", transport, operation, readOnly });
if (payload.item?.name) payload.item.name = "write_document";
}
fails(f, "completion-turns-only-report-actions");
assert.equal(gradeHiringTemplateTurns(f).actionEvidence.status, "violated");
});
}
it("rejects an attempted mutation even if it fails and a benign read follows", () => {
const f = fixture(), ledger = f.evidence.readRuns[0]!;
const mutation = structuredClone(ledger.events.slice(0, 4));
for (const event of mutation) {
const payload = event.payload.prpEvent.payload as Record<string, any>;
if (payload.executionId) payload.executionId = "mutation";
if (payload.item) { payload.item.id = "mutation"; if (payload.item.tool_use_id) payload.item.tool_use_id = "mutation"; }
if (payload.item?.input) payload.item.input.operationId = "POST /api/companies/{companyId}/issues";
if (payload.item?.result) Object.assign(payload.item.result, { apiOperationId: "POST /api/companies/{companyId}/issues", ok: false, status: 403 });
}
ledger.events.unshift(...mutation);
ledger.events.forEach((event, i) => event.seq = i + 1);
fails(f, "completion-turns-only-report-actions");
assert.equal(gradeHiringTemplateTurns(f).actionEvidence.status, "violated");
});
it("requires an accepted disposition and succeeded terminal in the action stream", () => {
for (const defect of ["accepted", "terminal"] as const) {
const f = fixture(), events = f.evidence.readRuns[0]!.events;
const payload = events[defect === "accepted" ? 4 : 5]!.payload.prpEvent.payload as Record<string, any>;
if (defect === "accepted") payload.result.schema = "wrong";
else payload.runTerminalState = "failed";
fails(f, "completion-turns-only-report-actions");
}
});
it("admits a correlated native finish without forcing Done versus yielded chat disposition", () => {
for (const disposition of ["done", "yielded"]) {
const f = fixture(), ledger = f.evidence.readRuns[0]!;
const start = structuredClone(ledger.events[0]!), finish = structuredClone(ledger.events[3]!);
for (const event of [start, finish]) Object.assign(event.payload.prpEvent.payload,
{ executionId: "finish", name: "paperclip_finish", transport: "dynamic" });
const proposal = { seq: 0, eventType: "run.result.proposed", payload: { prpEvent: {
schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", itemId: "finish",
payload: { schema: "paperclip.run_result.v1", reportedWorkDisposition: disposition },
} } };
ledger.events.splice(4, 0, start, proposal as unknown as typeof start, finish);
ledger.events.forEach((event, i) => event.seq = i + 1);
assert.equal(gradeHiringTemplateTurns(f).passed, true);
(proposal.payload.prpEvent as Record<string, unknown>).itemId = "unrelated";
fails(f, "completion-turns-only-report-actions");
}
});
tests/runner-e2e/hiring-template-turn-accounting.ts
+317
View File
@@ -0,0 +1,317 @@
/** Strict public lifecycle accounting for the bounded hiring-template journey.
* Three requested CEO turns and two coder executions remain exact. Up to two
* server completion turns are separately admitted and still count toward costs.
* Outcome artifacts and source/read coverage are graded by their own checks.
*/
export const HIRING_TEMPLATE_TURN_ACCOUNTING_VERSION = "paperclip.hiring-template-turn-accounting.v2";
const predicateIds = [
"known-fixture-context",
"complete-public-run-ledger",
"exact-five-required-work-turns",
"successful-native-without-retries",
"resolved-company-account-and-identity",
"three-distinct-requested-chat-turns",
"one-coder-execution-per-known-task",
"task-origins-are-first-two-requested-turns",
"bounded-server-completion-receipts",
"completion-runs-have-attributed-chat-replies",
"no-notification-created-extra-tasks",
"completion-turns-only-report-actions",
] as const;
export type HiringTemplateTurnPredicateId = typeof predicateIds[number];
export interface HiringTemplateTurnPredicate { id: HiringTemplateTurnPredicateId; passed: boolean }
export interface HiringTemplateTurnCounts {
requiredWorkTurns: 5; maximumCompletionTurns: 2; maximumTotalTurns: 7;
requestedLeadTurns: number; coderTurns: number; completionTurns: number; unclassifiedTurns: number;
snapshotRunCount: number; actualRunCount: number; costAccountingRunCount: number;
}
export interface HiringTemplateTurnAccountingResult {
version: typeof HIRING_TEMPLATE_TURN_ACCOUNTING_VERSION;
counts: HiringTemplateTurnCounts;
predicates: HiringTemplateTurnPredicate[];
passed: boolean;
actionEvidence: HiringCompletionActionEvidence;
}
export interface HiringCompletionActionEvidence {
status: "verified" | "violated" | "uncomparable"; notificationRuns: number;
canonicalExecutions: number; matchedNativeApiCalls: number; unknownExecutions: number;
}
type JsonRecord = Record<string, unknown>;
const object = (value: unknown): JsonRecord =>
value && typeof value === "object" && !Array.isArray(value) ? value as JsonRecord : {};
const rows = (value: unknown): JsonRecord[] => Array.isArray(value) ? value.map(object) : [];
const present = (value: unknown): value is string => typeof value === "string" && value.trim().length > 0;
const unique = (values: readonly unknown[]) => values.every(present) && new Set(values).size === values.length;
const sameSet = (a: readonly unknown[], b: readonly unknown[]) =>
a.length === b.length && unique(a) && unique(b) && a.every(v => b.includes(v));
const date = (value: unknown) => typeof value === "string" ? Date.parse(value) : NaN;
const zero = (value: unknown) => value == null || value === 0;
const noList = (value: unknown) => value == null || Array.isArray(value) && value.length === 0;
const context = (run: JsonRecord) => object(run.contextSnapshot);
const accepted = (run: JsonRecord) => rows(run.identityHistory).filter(i => i.status === "accepted");
const dispatchIdentity = (run: JsonRecord, cause: string, user: unknown) => accepted(run).some(i => i.cause === cause
&& i.responsibleUserId === user && !i.messageId && Number.isFinite(date(i.acceptedAt)));
// Compare only public accounting fields; malformed projections fail closed.
function accountingProjection(run: JsonRecord): string | null {
try {
return JSON.stringify({ id: run.id, companyId: run.companyId,
agentId: run.agentId, status: run.status, runtimeMode: run.runtimeMode,
responsibleUserId: run.responsibleUserId, invocationSource: run.invocationSource,
triggerDetail: run.triggerDetail, startedAt: run.startedAt, finishedAt: run.finishedAt,
retryOfRunId: run.retryOfRunId, processLossRetryCount: run.processLossRetryCount,
scheduledRetryAttempt: run.scheduledRetryAttempt, continuationAttempt: run.continuationAttempt,
context: Object.fromEntries(["source", "issueId", "wakeReason", "wakeSource", "wakeTriggerDetail", "wakeCommentId",
"wakeCommentIds", "conversationSessionGeneration", "chatCompletionDeliveryIds", "chatCompletionUpdates"]
.map(key => [key, context(run)[key]])),
account: Object.fromEntries(["connectionId", "responsibleUserId", "provider", "method", "mode"]
.map(key => [key, object(context(run).aiConnection)[key]])),
identities: rows(run.identityHistory).map(i => Object.fromEntries(["runId", "companyId", "responsibleUserId",
"status", "cause", "messageId", "acceptedAt"].map(key => [key, i[key]]))) });
} catch { return null; }
}
const notificationReadApiOperations = new Set([
"GET /api/issues/{id}", "GET /api/issues/{id}/comments", "GET /api/issues/{id}/documents", "GET /api/issues/{id}/documents/{key}",
]);
const discoveryNames = new Set(["search_api", "search_tasks"]);
const forbiddenWorkTools = new Set(["write_document", "create_task", "hire_agent", "register_deliverable", "create_skill",
"update_agent_instructions", "restore_agent_instructions", "create_project", "reassign_task", "set_dependencies", "set_task_title"]);
const errored = (value: JsonRecord) => Boolean(value.error || value.is_error || value.isError);
function eventEnvelope(event: JsonRecord) { return object(object(event.payload).prpEvent); }
/** Only exact per-run IDs join provider executions to successful native actions.
* ACPX host request IDs and provider stream IDs cannot be joined by order/name.
*/
export function gradeHiringCompletionActions(notifications: JsonRecord[], readRuns: unknown): HiringCompletionActionEvidence {
const ledgers = rows(readRuns);
const result: HiringCompletionActionEvidence = { status: "verified", notificationRuns: notifications.length,
canonicalExecutions: 0, matchedNativeApiCalls: 0, unknownExecutions: 0 };
const unresolved = () => { result.unknownExecutions++; if (result.status !== "violated") result.status = "uncomparable"; };
const violation = () => { result.status = "violated"; };
for (const run of notifications) {
const matching = ledgers.filter(ledger => ledger.runId === run.id && ledger.agentId === run.agentId);
if (matching.length !== 1) { unresolved(); continue; }
const events = rows(matching[0]!.events), terminals = events.filter(event => event.eventType === "run.terminal");
const acceptedResults = events.filter(event => event.eventType === "run.result.accepted");
const terminalEnvelope = eventEnvelope(terminals[0] ?? {}), terminal = object(terminalEnvelope.payload);
const acceptedEnvelope = eventEnvelope(acceptedResults[0] ?? {});
if (!events.length || terminals.length !== 1 || acceptedResults.length !== 1
|| terminalEnvelope.sourceKind !== "control_plane" || terminal.schema !== "paperclip.prp.terminal.v1"
|| terminal.runTerminalState !== "succeeded" || terminal.turnTerminalState !== "completed"
|| acceptedEnvelope.sourceKind !== "control_plane" || object(object(acceptedEnvelope.payload).result).schema !== "paperclip.run_result.v1"
|| !events.every((event, i) => Number.isSafeInteger(event.seq) && event.seq === i + 1)) { unresolved(); continue; }
const starts = new Map<unknown, JsonRecord>(), completed = new Map<unknown, JsonRecord>();
const nativeStarts = new Map<unknown, JsonRecord>(), nativeResults = new Map<unknown, JsonRecord>();
for (const event of events) {
const envelope = eventEnvelope(event), payload = object(envelope.payload), item = object(payload.item);
if (event.eventType?.toString().startsWith("tool.execution.")) {
if (payload.schema !== "paperclip.tool.execution.v1" || !present(payload.executionId)) { unresolved(); continue; }
if (event.eventType === "tool.execution.started") {
if (starts.has(payload.executionId)) violation();
starts.set(payload.executionId, payload);
} else if (event.eventType === "tool.execution.completed") {
if (completed.has(payload.executionId) || payload.status !== "completed" || errored(payload)
|| payload.exitCode != null && payload.exitCode !== 0) violation();
completed.set(payload.executionId, payload);
} else if (["tool.execution.failed", "tool.execution.cancelled", "tool.execution.interrupted"].includes(String(event.eventType))) violation();
}
if (item.type !== "tool_use" && item.type !== "tool_result") continue;
if (envelope.schema !== "paperclip.prp.event.v1" || envelope.schemaVersion !== 1 || envelope.sourceKind !== "runner"
|| !present(item.id) || errored(item)) { unresolved(); continue; }
if (event.eventType === "item.started" && item.type === "tool_use") {
if (nativeStarts.has(item.id)) violation();
nativeStarts.set(item.id, item);
} else if (event.eventType === "item.completed" && item.type === "tool_result") {
if (item.tool_use_id !== item.id || nativeResults.has(item.id) || errored(object(item.result))) violation();
nativeResults.set(item.id, item);
} else unresolved();
}
if (!sameSet([...starts.keys()], [...completed.keys()]) || !sameSet([...nativeStarts.keys()], [...nativeResults.keys()])) unresolved();
for (const [id, start] of nativeStarts) {
const receipt = object(nativeResults.get(id)?.result), input = object(start.input);
if (start.name === "call_api") {
if (present(input.operationId) && !notificationReadApiOperations.has(input.operationId)) violation();
if (!present(input.operationId) || input.operationId !== receipt.apiOperationId || receipt.ok !== true
|| !Number.isInteger(receipt.status) || Number(receipt.status) < 200 || Number(receipt.status) >= 300 || errored(receipt)) { unresolved(); continue; }
if (!notificationReadApiOperations.has(input.operationId)) violation();
} else if (start.name === "search_api") {
if (!Array.isArray(receipt.results) || !Number.isInteger(receipt.total) || Number(receipt.total) < 0 || typeof receipt.guidance !== "string"
|| receipt.nextCursor != null && typeof receipt.nextCursor !== "string") unresolved();
} else if (start.name === "search_tasks") {
if (!Array.isArray(receipt.tasks)) unresolved();
} else if (forbiddenWorkTools.has(String(start.name))) violation();
else unresolved();
}
for (const [id, start] of starts) {
result.canonicalExecutions++;
const finish = completed.get(id) ?? {}, native = nativeStarts.get(id);
if (start.name !== finish.name) { unresolved(); continue; }
if (forbiddenWorkTools.has(String(start.name))) { violation(); continue; }
if (start.name === "paperclip_finish") {
const proposals = events.filter(event => event.eventType === "run.result.proposed");
const proposal = eventEnvelope(proposals[0] ?? {});
if (start.transport !== "dynamic" || finish.transport !== "dynamic" || proposals.length !== 1
|| proposal.sourceKind !== "runner" || proposal.itemId !== id
|| object(proposal.payload).schema !== "paperclip.run_result.v1") unresolved();
continue;
}
if (start.name === "ToolSearch" && start.transport === "builtin" && finish.transport === "builtin") continue;
if (start.name === "call_api" || discoveryNames.has(String(start.name))) {
if (!native || native.name !== start.name) { unresolved(); continue; }
if (start.name === "call_api") result.matchedNativeApiCalls++;
} else if (start.readOnly === true && finish.readOnly === true && ["read", "search", "list"].includes(String(start.operation))
&& start.operation === finish.operation) continue;
else unresolved();
}
// Every native action must also have an exact canonical provider identity.
if ([...nativeStarts.keys()].some(id => !starts.has(id))) unresolved();
}
return result;
}
function evaluate(evidence: unknown, apiState: unknown): HiringTemplateTurnAccountingResult {
const e = object(evidence), api = object(apiState);
const chat = object(api.issue), user = chat.conversationUserId, company = chat.companyId;
const runs = rows(e.runs), publicRuns = rows(api.runs), comments = rows(api.comments);
const agents = rows(e.agents), tasks = rows(e.tasks), lead = e.leadId;
const coder = agents.find(a => a.id !== lead && a.name === e.hireName);
const taskIds = [object(e.first).issueId, object(e.second).issueId];
const taskById = new Map(tasks.map(t => [t.id, t]));
const requests = comments.filter(c => c.authorUserId).sort((a, b) => date(a.createdAt) - date(b.createdAt));
const requested: JsonRecord[] = [], workers: JsonRecord[] = [], notifications: JsonRecord[] = [], unknown: JsonRecord[] = [];
for (const run of runs) {
const c = context(run);
if (run.agentId === lead && c.issueId === e.chatIssueId && c.wakeReason === "issue_commented") requested.push(run);
else if (run.agentId === coder?.id && taskIds.includes(c.issueId) && c.wakeReason === "issue_assigned") workers.push(run);
else if (run.agentId === lead && c.issueId === e.chatIssueId && c.wakeReason === "chat_task_completed") notifications.push(run);
else unknown.push(run);
}
const predicates: HiringTemplateTurnPredicate[] = [];
const check = (id: HiringTemplateTurnPredicateId, passed: unknown) => predicates.push({ id, passed: Boolean(passed) });
const binding = object(e.binding);
check("known-fixture-context", present(company) && present(user) && present(lead) && present(coder?.id)
&& chat.id === e.chatIssueId && chat.conversationAgentId === lead
&& typeof chat.conversationSessionGeneration === "number" && Number.isInteger(chat.conversationSessionGeneration) && chat.conversationSessionGeneration >= 0
&& agents.length === 2 && agents.some(a => a.id === lead)
&& unique(agents.map(a => a.id)) && agents.every(a => a.companyId === company)
&& present(e.connectionId) && present(binding.provider) && present(binding.method) && binding.mode === "responsible_user"
&& unique(taskIds) && unique(tasks.map(t => t.id)) && sameSet(taskIds, tasks.map(t => t.id)));
check("complete-public-run-ledger", sameSet(runs.map(r => r.id), publicRuns.map(r => r.id))
&& runs.every(r => {
const projection = accountingProjection(r);
return projection !== null && projection === accountingProjection(publicRuns.find(p => p.id === r.id) ?? {});
}));
check("exact-five-required-work-turns", requested.length === 3 && workers.length === 2 && unknown.length === 0
&& notifications.length <= 2 && runs.length === 5 + notifications.length && runs.length <= 7);
check("successful-native-without-retries", runs.length > 0 && runs.every(r => r.status === "succeeded"
&& r.runtimeMode === "native" && Number.isFinite(date(r.startedAt)) && Number.isFinite(date(r.finishedAt))
&& date(r.finishedAt) >= date(r.startedAt) && !r.retryOfRunId && zero(r.processLossRetryCount)
&& zero(r.scheduledRetryAttempt) && zero(r.continuationAttempt)));
check("resolved-company-account-and-identity", runs.length > 0 && runs.every(r => {
const account = object(context(r).aiConnection), identities = accepted(r);
return r.companyId === company && r.responsibleUserId === user && account.responsibleUserId === user
&& account.connectionId === e.connectionId && ["provider", "method", "mode"].every(k => account[k] === binding[k])
&& identities.length > 0 && identities.every(i => i.responsibleUserId === user
&& i.runId === r.id && i.companyId === company);
}));
check("three-distinct-requested-chat-turns", requests.length === 3 && unique(requests.map(c => c.id))
&& sameSet(requested.map(r => context(r).wakeCommentId), requests.map(c => c.id))
&& requests.every(c => c.companyId === company && c.issueId === chat.id && c.authorUserId === user
&& !c.authorAgentId && present(c.body) && Number.isFinite(date(c.createdAt)))
&& requested.every(r => {
const c = context(r), comment = requests.find(p => p.id === c.wakeCommentId);
return r.invocationSource === "on_demand" && r.triggerDetail === "manual" && c.source === "issue.comment"
&& c.wakeSource === "on_demand" && c.wakeTriggerDetail === "manual"
&& c.conversationSessionGeneration === chat.conversationSessionGeneration
&& noList(c.chatCompletionDeliveryIds) && noList(c.chatCompletionUpdates)
&& sameSet(Array.isArray(c.wakeCommentIds) ? c.wakeCommentIds : [], [c.wakeCommentId])
&& comment && date(r.startedAt) >= date(comment.createdAt)
&& dispatchIdentity(r, "issue_commented", user)
&& accepted(r).some(i => i.cause === "instruction" && i.messageId === c.wakeCommentId
&& i.responsibleUserId === user && Number.isFinite(date(i.acceptedAt)));
}));
check("one-coder-execution-per-known-task", tasks.length === 2 && taskIds.every(id => {
const task = taskById.get(id), matching = workers.filter(r => context(r).issueId === id), run = matching[0] ?? {};
const c = context(run ?? {});
return task && matching.length === 1 && task.companyId === company && task.assigneeAgentId === coder?.id
&& task.projectId === e.projectId && !task.parentId && task.status === "done" && task.responsibleUserId === user
&& Number.isFinite(date(task.completedAt)) && date(task.completedAt) >= date(run.startedAt)
&& run.invocationSource === "assignment" && run.triggerDetail === "system"
&& c.source === "paperclip_runner.create_task" && c.wakeSource === "assignment" && c.wakeTriggerDetail === "system"
&& noList(c.chatCompletionDeliveryIds) && noList(c.chatCompletionUpdates)
&& dispatchIdentity(run, "issue_assigned", user);
}));
check("task-origins-are-first-two-requested-turns", taskIds.every((id, index) => {
const task = taskById.get(id), comment = requests[index];
const run = requested.find(r => context(r).wakeCommentId === comment?.id);
return task && run && task.createdByAgentId === lead && !task.createdByUserId && task.originRunId === run.id
&& date(task.createdAt) >= date(run.startedAt);
}));
const deliveryIds = new Set<unknown>(), notifiedTasks = new Set<unknown>();
check("bounded-server-completion-receipts", notifications.every(r => {
const c = context(r), ids = Array.isArray(c.chatCompletionDeliveryIds) ? c.chatCompletionDeliveryIds : [];
const updates = rows(c.chatCompletionUpdates);
let valid = r.invocationSource === "automation" && r.triggerDetail === "system"
&& c.wakeSource === "automation" && c.wakeTriggerDetail === "system" && !c.source
&& c.conversationSessionGeneration === chat.conversationSessionGeneration
&& !c.wakeCommentId && noList(c.wakeCommentIds)
&& ids.length > 0 && ids.length <= 2 && ids.length === updates.length && unique(ids)
&& ids.every(id => !deliveryIds.has(id)) && dispatchIdentity(r, "chat_task_completed", user);
for (const update of updates) {
const task = taskById.get(update.id);
valid = Boolean(valid && task && !notifiedTasks.has(update.id) && update.status === "done"
&& task.status === "done" && update.identifier === task.identifier && update.completedAt === task.completedAt
&& present(task.identifier) && update.url === `/issues/${task.identifier}` && update.hasSavedDocuments === true
&& date(r.startedAt) >= date(task.completedAt));
notifiedTasks.add(update.id);
}
ids.forEach(id => deliveryIds.add(id));
return valid;
}));
check("completion-runs-have-attributed-chat-replies", notifications.every(r => {
const updates = rows(context(r).chatCompletionUpdates);
const completedAt = Math.max(...updates.map(u => date(u.completedAt)));
return updates.length > 0 && comments.some(c => c.companyId === company && c.issueId === chat.id
&& c.authorAgentId === lead && !c.authorUserId && c.createdByRunId === r.id && present(c.body)
&& (c.conversationSessionGeneration == null || c.conversationSessionGeneration === chat.conversationSessionGeneration)
&& date(c.createdAt) >= completedAt && date(c.createdAt) >= date(r.startedAt));
}));
check("no-notification-created-extra-tasks", tasks.length === 2 && tasks.every(t =>
!notifications.some(r => r.id === t.originRunId)));
const actionEvidence = gradeHiringCompletionActions(notifications, e.readRuns);
check("completion-turns-only-report-actions", actionEvidence.status === "verified");
return {
version: HIRING_TEMPLATE_TURN_ACCOUNTING_VERSION,
counts: { requiredWorkTurns: 5, maximumCompletionTurns: 2, maximumTotalTurns: 7,
requestedLeadTurns: requested.length, coderTurns: workers.length, completionTurns: notifications.length,
unclassifiedTurns: unknown.length, snapshotRunCount: runs.length, actualRunCount: publicRuns.length,
costAccountingRunCount: publicRuns.length },
predicates, actionEvidence, passed: predicates.every(predicate => predicate.passed),
};
}
/** Accept untrusted evidence observations without needing an existing result. */
export function gradeHiringTemplateTurns(input: { evidence: unknown; apiState: unknown }): HiringTemplateTurnAccountingResult {
let snapshotRunCount = 0, actualRunCount = 0;
try {
const runs = object(object(input).apiState).runs;
actualRunCount = Array.isArray(runs) ? runs.length : 0;
} catch { /* An inaccessible public ledger is a failed observation. */ }
try {
const runs = object(object(input).evidence).runs;
snapshotRunCount = Array.isArray(runs) ? runs.length : 0;
} catch { /* Keep public cost accounting when the evidence is malformed. */ }
try {
const value = object(input);
return evaluate(value.evidence, value.apiState);
} catch {
return {
version: HIRING_TEMPLATE_TURN_ACCOUNTING_VERSION,
counts: { requiredWorkTurns: 5, maximumCompletionTurns: 2, maximumTotalTurns: 7,
requestedLeadTurns: 0, coderTurns: 0, completionTurns: 0, unclassifiedTurns: snapshotRunCount,
snapshotRunCount, actualRunCount, costAccountingRunCount: actualRunCount },
predicates: predicateIds.map(id => ({ id, passed: false })), passed: false,
actionEvidence: { status: "uncomparable", notificationRuns: 0, canonicalExecutions: 0, matchedNativeApiCalls: 0, unknownExecutions: 0 },
};
}
}
tests/runner-e2e/hiring-template-turn-fixture.ts
+110
View File
@@ -0,0 +1,110 @@
// Test-only public lifecycle facts shared by turn and outcome calibrations.
interface FixtureIdentity {
runId: string; companyId: string; responsibleUserId: string; status: string;
cause: string; messageId: string | null; acceptedAt: string;
}
interface FixtureUpdate {
id: string; identifier: string; status: string; completedAt: string;
url: string; hasSavedDocuments: boolean;
}
interface FixtureRun {
id: string; companyId: string; agentId: string; responsibleUserId: string;
invocationSource: string; triggerDetail: string; status: string; runtimeMode: string;
startedAt: string; finishedAt: string; retryOfRunId: string | null;
processLossRetryCount: number; scheduledRetryAttempt: number; continuationAttempt: number;
identityHistory: FixtureIdentity[];
contextSnapshot: {
source?: string; issueId: string; wakeReason: string; wakeSource: string;
wakeTriggerDetail: string; conversationSessionGeneration: number;
wakeCommentId?: string; wakeCommentIds: string[];
aiConnection: { provider: string; method: string; mode: string; connectionId: string; responsibleUserId: string };
chatCompletionDeliveryIds?: string[]; chatCompletionUpdates?: FixtureUpdate[];
};
[key: string]: unknown;
}
interface FixtureComment {
id: string; companyId: string; issueId: string; authorUserId: string | null;
authorAgentId: string | null; body: string; createdAt: string;
createdByRunId?: string; conversationSessionGeneration?: number | null;
}
export const hiringTurnFixtureTimestamp = (seconds: number) => new Date(Date.UTC(2026, 9, 2, 0, 0, seconds)).toISOString();
/** Complete canonical/native read-action stream with one exact execution ID. */
export function hiringNotificationActionEvents() {
const id = "read-action";
const payload = { schema: "paperclip.tool.execution.v1", executionId: id, transport: "dynamic",
operation: "unknown", name: "call_api", readOnly: null, status: "running", exitCode: null };
const items: Array<{ eventType: string; payload: Record<string, unknown> }> = [
{ eventType: "tool.execution.started", payload },
{ eventType: "item.started", payload: { kind: "tool", item: { id, type: "tool_use", name: "call_api",
input: { operationId: "GET /api/issues/{id}/documents", pathParams: {} } } } },
{ eventType: "item.completed", payload: { kind: "tool", item: { id, tool_use_id: id, type: "tool_result",
result: { apiOperationId: "GET /api/issues/{id}/documents", ok: true, status: 200 } } } },
{ eventType: "tool.execution.completed", payload: { ...payload, status: "completed" } },
{ eventType: "run.result.accepted", payload: { result: { schema: "paperclip.run_result.v1" } } },
{ eventType: "run.terminal", payload: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState: "completed" } },
];
return items.map((event, i) => ({ seq: i + 1, eventType: event.eventType, payload: { prpEvent: {
schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: event.eventType.startsWith("run.") ? "control_plane" : "runner", payload: event.payload,
} } }));
}
export function createHiringTemplateTurnFixture(notificationCount = 2) {
if (!Number.isInteger(notificationCount) || notificationCount < 0 || notificationCount > 2)
throw new RangeError("The positive lifecycle fixture accepts zero, one or two notification turns.");
const timestamp = hiringTurnFixtureTimestamp;
const company = "company", user = "board", lead = "lead", coder = "coder", chat = "chat";
const binding = { provider: "anthropic", method: "api_key", mode: "responsible_user" };
function run(id: string, agentId: string, issueId: string, reason: string, start: number, finish: number, source?: string): FixtureRun {
const invocationSource = reason === "issue_commented" ? "on_demand"
: reason === "issue_assigned" ? "assignment" : "automation";
const triggerDetail = reason === "issue_commented" ? "manual" : "system";
return { id, companyId: company, agentId, responsibleUserId: user, invocationSource, triggerDetail,
status: "succeeded", runtimeMode: "native", startedAt: timestamp(start), finishedAt: timestamp(finish),
retryOfRunId: null, processLossRetryCount: 0, scheduledRetryAttempt: 0, continuationAttempt: 0,
identityHistory: [{ runId: id, companyId: company, responsibleUserId: user, status: "accepted",
cause: reason, messageId: null, acceptedAt: timestamp(start) }],
contextSnapshot: { source, issueId, wakeReason: reason, wakeSource: invocationSource,
wakeTriggerDetail: triggerDetail, conversationSessionGeneration: 0, wakeCommentIds: [],
aiConnection: { ...binding, connectionId: "account", responsibleUserId: user } } };
}
const requested = [run("lead-first", lead, chat, "issue_commented", 1, 10, "issue.comment"),
run("lead-reuse", lead, chat, "issue_commented", 21, 30, "issue.comment"),
run("lead-status", lead, chat, "issue_commented", 51, 55, "issue.comment")];
const comments: FixtureComment[] = requested.map((r, i) => {
const id = `request-${i + 1}`;
r.contextSnapshot.wakeCommentId = id;
r.contextSnapshot.wakeCommentIds = [id];
r.identityHistory.push({ runId: r.id, companyId: company, responsibleUserId: user, status: "accepted",
cause: "instruction", messageId: id, acceptedAt: r.startedAt });
return { id, companyId: company, issueId: chat, authorUserId: user, authorAgentId: null,
body: `Fixture request ${i + 1}`, createdAt: timestamp([0, 20, 50][i]!) };
});
const workers = [run("worker-first", coder, "first-task", "issue_assigned", 4, 12, "paperclip_runner.create_task"),
run("worker-reuse", coder, "second-task", "issue_assigned", 25, 40, "paperclip_runner.create_task")];
const tasks = workers.map((r, i) => ({ id: r.contextSnapshot.issueId, companyId: company, projectId: "project",
title: r.contextSnapshot.issueId, parentId: null as string | null, status: "done", identifier: `RUN-${i + 2}`, responsibleUserId: user,
assigneeAgentId: coder, createdByAgentId: lead, createdByUserId: null as string | null, originRunId: requested[i]!.id,
createdAt: timestamp([2, 22][i]!), completedAt: timestamp([11, 39][i]!) }));
const notifications: FixtureRun[] = [];
for (let i = 0; i < notificationCount; i++) {
const r = run(`notify-${i + 1}`, lead, chat, "chat_task_completed", notificationCount === 1 ? 41 : [13, 41][i]!,
notificationCount === 1 ? 45 : [18, 45][i]!);
const covered = notificationCount === 1 ? tasks : [tasks[i]!];
r.contextSnapshot.chatCompletionDeliveryIds = covered.map(t => `delivery-${t.id}`);
r.contextSnapshot.chatCompletionUpdates = covered.map(t => ({ id: t.id, identifier: t.identifier,
status: "done", completedAt: t.completedAt, url: `/issues/${t.identifier}`, hasSavedDocuments: true }));
comments.push({ id: `reply-${i + 1}`, companyId: company, issueId: chat, authorAgentId: lead,
authorUserId: null, createdByRunId: r.id, body: "The saved fixture is complete.",
conversationSessionGeneration: null, createdAt: r.finishedAt });
notifications.push(r);
}
const runs = [...requested, ...workers, ...notifications];
const evidence = { leadId: lead, chatIssueId: chat, hireName: "Fixture Coder", projectId: "project", connectionId: "account", binding,
agents: [{ id: lead, companyId: company, name: "Fixture CEO" }, { id: coder, companyId: company, name: "Fixture Coder" }],
first: { issueId: tasks[0]!.id }, second: { issueId: tasks[1]!.id }, runs, tasks,
readRuns: notifications.map(run => ({ runId: run.id, agentId: run.agentId, events: hiringNotificationActionEvents() })) };
const apiState = { issue: { id: chat, companyId: company, conversationUserId: user,
conversationAgentId: lead, conversationSessionGeneration: 0 }, comments, runs: structuredClone(runs) };
return { evidence, apiState };
}
tests/runner-e2e/hiring-template.test.ts
+374
View File
@@ -0,0 +1,374 @@
import { readFile } from "node:fs/promises";
import { describe, expect, it } from "vitest";
import { runnerMatrix, runnerSuites } from "./catalog.js";
import { buildRunnerE2EProcessEnvironment } from "./harness-env.js";
import { canonicalProviderEventsFromAcpxRuntimeEvent, canonicalProviderEventsFromCodex } from "../../packages/paperclip-runner/src/provider-events.js";
import { loadDefaultAgentInstructionsBundle } from "../../server/src/services/default-agent-instructions.js";
import { HIRING_TEMPLATE_READ_FILES, HIRING_TEMPLATE_SKILL_KEY, hiringTemplateInputs, hiringTemplateScenario } from "./hiring-template-cases.js";
import { readHiringInstructions, readHiringTemplateSources, renderHiringCoderExample, waitForSettledHiringObservation } from "./hiring-template-flow.js";
import { gradeHiringTemplate, hiringTemplateHash, hiringTemplateReadReceipts, type HiringTemplateEvidence } from "./hiring-template-scoring.js";
import type { RunnerApi } from "./api.js";
import { createHiringTemplateTurnFixture } from "./hiring-template-turn-fixture.js";
import { assertChatFlowRunCount } from "./chat-flow.js";
const beforeHire = "2026-10-01T12:00:00.000Z", hiredAt = "2026-10-01T12:01:00.000Z";
const binding = { provider: "anthropic", method: "api_key", mode: "responsible_user" };
const ceo = { "AGENTS.md": "You are the CEO. Lead the company." };
const coder = "You are Casey, a software engineer at Fixture Company. Own software implementation and maintenance.";
// Expected values are stated independently of the implementation under test.
const values = ["launch-queue", "api-key-rotation", "mixed-case-42", "already-ready"];
const reuseValues = ["launch_queue", "api_key_rotation", "mixed_case_42", "already_ready"];
function commandEvents(file: string) {
return canonicalProviderEventsFromCodex("item/completed", { item: {
type: "commandExecution", id: `read-${file}`, status: "completed", exitCode: 0,
command: `cat /workspace/.agents/skills/paperclip-create-agent/${file}`, aggregatedOutput: "source bytes",
} }).map(event => ({ eventType: event.eventType, createdAt: beforeHire, payload: { prpEvent: event } }));
}
function validEvidence(notificationCount = 0): HiringTemplateEvidence {
const turnFixture = createHiringTemplateTurnFixture(notificationCount);
const sourceFiles = [...HIRING_TEMPLATE_READ_FILES.map(file => `skills/paperclip-create-agent/${file}`),
"skills/paperclip-create-agent/references/baseline-role-guide.md", "server/src/onboarding-assets/ceo/AGENTS.md"];
const hashes = Object.fromEntries(sourceFiles.map(file => [file, hiringTemplateHash(file)]));
const tasks = turnFixture.evidence.tasks;
const runs = turnFixture.evidence.runs;
const document = (issueId: string, reference: string, outputs: string[]) => ({ issueId, key: "fixture", latestRevisionId: `${issueId}-revision`, createdByAgentId: "coder",
body: JSON.stringify({ reference, entries: hiringTemplateInputs.map((input, index) => ({ input, value: outputs[index] })) }) });
const first = document("first-task", "HIREfixture", values);
return { leadId: "lead", chatIssueId: "chat", hireName: "Casey", marker: "HIREfixture", projectId: "project", inputs: hiringTemplateInputs,
expectedCeoFiles: ceo, leadInstructions: { mode: "managed", entryFile: "AGENTS.md", files: ceo },
expectedSourceHashes: hashes, servedSourceHashes: { ...hashes }, assignedSkills: [HIRING_TEMPLATE_SKILL_KEY], coderExample: coder,
hiredInstructions: { mode: "managed", entryFile: "AGENTS.md", files: { "AGENTS.md": coder } },
hiredInstructionsAfterReuse: { mode: "managed", entryFile: "AGENTS.md", files: { "AGENTS.md": coder } },
hiredSkills: [], hiredSkillsAfterReuse: [],
agents: [{ ...turnFixture.evidence.agents[0], id: "lead", name: "CEO", adapterConfig: { model: "model" } },
{ ...turnFixture.evidence.agents[1], id: "coder", name: "Casey", role: "engineer", reportsTo: "lead", adapterType: "paperclip_runner", createdAt: hiredAt,
adapterConfig: { model: "model" }, runtimeConfig: { aiConnection: binding } }],
connectionId: "account", binding, tasks, runs, first, firstAfterReuse: { ...first }, second: document("second-task", "REUSEHIREfixture", reuseValues),
turnApiState: turnFixture.apiState,
readRuns: [...turnFixture.evidence.readRuns, { runId: "lead-1", agentId: "lead", events: HIRING_TEMPLATE_READ_FILES.flatMap(commandEvents) }] };
}
function fails(evidence: HiringTemplateEvidence, id: string) {
expect(gradeHiringTemplate(evidence).checks.find(check => check.id === id)?.passed, id).toBe(false);
}
describe("production hiring template oracle", () => {
it("independently grades the child fixtures and accepts JSON object key order", () => {
const e = validEvidence();
expect(gradeHiringTemplate(e)).toMatchObject({ outcomePassed: true, comparisonStatus: "comparable" });
e.first!.body = `\`\`\`json\n${JSON.stringify({ entries: hiringTemplateInputs.map((input, index) => ({ value: values[index], input })), reference: e.marker })}\n\`\`\``;
e.firstAfterReuse = { ...e.first! };
expect(gradeHiringTemplate(e).outcomePassed).toBe(true);
for (const wrongBody of ["{}", "not JSON", JSON.stringify({ reference: e.marker, entries: [{ input: hiringTemplateInputs[0], value: "launch-queue" }] }),
JSON.stringify({ reference: e.marker, entries: hiringTemplateInputs.map(input => ({ input, value: input.toLowerCase() })) })]) {
fails({ ...e, first: { ...e.first!, body: wrongBody } }, "initial-json-artifact");
}
fails({ ...e, second: { ...e.second!, body: e.first!.body } }, "reused-json-artifact");
fails({ ...e, first: { ...e.first!, createdByAgentId: "lead" } }, "initial-json-artifact");
});
it("requires the real hired identity, account, two distinct tasks and exactly five required successful work turns", () => {
const e = validEvidence();
fails({ ...e, agents: [...e.agents, { ...e.agents[1]!, id: "replacement" }] }, "one-coder-hire");
for (const wrong of [{ role: "qa" }, { reportsTo: "somebody" }, { adapterType: "codex_local" }, { name: "Another coder" }]) {
fails({ ...e, agents: [e.agents[0]!, { ...e.agents[1]!, ...wrong }] }, "one-coder-hire");
}
fails({ ...e, agents: [e.agents[0]!, { ...e.agents[1]!, adapterConfig: { model: "different" } }] }, "execution-account");
fails({ ...e, agents: [e.agents[0]!, { ...e.agents[1]!, runtimeConfig: { aiConnection: { ...binding, mode: "company" } } }] }, "execution-account");
for (const patch of [{ assigneeAgentId: "lead" }, { parentId: "chat" }, { projectId: "other" }, { status: "backlog" }]) {
fails({ ...e, tasks: [e.tasks[0]!, { ...e.tasks[1]!, ...patch }] }, "two-worker-tasks");
}
fails({ ...e, runs: e.runs.slice(1) }, "bounded-work-and-completion-turns");
fails({ ...e, runs: [...e.runs, { ...e.runs[0]!, id: "extra" }] }, "bounded-work-and-completion-turns");
fails({ ...e, runs: e.runs.map(r => r.id === "worker-reuse" ? { ...r, contextSnapshot: { issueId: "second-task", aiConnection: { connectionId: "other" } } } : r) }, "bounded-work-and-completion-turns");
fails({ ...e, runs: e.runs.map(r => r.id === "lead-status" ? { ...r, agentId: "coder" } : r) }, "bounded-work-and-completion-turns");
fails({ ...e, firstAfterReuse: { ...e.first!, latestRevisionId: "modified" } }, "original-preserved");
});
it("compares each revision's bundle and example without imposing candidate length on baseline", async () => {
const historical = validEvidence();
const oldFiles = { "AGENTS.md": "A long historical CEO role.\n".repeat(40), "HEARTBEAT.md": "Heartbeat", "SOUL.md": "Identity", "TOOLS.md": "Tools" };
historical.expectedCeoFiles = oldFiles;
historical.leadInstructions!.files = oldFiles;
for (const file of Object.keys(oldFiles)) historical.expectedSourceHashes[`server/src/onboarding-assets/ceo/${file}`] = historical.servedSourceHashes[`server/src/onboarding-assets/ceo/${file}`] = hiringTemplateHash(oldFiles[file as keyof typeof oldFiles]);
// Exact baseline source: skills/paperclip-create-agent/references/agents/coder.md
// at d7bdfc422cadcc407f2945e211833a8382b80117. Its SHA-256 below
// preserves the actual historical text and all four placeholder kinds.
const historicalReference = await readFile(new URL("./fixtures/hiring-templates/coder.d7bdfc4.md", import.meta.url), "utf8");
expect(hiringTemplateHash(historicalReference)).toBe("766c6f5db907f3dc2e317d540feb80e77358202c59d759449228a32450fa6202");
historical.coderExample = renderHiringCoderExample(historicalReference, "Casey", "Fixture Company", "CEO", "FIX");
expect(historical.coderExample).not.toMatch(/\{\{[^}]+\}\}/);
expect(historical.coderExample).toContain("You report to CEO.");
expect(historical.coderExample.match(/\/FIX\/agents\//g)).toHaveLength(3);
historical.hiredInstructions!.files["AGENTS.md"] = historical.hiredInstructionsAfterReuse!.files["AGENTS.md"] = historical.coderExample;
const result = gradeHiringTemplate(historical);
expect(result).toMatchObject({ outcomePassed: true, comparisonStatus: "comparable" });
expect(result.instructionSizes.coder.words).toBeGreaterThan(200);
const e = validEvidence();
fails({ ...e, leadInstructions: { ...e.leadInstructions!, files: { "AGENTS.md": "Custom fixture lead" } } }, "production-ceo-bundle");
fails({ ...e, leadInstructions: { ...e.leadInstructions!, files: { ...ceo, "SOUL.md": "unexpected legacy file" } } }, "production-ceo-bundle");
fails({ ...e, hiredInstructions: { ...e.hiredInstructions!, files: { "AGENTS.md": "Generic default worker instructions" } } }, "supplied-coder-instructions");
fails({ ...e, hiredInstructionsAfterReuse: undefined }, "hired-instructions-durable");
fails({ ...e, hiredSkillsAfterReuse: ["changed"] }, "hired-skills-durable");
});
it("separates successful workflow outcomes from missing or mismatched source coverage", () => {
const e = validEvidence();
for (const patch of [{ readRuns: e.readRuns.filter(run => run.runId !== "lead-1") }, { assignedSkills: [] }, { expectedSourceHashes: {} }, { servedSourceHashes: {} },
{ servedSourceHashes: { ...e.servedSourceHashes, "skills/paperclip-create-agent/SKILL.md": hiringTemplateHash("other checkout") } }]) {
expect(gradeHiringTemplate({ ...e, ...patch })).toMatchObject({ outcomePassed: true, comparisonStatus: "uncomparable" });
}
const incomplete = { ...e.expectedSourceHashes };
delete incomplete["skills/paperclip-create-agent/references/agents/coder.md"];
fails({ ...e, expectedSourceHashes: incomplete }, "source-fingerprints");
});
it("requires a completed pre-hire lead read rather than an echoed path, failed read or listing", () => {
const e = validEvidence(), good = e.readRuns.find(run => run.runId === "lead-1")!;
for (const command of ["echo /workspace/.agents/skills/paperclip-create-agent/SKILL.md", "ls /workspace/.agents/skills/paperclip-create-agent/SKILL.md",
"cat /workspace/.agents/skills/wrong-skill/SKILL.md", "cat $SKILL/SKILL.md", "cat /workspace/.agents/skills/paperclip-create-agent/SKILL.md > /dev/null",
...["cat --help", "cat --version", "head --help", "tail --version", "head -n 0", "tail -c 0", "sed -n ''", "sed -n '1q'", "sed -n 'q'", "sed --help", "sed -n '1,200w /tmp/other'", "cat -n"].map(prefix => `${prefix} /workspace/.agents/skills/paperclip-create-agent/SKILL.md`),
'cat "/workspace/.agents/skills/paperclip-create-agent/SKILL.md""suffix"' ]) {
const events = commandEvents("SKILL.md");
const payload = events[0]!.payload.prpEvent.payload as Record<string, unknown>;
payload.name = command;
expect(hiringTemplateReadReceipts([{ ...good, events }], "lead"), command).toHaveLength(0);
}
fails({ ...e, readRuns: [{ ...good, agentId: "coder" }] }, "production-source-reads");
fails({ ...e, readRuns: [{ ...good, events: good.events.slice(1) }] }, "production-source-reads");
fails({ ...e, readRuns: [{ ...good, events: good.events.map(event => ({ ...event, createdAt: "2026-10-01T12:02:00Z" })) }] }, "production-source-reads");
fails({ ...e, readRuns: [{ ...good, events: good.events.map(event => ({ ...event, createdAt: undefined })) }] }, "production-source-reads");
const failed = commandEvents("SKILL.md");
(failed[0]!.payload.prpEvent.payload as Record<string, unknown>).exitCode = 1;
expect(hiringTemplateReadReceipts([{ ...good, events: failed }], "lead")).toHaveLength(0);
const noOutput = commandEvents("SKILL.md");
(noOutput[0]!.payload.prpEvent.payload as Record<string, unknown>).outputBytes = 0;
expect(hiringTemplateReadReceipts([{ ...good, events: noOutput }], "lead")).toHaveLength(0);
});
it("accepts only supported direct read arguments and does not trust process read hints", () => {
const path = "/workspace/.agents/skills/paperclip-create-agent/SKILL.md";
for (const command of [`cat ${path}`, `cat -- '${path}'`, `head -n 200 ${path}`, `head -n200 ${path}`, `tail -c 200 ${path}`,
`sed -n '1,200p' ${path}`, `sed -n -e 'p' ${path}`]) {
const events = commandEvents("SKILL.md");
(events[0]!.payload.prpEvent.payload as Record<string, unknown>).name = command;
expect(hiringTemplateReadReceipts([{ runId: "lead-1", agentId: "lead", events }], "lead"), command).toHaveLength(1);
}
const events = commandEvents("SKILL.md");
Object.assign(events[0]!.payload.prpEvent.payload, { name: `cat --help ${path}`, operation: "read", readOnly: true, target: ".agents/skills/paperclip-create-agent/SKILL.md" });
expect(hiringTemplateReadReceipts([{ runId: "lead-1", agentId: "lead", events }], "lead")).toHaveLength(0);
});
it("uses the production ACPX event mapper and leaves redacted absolute read paths uncomparable", () => {
const events = HIRING_TEMPLATE_READ_FILES.flatMap(file => canonicalProviderEventsFromAcpxRuntimeEvent({
type: "tool_call", tag: "tool_call_update", toolCallId: `read-${file}`, title: "Read", kind: "read", status: "completed",
locations: [{ path: `.agents/skills/paperclip-create-agent/${file}` }], rawOutput: "Source bytes",
} as never, `read-${file}`).map(event => ({ eventType: event.eventType, createdAt: beforeHire, payload: { prpEvent: event } })));
expect(gradeHiringTemplate({ ...validEvidence(), readRuns: [{ runId: "lead-1", agentId: "lead", events }] }).comparisonStatus).toBe("comparable");
const absolute = canonicalProviderEventsFromAcpxRuntimeEvent({ type: "tool_call", tag: "tool_call_update", toolCallId: "read", title: "Read", kind: "read", status: "completed",
locations: [{ path: "/workspace/.agents/skills/paperclip-create-agent/SKILL.md" }], rawOutput: "Source bytes" } as never, "read");
expect(hiringTemplateReadReceipts([{ runId: "lead-1", agentId: "lead", events: absolute.map(event => ({ eventType: event.eventType, payload: { prpEvent: event } })) }], "lead")).toHaveLength(0);
});
});
describe("executable hiring lifecycle count guards", () => {
const task = { expectedRunCount: 7, minimumExpectedRunCount: 5 };
it("grades and admits five required work turns with zero, batched or distinct completion turns", () => {
for (const count of [0, 1, 2]) {
const evidence = validEvidence(count);
const result = gradeHiringTemplate(evidence);
expect(result).toMatchObject({ outcomePassed: true, comparisonStatus: "comparable", turnAccounting: {
passed: true, counts: { requestedLeadTurns: 3, coderTurns: 2, completionTurns: count,
actualRunCount: 5 + count, costAccountingRunCount: 5 + count, unclassifiedTurns: 0 },
} });
expect(assertChatFlowRunCount({ suiteId: "hiring-templates", task, runs: evidence.runs,
hiringEvidence: evidence, hiringApiState: evidence.turnApiState })?.passed).toBe(true);
}
});
it("rejects an arbitrary same-count wake in both executable paths", () => {
const evidence = validEvidence(2);
const notification = evidence.runs.find(run => run.contextSnapshot?.wakeReason === "chat_task_completed")!;
notification.contextSnapshot!.wakeReason = "heartbeat_timer";
(evidence.turnApiState as { runs: unknown[] }).runs = structuredClone(evidence.runs);
fails(evidence, "bounded-work-and-completion-turns");
expect(() => assertChatFlowRunCount({ suiteId: "hiring-templates", task, runs: evidence.runs,
hiringEvidence: evidence, hiringApiState: evidence.turnApiState })).toThrow();
});
it("rejects unrelated notification document writes in both executable paths", () => {
const evidence = validEvidence(2);
const ledger = evidence.readRuns.find(run => run.runId === "notify-1")!;
for (const event of ledger.events) {
const payload = (event.payload as Record<string, any>).prpEvent.payload;
if (payload.name) payload.name = "write_document";
if (payload.item?.name) payload.item.name = "write_document";
}
fails(evidence, "bounded-work-and-completion-turns");
expect(() => assertChatFlowRunCount({ suiteId: "hiring-templates", task, runs: evidence.runs,
hiringEvidence: evidence, hiringApiState: evidence.turnApiState })).toThrow();
expect(gradeHiringTemplate(evidence).turnAccounting.actionEvidence.status).toBe("violated");
});
it("requires public lifecycle observations and preserves source/template coverage failures", () => {
const evidence = validEvidence(2);
fails({ ...evidence, turnApiState: undefined }, "bounded-work-and-completion-turns");
expect(() => assertChatFlowRunCount({ suiteId: "hiring-templates", task, runs: evidence.runs,
hiringEvidence: evidence })).toThrow();
const changedInstructions = { ...evidence.hiredInstructions!, files: { "AGENTS.md": `${coder} Changed punctuation.` } };
const result = gradeHiringTemplate({ ...evidence, readRuns: evidence.readRuns.filter(run => run.runId !== "lead-1"), hiredInstructions: changedInstructions,
hiredInstructionsAfterReuse: structuredClone(changedInstructions) });
expect(result).toMatchObject({ outcomePassed: true, comparisonStatus: "uncomparable", turnAccounting: { passed: true } });
expect(result.checks.filter(check => check.dimension === "coverage" && !check.passed).map(check => check.id))
.toEqual(["production-source-reads", "supplied-coder-instructions"]);
});
it("keeps every non-hiring chat count guard unchanged", () => {
const runs = validEvidence(2).runs;
for (const suiteId of ["agent-chat", "agent-chat-hardening", "agent-chat-qualification", "context-integrity"]) {
expect(() => assertChatFlowRunCount({ suiteId, task: { expectedRunCount: 2 }, runs: runs.slice(0, 2) })).not.toThrow();
expect(() => assertChatFlowRunCount({ suiteId, task: { expectedRunCount: 2 }, runs: runs.slice(0, 3) })).toThrow();
}
});
});
describe("production hiring fixture wiring and source observations", () => {
it("keeps two explicit local native cells, production permissions and bounded five-work-turn scope", () => {
const cells = runnerMatrix.filter(cell => cell.suite.id === "hiring-templates");
expect(cells.map(cell => cell.id)).toEqual(["hiring-templates.runner-codex.local.hire-coder-template-reuse", "hiring-templates.runner-acpx-claude.local.hire-coder-template-reuse"]);
expect(runnerSuites.find(suite => suite.id === "hiring-templates")?.manualOnly).toBe(true);
for (const cell of cells) {
expect(cell.task.expectedRunCount).toBe(7);
expect(cell.task.minimumExpectedRunCount).toBe(5);
expect(cell.task.attemptTimeoutMs?.local).toBe(15 * 60_000);
expect(buildRunnerE2EProcessEnvironment({}, [cell]).PAPERCLIP_RUNNER_API_TOOLS_ENABLED).toBe("true");
const payload = cell.profile.buildAgent({ executionId: "fixture", workspacePath: "/workspace", environmentId: "env", environmentFixtureId: "local",
secretRefs: { [cell.profile.credential]: { type: "secret_ref", secretId: "secret", version: "latest" } } });
expect(payload).toMatchObject({ role: "ceo", adapterType: "paperclip_runner" });
expect(payload).not.toHaveProperty("instructionsBundle");
expect(payload.adapterConfig).not.toHaveProperty("instructionsBundleMode");
expect(payload.adapterConfig).not.toHaveProperty("codexPermissionMode");
expect(payload.adapterConfig).not.toHaveProperty("acpxPermissionMode");
}
const scenario = hiringTemplateScenario("matched-fixture", "Project");
expect(scenario.initialPrompt).toBe(hiringTemplateScenario("matched-fixture", "Project").initialPrompt);
expect(scenario.reusePrompt("TASK-1")).toBe(hiringTemplateScenario("matched-fixture", "Project").reusePrompt("TASK-1"));
});
it("reads the actual default CEO selection and served hiring files through public APIs", async () => {
const files = await loadDefaultAgentInstructionsBundle("ceo");
const get = async (path: string) => {
if (path.endsWith("/instructions-bundle")) return { mode: "managed", entryFile: "AGENTS.md", files: Object.keys(files).map(path => ({ path })) };
if (path.includes("instructions-bundle/file?")) return { content: files[new URL(path, "http://fixture").searchParams.get("path")!] };
if (path === "/api/companies/company/skills") return [{ id: "creator", key: HIRING_TEMPLATE_SKILL_KEY }];
if (path === "/api/agents/lead/skills?companyId=company") return { desiredSkills: [HIRING_TEMPLATE_SKILL_KEY] };
if (path.startsWith("/api/companies/company/skills/creator/files?")) return { content: await readFile(new URL(`../../skills/paperclip-create-agent/${new URL(path, "http://fixture").searchParams.get("path")}`, import.meta.url), "utf8") };
throw new Error(`Unexpected public read ${path}`);
};
const api = { get } as Pick<RunnerApi, "get">;
const source = await readHiringTemplateSources(api, "company", "lead");
expect(source.leadInstructions.files).toEqual(files);
expect(source.expectedCeoFiles).toEqual(files);
expect(source.servedSourceHashes).toEqual(source.expectedSourceHashes);
expect(source.assignedSkills).toContain(HIRING_TEMPLATE_SKILL_KEY);
expect(source.loaderHash).toMatch(/^[a-f0-9]{64}$/);
expect(renderHiringCoderExample(source.coderReference, "Casey", "Company", "CEO", "FIX")).not.toContain("{{");
const wrongApi = { get: async (path: string) => path.includes("/files?") ? { content: "different revision" } : get(path) } as Pick<RunnerApi, "get">;
const wrongSource = await readHiringTemplateSources(wrongApi, "company", "lead");
expect(wrongSource.servedSourceHashes).not.toEqual(wrongSource.expectedSourceHashes);
});
it("retains baseline policy files while excluding binary files and generated personal notes", async () => {
const api = { get: async (path: string) => path.endsWith("/instructions-bundle")
? { mode: "managed", entryFile: "AGENTS.md", files: ["AGENTS.md", "HEARTBEAT.md", "SOUL.md", "TOOLS.md", "notes/today.md"].map(path => ({ path, binary: false })).concat([{ path: "image.png", binary: true }]) }
: { content: new URL(path, "http://fixture").searchParams.get("path") } } as Pick<RunnerApi, "get">;
expect(Object.keys((await readHiringInstructions(api, "lead")).files)).toEqual(["AGENTS.md", "HEARTBEAT.md", "SOUL.md", "TOOLS.md"]);
expect(renderHiringCoderExample("Historical example\n```md\nYou are {{agentName}} at {{companyName}}. Report to {{managerTitle}}.\nLarge manual content.\n```", "Casey", "Company", "CEO", "FIX"))
.toBe("You are Casey at Company. Report to CEO.\nLarge manual content.");
expect(() => renderHiringCoderExample("No example", "Casey", "Company", "CEO", "FIX")).toThrow();
});
});
describe("settled hiring observation", () => {
function observation(count = 2) {
const f = createHiringTemplateTurnFixture(count);
const apiState = { ...f.apiState, issue: { ...f.apiState.issue, title: "Hiring fixture", status: "in_review", conversationState: "waiting" },
wakes: { events: [], truncated: false } };
return { agents: f.evidence.agents, tasks: f.evidence.tasks, runs: f.evidence.runs,
readRuns: [], apiState, finalApiState: structuredClone(apiState) };
}
it("retries the whole snapshot when a completion appears between ledger reads", async () => {
const racing = observation(), settled = observation();
racing.runs = racing.runs.filter(run => run.id !== "notify-2");
racing.apiState.runs = structuredClone(racing.runs);
let reads = 0;
const value = await waitForSettledHiringObservation(async () => ++reads === 1 ? racing : settled,
{ deadlineAt: Date.now() + 1000, intervalMs: 0 });
expect(reads).toBe(3);
expect(value.runs).toHaveLength(7);
});
it("awaits pending completion wakes and then two stable observations", async () => {
const pending = observation(), settled = observation();
const wakes = { events: [{ kind: "wake_request", status: "queued", finishedAt: null }], truncated: false };
pending.apiState.wakes = pending.finalApiState.wakes = wakes as typeof pending.apiState.wakes;
let reads = 0;
await waitForSettledHiringObservation(async () => ++reads === 1 ? pending : settled,
{ deadlineAt: Date.now() + 1000, intervalMs: 0 });
expect(reads).toBe(3);
});
it("awaits an outbox completion not yet represented by a wake or run", async () => {
const awaitingCallback = observation(0), settled = observation(1);
let reads = 0;
const value = await waitForSettledHiringObservation(async () => ++reads === 1 ? awaitingCallback : settled,
{ deadlineAt: Date.now() + 1000, intervalMs: 0 });
expect(reads).toBe(3);
expect(value.runs).toHaveLength(6);
});
it("never admits truncated diagnostics, active runs or inconsistent observations", async () => {
for (const defect of ["truncated", "active", "inconsistent"] as const) {
const bad = observation();
if (defect === "truncated") bad.apiState.wakes.truncated = bad.finalApiState.wakes.truncated = true;
if (defect === "active") bad.runs[0]!.status = "running";
if (defect === "inconsistent") bad.finalApiState.comments.pop();
await expect(waitForSettledHiringObservation(async () => bad,
{ deadlineAt: Date.now() + 10, intervalMs: 0 })).rejects.toThrow(/Timed out/);
}
});
});
it("requires attributable callback replies and does not ignore unresolved coalesced or unknown wakes", async () => {
const f = createHiringTemplateTurnFixture(2);
const state = { ...f.apiState, issue: { ...f.apiState.issue, title: "Fixture", status: "in_review", conversationState: "waiting" },
wakes: { events: [] as Array<{ kind: string; status: string; runId?: string }>, truncated: false } };
const valid = { agents: f.evidence.agents, tasks: f.evidence.tasks, runs: f.evidence.runs,
readRuns: f.evidence.readRuns, apiState: state, finalApiState: structuredClone(state) };
for (const defect of ["reply", "coalesced", "unknown", "scheduled-retry", "duplicate-callback"] as const) {
const bad = structuredClone(valid);
if (defect === "reply") bad.apiState.comments = bad.finalApiState.comments = [];
if (defect === "coalesced") bad.apiState.wakes.events = bad.finalApiState.wakes.events = [{ kind: "wake_request", status: "coalesced", runId: "missing" }];
if (defect === "unknown") bad.apiState.wakes.events = bad.finalApiState.wakes.events = [{ kind: "wake_request", status: "other" }];
if (defect === "scheduled-retry") bad.runs[0]!.status = "scheduled_retry";
if (defect === "duplicate-callback") bad.runs.push(structuredClone(bad.runs.find(run => run.id === "notify-1")!));
await expect(waitForSettledHiringObservation(async () => bad,
{ deadlineAt: Date.now() + 10, intervalMs: 0 })).rejects.toThrow(/Timed out/);
}
});
it("admits completed and accounted coalesced wake bookkeeping after callbacks", async () => {
const f = createHiringTemplateTurnFixture(2);
const state = { ...f.apiState, issue: { ...f.apiState.issue, title: "Fixture", status: "in_review", conversationState: "waiting" },
wakes: { events: [{ kind: "wake_request", status: "completed", runId: "notify-1" },
{ kind: "wake_request", status: "coalesced", runId: "notify-2" }], truncated: false } };
const observation = { agents: f.evidence.agents, tasks: f.evidence.tasks, runs: f.evidence.runs,
readRuns: f.evidence.readRuns, apiState: state, finalApiState: structuredClone(state) };
let reads = 0;
await waitForSettledHiringObservation(async () => { reads++; return observation; },
{ deadlineAt: Date.now() + 1000, intervalMs: 0 });
expect(reads).toBe(2);
});
it("classifies absent notification action identity as uncomparable separately from unchanged source coverage", () => {
const evidence = validEvidence(2);
evidence.readRuns = evidence.readRuns.filter(run => run.runId === "lead-1");
const result = gradeHiringTemplate(evidence);
expect(result.comparisonStatus).toBe("uncomparable");
expect(result.turnAccounting.actionEvidence.status).toBe("uncomparable");
expect(result.checks.filter(check => check.dimension === "coverage" && !check.passed).map(check => check.id))
.toEqual(["completion-action-attribution"]);
expect(result.checks.find(check => check.id === "production-source-reads")?.passed).toBe(true);
});
tests/runner-e2e/history.test.ts
+2
View File
@@ -31,6 +31,7 @@ import {
import { renderRunnerHistoryIndex } from "./history-index.js";
import { renderPublicCampaignSummary } from "./public-summary-image.js";
import { isPublicRunnerScreenshotRoute } from "./screenshot-policy.js";
import { connectionCheckpoints, type ConnectionEvidence } from "./connection-evidence.js";
import type { MatrixExecution, RunnerE2EResult } from "./types.js";
const temporaryDirectories: string[] = [];
@@ -63,6 +64,7 @@ function result(execution: MatrixExecution, status: "passed" | "failed") {
finishedAt: "2026-08-28T00:00:01.000Z",
durationMs: 1_000,
cleanup: "passed",
...(execution.task.flow === "provider_connection" ? { providerConnection: { version: 1, outcome: status, phase: "finished", assisted: false, authFreshness: "signed-in", target: { mode: "attach", origin: "http://127.0.0.1:3100", commit: "a".repeat(40), deploymentMode: "local_trusted" }, entry: "agent", method: "api-key", checkpoints: Object.fromEntries(connectionCheckpoints.map(key => [key, true])), waits: [] } satisfies ConnectionEvidence } : {}),
} satisfies RunnerE2EResult;
}
tests/runner-e2e/installed-release.test.ts
+55
View File
@@ -0,0 +1,55 @@
import { mkdtempSync, mkdirSync, realpathSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import { installedReleaseDaytonaPlugin, installedReleaseEnvironment, installedReleaseLaunch } from "./installed-release.js";
const roots: string[] = [];
afterEach(() => roots.splice(0).forEach(root => rmSync(root, { recursive: true, force: true })));
describe("ordinary installed release smoke", () => {
it("launches the public compiled CLI from its own consumer root", () => {
const root = realpathSync(mkdtempSync(path.join(tmpdir(), "installed-paperclip-"))); roots.push(root);
mkdirSync(path.join(root, "dist"));
const cli = path.join(root, "dist", "index.js"); writeFileSync(cli, "");
writeFileSync(path.join(root, "package.json"), JSON.stringify({ name: "paperclipai", version: "0.0.0-verify" }));
expect(installedReleaseLaunch(cli)).toMatchObject({ cwd: root, args: [cli, "onboard", "--yes", "--run"], version: "0.0.0-verify" });
writeFileSync(path.join(root, "package.json"), JSON.stringify({ name: "other-cli", version: "1.0.0" }));
expect(() => installedReleaseLaunch(cli)).toThrow("public package");
expect(() => installedReleaseLaunch("relative/index.js")).toThrow("absolute path");
});
it("uses the public compiled Daytona plugin beside the installed CLI", () => {
const root = realpathSync(mkdtempSync(path.join(tmpdir(), "installed-daytona-"))); roots.push(root);
const cliRoot = path.join(root, "node_modules/paperclipai"), plugin = path.join(root, "node_modules/@paperclipai/plugin-daytona");
mkdirSync(path.join(cliRoot, "dist"), { recursive: true }); mkdirSync(path.join(plugin, "dist"), { recursive: true });
const cli = path.join(cliRoot, "dist/index.js"); writeFileSync(cli, "");
writeFileSync(path.join(cliRoot, "package.json"), JSON.stringify({ name: "paperclipai", version: "0.0.0-verify" }));
const manifest = { name: "@paperclipai/plugin-daytona", version: "0.0.0-verify", paperclipPlugin: { manifest: "./dist/manifest.js", worker: "./dist/worker.js" }, exports: { ".": { import: "./dist/index.js" } } };
for (const file of ["manifest.js", "worker.js", "index.js"]) writeFileSync(path.join(plugin, "dist", file), "");
writeFileSync(path.join(plugin, "package.json"), JSON.stringify(manifest));
expect(installedReleaseDaytonaPlugin(cli)).toBe(plugin);
expect(installedReleaseDaytonaPlugin(cli, plugin)).toBe(plugin);
expect(() => installedReleaseDaytonaPlugin(cli, "relative/plugin")).toThrow("absolute");
writeFileSync(path.join(plugin, "package.json"), JSON.stringify({ ...manifest, version: "0.0.0-stale" }));
expect(() => installedReleaseDaytonaPlugin(cli, plugin)).toThrow("qualified release version 0.0.0-verify");
expect(installedReleaseDaytonaPlugin(cli, plugin, "0.0.0-stale")).toBe(plugin);
writeFileSync(path.join(plugin, "package.json"), JSON.stringify({ ...manifest, version: undefined }));
expect(() => installedReleaseDaytonaPlugin(cli, plugin)).toThrow("qualified release version");
writeFileSync(path.join(plugin, "package.json"), JSON.stringify({ ...manifest, exports: { ".": "./src/index.ts" } }));
expect(() => installedReleaseDaytonaPlugin(cli)).toThrow("public compiled Daytona");
writeFileSync(path.join(plugin, "package.json"), JSON.stringify(manifest));
rmSync(path.join(plugin, "dist/worker.js"));
expect(() => installedReleaseDaytonaPlugin(cli)).toThrow();
});
it("removes repository executables, provider packs, native paths and loader injection", () => {
const source = { PATH: "/repo/packages/node_modules/.bin:/tmp/provider-bin:/usr/bin:/tmp/node/bin", NODE_OPTIONS: "--import /repo/loader", NODE_PATH: "/repo/node_modules",
PAPERCLIP_RUNNER_BINARY: "/repo/runner", PAPERCLIP_RUNNER_PROVIDER_PACK_ROOT: "/private/pack",
PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH: "/private/pack", PAPERCLIP_ACPX_BUILTIN_ROOT: "/repo/providers",
PAPERCLIP_NATIVE_RUNTIME_CONTEXT_PATH: "/repo/context", PAPERCLIP_RUNNER_E2E_PORT: "3100", PAPERCLIP_HOME: "/tmp/home" };
expect(installedReleaseEnvironment(source, "/repo", "/tmp/provider-bin")).toEqual({ PATH: "/usr/bin:/tmp/node/bin", PAPERCLIP_RUNNER_E2E_PORT: "3100", PAPERCLIP_HOME: "/tmp/home" });
expect(source.PAPERCLIP_RUNNER_BINARY).toBe("/repo/runner");
});
it("rejects qualification admission instead of silently relabeling it as production", () => {
expect(() => installedReleaseEnvironment({ PAPERCLIP_RUNNER_ACPX_QUALIFICATION: "cursor" }, "/repo", "/tmp/provider-bin")).toThrow("production admission");
});
});
tests/runner-e2e/installed-release.ts
+53
View File
@@ -0,0 +1,53 @@
import { readFileSync, realpathSync } from "node:fs";
import path from "node:path";
/** Test supervisor only: launch the actual public CLI, with its own module root. */
export function installedReleaseLaunch(cliPath: string) {
if (!path.isAbsolute(cliPath)) throw new Error("Installed release CLI must be an absolute path");
const cli = realpathSync(cliPath);
const root = path.resolve(path.dirname(cli), "..");
const manifest = JSON.parse(readFileSync(path.join(root, "package.json"), "utf8"));
if (path.basename(cli) !== "index.js" || path.basename(path.dirname(cli)) !== "dist" ||
manifest.name !== "paperclipai" || typeof manifest.version !== "string") {
throw new Error("Installed release must use paperclipai/dist/index.js from its public package");
}
return { cli, cwd: root, args: [cli, "onboard", "--yes", "--run"], version: manifest.version };
}
/** Verify the selected public plugin belongs to the release being qualified. */
export function installedReleaseDaytonaPlugin(cliPath: string, installedPluginPath?: string, expectedVersion?: string): string {
const cli = installedReleaseLaunch(cliPath);
if (installedPluginPath && !path.isAbsolute(installedPluginPath)) throw new Error("Installed Daytona plugin path must be absolute");
const plugin = realpathSync(installedPluginPath ?? path.join(path.dirname(cli.cwd), "@paperclipai/plugin-daytona"));
const manifest = JSON.parse(readFileSync(path.join(plugin, "package.json"), "utf8"));
if (manifest.name !== "@paperclipai/plugin-daytona" ||
manifest.paperclipPlugin?.manifest !== "./dist/manifest.js" ||
manifest.paperclipPlugin?.worker !== "./dist/worker.js" ||
manifest.exports?.["."]?.import !== "./dist/index.js") {
throw new Error("Installed release requires the public compiled Daytona plugin package");
}
const releaseVersion = expectedVersion ?? cli.version;
if (!releaseVersion || manifest.version !== releaseVersion) {
throw new Error(`Installed Daytona plugin must match qualified release version ${releaseVersion}`);
}
realpathSync(path.join(plugin, "dist/manifest.js"));
realpathSync(path.join(plugin, "dist/worker.js"));
realpathSync(path.join(plugin, "dist/index.js"));
return plugin;
}
/** Qualification assets must never make a public installed product smoke pass. */
export function installedReleaseEnvironment(source: NodeJS.ProcessEnv, repositoryRoot: string, providerBin: string): NodeJS.ProcessEnv {
if (source.PAPERCLIP_RUNNER_ACPX_QUALIFICATION) throw new Error("Installed release smoke requires production admission without qualification overrides");
const result = { ...source };
for (const key of Object.keys(result)) {
if ((key.startsWith("PAPERCLIP_RUNNER_") && !key.startsWith("PAPERCLIP_RUNNER_E2E_")) ||
key.startsWith("PAPERCLIP_ACPX_") || key.startsWith("PAPERCLIP_NATIVE_") ||
key === "NODE_PATH" || key === "NODE_OPTIONS") delete result[key];
}
result.PATH = source.PATH?.split(path.delimiter).filter(entry =>
entry && entry !== providerBin && entry !== repositoryRoot &&
!entry.startsWith(`${repositoryRoot}${path.sep}`) && !entry.includes(`${path.sep}node_modules${path.sep}`),
).join(path.delimiter);
return result;
}
tests/runner-e2e/instruction-persistence.ts
+2 -2
View File
@@ -68,9 +68,9 @@ export async function runInstructionPersistenceFlow(input: {
let issue: Row = {};
let runs: Row[] = [];
async function create(title: string, prompt: string) {
await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title, prompt, workMode: "standard", projectName: fixtures.project?.name });
const createdTask = await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title, prompt, workMode: "standard", projectName: fixtures.project?.name });
const found = await pollUntil({ label: `instruction task ${title}`, deadlineAt: input.deadlineAt,
load: async () => (await api.get<Row[]>(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(row => row.title === title), accept: row => Boolean(row) });
load: async () => (await api.get<Row[]>(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(row => row.id === createdTask.issueId), accept: row => Boolean(row) });
if (!found) throw new Error("Browser-created instruction task missing");
issue = found;
input.observe(issue, runs);
tests/runner-e2e/launch.ts
+70 -41
View File
@@ -25,7 +25,9 @@ import {
import { isImmutableDaytonaImage, runnerMatrix } from "./catalog.js";
import { renderRunnerE2EDashboard } from "./dashboard.js";
import { packageEvidence } from "./evidence.js";
import { classifyFailure, shouldRetryFailure } from "./failure-classifier.js";
import { classifyFailure } from "./failure-classifier.js";
import { mustPreserveRecoveryState, shouldKeepFailedDiagnostics } from "./cleanup-verification.js";
import { effectiveAutomaticRetryLimit, executeWithAutomaticRetry } from "./automatic-retry.js";
import { buildRunnerCampaign } from "./history.js";
import {
buildRunnerE2EProcessEnvironment,
@@ -54,6 +56,10 @@ import {
type RunnerE2EResult,
} from "./types.js";
import { assertRemoteNativeEvidencePrerequisites, assertRunnerE2EPrerequisites } from "./prerequisites.js";
import { assertNativeCompletionSelection, prepareNativeCompletionPreflight, NATIVE_COMPLETION_PREFLIGHT_ENV } from "./native-completion-admission.js";
import { assertNativeInstructionSelection, prepareNativeInstructionPreflight, NATIVE_INSTRUCTION_PREFLIGHT_ENV, NATIVE_INSTRUCTION_SUITE } from "./native-instruction-consolidation.js";
import { prepareStockHarnessPreflight, STOCK_PREFLIGHT_ENV } from "./stock-harness-admission.js";
import {
reapNewDetachedDarwinSharedMemory,
snapshotDarwinSharedMemory,
@@ -486,7 +492,7 @@ async function runAttempt(input: {
const publishedResultPaths = new Map<string, string>();
let attemptSecrets: string[] = [];
let processCleanupFailed = false;
let startupReceiptPath: string | undefined;
const rawCleanupResults: Array<{ cleanup: string; synthetic: boolean; status: string }> = [];
try {
const paperclipHome = path.join(temporaryRoot, "paperclip-home");
const workspace = path.join(temporaryRoot, "workspace");
@@ -536,7 +542,7 @@ async function runAttempt(input: {
executions.map((candidate) => candidate.id),
),
PAPERCLIP_RUNNER_E2E_ATTEMPT: String(attempt),
PAPERCLIP_RUNNER_E2E_INSTALLED_STARTUP_ONLY: options.installedStartupOnly ? "1" : undefined,
PAPERCLIP_RUNNER_E2E_PUBLIC_MCP: executions.some(candidate => candidate.task.flow === "public_mcp") ? "1" : "0",
PAPERCLIP_RUNNER_E2E_PORT: String(port),
PAPERCLIP_RUNNER_E2E_TEMP_ROOT: temporaryRoot,
PAPERCLIP_RUNNER_E2E_PRIVATE_DIR: privateDir,
@@ -643,6 +649,8 @@ async function runAttempt(input: {
processFailureClass,
);
const result = await readResult(resultPath, fallback);
// Keep cleanup authority before evidence copying/publication can fail.
rawCleanupResults.push({ cleanup: result.cleanup, synthetic: result === fallback, status: result.status });
return enforceResultProcessIntegrity(result, processResult);
}),
);
@@ -815,8 +823,22 @@ async function runAttempt(input: {
} finally {
if (!processCleanupFailed) reapNewDetachedDarwinSharedMemory(sharedMemoryBaseline);
let cleanupError: unknown;
if (processCleanupFailed) {
cleanupError = new Error(`Preserving temporary state after incomplete process cleanup: ${temporaryRoot}`);
const resourceAdmissionStarted = await access(path.join(temporaryRoot, "artifacts-private", "resource-admission-started"))
.then(() => true).catch((error: NodeJS.ErrnoException) => error.code !== "ENOENT");
if (mustPreserveRecoveryState({ processCleanupFailed, resourceAdmissionStarted, results: rawCleanupResults })) {
// Remote allocation cleanup is journaled in this instance's database.
// Deleting it after the controller exits would strand uncertain creates.
await chmod(temporaryRoot, 0o700);
cleanupError = new Error(`Preserving private recovery state after unconfirmed cleanup: ${temporaryRoot}`);
} else if (shouldKeepFailedDiagnostics({
enabled: process.env.PAPERCLIP_RUNNER_E2E_KEEP_FAILED_PRIVATE === "1",
expectedResults: executions.length,
results: publishedResults,
})) {
// Explicit diagnosis only. Confirmed resource cleanup stays confirmed;
// keep private traces for investigation without publishing provider data.
await chmod(temporaryRoot, 0o700);
console.warn(`Retained private failed-case diagnostics: ${temporaryRoot}`);
} else if (
temporaryRoot.startsWith(`${os.tmpdir()}${path.sep}paperclip-runner-e2e-`)
) {
@@ -856,7 +878,7 @@ async function runAttempt(input: {
Object.assign(publishedResult, {
status: "failed",
failureClass: "cleanup_failure",
error: message,
error: publishedResult.error ? `${publishedResult.error}; ${message}` : message,
cleanup: "failed",
} satisfies Partial<RunnerE2EResult>);
const safeResult = `${JSON.stringify(
@@ -897,36 +919,19 @@ async function runExecutionWithRetry(input: {
options: ReturnType<typeof parseRunnerSelectors>;
}): Promise<RunnerE2EResult> {
const { execution, campaignId, options } = input;
const [firstResult] = await runAttempt({
executions: [execution],
attempt: 1,
campaignId,
options,
return executeWithAutomaticRetry({
task: execution.task, options,
qualificationCandidate: execution.profile.qualificationCandidate !== undefined,
cancelled: () => cancelled,
onRetry: result => console.warn(
`Retrying ${execution.id} in a fresh isolated harness after ${result.failureClass!.replaceAll("_", " ")}`,
),
runAttempt: async attempt => {
const [result] = await runAttempt({ executions: [execution], attempt, campaignId, options });
if (!result) throw new Error(`No result produced for ${execution.id} attempt ${attempt}`);
return result;
},
});
if (!firstResult) throw new Error(`No result produced for ${execution.id}`);
if (
execution.profile.qualificationCandidate !== undefined ||
options.ui ||
options.debug ||
firstResult.status !== "failed" ||
!firstResult.failureClass ||
!shouldRetryFailure(firstResult.failureClass, options.maxAutomaticRetries)
) {
return firstResult;
}
if (cancelled) throw new Error("Runner E2E campaign cancelled");
console.warn(
`Retrying ${execution.id} in a fresh isolated harness after ${firstResult.failureClass.replaceAll("_", " ")}`,
);
const [retryResult] = await runAttempt({
executions: [execution],
attempt: 2,
campaignId,
options,
});
if (!retryResult)
throw new Error(`No retry result produced for ${execution.id}`);
return retryResult;
}
async function runWithConcurrency<T, R>(
@@ -986,9 +991,33 @@ async function main() {
return;
}
if (executions.some(execution => execution.task.flow === "provider_connection")) {
const { runConnectionCampaign } = await import("./connection-launch.js");
await runConnectionCampaign({ executions, catalog: runnerMatrix, configFile: options.connectionConfig, repositoryRoot });
return;
}
if (options.connectionConfig) throw new Error("--connection-config is only supported by the provider-connections suite");
// Keep admission before local-env loading and credential checks. Pending
// profiles remain discoverable, but cannot reach a provider.
assertRunnerE2EPrerequisites(executions);
assertNativeCompletionSelection(executions);
assertNativeInstructionSelection(executions);
const campaignId = cleanId(
process.env.PAPERCLIP_E2E_CAMPAIGN_ID ??
`local-${new Date().toISOString().replace(/[:.]/g, "-")}`,
);
const summaryDir = path.join(resultsRoot, campaignId);
await mkdir(summaryDir, { recursive: true });
if (executions.some(execution => execution.suite.id === "native-completion")) {
process.env[NATIVE_COMPLETION_PREFLIGHT_ENV] = prepareNativeCompletionPreflight(summaryDir);
}
if (executions.some(execution => execution.suite.id === NATIVE_INSTRUCTION_SUITE)) {
process.env[NATIVE_INSTRUCTION_PREFLIGHT_ENV] = prepareNativeInstructionPreflight(summaryDir);
}
if (executions.some(execution => execution.suite.id === "stock-harness")) {
process.env[STOCK_PREFLIGHT_ENV] = prepareStockHarnessPreflight(summaryDir);
}
await loadLocalEnvironment(process.env);
assertRemoteNativeEvidencePrerequisites(executions, process.env);
@@ -1011,12 +1040,7 @@ async function main() {
);
}
const campaignId = cleanId(
process.env.PAPERCLIP_E2E_CAMPAIGN_ID ??
`local-${new Date().toISOString().replace(/[:.]/g, "-")}`,
);
const summaryDir = path.join(resultsRoot, campaignId);
await mkdir(summaryDir, { recursive: true });
await writeFile(
path.join(summaryDir, "invocation-policy.json"),
`${JSON.stringify(
@@ -1024,6 +1048,11 @@ async function main() {
version: 1,
maxAutomaticRetries: options.maxAutomaticRetries,
retryClasses: ["transient_infrastructure", "provider_variance"],
executions: executions.map(execution => ({
executionId: execution.id,
automaticRetryPolicy: execution.task.automaticRetryPolicy ?? "default",
maxAutomaticRetries: effectiveAutomaticRetryLimit(execution.task, options.maxAutomaticRetries),
})),
},
null,
2,
tests/runner-e2e/lifecycle-baseline.test.ts
+101 -2
View File
@@ -22,7 +22,7 @@ function recording(): LifecycleCheckpoint[] {
{
...structuredClone(base),
phase: "initial",
issue: { id: "task", status: "in_review" },
issue: { id: "task", status: "in_review", assigneeAgentId: "agent" },
interactions: [
{ id: "question", kind: "ask_user_questions", status: "pending" },
],
@@ -30,7 +30,7 @@ function recording(): LifecycleCheckpoint[] {
{
...structuredClone(base),
phase: "final",
issue: { id: "task", status: "done" },
issue: { id: "task", status: "done", assigneeAgentId: "agent" },
interactions: [
{ id: "question", kind: "ask_user_questions", status: "answered" },
],
@@ -101,3 +101,102 @@ describe("LCA Product E2E evidence calibration", () => {
expect(failures(r)).toContain("lifecycle.initial.revision:plan-approval");
});
});
describe("settled waiting and every resumed question", () => {
it.each(["todo", "in_progress", "blocked", "done", "cancelled"])(
"rejects a settled question left %s even when the final result succeeds", (status) => {
const r = recording();
r[0].issue.status = status;
expect(failures(r)).toContain("lifecycle.initial.wait-state");
},
);
it.each(["executionRunId", "scheduledRetry", "activeRecoveryAction", "monitorNextCheckAt"] as const)(
"rejects a waiting task with leftover %s", (field) => {
const r = recording();
Object.assign(r[0].lifecycle!, { [field]: "unexpected" });
expect(failures(r)).toContain("lifecycle.initial.wait-state");
},
);
it.each(["queued", "running", "failed", "cancelled", "timed_out"])(
"rejects an unexplained %s run during a settled wait", (status) => {
const r = recording();
r[0].runs[0].status = status;
expect(failures(r)).toContain("lifecycle.initial.wait-state");
},
);
it.each(["id", "assigneeAgentId"] as const)("rejects changed task %s", field => {
const r = recording();
r[1].issue[field] = "other";
expect(failures(r)).toContain("lifecycle.task-owner-preserved");
});
it("fails closed when waiting-state or assignee evidence is absent", () => {
const r = recording();
delete r[0].lifecycle;
delete r[0].issue.assigneeAgentId;
expect(failures(r)).toContain("lifecycle.initial.wait-state");
expect(failures(r)).toContain("lifecycle.task-owner-preserved");
});
function pausedRecording() {
const r = recording();
r[0].issue.status = "in_progress";
r[0].runs[0].status = "running";
r[0].lifecycle!.executionRunId = "run-1";
r[0].interactions = [{ id: "question", kind: "ask_user_questions", status: "pending",
sourceRunId: "run-1", payload: { runtimeRequestId: "request-1" } }];
r[1].runs = [r[1].runs[0]];
return r;
}
it("accepts the native provider-question bridge paused in its original run", () => {
expect(failures(pausedRecording())).toEqual([]);
});
it.each(["sourceRunId", "runtimeRequestId", "runtimeMode", "executionRunId"])(
"rejects an unbound provider pause: %s", field => {
const r = pausedRecording();
if (field === "runtimeMode") r[0].runs[0].runtimeMode = "legacy";
else if (field === "executionRunId") r[0].lifecycle!.executionRunId = "other";
else {
const interaction = r[0].interactions[0] as Record<string, any>;
if (field === "sourceRunId") interaction.sourceRunId = "other";
else delete interaction.payload.runtimeRequestId;
}
expect(failures(r)).toContain("lifecycle.initial.wait-state");
},
);
function twoQuestions() {
const r = recording();
const middle = structuredClone(r[1]);
middle.phase = "answered";
middle.issue.status = "in_review";
middle.interactions.push({ id: "second-question", kind: "ask_user_questions", status: "pending" });
r[1].interactions.push({ id: "second-question", kind: "ask_user_questions", status: "answered" });
r[1].runs.push({ id: "run-3", status: "succeeded", runtimeMode: "native" });
return [r[0], middle, r[1]];
}
it("accepts two sequential questions with all run and answer receipts retained", () => {
expect(failures(twoQuestions())).toEqual([]);
});
it.each(["missing", "replaced", "cancelled", "duplicate"])(
"rejects a %s second answer although the first answer and final task are correct", mutation => {
const r = twoQuestions();
const final = r[2].interactions as Array<Record<string, any>>;
if (mutation === "missing") final.pop();
else if (mutation === "replaced") final[1].id = "replacement";
else if (mutation === "cancelled") final[1].status = "cancelled";
else final.push({ ...final[1] });
expect(failures(r)).toContain("lifecycle.answer:second-question");
},
);
it("rejects losing the intermediate run even when the first receipt survives", () => {
const r = twoQuestions();
r[2].runs = r[2].runs.filter(run => run.id !== "run-2");
expect(failures(r)).toContain("lifecycle.final.preserved-runs");
});
it("rejects missing or empty question IDs", () => {
const r = recording();
r[0].interactions = [{ id: "", kind: "ask_user_questions", status: "pending" }];
r[1].interactions = [{ id: "", kind: "ask_user_questions", status: "answered" }];
expect(failures(r)).toContain("lifecycle.initial.durable-wait");
expect(failures(r)).toContain("lifecycle.answer:");
});
});
tests/runner-e2e/lifecycle-baseline.ts
+52 -15
View File
@@ -7,6 +7,8 @@ export interface LifecycleSnapshot {
}
export type LifecycleCheckpoint = ContinuationCheckpoint & {
lifecycle?: LifecycleSnapshot;
// Successful persisted mutations; not a count of failed tool/HTTP attempts.
activity?: unknown[];
};
type Interaction = {
id?: string;
@@ -14,6 +16,7 @@ type Interaction = {
status?: string;
sourceRunId?: string;
payload?: {
runtimeRequestId?: string;
target?: {
type?: string;
issueId?: string;
@@ -22,6 +25,28 @@ type Interaction = {
};
};
};
const nonempty = (value: unknown): value is string =>
typeof value === "string" && value.trim().length > 0;
/** Semantic questions yield the run; provider-native questions instead pause
* inside it. Only a pending runtime request bound to that running native run
* permits retaining its execution lock at a waiting checkpoint. */
function hasHealthyWait(checkpoint: LifecycleCheckpoint, pending: Interaction[]) {
const state = checkpoint.lifecycle;
if (!state || state.scheduledRetry !== null || state.activeRecoveryAction !== null ||
state.monitorNextCheckAt !== null || checkpoint.runs.length === 0) return false;
const active = checkpoint.runs.filter(run => run.status !== "succeeded");
if (active.length === 0)
return checkpoint.issue.status === "in_review" && state.executionRunId === null;
if (active.length !== 1) return false;
const run = active[0];
return run.status === "running" && run.runtimeMode === "native" && nonempty(run.id) &&
state.executionRunId === run.id && ["in_progress", "in_review"].includes(checkpoint.issue.status) &&
pending.some(interaction => interaction.kind === "ask_user_questions" &&
nonempty(interaction.id) && interaction.sourceRunId === run.id &&
nonempty(interaction.payload?.runtimeRequestId));
}
/** Independent durable-state oracle. It never interprets response prose. */
export function gradeLifecycleBaseline(checkpoints: LifecycleCheckpoint[]) {
const checks: Array<{ id: string; passed: boolean; detail: string }> = [];
@@ -36,6 +61,14 @@ export function gradeLifecycleBaseline(checkpoints: LifecycleCheckpoint[]) {
waiting.every((c) => !!c.lifecycle),
"Every checkpoint must retain lifecycle evidence from the public task API.",
);
const first = waiting[0];
check(
"lifecycle.task-owner-preserved",
nonempty(first?.issue.id) && nonempty(first?.issue.assigneeAgentId) &&
checkpoints.every(c => c.issue.id === first.issue.id &&
c.issue.assigneeAgentId === first.issue.assigneeAgentId),
"Waiting and resuming retain the same task and assigned agent.",
);
for (const c of waiting) {
const pending = (c.interactions as Interaction[]).filter(
(i) => i.status === "pending",
@@ -44,7 +77,7 @@ export function gradeLifecycleBaseline(checkpoints: LifecycleCheckpoint[]) {
`lifecycle.${c.phase}.durable-wait`,
pending.some(
(i) =>
typeof i.id === "string" &&
nonempty(i.id) &&
[
"ask_user_questions",
"request_confirmation",
@@ -53,6 +86,11 @@ export function gradeLifecycleBaseline(checkpoints: LifecycleCheckpoint[]) {
),
"Waiting must have an identifiable pending interaction, not only an assistant message.",
);
check(
`lifecycle.${c.phase}.wait-state`,
hasHealthyWait(c, pending),
"A settled wait is in_review with no execution lock, retry, recovery or monitor. A bound native provider question may keep its paused run and lock.",
);
for (const i of pending.filter(
(i) => i.payload?.target?.type === "issue_document",
)) {
@@ -90,27 +128,26 @@ export function gradeLifecycleBaseline(checkpoints: LifecycleCheckpoint[]) {
),
"Completion has no unresolved interaction.",
);
const first = waiting[0];
if (first && final) {
for (const question of (first.interactions as Interaction[]).filter(
(i) => i.kind === "ask_user_questions" && i.status === "pending",
)) {
const questions = new Map(waiting.flatMap(c =>
(c.interactions as Interaction[]).filter(i => i.kind === "ask_user_questions" && i.status === "pending")
.map(i => [i.id, i] as const)));
for (const question of questions.values()) {
const answers = (final.interactions as Interaction[]).filter(i => i.id === question.id);
check(
`lifecycle.answer:${question.id}`,
typeof question.id === "string" &&
(final.interactions as Interaction[]).some(
(i) => i.id === question.id && i.status === "answered",
),
"The original question identity has a durable answer.",
nonempty(question.id) && answers.length === 1 &&
answers[0].kind === "ask_user_questions" && answers[0].status === "answered",
"Every observed pending question retains exactly one answered identity, including later questions.",
);
}
const originalIds = new Set(first.runs.map((r) => r.id));
const finalIds = new Set(final.runs.map(r => r.id));
check(
"lifecycle.final.preserved-runs",
first.runs.length > 0 &&
[...originalIds].every((id) => final.runs.some((r) => r.id === id)) &&
new Set(final.runs.map((r) => r.id)).size === final.runs.length,
"Original run receipts remain present without duplicated IDs.",
checkpoints.every(c => c.runs.length > 0 && c.runs.every(r => nonempty(r.id)) &&
new Set(c.runs.map(r => r.id)).size === c.runs.length) &&
waiting.every(c => c.runs.every(r => finalIds.has(r.id))),
"All intermediate run receipts survive in the final snapshot without duplicated or empty IDs.",
);
}
return checks;
tests/runner-e2e/live-fixtures.test.ts
+74 -3
View File
@@ -4,6 +4,75 @@ import { runnerMatrix } from "./catalog.js";
import { setupLiveFixtures } from "./live-fixtures.js";
describe("live runner fixtures", () => {
it.each(["runner-codex", "runner-acpx-claude", "runner-opencode"].flatMap(profile =>
["hire-reuse", "delegate-feedback"].map(task => [profile, task]))) (
"applies %s %s budget stops in actual setup with a nonce and only its selected credential", async (profile, task) => {
const execution = runnerMatrix.find(row => row.id === `everyday-workflows.${profile}.local.${task}`)!;
let agentBody: any;
let companyBody: any;
const api = {
async get() { return [{ id: "local", driver: "local" }]; },
async postSensitive(url: string) { return url.endsWith("/ai-connections") ? { connectionId: "account" } : { id: "secret" }; },
async post(url: string, data: any) {
if (url === "/api/companies") { companyBody = data; return { id: "company", name: "Test" }; }
if (url.endsWith("/agents")) { agentBody = data; return { id: "lead", ...data }; }
throw new Error(`Unexpected POST ${url}`);
},
} as unknown as RunnerApi;
const fixtures = await setupLiveFixtures({ api, execution, executionNonce: "random-nonce", workspacePath: "/tmp/test",
credentials: { [execution.profile.credential]: "test-value" } });
expect(companyBody.budgetMonthlyCents).toBe(1_000);
expect(agentBody.budgetMonthlyCents).toBe(1_000);
await fixtures.teardown();
},
);
it.each(["runner-codex", "runner-acpx-claude", "runner-opencode"])(
"sets both connection-guidance budget stops for %s before execution", async profile => {
const execution = runnerMatrix.find(row => row.suite.id === "native-connection-guidance" && row.profile.id === profile)!;
let companyBudget: unknown;
let agentBudget: unknown;
const api = {
async get() { return [{ id: "local", driver: "local" }]; },
async postSensitive() { return { id: "secret" }; },
async post(url: string, data: any) {
if (url === "/api/companies") { companyBudget = data.budgetMonthlyCents; return { id: "company", name: "Test" }; }
if (url.endsWith("/agents")) { agentBudget = data.budgetMonthlyCents; return { id: "lead", ...data }; }
throw new Error("Unexpected POST " + url);
},
} as unknown as RunnerApi;
const fixtures = await setupLiveFixtures({ api, execution, executionNonce: "nonce", workspacePath: "/tmp/test",
credentials: { [execution.profile.credential]: "test-value" } });
expect(companyBudget).toBe(1_000);
expect(agentBudget).toBe(1_000);
await fixtures.teardown();
},
);
it.each(["runner-codex", "legacy-codex", "runner-acpx-claude", "legacy-opencode"])(
"creates a production-default %s hire with company and agent budget stops", async (profile) => {
const execution = runnerMatrix.find(row => row.suite.id === "stock-harness" && row.profile.id === profile)!;
let agentBody: any;
const api = {
async get() { return [{ id: "local", driver: "local" }]; },
async postSensitive() { return { id: "secret" }; },
async post(url: string, data: any) {
if (url === "/api/companies") {
expect(data.budgetMonthlyCents).toBe(1_000);
return { id: "company", name: "Fixture" };
}
if (url.endsWith("/agents")) { agentBody = data; return { id: "agent", ...data }; }
throw new Error(`Unexpected POST ${url}`);
},
} as unknown as RunnerApi;
const fixtures = await setupLiveFixtures({ api, execution, executionNonce: "nonce", workspacePath: "/tmp/fixture",
credentials: { [execution.profile.credential]: "fixture-key" } });
expect(agentBody).not.toHaveProperty("instructionsBundle");
expect(agentBody.budgetMonthlyCents).toBe(1_000);
expect(agentBody.adapterConfig.env[execution.profile.credential]).toEqual({ type: "secret_ref", secretId: "secret", version: "latest" });
await fixtures.teardown();
},
);
it("anchors extended file validation to a public project workspace for local and remote copy-back", async () => {
const execution = runnerMatrix.find(e => e.id === "extended-harnesses.runner-acpx-pi.local.file-edit-validate")!;
let projectBody: any;
@@ -23,8 +92,11 @@ describe("live runner fixtures", () => {
});
it.each([
["runner-codex", "hiring-templates", "hire-coder-template-reuse"],
["runner-acpx-claude", "hiring-templates", "hire-coder-template-reuse"],
["runner-codex", "everyday-workflows", "hire-reuse"],
["runner-acpx-claude", "everyday-workflows", "hire-reuse"],
["runner-opencode", "everyday-workflows", "hire-reuse"],
["runner-codex", "agent-chat-hardening", "hire-delegate-reuse"],
["runner-acpx-claude", "agent-chat-hardening", "hire-delegate-reuse"],
])(
@@ -38,7 +110,7 @@ describe("live runner fixtures", () => {
e.environment.id === "local",
)!;
const provider =
profile === "runner-acpx-claude" ? "anthropic" : "openai";
profile === "runner-acpx-claude" ? "anthropic" : profile === "runner-opencode" ? "openrouter" : "openai";
let connected = false;
let agentBody: any;
const api = {
@@ -75,8 +147,7 @@ describe("live runner fixtures", () => {
executionNonce: "nonce",
workspacePath: "/tmp/test",
credentials: {
OPENAI_API_KEY: "test-value",
ANTHROPIC_API_KEY: "test-value",
[execution.profile.credential]: "test-value",
},
});
expect(connected).toBe(true);
tests/runner-e2e/live-fixtures.ts
+19 -8
View File
@@ -1,5 +1,6 @@
import { verifyInstalledDaytonaPlugin } from "./installed-daytona-plugin.js";
import { NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js";
import path from "node:path";
import { installedReleaseDaytonaPlugin } from "./installed-release.js";
import { isManagedHiringCase } from "./chat-cases.js";
import { FixtureRegistry } from "./fixture-registry.js";
import { TASK_TITLE_BUDGET_CENTS } from "./task-titles.js";
@@ -39,7 +40,7 @@ interface AgentRecord {
interface ManagedAccountFixture {
connectionId: string;
binding: {
provider: "openai" | "anthropic";
provider: "openai" | "anthropic" | "openrouter";
method: "api_key";
mode: "responsible_user";
};
@@ -116,7 +117,9 @@ export async function setupLiveFixtures(input: {
async setup() {
const installed = await verifyInstalledDaytonaPlugin(process.env, [execution]);
return api.post<PluginRecord>("/api/plugins/install", {
packageName: installed?.packageRoot ?? path.resolve(
packageName: process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI
? installedReleaseDaytonaPlugin(process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI, process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN, process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN_VERSION)
: path.resolve(
import.meta.dirname,
"../../packages/plugins/sandbox-providers/daytona",
),
@@ -137,7 +140,10 @@ export async function setupLiveFixtures(input: {
return api.post<CompanyRecord>("/api/companies", {
name: `Runner E2E ${execution.id} ${input.executionNonce}`,
description: "Ephemeral paid full-stack runner acceptance fixture",
budgetMonthlyCents: execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS : 0,
budgetMonthlyCents: ["native-completion", "native-instruction-consolidation", "native-connection-guidance"].includes(execution.suite.id)
|| (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id)) ? NATIVE_COMPLETION_BUDGET_CENTS
: execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS
: execution.suite.id === "stock-harness" ? 1_000 : 0,
});
},
async teardown() {
@@ -243,10 +249,11 @@ export async function setupLiveFixtures(input: {
dependencies: ["company"],
async setup(resolved) {
const company = value<CompanyRecord>(resolved, "company");
const provider =
execution.profile.provider === "acpx" ? "anthropic" : "openai";
const key =
provider === "anthropic" ? "ANTHROPIC_API_KEY" : "OPENAI_API_KEY";
const key = execution.profile.credential;
const provider = key === "ANTHROPIC_API_KEY" ? "anthropic"
: key === "OPENROUTER_API_KEY" ? "openrouter"
: key === "OPENAI_API_KEY" ? "openai" : null;
if (!provider) throw new Error(`Unsupported managed hiring credential ${key}`);
const apiKey = input.credentials[key];
if (!apiKey) throw new Error(`Missing credential ${key}`);
const account = await api.postSensitive<{ connectionId: string }>(
@@ -289,6 +296,10 @@ export async function setupLiveFixtures(input: {
secretRefs,
executionId: input.executionNonce,
});
if (["stock-harness", "native-connection-guidance"].includes(execution.suite.id)
|| (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id))) {
agent.budgetMonthlyCents = 1_000;
}
if (managedHiring) {
const account = value<ManagedAccountFixture>(resolved, "ai-connection");
const config = agent.adapterConfig as Record<string, unknown>;
tests/runner-e2e/matchers.ts
+9 -4
View File
@@ -27,7 +27,7 @@ export interface MatcherObservation {
runtimeMode?: string;
environment?: string;
files?: Record<string, string>;
artifacts?: Array<{ name: string; mimeType?: string }>;
artifacts?: Array<{ name: string; mimeType?: string; content?: string; contentVerified?: boolean }>;
json?: unknown;
}
@@ -132,12 +132,14 @@ export async function evaluateMatcher(
actual = undefined;
passed = false;
}
} else if (matcher.kind === "artifact_exists") {
} else if (matcher.kind === "artifact_exists" || matcher.kind === "artifact_exact") {
actual = observation.artifacts ?? [];
passed = (observation.artifacts ?? []).some(
(artifact) =>
artifact.name === matcher.name &&
(!matcher.mimeType || artifact.mimeType === matcher.mimeType),
(!matcher.mimeType || artifact.mimeType === matcher.mimeType) &&
(matcher.kind === "artifact_exists" ||
(artifact.contentVerified === true && artifact.content === matcher.expected)),
);
} else if (matcher.kind === "json_path") {
actual = readJsonPath(observation.json, matcher.path);
@@ -170,7 +172,7 @@ export async function evaluateMatchers(
export function persistedFinalRunMessage(
comments: Array<{ id: string; createdByRunId?: string | null; body?: string | null }>,
run: { id: string; resultJson?: Record<string, unknown> | null },
run: { id: string; runtimeMode?: string; resultJson?: Record<string, unknown> | null },
): string {
const runComments = comments.filter(comment => comment.createdByRunId === run.id);
const decision = run.resultJson?.presentationDecision;
@@ -181,5 +183,8 @@ export function persistedFinalRunMessage(
if (typeof selectedId === "string") {
return runComments.find(comment => comment.id === selectedId)?.body ?? "";
}
// Native finalization can publish a deliverable-preparation comment before
// its response decision. That earlier comment is not the selected reply.
if (run.runtimeMode === "native") return "";
return runComments.map(comment => comment.body ?? "").join("\n");
}
tests/runner-e2e/native-active-stop-evidence.ts
+9 -10
View File
@@ -2,14 +2,13 @@ import { createHash } from "node:crypto";
import { canonicalJson } from "../../packages/shared/src/portability-hash.js";
import { hasAcpxNativeOrigin } from "./acpx-native-origin.js";
import { isValidNativePrpEnvelope } from "./native-event-envelope.js";
import { assertCopilotRemoteRetirement, copilotRemoteDeniedSample, type CopilotRemoteSnapshot } from "./copilot-protection-evidence.js";
import { readCopilotToolEvidence } from "./copilot-evidence.js";
import { assertNativeRemoteRetirement, nativeRemoteDeniedSample, type NativeRemoteSnapshot } from "./native-remote-evidence.js";
import { readCursorToolEvidence } from "./cursor-native-evidence.js";
import { bootstrapReadExecutionId, withoutProvenBootstrapReads, type BootstrapReadProof } from "./native-bootstrap-read-proof.js";
type Row = Record<string, any>;
export interface ActiveStopCaller { type: "board"; userId: "local-board"; source: "local_implicit" }
export type ActiveStopProvider = "cursor" | "copilot";
export type ActiveStopProvider = "cursor";
export interface ActiveStopScope { provider: ActiveStopProvider; companyId: string; issueId: string; runId: string; target: string; commandSha256?: string }
export interface ActiveStopPending {
schema: "paperclip.e2e.native-active-stop-pending.v2";
@@ -62,7 +61,7 @@ const closures = new Set(["runtime_request.resolved", "runtime_request.cancelled
function origin(events: readonly unknown[], scope: ActiveStopScope, bootstrap?: BootstrapReadProof) {
fail([scope.companyId, scope.issueId, scope.runId, scope.target].every(id) && ["cursor", "copilot"].includes(scope.provider), "exact scope required");
const rows = canonicalRows(events, scope);
const notices = scope.provider === "cursor" ? readCursorToolEvidence(events, scope.runId) : readCopilotToolEvidence(events, scope.runId);
const notices = readCursorToolEvidence(events, scope.runId);
const requests = notices.filter(n => n.stage === "permission_requested");
fail(requests.length === 1, "one native permission required");
const notice = requests[0]!;
@@ -120,7 +119,7 @@ export function observeActiveStopPending(input: { events: readonly unknown[]; ru
fail(isActiveStopCaller(input.caller) && uuid(input.cancellationRequestId) && run.id === scope.runId && run.companyId === scope.companyId && run.nativeIssueId === scope.issueId
&& run.status === "running" && run.runtimeMode === "native" && issue.id === scope.issueId && issue.companyId === scope.companyId && issue.status === "in_progress"
&& run.resultJson?.startupCancellation == null && run.resultJson?.nativeCancellation == null, "run is not fresh active work");
fail(!(scope.provider === "cursor" ? readCursorToolEvidence(input.events, scope.runId) : readCopilotToolEvidence(input.events, scope.runId)).some(n => n.stage === "permission_delivered"), "permission already answered");
fail(!(readCursorToolEvidence(input.events, scope.runId)).some(n => n.stage === "permission_delivered"), "permission already answered");
fail(!proof.rows.some(x => terminals.has(x.event.eventType) || closures.has(x.event.eventType)), "request or provider already settled");
fail(!proof.rows.some(x => x.event.eventType === "tool.execution.completed" && rec(x.event.payload).executionId === bootstrapReadExecutionId(proof.notice.toolCallId)), "operation already ended");
return { schema: "paperclip.e2e.native-active-stop-pending.v2", scope, caller: input.caller, requestId: proof.notice.requestId!, toolCallId: proof.notice.toolCallId,
@@ -181,7 +180,7 @@ export function readActiveStopSettlement(input: { events: readonly unknown[]; ru
export interface ActiveStopRemoteObservation {
phase: "before-request" | "pending" | "owned-process-retirement";
source: "live-snapshot" | "retirement-seal";
snapshot: CopilotRemoteSnapshot;
snapshot: NativeRemoteSnapshot;
}
/** A per-turn Daytona observer seals itself when the owned tree retires. The
@@ -193,10 +192,10 @@ export function readActiveStopRemoteRetirement(input: {
fail(observations.length === 3
&& observations.map(o => `${o.phase}:${o.source}`).join(",") === "before-request:live-snapshot,pending:live-snapshot,owned-process-retirement:retirement-seal",
"remote seal cannot stand in for a fresh causal sample");
const [baseline, pending, terminal] = observations.map(o => o.snapshot) as [CopilotRemoteSnapshot, CopilotRemoteSnapshot, CopilotRemoteSnapshot];
const [baseline, pending, terminal] = observations.map(o => o.snapshot) as [NativeRemoteSnapshot, NativeRemoteSnapshot, NativeRemoteSnapshot];
fail(baseline.binding.companyId === input.scope.companyId && baseline.binding.runId === input.scope.runId, "remote observation belongs to another run");
for (const snapshot of [baseline, pending, terminal]) {
fail(!copilotRemoteDeniedSample(snapshot, baseline, input.scope.target, "pending").exists, "remote target changed");
fail(!nativeRemoteDeniedSample(snapshot, baseline, input.scope.target, "pending").exists, "remote target changed");
}
fail(BigInt(baseline.observedMonotonicNs) < BigInt(pending.observedMonotonicNs)
&& BigInt(pending.observedMonotonicNs) < BigInt(terminal.observedMonotonicNs), "remote observation reused or out of order");
@@ -204,8 +203,8 @@ export function readActiveStopRemoteRetirement(input: {
&& baseline.processes.captured && pending.processes.captured
&& baseline.processes.live.includes(baseline.processes.root?.pid ?? -1)
&& pending.processes.live.includes(pending.processes.root?.pid ?? -1), "remote pending lifetime unproven");
assertCopilotRemoteRetirement(terminal, baseline);
assertCopilotRemoteRetirement(terminal, pending);
assertNativeRemoteRetirement(terminal, baseline);
assertNativeRemoteRetirement(terminal, pending);
for (const snapshot of [baseline, pending]) {
fail(snapshot.processes.journal.every(p => terminal.processes.journal.some(q =>
p.pid === q.pid && p.startTicks === q.startTicks && p.bootId === q.bootId)), "remote descendant journal lost");
tests/runner-e2e/native-active-stop-flow.ts
+15 -12
View File
@@ -4,8 +4,8 @@ import { expect, type Page } from "@playwright/test";
import { pollUntil, type RunnerApi } from "./api.js";
import { collectRunEvents } from "./run-observations.js";
import { createTaskThroughUi } from "./user-actions.js";
import { createDeniedTargetFixture, exists, observeRunProcesses } from "./copilot-local-fixtures.js";
import { assertCopilotRemoteRetirement, copilotRemoteDeniedSample, prepareCopilotRemoteAction, type CopilotRemoteBootstrap, type CopilotRemoteFixture, type CopilotRemoteSnapshot } from "./copilot-protection-evidence.js";
import { createDeniedTargetFixture, exists, observeRunProcesses } from "./native-local-fixtures.js";
import { assertNativeRemoteRetirement, nativeRemoteDeniedSample, prepareNativeRemoteAction, type NativeRemoteBootstrap, type NativeRemoteFixture, type NativeRemoteSnapshot } from "./native-remote-evidence.js";
import { cursorDeniedCommand } from "./cursor-native-evidence.js";
import { assertActiveStopRetirement, readActiveStopCaller, observeActiveStopPending, readActiveStopSettlement, type ActiveStopRemoteObservation, type ActiveStopCaller, type ActiveStopPending, type ActiveStopScope } from "./native-active-stop-evidence.js";
import type { BootstrapReadProof } from "./native-bootstrap-read-proof.js";
@@ -43,7 +43,7 @@ export async function stopAtPendingPermission(input: {
}
export async function runNativeActiveStopFlow(input: {
page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution; nonce: string; workspacePath: string; deadlineAt: number;
remoteBootstrap?: CopilotRemoteBootstrap;
remoteBootstrap?: NativeRemoteBootstrap;
registerCleanupAssertion(callback: () => Promise<Check[]>): void;
registerBeforeEnvironmentTeardownAssertion(callback: () => Promise<Check[]>): void;
observe(issue: Row, runs: Row[]): void;
@@ -51,7 +51,7 @@ export async function runNativeActiveStopFlow(input: {
evidence(name: string, value: unknown): Promise<void>;
}) {
const { api, page, fixtures, execution, nonce } = input, provider = execution.profile.qualificationCandidate;
if ((provider !== "cursor" && provider !== "copilot") || execution.task.id !== "pending-permission-stop") throw new Error("Unknown native active Stop case");
if (provider !== "cursor" || execution.task.id !== "pending-permission-stop") throw new Error("Unknown native active Stop case");
const remote = execution.environment.id === "daytona";
if ((!remote && execution.environment.id !== "local") || (remote && !input.remoteBootstrap)) throw new Error("Active Stop requires an isolated admitted environment");
const checks: Check[] = []; let issue: Row = {}, runs: Row[] = [], events: Row[] = [];
@@ -65,12 +65,15 @@ export async function runNativeActiveStopFlow(input: {
const remoteObservations: ActiveStopRemoteObservation[] = [];
let retirement: ReturnType<typeof assertActiveStopRetirement> | undefined;
const samples: Array<{ phase: string; absent: boolean }> = [];
let fixture: CopilotRemoteFixture | undefined, baseline: CopilotRemoteSnapshot | undefined, sealed: CopilotRemoteSnapshot | undefined;
let fixture: NativeRemoteFixture | undefined, baseline: NativeRemoteSnapshot | undefined, sealed: NativeRemoteSnapshot | undefined;
let completed: Awaited<ReturnType<typeof stopAtPendingPermission>> | undefined;
const observeProcesses = () => {
// Remote ownership comes from the sandbox boot/start-tick journal. The
// API PID transitions from the controller command to the remote runner.
if (!observer) return;
const run = runs[0], authority = run?.processPid ? { pid: run.processPid, groupId: run.processGroupId, startedAt: run.processStartedAt, runId: run.id } : undefined;
if (authority) { const key = JSON.stringify(authority); if (processIdentity && processIdentity !== key) processError = true; processIdentity ??= key; }
if (observer) processes = observer.sample(authority);
processes = observer.sample(authority);
};
const load = async (): Promise<ActiveStopState> => {
if (issue.id) issue = await api.get<Row>(`/api/issues/${issue.id}`);
@@ -87,7 +90,7 @@ export async function runNativeActiveStopFlow(input: {
if (phase !== "pending") throw new Error("Remote filesystem phase requires a fresh live snapshot");
const snap = await fixture.snapshot(phase);
remoteObservations.push({ phase, source: "live-snapshot", snapshot: snap });
value = !copilotRemoteDeniedSample(snap, baseline, target, "pending").exists;
value = !nativeRemoteDeniedSample(snap, baseline, target, "pending").exists;
await input.evidence(`active-stop-${phase}-remote.json`, snap);
} else value = !await exists(local!.targetPath);
samples.push({ phase, absent: value }); check(`no-effect-${phase}`, value, "Exact target remains absent at the causal observation boundary");
@@ -100,7 +103,7 @@ export async function runNativeActiveStopFlow(input: {
await load();
if (remote) {
if (!fixture || !baseline) throw new Error("Remote observer was never armed");
sealed ??= await fixture.finish(); assertCopilotRemoteRetirement(sealed, baseline); processes = sealed.processes;
sealed ??= await fixture.finish(); assertNativeRemoteRetirement(sealed, baseline); processes = sealed.processes;
} else if (processes.captured && processes.live.length) {
await pollUntil({ label: "active Stop provider retirement", deadlineAt: Date.now() + 5000, intervalMs: 100, load: async () => { observeProcesses(); return processes; }, accept: p => p.live.length === 0 });
}
@@ -133,13 +136,13 @@ export async function runNativeActiveStopFlow(input: {
check("explicit-per-turn-policy", configured.adapterConfig?.acpxPermissionMode === "approve-reads" && configured.adapterConfig?.lifecycleMode === "per_turn" && (provider !== "cursor" || configured.adapterConfig?.acpxSessionMode === "agent"), "Agent mode and per-turn restrictive permission policy selected before startup");
const project = await api.post<Row>(`/api/companies/${fixtures.company.id}/projects`, { name: `Native active Stop ${nonce}`, executionWorkspacePolicy: { enabled: true, defaultMode: "shared_workspace", sharedWorkspaceConcurrency: "serialize", allowIssueOverride: false, environmentId: fixtures.environment.id, workspaceStrategy: { type: "project_primary" } }, workspace: { name: "Primary", sourceType: "local_path", cwd: input.workspacePath, isPrimary: true } });
if (!remote) await sample("before-request");
await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), prompt: remote ? input.remoteBootstrap!.prompt(nonce) : prompt(), workMode: "standard", projectName: project.name });
issue = (await pollUntil({ label: "browser-created active Stop task", deadlineAt: input.deadlineAt, load: async () => (await api.get<Row[]>(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(i => i.title === execution.task.buildTitle(nonce)), accept: Boolean }))!;
const createdTask = await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), prompt: remote ? input.remoteBootstrap!.prompt(nonce) : prompt(), workMode: "standard", projectName: project.name, requireExplicitTitle: true });
issue = (await pollUntil({ label: "browser-created active Stop task", deadlineAt: input.deadlineAt, load: async () => (await api.get<Row[]>(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(i => i.id === createdTask.issueId), accept: Boolean }))!;
if (remote) {
await pollUntil({ label: "active Stop remote bootstrap", deadlineAt: input.deadlineAt, load, accept: state => state.run.status === "running" });
const bound = await input.remoteBootstrap!.bindAndRelease({ issueId: issue.id, runId: runs[0]!.id, targets: [target], actionPrompt: async value => {
fixture = value; if (provider === "cursor") command = cursorDeniedCommand(join(value.remoteCwd, target));
const prepared = await prepareCopilotRemoteAction({ fixture: value, companyId: fixtures.company.id, environmentId: fixtures.environment.id, runId: runs[0]!.id, target, prompt: prompt() });
const prepared = await prepareNativeRemoteAction({ fixture: value, companyId: fixtures.company.id, environmentId: fixtures.environment.id, runId: runs[0]!.id, target, prompt: prompt() });
baseline = prepared.baseline;
remoteObservations.push({ phase: "before-request", source: "live-snapshot", snapshot: baseline });
await input.evidence("active-stop-before-request-remote.json", baseline); return prepared.prompt;
@@ -165,7 +168,7 @@ export async function runNativeActiveStopFlow(input: {
if (remote) {
// finish drains the observer's automatic retirement seal. It does not
// extend the remote watch through subsequent host UI/cleanup assertions.
sealed = await fixture!.finish(); assertCopilotRemoteRetirement(sealed, baseline!); processes = sealed.processes;
sealed = await fixture!.finish(); assertNativeRemoteRetirement(sealed, baseline!); processes = sealed.processes;
remoteObservations.push({ phase: "owned-process-retirement", source: "retirement-seal", snapshot: sealed });
await input.evidence("active-stop-owned-process-retirement-remote.json", sealed);
} else await sample("after-stop");
tests/runner-e2e/native-active-stop.test.ts
+9 -12
View File
@@ -5,7 +5,6 @@ import { stopAtPendingPermission } from "./native-active-stop-flow.js";
import { runnerMatrix, runnerSuites, suiteDefinitionHash } from "./catalog.js";
import { selectRunnerExecutions, parseRunnerSelectors } from "./selectors.js";
import { createCursorToolEvidence } from "../../packages/paperclip-runner/src/drivers/acpx/cursor-tool-evidence.js";
import { createCopilotToolEvidence } from "../../packages/paperclip-runner/src/drivers/acpx/copilot-tool-evidence.js";
import type { CanonicalProviderEvent } from "../../packages/paperclip-runner/src/provider-events.js";
import { CodexSessionState } from "../../packages/paperclip-runner/src/drivers/codex/codex-session-state.js";
@@ -14,7 +13,7 @@ import { mapTerminalTurn } from "../../packages/paperclip-runner/src/drivers/cod
const caller = readActiveStopCaller({ deploymentMode: "local_trusted" }, { session: { userId: "local-board", id: "paperclip:local_implicit:local-board" } });
const cancellationRequestId = "11111111-2222-4333-8444-555555555555";
type Row = Record<string, any>;
function fixture(provider: ActiveStopProvider = "copilot") {
function fixture(provider: ActiveStopProvider = "cursor") {
const scope = { provider, companyId: "company", issueId: "issue", runId: "run", target: "target.txt", ...(provider === "cursor" ? { commandSha256: `sha256:${"a".repeat(64)}` } : {}) };
const row = (seq: number, eventType: string, payload: Row): Row => ({ companyId: "company", runId: "run", seq, eventType, protocolSchemaVersion: 1,
payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", eventType, runId: "run", turnId: "turn",
@@ -82,7 +81,7 @@ function withProjectedArrivalOrder(provider: ActiveStopProvider, order: typeof a
}
f.events.length = 0;
const append = (eventType: string, value: Row) => f.events.push(f.row(f.events.length + 1, eventType, value));
const projector = (provider === "cursor" ? createCursorToolEvidence : createCopilotToolEvidence)({
const projector = (createCursorToolEvidence)({
sessionId: "native-session", turnId: "turn", workingDirectory: "/fixture", active: () => true,
emit: (event: CanonicalProviderEvent) => { append(event.eventType, event.payload); },
});
@@ -125,11 +124,10 @@ function withSessionPrefix(provider: ActiveStopProvider = "cursor") {
}
describe("definitely active native permission Stop", () => {
describe.each(["cursor", "copilot"] as const)("%s pre-Stop arrival orders", provider => {
describe.each(["cursor"] as const)("%s pre-Stop arrival orders", provider => {
it.each(arrivalOrders)("accepts %s only after both exact origins exist and retains them through settlement", order => {
const f = withProjectedArrivalOrder(provider, order), pending = f.pending();
expect(f.nativeStageOrder()).toEqual(provider === "copilot" && order === "permission-first"
? ["permission_requested", "tool"] : ["tool", "permission_requested"]);
expect(f.nativeStageOrder()).toEqual(["tool", "permission_requested"]);
expect(pending.schema).toBe("paperclip.e2e.native-active-stop-pending.v2");
expect(pending.toolOriginRowSha256).toMatch(/^sha256:[a-f0-9]{64}$/u);
expect(pending.toolStartedRowSha256).toMatch(/^sha256:[a-f0-9]{64}$/u);
@@ -189,7 +187,7 @@ describe("definitely active native permission Stop", () => {
});
expect(f.pending).toThrow("duplicate native operation lifecycle");
});
it.each(["cursor", "copilot"] as const)("accepts the mixed v1/v2 session prefix before strict %s pending proof", provider => {
it.each(["cursor"] as const)("accepts the mixed v1/v2 session prefix before strict %s pending proof", provider => {
const f = withSessionPrefix(provider);
expect(f.pending()).toMatchObject({ requestId: "request", toolCallId: "tool", permissionSourceSeq: 6, requestSourceSeq: 7 });
});
@@ -213,7 +211,7 @@ describe("definitely active native permission Stop", () => {
delete payload(f.events.find(row => row.eventType === "runtime_request.created")!).request.details.toolCallId;
expect(f.pending).toThrow("native notice/card identity mismatch");
});
it.each(["cursor", "copilot"] as const)("binds %s's unanswered callback to cancelled provider settlement and caller-owned Stop", provider => {
it.each(["cursor"] as const)("binds %s's unanswered callback to cancelled provider settlement and caller-owned Stop", provider => {
const f = fixture(provider), pending = f.pending(); f.settle();
expect(f.permissionResponse.mock.calls).toEqual([[{ action: "cancel" }]]);
expect(readActiveStopSettlement({ ...f.state(), pending, dispatchMonotonicNs: (BigInt(pending.observedMonotonicNs) + 1n).toString() })).toMatchObject({
@@ -329,7 +327,6 @@ describe("definitely active native permission Stop", () => {
describe("active Stop flow wiring", () => {
it.each([
["cursor", "tool-first"], ["cursor", "permission-first"],
["copilot", "tool-first"], ["copilot", "permission-first"],
] as const)("awaits retention and rechecks real %s %s evidence before issuing the single caller UUID request", async (provider, arrivalOrder) => {
const f = withProjectedArrivalOrder(provider, arrivalOrder); const order: string[] = []; let retainedId = "";
const stop = vi.fn(async (runId: string, id: string) => { expect(runId).toBe("run"); expect(id).toBe(retainedId); order.push("stop"); return f.settle(id); });
@@ -370,11 +367,11 @@ describe("active Stop flow wiring", () => {
});
describe("explicit active Stop discovery", () => {
it("adds exactly four versioned cells without enabling them in --all", () => {
it("adds exactly two versioned cells without enabling them in --all", () => {
const suite = runnerSuites.find(s => s.id === "native-active-stop")!;
const cells = runnerMatrix.filter(e => e.suite === suite);
expect(cells).toHaveLength(4); expect(suite.manualOnly).toBe(true);
expect(cells.map(e => `${e.profile.qualificationCandidate}/${e.environment.id}`).sort()).toEqual(["copilot/daytona", "copilot/local", "cursor/daytona", "cursor/local"]);
expect(cells).toHaveLength(2); expect(suite.manualOnly).toBe(true);
expect(cells.map(e => `${e.profile.qualificationCandidate}/${e.environment.id}`).sort()).toEqual(["cursor/daytona", "cursor/local"]);
expect(cells.every(e => e.task.expectedRunCount === 1 && e.task.flow === "native_active_stop" && e.task.expectedTerminalState?.run === "cancelled")).toBe(true);
expect(suite.definitionMetadata).toMatchObject({ version: 4, evidence: "paperclip.e2e.native-active-stop-settlement.v2", normalCompletionAccepted: false, providerDeath: "not-covered" });
expect(suiteDefinitionHash(suite)).not.toBe(suiteDefinitionHash({ ...suite, definitionMetadata: { ...suite.definitionMetadata, version: 3 } }));
tests/runner-e2e/native-blocker-visible.spec.ts
+28
View File
@@ -0,0 +1,28 @@
import { expect, test } from "@playwright/test";
import { assertNativeBlockerReply } from "./native-blocker-visible.js";
test("accepts a visible blocker explanation rather than requiring a marker-only reply", async ({ page }) => {
await page.setContent('<div data-testid="reply">Deployment is <strong>blocked</strong>.<br>Release Owner must Grant deployment access. BLOCKED_probe</div>');
await assertNativeBlockerReply(page.getByTestId("reply"), "BLOCKED_probe", 100);
});
test("accepts a future access condition while the task remains blocked", async ({ page }) => {
await page.setContent('<div data-testid="reply">Deployment remains blocked until access is granted. Release Owner must Grant deployment access. BLOCKED_probe</div>');
await assertNativeBlockerReply(page.getByTestId("reply"), "BLOCKED_probe", 100);
});
for (const [name, body] of [
["marker only", "BLOCKED_probe"],
["missing owner", "Blocked. Someone must Grant deployment access. BLOCKED_probe"],
["missing action", "Blocked. Release Owner must take action. BLOCKED_probe"],
["missing reason", "Release Owner: Grant deployment access. BLOCKED_probe"],
["contradictory disposition", "Deployment is not blocked. Release Owner completed Grant deployment access. BLOCKED_probe"],
["resolved blocker", "Deployment is no longer blocked. Release Owner must Grant deployment access. BLOCKED_probe"],
["granted access", "Blocked. Access already granted. Release Owner must Grant deployment access. BLOCKED_probe"],
["duplicate replies", "Blocked. Release Owner must Grant deployment access. BLOCKED_probe</div><div data-testid=reply>BLOCKED_probe"],
]) {
test(`rejects ${name}`, async ({ page }) => {
await page.setContent(`<div data-testid="reply">${body}</div>`);
await expect(assertNativeBlockerReply(page.getByTestId("reply"), "BLOCKED_probe", 100)).rejects.toThrow();
});
}
tests/runner-e2e/native-blocker-visible.ts
+18
View File
@@ -0,0 +1,18 @@
import { expect, type Locator } from "@playwright/test";
/** Blocker replies include an explanation; ordinary marker-only replies do not. */
export async function assertNativeBlockerReply(agentReplies: Locator, marker: string, timeout = 30_000): Promise<void> {
const reply = agentReplies.filter({ hasText: marker });
await expect(reply).toHaveCount(1, { timeout });
await expect(reply).toBeVisible({ timeout });
await expect(reply).toContainText("Release Owner", { timeout });
await expect(reply).toContainText("Grant deployment access", { timeout });
await expect(reply).not.toContainText(
/\b(?:not blocked|no longer blocked|access (?:is |has been |was )?already granted|completed Grant deployment access)\b/i,
{ timeout },
);
await expect(reply).toContainText(
/\b(?:blocked|cannot proceed|can't proceed|missing|required access|not (?:yet )?granted|awaiting|waiting|unavailable)\b/i,
{ timeout },
);
}
tests/runner-e2e/native-bootstrap-read-proof.test.ts
+3 -5
View File
@@ -81,12 +81,10 @@ it("accepts omitted terminal path/kind only with the retained single-path origin
}
});
it.each(["cursor", "copilot"] as const)("correlates actual %s passive notices through the public reader and canonical omitted-field updates", async provider => {
it.each(["cursor"] as const)("correlates actual %s passive notices through the public reader and canonical omitted-field updates", async provider => {
const { createCursorToolEvidence } = await import("../../packages/paperclip-runner/src/drivers/acpx/cursor-tool-evidence.js");
const { createCopilotToolEvidence } = await import("../../packages/paperclip-runner/src/drivers/acpx/copilot-tool-evidence.js");
const { readCursorToolEvidence } = await import("./cursor-native-evidence.js");
const { readCopilotToolEvidence } = await import("./copilot-evidence.js");
const create = provider === "cursor" ? createCursorToolEvidence : createCopilotToolEvidence;
const create = createCursorToolEvidence;
for (const variant of ["exact", "wrong-path", "multi-path", "ambiguous-update"]) {
const rows: any[] = [];
const projector = create({ sessionId: "session", turnId: "turn", workingDirectory: "/workspace", active: () => true,
@@ -98,7 +96,7 @@ it.each(["cursor", "copilot"] as const)("correlates actual %s passive notices th
projector.tool({ type: "tool_call", tag: "tool_call", toolCallId: "bootstrap", kind: "read", status: "pending", rawInput: variant === "multi-path" ? { paths: [path, "private.txt"] } : { path }, locations: [{ path }] });
projector.tool({ type: "tool_call", tag: "tool_call_update", toolCallId: "bootstrap", status: "failed", ...(variant === "ambiguous-update" ? { rawInput: { path, paths: [path, "private.txt"] } } : {}) });
const evaluate = () => {
const readNotices = provider === "cursor" ? readCursorToolEvidence(rows, "run") : readCopilotToolEvidence(rows, "run");
const readNotices = readCursorToolEvidence(rows, "run");
const started = receipt("started", "running", 2), end = receipt("completed", "failed", 4);
Object.assign(end.payload.prpEvent.payload, { target: null, operation: "unknown" });
return withoutProvenBootstrapReads([...readNotices, origin], origin, { actionFile, events: [...rows, started, end] });
tests/runner-e2e/native-completion-admission.test.ts
+108
View File
@@ -0,0 +1,108 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { spawnSync } from "node:child_process";
import { assertNativeCompletionSelection, prepareNativeCompletionPreflight, nativeCompletionPreflightEnvironment, verifyNativeCompletionPreflight } from "./native-completion-admission.js";
import { CREDENTIAL_NAMES, type MatrixExecution } from "./types.js";
vi.mock("node:child_process", () => ({ spawnSync: vi.fn() }));
const spawn = vi.mocked(spawnSync);
const campaignDirectory = "/fixture/results/gha-1-1-native-completion.fixture";
beforeEach(() => { vi.resetAllMocks(); vi.unstubAllEnvs(); });
describe("native completion credential-free admission", () => {
it("allows toolchain paths and excludes every present or future credential", () => {
const source = { PATH: "/bin", HOME: "/home/fixture", CARGO_HOME: "/cargo", CI: "true",
GH_TOKEN: "secret", FUTURE_PROVIDER_API_KEY: "secret", NODE_OPTIONS: "ambient override",
...Object.fromEntries(CREDENTIAL_NAMES.map(name => [name, "secret"])) };
expect(nativeCompletionPreflightEnvironment(source)).toEqual({ PATH: "/bin", HOME: "/home/fixture", CARGO_HOME: "/cargo", CI: "true" });
});
it("carries only public hosted identity through actual prepare and verify subprocess options", () => {
const identity = { GITHUB_ACTIONS: "true", PAPERCLIP_RUNNER_E2E_SOURCE_SHA: "a".repeat(40), GITHUB_RUN_ID: "123", GITHUB_RUN_ATTEMPT: "1" };
for (const [name, value] of Object.entries(identity)) vi.stubEnv(name, value);
for (const name of [...CREDENTIAL_NAMES, "GH_TOKEN", "FUTURE_PROVIDER_API_KEY", "NODE_OPTIONS"]) vi.stubEnv(name, "secret");
spawn.mockReturnValueOnce({ status: 0 } as ReturnType<typeof spawnSync>);
spawn.mockReturnValueOnce({ status: 0, stdout: JSON.stringify({ passed: true, sourceSha: identity.PAPERCLIP_RUNNER_E2E_SOURCE_SHA }) } as ReturnType<typeof spawnSync>);
prepareNativeCompletionPreflight(campaignDirectory);
expect(spawn).toHaveBeenCalledTimes(2);
for (const call of spawn.mock.calls) {
expect(call[2]?.env).toMatchObject(identity);
for (const name of [...CREDENTIAL_NAMES, "GH_TOKEN", "FUTURE_PROVIDER_API_KEY", "NODE_OPTIONS"]) expect(call[2]?.env).not.toHaveProperty(name);
}
expect(spawn.mock.calls[0]?.[1]?.some(arg => arg.startsWith("--output-dir="))).toBe(true);
expect(spawn.mock.calls[1]?.[1]?.some(arg => arg.startsWith("--verify="))).toBe(true);
});
it("rejects missing receipts without invoking a process", () => {
expect(() => verifyNativeCompletionPreflight(undefined)).toThrow("no prerequisite receipt");
expect(spawn).not.toHaveBeenCalled();
});
it("fails before provider execution when the prerequisite subprocess fails", () => {
spawn.mockReturnValue({ status: 1 } as ReturnType<typeof spawnSync>);
expect(() => prepareNativeCompletionPreflight(campaignDirectory)).toThrow("before provider execution");
expect(spawn).toHaveBeenCalledTimes(1);
});
it("retains credential-free prerequisites inside the exact campaign root and verifies them", () => {
const sourceSha = "c".repeat(40);
vi.stubEnv("PAPERCLIP_RUNNER_E2E_SOURCE_SHA", sourceSha);
vi.stubEnv("OPENAI_API_KEY", "secret"); vi.stubEnv("UNLISTED_PROVIDER_CREDENTIAL", "secret");
spawn.mockReturnValueOnce({ status: 0 } as ReturnType<typeof spawnSync>);
spawn.mockReturnValueOnce({ status: 0, stdout: JSON.stringify({ passed: true, sourceSha }) } as ReturnType<typeof spawnSync>);
const receipt = prepareNativeCompletionPreflight(campaignDirectory);
expect(receipt).toMatch(/^\/fixture\/results\/gha-1-1-native-completion\.fixture\/native-completion-prerequisites\/[^/]+\/preflight.json$/);
expect(spawn.mock.calls[1]?.[1]).toContain(`--verify=${receipt}`);
for (const call of spawn.mock.calls) {
expect(call[2]?.env).not.toHaveProperty("OPENAI_API_KEY");
expect(call[2]?.env).not.toHaveProperty("UNLISTED_PROVIDER_CREDENTIAL");
}
});
it("rejects a workflow source override that differs from the exact receipt SHA", () => {
vi.stubEnv("PAPERCLIP_RUNNER_E2E_SOURCE_SHA", "a".repeat(40));
spawn.mockReturnValue({ status: 0, stdout: JSON.stringify({ passed: true, sourceSha: "b".repeat(40) }) } as ReturnType<typeof spawnSync>);
expect(() => verifyNativeCompletionPreflight("/fixture/preflight.json")).toThrow("source SHA differs");
spawn.mockReturnValue({ status: 0, stdout: JSON.stringify({ passed: true, sourceSha: "a".repeat(40) }) } as ReturnType<typeof spawnSync>);
expect(verifyNativeCompletionPreflight("/fixture/preflight.json").sourceSha).toBe("a".repeat(40));
});
it("refuses failed or unreadable receipt verification", () => {
spawn.mockReturnValue({ status: 1, stderr: "stale source" } as ReturnType<typeof spawnSync>);
expect(() => verifyNativeCompletionPreflight("/fixture/preflight.json")).toThrow("stale source");
spawn.mockReturnValue({ status: 0, stdout: "malformed" } as ReturnType<typeof spawnSync>);
expect(() => verifyNativeCompletionPreflight("/fixture/preflight.json")).toThrow();
});
});
function selectedFixture() {
const tasks = ["assigned-skill-explicit-invocation", "native-blocked-report"].map(id =>
({ id, expectedRunCount: 1, automaticRetryPolicy: "single_attempt" }));
return { id: "native-completion.runner-codex.local.assigned-skill-explicit-invocation",
profile: { id: "runner-codex", generation: "native" }, environment: { id: "local" }, task: tasks[0],
suite: { id: "native-completion", manualOnly: true, expectedMatrixSize: 6, tasks } };
}
describe("native completion selected-fixture policy", () => {
it("admits an exact subset of the six declared one-turn single-attempt cells", () => {
const base = selectedFixture();
const executions = ["runner-codex", "runner-acpx-claude", "runner-opencode"].flatMap(id => base.suite.tasks.map(task =>
({ ...structuredClone(base), profile: { id, generation: "native" }, task,
id: `native-completion.${id}.local.${task.id}` })));
expect(() => assertNativeCompletionSelection(executions as unknown as MatrixExecution[])).not.toThrow();
expect(() => assertNativeCompletionSelection([executions[0]!] as unknown as MatrixExecution[])).not.toThrow();
});
const invalid: Array<[string, (fixture: ReturnType<typeof selectedFixture>) => void]> = [
["a legacy generation", f => { f.profile.generation = "legacy"; }],
["an unlisted profile", f => { f.profile.id = "runner-other"; }],
["a remote environment", f => { f.environment.id = "daytona"; }],
["an unexpected case", f => { f.task!.id = "other-task"; }],
["an extra expected turn", f => { f.task!.expectedRunCount = 2; }],
["a missing single-attempt policy", f => { f.task!.automaticRetryPolicy = "default"; }],
["a forged execution ID", f => { f.id = "other-cell"; }],
["an automatic suite", f => { f.suite.manualOnly = false; }],
["an expanded matrix", f => { f.suite.expectedMatrixSize = 7; }],
["a missing original completion case", f => { f.suite.tasks.shift(); }],
["an extra task", f => { f.suite.tasks.push({ ...f.suite.tasks[1]!, id: "extra-task" }); }],
];
for (const [label, mutate] of invalid) it(`rejects ${label} before credentials`, () => {
const f = selectedFixture(); mutate(f);
expect(() => assertNativeCompletionSelection([f] as unknown as MatrixExecution[])).toThrow("before provider execution");
});
it("leaves ordinary suites and their retry policies unchanged", () => {
const f = selectedFixture(); f.suite.id = "ordinary-suite"; f.task!.automaticRetryPolicy = "default";
expect(() => assertNativeCompletionSelection([f] as unknown as MatrixExecution[])).not.toThrow();
});
});
tests/runner-e2e/native-completion-admission.ts
+60
View File
@@ -0,0 +1,60 @@
import { spawnSync } from "node:child_process";
import { randomUUID } from "node:crypto";
import path from "node:path";
import type { MatrixExecution } from "./types.js";
const root = path.resolve(import.meta.dirname, "../..");
export const NATIVE_COMPLETION_PREFLIGHT_ENV = "PAPERCLIP_RUNNER_E2E_NATIVE_COMPLETION_PREFLIGHT";
/** Explicit fixture policy is enforced even when the workflow supplies only --id. */
export function assertNativeCompletionSelection(executions: readonly MatrixExecution[]): void {
const profiles = ["runner-codex", "runner-acpx-claude", "runner-opencode"];
const tasks = ["assigned-skill-explicit-invocation", "native-blocked-report"];
for (const execution of executions.filter(execution => execution.suite.id === "native-completion")) {
const suite = execution.suite;
if (!profiles.includes(execution.profile.id) || execution.profile.generation !== "native" ||
execution.profile.qualificationCandidate !== undefined || execution.environment.id !== "local" ||
!tasks.includes(execution.task.id) || execution.task.expectedRunCount !== 1 ||
(execution.task.minimumExpectedRunCount !== undefined && execution.task.minimumExpectedRunCount !== 1) ||
execution.task.automaticRetryPolicy !== "single_attempt" ||
execution.id !== `native-completion.${execution.profile.id}.local.${execution.task.id}` ||
suite.manualOnly !== true || suite.expectedMatrixSize !== 6 || suite.tasks.length !== 2 ||
!tasks.every(id => suite.tasks.filter(task => task.id === id && task.expectedRunCount === 1 &&
task.automaticRetryPolicy === "single_attempt").length === 1)) {
throw new Error("Native completion requires the declared local native fixtures, one turn and one attempt before provider execution.");
}
}
}
/** Future credentials and ambient auth overrides stay outside prerequisite children. */
export function nativeCompletionPreflightEnvironment(source: NodeJS.ProcessEnv): NodeJS.ProcessEnv {
return Object.fromEntries([
"PATH", "HOME", "TMPDIR", "TMP", "TEMP", "SYSTEMROOT", "LANG", "LC_ALL",
"CARGO_HOME", "RUSTUP_HOME", "CI", "GITHUB_ACTIONS",
"PAPERCLIP_RUNNER_E2E_SOURCE_SHA", "GITHUB_RUN_ID", "GITHUB_RUN_ATTEMPT",
].flatMap(name => source[name] === undefined ? [] : [[name, source[name]]]));
}
export function verifyNativeCompletionPreflight(receiptPath: string | undefined) {
if (!receiptPath) throw new Error("Native completion has no prerequisite receipt; use the live launcher.");
const run = spawnSync(process.execPath, [path.join(root, "tests/runner-e2e/native-completion-checks.mjs"), `--verify=${receiptPath}`], {
cwd: root, env: nativeCompletionPreflightEnvironment(process.env), encoding: "utf8", timeout: 90_000,
});
if (run.status !== 0) throw new Error(`Native completion prerequisite verification failed: ${run.stderr || run.error?.message || run.status}`);
const receipt = JSON.parse(run.stdout) as Record<string, unknown>;
const expectedSourceSha = process.env.PAPERCLIP_RUNNER_E2E_SOURCE_SHA?.trim();
if (expectedSourceSha && receipt.sourceSha !== expectedSourceSha)
throw new Error("Native completion prerequisite source SHA differs from the requested immutable source.");
return receipt;
}
export function prepareNativeCompletionPreflight(campaignDirectory: string) {
const output = path.join(campaignDirectory, "native-completion-prerequisites", randomUUID());
const receipt = path.join(output, "preflight.json");
const run = spawnSync(process.execPath, [path.join(root, "tests/runner-e2e/native-completion-checks.mjs"), `--output-dir=${output}`], {
cwd: root, env: nativeCompletionPreflightEnvironment(process.env), stdio: "inherit", timeout: 35 * 60_000,
});
if (run.status !== 0) throw new Error(`Native completion prerequisites failed before provider execution. Retained receipt: ${receipt}`);
verifyNativeCompletionPreflight(receipt);
return receipt;
}
tests/runner-e2e/native-completion-cases.ts
+47
View File
@@ -0,0 +1,47 @@
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import type { RunnerTaskFixture } from "./types.js";
import { contextIntegrityTasks } from "./context-integrity-cases.js";
export const nativeBlockedReportTask: RunnerTaskFixture = {
id: "native-blocked-report", label: "Native concrete blocker report", groups: ["native-completion"],
workMode: "standard", flow: "single_turn", expectedRunCount: 1, automaticRetryPolicy: "single_attempt",
attemptTimeoutMs: { local: 5 * 60_000, daytona: 5 * 60_000 },
expectedTerminalState: { issue: "blocked", run: "succeeded" },
buildTitle: nonce => `Native blocker report ${nonce}`,
buildVisibleMarker: nonce => `BLOCKED_${nonce.replace(/[^a-z0-9]/gi, "")}`,
buildPrompt: nonce => [
"Deployment cannot proceed because required release access has not been granted.",
"Release Owner is the external owner of that grant. Do not attempt deployment, run shell/process commands, change files, invoke external services, create documents, agents, interactions or subtasks, or claim the access exists.",
'Record the whole task as blocked and identify the owner as Release Owner. The exact unblock action is "Grant deployment access" (without quotation marks or trailing punctuation).',
`Include BLOCKED_${nonce.replace(/[^a-z0-9]/gi, "")} exactly once in your final explanation.`,
].join("\n"),
buildMatchers: nonce => [
{ kind: "issue_status", expected: "blocked" }, { kind: "run_status", expected: "succeeded" },
{ kind: "runtime_mode", expected: "native" },
{ kind: "message_contains", expected: `BLOCKED_${nonce.replace(/[^a-z0-9]/gi, "")}` },
{ kind: "message_contains", expected: "Release Owner" },
{ kind: "message_contains", expected: "Grant deployment access" },
{ kind: "message_regex", pattern: "^(?![\\s\\S]*\\b(?:not blocked|no longer blocked|access (?:is |has been |was )?already granted|completed Grant deployment access)\\b)[\\s\\S]*\\b(?:blocked|cannot proceed|can't proceed|missing|required access|not (?:yet )?granted|awaiting|waiting|unavailable)\\b", flags: "i" },
{ kind: "json_path", path: "run.resultJson.nativeResult.reportedWorkDisposition", expected: "blocked" },
{ kind: "json_path", path: "run.resultJson.nativeResult.blocker.owner.name", expected: "Release Owner" },
{ kind: "json_path", path: "run.resultJson.nativeResult.blocker.unblockAction", expected: "Grant deployment access" },
{ kind: "json_path", path: "run.resultJson.nativeResult.blocker.scope", expected: "task_wide" },
],
};
const assignedSkill = contextIntegrityTasks.find(task => task.id === "assigned-skill-explicit-invocation");
if (!assignedSkill) throw new Error("Missing original assigned-skill completion journey");
// Preserve the original task prompt, skill, durable-document oracle and deadline.
export const nativeCompletionTasks: readonly RunnerTaskFixture[] = [
{ ...assignedSkill, automaticRetryPolicy: "single_attempt" }, nativeBlockedReportTask,
];
export function nativeCompletionDefinitionDigest() {
const hash = createHash("sha256");
for (const file of ["native-completion-cases.ts", "native-completion-scoring.ts", "native-completion-content.ts", "native-completion-defaults.ts",
"native-blocker-visible.ts", "native-completion-admission.ts", "native-completion-checks.mjs",
"native-completion-source-contract.mjs", "automatic-retry.ts", "context-integrity-cases.ts", "context-integrity-flow.ts", "context-integrity-scoring.ts", "types.ts", "catalog.ts", "live-fixtures.ts", "runner.spec.ts", "launch.ts"]) {
hash.update(file); hash.update(readFileSync(new URL(file, import.meta.url)));
}
return hash.digest("hex");
}
tests/runner-e2e/native-completion-checks.mjs
+297
View File
@@ -0,0 +1,297 @@
// Independent, credential-free admission for the native completion comparison.
import { spawnSync } from "node:child_process";
import { createHash } from "node:crypto";
import { existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from "node:fs";
import { join, resolve } from "node:path";
import {
NATIVE_COMPLETION_SOURCE_CONTRACT, NATIVE_COMPLETION_SOURCE_FILES,
nativeCompletionSourceFingerprint, nativeSourceSha256,
} from "./native-completion-source-contract.mjs";
import { inspectNativeCompletionSourceMetadata, inspectNativeCompletionRunnerd, NATIVE_COMPLETION_RUNNERD_PATH } from "./native-completion-git-source.mjs";
const root = resolve(import.meta.dirname, "../..");
export const NATIVE_COMPLETION_PREFLIGHT_SCHEMA = "paperclip.native-completion-preflight.v1";
export const NATIVE_COMPLETION_CELL_IDS = ["runner-codex", "runner-acpx-claude", "runner-opencode"].flatMap(profile =>
["assigned-skill-explicit-invocation", "native-blocked-report"].map(task => `native-completion.${profile}.local.${task}`));
export function nativeCompletionGates(variant) {
if (!["candidate", "historical"].includes(variant)) throw new Error("Unknown native completion source variant");
return [
{ id: "NC-schemas", name: "Variant-native completion schemas", cwd: "packages/paperclip-runner",
files: ["src/contracts/completion-result.test.ts"],
required: ["distinguishes user-facing answer content", "propagates answer and wait descriptions", "requires a reason code for verification that was not run"] },
{ id: "NC-tools", name: "Native tool delivery and retained provider catalogs", cwd: "packages/paperclip-runner", files: [
"src/drivers/codex/codex-app-server-driver.lifecycle.test.ts", "src/drivers/runner-tool-bridge.test.ts",
"src/drivers/opencode/mcp-bridge.test.ts", "src/live/runnerd-codex-transport.test.ts"],
testPattern: "binds to loopback|preserves stock Codex instructions on|includes ACPX terminal tools|preserves answer and internal wait descriptions",
required: ["runner semantic MCP bridge binds to loopback", "OpenCode MCP bridge binds to loopback",
"preserves stock Codex instructions on task recovery", "preserves stock Codex instructions on prepared recovery",
"preserves stock Codex instructions on direct recovery", "includes ACPX terminal tools in the authenticated bridge catalog",
...(variant === "candidate" ? ["codex", "opencode", "claude_managed", "aws_agentcore", "acpx"].map(provider =>
`preserves answer and internal wait descriptions in the serialized native ${provider} tool catalog`) : [])] },
{ id: "NC-resume", name: "Variant-native fingerprint and checkpoint refresh", cwd: ".",
files: ["server/src/services/native-runtime/native-session-resume.test.ts"], testPattern: "refreshes retained",
required: ["refreshes retained 'completion descriptions'", "refreshes retained 'task-bound human-input description'",
...(variant === "candidate" ? ["native completion tool guidance"] : [])] },
{ id: "NC-eval", name: "Independent native oracle, defaults, admission and attempt policy", cwd: ".",
config: "tests/runner-e2e/vitest.config.ts", files: [
"tests/runner-e2e/native-completion-scoring.test.ts", "tests/runner-e2e/native-completion-defaults.test.ts",
"tests/runner-e2e/native-completion-admission.test.ts", "tests/runner-e2e/automatic-retry.test.ts",
"tests/runner-e2e/context-integrity.test.ts", "tests/runner-e2e/context-integrity-evidence.test.ts",
"tests/runner-e2e/context-integrity-flow-logs.test.ts"],
required: ["allows toolchain paths and excludes every present or future credential",
"retains credential-free prerequisites inside the exact campaign root",
"retains the first transient_infrastructure failure without a second attempt",
"retains the first provider_variance failure without a second attempt"] },
];
}
export const NATIVE_COMPLETION_COMMAND_GATE_IDS = ["NC-node", "NC-typecheck", "NC-manifest", "NC-discovery", "NC-rust-carrier"];
export function nativeCompletionCommandGateIds(mode) {
return NATIVE_COMPLETION_COMMAND_GATE_IDS.map(id => mode === "trusted_hosted_archive" && id === "NC-rust-carrier" ? "NC-hosted-runnerd" : id);
}
export function gradeNativeCompletionGate(gate, report, exitCode) {
const assertions = (report?.testResults ?? []).flatMap(file => file.assertionResults ?? []);
const requirements = gate.required.map(name => ({ name, passed: assertions.some(assertion =>
assertion.fullName?.includes(name) && assertion.status === "passed") }));
const files = gate.files.map(file => ({ file, passed: (report?.testResults ?? []).some(result =>
result.name?.endsWith(file) && result.status === "passed" && result.assertionResults?.some(assertion => assertion.status === "passed") &&
result.assertionResults.every(assertion => assertion.status === "passed" || (gate.testPattern && assertion.status === "skipped" &&
!gate.required.some(name => assertion.fullName?.includes(name))))) }));
return { id: gate.id, name: gate.name, passed: exitCode === 0 && requirements.every(row => row.passed) && files.every(row => row.passed),
exitCode, files, requirements, total: report?.numTotalTests ?? 0, passedTests: report?.numPassedTests ?? 0,
failedTests: report?.numFailedTests ?? 0, pendingTests: report?.numPendingTests ?? 0 };
}
export function gradeNativeCompletionDiscovery(output, exitCode) {
const lines = output.trim().split(/\r?\n/).filter(Boolean);
const rows = lines.slice(1).map(line => line.split("\t"));
const ids = rows.map(row => row[0]);
return { passed: exitCode === 0 && lines[0] === "ID\tSUITE\tGENERATION\tPROVIDER\tMODEL\tCREDENTIALS" &&
rows.length === 6 && new Set(ids).size === 6 && NATIVE_COMPLETION_CELL_IDS.every(id => ids.includes(id)) &&
rows.every(row => row.length === 6 && row[1] === "native-completion" && row[2] === "native" && row[4]?.trim() && row[5]?.trim()),
executionIds: ids, expectedCells: 6, expectedTurns: 6, maximumAttemptsPerCell: 1 };
}
/** capture() stores compact provenance JSON on its first line, then diagnostics. */
export function gradeNativeCompletionHostedRunnerdEvidence(text, expected) {
try { return JSON.stringify(JSON.parse(text.split(/\r?\n/, 1)[0])) === JSON.stringify(expected); }
catch { return false; }
}
export function gradeNativeCompletionRustCarrier(output, exitCode) {
return exitCode === 0 && /^test provider_events::tests::preserves_closed_compatibility_terminal_tool_identity \.\.\. ok$/m.test(output) &&
/test result: ok\. 1 passed; 0 failed;/.test(output);
}
export function nativeCompletionPrerequisiteEnvironment(source) {
return Object.fromEntries(["PATH", "HOME", "TMPDIR", "TMP", "TEMP", "SYSTEMROOT", "LANG", "LC_ALL",
"CARGO_HOME", "RUSTUP_HOME", "CI", "GITHUB_ACTIONS", "GITHUB_RUN_ID", "GITHUB_RUN_ATTEMPT", "PAPERCLIP_RUNNER_E2E_SOURCE_SHA"].flatMap(name => source[name] === undefined ? [] : [[name, source[name]]]));
}
function currentSource() {
const source = nativeCompletionSourceFingerprint();
return { ...source, ...inspectNativeCompletionSourceMetadata({ repositoryRoot: root,
sourceFiles: NATIVE_COMPLETION_SOURCE_FILES, baseSha: source.baseSha, variant: source.variant,
shallowParentAnchors: NATIVE_COMPLETION_SOURCE_CONTRACT.shallowParentAnchors,
}) };
}
function buildOutputFingerprint() {
const hash = createHash("sha256");
function visit(relative) {
for (const entry of readdirSync(join(root, relative), { withFileTypes: true }).sort((a, b) => a.name.localeCompare(b.name))) {
const file = join(relative, entry.name);
if (entry.isDirectory()) visit(file);
else if (entry.isFile()) { const bytes = readFileSync(join(root, file)); hash.update(JSON.stringify([file, bytes.length])).update(bytes); }
else throw new Error(`Unexpected build output: ${file}`);
}
}
for (const directory of ["packages/shared/dist", "packages/plugins/sdk/dist", "packages/paperclip-runner/dist"]) {
if (!existsSync(join(root, directory, "index.js"))) throw new Error(`Missing prerequisite build output: ${directory}`);
visit(directory);
}
const daemon = runnerdBinary();
const bytes = readFileSync(daemon);
hash.update(JSON.stringify(["debug/paperclip-runnerd", bytes.length])).update(bytes);
return hash.digest("hex");
}
function runnerdBinary() {
return join(root, `${NATIVE_COMPLETION_RUNNERD_PATH}${process.platform === "win32" ? ".exe" : ""}`);
}
function runnerdProvenance(source, env) {
return inspectNativeCompletionRunnerd({ repositoryRoot: root, sourceSha: source.sha,
sourceFingerprint: source.fingerprint, environment: env });
}
export function assertNativeCompletionPreflightReceipt(report, current) {
const expected = [...nativeCompletionGates(current.variant).map(gate => gate.id), ...nativeCompletionCommandGateIds(current.runnerdProvenance?.mode)];
const hostedGate = report?.gates?.find(gate => gate.id === "NC-hosted-runnerd");
const truthfulHosted = current.runnerdProvenance?.mode !== "trusted_hosted_archive" ||
hostedGate?.executed === false && hostedGate?.calibration === "not_executed" && hostedGate?.total === 0 &&
hostedGate?.passedTests === 0 && hostedGate?.reuse === "trusted_same_run_build";
if (!truthfulHosted || report?.schema !== NATIVE_COMPLETION_PREFLIGHT_SCHEMA || report.passed !== true || report.providerCalls !== 0 ||
report.live !== "not_run" || report.sourceSha !== current.sha || !/^[a-f0-9]{40}$/.test(current.sha ?? "") ||
![current.fingerprint, current.fixtureFingerprint, current.manifestFingerprint, current.buildOutputFingerprint]
.every(value => typeof value === "string" && /^[a-f0-9]{64}$/.test(value)) ||
report.sourceFingerprint !== current.fingerprint || report.fixtureFingerprint !== current.fixtureFingerprint ||
report.manifestFingerprint !== current.manifestFingerprint || report.variant !== current.variant ||
report.baseSha !== NATIVE_COMPLETION_SOURCE_CONTRACT.baseSha || report.archiveSha !== NATIVE_COMPLETION_SOURCE_CONTRACT.archiveSha ||
report.layering !== true || current.layering !== true || report.immutable !== true || current.immutable !== true ||
!/^[a-f0-9]{64}$/.test(current.sourceMetadataFingerprint ?? "") ||
report.sourceMetadataFingerprint !== current.sourceMetadataFingerprint ||
!Array.isArray(report.sourceMetadataErrors) || report.sourceMetadataErrors.length !== 0 || current.sourceMetadataErrors?.length !== 0 ||
!Array.isArray(report.sourceErrors) || report.sourceErrors.length !== 0 || current.sourceErrors?.length !== 0 ||
report.setup?.passed !== true || report.setup.sdkExitCode !== 0 || report.setup.runnerTypeScriptExitCode !== 0 ||
report.setup.runnerdExitCode !== (current.runnerdProvenance?.mode === "trusted_hosted_archive" ? null : 0) ||
current.runnerdProvenance?.passed !== true || !["trusted_hosted_archive", "fresh_local_build"].includes(current.runnerdProvenance?.mode) ||
JSON.stringify(report.setup.runnerdProvenance) !== JSON.stringify(current.runnerdProvenance) ||
report.setup.runnerdSelectedPath !== current.runnerdProvenance.selectedPath ||
report.setup.runnerdSha256 !== current.runnerdProvenance.binarySha256 ||
!/^[a-f0-9]{64}$/.test(current.runnerdProvenance.binarySha256 ?? "") ||
report.setup.runnerdSourceSha !== current.sha || report.setup.runnerdSourceFingerprint !== current.fingerprint ||
report.setup.buildOutputFingerprint !== current.buildOutputFingerprint ||
!Array.isArray(report.setup.evidence) || report.setup.evidence.length !== 3 ||
["setup-sdk.txt", "setup-runner-typescript.txt", "setup-runnerd.txt"].some(file => report.setup.evidence.filter(row =>
row.file === file && /^[a-f0-9]{64}$/.test(row.sha256 ?? "")).length !== 1) ||
!Array.isArray(report.gates) || report.gates.length !== expected.length ||
expected.some(id => report.gates.filter(gate => gate.id === id && gate.passed === true && gate.exitCode === 0).length !== 1) ||
report.expectedCells !== 6 || report.expectedTurns !== 6 || report.maximumAttemptsPerCell !== 1) {
throw new Error("Native completion requires passing prerequisites for this exact immutable source SHA, variant and fingerprint.");
}
return report;
}
function runCommand(command, args, env, timeout = 10 * 60_000, cwd = root) {
return spawnSync(command, args, { cwd, env, encoding: "utf8", timeout });
}
function capture(output, name, run) {
const file = `${name}.txt`, bytes = `${run?.stdout ?? ""}\n${run?.stderr ?? ""}`;
writeFileSync(join(output, file), bytes);
return { file, sha256: nativeSourceSha256(bytes) };
}
export function assertRetainedNativeCompletionEvidence(output, evidence) {
if (!evidence || typeof evidence.file !== "string" || !/^[A-Za-z0-9_.-]+$/.test(evidence.file) ||
nativeSourceSha256(readFileSync(join(output, evidence.file))) !== evidence.sha256)
throw new Error("Native completion retained prerequisite evidence changed or is unavailable.");
return readFileSync(join(output, evidence.file), "utf8");
}
function manifestCommands(env) {
return ["generate-capability-contract.mjs", "check-capability-inventory.mjs"].map(file => runCommand(process.execPath,
[join(root, "packages/paperclip-runner/scripts", file), ...(file.startsWith("generate-") ? ["--check"] : [])], env));
}
export function main(args = process.argv.slice(2)) {
if (args.includes("--list")) {
console.log(JSON.stringify({ variants: ["candidate", "historical"], gates: nativeCompletionGates("candidate"),
commands: { local: nativeCompletionCommandGateIds("fresh_local_build"), hosted: nativeCompletionCommandGateIds("trusted_hosted_archive") },
cells: NATIVE_COMPLETION_CELL_IDS, providerCalls: 0 }, null, 2)); return;
}
if (args.some(arg => !arg.startsWith("--output-dir=") && !arg.startsWith("--verify=")))
throw new Error("Use --list, --output-dir=<path> or --verify=<receipt>; never paid providers.");
const source = currentSource(), env = nativeCompletionPrerequisiteEnvironment(process.env);
const verify = args.find(arg => arg.startsWith("--verify="))?.slice("--verify=".length);
if (verify) {
const report = assertNativeCompletionPreflightReceipt(JSON.parse(readFileSync(verify, "utf8")), {
...source, buildOutputFingerprint: buildOutputFingerprint(), runnerdProvenance: runnerdProvenance(source, env),
});
const output = resolve(verify, "..");
for (const evidence of report.setup.evidence) assertRetainedNativeCompletionEvidence(output, evidence);
for (const gate of nativeCompletionGates(source.variant)) {
const retained = report.gates.find(row => row.id === gate.id);
const grade = gradeNativeCompletionGate(gate, JSON.parse(assertRetainedNativeCompletionEvidence(output, retained.evidence)), 0);
if (!grade.passed) throw new Error(`Missing or failed retained native prerequisite assertions: ${gate.id}`);
}
for (const id of nativeCompletionCommandGateIds(report.setup.runnerdProvenance.mode)) {
const retained = report.gates.find(row => row.id === id), text = assertRetainedNativeCompletionEvidence(output, retained.evidence);
if (id === "NC-node" && (!/# fail 0\b/.test(text) || !/# tests [1-9]\d*\b/.test(text)))
throw new Error("Missing passing native admission calibrations.");
if (id === "NC-discovery" && !gradeNativeCompletionDiscovery(text.split("\n\n")[0], 0).passed)
throw new Error("Native completion six-cell discovery changed.");
if (id === "NC-rust-carrier" && !gradeNativeCompletionRustCarrier(text, 0))
throw new Error("Missing passing retained Rust terminal-tool carrier calibration.");
if (id === "NC-hosted-runnerd" && (retained.executed !== false || retained.calibration !== "not_executed" ||
retained.total !== 0 || retained.passedTests !== 0 ||
!gradeNativeCompletionHostedRunnerdEvidence(text, report.setup.runnerdProvenance)))
throw new Error("Hosted runnerd reuse must retain exact binary proof and report Rust calibration not executed.");
if (id === "NC-manifest" && manifestCommands(env).some(run => run.status !== 0))
throw new Error("Generated native capability manifests are stale.");
}
console.log(JSON.stringify(report)); return report;
}
const output = resolve(args.find(arg => arg.startsWith("--output-dir="))?.slice("--output-dir=".length) ??
join(root, "tests/runner-e2e/results", `native-completion-preflight-${new Date().toISOString().replaceAll(":", "-")}`));
mkdirSync(output, { recursive: true });
const report = { schema: NATIVE_COMPLETION_PREFLIGHT_SCHEMA, sourceSha: source.sha, sourceFingerprint: source.fingerprint,
fixtureFingerprint: source.fixtureFingerprint, manifestFingerprint: source.manifestFingerprint,
variant: source.variant, baseSha: source.baseSha, archiveSha: source.archiveSha,
sourceErrors: source.sourceErrors, immutable: source.immutable, layering: source.layering,
sourceMetadata: source.sourceMetadata, sourceMetadataErrors: source.sourceMetadataErrors,
sourceMetadataFingerprint: source.sourceMetadataFingerprint,
measuredAt: new Date().toISOString(), providerCalls: 0, live: "not_run", expectedCells: 6, expectedTurns: 6, maximumAttemptsPerCell: 1,
setup: { passed: false, sdkExitCode: null, runnerTypeScriptExitCode: null, runnerdExitCode: null, evidence: [] }, gates: [], passed: false };
if (!source.sha || source.sourceErrors.length || !source.layering || !source.immutable) {
writeFileSync(join(output, "preflight.json"), JSON.stringify(report, null, 2) + "\n");
console.error(`Native completion source admission failed before providers: ${[...source.sourceErrors, ...source.sourceMetadataErrors].join("; ")}`);
process.exitCode = 1; return report;
}
const sdk = runCommand(process.execPath, [join(root, "scripts/ensure-plugin-build-deps.mjs")], env, 5 * 60_000);
report.setup.sdkExitCode = sdk.status; report.setup.evidence.push(capture(output, "setup-sdk", sdk));
const runner = sdk.status === 0 ? runCommand("pnpm", ["--filter", "@paperclipai/paperclip-runner", "build:typescript"], env) : null;
report.setup.runnerTypeScriptExitCode = runner?.status ?? null;
report.setup.evidence.push(capture(output, "setup-runner-typescript", runner));
const hosted = env.GITHUB_ACTIONS === "true";
const runnerd = sdk.status === 0 && runner?.status === 0 && !hosted ? runCommand("cargo",
["build", "--locked", "--offline", "--workspace", "--bins"], env,
10 * 60_000, join(root, "packages/paperclip-runner/runner")) : null;
report.setup.runnerdExitCode = runnerd?.status ?? null;
report.setup.runnerdProvenance = runnerdProvenance(source, env);
report.setup.evidence.push(capture(output, "setup-runnerd", hosted
? { stdout: JSON.stringify(report.setup.runnerdProvenance), stderr: "Rust build and unit calibration not executed in this hosted cell; reusing trusted same-run build." }
: runnerd));
report.setup.passed = sdk.status === 0 && runner?.status === 0 && (hosted || runnerd?.status === 0) && report.setup.runnerdProvenance.passed;
if (report.setup.passed) {
report.setup.buildOutputFingerprint = buildOutputFingerprint();
report.setup.runnerdSha256 = report.setup.runnerdProvenance.binarySha256;
report.setup.runnerdSelectedPath = report.setup.runnerdProvenance.selectedPath;
report.setup.runnerdSourceSha = source.sha;
report.setup.runnerdSourceFingerprint = source.fingerprint;
for (const gate of nativeCompletionGates(source.variant)) {
console.log(`Checking ${gate.id}: ${gate.name}`);
const file = `${gate.id}.json`, run = runCommand(process.execPath, [join(root, "node_modules/vitest/vitest.mjs"), "run", ...gate.files,
...(gate.config ? ["--config", gate.config] : []), ...(gate.testPattern ? ["--testNamePattern", gate.testPattern] : []),
"--reporter=default", "--reporter=json", `--outputFile.json=${join(output, file)}`], env, 10 * 60_000, resolve(root, gate.cwd));
capture(output, gate.id, run);
let result; try { result = JSON.parse(readFileSync(join(output, file), "utf8")); } catch { /* Missing discovery fails closed. */ }
report.gates.push({ ...gradeNativeCompletionGate(gate, result, run.status),
evidence: existsSync(join(output, file)) ? { file, sha256: nativeSourceSha256(readFileSync(join(output, file))) } : null });
}
const commands = [
{ id: "NC-node", command: process.execPath, args: ["--test", "--test-reporter=tap", "tests/runner-e2e/native-completion-checks.test.mjs", "tests/runner-e2e/native-completion-source-contract.test.mjs", "tests/runner-e2e/native-completion-git-source.test.mjs"] },
{ id: "NC-typecheck", command: process.execPath, args: ["node_modules/typescript/bin/tsc", "-p", "tests/runner-e2e/tsconfig.json"] },
...(!hosted ? [{ id: "NC-rust-carrier", command: "cargo", args: ["test", "--locked", "--offline",
"-p", "paperclip-runner-core", "--lib", "provider_events::tests::preserves_closed_compatibility_terminal_tool_identity", "--", "--exact"],
cwd: join(root, "packages/paperclip-runner/runner") }] : []),
{ id: "NC-discovery", command: process.execPath, args: ["--import", "./server/node_modules/tsx/dist/loader.mjs", "tests/runner-e2e/launch.ts", "--list", "--suite", "native-completion"] },
];
for (const command of commands) {
console.log(`Checking ${command.id}`);
const run = runCommand(command.command, command.args, env, 10 * 60_000, command.cwd ?? root);
report.gates.push({ id: command.id, passed: run.status === 0 &&
(command.id !== "NC-discovery" || gradeNativeCompletionDiscovery(run.stdout, run.status).passed) &&
(command.id !== "NC-rust-carrier" || gradeNativeCompletionRustCarrier(`${run.stdout ?? ""}\n${run.stderr ?? ""}`, run.status)) &&
(command.id !== "NC-node" || /# fail 0\b/.test(run.stdout) && /# tests [1-9]\d*\b/.test(run.stdout)),
exitCode: run.status, evidence: capture(output, command.id, run) });
}
if (hosted) report.gates.push({ id: "NC-hosted-runnerd", name: "Trusted hosted runnerd provenance; Rust calibration not executed",
passed: report.setup.runnerdProvenance.passed, exitCode: 0, executed: false, calibration: "not_executed",
reuse: "trusted_same_run_build", total: 0, passedTests: 0, evidence: capture(output, "NC-hosted-runnerd", {
stdout: JSON.stringify(report.setup.runnerdProvenance), stderr: "Rust unit calibration must be qualified by the separate exact-source local receipt and normal CI." }) });
const manifest = manifestCommands(env), combined = { stdout: manifest.map(run => run.stdout ?? "").join("\n"), stderr: manifest.map(run => run.stderr ?? "").join("\n") };
report.gates.push({ id: "NC-manifest", passed: manifest.every(run => run.status === 0),
exitCode: manifest.find(run => run.status !== 0)?.status ?? 0, evidence: capture(output, "NC-manifest", combined) });
const after = currentSource();
report.passed = report.gates.every(gate => gate.passed) && after.immutable && after.layering &&
after.sha === source.sha && after.fingerprint === source.fingerprint && after.sourceErrors.length === 0 &&
after.sourceMetadataFingerprint === source.sourceMetadataFingerprint && after.sourceMetadataErrors.length === 0 &&
buildOutputFingerprint() === report.setup.buildOutputFingerprint &&
JSON.stringify(runnerdProvenance(after, env)) === JSON.stringify(report.setup.runnerdProvenance);
}
writeFileSync(join(output, "preflight.json"), JSON.stringify(report, null, 2) + "\n");
console.log(`Native completion prerequisite evidence: ${join(output, "preflight.json")}`);
if (!report.passed) process.exitCode = 1;
return report;
}
if (process.argv[1] && resolve(process.argv[1]) === resolve(import.meta.filename)) main();
tests/runner-e2e/native-completion-checks.test.mjs
+182
View File
@@ -0,0 +1,182 @@
import assert from "node:assert/strict";
import test from "node:test";
import { mkdtempSync, writeFileSync, rmSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import {
nativeCompletionGates, gradeNativeCompletionGate, gradeNativeCompletionDiscovery, gradeNativeCompletionRustCarrier, gradeNativeCompletionHostedRunnerdEvidence,
assertNativeCompletionPreflightReceipt, assertRetainedNativeCompletionEvidence, nativeCompletionPrerequisiteEnvironment,
NATIVE_COMPLETION_PREFLIGHT_SCHEMA, NATIVE_COMPLETION_CELL_IDS, NATIVE_COMPLETION_COMMAND_GATE_IDS, nativeCompletionCommandGateIds,
} from "./native-completion-checks.mjs";
import { nativeSourceSha256, NATIVE_COMPLETION_SOURCE_CONTRACT } from "./native-completion-source-contract.mjs";
const gate = { id: "NC-test", name: "Boundary", files: ["boundary.test.ts"], required: ["native assertion"] };
const report = () => ({ testResults: [{ name: "/repo/boundary.test.ts", status: "passed", assertionResults: [
{ fullName: "Boundary native assertion", status: "passed" }] }], numTotalTests: 1, numPassedTests: 1 });
test("native completion prerequisite gates exclude reduced manual and shared legacy assertions", () => {
for (const variant of ["candidate", "historical"]) {
const gates = nativeCompletionGates(variant);
assert.deepEqual(gates.map(gate => gate.id), ["NC-schemas", "NC-tools", "NC-resume", "NC-eval"]);
assert.ok(gates.every(gate => gate.files.length && gate.required.length));
assert.ok(!gates.flatMap(gate => gate.files).some(file => /agent-skills-routes|onboarding-first-task-assets|server-utils\.test|prompt-sections/.test(file)));
}
assert.ok(nativeCompletionGates("candidate")[1].required.some(name => name.includes("serialized native codex")));
assert.ok(!nativeCompletionGates("historical")[1].required.some(name => name.includes("serialized native codex")));
assert.ok(!nativeCompletionGates("historical")[2].required.includes("native completion tool guidance"));
assert.throws(() => nativeCompletionGates("unknown"));
});
test("native completion prerequisite admits an executed passing boundary", () => assert.equal(gradeNativeCompletionGate(gate, report(), 0).passed, true));
for (const status of ["pending", "skipped", "failed", "todo"]) test(`native completion prerequisite rejects a ${status} required assertion`, () => {
const observation = report(); observation.testResults[0].assertionResults[0].status = status;
assert.equal(gradeNativeCompletionGate(gate, observation, 0).passed, false);
});
test("native completion prerequisite rejects missing discovery, unrelated assertions and failed commands", () => {
assert.equal(gradeNativeCompletionGate(gate, undefined, 0).passed, false);
const different = report(); different.testResults[0].assertionResults[0].fullName = "Other";
assert.equal(gradeNativeCompletionGate(gate, different, 0).passed, false);
assert.equal(gradeNativeCompletionGate(gate, report(), 1).passed, false);
assert.equal(gradeNativeCompletionGate({ ...gate, files: [...gate.files, "undiscovered.test.ts"] }, report(), 0).passed, false);
});
test("native completion prerequisite allows filtering unrelated assertions but never the required assertion", () => {
const observation = report(); observation.testResults[0].assertionResults.push({ fullName: "Unrelated", status: "skipped" });
assert.equal(gradeNativeCompletionGate({ ...gate, testPattern: "native assertion" }, observation, 0).passed, true);
assert.equal(gradeNativeCompletionGate(gate, observation, 0).passed, false);
observation.testResults[0].assertionResults[0].status = "skipped";
assert.equal(gradeNativeCompletionGate({ ...gate, testPattern: "native assertion" }, observation, 0).passed, false);
});
const discovery = () => ["ID\tSUITE\tGENERATION\tPROVIDER\tMODEL\tCREDENTIALS", ...NATIVE_COMPLETION_CELL_IDS.map(id =>
`${id}\tnative-completion\tnative\tprovider\tmodel\tAPI_KEY`)].join("\n");
test("native completion prerequisite discovers exactly six native single-turn cells", () => {
assert.deepEqual(gradeNativeCompletionDiscovery(discovery(), 0), { passed: true, executionIds: NATIVE_COMPLETION_CELL_IDS,
expectedCells: 6, expectedTurns: 6, maximumAttemptsPerCell: 1 });
});
for (const kind of ["missing", "duplicate", "legacy", "extra", "failed", "missing-model", "wrong-task"]) test(`native completion prerequisite rejects ${kind} discovery`, () => {
let output = discovery();
if (kind === "missing") output = output.split("\n").slice(0, -1).join("\n");
if (kind === "duplicate") output = output.replace(NATIVE_COMPLETION_CELL_IDS[1], NATIVE_COMPLETION_CELL_IDS[0]);
if (kind === "legacy") output = output.replace("\tnative\t", "\tlegacy\t");
if (kind === "extra") output += `\n${output.split("\n")[1]}`;
if (kind === "missing-model") output = output.replace("\tmodel\t", "\t\t");
if (kind === "wrong-task") output = output.replace("assigned-skill-explicit-invocation", "other-case");
assert.equal(gradeNativeCompletionDiscovery(output, kind === "failed" ? 1 : 0).passed, false);
});
test("native completion prerequisite excludes future credentials and auth overrides from every child", () => {
assert.deepEqual(nativeCompletionPrerequisiteEnvironment({ PATH: "/bin", HOME: "/fixture", CI: "true",
OPENAI_API_KEY: "secret", ANTHROPIC_API_KEY: "secret", FUTURE_TOKEN: "secret", NODE_OPTIONS: "secret", GH_TOKEN: "secret" }),
{ PATH: "/bin", HOME: "/fixture", CI: "true" });
});
const runnerdProof = () => ({ schema: "paperclip.native-completion-runnerd-provenance.v1", mode: "fresh_local_build", passed: true,
selectedPath: "packages/paperclip-runner/runner/target/debug/paperclip-runnerd", binarySha256: "9".repeat(64),
sourceSha: "a".repeat(40), sourceFingerprint: "b".repeat(64), archiveSha256: null, workflowRunId: null,
workflowRunAttempt: null, artifactName: null, errors: [] });
const current = variant => ({ runnerdProvenance: runnerdProof(), sha: "a".repeat(40), fingerprint: "b".repeat(64), fixtureFingerprint: "c".repeat(64),
manifestFingerprint: "d".repeat(64), buildOutputFingerprint: "e".repeat(64), runnerdSha256: "9".repeat(64), variant, immutable: true, layering: true, sourceErrors: [], sourceMetadataFingerprint: "f".repeat(64), sourceMetadataErrors: [] });
const receipt = variant => ({ schema: NATIVE_COMPLETION_PREFLIGHT_SCHEMA, passed: true, providerCalls: 0, live: "not_run",
sourceSha: current(variant).sha, sourceFingerprint: current(variant).fingerprint, fixtureFingerprint: current(variant).fixtureFingerprint,
manifestFingerprint: current(variant).manifestFingerprint, variant,
baseSha: NATIVE_COMPLETION_SOURCE_CONTRACT.baseSha, archiveSha: NATIVE_COMPLETION_SOURCE_CONTRACT.archiveSha, immutable: true, layering: true, sourceErrors: [],
sourceMetadataFingerprint: current(variant).sourceMetadataFingerprint, sourceMetadataErrors: [],
expectedCells: 6, expectedTurns: 6, maximumAttemptsPerCell: 1,
setup: { passed: true, sdkExitCode: 0, runnerTypeScriptExitCode: 0, buildOutputFingerprint: current(variant).buildOutputFingerprint,
runnerdExitCode: 0, runnerdSha256: current(variant).runnerdSha256, runnerdProvenance: runnerdProof(),
runnerdSelectedPath: runnerdProof().selectedPath,
runnerdSourceSha: current(variant).sha, runnerdSourceFingerprint: current(variant).fingerprint,
evidence: ["setup-sdk.txt", "setup-runner-typescript.txt", "setup-runnerd.txt"].map(file => ({ file, sha256: "f".repeat(64) })) },
gates: [...nativeCompletionGates(variant).map(gate => gate.id), ...NATIVE_COMPLETION_COMMAND_GATE_IDS].map(id => ({ id, passed: true, exitCode: 0 })) });
for (const variant of ["candidate", "historical"]) test(`native completion prerequisite admits only the exact immutable ${variant} receipt`, () => {
assert.equal(assertNativeCompletionPreflightReceipt(receipt(variant), current(variant)).passed, true);
});
const negatives = [
["old SHA", r => { r.sourceSha = "f".repeat(40); }], ["changed source", r => { r.sourceFingerprint = "f".repeat(64); }],
["changed fixtures", r => { r.fixtureFingerprint = "f".repeat(64); }], ["stale manifests", r => { r.manifestFingerprint = "f".repeat(64); }],
["wrong variant", r => { r.variant = "historical"; }], ["wrong base", r => { r.baseSha = "f".repeat(40); }],
["wrong archive", r => { r.archiveSha = "f".repeat(40); }], ["dirty source", r => { r.immutable = false; }],
["wrong master layering", r => { r.layering = false; }], ["source errors", r => { r.sourceErrors.push("missing.ts"); }],
["paid provider calls", r => { r.providerCalls = 1; }], ["paid live run", r => { r.live = "run"; }],
["missing gate", r => { r.gates.pop(); }], ["duplicate gate", r => { r.gates[1] = r.gates[0]; }],
["failed gate", r => { r.gates[0].passed = false; }], ["nonzero exit", r => { r.gates[0].exitCode = 1; }],
["failed SDK build", r => { r.setup.sdkExitCode = 1; }], ["missing Runner build", r => { delete r.setup.runnerTypeScriptExitCode; }],
["changed build output", r => { r.setup.buildOutputFingerprint = "f".repeat(64); }],
["extra cells", r => { r.expectedCells = 7; }], ["extra turns", r => { r.expectedTurns = 7; }],
["additional attempts", r => { r.maximumAttemptsPerCell = 2; }],
["missing runnerd provenance", r => { delete r.setup.runnerdProvenance; }],
["wrong selected runnerd path", r => { r.setup.runnerdSelectedPath = "packages/paperclip-runner/dist/bin/paperclip-runnerd"; }],
["claimed hosted reuse of local binary", r => { r.setup.runnerdProvenance.mode = "trusted_hosted_archive"; r.setup.runnerdExitCode = null; }],
["failed runnerd build", r => { r.setup.runnerdExitCode = 1; }],
["stale runnerd binary", r => { r.setup.runnerdSha256 = "8".repeat(64); }],
["wrong runnerd source SHA", r => { r.setup.runnerdSourceSha = "b".repeat(40); }],
["wrong runnerd source fingerprint", r => { r.setup.runnerdSourceFingerprint = "8".repeat(64); }],
["changed hosted source metadata", r => { r.sourceMetadataFingerprint = "a".repeat(64); }],
["source metadata errors", r => { r.sourceMetadataErrors.push("unknown lineage"); }],
["missing setup evidence", r => { r.setup.evidence.pop(); }],
["duplicate setup evidence", r => { r.setup.evidence[1] = r.setup.evidence[0]; }],
];
for (const [name, mutate] of negatives) test(`native completion prerequisite rejects ${name} before providers`, () => {
const r = receipt("candidate"); mutate(r);
assert.throws(() => assertNativeCompletionPreflightReceipt(r, current("candidate")), /exact immutable source SHA, variant and fingerprint/);
});
test("native completion prerequisite rejects a current dirty source and unavailable digests", () => {
assert.throws(() => assertNativeCompletionPreflightReceipt(receipt("candidate"), { ...current("candidate"), immutable: false }));
assert.throws(() => assertNativeCompletionPreflightReceipt(receipt("candidate"), { ...current("candidate"), fingerprint: null }));
});
test("native completion prerequisite rejects changed or missing retained evidence and unsafe paths", () => {
const output = mkdtempSync(join(tmpdir(), "native-completion-retained-"));
try {
const bytes = '{"passing":"native assertions"}', evidence = { file: "NC-tools.json", sha256: nativeSourceSha256(bytes) };
writeFileSync(join(output, evidence.file), bytes);
assert.equal(assertRetainedNativeCompletionEvidence(output, evidence), bytes);
writeFileSync(join(output, evidence.file), "Changed retained assertions");
assert.throws(() => assertRetainedNativeCompletionEvidence(output, evidence), /changed or is unavailable/);
assert.throws(() => assertRetainedNativeCompletionEvidence(output, { file: "missing.json", sha256: evidence.sha256 }));
assert.throws(() => assertRetainedNativeCompletionEvidence(output, { file: "../elsewhere.json", sha256: evidence.sha256 }));
assert.throws(() => assertRetainedNativeCompletionEvidence(output, null));
} finally { rmSync(output, { recursive: true, force: true }); }
});
test("native completion prerequisite requires the executed exact Rust carrier calibration", () => {
const output = "test provider_events::tests::preserves_closed_compatibility_terminal_tool_identity ... ok\ntest result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out;";
assert.equal(gradeNativeCompletionRustCarrier(output, 0), true);
assert.equal(gradeNativeCompletionRustCarrier(output, 1), false);
assert.equal(gradeNativeCompletionRustCarrier(output.replace(" ... ok", " ... FAILED"), 0), false);
assert.equal(gradeNativeCompletionRustCarrier(output.replace("1 passed", "0 passed"), 0), false);
assert.equal(gradeNativeCompletionRustCarrier(output.replace("preserves_closed_compatibility_terminal_tool_identity", "other_test"), 0), false);
});
function hostedFixture() {
const c = current("candidate"), r = receipt("candidate");
c.runnerdProvenance = { ...runnerdProof(), mode: "trusted_hosted_archive", archiveSha256: "7".repeat(64),
workflowRunId: "12345", workflowRunAttempt: "2", artifactName: `runner-e2e-build-${c.sha}-12345-2` };
r.setup.runnerdProvenance = structuredClone(c.runnerdProvenance); r.setup.runnerdExitCode = null;
r.gates = r.gates.map(gate => gate.id === "NC-rust-carrier" ? { ...gate, id: "NC-hosted-runnerd", executed: false,
calibration: "not_executed", total: 0, passedTests: 0, reuse: "trusted_same_run_build" } : gate);
return { c, r };
}
test("native hosted receipt explicitly reuses verified runnerd bytes without claiming Rust calibration execution", () => {
const {c, r} = hostedFixture(); assert.equal(assertNativeCompletionPreflightReceipt(r, c).passed, true);
assert.equal(nativeCompletionCommandGateIds("trusted_hosted_archive").includes("NC-rust-carrier"), false);
assert.equal(nativeCompletionCommandGateIds("trusted_hosted_archive").includes("NC-hosted-runnerd"), true);
});
for (const [name, mutate] of [
["executed Rust calibration", r => { r.gates.at(-1).executed = true; }],
["positive Rust test count", r => { r.gates.at(-1).passedTests = 1; }],
["missing explicit not-executed status", r => { delete r.gates.at(-1).calibration; }],
["fresh Rust build claim", r => { r.setup.runnerdExitCode = 0; }],
["different archive bytes", r => { r.setup.runnerdProvenance.archiveSha256 = "8".repeat(64); }],
["different workflow run", r => { r.setup.runnerdProvenance.workflowRunId = "999"; }],
]) test(`native hosted receipt rejects ${name}`, () => {
const {c,r} = hostedFixture();
// The hosted provenance gate is last in this deterministic fixture.
r.gates.sort((a,b) => a.id === "NC-hosted-runnerd" ? 1 : b.id === "NC-hosted-runnerd" ? -1 : 0);
mutate(r); assert.throws(() => assertNativeCompletionPreflightReceipt(r,c));
});
test("hosted runnerd evidence verifies the exact compact JSON capture before its diagnostic line", () => {
const proof = hostedFixture().c.runnerdProvenance;
const capture = `${JSON.stringify(proof)}\nRust unit calibration must be qualified by the separate exact-source local receipt and normal CI.`;
assert.equal(gradeNativeCompletionHostedRunnerdEvidence(capture, proof), true);
assert.equal(gradeNativeCompletionHostedRunnerdEvidence(capture.replace("\n", "\r\n"), proof), true);
assert.equal(gradeNativeCompletionHostedRunnerdEvidence(capture, { ...proof, binarySha256: "f".repeat(64) }), false);
assert.equal(gradeNativeCompletionHostedRunnerdEvidence(`not-json\n${JSON.stringify(proof)}`, proof), false);
assert.equal(gradeNativeCompletionHostedRunnerdEvidence("", proof), false);
});
Loaded 100 of 2195 files, more files were not shown because too many files have changed in this diff. Show more