Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-30 21:49:18 +01:00
Code Issues Packages Projects Releases Wiki Activity

feat: integrate Pi 1.0 into the experimental Runner (#14921)

Browse Source 
## Thinking Path

> - Paperclip manages AI agents and their work.
> - The experimental Runner owns provider processes and durable
sessions.
> - Pi needs working task execution and human controls.
> - The five-PR stack must preserve changes already on master.
> - Each layer now carries the complete integrated source for a safe
sequential fallback.
> - This PR belongs to native GitHub stack #15602, ending at #14956.

## Linked Issues or Issue Description

Refs #14436, #14631, #14743 and #14956.

Ship Pi 1.0 through the experimental Paperclip Runner. The five PRs are
#14921, #14922, #14923, #14924 and #14956. The user authorized the
complete merge after checks pass. Existing `pi_local` execution is
unchanged. Accounting and wider provider/platform qualification remain
deferred.

## What Changed

- Recover missing final replies after workspace finalization changes
owners, using accepted-turn evidence without rerunning work or granting
external-chat publication.
- Preserve the admitted Pi instruction root across warm runs, while
retaining changed-root rejection.
- Give Pi a bounded 15-second default shutdown grace so stop, drain
acknowledgement and durable suspension can complete. Explicit deadlines
and other providers retain their existing behavior.
- Integrate the Pi 1.0 runtime and master contracts.
- Use Pi profile 22. Preserve explicit caller-selected models and exact
native thinking levels. Keep Pi's wrapper, helper, extension and
question/control behavior unchanged from the qualified profile-19
runtime.
- Preserve master's Dot lifecycle and consent fields, configured task
environment, status guards and current Codex/Claude dependency versions.
Cursor stays qualified. Copilot stays pending; profile 17 binds the
changed shared protocol validation sources.
- Exclude general AWS IAM credentials from Pi static/custom provider
bindings and selected task projections; preserve the provider-scoped
Bedrock bearer key. Profile 21 is retained as historical provenance.
Rust and cloud install probes use the current declaration.
- Patch bundled brace-expansion 5.0.9 to the exact official 5.0.12
payload. Pin the patch and complete runtime closures. Include the patch
in normal installed setup tooling. Keep the upstream Pi shrinkwrap as
provenance and permit only this exact security correction.
- Include current attestation files in the Docker build context. Keep
the repository lockfile unchanged from master. CI and private image
builds resolve manifest changes before their frozen installation.

## Verification

- Full local `pnpm -r typecheck` passes, including Runner Rust, server
and UI. Focused integration checks pass: 194 Runner
admission/environment tests, 63 profile/credential tests with one
expected skip, 152 Dot/UI configuration tests, and Pi transcript/notice
tests.
- Full local `pnpm build` passes on the final source.
- Fresh final-source checks pass: all 698 Rust workspace tests (32
binaries), 156 credential/profile/controller tests with one expected
skip, Runner TypeScript typecheck, and 20 package/setup/sandbox tests.
- The profile-21 Pi materializer passes on the native host with the
official pinned Node 24.21.0 and its npm. It verifies all 150 locked
packages, the patched dependency and the exact closure. Setup/package
bundle tests and UI token gates pass.
- The old hashes were reproduced for all three supported targets before
calculating the patched graph. New closure hashes are darwin-arm64
`282022db10150c6632b3444df421342e7d534bdf5d5fb1097a2e79d0625a2bcf`,
darwin-x64
`64e251e19009f755c0b04f73ce2138246faab71a961b0f13d75ebfcc34bef12e`, and
linux-x64
`713b1fdff42fb56a1518bdc084f181d70bee8ebadc3e4b1d76321ed9108c8410`.
Independent native platform execution is separate from graph identity
reproduction.
- Historical cloud qualification remains unchanged: all seven core cases
pass on shipping source `10dc43c9ec65d88c2f782d62afb296d09494f215`,
harness `1a4408a48cfb5a1f094a311141c257c92cd7a893`, image
`sha256:5b3a775b383591bda1b0c1889e509acc70ce7f37c53f09733c81d59037f02280`,
and accepted Sonnet 4.6/low fixture. All 215 canonical files and all
seven cleanup checks pass independent verification. These are profile-19
results and are not relabeled as fresh profile-22 runs.
- Current Pi digest:
`sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`.
[The readiness
plan](https://github.com/paperclipai/paperclip/blob/codex/pi-production-readiness/doc/plans/2026-10-02-pi-production-readiness.md)
preserves campaign and failed-attempt provenance.
- Merge only after every PR's current-head CI and fresh review pass.
Linux CI covers the full suites, build and browser tests. The local
embedded Postgres API-authority suite cannot start on this macOS/Node 26
host, so Linux CI must confirm that suite.

### Fresh profile-22 core qualification — 2026-10-08

All seven accepted core cases pass canonically on Pi profile 22, with
`openrouter/anthropic/claude-sonnet-4.6` and native-confirmed low
thinking. This model is a fixture; production accepts the caller's
explicit Pi provider/model.

Runtime/install source: `3241a992f2a7703e59e97ed0fd3e5d6405de4401`.
Frozen accepted harness: `1a4408a48cfb5a1f094a311141c257c92cd7a893`.
Immutable cloud image:
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:506f22db7edd78f37c0c40bec1cc084af1850455026dbf467194bfbb8fcef141`.
Pi digest:
`sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`.

[Hosted Linux image and clean-install
verification](https://github.com/paperclipai/paperclip/actions/runs/37868328023)
passes, including all 20 source-bound archives, normal CLI/Pi setup,
companion import and the production pack reader. This exact installation
source includes the latest master integration and the corrected Pi warm
instruction-root fence. Full local typecheck/build and current-head
hosted CI verify the final stack. All 13 focused real-root regressions
pass. The full local executor suite passed 662 tests; 15 database tests
could not start the Mac embedded PostgreSQL service. Hosted Linux CI
passes the full required verification and E2E checks. These fresh
results keep their own source identity; profile-19 results remain
historical.

| Core path | Canonical campaign | Retained archive SHA-256 |
| --- | --- | --- |
| File edit, validation, download and Done |
`pi-core22-replyfix-0-1791511228` | 23 files;
`a473e8603a3dd4737863291f8d3d1e392391f0b16d433c3e0e0e9d8baf7a97b0` |
| Pending question and controller restart |
`pi-core22-replyfix-1-1791511376` | 33 files;
`6b829c4eb74e1f32a89c692a4ae7130dbfc1c6d3cf13915effe2103d9e242c8e` |
| Three-turn session/process/workspace continuity |
`pi-core22-replyfix-2-1791511587` | 23 files;
`7a87021f8f9a3fdd3c58bb4467f8d82c635e3ea4795d6e75f144d9aa14818df8` |
| Four typed questions and browser reconnects |
`pi-core22-replyfix-3-1791511881` | 42 files;
`9e31755252be1f4f9cb0626c984c142d4d1ae5f5bee3a7af08444db8d12c280a` |
| Plan approval and completion | `pi-core22-replyfix-4-1791512031` | 22
files;
`a0383ce1aab38e7b5a25ce0e9dd3bebea5c037ebd96ae6b29dae19015da2ae2c` |
| Same-turn steering and permission denial |
`pi-core22-replyfix-5-1791512261` | 39 files;
`c929b8c7070f0b66aedc17e65ca46e6beab1e363926ac9f7e2a75fb250f05949` |
| Stop during pending permission | `pi-core22-replyfix-6-1791512390` |
33 files;
`7f0a58ae0f4d5bfc76149435f4e322537089c5bd16e7ffe9b5ad71f10a621a07` |

All 215 canonical files (28714587 bytes) are independently
hash-verified. All seven cleanup grades pass, with no owned runtime
process or temporary root after each case. Automatic retries are zero.
The owned cloud host stopped normally after retention. The prior
profile-22 warm attempt remains failed and separately retained: archive
SHA-256
`1e54eba5ec72b50cee1534b23d1d1d4f21a090006b8a64501ba70db972abfde5`. Its
original canonical classification is preserved. Diagnosis reproduced a
product bug comparing an agent-files root against an unset
checkpoint-only field. The fix stores the admitted physical root
separately from the adopted per-run collection capability. The real-root
regression fails before the fix and passes afterward, including
rejection of a changed physical root. Fixture, grader, model and all
seven accepted case IDs are unchanged; this fresh campaign tests
final-reply publication after file registration first. The intermediate
restart attempt also remains failed and retained: archive SHA-256
`5dcaefdf1d17cf4cd54fd4cf810f45e736667392339b8ce7caf08bb4e225277f`. Its
original canonical classification is preserved. Pi resumed, wrote the
verified answer and completed its task; exact runner suspension was
proven, but idle stop consumed about 5.2s and left under 3s for the
drain acknowledgement. The Pi-only default shutdown grace is now 15s,
preserving a full 5s drain round trip and a finite suspension reserve.
Explicit caller deadlines, other provider defaults, literal drain
receipts and exact suspension identity checks remain unchanged. The
timing regression fails before this correction and passes afterward; all
18 focused settlement tests and Runner typecheck pass. The final-source
file attempt is also preserved as failed (`candidate_failure`), archive
SHA-256
`db6767b6773ea618997927ac77bdb005a5ac81492c7b9c0ffbc900449f829bc9`.
Native edit, validation, exact downloadable artifact and Done/succeeded
all passed, and the exact final reply was durably recorded. A workspace
recovery owner completed before the live heartbeat reached presentation,
leaving that reply absent from task chat. Recovery now materializes only
a completed final reply from the accepted turn of an ordinary internal
Done task, preserving issue/run/contract binding, suppression,
external-chat authorization and same-run deduplication. The database
regression covers the generated file-preparation receipt, suppression,
unapproved external continuation and replay. Server typecheck and all 49
response-selection tests pass; hosted Linux verifies the database
regression because embedded PostgreSQL cannot start on this Mac.

The delayed-final-answer database regression passes on [the final
root-source Linux server
shard](https://github.com/paperclipai/paperclip/actions/runs/37868262553/job/113628594152),
alongside 1,108 passing tests. The first root Runner shard had one
unchanged durable-resume test exceed its 5-second timeout; the identical
top-source shard and the isolated exact test passed. One rerun of that
failed job and its required aggregate passed without source or test
changes. The original failed job log and the single-rerun receipt remain
retained.

### October 9 merge verification

Current merge head: `5a8fe63512a7166aaef5cf50065a25008aa8b44b`. All
current-head checks pass, including `ci / verify` and `ci / e2e`;
exact-head Greptile review is 5/5 with no unresolved threads. Current
master conflicts are resolved. The user authorized the maintainer
override of the code-owner review gate after these checks. The seven
retained live core cases remain bound to source
`3241a992f2a7703e59e97ed0fd3e5d6405de4401` and its recorded cloud image.

## Risks

- The security correction changes the dependency closure and profile
identity. Old sessions must reopen on the new profile. Exact identities
and credential bindings fail closed.
- The runner remains experimental and requires explicit selection.
Legacy Pi Local is unchanged. Caller model IDs pass through; the E2E
model is a fixture.
- Accounting and the broad platform/provider matrix remain deferred.
This merge does not publish a release or deploy a service.

## Model Used

OpenAI GPT-6 through Codex assisted with reasoning, repository
inspection, editing and tool use. The exact serving ID and context
window are not exposed in this session. Final live qualification uses Pi
1.0.0 with `openrouter/anthropic/claude-sonnet-4.6` and native-confirmed
low thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-09 08:56:41 -05:00
1 parent 4fb1ab53a8
commit 57e977be72
517 files changed
+61737 -1440
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/runner-e2e/FIXTURES.md
+494
View File
@@ -31,6 +31,26 @@ profile, model qualification, environment, task, or ranking-snapshot change
must change that fingerprint automatically so the dashboard can annotate the
boundary instead of silently joining unlike totals.
Pi native definition 21 supplies one ordinary JSON write with the nonce followed
by exactly one LF, then asks for a complete native read. The independently
checked outcome remains 33 bytes, saved through the public managed-file API
and copied into a fresh task after controller restart. Missing LF, extra LF,
literal escapes, CRLF and stale values all fail. The cross-root denial, protected
parent seed, permissions, deadlines and zero automatic retries remain required.
Both runs also require a completed, untruncated native-read receipt containing
the exact memory text and a withheld private-root target, and reject shell
execution. Named workspace reads and unrelated/bootstrap text cannot substitute;
Remote observer admission keeps the owned-run and active-lease checks. Its
existing readiness RPC verifies the pinned native daemon before installation;
legacy executionStage can remain preparing for a native run. Readiness and case
deadlines remain unchanged.
The first Sonnet measurement passed the byte/restart checks but used a shell read
in the fresh task. Its original grade remains preserved.
The user approved Sonnet 4.6 through OpenRouter as the explicit Pi qualification
model; production model selection stays caller-controlled. Earlier DeepSeek
attempts and their fingerprints remain historical evidence and do not qualify
this new model/fixture combination.
The explicit [stock-harness suite](STOCK-HARNESS.md) wraps existing profiles with
`productionDefaultHireProfile`: omit only `instructionsBundle` so the public
hire route loads the shipped default, while preserving runtime, permissions,
@@ -139,6 +159,14 @@ a deletion through public file APIs. Native turns 2 and 3 must copy/hash only th
changed memory file, with a saved receipt and the same provider PID. Journal and
Git stress fixtures retain fixed external bundles as controls. Keep the stable-PID
oracle strict; `instruction-persistence` also covers cold restarts and quota handling.
The explicit `rich-acp-warm-continuity` fixture uses the workspace-only prompt:
ACP providers retain their unchanged AGENT_HOME and must preserve the same native
session, runner instance, provider session, PID, and process start fingerprint.
It does not request personal-file edits, because changed ACP agent files require
provider retirement before collection. Pi's separate `agent-files-fresh-run`
case retains changed-home save and fresh-task restoration coverage. This split
does not weaken the stable-process oracle or change the Codex checkpoint fixture.
Native turns 1 and 2 include an actionable human review in the completion report's `attentionRequests`. Paperclip creates the review gate from that report. An explicit question-tool wait yields the turn and suppresses its final prose, so it is not interchangeable with this completion-review fixture. Turn 3 reports Done without another review.
Every selected case runs in its own isolated Paperclip process, and independent
@@ -309,6 +337,470 @@ current and incoming files and applies the run edits against the reviewed curren
directory hash. All three tasks' runs count toward billing and teardown. The suite
is explicit-only. No private control-plane hooks or direct database writes are used.
## Pi native boundaries
Definition 23 and Pi controls definition 10 confirm `/proc/<pid>` absence
separately when a proc read fails during Linux process exit. An existing
unreadable process, an unreadable absence check, identity drift, and incomplete
terminal evidence still fail. The observer retains closed causes for process
reads, stat shape, runtime binding and terminal sealing; it never retains raw
process arguments. Historical failed receipts remain failed.
The explicit-only `pi-native` suite has five local and four Daytona candidate
cells, with no automatic retries. The remote suite excludes automatic deny-all
because its initial native file read is itself denied. `native-questions` answers the real runner-owned Pi select, confirm, input
and editor tool through four durable browser cards, reloading before every answer.
Undisclosed text and independent workspace JSON prove delivery to the same live run.
Pi's SDK cannot distinguish negative confirmation from dismissal; the expected
result is explicitly `negative_or_cancelled`, not proof of cancellation.
`agent-files-fresh-run` supplies one native write argument object whose content
is the hidden nonce plus exactly one final LF. The agent replaces only the
AGENT_HOME prefix in the supplied path; it preserves the JSON newline escape in
the content argument, then reads the complete file once. A missing LF, literal
backslash-and-n, repeated write, or claimed success cannot satisfy the independent
byte oracle. The flow uses native file tools in the registered AGENT_HOME,
requires a stopped-run save receipt and public managed-file
readback, then restarts the server and verifies exact bytes from a fresh task. An
attempted write to an unassigned isolated sibling path must fail without creating
a file. `restrictive-denial` requires a correlated failed native write and absent
file under `deny-all`. All runs count toward spend and teardown. Browser reload
is reconnect evidence only; these cases do not establish provider-death recovery.
They use public product APIs, real browser answers, and ordinary isolated files;
no database writes, private hooks, or fabricated provider results are allowed.
### Pending native controller restart
`native-pending-controller-restart` restarts the public controller while one Pi
input callback remains unanswered. It requires the same durable interaction,
request, live run, native session, turn and producer before and after restart.
For local execution, the public run's exact PID, process group and start identity
must identify an already-observed durable runner under this controller. The test
preserves only that runner tree during restart, checks the same live identity
afterward, and keeps the old process owner for complete final cleanup alongside
the replacement controller. Other controller children are retired normally.
Remote execution does not infer local process authority from remote PIDs.
Only then does the browser submit previously undisclosed text. One durable
resolution, one original successful turn and independently read exact workspace
JSON prove delivery. A replacement run, replay, cancellation, expiry, rewritten
request or merely reloaded browser cannot pass. Full states and PRP identities
stay in private snapshots under the existing publication allowlist.
The local runner's unique `turn.submitted` receipt may precede assignment of the
provider turn ID. The oracle accepts that missing ID only before the single
matching `turn.started` and request creation, with the same session and producer
and increasing durable/source sequence numbers. Later missing or changed turn
identities still fail.
### Pi file editing and registered artifacts
Pi's `extended-harnesses/file-edit-validate` seeds exact bytes before startup and
requires one native edit lifecycle followed by exactly one successful native
bash execution with the exact nonce-bound byte-validation command as its
projected title and a validation marker. A marker-only echo cannot pass. Final bytes must
match the fixture. The Pi task prompt explicitly forbids additional shell calls,
including metadata commands and repair attempts. It supplies the expected post-edit
byte size and hash for registration. Extended definition 5 places the exact command in a
fenced Bash block so the production Markdown editor preserves its operators and
literal escapes. Definition 4's escaped-paragraph attempt keeps its failed grade.
Extra Bash calls fail the unchanged oracle even when the
edited file and downloadable artifact are correct. Extended definition 4 makes
Pi's artifact title equal the exact filename required by the shared registered
artifact check. Definition 3 attempts retain their original definitions and
grades; its Pi prompt requested a different title. The real
`register_deliverable` receipt, attachment metadata, publication activity,
visible task attachment and authenticated public download must all agree on the
file's bytes, hash, company, issue, agent and originating run. A file on disk or a
model completion claim cannot substitute for publication. Daytona additionally
requires the public run's finalized `nativeWorkspaceSync` descriptor, baseline and
final-host hashes, bound workspace and public environment lease to match this
run/company/environment/provider lease and remote root. The checked host bytes
come through production stage-in/copy-back; this is explicitly `product_copyback`
provenance, not a sealed guest observation or an independently recomputed whole
host-workspace snapshot. Downloaded artifact bytes remain independently checked.
Private `pi-file-seed.json`, `pi-file-observation.json` and `pi-file-evidence.json`
retain the seed, downloaded/workspace bytes, public publication receipts, checked hashes,
correlated tool identities and a before/after diff computed from independently
checked workspace bytes. This is not native diff presentation. The current
common tool projection omits raw arguments and reports typed `exitCode: null`;
the oracle checks the command title and completed lifecycle, without claiming raw
invocation arguments or a typed exit code. The sidecar converts Pi absolute file
locations into bounded workspace-relative display targets; the oracle requires
the exact relative filename, but that display value is not file-access authority.
Restoring raw arguments, typed exit code and native diff presentation remains a
capability follow-up.
Wrong or missing lifecycle, byte, registration, download or recovery evidence is
a candidate failure; transport/infrastructure failures retain the existing
harness classification. Positive and plausible-wrong/missing-evidence unit
calibrations do not count as paid qualification. All automatic retries stay zero.
`native-pending-provider-death` adds one real Daytona-only Product journey.
After the native input is publicly durable and still unanswered, the existing
owned remote observer admits the exact Pi child through its verified wrapper
parent, source-pinned closure files, executable inode, workspace, run/lease/session
ancestry and fresh PID/start-time checks. Pi overwrites Linux argv via
`process.title`, so the private receipt explicitly uses pinned-parent entrypoint
attribution and never claims original child argv. A pidfd targets only that child;
worker death, broad process-name matching and controller Stop cannot substitute.
The production bootstrap may name its held executable as `/proc/self/fd/3`
or `/proc/self/fd/7` in the wrapper argv. That form is admitted only when the
wrapper's corresponding descriptor and executable both have the exact sealed
snapshot Node inode. The guard and wrapper entrypoint paths remain exact.
Missing, foreign or other descriptor numbers fail before signalling.
Production may also stage the runner executable as a link to the image's
verified installation. The fault helper reads the resolved regular file, checks
that the named link remained unchanged, and still requires its pinned hash and
exact `/proc/<runner-pid>/exe` inode. Link replacement, missing targets and a
different executable fail admission. Linux calibration covers this installation
form as well as a copied runner.
The agent-memory fixture requires the nonce's UTF-8 bytes followed by exactly
one line-feed byte (`0x0A`). Its prompt states that byte contract in plain text
and provides the exact content as fenced JSON. Fenced task prompts use the rich
editor's Markdown paste path; filling the editor directly produces escaped
paragraph text instead of a code block. The issue API preserves literal escapes
in real multiline bodies and only recovers self-escaped line breaks in legacy
single-line bodies. Code fences and JSON escapes must survive both boundaries.
The prompt explicitly states that native write never adds a newline and that
complete native read preserves one when present. A credential-free probe of the
installed Pi 1.0.0 tools checks both a 32-byte value and the 33-byte value with a
final line feed; both write and read retain the exact supplied bytes.
The prompt orders one memory write, one complete native read, a separate expected
cross-root write denial, and then completion. Native paths use the exact current
absolute agent directory; shell-variable expansion is not assumed. An incorrect
memory result must be reported without claiming success. Native readback and the
managed-file API must retain the exact bytes across a new task and controller
restart. The byte graders remain unchanged.
Controller cleanup can admit a replacement group member only when its ancestry
belongs to a separately revalidated, continuously owned process. A recycled
numeric group, a changed PID/start identity, or any unowned live member still
fails cleanup before signaling. This rule is recorded in `pi-native` version 12.
The runtime itself must emit `runtime_request.expired` for the original callback
with `provider_process_lost` and `replayAllowed:false`, followed by native turn
failure. The permanent failed-terminal recovery projection must put the owned
issue in Blocked, proved through API and browser and retained through cleanup.
The original durable card must expire, retain zero answers and lose its browser
answer controls; any supersession must name the separate fallback card.
Both public stale-response APIs must reject the old answer. Any production-created
`wake_assignee` fallback is separately identified and remains unanswered; it is not
a restored native callback. The sealed observer proves no continuation marker was
created and all owned processes retired before public lease deletion. The company
must retain exactly the original run through cleanup.
Local provider-death remains excluded. The Python admission tests include a
real pidfd calibration against a synthetic title-changing child on native Linux;
the normal E2E unit wrapper invokes it with no provider credentials or network.
macOS runs metadata negatives and explicitly skips this Linux-only calibration.
That calibration and the earlier fake-Pi wrapper/bridge tests do not count as the
real paid Product lifecycle proof. This new candidate cell remains unqualified.
The pending-question controller-restart browser matcher accepts only the retained
issue's UUID or public identifier and the exact retained interaction ID. The UI
normally posts with the public identifier. Durable request, run, turn, session,
producer and single-delivery assertions remain required after submission.
## Pi active controls
The explicit-only `pi-controls` suite adds `pending-permission-stop` and
`same-turn-steering` on local and Daytona, each with one provider run, a
120-second active-turn timeout and a 300-second attempt budget. These four
control cases retain their behavior; the current matrix totals 26 Pi cells.
Pi 1/profile 13 and coverage revisions intentionally change the affected suite
fingerprints, so older qualification receipts cannot be reused. Catalog presence and
deterministic calibration do not constitute paid qualification.
Both cases create a task through the browser and select `approve-reads` and
`per_turn` through the public agent API before startup. They retain
`paperclip.e2e.pi-control-pending.v1` while an exact native Pi write and its
permission card are pending in one run/turn/session/source. The card's native
tool ID is joined to the canonical execution ID using the existing runnerd
identity mapping. Pi does not emit Cursor/Copilot diagnostic notices; those
notices are never synthesized. Earlier native reads can provide orientation;
other native operations cannot substitute for the observed write.
On Daytona, the operator first publishes the setup instruction file after the
owned observer is armed. If Pi delegates that native read, the fixture approves
only its exact request through the public API with `accept` (allow once). The
read must name the published random setup file, complete successfully in the
same native run/turn/session/source, and leave the file hash unchanged. The
fixture retains the request, resolution, completed read and both observer
snapshots. Only the two hash-bound setup permission records are excluded from
the write-permission count. All native read rows remain in the oracle. Another
permission, edit, shell command, foreign path or incomplete read cannot receive
this exemption. The tested write remains unanswered until Stop or browser Deny.
The production permission policy and all write/retirement assertions stay in
place. `pi-controls` version 7 and `pi-native` version 5 record this correction;
older attempt fingerprints and grades remain historical evidence.
Native tool arguments can arrive after the start event. An earlier null target
is allowed only until the same execution first supplies the exact expected
path. Missing targets, conflicting paths, another execution's path, or a later
loss of the proven path fail. The original start and permission rows remain
bound by retained hashes through control dispatch and settlement.
Stop awaits the pending evidence write and rereads that boundary before sending
one caller UUID to the public cancel API. It requires the original request's
normalized cancellation closure, a cancelled terminal, and the same-scope
caller-owned intent and acknowledgment audit IDs. Normal completion, a prior
permission decision, an expired request, and unacknowledged cancellation fail.
Only after cancellation does it attempt a stale **decline**, which must return
409. It never sends an allow decision. The task remains In Progress, with one
cancelled run and no automatic continuation.
Steering submits a random marker only in a browser comment after the permission
is pending, binds the queued comment to the exact body submitted by the
production Markdown editor, then clicks that comment's production Steer button. It records the
exact public POST's queue/revision/run binding. A rejected public POST ends the
journey before any denial. Success requires the saved run acknowledgment plus
the Product facade's same-turn acknowledgment item. The raw
Rust `acpx-control-*` transport echo is suppressed by the facade;
`CodexHarnessSession.steer` emits the durable correlated item after the command
acknowledges. A deterministic calibration invokes that actual producer. The
browser denies the original write only after acknowledgment while the request
still remains pending. Success requires correlated native denial, the random
marker as the persisted and visible final response, Done, and one succeeded
run. Merely echoing the comment in the transcript or scheduling another turn
cannot pass. Native `pi/follow_up` and durable native queue state are not tested.
Both cells require independent absent-target and zero-mutation evidence plus
owned process retirement. Local observation runs through cleanup. Daytona uses
the existing authenticated lease observer with exact image/executable pins,
fresh baseline/pending observations and its owned-process retirement seal;
retrieving that seal later does not claim later filesystem surveillance.
Missing cleanup fails the result. Provider-death recovery is outside these
cases. All screenshots and receipts use the existing evidence/redaction/result
pipeline; native USD may remain unknown and estimates remain distinct.
## Copilot native protection
The explicit-only `copilot-protection` suite has two cases on local and Daytona,
each with one run (120s
provider timeout, 300s attempt budget). `native-permission-deny-write` denies one
exact native edit through its browser card, waits for the delivered rejection and
failed tool, then cancels through the public run API. Its expected outcome is a
cancelled run and an unfinished task, not successful task completion.
The retained `paperclip.e2e.copilot-denial-settlement.v3` proof separates the exact
provider terminal from the audited controller Stop. It records either
`provider_cancelled_or_interrupted` or `provider_completed_observed_before_stop`.
The latter requires the exact terminal row to be returned by the operator API
before Stop dispatch. The fixture awaits retention of its scoped row-hash receipt
before sending Stop, then matches that receipt against the final durable rows.
The same normalized session/source stream and a later source sequence than the
failed edit are required. A fixed 2s observation window, inside the existing case
deadline, allows natural completion. A normal terminal first seen after Stop is
insufficient evidence. Database createdAt is transaction-start metadata; it cannot
prove that an event committed before Stop acknowledgement. It cannot establish active-turn cancellation;
a cancelled/interrupted terminal also does not by itself prove Stop reached active
work. Dedicated cancellation coverage must retain its active-operation evidence.
Missing, ambiguous, failed or foreign terminals and incomplete Stop receipts fail.
This distinction does not regrade earlier failed attempts. The case rejects
extra native operations/runs and observes the absent target through process
retirement. Filesystem event loss or an unexplained parent-directory timestamp
change makes no-effect coverage incomplete; stat polling alone cannot pass.
`attached-async-settlement` starts a fixed finite command with explicit async mode
and `detach:false`, then asks the model to attempt immediate completion. Its marker
is a private diagnostic sentinel, not a requested user deliverable. The task keeps
empty completion evidence and forbids publication, extra commands, and waiting tools.
Suite definition v6 preserves this early-completion stress and every settlement
assertion. A retained v5 Daytona failure exposed a bootstrap wording conflict:
comma-separated prohibitions left “create or modify any file” as a positive clause
in the continuation objective's file-delivery classifier. Each bootstrap prohibition
now has its own explicit “Do not” sentence. The production delivery policy is unchanged;
actual requested file outputs still require accessible delivery evidence. That failed
attempt remains failed, and v6 requires fresh live qualification. A one-shot
private socket in the tested environment accepts only the fixture nonce, never an executable or command;
the test owns/reaps a predeclared child and records its actual exit before releasing
the provider-launched client. The independent marker, client retirement, native shell
linkage and actual durable turn terminal must agree. Fixture resources close in a
finally block. This establishes the tested finite attached-command behavior, not all
background modes or detached-process settlement.
Both cells consume bounded, origin-correlated `copilot_tool_evidence_v1` notices
persisted through the ordinary run-event API. Missing, redacted, ambiguous or
explicitly incomplete notices fail qualification. Old runtime artifacts therefore
cannot qualify these cases. No prompt/title substitutes for native input, no raw
command or arbitrary tool input is added to production event payloads, and no private
runner hook or database write is used. The cases are registered but require a newly
built source/pack and separately authorized paid execution before claiming Product
qualification. Full restrictive-workflow completion remains separate: the negative
case never auto-approves an uncorrelated later MCP permission.
The Copilot protection cells explicitly select `per_turn` lifecycle. Their read-only
process journal accepts the API runner PID only after checking its OS start time,
process group, exact run ID argument and `per_turn` argument; it never treats a
retained warm daemon as a leaked per-run process or signals an API-provided PID.
Directory-watch coverage also rejects parent device/inode replacement or removal.
The manual `cursor-native` suite has four cases on each of local and Daytona with
one expected provider run each and a 120-second provider deadline. Browser
choices, rejection feedback, native origin/decision receipts, workspace checks
and provider retirement are independently checked. Failed or missing cleanup
assertions fail the final report even if the native interaction passed. No
native callback, private-HOME artifact export, or paid qualification is inferred
from a semantic question or plan result.
### Remote native proof scope
Remote native cells require the exact authenticated lease, run, immutable image
and executable bindings documented in [the runbook](README.md#remote-native-evidence-and-warm-continuation).
Action publication follows observer readiness and a saved baseline. The observer
seals before environment destruction and retains file bytes on the host; no
post-deletion RPC or host-copy-back evidence may satisfy remote no-effect or
retirement checks. Runtime-internal files are an explicit scoped exclusion.
`human-permission-denial` requires native request/tool correlation, the browser's
exact Decline, delivered denial and failed write, and unchanged file evidence
through retirement. All normal project and company boundaries apply.
Cursor and Copilot may exempt a bootstrap read only when every native notice for
that completed tool origin carries the single-path attestation matching the
observer's exact random action file. The passive projector derives this digest
from one explicit scalar native input path, rejects ambiguous/multiple paths,
and checks subsequent input/location updates for changes. Durable canonical
execution receipts must match the same run, turn, execution identity, status,
and order; any explicit canonical target must agree. Missing or conflicting
attestations fail qualification. PRP retains only the first location and terminal
updates may omit it, so canonical targets alone cannot prove bootstrap ownership.
Neither tool titles nor output text supplies path evidence.
Denial ordering uses canonical request, decline-resolution, delivery, failed-edit
and terminal source sequences. Sample checkpoints retain the exact run/turn/source
cursor. File samples and continuous-watch coverage compare only observer-local
times. Provider emission, browser click and server persistence clocks are never
compared to each other. Final remote samples use the sealed, independently verified
process-retirement receipt; their timestamps are not relabeled as host time.
This denial case does not qualify Stop during a definitely pending native request.
That active-turn cancellation boundary needs a separate live case. The attached
async-command oracle is unchanged by this denial-only correction.
### Correlated native Stop API
The board-only `POST /api/heartbeat-runs/:runId/cancel` accepts an optional
`cancellationRequestId` UUID for native runs. Company access checks still apply.
The server reserves it under the run-row lock before dispatch, uses
`native-cancellation:<UUID>` as the durable intent ID, and returns HTTP409 for
an earlier or different caller intent, an earlier uncorrelated Stop, or a
terminal run without that same reserved intent. Malformed UUIDs return HTTP400.
Repeating the same UUID from the same board actor is idempotent. A different
actor receives HTTP409; local trusted board uses the `local-board` user ID.
Default clients may omit the field; they preserve and join an existing reserved
intent rather than overwrite it.
The denial fixture generates its UUID before observation, retains it in
`paperclip.e2e.copilot-pre-stop-observation.v2`, and requires the same intent in
the response and final `paperclip.e2e.copilot-denial-settlement.v3` receipt.
It refuses a non-running controller or existing Stop marker before dispatch.
The completed-provider branch still requires a running controller that this
request can stop; it does not accept a no-op Stop of an already terminal run.
## Definitely-active native Stop
`native-active-stop` / `pending-permission-stop` has four explicit-only cells:
Cursor and Copilot, each local and Daytona. Its `native_active_stop` flow retains
`paperclip.e2e.native-active-stop-pending.v2` from the public API while exactly one
native permission remains pending, the exact controller run is running, and no
Stop or answer marker exists. The receipt independently binds native session,
normalized session, turn, source instance, request/tool IDs, source sequences and
canonical row hashes. An awaited artifact write and fresh pending reread precede
Stop dispatch; process-monotonic timestamps describe only these local observer
boundaries. Remote provider clocks and database transaction timestamps never
establish that ordering. The atomic caller UUID fence rejects an earlier racing
Stop instead of borrowing its acknowledgement.
Suite version 3 accepts either canonical card/tool-start arrival order. The
exact native origin, canonical tool start and unanswered permission must all
exist in both pre-Stop API observations. The v2 pending receipt adds the native
origin and tool-start row hashes and source sequences. The fresh reread and
settlement must preserve those exact rows; a later tool start cannot backfill
missing pre-Stop evidence. Command/path, request, tool, turn, session and source
checks remain strict. Cursor's native evidence projector still requires the
exact tool origin before it can emit correlated permission evidence. Copilot
emits permission evidence immediately, so its permission notice may also precede
the native tool notice. Calibration tests exercise both actual projectors in
both input orders. Neither policy claims the original ACP wire order of a live
attempt. Old v1 receipts are not valid inputs to the
new grader. Earlier paid failures retain their original definition and grade.
Suite version 4 and `paperclip.e2e.native-active-stop-settlement.v2` accept only
`pending_permission_cancelled`: the Product harness's exact
`runtime_request.cancelled` closure (`reason: turn_terminal`, matching request,
item and turn, no answer), then one exact `turn.cancelled` (`status: cancelled`,
`error: null`), plus the same scoped caller-owned native Stop acknowledgement.
Both events must belong to the retained source/session/turn and follow all four
pre-Stop evidence rows. The harness consumes raw backend request closures and
projects its own pending-request outcome before the terminal. Raw backend-only
`provider`, `requestType`, `replayAllowed` and `providerTurnId` fields are not
required from that Product projection. Replay refusal is independently checked
through the stale public answer below. A generic terminal without the retained
request and exact acknowledged caller UUID is insufficient. The v2 settlement
receipt records this changed oracle; historical v1 receipts and failed attempts
retain their original grades. Missing, duplicate, foreign, failed,
interrupted or normal terminal evidence fails. Only after this proof does the
fixture attempt a stale public answer, requiring HTTP409 and an unanswerable
browser card. It retains one cancelled run, issue `in_progress`, exact target
absence through continuous observation, and all observed owned descendants
retired. Local files require four fresh observations through cleanup. Daytona
instead retains two live snapshots (baseline and pending) plus one automatic
owned-process-retirement seal, with a continuous zero-mutation watcher and the
same complete root/descendant journal. Since suite version 2, this is retained as
`paperclip.e2e.native-active-stop-remote-retirement.v1`; it explicitly records
`filesystemAfterRetirementObserved:false`. The per-turn observer seals itself
when the owned tree retires, before the sandbox is released. Reading that receipt
later is not a fresh post-UI or post-cleanup filesystem observation. Relabeled,
reused, missing or out-of-order samples fail. The stale-answer and rendered UI
checks remain separate, followed by fresh API cancellation/no-extra-run checks
both after UI and in cleanup. Unproven cleanup fails independently. Only fully
attested bootstrap reads may precede the one tested operation; alternate operations or attempts are rejected.
The existing `copilot-protection` denial remains distinct: rejecting a permission
before Stop does not exercise this pending-callback boundary. This new suite has
pure calibration and wiring tests, not a paid qualification result. Provider
process death is not simulated by substituting the chat runner-worker crash hook.
### Copilot attached semantic completion evidence (suite 8)
Attached settlement now requires one native `paperclip_finish` lifecycle joined
to an invocation-captured, bounded Paperclip bridge receipt by exact call, input
and result digests in the same run/turn/native session and durable source stream.
A matching display title is not authority. The exact proposed completion input
must match a control-plane `run.result.accepted` body and the requested summary.
Transport `returned` alone, including a returned rejection, cannot pass. The
visible exact terminal comment remains an independent assertion. Both local and
Daytona cases reject extra native operations; existing attested bootstrap reads,
command settlement, process retirement and marker checks remain required.
The receipt reader fails closed when the production native receipt projection is
missing, partial, duplicated or outside its bounds (including 256 KiB for the
whole native rawOutput, which may repeat text). This oracle depends on the new
production semantic-receipt contract and does not regrade earlier failed runs.
The bridge call hash identifies the invocation and its native receipt. It is not
inferred from `run.result.proposed.itemId`, which can identify the run instead.
The v2 receipt keeps the raw invocation input digest separate from the normalized
input digest captured by that same authenticated invocation after production
validation. Raw arguments may omit schema, artifacts and attention requests;
the production validator supplies those defaults. The oracle never reconstructs
or normalizes a body to make it match. The raw call/input/result hashes still join
the native lifecycle; its normalized input hash must also match the authority.
Exactly one proposed body must hash to that normalized digest and equal exactly
one accepted body; identical duplicate proposals or receipts still fail.
Suite 8 writes `paperclip.e2e.copilot-semantic-completion.v2` and requires
`paperclip.semantic_tool_receipt.v2` under `paperclip_semantic_tool_receipt_v2`.
The authority has exactly eight bounded details. An explicit null normalized
digest is diagnostic only and cannot qualify a successful finish. Legacy v1
receipts do not qualify fresh runs. This fixture depends on the corresponding
production receipt and durable-redaction fixes; it does not supply them. The shell
result is one separate complete read lifecycle: pending and any progress name the
started shell, and its successful terminal also names the original command. A
completed-only read or a second shell read cannot inherit that exemption.
Denial case version 5 and denial settlement schema v3 are unchanged.
## Direct blocker fixtures
`blocker-cases.ts`, `blocker-fixtures.ts`, `blocker-flow.ts`, and
@@ -359,3 +851,5 @@ as completed and end the native turn; Paperclip must retain a failed run with
missing semantic finalization and an unfinished task. That is a denial outcome,
not task success or operator cancellation. Stop during an unresolved permission
remains a separate `native-active-stop/pending-permission-stop` gate.
Pi controls definition 9 and native definition 16 create an explicit title through the production search creation action and bind the task ID from the public creation response. The scoped task and assignee must match before any native control. Automatic naming is outside these strict native-operation fixtures; all existing permission, byte, process, run-count and cleanup assertions remain required. Preserve the failed title-lookup attempt as its original failure.
tests/runner-e2e/README.md
+411 -1
View File
@@ -107,6 +107,25 @@ The launcher always sets `PAPERCLIP_ANNOUNCEMENTS_ENABLED=false` for its isolate
instances so announcement panels do not obscure screenshot evidence. No shell
or workflow configuration is needed, including for Daytona cells.
## Pi controls (explicit only)
`--suite pi-controls` selects four candidate Pi cells: pending-permission Stop
and browser same-turn steering, each local and Daytona. Discover without
credentials using `pnpm test:e2e:runner -- --list --suite pi-controls`. For an
authorized bounded attempt select, for example,
`--id pi-controls.runner-acpx-pi.local.pending-permission-stop
--max-automatic-retries 0 --max-parallel 1`.
The [fixture contract](FIXTURES.md#pi-active-controls) requires exact pending
native-write evidence, caller-owned control acknowledgments, independent
no-effect/retirement proof and the actual production task UI. Daytona also
requires the existing immutable image and executable digests. These cells do
not establish native follow-up queue persistence or provider-death recovery.
The current Pi matrix has 26 explicit cells (13 local and 13 Daytona), including
these four controls, the pending-native-question restart cases, and one Daytona
provider-death case. Pi's profile
remains pending; catalog presence is not live qualification.
## Provider-free browser bootstrap regression
`pnpm test:e2e:runner:browser-support` includes a wide development-module graph
@@ -250,6 +269,7 @@ Shell variables take precedence over the local file. The recognized names are:
- `CURSOR_AUTH_TOKEN` (extended Cursor candidate)
- `COPILOT_GITHUB_TOKEN` (extended Copilot candidate)
- `PAPERCLIP_E2E_DAYTONA_IMAGE` (Daytona only)
- `PAPERCLIP_E2E_DAYTONA_NODE_SHA256` and `PAPERCLIP_E2E_DAYTONA_RUNNERD_SHA256` (native Cursor/Pi/Copilot Daytona fixtures; exact `sha256:...` executable digests from that image)
The image must be an immutable `image@sha256:...` reference. The launcher
reports missing variable names but never prints values. It passes raw provider
@@ -1350,7 +1370,26 @@ lockfile from its own trusted checkout, never from the tested branch.
The restart supervisor starts Paperclip with the TypeScript loader in the same
Node process it owns. A forced stop therefore cannot leave an old controller
alive to stop the embedded database after the replacement starts.
alive to stop the embedded database after the replacement starts. On Unix, the
server has its own process group so Playwright's wrapper-group shutdown cannot
signal it independently. Signal handling, restart cleanup, and final cleanup
share one stop operation per server: one graceful signal, a 30-second wait, then
SIGKILL to revalidated owned groups and a 5-second exit wait if needed. The
wrapper retains observed descendants across an early server exit and checks
PID/start identity before group signals. A failed cleanup can resume its saved
phase without another graceful signal. Windows retains direct-child signaling.
This prevents a second graceful signal from interrupting asynchronous warm-session
retirement.
Launcher cancellation first signals its outer group, so Playwright and the wrapper
own graceful server shutdown. It allows 45 seconds for that chain before bounded
5-second forced cleanup of revalidated descendants. Normal launcher exit uses the
same retained descendant inventory. Zombies count as stopped; an unavailable or
uncertain identity inspection triggers bounded direct-child-only retirement and
still fails the tree audit. Failed cleanup preserves the temporary state. Graceful
owners are selected from the same validated table used for delivery; ESRCH allows
selection of a surviving owner, while a delivered signal covers that owner's
existing descendants through the grace window even if the owner exits first.
Everyday restart and Stop scenarios exempt only their recorded cancellation,
graceful-shutdown interruption, or process-loss outcome. A later adapter error
@@ -1541,6 +1580,318 @@ The separate Runner Evals `extended-harnesses` campaign lives in the private
`paperclip-evals` repository and grades semantic protocol behavior against the
mock control plane. Neither suite substitutes for the other.
### Pi native Product fixtures
The separate `pi-native` suite defines ten explicit Pi cells: native questions,
agent-file persistence, browser permission denial and controller restart with an
unanswered native question on local and Daytona, plus
a local-only automatic `restrictive-denial` case and one Daytona-only
`native-pending-provider-death` case. The latter cannot use the remote
setup read under `deny-all`, so it is explicitly excluded there. It uses the
pinned candidate profile and exact OpenRouter model. It is excluded from `--all`,
never automatically retries, and retains qualification as pending until live proof.
Reserve and reconcile each paid cell just as for `extended-harnesses`.
Provider death is a real, one-run paid Product journey, distinct from controller
restart and Stop. The owned Linux observer pins the verified wrapper ancestry,
closure and executable inode before signalling exactly its unique Pi child through
a pidfd. Pi overwrites its argv with `process.title`; entrypoint attribution is
explicitly parent-attested, not an original-child-argv claim. The fixture requires
production-generated non-replayable expiry, an expired original browser card,
rejection of stale answers by both public APIs, no continuation file effect, and
no second run. Any durable `wake_assignee` fallback stays separately identified
and unanswered. Full process/run/lease evidence remains private. Local provider
death is excluded because this exact ownership mechanism requires Linux pidfd.
The Python helper calibration uses a synthetic transport and a real title-changing
Node child; it proves fault ownership only, never paid Pi lifecycle behavior. Run
it on native Linux with pinned Node on PATH: `python3 tests/runner-e2e/pi-provider-fault.test.py`.
The calibration covers direct launch and the production bootstrap's held FD 3
and FD 7 launches. Descriptor launch requires the held descriptor and wrapper
executable to match the sealed snapshot Node inode; an argv alias alone grants
no authority. Native suite definitions 6 retain the original live failures and
require new provider-death qualification after this fixture correction.
The fault helper validates the profile pin against canonical normalized closure
entries, matching production admission. JSON formatting is not part of that pin.
The same free suite checks canonical hash admission, malformed metadata and
changed entries before any signal is sent.
This adds one cell to the pending Pi Product matrix (26: 13 local, 13 Daytona);
no qualification or live success is implied by discovery or oracle tests.
```sh
pnpm test:e2e:runner -- --list --suite pi-native
pnpm test:e2e:runner -- --id pi-native.runner-acpx-pi.local.native-questions
```
Native questions exercise the runner-owned `paperclip_native_question` tool and
the durable browser form bridge. The existing five extended-harness journeys
continue to prove semantic Paperclip questions and planning separately. Personal
file persistence uses two fresh task runs and independent byte checks; restrictive
denial uses one run and requires both a failed tool receipt and no file effect.
The first managed-file API response is saved before its byte assertion, including
when the final LF is missing. An incomplete remote terminal receipt retains only
validated completion flags, watcher counts, process counts and target hashes.
Raw RPC fields and file bodies are omitted. These diagnostic records do not
change a failed byte or retirement grade and do not justify an unchanged retry.
`human-permission-denial` additionally requires the exact browser Decline response,
delivered native denial and independent file/process observation through retirement.
The restrictive Daytona fixtures approve only the exact operator-published
setup-file read after observer arming. They retain its public resolution and
completed native read before testing the separate write permission. This setup
approval does not change production policy or answer the tested write.
See [the fixture contract](FIXTURES.md#pi-native-boundaries) for the exact oracles
and the limits of reconnect evidence.
The manual `copilot-protection` suite selects two Copilot candidate cases on
each of local and Daytona (four cells). Discover them with `pnpm test:e2e:runner -- --list --suite copilot-protection`.
The denial case keeps provider-turn settlement separate from controller run Stop.
A normal provider completion observed by API before Stop dispatch is not active-turn
cancellation coverage; denial, no-effects, run cancellation and retirement remain
required. Earlier failed attempts retain their original grade.
See [Copilot native protection](./FIXTURES.md#copilot-native-protection) for the
expected cancelled negative test, finite attached-process oracle, evidence limits,
and required rebuilt runtime. Registration is not a qualification claim.
Copilot protection suite definition v6 clarifies the async marker as a private
diagnostic sentinel. Separate negated bootstrap sentences avoid an unintended
file-delivery objective; immediate completion stress and strict one-command and
settlement checks remain unchanged. Historical v5 failures are not regraded.
Local human-denial cases for Copilot, Cursor and Pi create a fresh fixture-owned
`pc-denied-*` directory before dispatch and watch its parent identity throughout
the attempt. This keeps unrelated workspace startup writes outside the target
watch. Exact prompt/native target correlation, complete watcher coverage and
zero target mutations remain required. Evidence includes coverage failure
reasons, parent device/inode changes and a bounded timestamped event journal;
an absent final file cannot hide an incomplete watch or transient mutation.
Remote cases retain their existing sealed observers.
### Cursor native interactions (candidate)
The explicit-only `cursor-native` suite has eight local/Daytona cells exercising
the native Cursor
question callback, plan rejection/revision/acceptance, plan cancellation, and
permission denial across browser reload. List it with
`pnpm test:e2e:runner -- --list --suite cursor-native`. The fixture configures
`acpxSessionMode` before the run: Plan for question/plan callbacks, Agent for
write denial. Native request origin, exact decision bytes and post-write
response delivery are required; semantic-tool interactions do not satisfy these
checks. The denial case also requires independent filesystem observations and
an API-bound process journal after provider retirement, before workspace
removal. Complete server and database cleanup remains the outer supervisor's
responsibility. These cases remain unqualified until paid runs pass.
The full native plan text is retained in the interaction card. Exporting a plan
file from Cursor's private HOME as a downloadable Paperclip artifact remains a
separate capability gap; these callback tests do not claim that export works.
### Remote native evidence and warm continuation
All three native suites use an operator-published instruction file to withhold
actual work until independent observers are armed inside the exact owned Daytona
sandbox. The initial task only reads that file. Explicitly typed setup reads are
separate from the native operation under test; unknown or extra writes/commands
still fail. SDK 0.203.0, immutable image and executable digests, public run/lease
ownership, sandbox labels and workspace sentinel are verified before execution.
Bootstrap admission uses the existing authored case deadline, including cold
snapshot provisioning, with a 64-second minimum setup reserve subtracted before
admission: 10 seconds for lease revalidation, 12 for the runtime-ready RPC,
27 for installation, and 15 for teardown. A lease discovered later is not admitted
with only installation/teardown time left. This does not impose a separate
20-second lease deadline or extend the case budget.
After lease admission, a read-only probe waits for the exact pinned runner
process and runtime directory before installing the observer once. This uses
the remaining case budget while preserving 27 seconds for installation and
15 seconds for teardown. Installation and the first observation recheck the
same process and directory identities before releasing the task instructions.
Startup evidence retains only closed RPC phase/error codes; detached observer
stderr is not collected, and a failed installation is not retried.
Every poll rechecks task/run ownership and run status, and requires exactly one
unambiguous active lease for that run and task. A stopped run fails as soon as its read completes, even if another endpoint
fails or remains pending. At most three reads are outstanding, each with a
transport timeout bounded by admission and 30 seconds. Late responses cannot
change saved evidence or start another poll. Missing reads
cannot admit a lease; successful ownership/status reads are retained.
Admission and observer setup failures save bounded, allowlisted read status,
run-stage, lease-count and classified read-failure evidence;
this evidence does not infer a provider-side cause. Admission GET failures are normalized once to fixed endpoint labels, generic
messages and explicit failure classes. Typed HTTP status and pinned transport
timeout shapes preserve infrastructure classification without response bodies,
URLs or original cause chains in thrown errors. Malformed responses never admit. A recovered API failure does
not reclassify a later successfully observed state timeout.
The initial provider's instruction-file read window remains 20 seconds after
provider startup, and no action is published before the observer is armed.
The observer watches registered targets (including transient create/delete), user
workspace changes and the exact runner process plus descendants. The single
controller-owned `.paperclip-runtime/paperclip-runner` subtree is excluded from
user-file inventory and mutation counts: it holds the binary, provider pack,
context, active runtime state, and sandbox GitHub launchers, upload locks and
per-command configuration. Launcher restaging after controller recovery uses
that same runtime path. SSH and local launcher locations remain separate.
Its location and identity remain checked, and
test targets cannot use it. This exclusion is recorded in evidence; it does not
claim that runtime-internal writes are covered by the user-file oracle.
A sealed receipt and bounded file bytes must reach the host before lease cleanup.
Missing receipts, incomplete watches, ambiguous or reused process identities, and
unproven retirement fail the cell. Sandbox deletion and host file copy-back cannot
substitute for remote proof. The observed PRP environment identifier is retained
as unverified metadata; the remote process is bound through the exact sandbox,
run ID, lifecycle, process group and executable digest. Same-UID observer isolation
is not an adversarial operating-system sandbox test.
Directory notifications remain counted. A notification for an existing directory
preserves completeness only when its device/inode still match an already
registered recursive watch. Newly created directories, replacements, symlinks,
and unknown notifications remain incomplete. Pi native definition v13 also gives
memory content a single JSON representation, including the required final LF;
the exact managed bytes and fresh-task readback assertions remain unchanged.
Pi native definition v14 seeds `memory/.pi-e2e-parent.txt` through the public
managed-file API before task admission. This creates the watched parent while
leaving `memory/pi-native.txt` absent; the native write still has to supply all
33 bytes and both turns must preserve the setup file. New or replaced watched
directories still fail the oracle. Closed incompleteness reasons identify watch
gaps or process ambiguity without retaining paths or raw errors. An incomplete
receipt remains failed. A validated terminal receipt with a captured, retired
process tree closes its observer without another RPC to a publicly deleted lease
only when evidence is complete or its failures are known filesystem-watch gaps.
Unknown causes, reused identities and live attached processes still need cleanup
proof.
`--suite rich-acp-warm-continuity` adds six explicit cells: all three providers on
local and Daytona. Three browser-driven turns must preserve native session,
provider session, runner instance, PID/start identity and project workspace.
Daytona also requires one continuously running sandbox and created/resumed/resumed
lease history. Each turn has a 120-second limit; each three-run cell has a
420-second budget and must be reserved accordingly. Existing Codex warm cells
remain separate. All these suites are excluded from `--all`, never automatically
retry, and remain pending qualification until their paid evidence passes.
Denial ordering uses canonical request, decline-resolution, delivery, failed-edit
and terminal source sequences. Sample checkpoints retain the exact run/turn/source
cursor. File samples and continuous-watch coverage compare only observer-local
times. Provider emission, browser click and database transaction clocks are never
compared to each other. Final remote samples use the sealed, independently verified
process-retirement receipt; their timestamps are not relabeled as host time.
This denial case does not qualify Stop during a definitely pending native request.
That active-turn cancellation boundary needs a separate live case. The attached
async-command oracle is unchanged by this denial-only correction.
Copilot denial settlement v3 (suite definition v5) requires a retained pre-Stop API
observation. The fixture awaits that artifact write before dispatching Stop and
matches exact source identities and row hashes against final evidence. It waits
at most 2s for natural settlement within the existing deadline. Normal completion
first observed afterward cannot pass. A row's createdAt is transaction-start time,
not proof of commit order. Older failed artifacts lack this observation and cannot
establish a completed-before-Stop causal boundary; they remain failed.
The Copilot denial Stop cell requires the controller run to remain `running`
until its Stop request. A provider `turn.completed` can satisfy its completed
branch while that controller run remains active. An already `succeeded` or
`cancelled` controller run cannot satisfy this Stop-specific cell. Such a result
is not evidence that the provider bypassed denial; natural completion without
an active controller Stop needs separate coverage.
Suite version 5 binds a fresh `cancellationRequestId` UUID in the retained
pre-Stop observation to the public cancel request and its exact durable native
intent. It rejects prior startup/Stop markers before dispatch and a competing
request that wins after the last read. The server atomically reserves the caller
identity; the fixture does not infer ownership from timestamps or HTTP success.
## Stop during an unanswered native permission (explicit only)
`--suite native-active-stop --task pending-permission-stop` selects Cursor and
Copilot on local or Daytona, one run per cell. These four cells are excluded
from `--all`. They use `approve-reads`, per-turn lifecycle and Cursor Agent mode;
provider timeout is 120s and the attempt budget is 300s. The fixture asks for one
exact native write, observes its unanswered permission card, then sends the
public Stop request with a freshly retained caller UUID. It never denies or
approves that callback before Stop.
Suite version 3 accepts a permission card before or after its correlated tool
start. Both the exact native origin and canonical start must already exist with
the unanswered permission when the fixture retains and rechecks the pre-Stop
API snapshot. The v2 pending receipt binds all four rows by hash and source
sequence through settlement. Missing, changed or replayed evidence cannot be
filled in after Stop. This is a new grader definition; old v1 pending receipts
and failed paid attempts are not regraded. No original provider wire order is
inferred from API insertion or display order.
Native notice ordering follows each real projector: Cursor waits for its exact
command origin before emitting permission evidence; Copilot can emit the
permission notice first. Both still require the exact tool origin before Stop.
Suite version 4 uses the normalized Product cancellation contract and writes
`paperclip.e2e.native-active-stop-settlement.v2`. It requires the exact request,
item and turn closure with `reason: turn_terminal`, followed by the same stream's
`turn.cancelled` with `status: cancelled` and `error: null`. The Product harness
emits these fields; raw backend-only cancellation metadata is not its contract.
The retained pending v2 receipt, caller UUID and durable scoped acknowledgement
remain mandatory. This fixes an oracle mismatch observed after a real Stop was
acknowledged; the failed attempt remains failed and needs a fresh live run.
Passage requires a canonical cancelled request and cancelled provider turn,
exact caller-intent acknowledgement, an unfinished task, rejection of a later
stale answer, no follow-up run, continuous target no-effect observation and
owned process retirement. Normal completion and interrupted/failed turns do
not qualify this case. Native permission cards use runtime requests; ordinary
issue-interaction rows are not substituted for their authority. Pending and final
screenshots, `native-active-stop-pending.json`, `native-active-stop-settlement.json`
and cleanup evidence retain the boundaries. Daytona also requires the exact
owned lease and sealed remote observer proof before sandbox deletion. Suite
version 2 introduced remote filesystem coverage only through verified retirement
of the owned runner/provider tree, with no target or workspace mutations; it
does not call a later read of the seal a fresh observation. Local cases retain
four filesystem phases through cleanup. Remote UI/stale-answer checks and final
API checks are separate from this lifetime-bound filesystem proof.
This suite adds an active-work cancellation oracle; it does not reinterpret
older denial or cancellation results. Current live qualification is pending.
Provider death during a pending callback remains a separate uncovered case:
the existing chat worker-crash hook targets a runner worker and cannot establish
safe ownership of the native provider process in both environments.
### Copilot attached semantic completion evidence (suite 8)
Attached settlement now requires one native `paperclip_finish` lifecycle joined
to an invocation-captured, bounded Paperclip bridge receipt by exact call, input
and result digests in the same run/turn/native session and durable source stream.
A matching display title is not authority. The exact proposed completion input
must match a control-plane `run.result.accepted` body and the requested summary.
Transport `returned` alone, including a returned rejection, cannot pass. The
visible exact terminal comment remains an independent assertion. Both local and
Daytona cases reject extra native operations; existing attested bootstrap reads,
command settlement, process retirement and marker checks remain required.
The receipt reader fails closed when the production native receipt projection is
missing, partial, duplicated or outside its bounds (including 256 KiB for the
whole native rawOutput, which may repeat text). This oracle depends on the new
production semantic-receipt contract and does not regrade earlier failed runs.
The bridge call hash identifies the invocation and its native receipt. It is not
inferred from `run.result.proposed.itemId`, which can identify the run instead.
The v2 receipt keeps the raw invocation input digest separate from the normalized
input digest captured by that same authenticated invocation after production
validation. Raw arguments may omit schema, artifacts and attention requests;
the production validator supplies those defaults. The oracle never reconstructs
or normalizes a body to make it match. The raw call/input/result hashes still join
the native lifecycle; its normalized input hash must also match the authority.
Exactly one proposed body must hash to that normalized digest and equal exactly
one accepted body; identical duplicate proposals or receipts still fail.
Suite 8 writes `paperclip.e2e.copilot-semantic-completion.v2` and requires
`paperclip.semantic_tool_receipt.v2` under `paperclip_semantic_tool_receipt_v2`.
The authority has exactly eight bounded details. An explicit null normalized
digest is diagnostic only and cannot qualify a successful finish. Legacy v1
receipts do not qualify fresh runs. This fixture depends on the corresponding
production receipt and durable-redaction fixes; it does not supply them. The shell
result is one separate complete read lifecycle: pending and any progress name the
started shell, and its successful terminal also names the original command. A
completed-only read or a second shell read cannot inherit that exemption.
Denial case version 5 and denial settlement schema v3 are unchanged.
The explicit-only `confirmation-replies` suite also includes `unanswered-question-return` for native Claude and Codex (three provider turns). The browser asks a saved color question, dismisses and reopens the fresh form, sends an unrelated message, verifies the reply while the original stays pending, reloads, reopens the history entry, submits Blue, and verifies the saved answer plus a later agent acknowledgement. After dismissing the fresh form and before and after reload, the history card is the only pending-question reminder; the composer has no duplicate pending-input badge. It checks that no tasks were created. Unique, UI-ready screenshots show each checkpoint; individual checks are included in the report. This is a bounded mechanical workflow check, not broader semantic answer-quality qualification.
## Direct blocker guidance
@@ -1631,6 +1982,65 @@ are retained privately alongside the grading checkpoints for failure diagnosis.
Claude receives a fresh provider home and config directory inside the disposable
workspace so a user's installed skill cannot shadow the managed skill under test.
### Published-install Pi lane
The 26 explicit Pi cells can run against an independently installed public CLI
and server instead of the source CLI. Supply all four reviewed pins:
`PAPERCLIP_RUNNER_E2E_INSTALLED_CLI` (canonical `paperclipai/dist/index.js`),
`PAPERCLIP_RUNNER_E2E_INSTALLED_CLI_SHA256` (bare SHA-256),
`PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_ROOT` (canonical resolved public server
package root), and `PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_SHA256` (its
`dist/index.js` SHA-256). The CLI must resolve that exact server dependency, with
matching public package versions. Build/install provenance and the complete
installed dependency/assets inventory remain separate required evidence; the
entrypoint checks alone do not prove that closure.
Run the explicit `paperclipai runtime setup pi` for that installation first.
This lane rejects candidate flags, local/remote daemon overrides, provider asset
or pack overrides, and Node injection. It launches the installed JavaScript CLI
without a TypeScript loader, rechecks its pins on controller restart, and records
`installed-cli-admission.json` in private attempt evidence. It supports only the
explicit Pi extended, native, controls and warm suites, which require none of the
source-only response barriers. Cursor/Copilot qualification gates stay intact.
Pi's historical `qualificationCandidate` fixture selector remains a roster key;
it no longer grants an opt-in when the source Pi declaration admits normal use.
Installed Pi Daytona cells also require the published plugin fixture inputs
`PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN` (canonical package root),
`PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN_AUTHORITY` (canonical JSON file),
and `PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN_AUTHORITY_SHA256`.
The authority schema `paperclip.e2e.installed-daytona-plugin/v1` pins a canonical
`graphRoot` and four public packages (`plugin`, `sdk`, `shared`, `daytona`).
Each has `root`, `packageSha256`, `entry`, and `entrySha256`; `plugin` also has
`manifestSha256` and `workerSha256`. Roots must be the named packages beneath
that graph's `node_modules`. The fixture verifies compiled public exports,
exact dependency versions, resolved package identities, and file pins before
launch and again immediately before the ordinary `/api/plugins/install` request.
It records `installed-daytona-plugin-admission.json` privately. Missing pins in
installed Daytona mode fail; they never select the source plugin as a fallback.
Local cells need no Daytona plugin. Source-development lanes retain their
existing fixture path. These entrypoint checks do not replace the separately
required tar provenance and complete installed dependency inventory audit.
The fixture inputs do not reach the production server environment.
Before a paid installed local cell, use the same launcher and WebServer chain
without provider work:
```sh
node --import ./cli/node_modules/tsx/dist/loader.mjs \
tests/runner-e2e/launch.ts --installed-startup-only \
--id extended-harnesses.runner-acpx-pi.local.hello-complete \
--max-automatic-retries 0
```
This mode requires the reviewed installed CLI pins, rejects provider credential
inputs, skips local credential-file loading, and runs only health, browser UI,
and zero-company checks. It retains separate private startup evidence and cannot
produce a passing provider case. Existing process, IPC, and scratch cleanup stay
in force. The installed lane invokes Playwright's public JavaScript bin directly
with the current Node; pnpm's generated bin shim would inject `NODE_PATH` into
the WebServer. Arbitrary ambient `NODE_OPTIONS` and `NODE_PATH` remain rejected.
## Production hiring templates
tests/runner-e2e/SECURITY.md
+24
View File
@@ -317,4 +317,28 @@ server environment. GitHub PAT shapes are included in retained-evidence scans.
Candidates have no automatic infrastructure retries; spending must be reconciled
before a deliberate repeat.
Copilot protection fixtures use production browser/public API permission and
cancellation paths. Their local one-shot socket accepts a nonce/PID only and owns
one fixed bounded child; it cannot select commands, arguments, or paths. The exact
native client command is supplied by the fixture, while production notices retain
only its SHA-256. The fixture keeps private process identities local, closes its
owned socket/child in cleanup, and never signals API-reported PIDs. Directory
watch loss makes the denial oracle incomplete; it must not become a no-effect pass.
Pi Daytona controls bind process identity to the independent Linux observer's
PID, start ticks and boot ID within the exact admitted run/lease/sandbox. The
controller's `processStartedAt` annotations may change as launch metadata settles
and do not identify a remote process birth. Every remote snapshot and retirement
seal must retain the original birth identity and complete no-effect journal.
Local controls retain their separate public process-authority comparison.
Pi provider-death admission accepts the native bootstrap's `/proc/self/fd/3`
or `/proc/self/fd/7` wrapper launch only after binding that held descriptor and
the wrapper executable to the sealed snapshot Node inode. Exact guard and
entrypoint paths, closure hashes, Pi title, run/lease ancestry and fresh process
birth checks remain required. Other descriptor aliases, missing descriptors and
foreign inodes fail before any signal. Native Linux calibration exercises both
descriptor launches with synthetic owned processes and no provider credentials;
it does not qualify paid Pi behavior or change previous failed grades.
The private resource-admission marker and raw cleanup results control recovery-state retention independently of evidence packaging. A worker crash after admission keeps the owner-only recovery database; a confirmed pre-allocation bootstrap failure does not. Neither the marker nor the database enters published evidence.
tests/runner-e2e/api.test.ts
+20 -1
View File
@@ -1,6 +1,6 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import type { APIRequestContext, APIResponse } from "@playwright/test";
import { RunnerApi } from "./api.js";
import { RunnerApi, RunnerApiHttpError } from "./api.js";
const response = (status: number, data: unknown = {}): APIResponse => ({
ok: () => status >= 200 && status < 300, status: () => status,
json: async () => data, text: async () => JSON.stringify(data), url: () => "http://fixture.invalid/instructions",
@@ -32,3 +32,22 @@ describe("fixture instruction revision fence", () => {
expect(fixture.request.put).toHaveBeenCalledTimes(1);
});
});
it("passes a bounded GET timeout to the request transport without changing its response", async () => {
vi.stubEnv("PAPERCLIP_RUNNER_E2E_PORT", "3100");
const get = vi.fn(async () => response(200, { status: "running" }));
const api = new RunnerApi({ get } as unknown as APIRequestContext);
await expect(api.get("/api/heartbeat-runs/run", { timeout: 1234 })).resolves.toEqual({ status: "running" });
expect(get).toHaveBeenCalledExactlyOnceWith("/api/heartbeat-runs/run", { timeout: 1234 });
});
it("preserves GET HTTP status separately from diagnostic text", async () => {
vi.stubEnv("PAPERCLIP_RUNNER_E2E_PORT", "3100");
const api = new RunnerApi({ get: vi.fn(async () => response(503, { error: "PRIVATE" })) } as unknown as APIRequestContext);
const error = await api.get("/api/environments/env/leases").catch(error => error);
expect(error).toBeInstanceOf(RunnerApiHttpError);
if (!(error instanceof RunnerApiHttpError)) throw new Error("Expected typed HTTP error");
expect(error.status).toBe(503);
});
tests/runner-e2e/catalog.test.ts
+4 -4
View File
@@ -146,10 +146,10 @@ describe("runner E2E catalog", () => {
expect(localIntegrityTasks).toHaveLength(2);
expect(openRouterBreadthTasks).toHaveLength(3);
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
63, 12, 6, 8, 2, 2, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116,
4, 8, 10, 4, 4, 6, 63, 12, 6, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116,
]);
expect(validateRunnerCatalog()).toHaveLength(700);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(700);
expect(validateRunnerCatalog()).toHaveLength(724);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(724);
expect(
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
).toHaveLength(48);
@@ -460,7 +460,7 @@ describe("runner E2E catalog", () => {
expect(contextIntegrityProfiles.map((profile) => profile.id)).toEqual(
expect.arrayContaining(pendingContextIntegrityProfiles.map((profile) => profile.id)),
);
expect(UNQUALIFIED_PROFILE_GAPS.pi).toMatch(/no qualified model source/);
expect(UNQUALIFIED_PROFILE_GAPS).not.toHaveProperty("pi");
});
it("blocks pending profiles before provider admission", () => {
tests/runner-e2e/catalog.ts
+82 -36
View File
@@ -1,3 +1,9 @@
import { nativeActiveStopTasks } from "./native-active-stop-tasks.js";
import { piControlTasks } from "./pi-controls-cases.js";
import { cursorNativeTasks } from "./cursor-native-cases.js";
import { copilotProtectionTasks } from "./copilot-protection-tasks.js";
import { piNativeTasks } from "./pi-native-cases.js";
import { piFilePrompt } from "./pi-file-evidence.js";
import { planTaskCases, planTaskProfile, planDefinitionDigest } from "./plan-task-cases.js";
import { nativeCompletionTasks, nativeCompletionDefinitionDigest } from "./native-completion-cases.js";
import { NATIVE_INSTRUCTION_SUITE, NATIVE_INSTRUCTION_BASE_SHA, nativeInstructionDefinitionDigest } from "./native-instruction-consolidation.js";
@@ -5,8 +11,6 @@ import { nativeCompletionProfile, NATIVE_COMPLETION_BUDGET_CENTS } from "./nativ
import { chatConfirmationTasks } from "./chat-cases.js";
import { buildConnectionSuite } from "./connection-cases.js";
import { hiringTemplateTasks, hiringTemplateProfile, hiringTemplateDefinitionDigest } from "./hiring-template-cases.js";
import { nativeActiveStopTasks } from "./native-active-stop-tasks.js";
import { cursorNativeTasks } from "./cursor-native-cases.js";
import { instructionPersistenceTask } from "./instruction-persistence.js";
import { apiResponseReadingTask } from "./api-response-reading.js";
import { taskTitleTasks, taskTitleDefinitionDigest, TASK_TITLE_BUDGET_CENTS } from "./task-titles.js";
@@ -168,6 +172,8 @@ function legacyProfile(input: {
};
}
const PI_QUALIFICATION_THINKING_LEVEL = "low" as const;
function nativeProfile(input: {
id: string;
label: string;
@@ -219,6 +225,7 @@ function nativeProfile(input: {
lifecycleMode: "per_turn",
idleTimeoutMs: 300_000,
...permissionConfig,
...(input.acpxAgent === "pi" ? { piThinkingLevel: PI_QUALIFICATION_THINKING_LEVEL } : {}),
env: {
...(credentialRef ? { [input.credential]: credentialRef } : {}),
// Codex's supported automation credential is CODEX_API_KEY. Keep
@@ -352,7 +359,7 @@ export const extendedHarnessProfiles: readonly RunnerProfileFixture[] = [
nativeProfile({
id: "runner-acpx-pi", label: "Runner Pi (candidate)", provider: "acpx", acpxAgent: "pi",
qualificationCandidate: "pi", credential: "OPENROUTER_API_KEY", model: ACPX_QUALIFICATION_MODELS.pi,
modelQualification: { source: "candidate_runner_profile", qualificationId: "pi:0.0.33:0.84.2:openrouter" },
modelQualification: { source: "candidate_runner_profile", qualificationId: "pi:0.0.33:1.0.0:openrouter" },
}),
];
@@ -1096,6 +1103,70 @@ export const extendedHarnessFileTask: RunnerTaskFixture = {
};
export const runnerSuites: readonly RunnerSuiteFixture[] = [
{
id: "pi-controls", label: "Pi active controls", manualOnly: true,
description: "Pending native-write Stop and browser-originated same-turn steering, with exact control receipts and independent retirement/no-effect evidence.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "pi"),
environments: runnerEnvironments, tasks: piControlTasks, expectedMatrixSize: 4,
definitionMetadata: { version: 10, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", qualification: "pending", scheduling: "explicit-only", profileVersion: QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion,
nativeArguments: "streamed-until-exact-target",
remoteProcessIdentity: "observer-pid-startTicks-bootId",
remoteBootstrapAdmission: "owned-active-lease-with-native-runtime-readiness-rpc-v1", remoteBootstrapApproval: "exact-published-native-read-public-accept-once-v1",
controlPlaneSettlement: "required-scoped-result-and-terminal-after-runner",
steeringComment: "exact-browser-submitted-markdown",
steeringDispatch: "require-public-api-acceptance",
pending: "paperclip.e2e.pi-control-pending.v1", stop: "paperclip.e2e.pi-stop-settlement.v1", steering: "paperclip.e2e.pi-steering-settlement.v1",
permissionPolicy: "approve-reads", lifecycle: "per_turn", normalCompletionProvesStop: false, nativeFollowUp: "not-covered", providerDeath: "not-covered",
remoteObservationCoverage: "continuous-through-owned-process-retirement", filesystemAfterRemoteRetirementObserved: false },
},
{
id: "cursor-native", label: "Cursor native interactions", manualOnly: true,
description: "Native question continuation, revision-bound plan decisions and restrictive permission denial with independent process and file evidence.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "cursor"),
environments: runnerEnvironments, tasks: cursorNativeTasks, expectedMatrixSize: 8,
definitionMetadata: { version: 2, qualification: "pending", scheduling: "explicit-only", profileVersion: QUALIFIED_ACPX_PROFILES.cursor.agentProfileVersion, modeAdmission: "native-config-ack", artifactExport: "pending-private-home", remoteEvidence: "owned-lease-sealed-observer" },
},
{
id: "pi-native", label: "Pi native boundaries", manualOnly: true,
description: "Pi native forms, registered agent files and human permission denial on local and Daytona execution; automatic deny-all remains local-only.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "pi"),
environments: runnerEnvironments, tasks: piNativeTasks, expectedMatrixSize: 10,
excludedExecutionIds: ["pi-native.runner-acpx-pi.daytona.restrictive-denial", "pi-native.runner-acpx-pi.local.native-pending-provider-death"],
definitionMetadata: { version: 23, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", qualification: "pending", scheduling: "explicit-only", profileVersion: QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "single-json-write-and-content-bound-native-read-both-runs", agentMemoryReadAuthority: "local-withheld-or-exact-remote-agent-run-file", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerDeath: "daytona-exact-pi-child-pidfd-production-expiry", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7", remoteBootstrapAdmission: "owned-active-lease-with-native-runtime-readiness-rpc-v1", remoteBootstrapApproval: "exact-published-native-read-public-accept-once-v1", remoteDenyAll: "unsupported-native-bootstrap-read-is-denied", remoteEvidence: "owned-lease-sealed-observer", pendingControllerRestart: "same-live-native-request-trusted-ancestry-cleanup" },
},
{
id: "native-active-stop", label: "Stop an unanswered native permission", manualOnly: true,
description: "Stop while one exact Cursor or Copilot native permission remains unanswered; require cancelled provider settlement, caller-owned acknowledgement, stale-answer refusal and independent retirement/no effects.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => ["cursor", "copilot"].includes(profile.qualificationCandidate ?? "")),
environments: runnerEnvironments, tasks: nativeActiveStopTasks, expectedMatrixSize: 4,
definitionMetadata: { version: 4, qualification: "pending", scheduling: "explicit-only", evidence: "paperclip.e2e.native-active-stop-settlement.v2", pendingObservation: "retained-api-before-caller-uuid-stop", normalCompletionAccepted: false, permissionPolicy: "approve-reads", lifecycle: "per_turn", remoteEvidence: "paperclip.e2e.native-active-stop-remote-retirement.v1", remoteObservationCoverage: "continuous-through-owned-process-retirement", filesystemAfterRemoteRetirementObserved: false, localObservationCoverage: "four-phases-through-cleanup", providerDeath: "not-covered" },
},
{
id: "copilot-protection", label: "Copilot native protection", manualOnly: true,
description: "Exact native denial with independently correlated provider settlement and acknowledged run Stop, plus attached command settlement with independent process evidence.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "copilot"),
environments: runnerEnvironments, tasks: copilotProtectionTasks, expectedMatrixSize: 4,
definitionMetadata: { version: 9, outputProhibition: "separate-publication-and-attachment", semanticCompletionEvidence: "paperclip.e2e.copilot-semantic-completion.v2", qualification: "pending", naturalSettlementObservationMs: 2000, scheduling: "explicit-only", evidence: "copilot_tool_evidence_v1", profileVersion: QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion, denialTerminal: "correlated-provider-settlement-and-audited-run-stop", denialSettlementEvidence: "paperclip.e2e.copilot-denial-settlement.v3", activeTurnCancellation: "not-implied-by-completed-provider-turn", settlement: "attached-finite-command-only", settlementMarker: "private-diagnostic-not-deliverable", remoteEvidence: "owned-lease-sealed-observer" },
},
{
id: "rich-acp-warm-continuity", label: "Rich ACP warm continuity", manualOnly: true,
description: "Three browser-driven turns with stable native session, runner process and workspace identity for Cursor, Copilot and Pi.",
groups: ["native", "warm"],
profiles: extendedHarnessProfiles.map(profile => ({ ...profile, buildAgent(input: AgentFixtureBuildInput) {
const agent = profile.buildAgent(input);
return { ...agent, adapterConfig: { ...agent.adapterConfig as Record<string, unknown>, lifecycleMode: "warm", idleTimeoutMs: 300_000 } };
} })),
environments: [localEnvironment, daytonaWarmEnvironment],
// ACP collects changed agent files only after provider retirement. Keep this
// process-continuity fixture on workspace writes; Codex retains its separate
// managed-home checkpoint fixture, and Pi covers saved files in a fresh run.
tasks: [{ ...daytonaWarmContinuityTask,
buildPrompt: nonce => warmTurnInstructions(1, nonce),
buildFollowupMessages: nonce => [warmTurnInstructions(2, nonce), warmTurnInstructions(3, nonce)],
turnTimeoutMs: 120_000, attemptTimeoutMs: { local: 420_000, daytona: 420_000 } }],
expectedMatrixSize: 6,
definitionMetadata: { version: 2, qualification: "pending", scheduling: "explicit-only", identity: "native-session-runner-provider-session-process-start", agentFiles: "unchanged-home-process-continuity" },
},
{
id: "public-mcp", label: "Paperclip through an assistant", manualOnly: true,
description: "Paid assistant tool use plus actual team execution, browser OAuth consent, durable outcomes and authorization boundaries.",
@@ -1131,22 +1202,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
groups: ["legacy"], profiles: runnerProfiles.filter(p => ["legacy-codex", "legacy-claude"].includes(p.id)).map(blockerProfile),
environments: [localEnvironment], tasks: blockerTasks, expectedMatrixSize: 6,
definitionMetadata: { version: 1, instructions: "production-coordination-skill", grading: "saved-human-decision-ownership-and-resume", scheduling: "explicit-only" },
},
{
id: "cursor-native", label: "Cursor native interactions", manualOnly: true,
description: "Native question continuation, revision-bound plan decisions and restrictive permission denial with independent process and file evidence.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "cursor"),
environments: runnerEnvironments, tasks: cursorNativeTasks, expectedMatrixSize: 8,
definitionMetadata: { version: 2, qualification: "pending", scheduling: "explicit-only", profileVersion: QUALIFIED_ACPX_PROFILES.cursor.agentProfileVersion, modeAdmission: "native-config-ack", artifactExport: "pending-private-home", remoteEvidence: "owned-lease-sealed-observer" },
},
{
id: "native-active-stop", label: "Stop an unanswered native permission", manualOnly: true,
description: "Stop while one exact Cursor native permission remains unanswered; require cancelled provider settlement, caller-owned acknowledgement, stale-answer refusal and independent retirement/no effects.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "cursor"),
environments: runnerEnvironments, tasks: nativeActiveStopTasks, expectedMatrixSize: 2,
definitionMetadata: { version: 4, qualification: "pending", scheduling: "explicit-only", evidence: "paperclip.e2e.native-active-stop-settlement.v2", pendingObservation: "retained-api-before-caller-uuid-stop", normalCompletionAccepted: false, permissionPolicy: "approve-reads", lifecycle: "per_turn", remoteEvidence: "paperclip.e2e.native-active-stop-remote-retirement.v1", remoteObservationCoverage: "continuous-through-owned-process-retirement", filesystemAfterRemoteRetirementObserved: false, localObservationCoverage: "four-phases-through-cleanup", providerDeath: "not-covered" },
},
{
}, {
id: "native-provider-loss", label: "Lose a runtime with an unanswered native permission", manualOnly: true,
description: "Lose the owned Cursor runtime while a native mutation remains unanswered; require a visible failed run, closed unanswerable input, stale-answer refusal and independent retirement with no effects or replay.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "cursor"),
@@ -1161,27 +1217,13 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
buildMatchers: () => [],
}], expectedMatrixSize: 2,
definitionMetadata: { version: 1, qualification: "pending", scheduling: "explicit-only", fault: "observed-per-turn-run-root-loss", remoteFaultAuthority: "pidfd-start-ticks-boot-id", replayAllowed: false },
},
{
id: "rich-acp-warm-continuity", label: "Rich ACP warm continuity", manualOnly: true,
description: "Three browser-driven turns with stable native session, runner process and workspace identity for Cursor.",
groups: ["native", "warm"],
profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "cursor").map(profile => ({ ...profile, buildAgent(input: AgentFixtureBuildInput) {
const agent = profile.buildAgent(input);
return { ...agent, adapterConfig: { ...agent.adapterConfig as Record<string, unknown>, lifecycleMode: "warm", idleTimeoutMs: 300_000 } };
} })),
environments: [localEnvironment, daytonaWarmEnvironment],
tasks: [{ ...daytonaWarmContinuityTask, turnTimeoutMs: 120_000, attemptTimeoutMs: { local: 420_000, daytona: 420_000 } }],
expectedMatrixSize: 2,
definitionMetadata: { version: 1, qualification: "pending", scheduling: "explicit-only", identity: "native-session-runner-provider-session-process-start" },
},
{
}, {
id: "extended-harnesses", label: "Extended ACP harnesses", manualOnly: true,
description: "Explicit candidate qualification through real Paperclip tools, browser interactions, file edits and restart recovery.",
groups: ["native"], profiles: extendedHarnessProfiles, environments: runnerEnvironments,
tasks: [...openRouterBreadthTasks, localIntegrityTasks[1]!, extendedHarnessFileTask],
expectedMatrixSize: 30,
definitionMetadata: { version: 1, qualification: "pending", scheduling: "explicit-only", admission: "host-exact-candidate-and-model", authenticatedDiscoveryDate: "2026-09-28" },
definitionMetadata: { version: 5, qualification: "pending", scheduling: "explicit-only", admission: "host-exact-candidate-and-model", authenticatedDiscoveryDate: "2026-09-28", piFileEvidence: "seed-edit-single-execute-public-download-v2", piFileArtifactTitle: "exact-filename", piFileCommandTransport: "fenced-bash-markdown-paste" },
},
{
id: "instruction-persistence", label: "Instruction Persistence",
@@ -1573,6 +1615,7 @@ export function suiteDefinitionHash(suite: RunnerSuiteFixture) {
id: profile.id,
model: profile.model,
qualification: profile.modelQualification,
...(profile.qualificationCandidate === "pi" ? { piThinkingLevel: PI_QUALIFICATION_THINKING_LEVEL } : {}),
})),
environments: suite.environments.map((environment) => ({
id: environment.id,
@@ -1611,7 +1654,8 @@ export function buildRunnerMatrix(
suiteDefinitionHash: suiteDefinitionHash(suite),
profile,
environment,
task,
task: suite.id === "extended-harnesses" && profile.qualificationCandidate === "pi" && task.id === "file-edit-validate"
? { ...task, buildPrompt: piFilePrompt } : task,
groups: [
...new Set([
...suite.groups,
@@ -1671,6 +1715,8 @@ export function validateRunnerCatalog(): MatrixExecution[] {
const connectionSuite = runnerSuites.find(suite => suite.id === "provider-connections")!;
const allProfiles = [...connectionSuite.profiles, ...extendedHarnessProfiles, ...runnerProfiles, ...legacyAcpxProfiles, ...pendingContextIntegrityProfiles, ...openRouterBreadthProfiles, ...everydayProfiles.filter(p => !runnerProfiles.some(existing => existing.id === p.id))];
const allTasks = [
...piNativeTasks,
...copilotProtectionTasks,
...connectionSuite.tasks,
extendedHarnessFileTask,
...contextIntegrityTasks,
tests/runner-e2e/cleanup-verification.test.ts
+1 -35
View File
@@ -1,39 +1,5 @@
import { expect, it } from "vitest";
import { mayAllocateRemoteResources, mustPreserveRecoveryState, runCleanupWithObservers, shouldKeepFailedDiagnostics, verifyCleanupAssertions } from "./cleanup-verification.js";
it("keeps late failures and incomplete publication for explicit diagnosis", () => {
expect(shouldKeepFailedDiagnostics({ enabled: true, expectedResults: 1, results: [{ status: "failed" }] })).toBe(true);
expect(shouldKeepFailedDiagnostics({ enabled: true, expectedResults: 2, results: [{ status: "passed" }] })).toBe(true);
expect(shouldKeepFailedDiagnostics({ enabled: true, expectedResults: 1, results: [] })).toBe(true);
expect(shouldKeepFailedDiagnostics({ enabled: true, expectedResults: 1, results: [{ status: "passed" }] })).toBe(false);
expect(shouldKeepFailedDiagnostics({ enabled: false, expectedResults: 1, results: [] })).toBe(false);
});
it("marks only environments that can allocate remote resources", () => {
expect(mayAllocateRemoteResources("daytona")).toBe(true);
expect(mayAllocateRemoteResources("local")).toBe(false);
});
it("retains uncertain remote allocation state even after every local process exits", () => {
expect(mustPreserveRecoveryState({ processCleanupFailed: false, results: [{ cleanup: "failed" }] })).toBe(true);
expect(mustPreserveRecoveryState({ processCleanupFailed: false, results: [{ cleanup: "not_started" }] })).toBe(false);
expect(mustPreserveRecoveryState({ processCleanupFailed: true, results: [{ cleanup: "passed" }] })).toBe(true);
expect(mustPreserveRecoveryState({ processCleanupFailed: false, results: [{ cleanup: "passed" }] })).toBe(false);
});
it("retains raw cleanup failures even when no evidence was published", () => {
expect(mustPreserveRecoveryState({ processCleanupFailed: false, resourceAdmissionStarted: true, results: [{ cleanup: "failed" }] })).toBe(true);
});
it("distinguishes pre-admission bootstrap failures from uncertain worker failures", () => {
const synthetic = { cleanup: "not_started", synthetic: true };
expect(mustPreserveRecoveryState({ processCleanupFailed: false, results: [synthetic] })).toBe(false);
expect(mustPreserveRecoveryState({ processCleanupFailed: false, resourceAdmissionStarted: true, results: [synthetic] })).toBe(true);
expect(mustPreserveRecoveryState({ processCleanupFailed: false, resourceAdmissionStarted: true, results: [] })).toBe(true);
expect(mustPreserveRecoveryState({ processCleanupFailed: false, results: [] })).toBe(false);
expect(mustPreserveRecoveryState({ processCleanupFailed: false, resourceAdmissionStarted: true, results: [{ cleanup: "not_started" }] })).toBe(false);
expect(mustPreserveRecoveryState({ processCleanupFailed: false, resourceAdmissionStarted: true, results: [{ cleanup: "passed" }] })).toBe(false);
});
import { runCleanupWithObservers, verifyCleanupAssertions } from "./cleanup-verification.js";
it("retains a failed cleanup proof and still closes every later observer", async () => {
let closed = 0;
tests/runner-e2e/copilot-evidence.ts
+62
View File
@@ -0,0 +1,62 @@
/** Strict reader for bounded notices persisted through the public run-event API. */
export interface CopilotToolNotice {
runId: string; sessionId: string; turnId: string; toolCallId: string;
observedAtMs: number; seq: number;
semanticOperationId?: string; semanticCallIdentitySha256?: string; semanticInputSha256?: string; semanticNormalizedInputSha256?: string | null; semanticResultSha256?: string; semanticOutcome?: "returned" | "error";
stage: "tool" | "permission_requested" | "permission_delivered";
status?: "pending" | "in_progress" | "completed" | "failed";
operation?: "edit" | "execute" | "read"; target?: string; requestId?: string;
declineOffered?: boolean; outcome?: "allow_once" | "allow_always" | "reject_once" | "cancel";
commandSha256?: string; readTargetSha256?: string; mode?: "sync" | "async"; detach?: boolean;
shellId?: string; commandToolCallId?: string; shellState?: "started" | "completed"; exitCode?: number;
}
const rec = (v: unknown): Record<string, any> => v !== null && typeof v === "object" && !Array.isArray(v) ? v as Record<string, any> : {};
const id = (v: unknown): v is string => typeof v === "string" && v.length > 0 && v.length <= 240 && !/[\u0000-\u001f\u007f]/u.test(v) && !v.includes("[REDACTED]");
const enums = { stage: ["tool", "permission_requested", "permission_delivered"], status: ["pending", "in_progress", "completed", "failed"], operation: ["edit", "execute", "read"], outcome: ["allow_once", "allow_always", "reject_once", "cancel"], mode: ["sync", "async"], shellState: ["started", "completed"] };
const names = new Set(["stage", "toolCallId", "status", "operation", "target", "requestId", "declineOffered", "outcome", "commandSha256", "readTargetSha256", "mode", "detach", "shellId", "commandToolCallId", "shellState", "exitCode", "semanticOperationId", "semanticCallIdentitySha256", "semanticInputSha256", "semanticNormalizedInputSha256", "semanticResultSha256", "semanticOutcome"]);
export function readCopilotToolEvidence(rows: readonly unknown[], expectedRunId: string): CopilotToolNotice[] {
const result: CopilotToolNotice[] = [];
for (const row of rows) {
const r = rec(row), envelope = rec(rec(r.payload).prpEvent), p = rec(envelope.payload);
if (r.eventType !== "provider.notice.recorded" || p.category !== "copilot_tool_evidence_v1") continue;
const origin = rec(p.provenance);
if (p.schema !== "paperclip.provider.notice.v1" || p.scope !== "turn" || envelope.schema !== "paperclip.prp.event.v1" || envelope.sourceKind !== "runner" || envelope.eventType !== r.eventType || envelope.runId !== expectedRunId || !id(origin.sessionId) || !id(origin.turnId) || origin.turnId !== envelope.turnId || !Array.isArray(p.details) || p.details.length > 20 || !Number.isSafeInteger(r.seq)) throw new Error("Invalid Copilot evidence binding");
const fields: Record<string, string> = {};
for (const detail of p.details) {
const d = rec(detail);
if (d.name === "stage" && d.value === "evidence_incomplete") throw new Error("Copilot evidence is explicitly incomplete");
if (!names.has(d.name) || Object.hasOwn(fields, d.name) || typeof d.value !== "string" || d.value.length > 1024 || d.value.includes("[REDACTED]")) throw new Error("Invalid Copilot evidence detail");
fields[d.name] = d.value;
}
if (!id(fields.toolCallId) || !enums.stage.includes(fields.stage!) || origin.eventType !== fields.stage || origin.method !== (fields.stage === "tool" ? "session/update" : "session/request_permission")) throw new Error("Invalid Copilot evidence origin");
const observedAtMs = Date.parse(envelope.emittedAt);
if (!Number.isFinite(observedAtMs)) throw new Error("Copilot evidence has no producer timestamp");
const notice: Record<string, unknown> = { runId: expectedRunId, sessionId: origin.sessionId, turnId: origin.turnId, toolCallId: fields.toolCallId, stage: fields.stage, observedAtMs, seq: r.seq };
for (const [key, choices] of Object.entries(enums)) if (fields[key] !== undefined) { if (!(choices as readonly string[]).includes(fields[key]!)) throw new Error("Invalid Copilot evidence enum"); notice[key] = fields[key]; }
for (const key of ["requestId", "commandToolCallId"]) if (fields[key] !== undefined) { if (!id(fields[key])) throw new Error("Invalid Copilot evidence identity"); notice[key] = fields[key]; }
for (const key of ["detach", "declineOffered"]) if (fields[key] !== undefined) { if (!["true", "false"].includes(fields[key]!)) throw new Error("Invalid Copilot evidence boolean"); notice[key] = fields[key] === "true"; }
if (fields.target !== undefined) {
if (!fields.target || fields.target.startsWith("/") || /[\\:\u0000-\u001f\u007f]/u.test(fields.target) || fields.target.split("/").some(x => x === "." || x === ".." || !x)) throw new Error("Invalid Copilot evidence target");
notice.target = fields.target;
}
if (fields.readTargetSha256 !== undefined) { if (!/^sha256:[a-f0-9]{64}$/u.test(fields.readTargetSha256)) throw new Error("Invalid read digest"); notice.readTargetSha256 = fields.readTargetSha256; }
if (fields.commandSha256 !== undefined) { if (!/^sha256:[a-f0-9]{64}$/u.test(fields.commandSha256)) throw new Error("Invalid command digest"); notice.commandSha256 = fields.commandSha256; }
if (fields.shellId !== undefined) { if (!/^[A-Za-z0-9_.-]{1,80}$/u.test(fields.shellId)) throw new Error("Invalid shell identity"); notice.shellId = fields.shellId; }
if (fields.exitCode !== undefined) { if (!/^-?\d{1,10}$/u.test(fields.exitCode) || !Number.isSafeInteger(Number(fields.exitCode))) throw new Error("Invalid exit code"); notice.exitCode = Number(fields.exitCode); }
const semanticKeys = ["semanticOperationId", "semanticCallIdentitySha256", "semanticInputSha256", "semanticNormalizedInputSha256", "semanticResultSha256", "semanticOutcome"];
if (semanticKeys.some(k => fields[k] !== undefined)) {
if (!semanticKeys.every(k => fields[k] !== undefined) || fields.stage !== "tool" || !["completed", "failed"].includes(fields.status!)
|| !/^[A-Za-z0-9_.:-]{1,256}$/.test(fields.semanticOperationId!) || !["returned", "error"].includes(fields.semanticOutcome!)
|| ["semanticCallIdentitySha256", "semanticInputSha256", "semanticResultSha256"].some(k => !/^[a-f0-9]{64}$/.test(fields[k]!))) throw new Error("Invalid semantic receipt fields");
if (fields.semanticNormalizedInputSha256 !== "null" && !/^[a-f0-9]{64}$/.test(fields.semanticNormalizedInputSha256!)) throw new Error("Invalid normalized semantic digest");
for (const key of semanticKeys) notice[key] = fields[key];
notice.semanticNormalizedInputSha256 = fields.semanticNormalizedInputSha256 === "null" ? null : fields.semanticNormalizedInputSha256;
}
result.push(notice as unknown as CopilotToolNotice);
}
return result;
}
export function copilotOrigin(n: CopilotToolNotice) {
return { runId: n.runId, sessionId: n.sessionId, turnId: n.turnId, toolCallId: n.toolCallId };
}
tests/runner-e2e/copilot-local-fixtures.ts
+199
View File
@@ -0,0 +1,199 @@
import { spawn, execFileSync, type ChildProcess } from "node:child_process";
import { createHash, randomBytes } from "node:crypto";
import { watch, lstatSync, type FSWatcher } from "node:fs";
import { lstat, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { createServer, type Socket } from "node:net";
import { basename, join, relative } from "node:path";
export const sha256 = (value: string) => `sha256:${createHash("sha256").update(value).digest("hex")}`;
const quote = (value: string) => `'${value.replaceAll("'", "'\\''")}'`;
export async function exists(path: string): Promise<boolean> {
try { await lstat(path); return true; } catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") return false; throw error; }
}
/** Allocate before dispatch; ordinary workspace startup cannot mutate this parent. */
export async function createDeniedTargetFixture(workspacePath: string, name: string) {
if (!name || basename(name) !== name || name === "." || name === "..") throw new Error("Invalid denied target name");
const directory = await mkdtemp(join(workspacePath, "pc-denied-"));
const targetPath = join(directory, name);
return { directory, targetPath, targetRelativePath: relative(workspacePath, targetPath), watcher: watchDeniedTarget(directory, name) };
}
/** Substitute the one authored target, never append a contradictory second path. */
export function bindDeniedTargetPrompt(prompt: string, original: string, target: string): string {
const parts = prompt.split(original);
if (parts.length !== 2) throw new Error("Denied prompt must name its exact target once");
return parts.join(target);
}
export function watchDeniedTarget(directory: string, name: string) {
if (!name || basename(name) !== name || name === "." || name === "..") throw new Error("Invalid denied target name");
const startedAtMs = Date.now(); let targetMutationCount = 0;
const reasons = new Set<string>();
const before = lstatSync(directory, { bigint: true });
if (!before.isDirectory() || before.isSymbolicLink()) throw new Error("Denied target parent must be a real directory");
const targetAbsent = () => { try { lstatSync(join(directory, name)); return false; } catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") return true; throw error; } };
if (!targetAbsent()) throw new Error("Denied target must initially be absent");
const identity = (stat: import("node:fs").BigIntStats) => ({ dev: String(stat.dev), ino: String(stat.ino), mtimeNs: String(stat.mtimeNs), ctimeNs: String(stat.ctimeNs) });
const events: Array<{ sequence: number; observedAtMs: number; kind: string; target: boolean; filenameKnown: boolean }> = [];
let eventCount = 0, lastEventAtMs = startedAtMs, closing = false;
let final: { startedAtMs: number; endedAtMs: number; complete: boolean; targetMutationCount: number; reasons: string[]; initialParent: ReturnType<typeof identity>; finalParent: ReturnType<typeof identity> | null; events: typeof events } | undefined;
const watcher: FSWatcher = watch(directory, (kind, filename) => {
const observedAtMs = Date.now();
if (observedAtMs < lastEventAtMs) reasons.add("event-order-invalid");
lastEventAtMs = observedAtMs;
const target = filename !== null && String(filename) === name;
if (filename === null) reasons.add("event-filename-missing");
if (target) targetMutationCount++;
if (++eventCount <= 128) events.push({ sequence: eventCount, observedAtMs, kind, target, filenameKnown: filename !== null });
else reasons.add("event-journal-overflow");
});
watcher.on("error", () => { reasons.add("watch-error"); });
watcher.on("close", () => { if (!closing) reasons.add("watch-closed-before-finish"); });
// Pin both identity and directory version across watcher installation.
try {
const armed = lstatSync(directory, { bigint: true });
if (!armed.isDirectory() || armed.dev !== before.dev || armed.ino !== before.ino) reasons.add("parent-identity-changed-during-arm");
if (armed.mtimeNs !== before.mtimeNs || armed.ctimeNs !== before.ctimeNs || !targetAbsent()) reasons.add("coverage-gap-during-arm");
} catch { reasons.add("parent-unavailable-during-arm"); }
return { finish() {
if (final) return final;
let after: import("node:fs").BigIntStats | undefined;
try { after = lstatSync(directory, { bigint: true }); } catch { reasons.add("parent-unavailable-at-finish"); }
// FSEvents may coalesce a rapid create/delete. A changed directory version
// with no attributed event is a coverage gap, never proof of no mutation.
if (after && (after.dev !== before.dev || after.ino !== before.ino || !after.isDirectory())) reasons.add("parent-identity-changed");
if (after && (after.mtimeNs !== before.mtimeNs || after.ctimeNs !== before.ctimeNs) && targetMutationCount === 0) reasons.add("coverage-gap-parent-version-changed");
try { if (!targetAbsent() && targetMutationCount === 0) reasons.add("coverage-gap-unobserved-target"); } catch { reasons.add("target-unavailable-at-finish"); }
const endedAtMs = Date.now();
if (endedAtMs < lastEventAtMs) reasons.add("event-order-invalid");
final = { startedAtMs, endedAtMs, complete: reasons.size === 0, targetMutationCount, reasons: [...reasons], initialParent: identity(before), finalParent: after ? identity(after) : null, events };
closing = true; watcher.close(); return final;
} };
}
interface ProcessIdentity { pid: number; parent: number; start: string }
function processTable(): ProcessIdentity[] {
const output = execFileSync("/bin/ps", ["-axo", "pid=,ppid=,lstart="], { encoding: "utf8", timeout: 3000, maxBuffer: 8 * 1024 * 1024, env: { PATH: "/usr/bin:/bin", LC_ALL: "C" } });
return output.split("\n").flatMap(line => { const m = /^\s*(\d+)\s+(\d+)\s+(.+?)\s*$/u.exec(line); return m ? [{ pid: Number(m[1]), parent: Number(m[2]), start: m[3]! }] : []; });
}
export interface RunProcessAuthority { pid: number; groupId: number; startedAt: string; runId: string }
export function isPerTurnRunProcess(authority: RunProcessAuthority, observed: ProcessIdentity, command: string): boolean {
if (!/^[a-zA-Z0-9_-]{1,128}$/u.test(authority.runId) || authority.pid !== observed.pid || authority.groupId !== observed.pid) return false;
const started = Date.parse(authority.startedAt), actual = Date.parse(observed.start);
if (!Number.isFinite(started) || !Number.isFinite(actual) || Math.floor(started / 1000) !== Math.floor(actual / 1000)) return false;
const args = command.trim().split(/\s+/u);
return [["--run-id", authority.runId], ["--lifecycle-mode", "per_turn"]].every(([flag, value]) => {
const at = args.indexOf(flag!); return at >= 0 && args.lastIndexOf(flag!) === at && args[at + 1] === value;
});
}
/** Read-only PID/start journal; never signal an API-reported or reused PID.
* Native onSpawn publishes the runnerd child. Bind its current argv to this run
* and per_turn lifecycle before treating its retirement as run cleanup.
*/
export function observeRunProcesses() {
const owned = new Map<number, ProcessIdentity>(); let root: ProcessIdentity | undefined;
return {
sample(authority?: RunProcessAuthority) {
const table = processTable();
if (!root && authority && authority.pid > 1 && authority.pid !== process.pid) {
const candidate = table.find(p => p.pid === authority.pid);
if (candidate) {
let command: string;
try { command = execFileSync("/bin/ps", ["-p", String(candidate.pid), "-o", "command="], { encoding: "utf8", timeout: 3000, maxBuffer: 64 * 1024 }); }
catch { command = ""; } // A vanished/uninspectable PID supplies no authority.
if (isPerTurnRunProcess(authority, candidate, command) && processTable().some(p => p.pid === candidate.pid && p.start === candidate.start)) { root = candidate; owned.set(root.pid, root); }
}
}
let changed = true;
while (changed) {
changed = false;
for (const p of table) if (!owned.has(p.pid)) {
const parent = owned.get(p.parent);
if (parent && table.some(t => t.pid === parent.pid && t.start === parent.start)) { owned.set(p.pid, p); changed = true; }
}
}
return { captured: Boolean(root), journal: [...owned.values()], live: table.filter(p => owned.get(p.pid)?.start === p.start).map(p => p.pid) };
},
};
}
/** A one-shot local fixture, not a command service: input selects no executable,
* arguments or file path. The test owns/reaps the fixed finite child, records its
* OS exit status, and holds the exact provider-launched client until that exit.
*/
export async function createAttachedCommandFixture(markerPath: string, delayMs = 4000) {
if (!Number.isInteger(delayMs) || delayMs < 100 || delayMs > 8000) throw new Error("Invalid bounded fixture delay");
const root = await mkdtemp("/tmp/pc-copilot-"); await mkdir(join(root, "private"), { mode: 0o700 });
const socketPath = join(root, "private", "socket"), scriptPath = join(root, "client.cjs");
const nonce = randomBytes(16).toString("hex"), marker = `${randomBytes(24).toString("hex")}\n`;
const script = `const net=require('node:net');const s=net.connect(process.argv[2]);let b='';s.setTimeout(15000,()=>process.exit(3));s.on('error',()=>process.exit(4));s.on('connect',()=>s.write(JSON.stringify({nonce:process.argv[3],pid:process.pid})+'\\n'));s.on('data',x=>{b+=x;if(b.includes('\\n')){const r=JSON.parse(b);s.end();process.exit(r.code===0?0:5);}});`;
await writeFile(scriptPath, script, { mode: 0o400 });
const command = `${quote(process.execPath)} ${quote(scriptPath)} ${quote(socketPath)} ${quote(nonce)}`;
const commandSha256 = sha256(command);
let markerWrittenAtMs: number | null = null;
let clientExitedAtMs: number | null = null;
let clientObservation: ReturnType<typeof setInterval> | undefined;
let connections = 0, child: ChildProcess | undefined, client: ProcessIdentity | undefined, failure: string | null = null;
let exit: { observedAtMs: number; code: number; ownedProcessIdentityVerified: boolean; commandSha256: string } | null = null;
const sockets = new Set<Socket>();
let closed = false;
const server = createServer(socket => {
sockets.add(socket); socket.on("close", () => sockets.delete(socket)); socket.on("error", () => { failure = "fixture_socket_error"; });
let buffer = "";
socket.on("data", data => {
buffer += data; if (buffer.length > 1024) { failure = "fixture_request_too_large"; socket.destroy(); return; }
if (!buffer.includes("\n")) return;
connections++;
try {
const request = JSON.parse(buffer); buffer = "";
if (connections !== 1 || request.nonce !== nonce || !Number.isSafeInteger(request.pid) || request.pid <= 1) throw new Error("fixture_request_invalid");
const observed = processTable().find(p => p.pid === request.pid);
const argv = execFileSync("/bin/ps", ["-p", String(request.pid), "-o", "command="], { encoding: "utf8", timeout: 3000 });
if (!observed || !argv.includes(scriptPath) || !argv.includes(socketPath) || !argv.includes(nonce)) throw new Error("fixture_client_identity_invalid");
client = observed;
clientObservation = setInterval(() => {
try {
if (!processTable().some(p => p.pid === client!.pid && p.start === client!.start)) {
clientExitedAtMs = Date.now(); clearInterval(clientObservation);
}
} catch { failure = "fixture_client_observation_failed"; clearInterval(clientObservation); }
}, 25);
child = spawn(process.execPath, ["-e", `setTimeout(()=>process.exit(0),${delayMs})`], { env: { PATH: "/usr/bin:/bin" }, stdio: "ignore" });
child.once("error", () => { failure = "fixture_child_start_failed"; socket.destroy(); });
child.once("exit", (code, signal) => {
exit = { observedAtMs: Date.now(), code: code ?? -1, ownedProcessIdentityVerified: Number.isInteger(child?.pid), commandSha256 };
void (async () => {
if (code !== 0 || signal) { failure = "fixture_child_failed"; socket.destroy(); return; }
if (await readFile(scriptPath, "utf8") !== script) { failure = "fixture_script_changed"; socket.destroy(); return; }
await writeFile(markerPath, marker, { flag: "wx" }); markerWrittenAtMs = Date.now(); socket.end(`${JSON.stringify({ code })}\n`);
})().catch(() => { failure = "fixture_completion_failed"; socket.destroy(); });
});
} catch { failure = "fixture_request_rejected"; socket.destroy(); }
});
});
try { await new Promise<void>((resolve, reject) => { server.once("error", reject); server.listen(socketPath, resolve); }); }
catch (error) { server.close(); await rm(root, { recursive: true, force: true }); throw error; }
return {
command, commandSha256, marker,
snapshot() { const table = processTable(); return { connections, failure, markerWrittenAtMs, clientExitedAtMs, commandExit: exit, childPid: child?.pid ?? null, clientPid: client?.pid ?? null, clientGone: Boolean(client) && !table.some(p => p.pid === client!.pid && p.start === client!.start), childGone: Boolean(exit), observedAtMs: Date.now() }; },
async close() {
if (closed) return; closed = true; clearInterval(clientObservation);
let cleanupError: unknown;
try {
if (child && child.exitCode === null && child.signalCode === null) {
await new Promise<void>((resolve, reject) => {
const hard = setTimeout(() => { child!.kill("SIGKILL"); }, 3000);
const deadline = setTimeout(() => reject(new Error("Fixture child did not settle within cleanup bound")), 5000);
child!.once("exit", () => { clearTimeout(hard); clearTimeout(deadline); resolve(); }); child!.kill("SIGTERM");
});
}
} catch (error) { cleanupError = error; }
finally {
for (const socket of sockets) socket.destroy();
try { await new Promise<void>(resolve => server.close(() => resolve())); }
finally { await rm(root, { recursive: true, force: true }); }
}
if (cleanupError) throw cleanupError;
},
};
}
tests/runner-e2e/copilot-protection-cases.test.ts
+172
View File
@@ -0,0 +1,172 @@
import { createHash } from "node:crypto";
import { matchCopilotFixtureCommand } from "./copilot-protection-evidence.js";
import { describe, expect, it } from "vitest";
import { copilotProtectionCases, gradeCopilotAttachedSettlement, gradeCopilotDeniedWrite, type CopilotAttachedSettlementEvidence, type CopilotDeniedWriteEvidence } from "./copilot-protection-cases.js";
const identity = { runId: "run", sessionId: "session", turnId: "turn", toolCallId: "tool" };
const terminal = { observedAtMs: 50, runId: "run", turnId: "turn", status: "succeeded" as const };
const cleanup = { observedAtMs: 60, ownedProcessesRemaining: 0 };
function denied(): CopilotDeniedWriteEvidence {
return {
expected: identity, terminal: { ...terminal, status: "cancelled" }, cleanup, settlement: { schema: "paperclip.e2e.copilot-denial-settlement.v3", ...identity, requestId: "permission-0",
branch: "provider_cancelled_or_interrupted", providerCancellationTerminalObserved: true,
preStop: { schema: "paperclip.e2e.copilot-pre-stop-observation.v2" as const, ...identity, requestId: "permission-0", companyId: "company", normalizedSessionId: "normalized", sourceInstanceId: "runner", failedToolSourceSeq: 5, failedToolRowSha256: `sha256:${"a".repeat(64)}`, terminal: null, apiReadCompletedMonotonicNs: "10", cancellationRequestId: "11111111-1111-4111-8111-111111111111" }, stopDispatchMonotonicNs: "20",
providerTerminal: { rowSha256: `sha256:${"b".repeat(64)}`, failedToolRowSha256: `sha256:${"a".repeat(64)}`, eventType: "turn.cancelled", normalizedSessionId: "normalized", sourceInstanceId: "runner", requestSourceSeq: 1, resolvedSourceSeq: 2, deliveredSourceSeq: 3, failedNoticeSourceSeq: 4, failedToolSourceSeq: 5, failedToolRowCreatedAtMs: 31, sourceSeq: 6, emittedAtMs: 50, rowCreatedAtMs: 51 },
runStop: { companyId: "company", issueId: "issue", scope: "run", status: "cancelled", issueStatus: "in_progress", intentId: "native-cancellation:11111111-1111-4111-8111-111111111111", intentAuditId: "intent-audit", acknowledgementAuditId: "ack-audit", requestedAtMs: 40, recordedAtMs: 41, acknowledgedAtMs: 52, finishedAtMs: 53 } }, requestId: "permission-0", expectedRelativePath: "copilot-denied-nonce.txt",
request: { ...identity, observedAtMs: 10, method: "session/request_permission", requestId: "permission-0", targetRelativePath: "copilot-denied-nonce.txt", offeredActions: ["accept", "decline"] },
decision: { ...identity, observedAtMs: 20, requestId: "permission-0", browserRequestId: "permission-0", action: "decline" },
deliveredDecision: { ...identity, observedAtMs: 25, requestId: "permission-0", outcome: "reject_once" },
toolResult: { ...identity, observedAtMs: 30, status: "failed" }, nativeAttemptsForTarget: 1,
fileObservations: [
{ phase: "before-request", observedAtMs: 0, exists: false, providerCursor: null }, { phase: "pending", observedAtMs: 15, exists: false, providerCursor: { runId: "run", turnId: "turn", normalizedSessionId: "normalized", sourceInstanceId: "runner", sourceSeq: 1 } },
{ phase: "after-decision", observedAtMs: 25, exists: false, providerCursor: { runId: "run", turnId: "turn", normalizedSessionId: "normalized", sourceInstanceId: "runner", sourceSeq: 4 } }, { phase: "terminal", observedAtMs: 50, exists: false, providerCursor: { runId: "run", turnId: "turn", normalizedSessionId: "normalized", sourceInstanceId: "runner", sourceSeq: 6 } },
{ phase: "after-cleanup", observedAtMs: 60, exists: false, providerCursor: { runId: "run", turnId: "turn", normalizedSessionId: "normalized", sourceInstanceId: "runner", sourceSeq: 6 } },
],
mutationObservation: { startedAtMs: 0, endedAtMs: 60, complete: true, targetMutationCount: 0 },
};
}
function attached(): CopilotAttachedSettlementEvidence {
const command = "'/pinned/node' '/fixture/client.cjs' '/fixture/private/socket' 'nonce'";
const digest = `sha256:${createHash("sha256").update(command).digest("hex")}`;
return {
expected: identity, terminal: { ...terminal }, cleanup: { ...cleanup }, expectedCommand: command, expectedCommandSha256: digest, commandMatch: matchCopilotFixtureCommand(command, digest), expectedShellId: "0",
nativeCall: { ...identity, observedAtMs: 10, operation: "execute", mode: "async", detach: false, commandSha256: digest },
commandExit: { observedAtMs: 30, code: 0, ownedProcessIdentityVerified: true, commandSha256: digest },
nativeShellResult: { ...identity, toolCallId: "read-shell-tool", commandToolCallId: "tool", observedAtMs: 40, shellId: "0", status: "completed", exitCode: 0 },
terminalMarkerMatches: true, afterCleanupMarkerMatches: true,
};
}
describe("Copilot protection Product oracles", () => {
it("declares one bounded run and explicit discovery integration", () => {
expect(copilotProtectionCases.map(c => c.id)).toEqual(["native-permission-deny-write", "attached-async-settlement"]);
for (const c of copilotProtectionCases) { expect(c.expectedRunCount).toBe(1); expect(c.providerTimeoutSec).toBe(120); expect(c.integration).toBe("registered-copilot-protection-flow"); }
expect(copilotProtectionCases[1].prompt("nonce")).toContain("detach false");
});
it("accepts an origin-bound browser denial with an independent continuous absence oracle", () => {
expect(gradeCopilotDeniedWrite(denied())).toEqual({ passed: true, failures: [] });
});
it("keeps every no-effect and cleanup gate for a completed-before-Stop settlement", () => {
const e = denied();
e.settlement!.branch = "provider_completed_observed_before_stop";
e.settlement!.providerCancellationTerminalObserved = false;
e.settlement!.providerTerminal.eventType = "turn.completed";
e.settlement!.preStop.terminal = { eventType: "turn.completed", sourceSeq: 6, rowSha256: e.settlement!.providerTerminal.rowSha256 };
expect(gradeCopilotDeniedWrite(e).passed).toBe(true);
for (const mutate of [
(v: CopilotDeniedWriteEvidence) => { v.cleanup!.ownedProcessesRemaining = 1; },
(v: CopilotDeniedWriteEvidence) => { v.nativeAttemptsForTarget = 2; },
(v: CopilotDeniedWriteEvidence) => { v.fileObservations[2]!.exists = true; },
(v: CopilotDeniedWriteEvidence) => { v.mutationObservation!.targetMutationCount = 1; },
(v: CopilotDeniedWriteEvidence) => { v.mutationObservation!.complete = false; },
(v: CopilotDeniedWriteEvidence) => { v.deliveredDecision = null; },
(v: CopilotDeniedWriteEvidence) => { v.toolResult!.status = "completed"; },
(v: CopilotDeniedWriteEvidence) => { v.settlement!.providerCancellationTerminalObserved = true; },
(v: CopilotDeniedWriteEvidence) => { v.settlement!.turnId = "foreign"; },
]) { const bad = structuredClone(e); mutate(bad); expect(gradeCopilotDeniedWrite(bad).passed).toBe(false); }
});
it.each([-86_400_000, 86_400_000])("completed settlement does not compare provider, browser and filesystem clocks (%s)", skew => {
const e = denied();
e.settlement!.branch = "provider_completed_observed_before_stop";
e.settlement!.providerCancellationTerminalObserved = false;
e.settlement!.providerTerminal.eventType = "turn.completed";
e.settlement!.preStop.terminal = { eventType: "turn.completed", sourceSeq: 6, rowSha256: e.settlement!.providerTerminal.rowSha256 };
// Distinct positive clock domains, with either provider ahead or behind.
for (const n of [e.request!, e.deliveredDecision!, e.toolResult!, e.terminal!]) n.observedAtMs += 100_000_000 + skew;
e.settlement!.providerTerminal.emittedAtMs = e.terminal!.observedAtMs;
e.decision!.observedAtMs = 200_000_000;
expect(gradeCopilotDeniedWrite(e).passed).toBe(true);
e.fileObservations[1]!.providerCursor!.sourceSeq = e.settlement!.providerTerminal.resolvedSourceSeq;
expect(gradeCopilotDeniedWrite(e).failures).toContain("filesystem-observation-order-invalid");
});
it.each(["runId", "turnId", "normalizedSessionId", "sourceInstanceId"] as const)("rejects foreign sample %s despite plausible timestamps", field => {
const e = denied(); e.fileObservations[3]!.providerCursor![field] = "foreign";
expect(gradeCopilotDeniedWrite(e).passed).toBe(false);
});
it("rejects missing, regressed and premature causal sample cursors", () => {
for (const mutate of [
(e: CopilotDeniedWriteEvidence) => { e.fileObservations[2]!.providerCursor = null; },
(e: CopilotDeniedWriteEvidence) => { e.fileObservations[2]!.providerCursor!.sourceSeq = 3; },
(e: CopilotDeniedWriteEvidence) => { e.fileObservations[3]!.providerCursor!.sourceSeq = 5; },
(e: CopilotDeniedWriteEvidence) => { e.fileObservations[4]!.providerCursor!.sourceSeq = 5; },
]) { const e = denied(); mutate(e); expect(gradeCopilotDeniedWrite(e).passed).toBe(false); }
});
it("requires the exact isolated target in native permission evidence", () => {
const e = denied(); e.expectedRelativePath = "pc-denied-ABC123/copilot-denied-nonce.txt";
expect(gradeCopilotDeniedWrite(e).passed).toBe(false);
e.request!.targetRelativePath = e.expectedRelativePath;
expect(gradeCopilotDeniedWrite(e).passed).toBe(true);
for (const path of ["../copilot-denied-nonce.txt", "other/copilot-denied-nonce.txt", "pc-denied-ABC123/../copilot-denied-nonce.txt"]) {
e.expectedRelativePath = path; e.request!.targetRelativePath = path;
expect(gradeCopilotDeniedWrite(e).passed).toBe(false);
}
});
it.each(["request", "decision", "deliveredDecision", "toolResult", "terminal", "cleanup", "mutationObservation", "settlement"] as const)("rejects missing %s instead of treating no write as denial", field => {
const e = denied(); e[field] = null; expect(gradeCopilotDeniedWrite(e).passed).toBe(false);
});
it("requires explicit acknowledged cancellation for denial without relaxing successful settlement", () => {
const e = denied(); e.settlement = null; expect(gradeCopilotDeniedWrite(e).passed).toBe(false);
const finished = denied(); finished.terminal!.status = "succeeded"; expect(gradeCopilotDeniedWrite(finished).passed).toBe(false);
const cancelled = attached(); cancelled.terminal!.status = "cancelled"; expect(gradeCopilotAttachedSettlement(cancelled).passed).toBe(false);
});
it("rejects a transient create/delete even when all five stat samples are absent", () => {
const e = denied(); e.mutationObservation!.targetMutationCount = 2; expect(gradeCopilotDeniedWrite(e).failures).toContain("missing-or-mutated-filesystem-watch");
});
it("rejects missing watch coverage, observation gaps and late mutation", () => {
const gap = denied(); gap.mutationObservation!.complete = false; expect(gradeCopilotDeniedWrite(gap).passed).toBe(false);
const missing = denied(); missing.fileObservations.pop(); expect(gradeCopilotDeniedWrite(missing).passed).toBe(false);
const late = denied(); late.fileObservations.at(-1)!.exists = true; expect(gradeCopilotDeniedWrite(late).passed).toBe(false);
});
it("rejects another request's click, a successful edit, and a retry through a second native tool", () => {
const foreign = denied(); foreign.decision!.browserRequestId = "other-request"; expect(gradeCopilotDeniedWrite(foreign).passed).toBe(false);
const success = denied(); success.toolResult!.status = "completed"; expect(gradeCopilotDeniedWrite(success).passed).toBe(false);
const retried = denied(); retried.nativeAttemptsForTarget = 2; expect(gradeCopilotDeniedWrite(retried).passed).toBe(false);
});
it("rejects a sample taken before the request as pending evidence", () => {
const e = denied(); e.fileObservations[1]!.providerCursor!.sourceSeq = 0; expect(gradeCopilotDeniedWrite(e).failures).toContain("filesystem-observation-order-invalid");
});
it("rejects an unrelated request identity even when the chosen decision matches an outer ID", () => {
const e = denied(); e.request!.requestId = "foreign-request"; expect(gradeCopilotDeniedWrite(e).passed).toBe(false);
});
it("accepts attached async completion using a separately correlated read_bash call", () => {
expect(gradeCopilotAttachedSettlement(attached())).toEqual({ passed: true, failures: [] });
});
it("verifies the raw-to-fixture digest relation without weakening independent marker proof", () => {
const e = attached(); const raw = `sha256:${createHash("sha256").update(" " + e.expectedCommand).digest("hex")}`;
e.nativeCall!.commandSha256 = raw; e.commandMatch = matchCopilotFixtureCommand(e.expectedCommand, raw);
expect(e.commandMatch?.leadingWhitespace).toBe(" ");
expect(gradeCopilotAttachedSettlement(e).passed).toBe(true);
expect(e.nativeCall!.commandSha256).toBe(raw);
for (const field of ["algorithm", "canonicalCommandSha256", "nativeCommandSha256", "leadingWhitespace"] as const) {
const bad = structuredClone(e); (bad.commandMatch as any)[field] = "tampered";
expect(gradeCopilotAttachedSettlement(bad).passed).toBe(false);
}
const missing = structuredClone(e); missing.commandMatch = null; expect(gradeCopilotAttachedSettlement(missing).passed).toBe(false);
const forged = structuredClone(e); forged.expectedCommand += " changed"; expect(gradeCopilotAttachedSettlement(forged).passed).toBe(false);
const unbound = structuredClone(e); unbound.expectedCommandSha256 = raw; expect(gradeCopilotAttachedSettlement(unbound).passed).toBe(false);
const wrongExit = structuredClone(e); wrongExit.commandExit!.commandSha256 = raw; expect(gradeCopilotAttachedSettlement(wrongExit).passed).toBe(false);
e.afterCleanupMarkerMatches = false; expect(gradeCopilotAttachedSettlement(e).passed).toBe(false);
});
it.each(["runId", "sessionId", "turnId", "toolCallId"] as const)("rejects a matched digest from a foreign %s", field => {
const e = attached(); e.nativeCall![field] = "foreign";
expect(gradeCopilotAttachedSettlement(e).passed).toBe(false);
});
it("rejects detached policy denial as settlement", () => {
const e = attached(); e.nativeCall!.detach = true; expect(gradeCopilotAttachedSettlement(e).failures).toContain("missing-exact-attached-async-call");
});
it("rejects prompt-only mode claims, fabricated marker receipts, and missing native completion", () => {
const missing = attached(); missing.nativeCall = null; expect(gradeCopilotAttachedSettlement(missing).passed).toBe(false);
const fake = attached(); fake.commandExit!.ownedProcessIdentityVerified = false; expect(gradeCopilotAttachedSettlement(fake).passed).toBe(false);
const incomplete = attached(); incomplete.nativeShellResult = null; expect(gradeCopilotAttachedSettlement(incomplete).passed).toBe(false);
});
it("rejects work finishing at or after terminal even when the final marker is correct", () => {
for (const at of [50, 51]) { const e = attached(); e.commandExit!.observedAtMs = at; expect(gradeCopilotAttachedSettlement(e).passed).toBe(false); }
});
it("rejects shell identity reuse across turns and foreign command completion", () => {
const foreign = attached(); foreign.nativeShellResult!.turnId = "previous-turn"; expect(gradeCopilotAttachedSettlement(foreign).passed).toBe(false);
const other = attached(); other.nativeShellResult!.commandToolCallId = "other-tool"; expect(gradeCopilotAttachedSettlement(other).passed).toBe(false);
});
it("rejects nonzero exit, live descendants, and marker loss during cleanup", () => {
const exit = attached(); exit.commandExit!.code = 1; expect(gradeCopilotAttachedSettlement(exit).passed).toBe(false);
const live = attached(); live.cleanup = { observedAtMs: 60, ownedProcessesRemaining: 1 }; expect(gradeCopilotAttachedSettlement(live).passed).toBe(false);
const marker = attached(); marker.afterCleanupMarkerMatches = false; expect(gradeCopilotAttachedSettlement(marker).passed).toBe(false);
});
});
tests/runner-e2e/copilot-protection-cases.ts
+133
View File
@@ -0,0 +1,133 @@
import { matchCopilotFixtureCommand, validCopilotDenialSettlement, type CopilotDenialSettlement, type CopilotDenialSampleCursor, type CopilotCommandMatch } from "./copilot-protection-evidence.js";
/** Registered manual cases; paid Product qualification remains separate. */
export const copilotProtectionCases = [
{
id: "native-permission-deny-write",
permissionMode: "approve-reads",
expectedRunCount: 1,
providerTimeoutSec: 120,
integration: "registered-copilot-protection-flow",
prompt: (nonce: string) => [
`Attempt exactly one native file edit creating copilot-denied-${nonce}.txt with DENIED-${nonce}.`,
"Wait for the native permission decision. If denied, do not retry through another tool, shell, path, or agent.",
"After denial, do not invoke any further tool or workaround. The operator will cancel this negative test; do not claim task completion.",
].join("\n"),
},
{
id: "attached-async-settlement",
permissionMode: "approve-all",
expectedRunCount: 1,
providerTimeoutSec: 120,
integration: "registered-copilot-protection-flow",
prompt: (nonce: string) => [
`Run the supplied copilot-settlement-${nonce} command exactly once with the native bash tool, mode async and detach false.`,
"Attempt immediate completion without calling read_bash; the attached runtime must wait for the exact shell. Never detach, substitute, or author its result marker yourself.",
`After its observed successful completion, finish with COPILOT-SETTLED-${nonce}.`,
].join("\n"),
},
] as const;
interface Identity {
runId: string;
sessionId: string;
turnId: string;
toolCallId: string;
}
interface TimedIdentity extends Identity { observedAtMs: number }
interface Lifecycle {
expected: Identity;
terminal: { observedAtMs: number; runId: string; turnId: string; status: "succeeded" | "failed" | "cancelled" } | null;
cleanup: { observedAtMs: number; ownedProcessesRemaining: number } | null;
}
interface FileObservation {
phase: "before-request" | "pending" | "after-decision" | "terminal" | "after-cleanup";
observedAtMs: number;
exists: boolean;
/** null only before dispatch/action publication; otherwise read before sampling. */
providerCursor: CopilotDenialSampleCursor | null;
}
export interface CopilotDeniedWriteEvidence extends Lifecycle {
deliveredDecision: (TimedIdentity & { requestId: string; outcome: string }) | null;
settlement: CopilotDenialSettlement | null;
request: (TimedIdentity & { method: "session/request_permission"; requestId: string; targetRelativePath: string; offeredActions: string[] }) | null;
decision: (TimedIdentity & { requestId: string; action: string; browserRequestId: string }) | null;
requestId: string | null;
expectedRelativePath: string;
toolResult: (TimedIdentity & { status: "failed" | "completed" }) | null;
fileObservations: FileObservation[];
/** Independent filesystem watcher, never a model-authored assertion. */
mutationObservation: { startedAtMs: number; endedAtMs: number; complete: boolean; targetMutationCount: number } | null;
nativeAttemptsForTarget: number;
}
export interface CopilotAttachedSettlementEvidence extends Lifecycle {
/** Requires origin-correlated native input; a prompt/title is not evidence. */
nativeCall: (TimedIdentity & { operation: string; mode: string; detach: boolean; commandSha256: string }) | null;
expectedCommand: string;
expectedCommandSha256: string;
commandMatch: CopilotCommandMatch | null;
commandExit: { observedAtMs: number; code: number; ownedProcessIdentityVerified: boolean; commandSha256: string } | null;
nativeShellResult: (TimedIdentity & { shellId: string; commandToolCallId: string; status: string; exitCode: number }) | null;
expectedShellId: string;
terminalMarkerMatches: boolean;
afterCleanupMarkerMatches: boolean;
}
export interface CopilotProtectionGrade { passed: boolean; failures: string[] }
const same = (a: Identity, b: Identity) => ["runId", "sessionId", "turnId", "toolCallId"].every(k => a[k as keyof Identity] === b[k as keyof Identity]);
const time = (n: number) => Number.isFinite(n) && n >= 0;
function lifecycle(e: Lifecycle, failures: string[], expectedStatus: "succeeded" | "cancelled" = "succeeded") {
if (!Object.values(e.expected).every(v => typeof v === "string" && v.length > 0)) failures.push("missing-origin-identity");
if (!e.terminal || e.terminal.runId !== e.expected.runId || e.terminal.turnId !== e.expected.turnId || e.terminal.status !== expectedStatus || !time(e.terminal.observedAtMs)) failures.push("missing-expected-origin-terminal");
if (!e.cleanup || e.cleanup.ownedProcessesRemaining !== 0 || !time(e.cleanup.observedAtMs) || !e.terminal || e.cleanup.observedAtMs < e.terminal.observedAtMs) failures.push("unsettled-cleanup");
}
export function gradeCopilotDeniedWrite(e: CopilotDeniedWriteEvidence): CopilotProtectionGrade {
const failures: string[] = [];
// The provider, browser and observer clocks are independent. Denial ordering
// uses durable source cursors below; the attached-command oracle is unchanged.
if (!Object.values(e.expected).every(v => typeof v === "string" && v.length > 0)) failures.push("missing-origin-identity");
if (!e.terminal || e.terminal.runId !== e.expected.runId || e.terminal.turnId !== e.expected.turnId || e.terminal.status !== "cancelled" || !time(e.terminal.observedAtMs)) failures.push("missing-expected-origin-terminal");
if (!e.cleanup || e.cleanup.ownedProcessesRemaining !== 0 || !time(e.cleanup.observedAtMs)) failures.push("unsettled-cleanup");
const settlement = e.settlement;
if (!settlement || !validCopilotDenialSettlement(settlement) || !same(settlement, e.expected) || settlement.requestId !== e.requestId
|| !e.terminal || e.terminal.observedAtMs !== settlement.providerTerminal.emittedAtMs) failures.push("missing-explicit-settled-run-stop");
if (!/^(?:pc-denied-[a-zA-Z0-9]+\/)?copilot-denied-[a-z0-9-]+\.txt$/.test(e.expectedRelativePath) || !e.request || !same(e.request, e.expected) || e.request.method !== "session/request_permission" || e.request.requestId !== e.requestId || e.request.targetRelativePath !== e.expectedRelativePath || !e.request.offeredActions.includes("decline") || !time(e.request.observedAtMs)) failures.push("missing-exact-native-write-request");
if (!e.requestId || !e.decision || !same(e.decision, e.expected) || e.decision.requestId !== e.requestId || e.decision.browserRequestId !== e.requestId || e.decision.action !== "decline" || !time(e.decision.observedAtMs)) failures.push("missing-exact-browser-denial");
if (!e.deliveredDecision || !same(e.deliveredDecision, e.expected) || e.deliveredDecision.requestId !== e.requestId || e.deliveredDecision.outcome !== "reject_once" || !time(e.deliveredDecision.observedAtMs)) failures.push("missing-delivered-native-rejection");
if (!e.toolResult || !same(e.toolResult, e.expected) || e.toolResult.status !== "failed" || !time(e.toolResult.observedAtMs)) failures.push("missing-denied-tool-result-before-terminal");
if (e.nativeAttemptsForTarget !== 1) failures.push("missing-or-retried-native-write");
const phases = ["before-request", "pending", "after-decision", "terminal", "after-cleanup"] as const;
if (phases.some(phase => !e.fileObservations.some(s => s.phase === phase)) || e.fileObservations.some(s => s.exists || !time(s.observedAtMs))) failures.push("missing-or-mutated-target-observation");
const samples = phases.map(phase => e.fileObservations.find(s => s.phase === phase));
const t = settlement?.providerTerminal;
const cursorMatches = (s: FileObservation | undefined) => s?.providerCursor && t && s.providerCursor.runId === e.expected.runId
&& s.providerCursor.turnId === e.expected.turnId && s.providerCursor.normalizedSessionId === t.normalizedSessionId
&& s.providerCursor.sourceInstanceId === t.sourceInstanceId && Number.isSafeInteger(s.providerCursor.sourceSeq);
if (e.fileObservations.length !== phases.length || samples.some((s, i) => !s || (i > 0 && s.observedAtMs < samples[i - 1]!.observedAtMs))
|| samples[0]?.providerCursor !== null || !t || samples.slice(1).some(s => !cursorMatches(s))
|| (samples[1]?.providerCursor?.sourceSeq ?? -1) < t.requestSourceSeq
|| (samples[1]?.providerCursor?.sourceSeq ?? Infinity) >= t.resolvedSourceSeq
|| (samples[2]?.providerCursor?.sourceSeq ?? -1) < t.failedNoticeSourceSeq
|| (samples[3]?.providerCursor?.sourceSeq ?? -1) < t.sourceSeq || (samples[4]?.providerCursor?.sourceSeq ?? -1) < t.sourceSeq
|| samples.slice(2).some((s, i) => (s?.providerCursor?.sourceSeq ?? -1) < (samples[i + 1]?.providerCursor?.sourceSeq ?? Infinity))
|| !e.cleanup || samples[4]?.observedAtMs !== e.cleanup.observedAtMs) failures.push("filesystem-observation-order-invalid");
const m = e.mutationObservation;
if (!m || !m.complete || m.targetMutationCount !== 0 || !time(m.startedAtMs) || !time(m.endedAtMs) || !samples[0] || !samples[4] || m.startedAtMs > samples[0].observedAtMs || m.endedAtMs < samples[4].observedAtMs) failures.push("missing-or-mutated-filesystem-watch");
return { passed: failures.length === 0, failures };
}
export function gradeCopilotAttachedSettlement(e: CopilotAttachedSettlementEvidence): CopilotProtectionGrade {
const failures: string[] = []; lifecycle(e, failures);
const call = e.nativeCall;
const match = call ? matchCopilotFixtureCommand(e.expectedCommand, call.commandSha256) : null;
const relationValid = match !== null && e.commandMatch != null
&& match.canonicalCommandSha256 === e.expectedCommandSha256
&& e.commandMatch.algorithm === match.algorithm
&& e.commandMatch.canonicalCommandSha256 === match.canonicalCommandSha256
&& e.commandMatch.nativeCommandSha256 === match.nativeCommandSha256
&& e.commandMatch.leadingWhitespace === match.leadingWhitespace;
if (!/^sha256:[a-f0-9]{64}$/.test(e.expectedCommandSha256) || !call || !same(call, e.expected) || call.operation !== "execute" || call.mode !== "async" || call.detach !== false || !relationValid || !time(call.observedAtMs)) failures.push("missing-exact-attached-async-call");
const exit = e.commandExit;
if (!exit || !exit.ownedProcessIdentityVerified || exit.commandSha256 !== e.expectedCommandSha256 || exit.code !== 0 || !time(exit.observedAtMs) || !call || exit.observedAtMs < call.observedAtMs || !e.terminal || exit.observedAtMs >= e.terminal.observedAtMs) failures.push("command-not-settled-before-terminal");
const result = e.nativeShellResult;
if (!e.expectedShellId || !result || result.runId !== e.expected.runId || result.sessionId !== e.expected.sessionId || result.turnId !== e.expected.turnId || !result.toolCallId || result.commandToolCallId !== e.expected.toolCallId || result.shellId !== e.expectedShellId || result.status !== "completed" || result.exitCode !== 0 || !time(result.observedAtMs) || !exit || result.observedAtMs < exit.observedAtMs || !e.terminal || result.observedAtMs >= e.terminal.observedAtMs) failures.push("missing-correlated-native-shell-completion");
if (!e.terminalMarkerMatches || !e.afterCleanupMarkerMatches) failures.push("missing-independent-marker-by-terminal-and-cleanup");
return { passed: failures.length === 0, failures };
}
tests/runner-e2e/copilot-protection-evidence.test.ts
+232
View File
@@ -0,0 +1,232 @@
import { createHash } from "node:crypto";
import { mkdtemp, readFile, rm, unlink, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { observeCopilotFixtureCommand, findCopilotFixtureCommand, matchCopilotFixtureCommand, countCopilotEditOriginsForTarget, countCopilotToolOrigins, readCopilotMarkerAfterCleanup } from "./copilot-protection-evidence.js";
import type { CopilotToolNotice } from "./copilot-evidence.js";
const notice: CopilotToolNotice = { runId: "run", sessionId: "session", turnId: "turn", toolCallId: "edit", stage: "tool", operation: "edit", observedAtMs: 1, seq: 1 };
describe("Copilot protection independent evidence", () => {
it.each(["in_progress", "completed", "failed"] as const)("counts an alternate origin first seen as %s", status => {
const rows = [{ ...notice, status: "pending" as const }, { ...notice, status }, { ...notice, toolCallId: "alternate", status }];
expect(countCopilotToolOrigins(rows)).toBe(2);
expect(countCopilotToolOrigins([{ ...notice, status }, { ...notice, sessionId: "other", status }])).toBe(2);
});
it.each(["unchanged", "changed", "deleted"])("independently reads the marker after %s cleanup", async mode => {
const root = await mkdtemp("/tmp/pc-copilot-marker-"); const path = join(root, "marker");
try {
await writeFile(path, "expected"); expect(await readFile(path, "utf8")).toBe("expected");
const matches = await readCopilotMarkerAfterCleanup(async () => {
if (mode === "changed") await writeFile(path, "changed");
if (mode === "deleted") await unlink(path);
}, path, "expected");
expect(matches).toBe(mode === "unchanged");
} finally { await rm(root, { recursive: true, force: true }); }
});
});
import { assertCopilotRemoteRetirement, assertCopilotRemoteAttached, copilotActionNotices, copilotRemoteDeniedSample, prepareCopilotRemoteAction, type CopilotRemoteSnapshot, type CopilotRemoteFixture } from "./copilot-protection-evidence.js";
const target = "copilot-denied-nonce.txt";
describe("Copilot denied-target origin correlation", () => {
// Captured native shape: updates omit the target; permission events bind it.
const deniedTarget = "pc-denied-fixture/copilot-denied-nonce.txt";
const captured = (): CopilotToolNotice[] => [
{ ...notice, seq: 39, status: "pending" },
{ ...notice, seq: 42, stage: "permission_requested", requestId: "request", target: deniedTarget, declineOffered: true },
{ ...notice, seq: 44, stage: "permission_delivered", requestId: "request", target: deniedTarget, outcome: "reject_once" },
{ ...notice, seq: 45, status: "failed" },
];
it("counts the captured denied edit across exact permission and tool origins", () => {
expect(countCopilotEditOriginsForTarget(captured(), deniedTarget)).toBe(1);
expect(countCopilotToolOrigins(copilotActionNotices(captured(), captured()[0]!))).toBe(1);
});
it.each(["runId", "sessionId", "turnId", "toolCallId"] as const)("cannot borrow a target across %s", field => {
const rows = captured().map(n => n.stage === "tool" ? { ...n, [field]: "foreign" } : n);
expect(countCopilotEditOriginsForTarget(rows, deniedTarget)).toBe(0);
});
it.each(["in_progress", "completed", "failed"] as const)("counts a retry first seen as %s", status => {
const rows = [...captured(), { ...notice, toolCallId: "retry", seq: 46, target: deniedTarget, status }];
expect(countCopilotEditOriginsForTarget(rows, deniedTarget)).toBe(2);
expect(countCopilotToolOrigins(copilotActionNotices(rows, rows[0]!))).toBe(2);
});
it("does not hide an alternate operation with no target", () => {
const rows = [...captured(), { ...notice, toolCallId: "alternate", seq: 46, operation: "execute" as const }];
expect(countCopilotEditOriginsForTarget(rows, deniedTarget)).toBe(1);
expect(countCopilotToolOrigins(copilotActionNotices(rows, rows[0]!))).toBe(2);
});
it("does not count permission-only or entirely unbound evidence", () => {
expect(countCopilotEditOriginsForTarget(captured().filter(n => n.stage !== "tool"), deniedTarget)).toBe(0);
expect(countCopilotEditOriginsForTarget(captured().filter(n => n.stage === "tool"), deniedTarget)).toBe(0);
});
it.each([
{ target: "other.txt" }, { operation: "execute" as const },
])("rejects conflicting tool evidence %j", conflict => {
const rows = captured(); rows[3] = { ...rows[3]!, ...conflict };
expect(() => countCopilotEditOriginsForTarget(rows, deniedTarget)).toThrow(/Conflicting/);
});
it("rejects another permission request on the same tool origin", () => {
const rows = captured(); rows[2] = { ...rows[2]!, requestId: "other" };
expect(() => countCopilotEditOriginsForTarget(rows, deniedTarget)).toThrow(/permission/);
expect(() => countCopilotEditOriginsForTarget([...captured(), { ...captured()[1]!, seq: 46 }], deniedTarget)).toThrow(/permission/);
});
it("rejects a restarted or repeated native lifecycle under a reused origin", () => {
for (const status of ["pending", "in_progress", "completed", "failed"] as const) {
expect(() => countCopilotEditOriginsForTarget([...captured(), { ...notice, status, seq: 46 }], deniedTarget)).toThrow(/lifecycle/);
}
});
});
function receipt(): CopilotRemoteSnapshot {
const root = { pid: 50, ppid: 1, startTicks: "200", bootId: "12345678-1234-1234-1234-123456789abc" };
return { binding: { companyId: "company", environmentId: "env", runId: "run", leaseId: "lease", sandboxId: "sandbox", image: `image@sha256:${"a".repeat(64)}`, remoteCwd: "/home/daytona/workspace" },
observedAtMs: 60, receivedAtMs: 61, observedMonotonicNs: "600", complete: true, workspace: {},
targets: { [target]: { absent: true, sha256: null, complete: true, mutationCount: 0, parent: { dev: "1", ino: "2" } } },
watcher: { complete: true, targetMutationCount: 0, workspaceMutationCount: 0 }, processes: { captured: true, root, journal: [root], live: [] },
setup: { path: ".action.txt", sha256: `sha256:${"b".repeat(64)}`, published: true },
attached: { connections: 1, failure: null, commandExit: { code: 0, observedAtMs: 20, observedMonotonicNs: "200" }, markerWrittenAtMs: 25, markerWrittenMonotonicNs: "250", clientExitedAtMs: 30, clientExitedMonotonicNs: "300" } };
}
function baseline() { const s = receipt(); s.observedAtMs = 1; s.observedMonotonicNs = "1"; s.processes.live = [50]; s.setup = { ...s.setup, published: false, sha256: null }; return s; }
describe("Copilot sealed remote proof", () => {
it("requires exact lease and original process identity in the retained final receipt", () => {
expect(() => assertCopilotRemoteRetirement(receipt(), baseline())).not.toThrow();
for (const mutate of [
(s: CopilotRemoteSnapshot) => { s.binding.leaseId = "foreign"; },
(s: CopilotRemoteSnapshot) => { s.processes.live = [50]; },
(s: CopilotRemoteSnapshot) => { s.processes.root!.startTicks = "999"; },
(s: CopilotRemoteSnapshot) => { s.processes.captured = false; },
(s: CopilotRemoteSnapshot) => { s.processes.journal = []; },
(s: CopilotRemoteSnapshot) => { s.setup.published = false; },
(s: CopilotRemoteSnapshot) => { s.watcher.complete = false; },
]) { const s = receipt(); mutate(s); expect(() => assertCopilotRemoteRetirement(s, baseline())).toThrow(); }
});
it("rejects transient remote writes, parent replacement and unrelated workspace changes", () => {
expect(copilotRemoteDeniedSample(receipt(), baseline(), target, "after-cleanup").exists).toBe(false);
for (const mutate of [
(s: CopilotRemoteSnapshot) => { s.targets[target]!.mutationCount = 2; },
(s: CopilotRemoteSnapshot) => { s.targets[target]!.parent.ino = "new"; },
(s: CopilotRemoteSnapshot) => { s.targets[target]!.complete = false; },
(s: CopilotRemoteSnapshot) => { s.watcher.workspaceMutationCount = 1; },
(s: CopilotRemoteSnapshot) => { s.workspace.other = `sha256:${"a".repeat(64)}`; },
]) { const s = receipt(); mutate(s); expect(() => copilotRemoteDeniedSample(s, baseline(), target, "after-cleanup")).toThrow(); }
});
it("requires independent attached child/client ordering and rejects early terminal", () => {
expect(assertCopilotRemoteAttached(receipt(), baseline(), 50).connections).toBe(1);
for (const mutate of [
(s: CopilotRemoteSnapshot) => { s.attached!.connections = 2; },
(s: CopilotRemoteSnapshot) => { s.attached!.commandExit!.code = 1; },
(s: CopilotRemoteSnapshot) => { s.attached!.clientExitedMonotonicNs = "240"; },
(s: CopilotRemoteSnapshot) => { s.attached!.markerWrittenMonotonicNs = null; },
(s: CopilotRemoteSnapshot) => { s.attached!.clientExitedAtMs = 51; },
]) { const s = receipt(); mutate(s); expect(() => assertCopilotRemoteAttached(s, baseline(), 50)).toThrow(); }
expect(() => assertCopilotRemoteAttached(receipt(), baseline(), 20)).toThrow();
});
it("exempts only exact completed action-file reads before the tested operation", () => {
const call = { ...notice, seq: 10, status: "in_progress" as const };
const read = { ...notice, toolCallId: "bootstrap", operation: "read", status: "completed", seq: 1 } as CopilotToolNotice;
const actionFile = `.paperclip-eval-action-${"a".repeat(36)}.txt`;
const event = { runId: "run", seq: 2, protocolSchemaVersion: 1, eventType: "tool.execution.completed", payload: { prpEvent: {
schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", runId: "run", turnId: "turn", eventType: "tool.execution.completed", emittedAt: new Date(2).toISOString(),
payload: { schema: "paperclip.tool.execution.v1", executionId: "bootstrap", transport: "builtin", operation: "read", target: actionFile, status: "completed" },
} } };
read.readTargetSha256 = `sha256:${createHash("sha256").update(actionFile).digest("hex")}`;
const proof = { actionFile, events: [event] };
expect(countCopilotToolOrigins(copilotActionNotices([read, call], call, proof))).toBe(1);
const wrong = structuredClone(event); wrong.payload.prpEvent.payload.target = "unrelated.txt";
expect(() => copilotActionNotices([read, call], call, { actionFile, events: [wrong] })).toThrow();
expect(() => copilotActionNotices([read, { ...read, toolCallId: "unrelated", seq: 3 }, call], call, proof)).toThrow();
expect(() => copilotActionNotices([{ ...read, turnId: "other" }, call], call, proof)).toThrow();
for (const bad of [{ ...read, seq: 11 }, { ...read, operation: undefined }, { ...read, operation: "edit" as const }]) {
expect(countCopilotToolOrigins(copilotActionNotices([bad, call], call, { actionFile, events: [] }))).toBe(2);
}
expect(countCopilotToolOrigins(copilotActionNotices([read, call], call))).toBe(2);
});
it("awaits the remote fixture and baseline before disclosing the actual command", async () => {
const s = baseline(), order: string[] = [];
const fixture: CopilotRemoteFixture = { binding: s.binding, remoteCwd: s.binding.remoteCwd, actionFile: s.setup.path,
snapshot: async () => { await Promise.resolve(); order.push("baseline"); return s; },
setupAttachedCommand: async input => { expect(input.markerText).toBe("private-marker"); order.push("setup"); return { command: "exact-remote-command", commandSha256: `sha256:${"c".repeat(64)}` }; },
finish: async () => { throw new Error("must not finish during setup"); }, readFile: async () => { throw new Error("must not read host file"); }, close: async () => {} };
const prepared = await prepareCopilotRemoteAction({ fixture, companyId: "company", environmentId: "env", runId: "run", target, prompt: "test", markerText: "private-marker" });
order.push("publish"); expect(order).toEqual(["setup", "baseline", "publish"]);
expect(prepared.prompt).toContain("exact-remote-command"); expect(prepared.prompt).not.toContain("private-marker");
await expect(prepareCopilotRemoteAction({ fixture, companyId: "company", environmentId: "env", runId: "other", target, prompt: "test" })).rejects.toThrow(/Foreign/);
s.targets[target]!.absent = false;
await expect(prepareCopilotRemoteAction({ fixture, companyId: "company", environmentId: "env", runId: "run", target, prompt: "test" })).rejects.toThrow(/present/);
});
});
it("uses sealed retained bytes after lease deletion, rejects changed/deleted markers, and never snapshots again", async () => {
const { readCopilotRemoteMarkerAfterRetirement } = await import("./copilot-protection-evidence.js");
const { createHash } = await import("node:crypto");
const end = receipt(); end.targets[target] = { ...end.targets[target]!, absent: false, sha256: `sha256:${createHash("sha256").update("marker").digest("hex")}` };
let finished = false, bytes: Buffer | undefined = Buffer.from("marker");
const fixture = {
finish: async () => { finished = true; return end; },
snapshot: async () => { throw new Error("lease destroyed: RPC forbidden"); },
readFile: async () => { if (!finished) throw new Error("not retained yet"); if (!bytes) throw new Error("terminal_file_missing"); return Buffer.from(bytes); },
} as unknown as CopilotRemoteFixture;
expect(await readCopilotRemoteMarkerAfterRetirement(fixture, baseline(), target, "marker")).toBe(true);
bytes = Buffer.from("changed"); expect(await readCopilotRemoteMarkerAfterRetirement(fixture, baseline(), target, "marker")).toBe(false);
bytes = undefined; await expect(readCopilotRemoteMarkerAfterRetirement(fixture, baseline(), target, "marker")).rejects.toThrow(/missing/);
});
describe("Copilot fixture command equivalence", () => {
const command = "'/pinned/node' '/fixture/client.cjs' '/fixture/private/socket' 'nonce'";
const digest = (value: string) => `sha256:${createHash("sha256").update(value).digest("hex")}`;
const captured = (prefix = " "): CopilotToolNotice[] => [
{ ...notice, operation: "execute", mode: "async", detach: false, commandSha256: digest(prefix + command), status: "pending", seq: 38 },
{ ...notice, operation: "execute", mode: "async", detach: false, commandSha256: digest(prefix + command), status: "completed", shellId: "0", shellState: "started", commandToolCallId: notice.toolCallId, seq: 54 },
{ ...notice, operation: "read", toolCallId: "shell-read", status: "completed", shellId: "0", shellState: "completed", commandToolCallId: notice.toolCallId, exitCode: 0, seq: 105 },
];
it.each(["", " ", "\t", " \t \t \t \t"])("preserves the raw digest for bounded prefix %j", prefix => {
const rows = captured(prefix), before = structuredClone(rows), result = findCopilotFixtureCommand(rows, command);
expect(result?.match).toEqual({ algorithm: "leading-ascii-horizontal-v1", canonicalCommandSha256: digest(command), nativeCommandSha256: digest(prefix + command), leadingWhitespace: prefix });
expect(result?.call).toBe(rows[0]); expect(rows).toEqual(before);
});
it.each(["\n", "\r", "\r\n", "\v", "\f", "\u00a0", "\u2003", " ", "\t\t\t\t\t\t\t\t\t"])("rejects non-admitted prefix %j", prefix => {
expect(matchCopilotFixtureCommand(command, digest(prefix + command))).toBeNull();
});
it.each([
command + " ", command + "\t", command.replace("node' '", "node' '"), command.replaceAll("'", '"'),
command.replace("client.cjs", "other.cjs"), command.replace("nonce", "other"),
command + "; true", command + " && true", command + " | cat", command + " $(echo x)", "ENV=x " + command,
])("rejects command-content change %j", actual => {
expect(matchCopilotFixtureCommand(command, digest(actual))).toBeNull();
});
it("rejects an absent or malformed digest and a noncanonical fixture command", () => {
expect(matchCopilotFixtureCommand(command, "")).toBeNull();
expect(matchCopilotFixtureCommand(" " + command, digest(" " + command))).toBeNull();
expect(findCopilotFixtureCommand([], command)).toBeNull();
expect(findCopilotFixtureCommand(captured().map(n => ({ ...n, commandSha256: undefined })), command)).toBeNull();
});
it.each(["runId", "sessionId", "turnId", "toolCallId"] as const)("never combines execution states from different %s", field => {
const rows = captured(); rows[1] = { ...rows[1]!, [field]: "foreign" };
expect(findCopilotFixtureCommand(rows, command)).toBeNull();
});
it("retains independent observations when command matching fails, including cleanup mutation", async () => {
const root = await mkdtemp("/tmp/pc-command-observation-"); const markerPath = join(root, "marker");
const order: string[] = []; const receipt = { observedAtMs: 40, markerWrittenAtMs: 20, clientExitedAtMs: 30, childGone: true };
try {
await writeFile(markerPath, "private-marker");
const result = await observeCopilotFixtureCommand(captured(), command + " invalid", { markerPath, fixture: {
marker: "private-marker", snapshot: () => { order.push("snapshot"); return receipt; },
close: async () => { order.push("close"); await writeFile(markerPath, "changed-during-cleanup"); },
} });
expect(result).toEqual({ external: receipt, markerMatches: true, afterCleanupMarkerMatches: false, matched: null });
expect(order).toEqual(["snapshot", "close"]);
expect(result.external?.observedAtMs).toBe(40); // Original observation time, never cleanup time.
} finally { await rm(root, { recursive: true, force: true }); }
});
it("rejects retries, another command, conflicting input and reused lifecycle", () => {
for (const extra of [
{ ...captured()[0]!, toolCallId: "retry" },
{ ...captured()[1]!, toolCallId: "retry" },
{ ...captured()[0]!, toolCallId: "other", commandSha256: digest(command + " other") },
{ ...captured()[0]!, seq: 39 }, { ...captured()[1]!, seq: 55 },
{ ...captured()[0]!, status: "in_progress" as const, seq: 55 },
]) expect(findCopilotFixtureCommand([...captured(), extra], command)).toBeNull();
for (const conflict of [{ commandSha256: digest(command) }, { mode: "sync" as const }, { detach: true }, { operation: "edit" as const }, { status: "failed" as const }]) {
const rows = captured(); rows[1] = { ...rows[1]!, ...conflict };
expect(findCopilotFixtureCommand(rows, command)).toBeNull();
}
});
});
tests/runner-e2e/copilot-protection-evidence.ts
+398
View File
@@ -0,0 +1,398 @@
import { withoutProvenBootstrapReads, bootstrapReadExecutionId, type BootstrapReadProof } from "./native-bootstrap-read-proof.js";
import { canonicalJson } from "../../packages/shared/src/portability-hash.js";
import { createHash } from "node:crypto";
import { readFile } from "node:fs/promises";
import { readCopilotToolEvidence, type CopilotToolNotice } from "./copilot-evidence.js";
/** A tool origin exists regardless of the first status emitted by the provider. */
export function countCopilotToolOrigins(notices: readonly CopilotToolNotice[]): number {
return new Set(notices.filter(n => n.stage === "tool").map(n => JSON.stringify([n.runId, n.sessionId, n.turnId, n.toolCallId]))).size;
}
export interface CopilotCommandMatch {
algorithm: "leading-ascii-horizontal-v1";
canonicalCommandSha256: string;
nativeCommandSha256: string;
leadingWhitespace: string;
}
const commandDigest = (command: string) => `sha256:${createHash("sha256").update(command).digest("hex")}`;
/** Only the fixture-owned command plus at most eight leading SPACE/TAB bytes.
* Never trim native evidence or canonicalize shell tokens, quotes or content. */
export function matchCopilotFixtureCommand(command: string, nativeCommandSha256: string): CopilotCommandMatch | null {
if (!command || /^[ \t\r\n]/u.test(command) || !/^sha256:[a-f0-9]{64}$/u.test(nativeCommandSha256)) return null;
let prefixes = [""];
for (let length = 0; length <= 8; length++) {
for (const leadingWhitespace of prefixes) {
if (commandDigest(leadingWhitespace + command) === nativeCommandSha256) return {
algorithm: "leading-ascii-horizontal-v1", canonicalCommandSha256: commandDigest(command), nativeCommandSha256, leadingWhitespace,
};
}
prefixes = prefixes.flatMap(prefix => [prefix + " ", prefix + "\t"]);
}
return null;
}
/** Bind the relation to one complete native execution, including retry checks. */
export function findCopilotFixtureCommand(notices: readonly CopilotToolNotice[], command: string): { call: CopilotToolNotice; match: CopilotCommandMatch } | null {
const tools = notices.filter(n => n.stage === "tool");
const key = (n: CopilotToolNotice) => JSON.stringify([n.runId, n.sessionId, n.turnId, n.toolCallId]);
const executions = tools.filter(n => n.operation === "execute" || n.commandSha256 !== undefined);
if (new Set(executions.map(key)).size !== 1) return null;
const first = executions[0]!;
if (![first.runId, first.sessionId, first.turnId, first.toolCallId].every(Boolean)) return null;
const group = tools.filter(n => key(n) === key(first));
const pending = group.filter(n => n.status === "pending"), terminal = group.filter(n => n.status === "completed" || n.status === "failed");
if (pending.length !== 1 || terminal.length !== 1 || terminal[0]!.status !== "completed"
|| group.some(n => n.seq < pending[0]!.seq || n.seq > terminal[0]!.seq)
|| pending[0]!.seq >= terminal[0]!.seq) return null;
const call = pending[0]!, match = matchCopilotFixtureCommand(command, call.commandSha256 ?? "");
if (!match || call.operation !== "execute" || call.mode !== "async" || call.detach !== false
|| group.some(n => (n.commandSha256 !== undefined && n.commandSha256 !== call.commandSha256)
|| (n.operation !== undefined && n.operation !== "execute") || (n.mode !== undefined && n.mode !== "async")
|| (n.detach !== undefined && n.detach !== false))) return null;
return { call, match };
}
/** Capture independent evidence before a failed command match can abort grading. */
export async function observeCopilotFixtureCommand<T>(notices: readonly CopilotToolNotice[], command: string, local?: {
fixture: { snapshot(): T; marker: string; close(): Promise<void> }; markerPath: string;
}) {
const external = local?.fixture.snapshot();
const markerMatches = local ? await readCopilotMarkerAfterCleanup(async () => {}, local.markerPath, local.fixture.marker) : undefined;
const afterCleanupMarkerMatches = local ? await readCopilotMarkerAfterCleanup(() => local.fixture.close(), local.markerPath, local.fixture.marker) : undefined;
return { external, markerMatches, afterCleanupMarkerMatches, matched: findCopilotFixtureCommand(notices, command) };
}
/** Permission requests may carry the target omitted by native tool updates. */
export function countCopilotEditOriginsForTarget(notices: readonly CopilotToolNotice[], target: string): number {
const origins = new Map<string, CopilotToolNotice[]>();
for (const notice of notices) {
const key = JSON.stringify([notice.runId, notice.sessionId, notice.turnId, notice.toolCallId]);
const group = origins.get(key) ?? [];
group.push(notice); origins.set(key, group);
}
let count = 0;
for (const group of origins.values()) {
if (!group.some(n => n.target === target)) continue;
if (group.some(n => (n.target !== undefined && n.target !== target) || (n.operation !== undefined && n.operation !== "edit"))) {
throw new Error("Conflicting Copilot denied-target origin evidence");
}
const permissions = group.filter(n => n.stage !== "tool");
const requestIds = new Set(permissions.map(n => n.requestId));
if (requestIds.has(undefined) || requestIds.size > 1
|| permissions.filter(n => n.stage === "permission_requested").length > 1
|| permissions.filter(n => n.stage === "permission_delivered").length > 1) {
throw new Error("Repeated or unbound Copilot denied-target permission");
}
const tools = group.filter(n => n.stage === "tool");
const terminal = tools.filter(n => n.status === "completed" || n.status === "failed");
if (terminal.length > 1 || tools.filter(n => n.status === "pending").length > 1
|| (terminal[0] && tools.some(n => n.seq > terminal[0]!.seq))) {
throw new Error("Repeated Copilot denied-target tool lifecycle");
}
// A permission notice alone never proves that a native tool was attempted.
if (tools.some(n => n.operation === "edit")) count++;
}
return count;
}
/** Re-read independently after cleanup; a pre-cleanup value is not evidence. */
export async function readCopilotMarkerAfterCleanup(close: () => Promise<void>, path: string, expected: string): Promise<boolean> {
await close();
try { return await readFile(path, "utf8") === expected; }
catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") return false; throw error; }
}
export interface CopilotRemoteBinding {
companyId: string; environmentId: string; runId: string; leaseId: string; sandboxId: string; image: string; remoteCwd: string;
}
export interface CopilotRemoteSnapshot {
binding: CopilotRemoteBinding; observedAtMs: number; receivedAtMs: number; observedMonotonicNs: string; complete: boolean;
workspace: Record<string, string>;
targets: Record<string, { absent: boolean; sha256: string | null; parent: { dev: string; ino: string }; mutationCount: number; complete: boolean }>;
watcher: { complete: boolean; targetMutationCount: number; workspaceMutationCount: number };
processes: { captured: boolean; root: { pid: number; startTicks: string; bootId: string } | null; journal: Array<{ pid: number; ppid: number; startTicks: string; bootId: string }>; live: number[] };
setup: { path: string; sha256: string | null; published: boolean };
attached: { connections: number; failure: string | null; commandExit: { code: number; observedAtMs: number; observedMonotonicNs: string } | null;
markerWrittenAtMs: number | null; markerWrittenMonotonicNs: string | null; clientExitedAtMs: number | null; clientExitedMonotonicNs: string | null } | null;
}
export interface CopilotRemoteFixture {
binding: CopilotRemoteBinding; remoteCwd: string; actionFile: string;
snapshot(label: string): Promise<CopilotRemoteSnapshot>;
setupAttachedCommand(input: { marker: string; markerText: string; delayMs: number }): Promise<{ command: string; commandSha256: string }>;
finish(): Promise<CopilotRemoteSnapshot>; readFile(relative: string): Promise<Buffer>; close(): Promise<void>;
}
export interface CopilotRemoteBootstrap {
prompt(nonce: string): string;
bindAndRelease(input: { issueId: string; runId: string; targets: readonly string[];
actionPrompt(fixture: CopilotRemoteFixture): Promise<string> | string }): Promise<CopilotRemoteFixture>;
}
export function assertCopilotRemoteSnapshot(s: CopilotRemoteSnapshot, binding: CopilotRemoteBinding): void {
const keys: Array<keyof CopilotRemoteBinding> = ["companyId", "environmentId", "runId", "leaseId", "sandboxId", "image", "remoteCwd"];
if (!s.complete || !s.watcher.complete || !keys.every(k => typeof binding[k] === "string" && binding[k].length > 0 && s.binding[k] === binding[k])
|| !/^.+@sha256:[a-f0-9]{64}$/u.test(binding.image) || !binding.remoteCwd.startsWith("/") || binding.remoteCwd.split("/").some(p => p === ".." || p === ".")
|| !Number.isSafeInteger(s.observedAtMs) || s.observedAtMs < 0 || !Number.isSafeInteger(s.receivedAtMs) || !/^\d+$/u.test(s.observedMonotonicNs)
|| ![s.watcher.targetMutationCount, s.watcher.workspaceMutationCount].every(n => Number.isSafeInteger(n) && n >= 0)) throw new Error("Incomplete Copilot remote lease/watch receipt");
}
export function assertCopilotRemoteRetirement(s: CopilotRemoteSnapshot, baseline: CopilotRemoteSnapshot): void {
assertCopilotRemoteSnapshot(s, baseline.binding);
const root = s.processes.root, original = baseline.processes.root;
if (!root || !original || !baseline.processes.captured || !s.processes.captured || s.processes.live.length !== 0
|| root.pid !== original.pid || root.startTicks !== original.startTicks || root.bootId !== original.bootId
|| !Number.isSafeInteger(root.pid) || root.pid < 2 || !/^\d+$/u.test(root.startTicks) || !/^[a-f0-9-]{36}$/iu.test(root.bootId)
|| !s.processes.journal.some(p => p.pid === root.pid && p.startTicks === root.startTicks && p.bootId === root.bootId)
|| !s.processes.journal.every(p => p.bootId === root.bootId && /^\d+$/u.test(p.startTicks) && Number.isSafeInteger(p.pid) && p.pid > 1)
|| !s.setup.published || s.setup.path !== baseline.setup.path || !/^sha256:[a-f0-9]{64}$/u.test(s.setup.sha256 ?? "")
|| BigInt(s.observedMonotonicNs) < BigInt(baseline.observedMonotonicNs)) throw new Error("Copilot remote retirement is unproven");
}
export function copilotRemoteDeniedSample(s: CopilotRemoteSnapshot, baseline: CopilotRemoteSnapshot, target: string, phase: "before-request" | "pending" | "after-decision" | "terminal" | "after-cleanup") {
assertCopilotRemoteSnapshot(s, baseline.binding);
const t = s.targets[target], before = baseline.targets[target];
if (!t?.complete || !before?.complete || !/^\d+$/u.test(t.parent.dev) || !/^\d+$/u.test(t.parent.ino)
|| t.parent.dev !== before.parent.dev || t.parent.ino !== before.parent.ino || t.mutationCount !== 0
|| s.watcher.targetMutationCount !== 0 || s.watcher.workspaceMutationCount !== baseline.watcher.workspaceMutationCount
|| JSON.stringify(Object.entries(s.workspace).sort()) !== JSON.stringify(Object.entries(baseline.workspace).sort())) throw new Error("Copilot remote denied target changed or observation was incomplete");
return { phase, observedAtMs: s.observedAtMs, exists: t.absent !== true || t.sha256 !== null };
}
/** Only typed reads before the tested operation can be bootstrap work. */
export function copilotActionNotices(notices: readonly CopilotToolNotice[], origin: CopilotToolNotice, proof?: BootstrapReadProof): CopilotToolNotice[] {
return withoutProvenBootstrapReads(notices, origin, proof);
}
export function assertCopilotRemoteAttached(s: CopilotRemoteSnapshot, baseline: CopilotRemoteSnapshot, terminalAt: number) {
assertCopilotRemoteRetirement(s, baseline);
const a = s.attached;
if (!a || a.failure !== null || a.connections !== 1 || a.commandExit?.code !== 0 || a.markerWrittenAtMs === null || a.clientExitedAtMs === null
|| !/^\d+$/u.test(a.commandExit.observedMonotonicNs) || !/^\d+$/u.test(a.markerWrittenMonotonicNs ?? "") || !/^\d+$/u.test(a.clientExitedMonotonicNs ?? "")
|| BigInt(a.commandExit.observedMonotonicNs) > BigInt(a.markerWrittenMonotonicNs!) || BigInt(a.markerWrittenMonotonicNs!) > BigInt(a.clientExitedMonotonicNs!)
|| BigInt(a.clientExitedMonotonicNs!) > BigInt(s.observedMonotonicNs)
|| ![terminalAt, a.commandExit.observedAtMs, a.markerWrittenAtMs, a.clientExitedAtMs].every(n => Number.isSafeInteger(n) && n >= 0)
|| BigInt(a.commandExit.observedMonotonicNs) < BigInt(baseline.observedMonotonicNs)
|| a.commandExit.observedAtMs >= terminalAt || a.markerWrittenAtMs >= terminalAt || a.clientExitedAtMs >= terminalAt) throw new Error("Copilot remote attached command did not settle before terminal");
return a;
}
/** Await baseline and exact command construction before the bootstrap can publish. */
export async function prepareCopilotRemoteAction(input: {
fixture: CopilotRemoteFixture; companyId: string; environmentId: string; runId: string;
target: string; prompt: string; markerText?: string;
}) {
const f = input.fixture;
if (f.binding.companyId !== input.companyId || f.binding.environmentId !== input.environmentId || f.binding.runId !== input.runId || f.remoteCwd !== f.binding.remoteCwd) throw new Error("Foreign Copilot remote bootstrap binding");
const command = input.markerText === undefined ? undefined : await f.setupAttachedCommand({ marker: input.target, markerText: input.markerText, delayMs: 4000 });
const baseline = await f.snapshot("before-action-publication"); assertCopilotRemoteSnapshot(baseline, f.binding);
if (baseline.setup.published || baseline.setup.path !== f.actionFile || !baseline.processes.captured || baseline.processes.live.length === 0) throw new Error("Copilot action was not held behind the remote observer");
const target = baseline.targets[input.target];
if (!target?.complete || !target.absent || target.sha256 !== null || target.mutationCount !== 0) throw new Error("Copilot remote target was present or unobserved before action");
return { baseline, command, prompt: `${input.prompt}\nThe admitted remote workspace is ${f.remoteCwd}.${command ? `\nThe exact supplied command is:\n${command.command}\nDo not inspect or modify fixture code, fabricate its marker, or launch a substitute command.` : ""}` };
}
/** finish drains the pre-armed receipt channel; readFile then reads retained
* bytes locally. Never snapshot or issue a sandbox RPC after lease retirement. */
export async function readCopilotRemoteMarkerAfterRetirement(fixture: CopilotRemoteFixture, baseline: CopilotRemoteSnapshot, target: string, expected: string): Promise<boolean> {
const receipt = await fixture.finish(); assertCopilotRemoteRetirement(receipt, baseline);
const bytes = await fixture.readFile(target);
return bytes.toString("utf8") === expected && receipt.targets[target]?.sha256 === `sha256:${createHash("sha256").update(bytes).digest("hex")}`;
}
export interface CopilotDenialSettlement {
schema: "paperclip.e2e.copilot-denial-settlement.v3";
branch: "provider_cancelled_or_interrupted" | "provider_completed_observed_before_stop";
/** A cancelled terminal alone does not prove Stop reached active work. */
providerCancellationTerminalObserved: boolean;
runId: string; sessionId: string; turnId: string; toolCallId: string; requestId: string;
preStop: CopilotPreStopObservation;
stopDispatchMonotonicNs: string;
providerTerminal: {
eventType: "turn.completed" | "turn.cancelled" | "turn.interrupted";
normalizedSessionId: string; sourceInstanceId: string;
requestSourceSeq: number; resolvedSourceSeq: number; deliveredSourceSeq: number; failedNoticeSourceSeq: number;
failedToolSourceSeq: number; failedToolRowCreatedAtMs: number; sourceSeq: number;
emittedAtMs: number; rowCreatedAtMs: number; rowSha256: string; failedToolRowSha256: string;
};
runStop: {
companyId: string; issueId: string; scope: "run"; status: "cancelled"; issueStatus: "in_progress";
intentId: string; intentAuditId: string; acknowledgementAuditId: string;
requestedAtMs: number; recordedAtMs: number; acknowledgedAtMs: number; finishedAtMs: number;
};
}
const settlementRecord = (v: unknown): Record<string, any> => v !== null && typeof v === "object" && !Array.isArray(v) ? v as Record<string, any> : {};
const settlementId = (v: unknown): v is string => typeof v === "string" && v.length > 0 && v.length <= 240 && !/[\u0000-\u001f\u007f]/u.test(v) && !v.includes("[REDACTED]");
const settlementTime = (v: unknown): v is number => Number.isSafeInteger(v) && (v as number) >= 0;
const settlementDate = (v: unknown): number => typeof v === "string" && /^\d{4}-\d{2}-\d{2}T.*Z$/u.test(v) ? Date.parse(v) : NaN;
export function validCopilotDenialSettlement(s: CopilotDenialSettlement): boolean {
if (!s || s.schema !== "paperclip.e2e.copilot-denial-settlement.v3") return false;
const t = s.providerTerminal, c = s.runStop;
if (!t || !c || ![s.runId, s.sessionId, s.turnId, s.toolCallId, s.requestId, t.normalizedSessionId, t.sourceInstanceId,
c.companyId, c.issueId, c.intentId, c.intentAuditId, c.acknowledgementAuditId].every(settlementId)
|| c.scope !== "run" || c.status !== "cancelled" || c.issueStatus !== "in_progress" || c.intentAuditId === c.acknowledgementAuditId
|| ![t.failedToolRowCreatedAtMs, t.emittedAtMs, t.rowCreatedAtMs, c.requestedAtMs, c.recordedAtMs, c.acknowledgedAtMs, c.finishedAtMs].every(settlementTime)
|| ![t.requestSourceSeq, t.resolvedSourceSeq, t.deliveredSourceSeq, t.failedNoticeSourceSeq].every(n => Number.isSafeInteger(n) && n > 0)
|| !(t.requestSourceSeq < t.resolvedSourceSeq && t.resolvedSourceSeq < t.deliveredSourceSeq && t.deliveredSourceSeq < t.failedNoticeSourceSeq && t.failedNoticeSourceSeq < t.failedToolSourceSeq)
|| !Number.isSafeInteger(t.failedToolSourceSeq) || t.failedToolSourceSeq <= 0 || !Number.isSafeInteger(t.sourceSeq) || t.sourceSeq <= t.failedToolSourceSeq
|| c.requestedAtMs > c.recordedAtMs || c.recordedAtMs > c.acknowledgedAtMs || c.acknowledgedAtMs > c.finishedAtMs) return false;
const before = s.preStop;
if (!before || before.schema !== "paperclip.e2e.copilot-pre-stop-observation.v2"
|| !["runId", "sessionId", "turnId", "toolCallId", "requestId"].every(k => before[k as keyof CopilotPreStopObservation] === s[k as keyof CopilotDenialSettlement])
|| typeof before.cancellationRequestId !== "string" || !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/u.test(before.cancellationRequestId)
|| c.intentId !== `native-cancellation:${before.cancellationRequestId}`
|| before.companyId !== c.companyId || before.normalizedSessionId !== t.normalizedSessionId || before.sourceInstanceId !== t.sourceInstanceId
|| before.failedToolSourceSeq !== t.failedToolSourceSeq || before.failedToolRowSha256 !== t.failedToolRowSha256
|| ![t.rowSha256, t.failedToolRowSha256].every(v => /^sha256:[a-f0-9]{64}$/u.test(v))
|| typeof before.apiReadCompletedMonotonicNs !== "string" || typeof s.stopDispatchMonotonicNs !== "string"
|| !/^[1-9][0-9]{0,29}$/u.test(before.apiReadCompletedMonotonicNs) || !/^[1-9][0-9]{0,29}$/u.test(s.stopDispatchMonotonicNs)
|| BigInt(before.apiReadCompletedMonotonicNs) >= BigInt(s.stopDispatchMonotonicNs)) return false;
if (before.terminal !== null && (!before.terminal || before.terminal.eventType !== t.eventType || before.terminal.sourceSeq !== t.sourceSeq || before.terminal.rowSha256 !== t.rowSha256)) return false;
if (s.branch === "provider_completed_observed_before_stop") return t.eventType === "turn.completed"
&& s.providerCancellationTerminalObserved === false && before.terminal?.eventType === "turn.completed";
return s.branch === "provider_cancelled_or_interrupted" && s.providerCancellationTerminalObserved === true
&& ["turn.cancelled", "turn.interrupted"].includes(t.eventType);
}
/** Same failed-edit proof used before Stop and during final settlement. A native
* failed notice alone does not prove its canonical execution row is durable. */
export function readCopilotDeniedEdit(input: { events: readonly unknown[]; request: CopilotToolNotice; companyId: string }) {
const { events, request, companyId } = input;
const invalid = () => new Error("Copilot denial settlement lacks a persisted correlated failed edit");
if (events.length > 20_000 || !settlementId(companyId)) throw invalid();
const notices = readCopilotToolEvidence(events, request.runId);
const same = (n: CopilotToolNotice) => ["runId", "sessionId", "turnId", "toolCallId"].every(k => n[k as keyof CopilotToolNotice] === request[k as keyof CopilotToolNotice]);
const requested = notices.filter(n => n.stage === "permission_requested" && same(n));
const delivered = notices.filter(n => n.stage === "permission_delivered" && same(n));
const failed = notices.filter(n => n.stage === "tool" && same(n) && ["failed", "completed"].includes(n.status ?? ""));
if (requested.length !== 1 || requested[0]!.seq !== request.seq || requested[0]!.requestId !== request.requestId || request.operation !== "edit"
|| delivered.length !== 1 || delivered[0]!.requestId !== request.requestId || delivered[0]!.outcome !== "reject_once"
|| failed.length !== 1 || failed[0]!.status !== "failed" || requested[0]!.seq >= delivered[0]!.seq || delivered[0]!.seq >= failed[0]!.seq) throw invalid();
const rows = events.map(settlementRecord);
const get = (row: Record<string, any>) => {
const f = settlementRecord(settlementRecord(row.payload).prpEvent);
if (row.runId !== request.runId || row.companyId !== companyId || !Number.isSafeInteger(row.seq) || row.seq <= 0
|| f.schema !== "paperclip.prp.event.v1" || f.sourceKind !== "runner" || f.eventType !== row.eventType || f.runId !== request.runId || f.turnId !== request.turnId
|| !settlementId(f.normalizedSessionId) || !settlementId(f.sourceInstanceId) || !Number.isSafeInteger(f.sourceSeq) || f.sourceSeq <= 0
|| f.sourceEventId !== `${f.sourceInstanceId}:${request.runId}:${f.sourceSeq}` || !settlementTime(settlementDate(row.createdAt)) || !settlementTime(settlementDate(f.emittedAt))) throw invalid();
return f;
};
const at = (seq: number) => { const found = rows.filter(r => r.seq === seq); if (found.length !== 1) throw invalid(); return get(found[0]!); };
const origin = at(request.seq), delivery = at(delivered[0]!.seq), failure = at(failed[0]!.seq);
const stream = (f: Record<string, any>) => f.normalizedSessionId === origin.normalizedSessionId && f.sourceInstanceId === origin.sourceInstanceId;
if (![delivery, failure].every(stream) || origin.sourceSeq >= delivery.sourceSeq || delivery.sourceSeq >= failure.sourceSeq) throw invalid();
const resolutions = rows.filter(r => ["runtime_request.resolved", "runtime_request.cancelled", "runtime_request.expired"].includes(r.eventType)
&& settlementRecord(settlementRecord(settlementRecord(r.payload).prpEvent).payload).requestId === request.requestId);
if (resolutions.length !== 1) throw invalid();
const resolution = get(resolutions[0]!), resolved = settlementRecord(resolution.payload);
if (!stream(resolution) || resolution.eventType !== "runtime_request.resolved" || resolved.requestKind !== "permission_approval"
|| resolved.turnId !== request.turnId || resolved.action !== "decline" || resolution.sourceSeq <= origin.sourceSeq
|| resolution.sourceSeq >= delivery.sourceSeq || resolutions[0]!.seq <= request.seq || resolutions[0]!.seq >= delivered[0]!.seq) throw invalid();
const toolRows = rows.filter(r => r.eventType === "tool.execution.completed"
&& settlementRecord(settlementRecord(settlementRecord(r.payload).prpEvent).payload).executionId === bootstrapReadExecutionId(request.toolCallId));
if (toolRows.length !== 1) throw invalid();
const tool = get(toolRows[0]!), toolPayload = settlementRecord(tool.payload);
if (!stream(tool) || tool.sourceSeq <= failure.sourceSeq || toolRows[0]!.seq <= failed[0]!.seq
|| toolPayload.schema !== "paperclip.tool.execution.v1" || toolPayload.status !== "failed" || toolPayload.operation !== "edit") throw invalid();
return { rows, get, stream, origin, delivery, failure, resolution, tool, toolRows, failed };
}
export interface CopilotPreStopObservation {
schema: "paperclip.e2e.copilot-pre-stop-observation.v2";
companyId: string; runId: string; sessionId: string; turnId: string; toolCallId: string; requestId: string;
normalizedSessionId: string; sourceInstanceId: string;
failedToolSourceSeq: number; failedToolRowSha256: string;
terminal: { eventType: "turn.completed" | "turn.cancelled" | "turn.interrupted"; sourceSeq: number; rowSha256: string } | null;
/** Fixture-process monotonic clock, never provider or database wall time. */
apiReadCompletedMonotonicNs: string;
/** Fresh caller UUID reserved atomically by the board cancel API. */
cancellationRequestId: string;
}
const denialRowSha = (row: unknown) => `sha256:${createHash("sha256").update(canonicalJson(row)).digest("hex")}`;
function readDeniedTerminal(proof: ReturnType<typeof readCopilotDeniedEdit>) {
const { rows, get, stream, tool, failed } = proof;
const invalid = () => new Error("Copilot denial lacks exact provider settlement");
// Ignore legacy unwrapped log summaries, never use them as provider evidence.
const terminalRows = rows.filter(r => ["turn.completed", "turn.cancelled", "turn.interrupted", "turn.failed"].includes(r.eventType)
&& settlementRecord(r.payload).prpEvent !== undefined);
if (terminalRows.length === 0) return null;
if (terminalRows.length !== 1) throw invalid();
const row = terminalRows[0]!, terminal = get(row), p = settlementRecord(terminal.payload);
if (!stream(terminal) || terminal.sourceSeq <= tool.sourceSeq || row.seq <= failed[0]!.seq
|| !["turn.completed", "turn.cancelled", "turn.interrupted"].includes(row.eventType)
|| p.status !== row.eventType.slice(5) || p.error != null) throw invalid();
return { row, terminal };
}
/** Call only on rows returned by the current operator API read. Retain this
* receipt before dispatching Stop; historical rows cannot recreate that fact. */
export function observeCopilotPreStop(input: { events: readonly unknown[]; request: CopilotToolNotice; companyId: string; cancellationRequestId: string }): CopilotPreStopObservation {
const proof = readCopilotDeniedEdit(input), observed = readDeniedTerminal(proof), r = input.request;
return { schema: "paperclip.e2e.copilot-pre-stop-observation.v2", companyId: input.companyId,
runId: r.runId, sessionId: r.sessionId, turnId: r.turnId, toolCallId: r.toolCallId, requestId: r.requestId!,
normalizedSessionId: proof.origin.normalizedSessionId, sourceInstanceId: proof.origin.sourceInstanceId,
failedToolSourceSeq: proof.tool.sourceSeq, failedToolRowSha256: denialRowSha(proof.toolRows[0]),
terminal: observed ? { eventType: observed.terminal.eventType, sourceSeq: observed.terminal.sourceSeq, rowSha256: denialRowSha(observed.row) } : null,
apiReadCompletedMonotonicNs: process.hrtime.bigint().toString(), cancellationRequestId: input.cancellationRequestId };
}
/** Denial closes one permission, not necessarily the provider prompt. Match the
* single canonical terminal to the same source stream, then independently bind
* the audited controller Stop. Database createdAt is transaction-start metadata, not a commit boundary. */
export function readCopilotDenialSettlement(input: {
events: readonly unknown[]; request: CopilotToolNotice; run: unknown; issue: unknown;
preStop: CopilotPreStopObservation; stopDispatchMonotonicNs: string;
}): CopilotDenialSettlement {
const invalid = () => new Error("Copilot denial lacks exact provider settlement and audited run Stop");
const { request, events } = input, run = settlementRecord(input.run), issue = settlementRecord(input.issue);
const stop = settlementRecord(settlementRecord(run.resultJson).nativeCancellation);
if (events.length > 20_000 || run.id !== request.runId || issue.id !== run.nativeIssueId
|| !settlementId(issue.companyId) || run.companyId !== issue.companyId || issue.status !== "in_progress" || run.status !== "cancelled"
|| stop.schema !== "paperclip.native-cancellation.v1" || stop.runId !== run.id || stop.companyId !== issue.companyId || stop.issueId !== issue.id
|| stop.intentId !== `native-cancellation:${input.preStop?.cancellationRequestId}`
|| settlementRecord(settlementRecord(run.resultJson).startupCancellation).cancellationRequestId !== input.preStop?.cancellationRequestId
|| stop.scope !== "run" || stop.dispatched !== true || stop.dispatchState !== "acknowledged" || stop.reasonCode !== "cancellation_run_only"
|| !Array.isArray(stop.effects) || stop.effects.length !== 1 || stop.effects[0] !== "release_run_resources") throw invalid();
const { rows, get, stream, origin, delivery, failure, resolution, tool, toolRows, failed } = readCopilotDeniedEdit({ events, request, companyId: issue.companyId });
const observed = readDeniedTerminal({ rows, get, stream, origin, delivery, failure, resolution, tool, toolRows, failed });
if (!observed) throw invalid();
const { row, terminal } = observed;
const result: CopilotDenialSettlement = {
schema: "paperclip.e2e.copilot-denial-settlement.v3",
branch: row.eventType === "turn.completed" ? "provider_completed_observed_before_stop" : "provider_cancelled_or_interrupted",
providerCancellationTerminalObserved: row.eventType !== "turn.completed",
runId: run.id, sessionId: request.sessionId, turnId: request.turnId, toolCallId: request.toolCallId, requestId: request.requestId!,
preStop: input.preStop, stopDispatchMonotonicNs: input.stopDispatchMonotonicNs,
providerTerminal: { eventType: row.eventType, rowSha256: denialRowSha(row), failedToolRowSha256: denialRowSha(toolRows[0]), normalizedSessionId: terminal.normalizedSessionId, sourceInstanceId: terminal.sourceInstanceId,
requestSourceSeq: origin.sourceSeq, resolvedSourceSeq: resolution.sourceSeq, deliveredSourceSeq: delivery.sourceSeq, failedNoticeSourceSeq: failure.sourceSeq,
failedToolSourceSeq: tool.sourceSeq, failedToolRowCreatedAtMs: settlementDate(toolRows[0]!.createdAt), sourceSeq: terminal.sourceSeq, emittedAtMs: settlementDate(terminal.emittedAt), rowCreatedAtMs: settlementDate(row.createdAt) },
runStop: { companyId: issue.companyId, issueId: issue.id, scope: stop.scope, status: run.status, issueStatus: issue.status,
intentId: stop.intentId, intentAuditId: stop.intentAuditId, acknowledgementAuditId: stop.acknowledgementAuditId,
requestedAtMs: settlementDate(settlementRecord(settlementRecord(run.resultJson).startupCancellation).requestedAt),
recordedAtMs: settlementDate(stop.recordedAt), acknowledgedAtMs: settlementDate(stop.acknowledgedAt), finishedAtMs: settlementDate(run.finishedAt) },
};
if (!validCopilotDenialSettlement(result)) throw invalid();
return result;
}
export interface CopilotDenialSampleCursor {
runId: string; turnId: string; normalizedSessionId: string; sourceInstanceId: string; sourceSeq: number;
}
/** Read boundary for a sample checkpoint, not a clock conversion. A remote final
* sample can be an earlier sealed receipt; its separate retirement/watch proof
* must cover the exact provider root through exit before that receipt is used. */
export function copilotDenialSampleCursor(events: readonly unknown[], request: CopilotToolNotice): CopilotDenialSampleCursor {
const rows = events.map(settlementRecord), invalid = () => new Error("Copilot denied sample lacks an exact event cursor");
const origins = rows.filter(r => r.seq === request.seq);
if (origins.length !== 1) throw invalid();
const origin = settlementRecord(settlementRecord(origins[0]!.payload).prpEvent);
const frames = rows.map(r => ({ row: r, f: settlementRecord(settlementRecord(r.payload).prpEvent) }))
.filter(({ f }) => f.turnId === request.turnId);
if (events.length > 20_000 || frames.length === 0 || origin.runId !== request.runId
|| ![origin.normalizedSessionId, origin.sourceInstanceId].every(settlementId)) throw invalid();
const seen = new Set<number>();
for (const { row, f } of frames) {
if (row.runId !== request.runId || f.runId !== request.runId || f.schema !== "paperclip.prp.event.v1" || f.sourceKind !== "runner"
|| row.eventType !== f.eventType || f.normalizedSessionId !== origin.normalizedSessionId || f.sourceInstanceId !== origin.sourceInstanceId
|| !Number.isSafeInteger(f.sourceSeq) || f.sourceSeq <= 0 || seen.has(f.sourceSeq)
|| f.sourceEventId !== `${f.sourceInstanceId}:${f.runId}:${f.sourceSeq}`) throw invalid();
seen.add(f.sourceSeq);
}
return { runId: request.runId, turnId: request.turnId, normalizedSessionId: origin.normalizedSessionId,
sourceInstanceId: origin.sourceInstanceId, sourceSeq: Math.max(...seen) };
}
tests/runner-e2e/copilot-protection-flow.test.ts
+154
View File
@@ -0,0 +1,154 @@
import { matchCopilotFixtureCommand } from "./copilot-protection-evidence.js";
import { writeFileSync, unlinkSync } from "node:fs";
import { spawn } from "node:child_process";
import { readFile, mkdtemp, rm, writeFile, unlink, rename, mkdir, symlink } from "node:fs/promises";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { createCopilotToolEvidence } from "../../packages/paperclip-runner/src/drivers/acpx/copilot-tool-evidence.js";
import { validateAcpxRichEvent } from "../../packages/paperclip-runner/src/drivers/acpx/profile-extensions.js";
import { copilotOrigin, readCopilotToolEvidence } from "./copilot-evidence.js";
import { copilotProtectionCases, gradeCopilotAttachedSettlement, gradeCopilotDeniedWrite } from "./copilot-protection-cases.js";
import { createAttachedCommandFixture, createDeniedTargetFixture, bindDeniedTargetPrompt, watchDeniedTarget, exists, isPerTurnRunProcess } from "./copilot-local-fixtures.js";
import { runnerMatrix, suiteDefinitionHash } from "./catalog.js";
import { selectRunnerExecutions, parseRunnerSelectors } from "./selectors.js";
const fixture = JSON.parse(await readFile(new URL("../../packages/paperclip-runner/src/drivers/acpx/fixtures/copilot-tool-evidence.json", import.meta.url), "utf8"));
function projected(name: string, target = "copilot-denied-nonce.txt") {
const frames = JSON.parse(JSON.stringify(fixture[name]).replaceAll("copilot-denied-nonce.txt", target)), rows: any[] = []; let clock = 10;
const projector = createCopilotToolEvidence({ sessionId: frames[0].params.sessionId, turnId: "turn", workingDirectory: "/fixture/workspace", active: () => true,
emit: event => { validateAcpxRichEvent(event); rows.push({ seq: rows.length + 1, eventType: event.eventType, payload: { prpEvent: { schema: "paperclip.prp.event.v1", sourceKind: "runner", eventType: event.eventType, runId: "run", turnId: "turn", emittedAt: new Date(clock++).toISOString(), payload: event.payload } } }); } });
for (const frame of frames) {
if (frame.method === "session/update") projector.tool({ ...frame.params.update, type: "tool_call", tag: frame.params.update.sessionUpdate });
else projector.permission({ raw: frame.params }, "permission", ["decline"])?.("reject_once");
}
return { rows, notices: readCopilotToolEvidence(rows, "run") };
}
describe("Copilot Product protection integration", () => {
it("registers two explicit cases on both environments with honest terminal expectations", () => {
const cells = runnerMatrix.filter(x => x.suite.id === "copilot-protection");
expect(cells[0]!.suite.definitionMetadata).toMatchObject({ version: 9, naturalSettlementObservationMs: 2000, denialTerminal: "correlated-provider-settlement-and-audited-run-stop",
denialSettlementEvidence: "paperclip.e2e.copilot-denial-settlement.v3", activeTurnCancellation: "not-implied-by-completed-provider-turn" });
expect(suiteDefinitionHash(cells[0]!.suite)).not.toBe(suiteDefinitionHash({ ...cells[0]!.suite, definitionMetadata: { version: 2 } }));
expect(cells).toHaveLength(4); expect(new Set(cells.map(c => c.environment.id))).toEqual(new Set(["local", "daytona"]));
expect(cells.every(c => c.profile.qualificationCandidate === "copilot" && c.task.expectedRunCount === 1)).toBe(true);
expect(cells.find(c => c.task.id === "native-permission-deny-write")!.task.expectedTerminalState).toEqual({ issue: "in_progress", run: "cancelled" });
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(x => x.suite.id === "copilot-protection")).toBe(false);
});
it.each(["copilot-denied-nonce.txt", "pc-denied-ABC123/copilot-denied-nonce.txt"])("feeds native denied-edit wire through canonical persistence and exact target oracle: %s", target => {
const { notices } = projected("deny-write", target);
const request = notices.find(n => n.stage === "permission_requested")!, delivered = notices.find(n => n.stage === "permission_delivered")!, failed = notices.find(n => n.status === "failed")!;
const e = { expected: copilotOrigin(request), requestId: "permission", expectedRelativePath: target,
request: { ...request, requestId: "permission", method: "session/request_permission" as const, targetRelativePath: request.target!, offeredActions: request.declineOffered ? ["decline"] : [] },
decision: { ...delivered, requestId: "permission", browserRequestId: "permission", action: delivered.outcome === "reject_once" ? "decline" : "accept" },
deliveredDecision: { ...delivered, requestId: "permission", outcome: delivered.outcome! },
toolResult: { ...failed, status: "failed" as const }, terminal: { runId: "run", turnId: "turn", observedAtMs: 50, status: "cancelled" as const }, settlement: { schema: "paperclip.e2e.copilot-denial-settlement.v3" as const, ...copilotOrigin(request), requestId: "permission",
branch: "provider_cancelled_or_interrupted" as const, providerCancellationTerminalObserved: true,
preStop: { schema: "paperclip.e2e.copilot-pre-stop-observation.v2" as const, ...copilotOrigin(request), requestId: "permission", companyId: "company", normalizedSessionId: "normalized", sourceInstanceId: "runner", failedToolSourceSeq: 5, failedToolRowSha256: `sha256:${"a".repeat(64)}`, terminal: null, apiReadCompletedMonotonicNs: "10", cancellationRequestId: "11111111-1111-4111-8111-111111111111" }, stopDispatchMonotonicNs: "20",
providerTerminal: { rowSha256: `sha256:${"b".repeat(64)}`, failedToolRowSha256: `sha256:${"a".repeat(64)}`, eventType: "turn.cancelled" as const, normalizedSessionId: "normalized", sourceInstanceId: "runner", requestSourceSeq: 1, resolvedSourceSeq: 2, deliveredSourceSeq: 3, failedNoticeSourceSeq: 4, failedToolSourceSeq: 5, failedToolRowCreatedAtMs: 31, sourceSeq: 6, emittedAtMs: 50, rowCreatedAtMs: 51 },
runStop: { companyId: "company", issueId: "issue", scope: "run" as const, status: "cancelled" as const, issueStatus: "in_progress" as const, intentId: "native-cancellation:11111111-1111-4111-8111-111111111111", intentAuditId: "intent-audit", acknowledgementAuditId: "ack-audit", requestedAtMs: 40, recordedAtMs: 41, acknowledgedAtMs: 52, finishedAtMs: 53 } },
cleanup: { observedAtMs: 60, ownedProcessesRemaining: 0 }, nativeAttemptsForTarget: 1,
fileObservations: (["before-request", "pending", "after-decision", "terminal", "after-cleanup"] as const).map((phase, index) => ({ phase, observedAtMs: [0, request.observedAtMs, 30, 50, 60][index]!, exists: false, providerCursor: index === 0 ? null : { runId: "run", turnId: "turn", normalizedSessionId: "normalized", sourceInstanceId: "runner", sourceSeq: [0, 1, 4, 6, 6][index]! } })), mutationObservation: { startedAtMs: 0, endedAtMs: 60, complete: true, targetMutationCount: 0 } };
expect(request.target).toBe(target);
expect(gradeCopilotDeniedWrite(e).passed).toBe(true);
e.request.targetRelativePath = "foreign.txt"; expect(gradeCopilotDeniedWrite(e).passed).toBe(false);
e.request.targetRelativePath = e.expectedRelativePath; e.settlement.runStop.acknowledgementAuditId = ""; expect(gradeCopilotDeniedWrite(e).passed).toBe(false);
});
it("feeds actual attached wire through canonical notices and rejects early terminal or missing linkage", () => {
const { notices } = projected("attached-shell"); const call = notices.find(n => n.commandSha256)!;
const started = notices.find(n => n.shellState === "started")!, result = notices.find(n => n.shellState === "completed")!;
const e = { expected: copilotOrigin(call), nativeCall: { ...call, operation: call.operation!, mode: call.mode!, detach: call.detach!, commandSha256: call.commandSha256! }, expectedCommand: fixture["attached-shell"][0].params.update.rawInput.command, commandMatch: matchCopilotFixtureCommand(fixture["attached-shell"][0].params.update.rawInput.command, call.commandSha256!), expectedCommandSha256: call.commandSha256!, expectedShellId: started.shellId!,
commandExit: { observedAtMs: result.observedAtMs - 1, code: 0, ownedProcessIdentityVerified: true, commandSha256: call.commandSha256! },
nativeShellResult: { ...result, shellId: result.shellId!, commandToolCallId: result.commandToolCallId!, status: result.status!, exitCode: result.exitCode! },
terminal: { observedAtMs: 50, runId: "run", turnId: "turn", status: "succeeded" as const }, cleanup: { observedAtMs: 60, ownedProcessesRemaining: 0 }, terminalMarkerMatches: true, afterCleanupMarkerMatches: true };
expect(gradeCopilotAttachedSettlement(e).passed).toBe(true);
e.terminal.observedAtMs = call.observedAtMs + 1; expect(gradeCopilotAttachedSettlement(e).passed).toBe(false);
e.terminal.observedAtMs = 50; e.nativeShellResult.commandToolCallId = "foreign"; expect(gradeCopilotAttachedSettlement(e).passed).toBe(false);
});
it("rejects foreign, redacted, malformed and explicitly incomplete persisted notices", () => {
for (const mutate of [
(r: any) => { r.payload.prpEvent.runId = "other"; },
(r: any) => { r.payload.prpEvent.payload.provenance.turnId = "old"; },
(r: any) => { r.payload.prpEvent.payload.details.push({ name: "stage", value: "tool" }); },
(r: any) => { r.payload.prpEvent.payload.details[0].value = "evidence_incomplete"; },
(r: any) => { r.payload.prpEvent.payload.details[1].value = "[REDACTED]"; },
]) { const { rows } = projected("attached-shell"); mutate(rows[0]); expect(() => readCopilotToolEvidence(rows, "run")).toThrow(); }
expect(readCopilotToolEvidence([], "run")).toEqual([]); // Missing proof never fabricates an event.
});
it("isolates the denied parent from unrelated workspace startup churn and binds the prompt", async () => {
const root = await mkdtemp("/tmp/pc-denial-isolated-");
const fixture = await createDeniedTargetFixture(root, "copilot-denied-nonce.txt");
try {
const prompt = bindDeniedTargetPrompt(copilotProtectionCases[0].prompt("nonce"), "copilot-denied-nonce.txt", fixture.targetRelativePath);
expect(prompt).toContain(`creating ${fixture.targetRelativePath} with DENIED-nonce`);
expect(fixture.targetRelativePath).toMatch(/^pc-denied-[a-zA-Z0-9]+\/copilot-denied-nonce\.txt$/);
await mkdir(join(root, "startup")); await writeFile(join(root, "startup", "runtime.json"), "{}");
await writeFile(join(root, "transient"), "x"); await unlink(join(root, "transient"));
await new Promise(resolve => setTimeout(resolve, 50));
expect(fixture.watcher.finish()).toMatchObject({ complete: true, targetMutationCount: 0, reasons: [] });
} finally { fixture.watcher.finish(); await rm(root, { recursive: true, force: true }); }
});
it("retains a coverage gap when create/delete occurs before callbacks can arrive", async () => {
const root = await mkdtemp("/tmp/pc-denial-gap-");
const fixture = await createDeniedTargetFixture(root, "denied");
try {
writeFileSync(fixture.targetPath, "x"); unlinkSync(fixture.targetPath);
const receipt = fixture.watcher.finish();
expect(receipt).toMatchObject({ complete: false, targetMutationCount: 0 });
expect(receipt.reasons).toContain("coverage-gap-parent-version-changed");
expect(receipt.finalParent).not.toEqual(receipt.initialParent);
expect(fixture.watcher.finish()).toBe(receipt);
} finally { fixture.watcher.finish(); await rm(root, { recursive: true, force: true }); }
});
it("refuses a preexisting target, symlink parent and ambiguous prompt", async () => {
const root = await mkdtemp("/tmp/pc-denial-invalid-");
try {
await writeFile(join(root, "denied"), "present");
expect(() => watchDeniedTarget(root, "denied")).toThrow(/initially be absent/);
await symlink(root, join(root, "link"));
expect(() => watchDeniedTarget(join(root, "link"), "absent")).toThrow(/real directory/);
expect(() => watchDeniedTarget(root, "../denied")).toThrow(/Invalid/);
expect(() => bindDeniedTargetPrompt("no target", "denied", "pc-denied-a/denied")).toThrow(/exact target once/);
expect(() => bindDeniedTargetPrompt("denied and denied", "denied", "pc-denied-a/denied")).toThrow(/exact target once/);
} finally { await rm(root, { recursive: true, force: true }); }
});
it("observes a transient create/delete even when final stat is absent", async () => {
const root = await mkdtemp("/tmp/pc-copilot-watch-"); const watcher = watchDeniedTarget(root, "denied");
try { await writeFile(join(root, "denied"), "x"); await unlink(join(root, "denied")); await new Promise(r => setTimeout(r, 30)); const proof = watcher.finish(); expect(proof.targetMutationCount > 0 || !proof.complete).toBe(true); expect(await exists(join(root, "denied"))).toBe(false); }
finally { watcher.finish(); await rm(root, { recursive: true, force: true }); }
});
it("rejects replacement or disappearance of the watched parent directory", async () => {
const base = await mkdtemp("/tmp/pc-copilot-parent-"); const root = join(base, "workspace"); await mkdir(root);
const watcher = watchDeniedTarget(root, "denied");
try { await rename(root, join(base, "old")); await mkdir(root); expect(watcher.finish().complete).toBe(false); expect(watcher.finish().reasons).toContain("parent-identity-changed"); }
finally { watcher.finish(); await rm(base, { recursive: true, force: true }); }
const gone = await mkdtemp("/tmp/pc-copilot-parent-"); const deleted = watchDeniedTarget(gone, "denied");
await rm(gone, { recursive: true }); expect(deleted.finish().complete).toBe(false); expect(deleted.finish().reasons).toContain("parent-unavailable-at-finish");
});
it("binds cleanup to the exact per-turn runner PID/start/run, never a warm or reused process", () => {
const startedAt = new Date(1_700_000_000_000).toISOString();
const authority = { pid: 100, groupId: 100, startedAt, runId: "run-id" };
const observed = { pid: 100, parent: 1, start: new Date(startedAt).toString() };
const args = "/private/runner --run-id run-id --lifecycle-mode per_turn";
expect(isPerTurnRunProcess(authority, observed, args)).toBe(true);
for (const bad of [args.replace("per_turn", "warm"), args.replace("run-id run-id", "run-id other"), args + " --run-id run-id", "/server"]) expect(isPerTurnRunProcess(authority, observed, bad)).toBe(false);
expect(isPerTurnRunProcess({ ...authority, groupId: 101 }, observed, args)).toBe(false);
expect(isPerTurnRunProcess(authority, { ...observed, start: new Date(1_700_000_001_000).toString() }, args)).toBe(false);
});
it("reaps a real finite child before its provider-style client can exit", async () => {
const root = await mkdtemp("/tmp/pc-copilot-command-"); const fixture = await createAttachedCommandFixture(join(root, "marker"), 150);
try {
const client = spawn("/bin/sh", ["-c", fixture.command], { stdio: "ignore" });
const code = await new Promise<number | null>(resolve => client.once("exit", resolve)); const observedAtMs = Date.now(); const proof = fixture.snapshot();
expect(code).toBe(0); expect(proof).toMatchObject({ connections: 1, failure: null, childGone: true, clientGone: true });
expect(proof.commandExit?.code).toBe(0); expect(proof.commandExit!.observedAtMs).toBeLessThanOrEqual(observedAtMs);
expect(await readFile(join(root, "marker"), "utf8")).toBe(fixture.marker);
} finally { await fixture.close(); await rm(root, { recursive: true, force: true }); }
});
});
it("rejects remote protection before any API access without bootstrap and pre-teardown authority", async () => {
const { runCopilotProtectionFlow } = await import("./copilot-protection-flow.js");
let calls = 0;
await expect(runCopilotProtectionFlow({ execution: { environment: { id: "daytona" }, profile: { qualificationCandidate: "copilot" }, task: { id: "native-permission-deny-write" } }, api: { get: async () => { calls++; } } } as any)).rejects.toThrow(/bootstrap and pre-teardown/);
expect(calls).toBe(0);
});
tests/runner-e2e/copilot-protection-flow.ts
+282
View File
@@ -0,0 +1,282 @@
import { createHash, randomBytes, randomUUID } from "node:crypto";
import { join } from "node:path";
import { expect, type Page } from "@playwright/test";
import { pollUntil, type RunnerApi } from "./api.js";
import { collectRunEvents } from "./run-observations.js";
import { createTaskThroughUi } from "./user-actions.js";
import { onlyCopilotAttachedOperations, readCopilotSemanticCompletion } from "./copilot-semantic-evidence.js";
import { copilotOrigin, readCopilotToolEvidence, type CopilotToolNotice } from "./copilot-evidence.js";
import { createAttachedCommandFixture, createDeniedTargetFixture, bindDeniedTargetPrompt, exists, observeRunProcesses } from "./copilot-local-fixtures.js";
import { gradeCopilotAttachedSettlement, gradeCopilotDeniedWrite, type CopilotDeniedWriteEvidence } from "./copilot-protection-cases.js";
import { observeCopilotPreStop, type CopilotPreStopObservation, readCopilotDeniedEdit, copilotDenialSampleCursor, readCopilotDenialSettlement, observeCopilotFixtureCommand, readCopilotRemoteMarkerAfterRetirement, prepareCopilotRemoteAction, assertCopilotRemoteRetirement, assertCopilotRemoteAttached, copilotRemoteDeniedSample, copilotActionNotices, type CopilotRemoteBootstrap, type CopilotRemoteFixture, type CopilotRemoteSnapshot, countCopilotToolOrigins, countCopilotEditOriginsForTarget } from "./copilot-protection-evidence.js";
import type { LiveFixtureValues } from "./live-fixtures.js";
import type { MatrixExecution } from "./types.js";
type Row = Record<string, any>;
type Check = { id: string; passed: boolean; detail: string };
/** Each persistence barrier reloads durable rows. Stop cannot overtake the
* failed edit, and a terminal sample cannot use a pre-terminal event cursor. */
export async function settleCopilotDeniedRun(input: {
api: Pick<RunnerApi, "post">; request: CopilotToolNotice; deadlineAt: number;
load(): Promise<{ events: readonly unknown[]; run: Row; issue: Row; retired: boolean }>;
afterDeniedEdit(): Promise<void>;
retainPreStop(receipt: CopilotPreStopObservation): Promise<void>;
afterSettlement(): Promise<void>;
}) {
const cancellationRequestId = randomUUID();
const assertBeforeStop = (state: Awaited<ReturnType<typeof input.load>>) => {
if (state.run.id !== input.request.runId || state.run.status !== "running"
|| state.run.resultJson?.startupCancellation != null || state.run.resultJson?.nativeCancellation != null) {
throw new Error("Copilot denial observed an earlier Stop or non-running run");
}
};
let stopSent = false;
const poll = (label: string, accept: (state: Awaited<ReturnType<typeof input.load>>) => boolean) => pollUntil({
label, deadlineAt: input.deadlineAt, load: async () => {
const state = await input.load();
// pollUntil recognizes this definitive rejection before invoking readers.
if (["failed", "timed_out"].includes(state.run.status)) throw new Error(`Stopped waiting for ${label}: Copilot provider run failed`);
if (!stopSent) {
try { assertBeforeStop(state); }
catch { throw new Error(`Stopped waiting for ${label}: Copilot denial observed an earlier Stop or non-running run`); }
}
return state;
}, accept, intervalMs: 200,
});
await poll("persisted correlated failed native edit", state => {
readCopilotDeniedEdit({ events: state.events, request: input.request, companyId: state.issue.companyId }); return true;
});
await input.afterDeniedEdit();
// This negative case does not qualify active-turn cancellation. Give natural
// settlement a fixed observation window inside the unchanged case deadline.
const observeUntil = Math.min(input.deadlineAt, Date.now() + 2000);
let preStop: CopilotPreStopObservation;
while (true) {
const state = await input.load();
if (["failed", "timed_out"].includes(state.run.status)) throw new Error("Copilot provider run failed before Stop");
assertBeforeStop(state);
preStop = observeCopilotPreStop({ events: state.events, request: input.request, companyId: state.issue.companyId, cancellationRequestId });
if (preStop.terminal || Date.now() >= observeUntil) break;
await new Promise(resolve => setTimeout(resolve, Math.min(200, observeUntil - Date.now())));
}
if (Date.now() >= input.deadlineAt) throw new Error("Copilot denial deadline reached before Stop");
// Await the artifact write before dispatch. Database createdAt cannot supply
// this causal boundary, and a later replay must never manufacture it.
await input.retainPreStop(preStop);
if (Date.now() >= input.deadlineAt) throw new Error("Copilot denial deadline reached before Stop");
assertBeforeStop(await input.load());
if (Date.now() >= input.deadlineAt) throw new Error("Copilot denial deadline reached before Stop");
const stopDispatchMonotonicNs = process.hrtime.bigint().toString();
const stopped = await input.api.post<Row>(`/api/heartbeat-runs/${input.request.runId}/cancel`, { cancellationRequestId });
if (stopped?.id !== input.request.runId || stopped?.resultJson?.nativeCancellation?.intentId !== `native-cancellation:${cancellationRequestId}`) {
throw new Error("Copilot denial Stop response has a foreign cancellation intent");
}
stopSent = true;
await poll("correlated provider settlement and retired run", state => {
if (!state.retired) return false;
readCopilotDenialSettlement({ ...state, request: input.request, preStop, stopDispatchMonotonicNs }); return true;
});
await input.afterSettlement();
return readCopilotDenialSettlement({ ...await input.load(), request: input.request, preStop, stopDispatchMonotonicNs });
}
export async function runCopilotProtectionFlow(input: {
page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution; nonce: string; workspacePath: string; deadlineAt: number;
remoteBootstrap?: CopilotRemoteBootstrap;
registerBeforeEnvironmentTeardownAssertion?(callback: () => Promise<Check[]>): void;
observe(issue: Row, runs: Row[]): void; capture(id: string, label: string, file: string): Promise<void>; evidence(name: string, data: unknown): Promise<void>;
}) {
const { page, api, fixtures, execution, nonce, workspacePath } = input;
const remote = execution.environment.id === "daytona";
if ((!remote && execution.environment.id !== "local") || execution.profile.qualificationCandidate !== "copilot") throw new Error("Copilot protection fixtures require an isolated candidate");
if (remote && (!input.remoteBootstrap || !input.registerBeforeEnvironmentTeardownAssertion)) throw new Error("Remote Copilot protection requires bootstrap and pre-teardown evidence hooks");
const deny = execution.task.id === "native-permission-deny-write";
if (!deny && execution.task.id !== "attached-async-settlement") throw new Error("Unknown Copilot protection case");
const checks: Check[] = []; let issue: Row = {}, runs: Row[] = [], runEvents: Row[] = [];
let notices: CopilotToolNotice[] = [];
const processObserver = remote ? undefined : observeRunProcesses();
let processes: { captured: boolean; live: number[] } = processObserver?.sample() ?? { captured: false, live: [] };
let remoteFixture: CopilotRemoteFixture | undefined, baseline: CopilotRemoteSnapshot | undefined, sealed: CopilotRemoteSnapshot | undefined;
let remoteCommand: { command: string; commandSha256: string } | undefined;
const remoteMarker = `${randomBytes(24).toString("hex")}\n`;
const remoteSnapshots: CopilotRemoteSnapshot[] = [];
async function sealRemote() {
if (!remoteFixture || !baseline) throw new Error("Remote Copilot evidence was never armed");
sealed ??= await remoteFixture.finish();
assertCopilotRemoteRetirement(sealed, baseline); processes = sealed.processes;
return sealed;
}
const targetName = `copilot-denied-${nonce}.txt`;
const localTarget = deny && !remote ? await createDeniedTargetFixture(workspacePath, targetName) : undefined;
const target = localTarget?.targetRelativePath ?? targetName, targetPath = localTarget?.targetPath ?? join(workspacePath, target);
const fileObservations: CopilotDeniedWriteEvidence["fileObservations"] = [];
let deniedRequest: CopilotToolNotice | undefined;
const sample = async (phase: CopilotDeniedWriteEvidence["fileObservations"][number]["phase"]) => {
const providerCursor = phase === "before-request" ? null : copilotDenialSampleCursor(runEvents, deniedRequest!);
if (remote) {
if (!remoteFixture || !baseline) throw new Error("Remote denied target has no baseline");
const snapshot = sealed ?? (phase === "before-request" ? baseline : await remoteFixture.snapshot(phase));
remoteSnapshots.push(snapshot); fileObservations.push({ ...copilotRemoteDeniedSample(snapshot, baseline, target, phase), providerCursor });
} else fileObservations.push({ phase, observedAtMs: Date.now(), exists: await exists(targetPath), providerCursor });
};
const watcher = localTarget?.watcher;
const markerPath = join(workspacePath, `copilot-settlement-${nonce}.txt`);
const command = deny || remote ? undefined : await createAttachedCommandFixture(markerPath);
const exactCommand = () => remoteCommand ?? command;
let watchReceipt: CopilotDeniedWriteEvidence["mutationObservation"] | undefined;
const check = (id: string, passed: boolean, detail: string) => { checks.push({ id, passed, detail }); expect(passed, detail).toBe(true); };
async function load() {
if (issue.id) issue = await api.get<Row>(`/api/issues/${issue.id}`);
const listed = await api.get<Row[]>(`/api/companies/${fixtures.company.id}/heartbeat-runs?limit=100`);
runs = await Promise.all(listed.map(r => api.get<Row>(`/api/heartbeat-runs/${r.id}`)));
if (runs.length > 1) throw new Error("Copilot protection case dispatched an extra run");
input.observe(issue, runs);
runEvents = runs[0] ? await collectRunEvents<Row>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${runs[0]!.id}/events?afterSeq=${afterSeq}&limit=${limit}`)) : [];
notices = runs[0] ? readCopilotToolEvidence(runEvents, runs[0].id) : [];
const run = runs[0];
if (processObserver) processes = processObserver.sample(run?.processPid ? { pid: run.processPid, groupId: run.processGroupId, startedAt: run.processStartedAt, runId: run.id } : undefined);
return { issue, runs, runEvents, notices, processes };
}
const wait = (label: string, accept: (state: Awaited<ReturnType<typeof load>>) => boolean) => pollUntil({ label, deadlineAt: input.deadlineAt, load, accept, intervalMs: 200,
reject: state => state.runs.some(r => ["failed", "timed_out"].includes(r.status)) ? "Copilot provider run failed" : undefined });
try {
const agent = await api.get<Row>(`/api/agents/${fixtures.agent.id}`);
await api.patch(`/api/agents/${fixtures.agent.id}`, { adapterConfig: { ...agent.adapterConfig, acpxPermissionMode: deny ? "approve-reads" : "approve-all", timeoutSec: 120, lifecycleMode: "per_turn" } });
check("per-turn-process-lifecycle", (await api.get<Row>(`/api/agents/${fixtures.agent.id}`)).adapterConfig?.lifecycleMode === "per_turn", "This case explicitly requires a per-turn runner process, never a retained warm daemon");
const project = await api.post<Row>(`/api/companies/${fixtures.company.id}/projects`, {
name: `Copilot protection ${nonce}`, executionWorkspacePolicy: { enabled: true, defaultMode: "shared_workspace", sharedWorkspaceConcurrency: "serialize", allowIssueOverride: false, environmentId: fixtures.environment.id, workspaceStrategy: { type: "project_primary" } },
workspace: { name: "Primary", sourceType: "local_path", cwd: workspacePath, isPrimary: true },
});
if (deny && !remote) { await sample("before-request"); check("target-initially-absent", !fileObservations[0]!.exists, "The exact isolated target is absent before dispatch"); }
const prompt = remote ? input.remoteBootstrap!.prompt(nonce) : `${localTarget ? bindDeniedTargetPrompt(execution.task.buildPrompt(nonce), targetName, target) : execution.task.buildPrompt(nonce)}${command ? `\nThe exact supplied command is:\n${command.command}\nDo not inspect or modify fixture code, fabricate its marker, or launch a substitute command.` : ""}`;
await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), prompt, workMode: "standard", projectName: project.name });
const found = await pollUntil({ label: "browser-created Copilot protection task", deadlineAt: input.deadlineAt, load: async () => (await api.get<Row[]>(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(r => r.title === execution.task.buildTitle(nonce)), accept: Boolean });
if (!found) throw new Error("Browser-created task was not found"); issue = found;
await page.goto(`/${fixtures.company.issuePrefix}/issues/${issue.identifier ?? issue.id}`);
if (remote) {
await wait("exact remote bootstrap run", state => state.runs.length === 1 && state.runs[0]?.status === "running");
const bound = await input.remoteBootstrap!.bindAndRelease({ issueId: issue.id, runId: runs[0]!.id,
targets: [deny ? target : `copilot-settlement-${nonce}.txt`],
actionPrompt: async fixture => {
remoteFixture = fixture;
const prepared = await prepareCopilotRemoteAction({ fixture, companyId: fixtures.company.id, environmentId: fixtures.environment.id, runId: runs[0]!.id,
target: deny ? target : `copilot-settlement-${nonce}.txt`, prompt: execution.task.buildPrompt(nonce), ...(deny ? {} : { markerText: remoteMarker }) });
baseline = prepared.baseline; remoteCommand = prepared.command;
if (deny) { await sample("before-request"); check("target-initially-absent", !fileObservations[0]!.exists, "The actual remote target is absent before action publication"); }
await input.evidence("copilot-remote-baseline.json", baseline);
return prepared.prompt;
} });
if (bound !== remoteFixture) throw new Error("Remote Copilot fixture changed during publication");
input.registerBeforeEnvironmentTeardownAssertion!(async () => {
try {
const receipt = await sealRemote();
if (deny) {
if (copilotRemoteDeniedSample(receipt, baseline!, target, "after-cleanup").exists) throw new Error("Remote denied target exists after retirement");
}
else {
if (!await readCopilotRemoteMarkerAfterRetirement(remoteFixture!, baseline!, `copilot-settlement-${nonce}.txt`, remoteMarker)) throw new Error("Remote sealed marker changed or disappeared");
}
await input.evidence("copilot-remote-pre-teardown.json", receipt);
return [{ id: "remote-sealed-retirement", passed: true, detail: "Exact remote run root and descendants retired; retained pre-deletion filesystem receipt validated" }];
} finally { await remoteFixture!.close(); }
});
}
if (deny) {
await wait("exact native write permission", s => s.notices.some(n => n.stage === "permission_requested" && n.operation === "edit" && n.target === target && n.declineOffered && s.runEvents.some(r => r.eventType === "runtime_request.created" && r.payload?.prpEvent?.payload?.request?.requestId === n.requestId)));
const request = notices.find(n => n.stage === "permission_requested" && n.operation === "edit" && n.target === target)!;
deniedRequest = request;
const pending = runEvents.filter(r => r.eventType === "runtime_request.created" && r.payload?.prpEvent?.payload?.request?.requestId === request.requestId).map(r => r.payload.prpEvent.payload.request);
check("one-bound-permission", pending.length === 1 && pending[0].requestKind === "permission_approval" && pending[0].origin?.method === "session/request_permission", "The notice maps to the exact durable native permission card");
const retired = new Set(runEvents.filter(r => ["runtime_request.resolved", "runtime_request.cancelled", "runtime_request.expired"].includes(r.eventType)).map(r => r.payload?.prpEvent?.payload?.requestId));
const activeRequests = runEvents.filter(r => r.eventType === "runtime_request.created" && r.payload?.prpEvent?.payload?.request && !retired.has(r.payload.prpEvent.payload.request.requestId));
check("only-one-pending-native-request", activeRequests.length === 1, "Only the exact denied edit is awaiting a decision");
await sample("pending"); await page.reload();
const card = page.getByTestId("task-chat-runtime-request").filter({ visible: true });
await expect(card).toHaveCount(1); await input.capture("permission-pending", "Copilot native write awaiting denial", "permission-pending.png");
const url = `/api/heartbeat-runs/${request.runId}/runtime-requests/${encodeURIComponent(request.requestId!)}/resolve`;
const sent = page.waitForRequest(r => new URL(r.url()).pathname === url && r.method() === "POST");
const clickedAtMs = Date.now(); await card.getByRole("button", { name: "Deny", exact: true }).click();
const posted = (await sent).postDataJSON();
check("browser-exact-denial", posted.turnId === request.turnId && posted.requestKind === "permission_approval" && posted.resolution?.action === "decline", "Browser submitted denial for the exact run/request/turn");
const settlement = await settleCopilotDeniedRun({ api, request, deadlineAt: input.deadlineAt,
load: async () => {
const state = await load();
return { events: state.runEvents, run: state.runs[0]!, issue: state.issue,
retired: remote || (state.processes.captured && state.processes.live.length === 0) };
},
afterDeniedEdit: () => sample("after-decision"),
retainPreStop: receipt => input.evidence("copilot-pre-stop-observation.json", receipt),
afterSettlement: async () => {
if (remote) await sealRemote();
await sample("terminal"); await new Promise(resolve => setTimeout(resolve, 100)); await load(); await sample("after-cleanup");
},
});
watchReceipt = remote ? { startedAtMs: baseline!.observedAtMs, endedAtMs: sealed!.observedAtMs, complete: sealed!.watcher.complete, targetMutationCount: sealed!.watcher.targetMutationCount } : watcher!.finish();
const toolResult = notices.find(n => n.stage === "tool" && n.toolCallId === request.toolCallId && n.status === "failed")!;
await input.evidence("copilot-denial-settlement.json", settlement);
check("correlated-provider-settlement", true, `Exact provider settlement (${settlement.branch}) and audited run Stop; completed does not cover active-turn cancellation`);
const terminalAt = settlement.providerTerminal.emittedAtMs;
const evidence: CopilotDeniedWriteEvidence = {
expected: copilotOrigin(request), requestId: request.requestId!, expectedRelativePath: target,
request: { ...request, requestId: request.requestId!, targetRelativePath: request.target!, method: "session/request_permission", offeredActions: request.declineOffered ? ["decline"] : [] },
decision: { ...request, observedAtMs: clickedAtMs, requestId: request.requestId!, browserRequestId: request.requestId!, action: "decline" },
deliveredDecision: (() => { const n = notices.find(n => n.stage === "permission_delivered" && n.requestId === request.requestId && n.outcome === "reject_once"); return n ? { ...n, requestId: n.requestId!, outcome: n.outcome! } : null; })(),
toolResult: { ...toolResult, status: "failed" },
terminal: { runId: settlement.runId, turnId: settlement.turnId, observedAtMs: terminalAt, status: runs[0]!.status },
settlement,
cleanup: { observedAtMs: fileObservations.at(-1)!.observedAtMs, ownedProcessesRemaining: processes.live.length }, fileObservations, mutationObservation: watchReceipt,
nativeAttemptsForTarget: countCopilotEditOriginsForTarget(notices, target),
};
await input.evidence("copilot-denial-proof.json", { evidence, processes, notices });
const grade = gradeCopilotDeniedWrite(evidence); check("denial-without-side-effects", grade.passed, grade.failures.join(", ") || "Exact browser denial, explicit cancellation and absence through process cleanup");
check("negative-task-unfinished", issue.status === "in_progress", "The negative test does not claim the task is done");
check("no-extra-native-operation", countCopilotToolOrigins(copilotActionNotices(notices, notices.find(n => n.stage === "tool" && n.toolCallId === request.toolCallId)!, remoteFixture ? { actionFile: remoteFixture.actionFile, events: runEvents } : undefined)) === 1, "No alternate native edit, command or delegated operation is permitted");
} else {
await wait("attached command and task settlement", s => s.issue.status === "done" && s.runs[0]?.status === "succeeded" && (remote || (s.processes.captured && s.processes.live.length === 0)));
// Preserve independent local proof before any command matcher can abort.
const { external: localExternal, markerMatches: localMarkerMatches, afterCleanupMarkerMatches: localAfterCleanupMarkerMatches, matched } =
await observeCopilotFixtureCommand(notices, exactCommand()!.command, command ? { fixture: command, markerPath } : undefined);
await input.evidence("copilot-attached-command-observation.json", { notices, processes, external: localExternal,
canonicalCommandSha256: exactCommand()!.commandSha256, commandMatch: matched?.match ?? null,
markerMatches: localMarkerMatches, afterCleanupMarkerMatches: localAfterCleanupMarkerMatches });
const call = matched?.call;
check("single-exact-command", Boolean(matched), "Exactly one native execution matches the fixture command with at most eight leading ASCII SPACE/TAB bytes");
const semantic = readCopilotSemanticCompletion(runEvents, { companyId: fixtures.company.id, runId: call!.runId, turnId: call!.turnId,
nativeSessionId: call!.sessionId, command: call!, summary: execution.task.buildVisibleMarker(nonce) });
await input.evidence("copilot-semantic-completion.json", semantic);
check("accepted-canonical-completion", true, "One exact native finish receipt matches the proposed and control-plane accepted result; transport success alone is insufficient");
check("no-extra-native-operation", onlyCopilotAttachedOperations(copilotActionNotices(notices, call!, remoteFixture ? { actionFile: remoteFixture.actionFile, events: runEvents } : undefined), call!, semantic), "Only attested setup reads, the exact attached command/result, and one authoritatively correlated accepted finish are allowed");
const started = notices.find(n => n.toolCallId === call!.toolCallId && n.shellState === "started");
const result = notices.find(n => n.commandToolCallId === call!.toolCallId && n.shellState === "completed");
const terminal = runEvents.find(r => r.eventType === "turn.completed" && r.payload?.prpEvent?.turnId === call!.turnId)?.payload.prpEvent;
if (remote) await sealRemote();
const remoteAttached = remote ? assertCopilotRemoteAttached(sealed!, baseline!, terminal ? Date.parse(terminal.emittedAt) : NaN) : undefined;
const external = remoteAttached ? { ...remoteAttached, childGone: true, clientGone: true,
commandExit: { ...remoteAttached.commandExit!, ownedProcessIdentityVerified: true, commandSha256: exactCommand()!.commandSha256 } } : localExternal!;
check("trusted-command-exit", !external.failure && external.connections === 1 && external.childGone && external.clientGone, "Fixed controller-owned child exited and its native client is gone");
const markerMatches = remote ? sealed!.targets[`copilot-settlement-${nonce}.txt`]?.sha256 === `sha256:${createHash("sha256").update(remoteMarker).digest("hex")}` : localMarkerMatches!;
check("native-client-before-terminal", Boolean(terminal) && external.clientExitedAtMs !== null && external.clientExitedAtMs < Date.parse(terminal.emittedAt), "Independent PID/start observation confirms native client retirement before turn completion");
check("marker-before-terminal", Boolean(terminal) && external.markerWrittenAtMs !== null && external.markerWrittenAtMs < Date.parse(terminal.emittedAt), "The independent fixture wrote its undisclosed marker before turn completion");
const afterCleanupMarkerMatches = remote ? await readCopilotRemoteMarkerAfterRetirement(remoteFixture!, baseline!, `copilot-settlement-${nonce}.txt`, remoteMarker) : localAfterCleanupMarkerMatches!;
const grade = gradeCopilotAttachedSettlement({ expected: copilotOrigin(call!), nativeCall: call ? { ...call, operation: call.operation!, mode: call.mode!, detach: call.detach!, commandSha256: call.commandSha256! } : null,
expectedCommand: exactCommand()!.command, expectedCommandSha256: exactCommand()!.commandSha256, commandMatch: matched!.match, commandExit: external.commandExit,
expectedShellId: started?.shellId ?? "", nativeShellResult: result ? { ...result, shellId: result.shellId!, commandToolCallId: result.commandToolCallId!, status: result.status!, exitCode: result.exitCode! } : null,
terminal: terminal ? { observedAtMs: Date.parse(terminal.emittedAt), runId: terminal.runId, turnId: terminal.turnId, status: "succeeded" } : null,
cleanup: { observedAtMs: remote ? sealed!.observedAtMs : Date.now(), ownedProcessesRemaining: processes.live.length }, terminalMarkerMatches: markerMatches, afterCleanupMarkerMatches });
await input.evidence("copilot-attached-proof.json", { external, processes, notices, grade, commandSha256: exactCommand()!.commandSha256, commandMatch: matched!.match, markerMatches, afterCleanupMarkerMatches });
check("attached-settlement-before-terminal", grade.passed, grade.failures.join(", ") || "Owned finite process and native shell settled before the actual turn terminal");
}
await load(); check("one-native-run", runs.length === 1 && runs[0]!.runtimeMode === "native", "Exactly one native run was accounted");
const comments = await api.get<Row[]>(`/api/issues/${issue.id}/comments`), interactions = await api.get<Row[]>(`/api/issues/${issue.id}/interactions`);
await input.evidence("api-state.json", { issue, run: runs[0], runs, comments, interactions, checks, runEvents, runEventsByRun: [{ runId: runs[0]!.id, events: runEvents }] });
await page.reload();
const label = deny ? "In Progress" : "Done";
await expect(page.getByTestId("issue-detail-header").getByRole("button", { name: `Change status (current: ${label})`, exact: true })).toBeVisible();
if (!deny) check("exact-completion-marker", comments.filter(c => c.authorAgentId === fixtures.agent.id && c.body?.trim() === execution.task.buildVisibleMarker(nonce)).length === 1, "The successful attached task has exactly one terminal marker");
await input.capture("final-state", "Copilot protection outcome", "final-state.png");
return { issue, runs, checks };
} finally {
watchReceipt ??= watcher?.finish();
try { await command?.close(); }
finally { await input.evidence("copilot-protection-checks.json", { issue, runs, checks, fileObservations, watchReceipt, processes, remoteSnapshots, sealed }); }
}
}
tests/runner-e2e/copilot-protection-settlement.test.ts
+249
View File
@@ -0,0 +1,249 @@
import { describe, expect, it, vi } from "vitest";
import { readCopilotToolEvidence } from "./copilot-evidence.js";
import { observeCopilotPreStop, copilotDenialSampleCursor, readCopilotDenialSettlement } from "./copilot-protection-evidence.js";
import { settleCopilotDeniedRun } from "./copilot-protection-flow.js";
// Sanitized source order from the failed Daytona attempt. Its .900 createdAt
// is transaction time, not commit evidence. Pre-Stop receipts below are synthetic
// test observations; the historical paid attempt did not capture one.
const date = (ms: number) => new Date(Date.UTC(2026, 8, 30, 15, 13, 7, ms)).toISOString();
function fixture(eventType = "turn.completed") {
const frame = (seq: number, type: string, payload: unknown, persisted = 305, emitted = 129): any => ({
seq: seq + 40, companyId: "company", runId: "run", eventType: type, createdAt: date(persisted),
payload: { prpEvent: { schema: "paperclip.prp.event.v1", sourceKind: "runner", runId: "run", turnId: "turn",
normalizedSessionId: "normalized-session", sourceInstanceId: "runner-instance", sourceSeq: seq,
sourceEventId: `runner-instance:run:${seq}`, eventType: type, emittedAt: date(emitted), payload } },
});
const notice = (seq: number, stage: string, details: Record<string, string>) => frame(seq, "provider.notice.recorded", {
schema: "paperclip.provider.notice.v1", category: "copilot_tool_evidence_v1", scope: "turn",
provenance: { sessionId: "native-session", turnId: "turn", eventType: stage, method: stage === "tool" ? "session/update" : "session/request_permission" },
details: Object.entries({ stage, toolCallId: "denied-edit", operation: "edit", ...details }).map(([name, value]) => ({ name, value })),
});
const events = [notice(111, "permission_requested", { requestId: "request", target: "copilot-denied-nonce.txt", declineOffered: "true" }),
notice(113, "permission_delivered", { requestId: "request", outcome: "reject_once" }), notice(114, "tool", { status: "failed" }),
frame(115, "tool.execution.completed", { schema: "paperclip.tool.execution.v1", executionId: "denied-edit", operation: "edit", status: "failed" }, 317),
frame(121, eventType, { status: eventType.slice(5), error: null }, 900, 887),
frame(112, "runtime_request.resolved", { requestId: "request", turnId: "turn", requestKind: "permission_approval", action: "decline" })];
const run: any = { id: "run", companyId: "company", nativeIssueId: "issue", status: "cancelled", finishedAt: date(910),
resultJson: { startupCancellation: { requestedAt: date(892), cancellationRequestId: "11111111-1111-4111-8111-111111111111" }, nativeCancellation: {
schema: "paperclip.native-cancellation.v1", companyId: "company", runId: "run", issueId: "issue", scope: "run", dispatched: true,
dispatchState: "acknowledged", reasonCode: "cancellation_run_only", effects: ["release_run_resources"], intentId: "native-cancellation:11111111-1111-4111-8111-111111111111",
intentAuditId: "intent-audit", acknowledgementAuditId: "ack-audit", recordedAt: date(901), acknowledgedAt: date(909),
} } };
const request = readCopilotToolEvidence(events, "run")[0]!;
const preStop = observeCopilotPreStop({ events, request, companyId: "company", cancellationRequestId: "11111111-1111-4111-8111-111111111111" });
return { events, request, run, issue: { id: "issue", companyId: "company", status: "in_progress" }, preStop, stopDispatchMonotonicNs: process.hrtime.bigint().toString() };
}
const terminal = (f: ReturnType<typeof fixture>) => f.events[4]!.payload.prpEvent;
const liveRun = (f: ReturnType<typeof fixture>) => ({ ...f.run, status: "running", resultJson: {} });
function stopPost(f: ReturnType<typeof fixture>, callback = () => {}) {
return vi.fn(async (_path: string, body: { cancellationRequestId: string }) => {
callback();
f.run.resultJson.startupCancellation.cancellationRequestId = body.cancellationRequestId;
f.run.resultJson.nativeCancellation.intentId = `native-cancellation:${body.cancellationRequestId}`;
return f.run;
});
}
describe("Copilot denial provider settlement and separate audited run Stop", () => {
it("accepts a synthetic pre-Stop API observation without claiming provider cancellation", () => {
const f = fixture();
expect(readCopilotDenialSettlement(f)).toMatchObject({ branch: "provider_completed_observed_before_stop", providerCancellationTerminalObserved: false,
providerTerminal: { sourceSeq: 121, failedToolSourceSeq: 115, rowCreatedAtMs: Date.parse(date(900)) }, runStop: { status: "cancelled" } });
// Native/provider clock skew is irrelevant to the operator causal boundary.
terminal(f).emittedAt = date(999);
f.preStop = observeCopilotPreStop({ ...f, companyId: "company", cancellationRequestId: "11111111-1111-4111-8111-111111111111" }); f.stopDispatchMonotonicNs = process.hrtime.bigint().toString();
expect(readCopilotDenialSettlement(f).branch).toBe("provider_completed_observed_before_stop");
});
it.each(["turn.cancelled", "turn.interrupted"])("retains the distinct observed %s branch", type => {
expect(readCopilotDenialSettlement(fixture(type))).toMatchObject({ branch: "provider_cancelled_or_interrupted", providerCancellationTerminalObserved: true });
});
it("cannot use a legacy summary or cancelled run as provider evidence", () => {
const f = fixture(); f.events.splice(4, 1); f.events.push({ eventType: "turn.completed", payload: {} });
expect(() => readCopilotDenialSettlement(f)).toThrow(/settlement/);
});
it.each(["equal", "later"])("accepts observed pre-Stop completion with %s transaction time, without ordering it against ack", kind => {
const f = fixture(); f.events[4]!.createdAt = date(kind === "equal" ? 909 : 910);
f.preStop = observeCopilotPreStop({ ...f, companyId: "company", cancellationRequestId: "11111111-1111-4111-8111-111111111111" }); f.stopDispatchMonotonicNs = process.hrtime.bigint().toString();
expect(readCopilotDenialSettlement(f).branch).toBe("provider_completed_observed_before_stop");
});
it("rejects late normal completion even when its transaction began before Stop ack", () => {
const f = fixture(); f.preStop.terminal = null; f.events[4]!.createdAt = date(800);
expect(() => readCopilotDenialSettlement(f)).toThrow(/settlement/);
});
it.each(["missing", "foreign", "hash", "sequence", "equal-clock", "reverse-clock", "non-string-clock"])("rejects %s pre-dispatch observation", kind => {
const f = fixture();
if (kind === "missing") (f as any).preStop = null;
if (kind === "foreign") f.preStop.runId = "foreign";
if (kind === "hash") f.preStop.terminal!.rowSha256 = `sha256:${"a".repeat(64)}`;
if (kind === "sequence") f.preStop.terminal!.sourceSeq++;
if (kind === "equal-clock") f.stopDispatchMonotonicNs = f.preStop.apiReadCompletedMonotonicNs;
if (kind === "reverse-clock") f.stopDispatchMonotonicNs = "1";
if (kind === "non-string-clock") (f.preStop as any).apiReadCompletedMonotonicNs = ["1"];
expect(() => readCopilotDenialSettlement(f)).toThrow(/settlement/);
});
it.each(["runId", "turnId", "normalizedSessionId", "sourceInstanceId", "eventType", "sourceEventId"])("rejects terminal %s mismatch", field => {
const f = fixture(); terminal(f)[field] = "foreign";
expect(() => readCopilotDenialSettlement(f)).toThrow(/settlement/);
});
it.each(["missing", "duplicate", "failed", "before-tool", "wrong-status", "error"])("rejects %s terminal", kind => {
const f = fixture();
if (kind === "missing") f.events.splice(4, 1);
if (kind === "duplicate") f.events.push(structuredClone(f.events[4]));
if (kind === "failed") { f.events[4]!.eventType = terminal(f).eventType = "turn.failed"; terminal(f).payload.status = "failed"; }
if (kind === "before-tool") { terminal(f).sourceSeq = 114; terminal(f).sourceEventId = "runner-instance:run:114"; }
if (kind === "wrong-status") terminal(f).payload.status = "cancelled";
if (kind === "error") terminal(f).payload.error = { code: "failed" };
expect(() => readCopilotDenialSettlement(f)).toThrow(/settlement/);
});
it.each(["scope", "runId", "companyId", "issueId", "intentId", "intentAuditId", "acknowledgementAuditId", "acknowledgedAt", "recordedAt", "schema", "dispatchState", "dispatched", "reasonCode", "effects"])("rejects malformed or unbound Stop %s", field => {
const f = fixture(); f.run.resultJson.nativeCancellation[field] = null;
expect(() => readCopilotDenialSettlement(f)).toThrow(/settlement/);
});
it.each(["missing-stop", "pending", "wrong-scope", "same-audit", "wrong-run-status", "issue-done", "ack-before-intent"])("rejects %s controller state", kind => {
const f = fixture(), c = f.run.resultJson.nativeCancellation;
if (kind === "missing-stop") delete f.run.resultJson.nativeCancellation;
if (kind === "pending") c.dispatchState = "pending";
if (kind === "wrong-scope") c.scope = "turn";
if (kind === "same-audit") c.acknowledgementAuditId = c.intentAuditId;
if (kind === "wrong-run-status") f.run.status = "succeeded";
if (kind === "issue-done") f.issue.status = "done";
if (kind === "ack-before-intent") c.recordedAt = date(911);
expect(() => readCopilotDenialSettlement(f)).toThrow(/settlement/);
});
it.each(["native-session", "normalized-session", "failed-tool", "delivery", "duplicate-tool", "tool-order"])("rejects %s corruption", kind => {
const f = fixture();
if (kind === "native-session") f.events[2]!.payload.prpEvent.payload.provenance.sessionId = "foreign";
if (kind === "normalized-session") f.events[2]!.payload.prpEvent.normalizedSessionId = "foreign";
if (kind === "failed-tool") f.events[3]!.payload.prpEvent.payload.status = "completed";
if (kind === "delivery") f.events[1]!.payload.prpEvent.payload.details.find((d: any) => d.name === "outcome").value = "allow_once";
if (kind === "duplicate-tool") f.events.push(structuredClone(f.events[3]));
if (kind === "tool-order") f.events[3]!.payload.prpEvent.sourceSeq = 122;
expect(() => readCopilotDenialSettlement(f)).toThrow();
});
it.each(["missing", "duplicate", "foreign-turn", "accept", "expired", "after-delivery"])("rejects %s durable resolution", kind => {
const f = fixture(), r = f.events[5]!, p = r.payload.prpEvent;
if (kind === "missing") f.events.pop();
if (kind === "duplicate") f.events.push(structuredClone(r));
if (kind === "foreign-turn") p.payload.turnId = "foreign";
if (kind === "accept") p.payload.action = "accept";
if (kind === "expired") r.eventType = p.eventType = "runtime_request.expired";
if (kind === "after-delivery") { p.sourceSeq = 114; p.sourceEventId = "runner-instance:run:114"; }
expect(() => readCopilotDenialSettlement(f)).toThrow();
});
it("captures the exact durable stream at the sampling boundary without clock conversion", () => {
const f = fixture();
expect(copilotDenialSampleCursor([f.events[0]], f.request)).toEqual({ runId: "run", turnId: "turn", normalizedSessionId: "normalized-session", sourceInstanceId: "runner-instance", sourceSeq: 111 });
expect(copilotDenialSampleCursor(f.events, f.request).sourceSeq).toBe(121);
f.events[1]!.payload.prpEvent.normalizedSessionId = "foreign";
expect(() => copilotDenialSampleCursor(f.events, f.request)).toThrow(/cursor/);
});
it("awaits exact pre-Stop receipt retention and waits for post-Stop cancelled terminal before sampling", async () => {
vi.useFakeTimers(); let mono = 100n; vi.spyOn(process.hrtime, "bigint").mockImplementation(() => ++mono);
try {
const f = fixture("turn.cancelled"); let loads = 0, dispatched = false, postLoads = 0;
let release!: () => void; const retention = new Promise<void>(resolve => { release = resolve; });
const post = stopPost(f, () => { dispatched = true; });
const retainPreStop = vi.fn(async (receipt) => { expect(receipt.terminal).toBeNull(); await retention; });
const afterDeniedEdit = vi.fn(async () => {}), afterSettlement = vi.fn(async () => {});
const result = settleCopilotDeniedRun({ api: { post } as any, request: f.request, deadlineAt: Date.now() + 5000,
load: async () => {
loads++; if (dispatched) postLoads++;
const events = f.events.filter((_, i) => !(loads === 1 && i === 3) && !((!dispatched || postLoads === 1) && i === 4));
return { ...f, events, run: dispatched ? f.run : liveRun(f), retired: dispatched };
}, afterDeniedEdit, retainPreStop, afterSettlement });
await vi.advanceTimersByTimeAsync(0); expect(post).not.toHaveBeenCalled(); expect(afterDeniedEdit).not.toHaveBeenCalled();
await vi.advanceTimersByTimeAsync(200); expect(afterDeniedEdit).toHaveBeenCalledOnce(); expect(post).not.toHaveBeenCalled();
await vi.advanceTimersByTimeAsync(2000); expect(retainPreStop).toHaveBeenCalledOnce(); expect(post).not.toHaveBeenCalled();
release(); await vi.advanceTimersByTimeAsync(0); expect(post).toHaveBeenCalledOnce(); expect(afterSettlement).not.toHaveBeenCalled();
await vi.advanceTimersByTimeAsync(200); expect((await result).branch).toBe("provider_cancelled_or_interrupted"); expect(afterSettlement).toHaveBeenCalledOnce();
} finally { vi.restoreAllMocks(); vi.useRealTimers(); }
});
it("rejects normal completion first returned after Stop even with an earlier transaction timestamp", async () => {
vi.useFakeTimers(); let mono = 100n; vi.spyOn(process.hrtime, "bigint").mockImplementation(() => ++mono);
try {
const f = fixture(); f.events[4]!.createdAt = date(1); let dispatched = false;
const sample = vi.fn(async () => {}), post = stopPost(f, () => { dispatched = true; });
const retain = vi.fn(async (receipt) => { expect(receipt.terminal).toBeNull(); });
const result = settleCopilotDeniedRun({ api: { post } as any, request: f.request, deadlineAt: Date.now() + 3000,
load: async () => ({ ...f, run: dispatched ? f.run : liveRun(f), events: dispatched ? f.events : f.events.filter((_, i) => i !== 4), retired: dispatched }),
afterDeniedEdit: async () => {}, retainPreStop: retain, afterSettlement: sample });
const rejected = expect(result).rejects.toThrow(/Timed out waiting for correlated provider settlement/);
await vi.advanceTimersByTimeAsync(3200); await rejected;
expect(retain).toHaveBeenCalledOnce(); expect(post).toHaveBeenCalledOnce(); expect(sample).not.toHaveBeenCalled();
} finally { vi.restoreAllMocks(); vi.useRealTimers(); }
});
it.each(["missing-edit", "foreign-edit", "missing-terminal", "foreign-terminal", "live-process"])("fails closed on %s without premature side effects or terminal samples", async kind => {
vi.useFakeTimers(); let mono = 100n; vi.spyOn(process.hrtime, "bigint").mockImplementation(() => ++mono);
try {
const f = fixture(), post = stopPost(f), sample = vi.fn(async () => {});
if (kind === "missing-edit") f.events.splice(3, 1);
if (kind === "foreign-edit") f.events[3]!.payload.prpEvent.normalizedSessionId = "foreign";
if (kind === "missing-terminal") f.events.splice(4, 1);
if (kind === "foreign-terminal") terminal(f).sourceInstanceId = "foreign";
const result = settleCopilotDeniedRun({ api: { post } as any, request: f.request, deadlineAt: Date.now() + 3000,
load: async () => ({ ...f, run: post.mock.calls.length ? f.run : liveRun(f), retired: kind !== "live-process" }), afterDeniedEdit: async () => {}, retainPreStop: async () => {}, afterSettlement: sample });
const rejected = expect(result).rejects.toThrow();
await vi.advanceTimersByTimeAsync(3200); await rejected;
expect(post).toHaveBeenCalledTimes(kind.endsWith("edit") || kind === "foreign-terminal" ? 0 : 1);
expect(sample).not.toHaveBeenCalled();
} finally { vi.restoreAllMocks(); vi.useRealTimers(); }
});
it.each(["failed", "timed_out"])("reports %s promptly even when required rows are missing", async status => {
const f = fixture(), post = stopPost(f), sample = vi.fn(async () => {});
const load = vi.fn(async () => ({ ...f, events: [], run: { ...f.run, status }, retired: false }));
await expect(settleCopilotDeniedRun({ api: { post } as any, request: f.request, deadlineAt: Date.now() + 10000,
load, afterDeniedEdit: sample, retainPreStop: sample, afterSettlement: sample })).rejects.toThrow(`Stopped waiting for persisted correlated failed native edit: Copilot provider run failed`);
expect(load).toHaveBeenCalledOnce(); expect(post).not.toHaveBeenCalled(); expect(sample).not.toHaveBeenCalled();
});
it.each(["failed", "timed_out"])("reports post-Stop %s before attempting a missing terminal reader", async status => {
const f = fixture(), post = stopPost(f); let loads = 0;
const load = vi.fn(async () => ({ ...f, events: ++loads < 4 ? f.events : [], run: loads < 4 ? liveRun(f) : { ...f.run, status }, retired: true }));
await expect(settleCopilotDeniedRun({ api: { post } as any, request: f.request, deadlineAt: Date.now() + 10000,
load, afterDeniedEdit: async () => {}, retainPreStop: async () => {}, afterSettlement: async () => {} })).rejects.toThrow("Stopped waiting for correlated provider settlement and retired run: Copilot provider run failed");
expect(load).toHaveBeenCalledTimes(4); expect(post).toHaveBeenCalledOnce();
});
it("does not dispatch Stop if the pre-dispatch receipt cannot be retained", async () => {
const f = fixture(), post = stopPost(f);
await expect(settleCopilotDeniedRun({ api: { post } as any, request: f.request, deadlineAt: Date.now() + 1000,
load: async () => ({ ...f, run: post.mock.calls.length ? f.run : liveRun(f), retired: true }), afterDeniedEdit: async () => {}, afterSettlement: async () => {},
retainPreStop: async () => { throw new Error("artifact write failed"); } })).rejects.toThrow("artifact write failed");
expect(post).not.toHaveBeenCalled();
});
it("retains completed-before-Stop settlement and propagates retirement sampling failure", async () => {
const f = fixture(), post = stopPost(f);
const input = { api: { post } as any, request: f.request, deadlineAt: Date.now() + 1000,
load: async () => ({ ...f, run: post.mock.calls.length ? f.run : liveRun(f), retired: true }), afterDeniedEdit: async () => {}, retainPreStop: async () => {}, afterSettlement: async () => {} };
expect((await settleCopilotDeniedRun(input)).branch).toBe("provider_completed_observed_before_stop");
post.mockClear();
await expect(settleCopilotDeniedRun({ ...input, afterSettlement: async () => { throw new Error("remote descendants live"); } })).rejects.toThrow("remote descendants live");
});
it.each(["cancelled", "pending-intent", "startup-only"])("rejects visible earlier Stop %s before dispatch", async kind => {
const f = fixture(), post = stopPost(f);
const run = kind === "cancelled" ? f.run : { ...liveRun(f), resultJson: kind === "startup-only"
? { startupCancellation: { requestedAt: date(1) } } : { nativeCancellation: { intentId: "earlier", dispatchState: "pending" } } };
await expect(settleCopilotDeniedRun({ api: { post } as any, request: f.request, deadlineAt: Date.now() + 1000,
load: async () => ({ ...f, run, retired: false }), afterDeniedEdit: async () => {}, retainPreStop: async () => {}, afterSettlement: async () => {},
})).rejects.toThrow("earlier Stop");
expect(post).not.toHaveBeenCalled();
});
it("rejects Stop racing during artifact retention", async () => {
const f = fixture(), post = stopPost(f); let earlier = false;
await expect(settleCopilotDeniedRun({ api: { post } as any, request: f.request, deadlineAt: Date.now() + 1000,
load: async () => ({ ...f, run: earlier ? f.run : liveRun(f), retired: false }), afterDeniedEdit: async () => {},
retainPreStop: async () => { earlier = true; }, afterSettlement: async () => {},
})).rejects.toThrow("earlier Stop");
expect(post).not.toHaveBeenCalled();
});
it.each(["conflict", "foreign-response"])("cannot pass a Stop racing after the last API read: %s", async kind => {
const f = fixture(); const post = vi.fn(async (_path: string, _body: { cancellationRequestId: string }) => { if (kind === "conflict") throw new Error("409 cancellation intent conflict"); return f.run; });
const sampled = vi.fn(async () => {});
await expect(settleCopilotDeniedRun({ api: { post } as any, request: f.request, deadlineAt: Date.now() + 1000,
load: async () => ({ ...f, run: liveRun(f), retired: false }), afterDeniedEdit: async () => {}, retainPreStop: async () => {}, afterSettlement: sampled,
})).rejects.toThrow(kind === "conflict" ? "409" : "foreign cancellation intent");
expect(post).toHaveBeenCalledOnce(); expect(sampled).not.toHaveBeenCalled();
expect(post.mock.calls[0]?.[1]).toMatchObject({ cancellationRequestId: expect.stringMatching(/^[a-f0-9-]{36}$/) });
});
});
tests/runner-e2e/copilot-protection-tasks.ts
+18
View File
@@ -0,0 +1,18 @@
import type { RunnerTaskFixture } from "./types.js";
import { copilotProtectionCases } from "./copilot-protection-cases.js";
export const copilotProtectionTasks: readonly RunnerTaskFixture[] = copilotProtectionCases.map(c => ({
id: c.id, label: c.id === "native-permission-deny-write" ? "Deny one native write, then stop safely" : "Attached command settles before completion",
groups: [], workMode: "standard", flow: "copilot_protection", expectedRunCount: 1,
attemptTimeoutMs: { local: 300_000, daytona: 300_000 },
expectedTerminalState: { issue: c.id === "native-permission-deny-write" ? "in_progress" : "done", run: c.id === "native-permission-deny-write" ? "cancelled" : "succeeded" },
buildTitle: nonce => `Copilot ${c.id} ${nonce}`,
buildVisibleMarker: nonce => `COPILOT-PROTECTION-${c.id}-${nonce}`,
buildPrompt: nonce => c.id === "native-permission-deny-write" ? c.prompt(nonce) : [
"Run the supplied exact command once using native bash with mode async and detach false.",
"After starting it, attempt to finish immediately without calling read_bash or another waiting tool. The attached-process runtime must settle the command before ending the turn.",
"Do not modify fixture code, manufacture its result, or start another command.",
"The fixture-owned marker is a private diagnostic sentinel, not a requested file deliverable. Do not publish it. Do not attach it.",
`Call paperclip_finish with reportedWorkDisposition done, summary COPILOT-PROTECTION-${c.id}-${nonce}, current completion contract revision, satisfied objective criterion, no remaining work, evidence [], verification []. After acceptance reply with that exact summary.`,
].join("\n"),
buildMatchers: () => [], // Flow grades public durable origin and independent OS/filesystem observations.
}));
tests/runner-e2e/copilot-semantic-evidence.test.ts
+191
View File
@@ -0,0 +1,191 @@
import { createHash } from "node:crypto";
import { readFile } from "node:fs/promises";
import { describe, expect, it } from "vitest";
import { onlyCopilotAttachedOperations, readCopilotSemanticCompletion } from "./copilot-semantic-evidence.js";
import { readCopilotToolEvidence } from "./copilot-evidence.js";
import { validatePrpStructuredRunResult } from "../../packages/paperclip-runner/src/protocol/replay-contract.js";
import { runnerSuites, suiteDefinitionHash } from "./catalog.js";
const hash = (v: string) => createHash("sha256").update(v).digest("hex");
const canonical = (v: any): string => Array.isArray(v) ? `[${v.map(canonical).join(",")}]` : v !== null && typeof v === "object" ? `{${Object.keys(v).sort().map(k => `${JSON.stringify(k)}:${canonical(v[k])}`).join(",")}}` : JSON.stringify(v);
const rawInput = { summary: "EXACT-MARKER", reportedWorkDisposition: "done", completionClaim: { contractRevision: "1", objectiveSatisfied: true, criteria: [{ criterionId: "objective", status: "satisfied", evidenceRefs: [] }], remainingWork: [] }, evidence: [], verification: [] };
const validation = validatePrpStructuredRunResult(rawInput);
if (!validation.ok) throw new Error("Semantic fixture input must pass the production validator");
const result = validation.result;
const receipt = { schema: "paperclip.semantic_tool_receipt.v2", operationId: "paperclip_finish", callIdentitySha256: hash("3"), inputSha256: hash(canonical(rawInput)), normalizedInputSha256: hash(canonical(result)), resultSha256: hash("opaque original returned encoding"), outcome: "returned" };
function row(seq: number, eventType: string, payload: any, overrides: any = {}) {
const e = { schema: "paperclip.prp.event.v1", eventType, runId: "run", turnId: "turn", normalizedSessionId: "normalized", sourceInstanceId: "source", sourceKind: "runner", sourceSeq: seq, emittedAt: "2026-09-30T12:00:00Z", payload, ...overrides };
return { companyId: "company", runId: "run", seq, eventType, sourceInstanceId: e.sourceInstanceId, sourceSeq: e.sourceSeq, payload: { prpEvent: e } };
}
function notice(seq: number, toolCallId: string, status: string, fields: any = {}) {
return row(seq, "provider.notice.recorded", { schema: "paperclip.provider.notice.v1", category: "copilot_tool_evidence_v1", scope: "turn", provenance: { method: "session/update", eventType: "tool", sessionId: "native", turnId: "turn" }, details: Object.entries({ stage: "tool", toolCallId, status, ...fields }).map(([name, value]) => ({ name, value: String(value) })) });
}
function fixture() {
const rows = [notice(1, "command", "pending", { operation: "execute" }), notice(2, "finish-native", "pending"), row(3, "run.result.proposed", structuredClone(result), { itemId: "item-run" }),
row(4, "provider.notice.recorded", { schema: "paperclip.provider.notice.v1", category: "paperclip_semantic_tool_receipt_v2", scope: "turn", provenance: { method: "paperclip/semantic_tool_result", eventType: "semantic_result", sessionId: "native", turnId: "turn" }, details: Object.entries({ stage: "semantic_result", ...receipt }).map(([name, value]) => ({ name, value })) }),
notice(5, "finish-native", "completed", { semanticOperationId: receipt.operationId, semanticCallIdentitySha256: receipt.callIdentitySha256, semanticInputSha256: receipt.inputSha256, semanticNormalizedInputSha256: receipt.normalizedInputSha256, semanticResultSha256: receipt.resultSha256, semanticOutcome: receipt.outcome }),
row(6, "turn.completed", {}), row(7, "run.result.accepted", { result: structuredClone(result) }, { sourceKind: "control_plane", sourceInstanceId: "source:control", sourceSeq: 1 })];
const expected = { companyId: "company", runId: "run", turnId: "turn", nativeSessionId: "native", command: readCopilotToolEvidence(rows, "run")[0]!, summary: "EXACT-MARKER" };
return { rows, expected };
}
const frame = (r: any) => r.payload.prpEvent;
function setField(r: any, key: string, value: string) { frame(r).payload.details.find((d: any) => d.name === key).value = value; }
describe("Copilot semantic completion public-event oracle", () => {
it("versions the actual Copilot suite oracle without changing denial settlement", () => {
const suite = runnerSuites.find(s => s.id === "copilot-protection")!;
expect(suite.definitionMetadata).toMatchObject({ version: 9, semanticCompletionEvidence: "paperclip.e2e.copilot-semantic-completion.v2", denialSettlementEvidence: "paperclip.e2e.copilot-denial-settlement.v3" });
expect(suiteDefinitionHash(suite)).not.toBe(suiteDefinitionHash({ ...suite, definitionMetadata: { ...suite.definitionMetadata, version: 7 } }));
});
it("joins exact native lifecycle, authoritative callback, proposed content and accepted control-plane result", () => {
const { rows, expected } = fixture(); const proof = readCopilotSemanticCompletion(rows, expected);
expect(proof.nativeToolCallId).toBe("finish-native"); expect(proof.acceptedSeq).toBe(7); expect(proof.summarySha256).toBe(hash("EXACT-MARKER"));
expect(JSON.stringify(proof)).not.toContain("EXACT-MARKER");
});
it("binds distinct raw and production-normalized input digests without filling defaults in the oracle", () => {
const { rows, expected } = fixture();
expect(rawInput).not.toHaveProperty("schema");
expect(rawInput).not.toHaveProperty("artifacts");
expect(rawInput).not.toHaveProperty("attentionRequests");
expect(result).toMatchObject({ schema: "paperclip.run_result.v1", artifacts: [], attentionRequests: [] });
expect(receipt.inputSha256).not.toBe(receipt.normalizedInputSha256);
expect(readCopilotSemanticCompletion(rows, expected)).toMatchObject({
schema: "paperclip.e2e.copilot-semantic-completion.v2",
inputSha256: receipt.inputSha256, normalizedInputSha256: receipt.normalizedInputSha256,
});
// Same authenticated raw/native join is necessary but cannot substitute for
// the invocation-captured normalized body digest.
setField(rows[3], "normalizedInputSha256", receipt.inputSha256);
setField(rows[4], "semanticNormalizedInputSha256", receipt.inputSha256);
expect(() => readCopilotSemanticCompletion(rows, expected)).toThrow("canonical accepted result");
});
it.each(["missing", "null", "wrong", "uppercase", "malformed"])("rejects %s normalized finish digest", value => {
const f = fixture();
if (value === "missing") {
frame(f.rows[3]!).payload.details = frame(f.rows[3]!).payload.details.filter((d: any) => d.name !== "normalizedInputSha256");
frame(f.rows[4]!).payload.details = frame(f.rows[4]!).payload.details.filter((d: any) => d.name !== "semanticNormalizedInputSha256");
} else {
const digest = value === "null" ? "null" : value === "wrong" ? hash("foreign-normalized-input") : value === "uppercase" ? "A".repeat(64) : "not-a-digest";
setField(f.rows[3], "normalizedInputSha256", digest); setField(f.rows[4], "semanticNormalizedInputSha256", digest);
}
expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow();
});
it("allows an explicit null native digest as diagnostic data, never as successful finish authority", () => {
const f = fixture(); setField(f.rows[4], "semanticNormalizedInputSha256", "null");
expect(readCopilotToolEvidence(f.rows, "run").at(-1)?.semanticNormalizedInputSha256).toBeNull();
expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow("native receipt mismatch");
});
it.each(["only", "alongside-v2"])("rejects legacy v1 authority %s as fresh qualification", mode => {
const f = fixture(), old = structuredClone(f.rows[3]!);
frame(old).payload.category = "paperclip_semantic_tool_receipt_v1";
setField(old, "schema", "paperclip.semantic_tool_receipt.v1");
frame(old).payload.details = frame(old).payload.details.filter((d: any) => d.name !== "normalizedInputSha256");
if (mode === "only") f.rows[3] = old; else f.rows.push(old);
expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow();
});
it("does not repair an unnormalized proposed and accepted body by adding defaults", () => {
const f = fixture(); frame(f.rows[2]!).payload = structuredClone(rawInput); frame(f.rows[6]!).payload.result = structuredClone(rawInput);
expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow("canonical accepted result");
});
it.each([0, 1, 2, 3, 4, 5, 6])("rejects missing required row %s", index => { const f = fixture(); f.rows.splice(index, 1); expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow(); });
it.each([1, 2, 3, 4, 5, 6])("rejects duplicate required row %s", index => { const f = fixture(); f.rows.push(structuredClone(f.rows[index]!)); expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow(); });
it.each(["companyId", "runId"])("rejects foreign durable %s", key => { const f = fixture(); (f.rows[3] as any)[key] = "foreign"; expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow(); });
it.each(["turnId", "normalizedSessionId", "sourceInstanceId", "sourceKind", "eventType", "schema"])("rejects foreign or malformed envelope %s", key => {
for (const index of [1, 2, 3, 4, 5, 6]) { const f = fixture(); (frame(f.rows[index]!))[key] = "foreign"; expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow(); }
});
it.each([null, undefined])("rejects missing envelope turn %s even with native provenance intact", turn => { const f = fixture(); frame(f.rows[4]!).turnId = turn; expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow(); });
it.each(["sessionId", "turnId", "method", "eventType"])("rejects authoritative provenance %s mismatch", key => { const f = fixture(); frame(f.rows[3]!).payload.provenance[key] = "foreign"; expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow(); });
it.each(["callIdentitySha256", "inputSha256", "normalizedInputSha256", "resultSha256", "operationId", "outcome"])("rejects native %s disagreement", key => { const f = fixture(); setField(f.rows[4], `semantic${key[0]!.toUpperCase()}${key.slice(1)}`, key.endsWith("Sha256") ? "a".repeat(64) : "wrong"); expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow(); });
it("does not infer acceptance from returned receipt or a tool named paperclip_finish", () => {
const f = fixture(); frame(f.rows[6]!).payload.result = { accepted: false, summary: "EXACT-MARKER" }; expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow();
const g = fixture(); frame(g.rows[4]!).payload.title = "paperclip_finish"; frame(g.rows[4]!).payload.details = frame(g.rows[4]!).payload.details.filter((d: any) => !d.name.startsWith("semantic")); expect(() => readCopilotSemanticCompletion(g.rows, g.expected)).toThrow();
});
it.each(["failed", "cancelled", "interrupted"])("rejects %s terminal", status => { const f = fixture(); f.rows[5]!.eventType = `turn.${status}`; frame(f.rows[5]!).eventType = `turn.${status}`; expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow(); });
it("rejects changed accepted body, forged call ID, wrong summary and rejected result", () => {
for (const mutate of [(f: ReturnType<typeof fixture>) => { frame(f.rows[6]!).payload.result.completionClaim.remainingWork.push("unfinished"); }, (f: ReturnType<typeof fixture>) => { setField(f.rows[3], "callIdentitySha256", hash("foreign-call")); }, (f: ReturnType<typeof fixture>) => { f.expected.summary = "other"; }, (f: ReturnType<typeof fixture>) => { f.rows.push(row(8, "run.result.rejected", {})); }]) { const f = fixture(); mutate(f); expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow(); }
});
it("rejects duplicate/unknown/oversized receipt details and partial native fields", () => {
for (const mutate of [(r: any) => frame(r).payload.details.push({ name: "secret", value: "untrusted" }), (r: any) => frame(r).payload.details.push(frame(r).payload.details[0]), (r: any) => setField(r, "operationId", "x".repeat(300))]) { const f = fixture(); mutate(f.rows[3]); expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow(); }
const f = fixture(); frame(f.rows[4]!).payload.details.pop(); expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow();
});
it("rejects mutation metadata, extra lifecycle, reordered receipt and source sequence drift", () => {
for (const mutate of [(f: ReturnType<typeof fixture>) => frame(f.rows[1]!).payload.details.push({ name: "operation", value: "edit" }), (f: ReturnType<typeof fixture>) => f.rows.push(notice(8, "finish-native", "pending")), (f: ReturnType<typeof fixture>) => { f.rows[3]!.seq = 8; }, (f: ReturnType<typeof fixture>) => { frame(f.rows[3]!).sourceSeq = 20; f.rows[3]!.sourceSeq = 20; }, (f: ReturnType<typeof fixture>) => { frame(f.rows[1]!).sourceSeq = 20; f.rows[1]!.sourceSeq = 20; }]) { const f = fixture(); mutate(f); expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow(); }
});
it("does not treat canonical run-level itemId as semantic call identity", () => {
for (const itemId of ["item-run", "3", "unrelated-item", null]) {
const f = fixture(); frame(f.rows[2]!).itemId = itemId;
expect(readCopilotSemanticCompletion(f.rows, f.expected).callIdentitySha256).toBe(receipt.callIdentitySha256);
}
});
it("rejects a different finish input even when proposal and acceptance agree", () => {
const f = fixture(); frame(f.rows[2]!).payload.completionClaim.criteria[0].evidenceRefs.push("different");
frame(f.rows[6]!).payload.result = structuredClone(frame(f.rows[2]!).payload);
expect(() => readCopilotSemanticCompletion(f.rows, f.expected)).toThrow();
});
it("accepts the retained shell-read ordering: pending has shellId only, completion gains command identity after finish", () => {
const f = attachedFixture();
expect(f.notices.find(n => n.toolCallId === "shell-read" && n.status === "pending")?.commandToolCallId).toBeUndefined();
expect(onlyCopilotAttachedOperations(f.notices, f.command, f.proof)).toBe(true);
});
it("accepts bounded in-progress shell updates without inventing terminal command identity", () => {
const f = attachedFixture(), pending = f.notices.find(n => n.toolCallId === "shell-read" && n.status === "pending")!;
f.notices.push({ ...pending, status: "in_progress", seq: 53 });
expect(onlyCopilotAttachedOperations(f.notices, f.command, f.proof)).toBe(true);
});
it("rejects shell-read evidence from a different durable stream despite matching native identity", () => {
const f = attachedFixture(), r = f.rows.find(r => r.seq === 52)!;
frame(r).sourceInstanceId = "foreign"; r.sourceInstanceId = "foreign";
expect(() => readCopilotSemanticCompletion(f.rows, { companyId: "company", runId: "run", turnId: "turn", nativeSessionId: "native", command: f.command, summary: "EXACT-MARKER" })).toThrow();
});
it.each(["missing-pending", "missing-terminal", "duplicate-pending", "duplicate-terminal", "extra-completed-only", "extra-complete-read", "wrong-shell", "wrong-command", "failed", "nonzero", "reordered", "late", "wrong-operation", "mutation", "read-path", "foreign-session", "foreign-turn", "foreign-run", "permission", "semantic-fields", "normalized-semantic-field"])("rejects invalid shell-read lifecycle: %s", failure => {
const f = attachedFixture(), start = f.notices.find(n => n.toolCallId === "shell-read" && n.status === "pending")!, end = f.notices.find(n => n.toolCallId === "shell-read" && n.status === "completed")!;
if (failure === "missing-pending") f.notices.splice(f.notices.indexOf(start), 1);
if (failure === "missing-terminal") f.notices.splice(f.notices.indexOf(end), 1);
if (failure === "duplicate-pending") f.notices.push({ ...start, seq: 53 });
if (failure === "duplicate-terminal") f.notices.push({ ...end, seq: 56 });
if (failure === "extra-completed-only") f.notices.push({ ...end, toolCallId: "extra", seq: 57 });
if (failure === "extra-complete-read") f.notices.push({ ...start, toolCallId: "extra", seq: 56 }, { ...end, toolCallId: "extra", seq: 57 });
if (failure === "wrong-shell") start.shellId = "foreign";
if (failure === "wrong-command") end.commandToolCallId = "foreign";
if (failure === "failed") end.status = "failed";
if (failure === "nonzero") end.exitCode = 1;
if (failure === "reordered") start.seq = 14;
if (failure === "late") end.seq = 61;
if (failure === "wrong-operation") start.operation = "execute";
if (failure === "mutation") end.target = "file.txt";
if (failure === "read-path") start.readTargetSha256 = `sha256:${"a".repeat(64)}`;
if (failure === "foreign-session") start.sessionId = "foreign";
if (failure === "foreign-turn") start.turnId = "foreign";
if (failure === "foreign-run") start.runId = "foreign";
if (failure === "permission") start.stage = "permission_requested";
if (failure === "semantic-fields") end.semanticCallIdentitySha256 = receipt.callIdentitySha256;
if (failure === "normalized-semantic-field") end.semanticNormalizedInputSha256 = receipt.normalizedInputSha256;
expect(onlyCopilotAttachedOperations(f.notices, f.command, f.proof)).toBe(false);
});
it("rejects duplicate shell origins and arbitrary extra operations", () => {
const f = attachedFixture();
const started = f.notices.find(n => n.shellState === "started")!;
expect(onlyCopilotAttachedOperations([...f.notices, { ...started, seq: 16 }], f.command, f.proof)).toBe(false);
for (const operation of [undefined, "read", "edit", "execute"] as const) expect(onlyCopilotAttachedOperations([...f.notices, { ...f.command, toolCallId: "extra", operation, seq: 17 }], f.command, f.proof)).toBe(false);
});
it("keeps actual flow correlation before no-extra-operation gate and keeps visible marker independent", async () => {
const source = await readFile(new URL("./copilot-protection-flow.ts", import.meta.url), "utf8");
expect(source.indexOf("readCopilotSemanticCompletion(runEvents")).toBeLessThan(source.indexOf('check("no-extra-native-operation", onlyCopilotAttachedOperations('));
expect(source).toContain("undefined), call!, semantic)");
expect(source).toContain('check("exact-completion-marker", comments.filter');
expect(source).not.toContain('if (remote) check("no-extra-native-operation"');
});
});
// Mirrors retained native order: command pending→shell started; finish; then
// read_bash pending(shellId only)→completed(commandToolCallId), before turn end.
function attachedFixture() {
const f = fixture(); frame(f.rows[2]!).itemId = "3";
for (const r of f.rows) { r.seq *= 10; if (frame(r).sourceKind === "runner") { r.sourceSeq *= 10; frame(r).sourceSeq *= 10; } }
const fields = { operation: "execute", commandSha256: `sha256:${"a".repeat(64)}`, mode: "async", detach: false };
f.rows[0] = notice(10, "command", "pending", fields);
f.rows.push(notice(15, "command", "completed", { ...fields, shellId: "0", commandToolCallId: "command", shellState: "started" }),
notice(52, "shell-read", "pending", { operation: "read", shellId: "0" }),
notice(55, "shell-read", "completed", { operation: "read", shellId: "0", commandToolCallId: "command", shellState: "completed", exitCode: 0 }));
const notices = readCopilotToolEvidence(f.rows, "run"), command = notices.find(n => n.seq === 10)!;
const proof = readCopilotSemanticCompletion(f.rows, { ...f.expected, command });
return { rows: f.rows, notices, command, proof };
}
tests/runner-e2e/copilot-semantic-evidence.ts
+139
View File
@@ -0,0 +1,139 @@
import { createHash } from "node:crypto";
import { readCopilotToolEvidence, type CopilotToolNotice } from "./copilot-evidence.js";
type Row = Record<string, any>;
const rec = (v: unknown): Row => v !== null && typeof v === "object" && !Array.isArray(v) ? v as Row : {};
const id = (v: unknown): v is string => typeof v === "string" && v.length > 0 && v.length <= 240 && !/[\u0000-\u001f\u007f]/u.test(v) && !v.includes("[REDACTED]");
const hash = (v: string) => createHash("sha256").update(v).digest("hex");
const canonical = (v: unknown): string => Array.isArray(v) ? `[${v.map(canonical).join(",")}]` : v !== null && typeof v === "object"
? `{${Object.keys(v).sort().map(k => `${JSON.stringify(k)}:${canonical((v as Row)[k])}`).join(",")}}` : JSON.stringify(v) ?? "undefined";
const digest = (v: unknown) => hash(canonical(v));
const receiptKeys = ["callIdentitySha256", "inputSha256", "normalizedInputSha256", "operationId", "outcome", "resultSha256", "schema", "stage"].sort().join(",");
export interface CopilotSemanticCompletionProof {
schema: "paperclip.e2e.copilot-semantic-completion.v2";
runId: string; turnId: string; nativeSessionId: string; normalizedSessionId: string; sourceInstanceId: string;
nativeToolCallId: string; callIdentitySha256: string; inputSha256: string; normalizedInputSha256: string; resultSha256: string;
proposedSeq: number; receiptSeq: number; nativeCompletedSeq: number; turnCompletedSeq: number; acceptedSeq: number;
summarySha256: string;
}
/** Independent public-event oracle. A transport receipt alone never proves completion acceptance. */
export function readCopilotSemanticCompletion(rows: readonly unknown[], expected: {
companyId: string; runId: string; turnId: string; nativeSessionId: string; command: CopilotToolNotice; summary: string;
}): CopilotSemanticCompletionProof {
const fail = (message: string): never => { throw new Error(`Copilot semantic completion: ${message}`); };
const all = rows.map(rec), notices = readCopilotToolEvidence(rows, expected.runId);
const commandRows = all.filter(r => r.seq === expected.command.seq);
if (commandRows.length !== 1) fail("missing exact command origin");
const base = rec(rec(commandRows[0]!.payload).prpEvent);
if (!id(base.normalizedSessionId) || !id(base.sourceInstanceId)) fail("missing command stream");
function frame(r: Row, sourceKind: "runner" | "control_plane") {
const e = rec(rec(r.payload).prpEvent);
if (r.companyId !== expected.companyId || r.runId !== expected.runId || !Number.isSafeInteger(r.seq) || r.seq < 1
|| e.schema !== "paperclip.prp.event.v1" || e.eventType !== r.eventType || e.sourceKind !== sourceKind
|| e.runId !== expected.runId || e.turnId !== expected.turnId || e.normalizedSessionId !== base.normalizedSessionId
|| !id(e.sourceInstanceId) || r.sourceInstanceId !== e.sourceInstanceId || r.sourceSeq !== e.sourceSeq
|| !Number.isSafeInteger(e.sourceSeq) || e.sourceSeq < 1
|| (sourceKind === "runner" && e.sourceInstanceId !== base.sourceInstanceId)) fail("foreign or malformed durable frame");
return e;
}
frame(commandRows[0]!, "runner");
// Every operation admitted later must belong to this durable stream, including
// shell reads whose command identity is only known on their terminal notice.
for (const n of notices) {
const rs = all.filter(r => r.seq === n.seq);
if (rs.length !== 1) fail("duplicate native row");
frame(rs[0]!, "runner");
}
if (expected.command.runId !== expected.runId || expected.command.turnId !== expected.turnId || expected.command.sessionId !== expected.nativeSessionId) fail("foreign command");
const authorityRows = all.filter(r => rec(rec(rec(r.payload).prpEvent).payload).category === "paperclip_semantic_tool_receipt_v2" || rec(rec(rec(r.payload).prpEvent).payload).category === "paperclip_semantic_tool_receipt_v1");
// This authored case asks for one finish, not arbitrary semantic actions.
if (authorityRows.length !== 1) fail("expected one authoritative finish receipt");
const authorityRow = authorityRows[0]!, authority = frame(authorityRow, "runner"), payload = rec(authority.payload), origin = rec(payload.provenance);
if (payload.category !== "paperclip_semantic_tool_receipt_v2" || payload.schema !== "paperclip.provider.notice.v1" || payload.scope !== "turn" || authorityRow.eventType !== "provider.notice.recorded"
|| origin.method !== "paperclip/semantic_tool_result" || origin.eventType !== "semantic_result"
|| origin.sessionId !== expected.nativeSessionId || origin.turnId !== expected.turnId || !Array.isArray(payload.details) || payload.details.length !== 8) fail("invalid receipt provenance");
const fields: Record<string, string> = {};
for (const d of payload.details) {
if (Object.keys(rec(d)).sort().join(",") !== "name,value" || typeof d.name !== "string" || typeof d.value !== "string" || d.value.length > 256 || Object.hasOwn(fields, d.name)) fail("malformed receipt detail");
fields[d.name] = d.value;
}
if (Object.keys(fields).sort().join(",") !== receiptKeys || fields.stage !== "semantic_result" || fields.schema !== "paperclip.semantic_tool_receipt.v2"
|| fields.operationId !== "paperclip_finish" || fields.outcome !== "returned"
|| ![fields.callIdentitySha256, fields.inputSha256, fields.normalizedInputSha256, fields.resultSha256].every(v => /^[a-f0-9]{64}$/.test(v!))) fail("invalid finish receipt");
const matches = notices.filter(n => n.semanticCallIdentitySha256 === fields.callIdentitySha256);
if (matches.length !== 1) fail("missing or duplicate native match");
const native = matches[0]!;
if (native.stage !== "tool" || native.status !== "completed" || native.sessionId !== expected.nativeSessionId || native.turnId !== expected.turnId
|| native.semanticOperationId !== fields.operationId || native.semanticInputSha256 !== fields.inputSha256 || native.semanticNormalizedInputSha256 !== fields.normalizedInputSha256 || native.semanticResultSha256 !== fields.resultSha256 || native.semanticOutcome !== "returned") fail("native receipt mismatch");
const group = notices.filter(n => n.toolCallId === native.toolCallId);
const pending = group.filter(n => n.stage === "tool" && n.status === "pending");
if (pending.length !== 1 || group.filter(n => n.status === "completed" || n.status === "failed").length !== 1
|| group.some(n => n.stage !== "tool" || n.sessionId !== expected.nativeSessionId || n.turnId !== expected.turnId
|| n.operation !== undefined || n.target !== undefined || n.commandSha256 !== undefined || n.shellId !== undefined || n.commandToolCallId !== undefined
|| !["pending", "in_progress", "completed"].includes(n.status ?? "") || n.seq < pending[0]!.seq || n.seq > native.seq)) fail("not one semantic-only native lifecycle");
for (const n of group) {
const rs = all.filter(r => r.seq === n.seq); if (rs.length !== 1) fail("duplicate native row"); frame(rs[0]!, "runner");
}
const proposedRows = all.filter(r => r.eventType === "run.result.proposed"), acceptedRows = all.filter(r => r.eventType === "run.result.accepted"), terminals = all.filter(r => ["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"].includes(r.eventType) && rec(rec(r.payload).prpEvent).turnId === expected.turnId);
if (proposedRows.length !== 1 || acceptedRows.length !== 1 || terminals.length !== 1 || terminals[0]!.eventType !== "turn.completed"
|| all.some(r => r.eventType === "run.result.rejected")) fail("missing, rejected or ambiguous completion");
const proposed = frame(proposedRows[0]!, "runner"), accepted = frame(acceptedRows[0]!, "control_plane"), terminal = frame(terminals[0]!, "runner");
const result = rec(proposed.payload), acceptedResult = rec(rec(accepted.payload).result);
// Native correlation uses the invocation-captured callback call hash. The
// canonical proposal itemId may instead identify the run: never interpret it
// as a bridge call ID. Unique same-turn result bodies supply the independent
// acceptance proof, joined to the normalized digest captured by the same
// invocation after validation. Raw provider arguments may omit defaults;
// their separate digest remains joined to the native receipt above. Never
// reconstruct or normalize a proposal here to make its digest match.
if (digest(result) !== fields.normalizedInputSha256
|| result.schema !== "paperclip.run_result.v1" || result.reportedWorkDisposition !== "done" || result.summary !== expected.summary
|| canonical(result) !== canonical(acceptedResult)) fail("canonical accepted result does not match exact finish");
const pendingFrame = frame(all.find(r => r.seq === pending[0]!.seq)!, "runner");
const completedFrame = frame(all.find(r => r.seq === native.seq)!, "runner");
if (!(expected.command.seq < pending[0]!.seq && pending[0]!.seq < proposedRows[0]!.seq && proposedRows[0]!.seq < authorityRow.seq
&& authorityRow.seq < native.seq && native.seq < terminals[0]!.seq && terminals[0]!.seq < acceptedRows[0]!.seq)
|| !(base.sourceSeq < pendingFrame.sourceSeq && pendingFrame.sourceSeq < proposed.sourceSeq && proposed.sourceSeq < authority.sourceSeq
&& authority.sourceSeq < completedFrame.sourceSeq && completedFrame.sourceSeq < terminal.sourceSeq)) fail("invalid causal order");
return { schema: "paperclip.e2e.copilot-semantic-completion.v2", runId: expected.runId, turnId: expected.turnId, nativeSessionId: expected.nativeSessionId,
normalizedSessionId: base.normalizedSessionId, sourceInstanceId: base.sourceInstanceId, nativeToolCallId: native.toolCallId,
callIdentitySha256: fields.callIdentitySha256!, inputSha256: fields.inputSha256!, normalizedInputSha256: fields.normalizedInputSha256!, resultSha256: fields.resultSha256!,
proposedSeq: proposedRows[0]!.seq, receiptSeq: authorityRow.seq, nativeCompletedSeq: native.seq, turnCompletedSeq: terminals[0]!.seq, acceptedSeq: acceptedRows[0]!.seq, summarySha256: hash(expected.summary) };
}
/** Called only after the completion proof and bootstrap-read oracle succeed. */
export function onlyCopilotAttachedOperations(notices: readonly CopilotToolNotice[], command: CopilotToolNotice, proof: CopilotSemanticCompletionProof): boolean {
if (proof.runId !== command.runId || proof.turnId !== command.turnId || proof.nativeSessionId !== command.sessionId
|| proof.nativeToolCallId === command.toolCallId || notices.length > 2048
|| notices.some(n => n.runId !== command.runId || n.sessionId !== command.sessionId || n.turnId !== command.turnId
|| !Number.isSafeInteger(n.seq) || n.seq < 1 || n.seq >= proof.turnCompletedSeq)
|| new Set(notices.map(n => n.seq)).size !== notices.length) return false;
const groups = new Map<string, CopilotToolNotice[]>();
for (const n of notices) { const group = groups.get(n.toolCallId) ?? []; group.push(n); groups.set(n.toolCallId, group); }
// Exactly one command, one shell-result read, and the already-proven finish.
// A completed-only extra read cannot borrow the original command's identity.
if (groups.size !== 3 || !groups.has(proof.nativeToolCallId)) return false;
const complete = (group: CopilotToolNotice[]): boolean => {
group.sort((a, b) => a.seq - b.seq);
return group.length >= 2 && group[0]!.status === "pending" && group.at(-1)!.status === "completed"
&& group.every((n, i) => n.stage === "tool" && (i === 0 ? n.status === "pending"
: i === group.length - 1 ? n.status === "completed" : n.status === "in_progress"));
};
const semanticFree = (n: CopilotToolNotice) => n.semanticOperationId === undefined && n.semanticCallIdentitySha256 === undefined
&& n.semanticInputSha256 === undefined && n.semanticNormalizedInputSha256 === undefined && n.semanticResultSha256 === undefined && n.semanticOutcome === undefined;
const commandGroup = groups.get(command.toolCallId) ?? [];
if (!complete(commandGroup) || commandGroup[0]!.seq !== command.seq) return false;
const started = commandGroup.at(-1)!;
if (!id(started.shellId) || started.shellState !== "started" || started.commandToolCallId !== command.toolCallId
|| commandGroup.some(n => n.operation !== "execute" || n.commandSha256 !== command.commandSha256 || n.mode !== "async" || n.detach !== false
|| n.target !== undefined || n.readTargetSha256 !== undefined || n.exitCode !== undefined || !semanticFree(n)
|| (n !== started && (n.shellId !== undefined || n.shellState !== undefined || n.commandToolCallId !== undefined)))) return false;
const shellGroup = [...groups].find(([key]) => key !== command.toolCallId && key !== proof.nativeToolCallId)![1];
if (!complete(shellGroup)) return false;
const terminal = shellGroup.at(-1)!;
return shellGroup[0]!.seq > started.seq && terminal.seq < proof.turnCompletedSeq
&& terminal.commandToolCallId === command.toolCallId && terminal.shellState === "completed" && terminal.exitCode === 0
&& shellGroup.every(n => n.operation === "read" && n.shellId === started.shellId && n.target === undefined && n.readTargetSha256 === undefined
&& n.commandSha256 === undefined && n.mode === undefined && n.detach === undefined && semanticFree(n)
&& (n === terminal || (n.commandToolCallId === undefined && n.shellState === undefined && n.exitCode === undefined)));
}
tests/runner-e2e/cursor-native-cases.ts
+4 -4
View File
@@ -9,10 +9,10 @@ import type { RunnerTaskFixture } from "./types.js";
* authenticated model/mode; a prompt cannot force an absent provider tool.
*/
export const cursorNativeCaseDesigns = [
{ id: "native-question-reconnect", method: "cursor/ask_question", cursorMode: "plan", permissionMode: "approve-all", expectedRunCount: 1 },
{ id: "native-plan-reject-revise-accept", method: "cursor/create_plan", cursorMode: "plan", permissionMode: "approve-all", expectedRunCount: 1 },
{ id: "native-plan-cancel", method: "cursor/create_plan", cursorMode: "plan", permissionMode: "approve-all", expectedRunCount: 1 },
{ id: "native-write-deny-reconnect", method: "session/request_permission", cursorMode: "agent", permissionMode: "approve-reads", expectedRunCount: 1 },
{ id: "native-question-reconnect", method: "cursor/ask_question", mode: "plan", permissionMode: "approve-all", expectedRunCount: 1 },
{ id: "native-plan-reject-revise-accept", method: "cursor/create_plan", mode: "plan", permissionMode: "approve-all", expectedRunCount: 1 },
{ id: "native-plan-cancel", method: "cursor/create_plan", mode: "plan", permissionMode: "approve-all", expectedRunCount: 1 },
{ id: "native-write-deny-reconnect", method: "session/request_permission", mode: "agent", permissionMode: "approve-reads", expectedRunCount: 1 },
] as const;
export type CursorNativeMethod = typeof cursorNativeCaseDesigns[number]["method"];
tests/runner-e2e/cursor-native-flow.test.ts
+2 -2
View File
@@ -10,8 +10,8 @@ it("keeps native mode/permission choices explicit and artifact export pending",
for (const task of cursorNativeTasks) {
expect(task.flow).toBe("cursor_native"); expect(task.expectedRunCount).toBe(1); expect(task.turnTimeoutMs).toBe(120_000);
}
expect(cursorNativeCaseDesigns.filter(row => row.method !== "session/request_permission").every(row => row.cursorMode === "plan")).toBe(true);
expect(cursorNativeCaseDesigns.find(row => row.method === "session/request_permission")).toMatchObject({ cursorMode: "agent", permissionMode: "approve-reads" });
expect(cursorNativeCaseDesigns.filter(row => row.method !== "session/request_permission").every(row => row.mode === "plan")).toBe(true);
expect(cursorNativeCaseDesigns.find(row => row.method === "session/request_permission")).toMatchObject({ mode: "agent", permissionMode: "approve-reads" });
expect(cursorNativeTasks.find(task => task.id === "native-write-deny-reconnect")!.expectedTerminalState).toEqual({ issue: "in_progress", run: "failed" });
expect(cursorNativePlanArtifactGate.status).toBe("pending");
expect(cursorNativePlanArtifactGate.nativePath).toContain("<private provider HOME>");
tests/runner-e2e/cursor-native-flow.ts
+3 -3
View File
@@ -113,10 +113,10 @@ export async function runCursorNativeFlow(input: {
const events = (runId: string) => collectRunEvents<Row>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${runId}/events?afterSeq=${afterSeq}&limit=${limit}`));
const absent = async (path: string) => { try { await lstat(path); return false; } catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") return true; throw error; } };
const agent = await api.get<Row>(`/api/agents/${fixtures.agent.id}`);
const configured = await api.patch<Row>(`/api/agents/${fixtures.agent.id}`, { adapterConfig: { ...agent.adapterConfig, acpxSessionMode: design.cursorMode, acpxPermissionMode: design.permissionMode, ...(remote || design.id === "native-write-deny-reconnect" ? { lifecycleMode: "per_turn", timeoutSec: 120 } : {}) } });
check("explicit-mode-policy", configured.adapterConfig.acpxSessionMode === design.cursorMode && configured.adapterConfig.acpxPermissionMode === design.permissionMode, "Public agent configuration selected mode and permission policy separately before provider startup");
const configured = await api.patch<Row>(`/api/agents/${fixtures.agent.id}`, { adapterConfig: { ...agent.adapterConfig, acpxSessionMode: design.mode, acpxPermissionMode: design.permissionMode, ...(remote || design.id === "native-write-deny-reconnect" ? { lifecycleMode: "per_turn", timeoutSec: 120 } : {}) } });
check("explicit-mode-policy", configured.adapterConfig.acpxSessionMode === design.mode && configured.adapterConfig.acpxPermissionMode === design.permissionMode, "Public agent configuration selected mode and permission policy separately before provider startup");
if (remote || design.id === "native-write-deny-reconnect") check("per-turn-process-authority", configured.adapterConfig.lifecycleMode === "per_turn" && configured.adapterConfig.timeoutSec === 120, "Public configuration admits a bounded per-turn provider process before startup");
await input.evidence("cursor-native-contract.json", { caseId: design.id, mode: design.cursorMode, permissionMode: design.permissionMode, method: design.method, expectedRunCount: 1, nativeCallbackRequired: true, artifactGate: cursorNativePlanArtifactGate });
await input.evidence("cursor-native-contract.json", { caseId: design.id, mode: design.mode, permissionMode: design.permissionMode, method: design.method, expectedRunCount: 1, nativeCallbackRequired: true, artifactGate: cursorNativePlanArtifactGate });
const project = await api.post<Row>(`/api/companies/${fixtures.company.id}/projects`, {
name: `Cursor native workspace ${nonce}`, executionWorkspacePolicy: { enabled: true, defaultMode: "shared_workspace", sharedWorkspaceConcurrency: "serialize", allowIssueOverride: false, environmentId: fixtures.environment.id, workspaceStrategy: { type: "project_primary" } },
workspace: { name: "Primary", sourceType: "local_path", cwd: input.workspacePath, isPrimary: true },
tests/runner-e2e/daytona-image-content.ts
+7 -3
View File
@@ -40,16 +40,20 @@ export const DAYTONA_IMAGE_INPUT_PATHS = [
"packages/paperclip-runner/scripts/provider-pack-executable-shims.mjs",
"packages/paperclip-runner/scripts/build-node-startup-timeout.mjs",
"packages/paperclip-runner/scripts/candidate-provider-pack.mjs",
"packages/paperclip-runner/scripts/build-copilot-distribution.mjs",
"packages/paperclip-runner/scripts/materialize-copilot-binary.mjs",
"packages/paperclip-runner/scripts/materialize-cursor-distribution.mjs",
"packages/paperclip-runner/scripts/cursor-runtime-patch.mjs",
"packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs",
"packages/paperclip-runner/scripts/generate-acpx-sidecar-contract.mjs",
"packages/paperclip-runner/scripts/generate-protocol-schema-module.mjs",
"packages/paperclip-runner/scripts/materialize-pi-distribution.mjs",
"packages/paperclip-runner/scripts/pi-distribution",
"packages/paperclip-runner/scripts/cursor-native-usage.mjs",
"packages/paperclip-runner/scripts/cursor-native-usage-source.json",
"packages/paperclip-runner/scripts/provision-cursor.mjs",
"packages/paperclip-runner/cursor-distributions.json",
"packages/paperclip-runner/cursor-contract.json",
"packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs",
"packages/paperclip-runner/scripts/generate-acpx-sidecar-contract.mjs",
"packages/paperclip-runner/scripts/generate-protocol-schema-module.mjs",
// Pi runtime/extension/Node/closure pins are included by the src tree below.
"packages/paperclip-runner/src",
"packages/paperclip-runner/styles.css",
tests/runner-e2e/daytona-image.test.ts
+21
View File
@@ -88,6 +88,14 @@ describe("runner E2E Daytona image contract", () => {
expect(dockerignore).toContain("**/node_modules");
expect(dockerignore).toContain("packages/paperclip-runner/dist");
expect(dockerignore).toContain("packages/paperclip-runner/runner/target");
expect(await readFile(path.join(repositoryRoot, ".github/docker-context-checks.Dockerfile"), "utf8")).toContain("node packages/paperclip-runner/scripts/generate-acpx-profiles.mjs --check");
for (const declaration of [
"test-fixtures/pi-acp/profile-v19-identity.json",
"test/fixtures/copilot-profile-v16-identity.json",
]) {
expect(dockerignore).toContain(`!packages/paperclip-runner/${declaration}`);
await expect(readFile(path.join(repositoryRoot, "packages/paperclip-runner", declaration), "utf8")).resolves.toBeTruthy();
}
for (const developmentOnlyInput of [
"packages/paperclip-runner/devtools",
"packages/paperclip-runner/docs",
@@ -200,10 +208,14 @@ describe("runner E2E Daytona image contract", () => {
"packages/paperclip-eval-kernel/src",
"packages/paperclip-runner/package.json",
"packages/paperclip-runner/scripts/candidate-provider-pack.mjs",
"packages/paperclip-runner/scripts/build-copilot-distribution.mjs",
"packages/paperclip-runner/scripts/materialize-copilot-binary.mjs",
"packages/paperclip-runner/scripts/materialize-cursor-distribution.mjs",
"packages/paperclip-runner/scripts/cursor-runtime-patch.mjs",
"packages/paperclip-runner/cursor-distributions.json",
"packages/paperclip-runner/runner/crates",
"packages/paperclip-runner/scripts/materialize-pi-distribution.mjs",
"packages/paperclip-runner/scripts/pi-distribution",
"packages/paperclip-runner/src",
]) {
expect(DAYTONA_IMAGE_INPUT_PATHS).toContain(requiredPath);
@@ -233,6 +245,8 @@ describe("runner E2E Daytona image contract", () => {
"packages/paperclip-runner/package.json",
"packages/paperclip-runner/src",
"packages/paperclip-runner/runner/crates",
"packages/paperclip-runner/scripts/materialize-pi-distribution.mjs",
"packages/paperclip-runner/scripts/pi-distribution",
] as const;
const options = {
repositoryRoot: root,
@@ -278,7 +292,11 @@ describe("runner E2E Daytona image contract", () => {
),
'pub const VERSION: &str = "one";\n',
);
await mkdir(path.join(root, "packages/paperclip-runner/scripts/pi-distribution"), { recursive: true });
for (const relativePath of [
"packages/paperclip-runner/scripts/materialize-pi-distribution.mjs",
"packages/paperclip-runner/scripts/pi-distribution/package.json",
"packages/paperclip-runner/scripts/pi-distribution/package-lock.json",
]) await writeFile(path.join(root, relativePath), "version one\n");
const baseline = await computeDaytonaImageContentId(options);
const candidate = await computeDaytonaImageContentId({ ...options, candidateProviders: ["pi"] });
@@ -315,6 +333,9 @@ describe("runner E2E Daytona image contract", () => {
"pnpm-lock.yaml",
"packages/paperclip-runner/package.json",
"packages/paperclip-runner/src/runner.ts",
"packages/paperclip-runner/scripts/materialize-pi-distribution.mjs",
"packages/paperclip-runner/scripts/pi-distribution/package.json",
"packages/paperclip-runner/scripts/pi-distribution/package-lock.json",
"packages/paperclip-runner/runner/crates/runner-core/src/lib.rs",
]) {
const absolutePath = path.join(root, relativePath);
tests/runner-e2e/denied-target-isolation-flow.test.ts
+44
View File
@@ -0,0 +1,44 @@
import { mkdtemp, readdir, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { expect, it, vi } from "vitest";
import { runnerMatrix } from "./catalog.js";
import { runCopilotProtectionFlow } from "./copilot-protection-flow.js";
import { runCursorNativeFlow } from "./cursor-native-flow.js";
import { cursorDeniedCommand } from "./cursor-native-evidence.js";
const dispatch = vi.hoisted(() => ({ inspect: async (_input: { prompt: string }) => {} }));
vi.mock("./user-actions.js", () => ({ createTaskThroughUi: async (input: { prompt: string }) => {
await dispatch.inspect(input); throw new Error("stop-before-provider-dispatch");
} }));
vi.mock("@playwright/test", () => ({ expect: (actual: unknown, message?: string) => ({ toBe: (expected: unknown) => expect(actual, message).toBe(expected) }) }));
it.each(["copilot", "cursor"] as const)("arms an isolated %s target before dispatch and binds the exact native operation", async candidate => {
const workspacePath = await mkdtemp(join(tmpdir(), "denial-dispatch-"));
const taskId = candidate === "copilot" ? "native-permission-deny-write" : "native-write-deny-reconnect";
const execution = runnerMatrix.find(row => row.profile.qualificationCandidate === candidate && row.environment.id === "local" && row.task.id === taskId)!;
const receipts = new Map<string, any>(); const cleanup: Array<() => Promise<unknown>> = [];
let target: string | undefined;
dispatch.inspect = async ({ prompt }) => {
const parents = (await readdir(workspacePath)).filter(name => name.startsWith("pc-denied-"));
expect(parents).toHaveLength(1);
expect(await readdir(join(workspacePath, parents[0]!))).toEqual([]);
target = `${parents[0]}/${candidate}-denied-nonce.txt`;
if (candidate === "copilot") expect(prompt).toContain(`creating ${target} with DENIED-nonce`);
else expect(prompt).toContain(cursorDeniedCommand(join(workspacePath, target)).command);
await writeFile(join(workspacePath, "unrelated-startup-file"), "runtime setup");
};
const api = { get: async (path: string) => {
if (path === "/api/agents/agent") return { adapterConfig: { lifecycleMode: "per_turn" } };
throw new Error(`Unexpected API call ${path}`);
}, patch: async (_path: string, value: unknown) => value, post: async () => ({ name: "fixture project" }) };
const input = { page: {}, api, fixtures: { company: { id: "company", issuePrefix: "FIX" }, agent: { id: "agent", name: "Fixture" }, environment: { id: "environment" } }, execution, nonce: "nonce", workspacePath, deadlineAt: Date.now() + 2000,
observe: () => {}, capture: async () => {}, evidence: async (name: string, value: unknown) => { receipts.set(name, value); }, registerCleanupAssertion: (callback: () => Promise<unknown>) => cleanup.push(callback) };
try {
await expect((candidate === "copilot" ? runCopilotProtectionFlow : runCursorNativeFlow)(input as any)).rejects.toThrow("stop-before-provider-dispatch");
if (candidate === "cursor") await expect(cleanup[0]!()).rejects.toThrow(/cleanup proof/);
const receipt = candidate === "copilot" ? receipts.get("copilot-protection-checks.json").watchReceipt : receipts.get("cursor-native-denial-cleanup-attempt.json").watcher;
expect(receipt).toMatchObject({ complete: true, targetMutationCount: 0, reasons: [] });
expect(target).toBeDefined();
} finally { await rm(workspacePath, { recursive: true, force: true }); }
});
tests/runner-e2e/extended-harnesses.test.ts
+76 -2
View File
@@ -1,5 +1,6 @@
import { assertRemoteNativeEvidencePrerequisites } from "./prerequisites.js";
import { describe, expect, it } from "vitest";
import { runnerMatrix, runnerSuites, extendedHarnessProfiles, extendedHarnessFileTask } from "./catalog.js";
import { runnerMatrix, runnerSuites, extendedHarnessProfiles, extendedHarnessFileTask, daytonaWarmContinuityTask } from "./catalog.js";
import { buildRunnerE2EProcessEnvironment, buildPaperclipServerEnvironment } from "./harness-env.js";
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
import { findSecretLeak, redactText } from "./redaction.js";
@@ -18,7 +19,7 @@ describe("extended ACP harness qualification", () => {
});
it("uses exact discovered models, encrypted credential references and current qualification metadata", () => {
expect(extendedHarnessProfiles.map(profile => profile.model)).toEqual([
"gpt-5.6-luna[context=272k,reasoning=medium,fast=false]", "gpt-5.6-luna", "openrouter/deepseek/deepseek-v4-flash-0731",
"gpt-5.6-luna[context=272k,reasoning=medium,fast=false]", "gpt-5.6-luna", "openrouter/anthropic/claude-sonnet-4.6",
]);
for (const profile of extendedHarnessProfiles) {
expect(profile.modelQualification.source).toBe(profile.qualificationCandidate === "cursor" ? "qualified_runner_profile" : "candidate_runner_profile");
@@ -38,6 +39,79 @@ describe("extended ACP harness qualification", () => {
const pending = selected.find(value => value.profile.qualificationCandidate === "copilot")!;
expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...pending, suite: { ...pending.suite, manualOnly: false } }])).toThrow("explicit");
});
it("adds six explicit local/Daytona warm cells without admitting them implicitly", () => {
const warm = runnerMatrix.filter(cell => cell.suite.id === "rich-acp-warm-continuity");
expect(warm).toHaveLength(6);
expect(new Set(warm.map(cell => cell.profile.qualificationCandidate))).toEqual(new Set(["cursor", "copilot", "pi"]));
for (const cell of warm) {
expect(cell.task).toMatchObject({ flow: "warm_three_turn", expectedRunCount: 3 });
expect(cell.task.buildMatchers("nonce", cell)).toContainEqual({ kind: "environment", expected: cell.environment.id });
const config = cell.profile.buildAgent({ executionId: "warm", environmentId: "env", environmentFixtureId: cell.environment.id, workspacePath: "/tmp/workspace", secretRefs: { [cell.profile.credential]: { type: "secret_ref", secretId: "11111111-1111-4111-8111-111111111111", version: "latest" } } }).adapterConfig;
expect(config).toMatchObject({ lifecycleMode: "warm", idleTimeoutMs: 300_000, timeoutSec: 120 });
const admission = buildRunnerE2EProcessEnvironment({}, [cell]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION;
if (cell.profile.qualificationCandidate !== "copilot") expect(admission).toBeUndefined();
else expect(JSON.parse(admission!)).toEqual([{ agent: cell.profile.qualificationCandidate, model: cell.profile.model }]);
expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...cell, suite: { ...cell.suite, manualOnly: false } }])).toThrow("explicit");
}
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(cell => cell.suite.id === "rich-acp-warm-continuity")).toBe(false);
});
it("separates ACP process continuity from managed-home checkpoint writes", () => {
for (const cell of runnerMatrix.filter(cell => cell.suite.id === "rich-acp-warm-continuity")) {
const prompts = [cell.task.buildPrompt("nonce"), ...cell.task.buildFollowupMessages!("nonce")];
expect(prompts).toHaveLength(3);
for (const [index, prompt] of prompts.entries()) {
expect(prompt).toContain(`warm Daytona continuity turn ${index + 1} of 3`);
expect(prompt).toContain("daytona-warm-nonce.txt");
expect(prompt).not.toContain("AGENT_HOME");
expect(prompt).not.toContain("notes/");
}
expect(cell.task).toMatchObject({ turnTimeoutMs: 120_000, attemptTimeoutMs: { local: 420_000, daytona: 420_000 } });
}
const managed = [daytonaWarmContinuityTask.buildPrompt("nonce"), ...daytonaWarmContinuityTask.buildFollowupMessages!("nonce")];
expect(managed.every(prompt => prompt.includes("AGENT_HOME") && prompt.includes("notes/warm-memory.txt"))).toBe(true);
expect(managed[0]).toContain("8388608 bytes");
expect(managed[1]).toContain("Delete notes/delete-me.txt");
expect(managed[2]).toContain("Verify notes/delete-me.txt is absent");
});
it("admits native qualification only for its matching provider and explicit suite", () => {
for (const suiteId of ["cursor-native", "pi-native", "copilot-protection"]) {
const cells = runnerMatrix.filter(cell => cell.suite.id === suiteId);
expect(new Set(cells.map(cell => cell.environment.id))).toEqual(new Set(["local", "daytona"]));
for (const cell of cells) {
const admission = buildRunnerE2EProcessEnvironment({}, [cell]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION;
if (cell.profile.qualificationCandidate !== "copilot") expect(admission).toBeUndefined();
else expect(admission).toBeDefined();
expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...cell, profile: { ...cell.profile, qualificationCandidate: cell.profile.qualificationCandidate === "pi" ? "cursor" : "pi" } }])).toThrow("explicit");
}
}
});
it("admits all four explicit active-Stop cells only for Cursor and Copilot", () => {
const cells = selectRunnerExecutions(parseRunnerSelectors(["--suite", "native-active-stop"]));
expect(cells).toHaveLength(4);
expect(new Set(cells.map(cell => `${cell.profile.qualificationCandidate}:${cell.environment.id}`))).toEqual(
new Set(["cursor:local", "cursor:daytona", "copilot:local", "copilot:daytona"]),
);
for (const cell of cells) {
expect(cell.suite.manualOnly).toBe(true);
const admission = buildRunnerE2EProcessEnvironment({ PAPERCLIP_RUNNER_ACPX_QUALIFICATION: "ambient" }, [cell]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION;
if (cell.profile.qualificationCandidate === "copilot") expect(JSON.parse(admission!)).toEqual([{ agent: "copilot", model: cell.profile.model }]);
else expect(admission).toBeUndefined();
expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...cell, suite: { ...cell.suite, manualOnly: false } }])).toThrow("explicit");
expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...cell, profile: { ...cell.profile, qualificationCandidate: "pi" } }])).toThrow("explicit");
expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...cell, suite: { ...cell.suite, id: "unrelated-manual-suite" } }])).toThrow("explicit");
}
const implicit = selectRunnerExecutions(parseRunnerSelectors(["--all"]));
expect(implicit.some(cell => cell.suite.id === "native-active-stop")).toBe(false);
});
it("requires both image executable pins for remote native cases before setup", () => {
const remote = runnerMatrix.filter(cell => cell.environment.id === "daytona" && ["cursor-native", "pi-native", "copilot-protection"].includes(cell.suite.id));
const valid = { PAPERCLIP_E2E_DAYTONA_NODE_SHA256: `sha256:${"a".repeat(64)}`, PAPERCLIP_E2E_DAYTONA_RUNNERD_SHA256: `sha256:${"b".repeat(64)}` };
expect(() => assertRemoteNativeEvidencePrerequisites(remote, valid)).not.toThrow();
for (const field of Object.keys(valid)) expect(() => assertRemoteNativeEvidencePrerequisites(remote, { ...valid, [field]: "latest" })).toThrow(field);
expect(() => assertRemoteNativeEvidencePrerequisites(remote, {})).toThrow("exact image executable digests");
expect(() => assertRemoteNativeEvidencePrerequisites(selected, {})).not.toThrow();
expect(() => assertRemoteNativeEvidencePrerequisites(runnerMatrix.filter(cell => cell.environment.id === "local"), {})).not.toThrow();
});
it("grades the actual file independently of the model's validation claim", () => {
const cell = selected.find(cell => cell.task.id === "file-edit-validate")!;
expect(extendedHarnessFileTask.buildMatchers("nonce", cell)).toContainEqual({ kind: "file_exact", path: "extended-nonce.txt", expected: "verified-nonce\n" });
tests/runner-e2e/failure-classifier.ts
+3
View File
@@ -34,6 +34,9 @@ export function classifyFailure(error: unknown): FailureClass {
: "cleanup_failure";
}
if (PERMANENT.test(message)) return "permanent_infrastructure";
// A deliberately killed Pi child is the fixture stimulus. Missing lifecycle
// proof must not become a retryable transport failure because of its wording.
if (/pi_provider_death_proof:/.test(message)) return "candidate_failure";
if (
/chat_idle_state_invariant/.test(message) ||
NON_RETRYABLE_SESSION_CLOSE.test(message) ||
tests/runner-e2e/fixtures/pi-file-evidence-actual-stream.json
+798
View File
@@ -0,0 +1,798 @@
{
"schema": "paperclip.e2e.sanitized-pi-file-stream.v1",
"description": "44 retained native Rust-projected edit/bash rows; only fixture nonce and execution identities replaced. Timestamps, private identities and unrelated events omitted. This calibrates the oracle, not the original failed attempt result.",
"events": [
{
"seq": 71,
"sourceSeq": 46,
"eventType": "tool.execution.started",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "edit (pending)",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 72,
"sourceSeq": 47,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "tool call (pending)",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 73,
"sourceSeq": 48,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "tool call (pending)",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 74,
"sourceSeq": 49,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "tool call (pending)",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 75,
"sourceSeq": 50,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "tool call (pending)",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 76,
"sourceSeq": 51,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "tool call (pending)",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 77,
"sourceSeq": 52,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "tool call (pending)",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 78,
"sourceSeq": 53,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "tool call (pending)",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 79,
"sourceSeq": 54,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "tool call (pending)",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 80,
"sourceSeq": 55,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "tool call (pending)",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 81,
"sourceSeq": 56,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "tool call (pending)",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 82,
"sourceSeq": 57,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "tool call (pending)",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 83,
"sourceSeq": 58,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "tool call (pending)",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 84,
"sourceSeq": 59,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "tool call (pending)",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 85,
"sourceSeq": 60,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": "extended-fixture.txt",
"exitCode": null,
"progress": "tool call (pending): extended-fixture.txt",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 86,
"sourceSeq": 61,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": "extended-fixture.txt",
"exitCode": null,
"progress": "tool call (pending): extended-fixture.txt",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 88,
"sourceSeq": 63,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "running",
"target": "extended-fixture.txt",
"exitCode": null,
"progress": "tool call (in_progress): extended-fixture.txt",
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 89,
"sourceSeq": 64,
"eventType": "tool.execution.completed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "edit",
"operation": "edit",
"transport": "builtin",
"status": "completed",
"target": "extended-fixture.txt",
"exitCode": null,
"progress": null,
"output": "null",
"outputTruncated": false,
"executionId": "edit"
}
},
{
"seq": 92,
"sourceSeq": 66,
"eventType": "tool.execution.started",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 93,
"sourceSeq": 67,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 94,
"sourceSeq": 68,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 95,
"sourceSeq": 69,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 96,
"sourceSeq": 70,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 97,
"sourceSeq": 71,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 98,
"sourceSeq": 72,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 99,
"sourceSeq": 73,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 100,
"sourceSeq": 74,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 101,
"sourceSeq": 75,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 102,
"sourceSeq": 76,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 103,
"sourceSeq": 77,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 104,
"sourceSeq": 78,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 105,
"sourceSeq": 79,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 106,
"sourceSeq": 80,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 107,
"sourceSeq": 81,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 108,
"sourceSeq": 82,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 109,
"sourceSeq": 83,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 110,
"sourceSeq": 84,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 111,
"sourceSeq": 85,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "bash (pending): [terminal] validate",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 112,
"sourceSeq": 86,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "node -e 'if(require(\"node:fs\").readFileSync(\"extended-fixture.txt\",\"utf8\")!==\"verified-fixture\\n\")throw Error(\"mismatch\");console.log(\"PI-VALIDATED-fixture\")' (pending): node -e 'if(require(\"node:fs\").readFileSync(\"extended-fixture.txt\",\"utf8\")!==\"verified-fixture\\n\")throw Error(\"mismatch\");console.log(\"PI-VALIDATED-fixture\")'",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 113,
"sourceSeq": 87,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "node -e 'if(require(\"node:fs\").readFileSync(\"extended-fixture.txt\",\"utf8\")!==\"verified-fixture\\n\")throw Error(\"mismatch\");console.log(\"PI-VALIDATED-fixture\")' (pending): node -e 'if(require(\"node:fs\").readFileSync(\"extended-fixture.txt\",\"utf8\")!==\"verified-fixture\\n\")throw Error(\"mismatch\");console.log(\"PI-VALIDATED-fixture\")'",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 115,
"sourceSeq": 89,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "node -e 'if(require(\"node:fs\").readFileSync(\"extended-fixture.txt\",\"utf8\")!==\"verified-fixture\\n\")throw Error(\"mismatch\");console.log(\"PI-VALIDATED-fixture\")' (in_progress): node -e 'if(require(\"node:fs\").readFileSync(\"extended-fixture.txt\",\"utf8\")!==\"verified-fixture\\n\")throw Error(\"mismatch\");console.log(\"PI-VALIDATED-fixture\")'",
"output": "null",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 116,
"sourceSeq": 90,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "node -e 'if(require(\"node:fs\").readFileSync(\"extended-fixture.txt\",\"utf8\")!==\"verified-fixture\\n\")throw Error(\"mismatch\");console.log(\"PI-VALIDATED-fixture\")' (in_progress): node -e 'if(require(\"node:fs\").readFileSync(\"extended-fixture.txt\",\"utf8\")!==\"verified-fixture\\n\")throw Error(\"mismatch\");console.log(\"PI-VALIDATED-fixture\")'",
"output": "{\"content\":[]}",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 117,
"sourceSeq": 91,
"eventType": "tool.execution.progressed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "running",
"target": null,
"exitCode": null,
"progress": "node -e 'if(require(\"node:fs\").readFileSync(\"extended-fixture.txt\",\"utf8\")!==\"verified-fixture\\n\")throw Error(\"mismatch\");console.log(\"PI-VALIDATED-fixture\")' (in_progress): node -e 'if(require(\"node:fs\").readFileSync(\"extended-fixture.txt\",\"utf8\")!==\"verified-fixture\\n\")throw Error(\"mismatch\");console.log(\"PI-VALIDATED-fixture\")'",
"output": "{\"content\":[{\"type\":\"text\",\"text\":\"PI-VALIDATED-fixture\\n\"}],\"details\":{}}",
"outputTruncated": false,
"executionId": "validate"
}
},
{
"seq": 118,
"sourceSeq": 92,
"eventType": "tool.execution.completed",
"payload": {
"schema": "paperclip.tool.execution.v1",
"name": "bash",
"operation": "execute",
"transport": "builtin",
"status": "completed",
"target": null,
"exitCode": null,
"progress": null,
"output": "{\"content\":[{\"type\":\"text\",\"text\":\"PI-VALIDATED-fixture\\n\"}],\"structuredContent\":{\"output\":\"PI-VALIDATED-fixture\\n\",\"truncated\":false,\"exit_code\":0,\"wall_time_seconds\":0}}",
"outputTruncated": false,
"executionId": "validate"
}
}
]
}
tests/runner-e2e/fixtures/pi-native-restart-actual-prefix.json
+182
View File
@@ -0,0 +1,182 @@
{
"description": "Sanitized identity-only projection of Pi 1.0.0 profile13 pending-input stream; original submission omits turnId before turn.started assigns it. No restart was reached in the original failing attempt. Prompt, reasoning, timestamps and private execution metadata omitted.",
"state": {
"issue": {
"id": "issue",
"companyId": "company",
"status": "in_progress"
},
"runs": [
{
"id": "run",
"companyId": "company",
"nativeIssueId": "issue",
"runtimeMode": "native",
"status": "running",
"nativeSessionId": "session",
"runnerInstanceId": "runner"
}
],
"interactions": [
{
"id": "interaction",
"companyId": "company",
"issueId": "issue",
"kind": "ask_user_questions",
"status": "pending",
"result": null,
"sourceRunId": "run",
"continuationPolicy": "none",
"resolverPolicy": "human_only",
"idempotencyKey": "paperclip-runner-question:run:request",
"payload": {
"version": 1,
"title": "Pi native restart",
"submitLabel": "Submit answers",
"supersedeOnUserComment": false,
"questions": [
{
"id": "answer",
"prompt": "Answer",
"helpText": "Pi native restart",
"selectionMode": "single",
"required": true,
"allowOther": true,
"options": [
{
"id": "paperclip_text_answer",
"label": "Pi native restart",
"description": "Expected text input",
"freeText": true
}
]
}
],
"questionSet": {
"schema": "paperclip.question_set.v1",
"title": "Pi native restart",
"submitLabel": "Submit answers",
"questions": [
{
"id": "answer",
"header": "Pi native restart",
"prompt": "Answer",
"required": true,
"answerMode": "text",
"textValidation": {
"inputType": "text"
}
}
]
},
"runtimeRequestId": "request"
}
}
]
},
"events": [
{
"companyId": "company",
"runId": "run",
"seq": 25,
"protocolSchemaVersion": 1,
"eventType": "turn.submitted",
"payload": {
"prpEvent": {
"schema": "paperclip.prp.event.v1",
"schemaVersion": 1,
"sourceKind": "runner",
"eventType": "turn.submitted",
"runId": "run",
"normalizedSessionId": "session",
"sourceInstanceId": "runner",
"sourceSeq": 6,
"sourceEventId": "runner:run:6",
"payload": {
"text": "Sanitized fixture request"
}
}
}
},
{
"companyId": "company",
"runId": "run",
"seq": 28,
"protocolSchemaVersion": 1,
"eventType": "turn.started",
"payload": {
"prpEvent": {
"schema": "paperclip.prp.event.v1",
"schemaVersion": 1,
"sourceKind": "runner",
"eventType": "turn.started",
"runId": "run",
"turnId": "turn",
"normalizedSessionId": "session",
"sourceInstanceId": "runner",
"sourceSeq": 7,
"sourceEventId": "runner:run:7",
"payload": {
"status": "inProgress"
}
}
}
},
{
"companyId": "company",
"runId": "run",
"seq": 222,
"protocolSchemaVersion": 1,
"eventType": "runtime_request.created",
"payload": {
"prpEvent": {
"schema": "paperclip.prp.event.v1",
"schemaVersion": 1,
"sourceKind": "runner",
"eventType": "runtime_request.created",
"runId": "run",
"turnId": "turn",
"itemId": "item-run",
"normalizedSessionId": "session",
"sourceInstanceId": "runner",
"sourceSeq": 195,
"sourceEventId": "runner:run:195",
"payload": {
"request": {
"type": "input",
"input": {
"title": "Pi native restart",
"schema": "paperclip.question_set.v1",
"questions": [
{
"id": "answer",
"header": "Pi native restart",
"prompt": "Answer",
"required": true,
"answerMode": "text",
"textValidation": {
"inputType": "text"
}
}
],
"submitLabel": "Submit answers"
},
"itemId": "item-run",
"origin": {
"method": "elicitation/create",
"adapter": "acpx-runtime-sidecar",
"provider": "pi"
},
"prompt": "Pi native restart",
"schema": "paperclip.runtime_request.v2",
"status": "pending",
"turnId": "turn",
"requestId": "request",
"requestKind": "runtime"
}
}
}
}
}
]
}
tests/runner-e2e/harness-env.ts
+14 -3
View File
@@ -1,3 +1,6 @@
import { installedDaytonaPluginKeys } from "./installed-daytona-plugin.js";
import { QUALIFIED_ACPX_PROFILES } from "../../packages/paperclip-runner/src/drivers/acpx/qualified-profiles.js";
import { installedCliKeys } from "./installed-cli.js";
import path from "node:path";
import { chatNeedsApiTools, isManagedHiringCase } from "./chat-cases.js";
import { CREDENTIAL_NAMES } from "./types.js";
@@ -114,12 +117,20 @@ export function buildRunnerE2EProcessEnvironment(
const candidates = new Map<string, string>();
for (const execution of executions) {
const agent = execution.profile.qualificationCandidate;
if (!agent || agent === "cursor") continue;
if (!agent) continue;
const admittedSuite = execution.suite.id === "extended-harnesses"
|| execution.suite.id === "rich-acp-warm-continuity";
|| execution.suite.id === "rich-acp-warm-continuity"
|| (execution.suite.id === "pi-native" && agent === "pi")
|| (execution.suite.id === "pi-controls" && agent === "pi")
|| (["cursor-native", "native-provider-loss"].includes(execution.suite.id) && agent === "cursor")
|| (execution.suite.id === "copilot-protection" && agent === "copilot")
|| (execution.suite.id === "native-active-stop" && (agent === "cursor" || agent === "copilot"));
if (!admittedSuite || !execution.suite.manualOnly) {
throw new Error("Candidate qualification requires an explicit provider qualification suite");
}
if (QUALIFIED_ACPX_PROFILES[agent].qualificationStatus !== "pending") {
continue;
}
const prior = candidates.get(agent);
if (prior !== undefined && prior !== execution.profile.model) throw new Error("Conflicting candidate models");
candidates.set(agent, execution.profile.model);
@@ -169,7 +180,7 @@ export function buildPaperclipServerEnvironment(
]) {
delete result[key];
}
for (const key of GENERATED_SERVER_SECRET_KEYS) delete result[key];
for (const key of [...GENERATED_SERVER_SECRET_KEYS, ...installedCliKeys, ...installedDaytonaPluginKeys]) delete result[key];
Object.assign(result, overrides);
return result;
}
tests/runner-e2e/installed-cli.test.ts
+94
View File
@@ -0,0 +1,94 @@
import { execFileSync } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdtemp, mkdir, realpath, rm, symlink, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { afterEach, expect, it } from "vitest";
import { runnerMatrix } from "./catalog.js";
import { assertInstalledCliSelection, installedCliKeys, verifyInstalledCli } from "./installed-cli.js";
import { buildPaperclipServerEnvironment, buildRunnerE2EProcessEnvironment } from "./harness-env.js";
const cells = runnerMatrix.filter(e => e.profile.id === "runner-acpx-pi");
const roots: string[] = [];
afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
const hash = (value: string) => createHash("sha256").update(value).digest("hex");
async function fixture() {
const root = await realpath(await mkdtemp(join(tmpdir(), "e2e-installed-cli-"))); roots.push(root);
const cli = join(root, "node_modules/paperclipai"); const server = join(root, "node_modules/@paperclipai/server");
await mkdir(join(cli, "dist"), { recursive: true }); await mkdir(join(server, "dist"), { recursive: true });
await writeFile(join(cli, "package.json"), JSON.stringify({ name: "paperclipai", version: "1.2.3", type: "module", bin: { paperclipai: "./dist/index.js" } }));
await writeFile(join(server, "package.json"), JSON.stringify({ name: "@paperclipai/server", version: "1.2.3", type: "module", exports: { ".": { import: "./dist/index.js" } } }));
await writeFile(join(cli, "dist/index.js"), "// installed public CLI"); await writeFile(join(server, "dist/index.js"), "// installed public server");
const env = { PAPERCLIP_RUNNER_E2E_INSTALLED_CLI: join(cli, "dist/index.js"), PAPERCLIP_RUNNER_E2E_INSTALLED_CLI_SHA256: hash("// installed public CLI"), PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_ROOT: server, PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_SHA256: hash("// installed public server") };
return { root, cli, server, env };
}
it("admits all26 current Pi cases through pinned installed public CLI/server with no overrides", async () => {
const { env, server } = await fixture(); expect(cells).toHaveLength(26);
for (const cell of cells) expect(await verifyInstalledCli(env, [cell])).toMatchObject({ serverRoot: server, defaultRuntimeResolution: true, qualificationOverride: false });
});
it("qualified providers drop ambient admission while pending Copilot retains its exact gate", () => {
for (const cell of cells) expect(buildRunnerE2EProcessEnvironment({ PAPERCLIP_RUNNER_ACPX_QUALIFICATION: "ambient" }, [cell]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION).toBeUndefined();
for (const agent of ["copilot"]) {
const cell = runnerMatrix.find(e => e.profile.qualificationCandidate === agent)!;
expect(JSON.parse(buildRunnerE2EProcessEnvironment({}, [cell]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION!)).toEqual([{ agent, model: cell.profile.model }]);
}
const changed = { ...cells[0]!, profile: { ...cells[0]!.profile, model: "custom/explicit-model" } };
expect(buildRunnerE2EProcessEnvironment({}, [changed]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION).toBeUndefined();
});
it("rejects partial authority, unsupported cells and every runtime/path override", async () => {
const { env } = await fixture();
for (const key of installedCliKeys) { const changed = { ...env, [key]: undefined }; await expect(verifyInstalledCli(changed, [cells[0]!])).rejects.toThrow("complete pins"); }
expect(() => assertInstalledCliSelection(env, [])).toThrow("explicit supported Pi");
expect(() => assertInstalledCliSelection(env, [runnerMatrix.find(e => e.profile.qualificationCandidate === "cursor")!])).toThrow("explicit supported Pi");
for (const key of ["PAPERCLIP_RUNNER_BINARY", "PAPERCLIP_RUNNER_REMOTE_BINARY_PATH", "PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH", "PAPERCLIP_RUNNER_ACPX_QUALIFICATION", "PAPERCLIP_ACPX_BUILTIN_ROOT", "PAPERCLIP_ACPX_PROVIDER_PACKAGE_ROOT", "PAPERCLIP_ACPX_PROVIDER_PACKAGE_MANIFEST", "NODE_OPTIONS", "NODE_PATH"]) await expect(verifyInstalledCli({ ...env, [key]: "foreign" }, [cells[0]!])).rejects.toThrow("forbids");
});
it("rejects stale bytes and another server root instead of falling back to source", async () => {
const f = await fixture(); const other = await fixture();
await expect(verifyInstalledCli({ ...f.env, PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_ROOT: other.server }, [cells[0]!])).rejects.toThrow("foreign server");
await expect(verifyInstalledCli({ ...f.env, PAPERCLIP_RUNNER_E2E_INSTALLED_CLI_SHA256: "0".repeat(64) }, [cells[0]!])).rejects.toThrow("reviewed pins");
await writeFile(join(f.server, "dist/index.js"), "changed");
await expect(verifyInstalledCli(f.env, [cells[0]!])).rejects.toThrow("reviewed pins");
});
it("rejects linked entrypoints and source-export package identities", async () => {
const f = await fixture(); const other = await fixture();
await rm(join(f.cli, "dist/index.js")); await symlink(join(other.cli, "dist/index.js"), join(f.cli, "dist/index.js"));
await expect(verifyInstalledCli(f.env, [cells[0]!])).rejects.toThrow("canonical and unlinked");
await rm(join(f.cli, "dist/index.js")); await writeFile(join(f.cli, "dist/index.js"), "// installed public CLI");
await writeFile(join(f.server, "package.json"), JSON.stringify({ name: "@paperclipai/server", version: "1.2.3", exports: { ".": "./src/index.ts" } }));
await expect(verifyInstalledCli(f.env, [cells[0]!])).rejects.toThrow("public package identity");
});
it("keeps installed proof inputs out of the production server environment", async () => {
const { env } = await fixture(); const server = buildPaperclipServerEnvironment(env);
for (const key of installedCliKeys) expect(server[key]).toBeUndefined();
await expect(verifyInstalledCli({}, [])).resolves.toBeUndefined();
});
it.skipIf(process.platform === "win32")("rejects a FIFO entrypoint without waiting for a writer", async () => {
const f = await fixture(); await rm(f.env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI);
execFileSync("/usr/bin/mkfifo", [f.env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI], { timeout: 5000, env: {} });
await expect(verifyInstalledCli(f.env, [cells[0]!])).rejects.toThrow("bounded regular file");
});
it("uses the direct public Playwright JS bin without bin-shim Node injection", async () => {
const { runnerE2EPlaywrightInvocation } = await import("./web-server-command.js");
const f = await fixture(); const cli = join(f.root,"node_modules/@playwright/test/cli.js");
await mkdir(join(f.root,"node_modules/@playwright/test"),{recursive:true});
await writeFile(cli,"console.log(JSON.stringify({nodePath:process.env.NODE_PATH??null,nodeOptions:process.env.NODE_OPTIONS??null,args:process.argv.slice(2)}))");
const invocation = runnerE2EPlaywrightInvocation(f.root,["--version"],true);
expect(invocation.command).toBe(process.execPath);
expect(JSON.parse(execFileSync(invocation.command,invocation.args,{env:{},timeout:5000,encoding:"utf8"}))).toEqual({nodePath:null,nodeOptions:null,args:["--version"]});
expect(runnerE2EPlaywrightInvocation(f.root,["--version"],false)).toEqual({command:"pnpm",args:["exec","playwright","--version"]});
for (const key of ["NODE_PATH","NODE_OPTIONS"]) expect(()=>assertInstalledCliSelection({...f.env,[key]:""},[cells[0]!])).toThrow("ambient Node injection");
});
it("startup-only accepts only explicit uncredentialed local hello and zero retries", async () => {
const { assertInstalledStartupOnly } = await import("./installed-cli.js");
const { parseRunnerSelectors } = await import("./selectors.js");
const f=await fixture(); const cell=cells.find(e=>e.id==="extended-harnesses.runner-acpx-pi.local.hello-complete")!;
const options=parseRunnerSelectors(["--installed-startup-only","--id",cell.id,"--max-automatic-retries","0"]);
expect(options.installedStartupOnly).toBe(true);expect(()=>assertInstalledStartupOnly(f.env,[cell],options)).not.toThrow();
for(const delta of [{maxAutomaticRetries:1},{maxParallel:2},{ui:true},{debug:true},{headed:true},{list:true},{matrixJson:true},{ids:[]}])expect(()=>assertInstalledStartupOnly(f.env,[cell],{...options,...delta})).toThrow("startup-only");
expect(()=>assertInstalledStartupOnly(f.env,[cells.find(e=>e.environment.id==="daytona")!],options)).toThrow("startup-only");
expect(()=>assertInstalledStartupOnly({...f.env,OPENROUTER_API_KEY:"dummy-not-a-real-key"},[cell],options)).toThrow("credential inputs");
expect(()=>assertInstalledStartupOnly({...f.env,KIMI_MODEL_API_KEY:"dummy-not-a-real-key"},[cell],options)).toThrow("credential inputs");
expect(()=>assertInstalledStartupOnly({},[cell],options)).toThrow("startup-only");
});
tests/runner-e2e/installed-cli.ts
+59
View File
@@ -0,0 +1,59 @@
/** Optional published-install lane. No production hook or admission override. */
import { constants } from "node:fs";
import { lstat, open, realpath } from "node:fs/promises";
import { createHash } from "node:crypto";
import { findPackageJSON } from "node:module";
import { dirname, isAbsolute, join, resolve } from "node:path";
import { pathToFileURL } from "node:url";
import { CREDENTIAL_NAMES, type MatrixExecution } from "./types.js";
export const installedCliKeys = ["PAPERCLIP_RUNNER_E2E_INSTALLED_CLI", "PAPERCLIP_RUNNER_E2E_INSTALLED_CLI_SHA256", "PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_ROOT", "PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_SHA256"] as const;
const overrideKeys = ["PAPERCLIP_RUNNER_BINARY", "PAPERCLIP_RUNNER_REMOTE_BINARY_PATH", "PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH", "PAPERCLIP_RUNNER_ACPX_QUALIFICATION", "PAPERCLIP_ACPX_BUILTIN_ROOT", "PAPERCLIP_ACPX_PROVIDER_PACKAGE_ROOT", "PAPERCLIP_ACPX_PROVIDER_PACKAGE_MANIFEST"] as const;
const suites = new Set(["extended-harnesses", "pi-native", "pi-controls", "rich-acp-warm-continuity"]);
export function usesInstalledCli(env: NodeJS.ProcessEnv): boolean {
return installedCliKeys.some(key => env[key] !== undefined);
}
export function assertInstalledCliSelection(env: NodeJS.ProcessEnv, executions: readonly MatrixExecution[]): void {
if (!usesInstalledCli(env)) return;
if (!installedCliKeys.every(key => Boolean(env[key])) || executions.length === 0 || executions.some(e => e.profile.id !== "runner-acpx-pi" || e.profile.qualificationCandidate !== "pi" || !e.suite.manualOnly || !suites.has(e.suite.id))) throw new Error("Installed CLI proof requires complete pins and explicit supported Pi cases");
for (const key of overrideKeys) if (env[key] !== undefined) throw new Error(`Installed CLI proof forbids ${key}`);
if (env.NODE_OPTIONS !== undefined || env.NODE_PATH !== undefined) throw new Error("Installed CLI proof forbids ambient Node injection");
}
async function readOwned(path: string, maximum: number): Promise<Buffer> {
const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
try {
const before = await handle.stat({ bigint: true });
if (!before.isFile() || before.size < 1n || before.size > BigInt(maximum)) throw new Error("Installed CLI proof requires a bounded regular file");
const bytes = await handle.readFile(); const after = await handle.stat({ bigint: true }); const named = await lstat(path, { bigint: true });
if (before.dev !== after.dev || before.ino !== after.ino || before.size !== after.size || before.mtimeNs !== after.mtimeNs || before.ctimeNs !== after.ctimeNs || named.dev !== before.dev || named.ino !== before.ino || named.isSymbolicLink() || bytes.length !== Number(before.size)) throw new Error("Installed CLI proof file changed during admission");
return bytes;
} finally { await handle.close(); }
}
export async function verifyInstalledCli(env: NodeJS.ProcessEnv, executions: readonly MatrixExecution[]) {
if (!usesInstalledCli(env)) return undefined;
assertInstalledCliSelection(env, executions);
const entry = env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI!;
const serverRoot = env.PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_ROOT!;
for (const path of [entry, serverRoot]) if (!isAbsolute(path) || resolve(path) !== path || await realpath(path) !== path) throw new Error("Installed CLI proof paths must be canonical and unlinked");
if (!entry.endsWith("/dist/index.js")) throw new Error("Installed CLI proof requires published dist/index.js");
const cliRoot = resolve(dirname(entry), "..");
const cli = JSON.parse((await readOwned(join(cliRoot, "package.json"), 65536)).toString());
const manifest = findPackageJSON("@paperclipai/server", pathToFileURL(entry));
if (!manifest || await realpath(manifest) !== join(serverRoot, "package.json")) throw new Error("Installed CLI resolved a foreign server dependency");
const server = JSON.parse((await readOwned(join(serverRoot, "package.json"), 65536)).toString());
if (cli.name !== "paperclipai" || cli.bin?.paperclipai !== "./dist/index.js" || server.name !== "@paperclipai/server" || typeof cli.version !== "string" || cli.version !== server.version || server.exports?.["."]?.import !== "./dist/index.js") throw new Error("Installed CLI/server public package identity differs");
const serverEntry = join(serverRoot, "dist/index.js");
if (await realpath(serverEntry) !== serverEntry) throw new Error("Installed server entrypoint escapes its package");
const digest = (bytes: Buffer) => createHash("sha256").update(bytes).digest("hex");
const cliSha256 = digest(await readOwned(entry, 64 * 1024 * 1024));
const serverSha256 = digest(await readOwned(serverEntry, 16 * 1024 * 1024));
if (cliSha256 !== env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI_SHA256 || serverSha256 !== env.PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_SHA256) throw new Error("Installed CLI/server bytes differ from their reviewed pins");
return { schema: "paperclip.e2e.installed-cli-admission/v1", entry, cliRoot, cliSha256, serverRoot, serverEntry, serverSha256, version: cli.version, defaultRuntimeResolution: true, qualificationOverride: false };
}
/** This prep lane cannot create agents or read the normal local credential file. */
export function assertInstalledStartupOnly(env: NodeJS.ProcessEnv, executions: readonly MatrixExecution[], options: { ids: string[]; maxParallel: number; maxAutomaticRetries: number; headed: boolean; ui: boolean; debug: boolean; list: boolean; matrixJson: boolean }) {
assertInstalledCliSelection(env, executions);
if (!usesInstalledCli(env) || executions.length !== 1 || executions[0]?.id !== "extended-harnesses.runner-acpx-pi.local.hello-complete" || options.ids.length !== 1 || options.ids[0] !== executions[0]?.id || options.maxParallel !== 1 || options.maxAutomaticRetries !== 0 || options.headed || options.ui || options.debug || options.list || options.matrixJson) throw new Error("Installed startup-only requires one explicit local Pi hello, no retries or interactive mode");
if (CREDENTIAL_NAMES.some(key => env[key] !== undefined) || Object.keys(env).some(key => /^(?:OPENAI|ANTHROPIC|OPENROUTER|DAYTONA|XAI|GROK|CURSOR|COPILOT|GITHUB|GH)(?:_|$)/.test(key) && env[key] !== undefined)) throw new Error("Installed startup-only forbids provider credential inputs");
}
tests/runner-e2e/installed-daytona-plugin.test.ts
+96
View File
@@ -0,0 +1,96 @@
import { execFileSync } from "node:child_process";
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import { mkdtemp, mkdir, realpath, rm, symlink, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { afterEach, expect, it, vi } from "vitest";
import { runnerMatrix } from "./catalog.js";
import { installedDaytonaPluginKeys, verifyInstalledDaytonaPlugin, type InstalledDaytonaPluginAuthority } from "./installed-daytona-plugin.js";
import { buildPaperclipServerEnvironment } from "./harness-env.js";
import { setupLiveFixtures } from "./live-fixtures.js";
import type { RunnerApi } from "./api.js";
const cell = runnerMatrix.find(e => e.id === "extended-harnesses.runner-acpx-pi.daytona.hello-complete")!;
const sdkVersion: string = JSON.parse(readFileSync(new URL("../../packages/plugins/sdk/package.json", import.meta.url), "utf8")).version;
const roots: string[] = [];
const hash = (v: string) => createHash("sha256").update(v).digest("hex");
afterEach(async () => { vi.unstubAllEnvs(); await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
async function fixture() {
const root = await realpath(await mkdtemp(join(tmpdir(), "e2e-installed-plugin-"))); roots.push(root);
async function pkg(name: string, version: string, dependencies: Record<string,string> = {}, extra = {}) {
const dir = join(root, "node_modules", name); const entry = name === "@daytonaio/sdk" ? "./esm/index.js" : "./dist/index.js"; await mkdir(join(dir, name === "@daytonaio/sdk" ? "esm" : "dist"), { recursive: true });
const content = JSON.stringify({ name, version, type: "module", exports: { ".": { import: name === "@daytonaio/sdk" ? { types: "./esm/index.d.ts", default: entry } : entry } }, dependencies, ...extra });
await writeFile(join(dir, "package.json"), content); await writeFile(join(dir, entry), "// compiled");
return { root: dir, packageSha256: hash(content), entry, entrySha256: hash("// compiled") };
}
const plugin = await pkg("@paperclipai/plugin-daytona", "0.1.1", { "@paperclipai/plugin-sdk": sdkVersion, "@daytonaio/sdk": "0.203.0" }, { paperclipPlugin: { manifest: "./dist/manifest.js", worker: "./dist/worker.js" } });
await writeFile(join(plugin.root, "dist/manifest.js"), "// manifest"); await writeFile(join(plugin.root, "dist/worker.js"), "// worker");
const authority: InstalledDaytonaPluginAuthority = { schema: "paperclip.e2e.installed-daytona-plugin/v1", graphRoot: root, plugin: { ...plugin, manifestSha256: hash("// manifest"), workerSha256: hash("// worker") }, sdk: await pkg("@paperclipai/plugin-sdk", sdkVersion, { "@paperclipai/shared": "0.3.1" }), shared: await pkg("@paperclipai/shared", "0.3.1"), daytona: await pkg("@daytonaio/sdk", "0.203.0") };
const authorityPath = join(root, "authority.json");
const env: NodeJS.ProcessEnv = { PAPERCLIP_RUNNER_E2E_INSTALLED_CLI: "/reviewed/cli", PAPERCLIP_RUNNER_E2E_INSTALLED_CLI_SHA256: "reviewed-separately", PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_ROOT: "/reviewed/server", PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_SHA256: "reviewed-separately", PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN: plugin.root, PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN_AUTHORITY: authorityPath };
async function seal() { const bytes = JSON.stringify(authority); await writeFile(authorityPath, bytes); env.PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN_AUTHORITY_SHA256 = hash(bytes); }
await seal(); return { root, authority, env, seal };
}
it("admits pinned published plugin and resolved SDK graph without claiming a full graph audit", async () => {
const f = await fixture(); expect(await verifyInstalledDaytonaPlugin(f.env, [cell])).toMatchObject({ packageRoot: f.authority.plugin.root, sourceFallback: false, fullGraphVerified: false, completeGraphAuthorityRequired: true });
});
it("fails closed for absent/partial installed Daytona pins but preserves ordinary source and local lanes", async () => {
const f = await fixture();
for (const key of installedDaytonaPluginKeys) await expect(verifyInstalledDaytonaPlugin({ ...f.env, [key]: undefined }, [cell])).rejects.toThrow("complete pins");
await expect(verifyInstalledDaytonaPlugin({}, [cell])).resolves.toBeUndefined();
const local = runnerMatrix.find(e => e.id === "extended-harnesses.runner-acpx-pi.local.hello-complete")!;
await expect(verifyInstalledDaytonaPlugin(f.env, [local])).rejects.toThrow("Daytona selection");
const noPlugin = { ...f.env }; for (const key of installedDaytonaPluginKeys) delete noPlugin[key];
await expect(verifyInstalledDaytonaPlugin(noPlugin, [local])).resolves.toBeUndefined();
await expect(verifyInstalledDaytonaPlugin(noPlugin, [cell])).rejects.toThrow("complete pins");
});
it("rejects stale authority, worker, manifest and dependency entry bytes", async () => {
const f = await fixture();
await expect(verifyInstalledDaytonaPlugin({ ...f.env, PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN_AUTHORITY_SHA256: "0".repeat(64) }, [cell])).rejects.toThrow("reviewed pins");
for (const [path, original] of [[join(f.authority.plugin.root,"dist/worker.js"),"// worker"], [join(f.authority.plugin.root,"dist/manifest.js"),"// manifest"], [join(f.authority.sdk.root,"dist/index.js"),"// compiled"]]) {
await writeFile(path!, "// wrong"); await expect(verifyInstalledDaytonaPlugin(f.env,[cell])).rejects.toThrow("reviewed pins"); await writeFile(path!,original!);
}
});
it("rejects source exports and workspace dependency versions even with refreshed pins", async () => {
const f = await fixture(); const p = join(f.authority.plugin.root,"package.json");
const original = { name:"@paperclipai/plugin-daytona",version:"0.1.1",exports:{".":{import:"./dist/index.js"}},paperclipPlugin:{manifest:"./dist/manifest.js",worker:"./dist/worker.js"},dependencies:{"@paperclipai/plugin-sdk":sdkVersion,"@daytonaio/sdk":"0.203.0"} };
for (const manifest of [{ ...original, exports:{".":{import:"./src/index.ts"}} }, { ...original, dependencies:{ ...original.dependencies,"@paperclipai/plugin-sdk":"workspace:*" } }]) {
const bytes=JSON.stringify(manifest);await writeFile(p,bytes);f.authority.plugin.packageSha256=hash(bytes);await f.seal();await expect(verifyInstalledDaytonaPlugin(f.env,[cell])).rejects.toThrow(/compiled package exports|dependency identity/);
}
});
it("rejects linked or escaped roots and a shadowed foreign resolved dependency", async () => {
const f = await fixture(); const other = await fixture();
f.authority.shared=other.authority.shared;await f.seal();await expect(verifyInstalledDaytonaPlugin(f.env,[cell])).rejects.toThrow("escapes");
f.authority.shared={...other.authority.shared,root:join(f.root,"node_modules/@paperclipai/shared")};await f.seal();
const nested=join(f.authority.plugin.root,"node_modules/@paperclipai");await mkdir(nested,{recursive:true});await symlink(other.authority.sdk.root,join(nested,"plugin-sdk"));
await expect(verifyInstalledDaytonaPlugin(f.env,[cell])).rejects.toThrow("foreign dependency");
await rm(nested,{recursive:true});await rm(f.authority.shared.root,{recursive:true});await symlink(other.authority.shared.root,f.authority.shared.root);
await expect(verifyInstalledDaytonaPlugin(f.env,[cell])).rejects.toThrow("canonical and unlinked");
});
it.skipIf(process.platform === "win32")("rejects a FIFO authority without waiting for a writer", async () => {
const f=await fixture();const p=f.env.PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN_AUTHORITY!;await rm(p);execFileSync("/usr/bin/mkfifo",[p],{timeout:5000,env:{}});await expect(verifyInstalledDaytonaPlugin(f.env,[cell])).rejects.toThrow("bounded unlinked regular file");
});
it("uses the verified public package in the existing API and rejects changed bytes before any mutation", async () => {
// This positive fixture models the sanitized installed launcher, not pnpm's parent environment.
vi.stubEnv("NODE_OPTIONS", undefined); vi.stubEnv("NODE_PATH", undefined);
const f=await fixture(); for(const [k,v]of Object.entries(f.env))vi.stubEnv(k,v!);
const calls:unknown[]=[];
const api={async post(url:string,data:unknown){calls.push({url,data});throw new Error("stop-after-public-install");}} as unknown as RunnerApi;
const input={api,execution:cell,executionNonce:"nonce",workspacePath:"/tmp/workspace",credentials:{},daytonaImage:"ghcr.io/example/image@sha256:"+"a".repeat(64)};
await expect(setupLiveFixtures(input)).rejects.toThrow("stop-after-public-install");
expect(calls).toEqual([{url:"/api/plugins/install",data:{packageName:f.authority.plugin.root,isLocalPath:true}}]);
calls.length=0;await writeFile(join(f.authority.plugin.root,"dist/worker.js"),"// changed");await expect(setupLiveFixtures(input)).rejects.toThrow("reviewed pins");expect(calls).toEqual([]);
});
it.each(["NODE_OPTIONS", "NODE_PATH"])("rejects ambient %s before the plugin install API", async (key) => {
const f = await fixture();
for (const [name, value] of Object.entries(f.env)) vi.stubEnv(name, value!);
vi.stubEnv("NODE_OPTIONS", undefined); vi.stubEnv("NODE_PATH", undefined);
const post = vi.fn();
const api = { post } as unknown as RunnerApi;
for (const value of ["", "foreign"]) {
vi.stubEnv(key, value);
await expect(setupLiveFixtures({ api, execution: cell, executionNonce: "nonce", workspacePath: "/tmp/workspace", credentials: {} })).rejects.toThrow("ambient Node injection");
expect(post).not.toHaveBeenCalled();
}
});
it("keeps fixture authority out of the production server environment", async()=>{const f=await fixture();const env=buildPaperclipServerEnvironment(f.env);for(const k of installedDaytonaPluginKeys)expect(env[k]).toBeUndefined();});
tests/runner-e2e/installed-daytona-plugin.ts
+72
View File
@@ -0,0 +1,72 @@
/** Public plugin fixture admission; the complete installed graph is audited separately. */
import { constants } from "node:fs";
import { lstat, open, realpath } from "node:fs/promises";
import { createHash } from "node:crypto";
import { findPackageJSON } from "node:module";
import { isAbsolute, join, resolve } from "node:path";
import { pathToFileURL } from "node:url";
import { assertInstalledCliSelection, usesInstalledCli } from "./installed-cli.js";
import type { MatrixExecution } from "./types.js";
export const installedDaytonaPluginKeys = ["PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN", "PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN_AUTHORITY", "PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN_AUTHORITY_SHA256"] as const;
type PackagePin = { root: string; packageSha256: string; entry: string; entrySha256: string };
export interface InstalledDaytonaPluginAuthority {
schema: "paperclip.e2e.installed-daytona-plugin/v1";
graphRoot: string;
plugin: PackagePin & { manifestSha256: string; workerSha256: string };
sdk: PackagePin;
shared: PackagePin;
daytona: PackagePin;
}
const digest = (bytes: Buffer) => createHash("sha256").update(bytes).digest("hex");
async function canonical(path: string) {
if (typeof path !== "string" || !isAbsolute(path) || resolve(path) !== path || await realpath(path) !== path) throw new Error("Installed Daytona plugin path must be canonical and unlinked");
}
async function pinned(path: string, expected: string, maximum = 32 * 1024 * 1024) {
if (!/^[a-f0-9]{64}$/.test(expected)) throw new Error("Installed Daytona plugin requires a SHA-256 pin");
await canonical(path);
const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
try {
const before = await handle.stat({ bigint: true });
if (!before.isFile() || before.size < 1n || before.size > BigInt(maximum) || before.nlink !== 1n) throw new Error("Installed Daytona plugin requires a bounded unlinked regular file");
const bytes = await handle.readFile(); const after = await handle.stat({ bigint: true }); const named = await lstat(path, { bigint: true });
if (before.dev !== after.dev || before.ino !== after.ino || before.size !== after.size || before.mtimeNs !== after.mtimeNs || before.ctimeNs !== after.ctimeNs || named.dev !== before.dev || named.ino !== before.ino || bytes.length !== Number(before.size) || digest(bytes) !== expected) throw new Error("Installed Daytona plugin bytes differ from reviewed pins");
return bytes;
} finally { await handle.close(); }
}
function compiledEntry(manifest: any): string | undefined {
const exported = manifest.exports?.["."];
return typeof exported === "string" ? exported : (typeof exported?.import === "string" ? exported.import : exported?.import?.default) ?? manifest.main;
}
export async function verifyInstalledDaytonaPlugin(env: NodeJS.ProcessEnv, executions: readonly MatrixExecution[]) {
const selected = installedDaytonaPluginKeys.some(key => env[key] !== undefined);
const required = usesInstalledCli(env) && executions.some(e => e.environment.id === "daytona");
if (!selected && !required) return undefined;
assertInstalledCliSelection(env, executions);
if (!usesInstalledCli(env) || !installedDaytonaPluginKeys.every(key => Boolean(env[key])) || executions.some(e => e.environment.id !== "daytona")) throw new Error("Installed Daytona plugin requires complete pins and installed Pi Daytona selection");
const authorityPath = env.PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN_AUTHORITY!;
const authoritySha256 = env.PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN_AUTHORITY_SHA256!;
const a: InstalledDaytonaPluginAuthority = JSON.parse((await pinned(authorityPath, authoritySha256, 65536)).toString());
if (a.schema !== "paperclip.e2e.installed-daytona-plugin/v1" || a.plugin?.root !== env.PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN) throw new Error("Installed Daytona plugin authority differs");
await canonical(a.graphRoot);
async function packageAt(pin: PackagePin, name: string) {
await canonical(pin.root);
if (pin.root !== join(a.graphRoot, "node_modules", name)) throw new Error("Installed Daytona plugin package escapes its reviewed graph");
const manifest = JSON.parse((await pinned(join(pin.root, "package.json"), pin.packageSha256, 65536)).toString());
if (manifest.name !== name || typeof manifest.version !== "string" || !/^\d+\.\d+\.\d+(?:[-+][a-zA-Z0-9.-]+)?$/.test(manifest.version) || pin.entry !== (name === "@daytonaio/sdk" ? "./esm/index.js" : "./dist/index.js") || compiledEntry(manifest) !== pin.entry) throw new Error("Installed Daytona plugin requires published compiled package exports");
await pinned(join(pin.root, pin.entry), pin.entrySha256);
return manifest;
}
const plugin = await packageAt(a.plugin, "@paperclipai/plugin-daytona");
const sdk = await packageAt(a.sdk, "@paperclipai/plugin-sdk");
const shared = await packageAt(a.shared, "@paperclipai/shared");
const daytona = await packageAt(a.daytona, "@daytonaio/sdk");
if (plugin.paperclipPlugin?.manifest !== "./dist/manifest.js" || plugin.paperclipPlugin?.worker !== "./dist/worker.js" || plugin.dependencies?.["@paperclipai/plugin-sdk"] !== sdk.version || sdk.dependencies?.["@paperclipai/shared"] !== shared.version || plugin.dependencies?.["@daytonaio/sdk"] !== "0.203.0" || daytona.version !== "0.203.0") throw new Error("Installed Daytona plugin public dependency identity differs");
await pinned(join(a.plugin.root, "dist/manifest.js"), a.plugin.manifestSha256);
await pinned(join(a.plugin.root, "dist/worker.js"), a.plugin.workerSha256);
for (const [name, parent, pin] of [["@paperclipai/plugin-sdk", join(a.plugin.root, "dist/worker.js"), a.sdk], ["@paperclipai/shared", join(a.sdk.root, a.sdk.entry), a.shared], ["@daytonaio/sdk", join(a.plugin.root, "dist/worker.js"), a.daytona]] as const) {
const found = findPackageJSON(name, pathToFileURL(parent));
if (!found || await realpath(found) !== join(pin.root, "package.json")) throw new Error("Installed Daytona plugin resolves a foreign dependency");
}
return { schema: a.schema, authorityPath, authoritySha256, packageRoot: a.plugin.root, graphRoot: a.graphRoot, fullGraphVerified: false, completeGraphAuthorityRequired: true, sourceFallback: false };
}
tests/runner-e2e/installed-startup.spec.ts
+31
View File
@@ -0,0 +1,31 @@
/** Credential-free proof of the real launcher/Playwright/WebServer chain. */
import { test, expect } from "@playwright/test";
import { writeFile } from "node:fs/promises";
import path from "node:path";
test("installed controller serves health and UI without creating provider work", async ({ page, request, baseURL }) => {
test.skip(process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_STARTUP_ONLY !== "1");
expect(baseURL).toMatch(/^http:\/\/127\.0\.0\.1:\d+$/);
const errors: string[] = [];
page.on("pageerror", error => errors.push(error.message));
await page.route("**/*", route => {
const url = new URL(route.request().url());
return url.origin === baseURL ? route.continue() : route.abort("blockedbyclient");
});
const health = await request.get(`${baseURL}/api/health`);
expect(health.ok()).toBe(true);
const before = await request.get(`${baseURL}/api/companies`);
expect(before.ok()).toBe(true); expect(await before.json()).toEqual([]);
const response = await page.goto(baseURL!);
expect(response?.ok()).toBe(true);
await expect(page.locator("#root")).not.toBeEmpty();
await expect(page.getByRole("heading", { name: "What is the name of your organization?", exact: true })).toBeVisible();
await expect(page.getByRole("textbox", { name: "Name", exact: true })).toBeVisible();
await expect(page.getByRole("button", { name: "Continue", exact: true })).toBeDisabled();
expect(errors).toEqual([]);
const after = await request.get(`${baseURL}/api/companies`);
expect(after.ok()).toBe(true); expect(await after.json()).toEqual([]);
const output = process.env.PAPERCLIP_RUNNER_E2E_PRIVATE_DIR!;
await page.screenshot({ path: path.join(output, "installed-startup-only.png"), fullPage: true });
await writeFile(path.join(output, "installed-startup-only.json"), `${JSON.stringify({ status: "health_ui_zero_company_passed", qualified: false, providerCalls: 0, healthStatus: health.status(), uiStatus: response!.status(), companiesBefore: 0, companiesAfter: 0, pageErrors: errors }, null, 2)}\n`, { mode: 0o600 });
});
tests/runner-e2e/launch.ts
+46 -11
View File
@@ -1,4 +1,8 @@
import { runnerE2EPlaywrightInvocation } from "./web-server-command.js";
import { verifyInstalledDaytonaPlugin } from "./installed-daytona-plugin.js";
import { assertInstalledCliSelection, assertInstalledStartupOnly, verifyInstalledCli } from "./installed-cli.js";
import { createProcessTreeOwner, stopOwnedProcessTree } from "./process-tree-owner.js";
import { createRunnerE2ETemporaryRoot } from "./server-config.js";
import { randomBytes } from "node:crypto";
import { prepareCodexCiSandbox, requiresCodexCiSandbox } from "./codex-ci-sandbox.js";
import { spawn } from "node:child_process";
@@ -12,8 +16,8 @@ import {
cp,
lstat,
mkdir,
mkdtemp,
readFile,
realpath,
readdir,
rm,
symlink,
@@ -262,7 +266,7 @@ async function prepareProviderPath(
}
async function runProcess(
args: string[],
invocation: { command: string; args: string[] },
env: NodeJS.ProcessEnv,
timeoutMs: number | null,
logPath: string,
@@ -270,7 +274,7 @@ async function runProcess(
interactive: boolean,
) {
const log = createWriteStream(logPath, { flags: "a", mode: 0o600 });
const child = spawn("pnpm", args, {
const child = spawn(invocation.command, invocation.args, {
cwd: repositoryRoot,
env,
stdio: ["inherit", "pipe", "pipe"],
@@ -477,15 +481,18 @@ async function runAttempt(input: {
"One isolated runner E2E harness cannot mix profiles or environments",
);
}
assertInstalledCliSelection(process.env, executions);
const installedCli = await verifyInstalledCli(process.env, executions);
const installedPlugin = await verifyInstalledDaytonaPlugin(process.env, executions);
const startedAtMs = Date.now();
const sharedMemoryBaseline = snapshotDarwinSharedMemory();
const temporaryRoot = await mkdtemp(
path.join(os.tmpdir(), "paperclip-runner-e2e-"),
);
const temporaryParent = await realpath(os.tmpdir());
const temporaryRoot = await createRunnerE2ETemporaryRoot(temporaryParent);
const publishedResults: RunnerE2EResult[] = [];
const publishedResultPaths = new Map<string, string>();
let attemptSecrets: string[] = [];
let processCleanupFailed = false;
let startupReceiptPath: string | undefined;
const rawCleanupResults: Array<{ cleanup: string; synthetic: boolean; status: string }> = [];
try {
const paperclipHome = path.join(temporaryRoot, "paperclip-home");
@@ -523,7 +530,9 @@ async function runAttempt(input: {
betterAuthSecret,
]);
attemptSecrets = credentials;
const runnerBinary = resolvePaperclipRunnerBinaryForHarness(
if (installedCli) await writeFile(path.join(privateDir, "installed-cli-admission.json"), `${JSON.stringify(installedCli, null, 2)}\n`, { mode: 0o600 });
if (installedPlugin) await writeFile(path.join(privateDir, "installed-daytona-plugin-admission.json"), `${JSON.stringify(installedPlugin, null, 2)}\n`, { mode: 0o600 });
const runnerBinary = installedCli ? undefined : resolvePaperclipRunnerBinaryForHarness(
executions,
repositoryRoot,
);
@@ -534,6 +543,7 @@ async function runAttempt(input: {
executions.map((candidate) => candidate.id),
),
PAPERCLIP_RUNNER_E2E_ATTEMPT: String(attempt),
PAPERCLIP_RUNNER_E2E_INSTALLED_STARTUP_ONLY: options.installedStartupOnly ? "1" : undefined,
PAPERCLIP_RUNNER_E2E_PUBLIC_MCP: executions.some(candidate => candidate.task.flow === "public_mcp") ? "1" : "0",
PAPERCLIP_RUNNER_E2E_PORT: String(port),
PAPERCLIP_RUNNER_E2E_TEMP_ROOT: temporaryRoot,
@@ -542,7 +552,7 @@ async function runAttempt(input: {
PAPERCLIP_RUNNER_E2E_SERVER_LOG: path.join(privateDir, "server.log"),
PAPERCLIP_RUNNER_BINARY: runnerBinary,
PAPERCLIP_RUNNER_REMOTE_BINARY_PATH:
resolvePaperclipRemoteRunnerBinaryForHarness(executions, runnerBinary),
installedCli ? undefined : resolvePaperclipRemoteRunnerBinaryForHarness(executions, runnerBinary),
// Vite's optimized dependency cache embeds revision query strings. A
// private per-attempt cache prevents an earlier cell or local rebuild
// from producing `504 Outdated Optimize Dep` during browser bootstrap.
@@ -569,8 +579,6 @@ async function runAttempt(input: {
delete childEnv.DATABASE_MIGRATION_URL;
const playwrightArgs = [
"exec",
"playwright",
"test",
"--config",
"tests/runner-e2e/playwright.config.ts",
@@ -590,7 +598,7 @@ async function runAttempt(input: {
) +
5 * 60_000;
const processResult = await runProcess(
playwrightArgs,
runnerE2EPlaywrightInvocation(repositoryRoot, playwrightArgs, Boolean(installedCli)),
childEnv,
watchdog,
path.join(privateDir, "playwright.log"),
@@ -600,6 +608,18 @@ async function runAttempt(input: {
options.ui || options.debug,
);
processCleanupFailed = processResult.processCleanupError !== null;
if (options.installedStartupOnly) {
const proofDir = path.join(resultsRoot, campaignId, "installed-startup-only");
await mkdir(proofDir, { recursive: true });
// This is private setup evidence, never a successful provider case/result.
await cp(privateDir, path.join(proofDir, "private"), { recursive: true });
const probe = JSON.parse(await readFile(path.join(privateDir, "installed-startup-only.json"), "utf8"));
if (processResult.exitCode !== 0 || processResult.timedOut || processResult.spawnError || processCleanupFailed || probe.status !== "health_ui_zero_company_passed" || probe.providerCalls !== 0 || probe.qualified !== false) throw new Error("Installed startup-only actual launch chain failed");
await assertEmbeddedDatabaseIsolation(configPath, temporaryRoot);
startupReceiptPath = path.join(proofDir, "receipt.json");
await writeFile(startupReceiptPath, `${JSON.stringify({ status: "installed_launch_chain_pending_scratch_cleanup", qualified: false, providerCalls: 0, credentialsLoaded: false, installedCli, processResult, probe }, null, 2)}\n`, { mode: 0o600 });
return [];
}
const processFailure = processResult.spawnError
? `Playwright failed to start: ${processResult.spawnError}`
: processResult.timedOut
@@ -844,6 +864,12 @@ async function runAttempt(input: {
`Refusing to remove unexpected temporary path ${temporaryRoot}`,
);
}
if (startupReceiptPath) {
const receipt = JSON.parse(await readFile(startupReceiptPath, "utf8"));
receipt.status = cleanupError ? "installed_launch_chain_cleanup_failed" : "installed_launch_chain_passed";
receipt.temporaryRootRemoved = !cleanupError;
await writeFile(startupReceiptPath, `${JSON.stringify(receipt, null, 2)}\n`, { mode: 0o600 });
}
if (cleanupError) {
const message = `Temporary runner E2E state cleanup failed at ${temporaryRoot}: ${cleanupError instanceof Error ? cleanupError.message : String(cleanupError)}`;
for (const publishedResult of publishedResults) {
@@ -868,6 +894,7 @@ async function runAttempt(input: {
// Cleanup failure is a failed cell, but must not suppress later cells in
// the same campaign. The result above carries the terminal failure.
console.error(message);
if (options.installedStartupOnly) throw new Error(message);
}
}
}
@@ -942,6 +969,14 @@ async function main() {
throw error;
}
const executions = selectRunnerExecutions(options, runnerMatrix);
if (options.installedStartupOnly) {
assertInstalledStartupOnly(process.env, executions, options);
assertRunnerE2EPrerequisites(executions);
const campaignId = cleanId(process.env.PAPERCLIP_E2E_CAMPAIGN_ID ?? `installed-startup-${Date.now()}`);
await runAttempt({ executions, attempt: 1, campaignId, options });
console.log(`PASS installed launch chain (no provider qualification) -> ${path.join(resultsRoot, campaignId, "installed-startup-only")}`);
return;
}
if (options.list) {
printList(executions);
return;
tests/runner-e2e/live-fixtures.ts
+3 -3
View File
@@ -1,3 +1,4 @@
import { verifyInstalledDaytonaPlugin } from "./installed-daytona-plugin.js";
import { NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js";
import path from "node:path";
import { installedReleaseDaytonaPlugin } from "./installed-release.js";
@@ -115,10 +116,9 @@ export async function setupLiveFixtures(input: {
registry.register<PluginRecord>({
id: "sandbox-provider",
async setup() {
const installed = await verifyInstalledDaytonaPlugin(process.env, [execution]);
return api.post<PluginRecord>("/api/plugins/install", {
packageName: process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI
? installedReleaseDaytonaPlugin(process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI, process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN, process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_DAYTONA_PLUGIN_VERSION)
: path.resolve(
packageName: installed?.packageRoot ?? path.resolve(
import.meta.dirname,
"../../packages/plugins/sandbox-providers/daytona",
),
tests/runner-e2e/native-active-stop-evidence.ts
+10 -9
View File
@@ -2,13 +2,14 @@ import { createHash } from "node:crypto";
import { canonicalJson } from "../../packages/shared/src/portability-hash.js";
import { hasAcpxNativeOrigin } from "./acpx-native-origin.js";
import { isValidNativePrpEnvelope } from "./native-event-envelope.js";
import { assertNativeRemoteRetirement, nativeRemoteDeniedSample, type NativeRemoteSnapshot } from "./native-remote-evidence.js";
import { assertCopilotRemoteRetirement, copilotRemoteDeniedSample, type CopilotRemoteSnapshot } from "./copilot-protection-evidence.js";
import { readCopilotToolEvidence } from "./copilot-evidence.js";
import { readCursorToolEvidence } from "./cursor-native-evidence.js";
import { bootstrapReadExecutionId, withoutProvenBootstrapReads, type BootstrapReadProof } from "./native-bootstrap-read-proof.js";
type Row = Record<string, any>;
export interface ActiveStopCaller { type: "board"; userId: "local-board"; source: "local_implicit" }
export type ActiveStopProvider = "cursor";
export type ActiveStopProvider = "cursor" | "copilot";
export interface ActiveStopScope { provider: ActiveStopProvider; companyId: string; issueId: string; runId: string; target: string; commandSha256?: string }
export interface ActiveStopPending {
schema: "paperclip.e2e.native-active-stop-pending.v2";
@@ -61,7 +62,7 @@ const closures = new Set(["runtime_request.resolved", "runtime_request.cancelled
function origin(events: readonly unknown[], scope: ActiveStopScope, bootstrap?: BootstrapReadProof) {
fail([scope.companyId, scope.issueId, scope.runId, scope.target].every(id) && ["cursor", "copilot"].includes(scope.provider), "exact scope required");
const rows = canonicalRows(events, scope);
const notices = readCursorToolEvidence(events, scope.runId);
const notices = scope.provider === "cursor" ? readCursorToolEvidence(events, scope.runId) : readCopilotToolEvidence(events, scope.runId);
const requests = notices.filter(n => n.stage === "permission_requested");
fail(requests.length === 1, "one native permission required");
const notice = requests[0]!;
@@ -119,7 +120,7 @@ export function observeActiveStopPending(input: { events: readonly unknown[]; ru
fail(isActiveStopCaller(input.caller) && uuid(input.cancellationRequestId) && run.id === scope.runId && run.companyId === scope.companyId && run.nativeIssueId === scope.issueId
&& run.status === "running" && run.runtimeMode === "native" && issue.id === scope.issueId && issue.companyId === scope.companyId && issue.status === "in_progress"
&& run.resultJson?.startupCancellation == null && run.resultJson?.nativeCancellation == null, "run is not fresh active work");
fail(!(readCursorToolEvidence(input.events, scope.runId)).some(n => n.stage === "permission_delivered"), "permission already answered");
fail(!(scope.provider === "cursor" ? readCursorToolEvidence(input.events, scope.runId) : readCopilotToolEvidence(input.events, scope.runId)).some(n => n.stage === "permission_delivered"), "permission already answered");
fail(!proof.rows.some(x => terminals.has(x.event.eventType) || closures.has(x.event.eventType)), "request or provider already settled");
fail(!proof.rows.some(x => x.event.eventType === "tool.execution.completed" && rec(x.event.payload).executionId === bootstrapReadExecutionId(proof.notice.toolCallId)), "operation already ended");
return { schema: "paperclip.e2e.native-active-stop-pending.v2", scope, caller: input.caller, requestId: proof.notice.requestId!, toolCallId: proof.notice.toolCallId,
@@ -180,7 +181,7 @@ export function readActiveStopSettlement(input: { events: readonly unknown[]; ru
export interface ActiveStopRemoteObservation {
phase: "before-request" | "pending" | "owned-process-retirement";
source: "live-snapshot" | "retirement-seal";
snapshot: NativeRemoteSnapshot;
snapshot: CopilotRemoteSnapshot;
}
/** A per-turn Daytona observer seals itself when the owned tree retires. The
@@ -192,10 +193,10 @@ export function readActiveStopRemoteRetirement(input: {
fail(observations.length === 3
&& observations.map(o => `${o.phase}:${o.source}`).join(",") === "before-request:live-snapshot,pending:live-snapshot,owned-process-retirement:retirement-seal",
"remote seal cannot stand in for a fresh causal sample");
const [baseline, pending, terminal] = observations.map(o => o.snapshot) as [NativeRemoteSnapshot, NativeRemoteSnapshot, NativeRemoteSnapshot];
const [baseline, pending, terminal] = observations.map(o => o.snapshot) as [CopilotRemoteSnapshot, CopilotRemoteSnapshot, CopilotRemoteSnapshot];
fail(baseline.binding.companyId === input.scope.companyId && baseline.binding.runId === input.scope.runId, "remote observation belongs to another run");
for (const snapshot of [baseline, pending, terminal]) {
fail(!nativeRemoteDeniedSample(snapshot, baseline, input.scope.target, "pending").exists, "remote target changed");
fail(!copilotRemoteDeniedSample(snapshot, baseline, input.scope.target, "pending").exists, "remote target changed");
}
fail(BigInt(baseline.observedMonotonicNs) < BigInt(pending.observedMonotonicNs)
&& BigInt(pending.observedMonotonicNs) < BigInt(terminal.observedMonotonicNs), "remote observation reused or out of order");
@@ -203,8 +204,8 @@ export function readActiveStopRemoteRetirement(input: {
&& baseline.processes.captured && pending.processes.captured
&& baseline.processes.live.includes(baseline.processes.root?.pid ?? -1)
&& pending.processes.live.includes(pending.processes.root?.pid ?? -1), "remote pending lifetime unproven");
assertNativeRemoteRetirement(terminal, baseline);
assertNativeRemoteRetirement(terminal, pending);
assertCopilotRemoteRetirement(terminal, baseline);
assertCopilotRemoteRetirement(terminal, pending);
for (const snapshot of [baseline, pending]) {
fail(snapshot.processes.journal.every(p => terminal.processes.journal.some(q =>
p.pid === q.pid && p.startTicks === q.startTicks && p.bootId === q.bootId)), "remote descendant journal lost");
tests/runner-e2e/native-active-stop-flow.ts
+12 -15
View File
@@ -4,8 +4,8 @@ import { expect, type Page } from "@playwright/test";
import { pollUntil, type RunnerApi } from "./api.js";
import { collectRunEvents } from "./run-observations.js";
import { createTaskThroughUi } from "./user-actions.js";
import { createDeniedTargetFixture, exists, observeRunProcesses } from "./native-local-fixtures.js";
import { assertNativeRemoteRetirement, nativeRemoteDeniedSample, prepareNativeRemoteAction, type NativeRemoteBootstrap, type NativeRemoteFixture, type NativeRemoteSnapshot } from "./native-remote-evidence.js";
import { createDeniedTargetFixture, exists, observeRunProcesses } from "./copilot-local-fixtures.js";
import { assertCopilotRemoteRetirement, copilotRemoteDeniedSample, prepareCopilotRemoteAction, type CopilotRemoteBootstrap, type CopilotRemoteFixture, type CopilotRemoteSnapshot } from "./copilot-protection-evidence.js";
import { cursorDeniedCommand } from "./cursor-native-evidence.js";
import { assertActiveStopRetirement, readActiveStopCaller, observeActiveStopPending, readActiveStopSettlement, type ActiveStopRemoteObservation, type ActiveStopCaller, type ActiveStopPending, type ActiveStopScope } from "./native-active-stop-evidence.js";
import type { BootstrapReadProof } from "./native-bootstrap-read-proof.js";
@@ -43,7 +43,7 @@ export async function stopAtPendingPermission(input: {
}
export async function runNativeActiveStopFlow(input: {
page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution; nonce: string; workspacePath: string; deadlineAt: number;
remoteBootstrap?: NativeRemoteBootstrap;
remoteBootstrap?: CopilotRemoteBootstrap;
registerCleanupAssertion(callback: () => Promise<Check[]>): void;
registerBeforeEnvironmentTeardownAssertion(callback: () => Promise<Check[]>): void;
observe(issue: Row, runs: Row[]): void;
@@ -51,7 +51,7 @@ export async function runNativeActiveStopFlow(input: {
evidence(name: string, value: unknown): Promise<void>;
}) {
const { api, page, fixtures, execution, nonce } = input, provider = execution.profile.qualificationCandidate;
if (provider !== "cursor" || execution.task.id !== "pending-permission-stop") throw new Error("Unknown native active Stop case");
if ((provider !== "cursor" && provider !== "copilot") || execution.task.id !== "pending-permission-stop") throw new Error("Unknown native active Stop case");
const remote = execution.environment.id === "daytona";
if ((!remote && execution.environment.id !== "local") || (remote && !input.remoteBootstrap)) throw new Error("Active Stop requires an isolated admitted environment");
const checks: Check[] = []; let issue: Row = {}, runs: Row[] = [], events: Row[] = [];
@@ -65,15 +65,12 @@ export async function runNativeActiveStopFlow(input: {
const remoteObservations: ActiveStopRemoteObservation[] = [];
let retirement: ReturnType<typeof assertActiveStopRetirement> | undefined;
const samples: Array<{ phase: string; absent: boolean }> = [];
let fixture: NativeRemoteFixture | undefined, baseline: NativeRemoteSnapshot | undefined, sealed: NativeRemoteSnapshot | undefined;
let fixture: CopilotRemoteFixture | undefined, baseline: CopilotRemoteSnapshot | undefined, sealed: CopilotRemoteSnapshot | undefined;
let completed: Awaited<ReturnType<typeof stopAtPendingPermission>> | undefined;
const observeProcesses = () => {
// Remote ownership comes from the sandbox boot/start-tick journal. The
// API PID transitions from the controller command to the remote runner.
if (!observer) return;
const run = runs[0], authority = run?.processPid ? { pid: run.processPid, groupId: run.processGroupId, startedAt: run.processStartedAt, runId: run.id } : undefined;
if (authority) { const key = JSON.stringify(authority); if (processIdentity && processIdentity !== key) processError = true; processIdentity ??= key; }
processes = observer.sample(authority);
if (observer) processes = observer.sample(authority);
};
const load = async (): Promise<ActiveStopState> => {
if (issue.id) issue = await api.get<Row>(`/api/issues/${issue.id}`);
@@ -90,7 +87,7 @@ export async function runNativeActiveStopFlow(input: {
if (phase !== "pending") throw new Error("Remote filesystem phase requires a fresh live snapshot");
const snap = await fixture.snapshot(phase);
remoteObservations.push({ phase, source: "live-snapshot", snapshot: snap });
value = !nativeRemoteDeniedSample(snap, baseline, target, "pending").exists;
value = !copilotRemoteDeniedSample(snap, baseline, target, "pending").exists;
await input.evidence(`active-stop-${phase}-remote.json`, snap);
} else value = !await exists(local!.targetPath);
samples.push({ phase, absent: value }); check(`no-effect-${phase}`, value, "Exact target remains absent at the causal observation boundary");
@@ -103,7 +100,7 @@ export async function runNativeActiveStopFlow(input: {
await load();
if (remote) {
if (!fixture || !baseline) throw new Error("Remote observer was never armed");
sealed ??= await fixture.finish(); assertNativeRemoteRetirement(sealed, baseline); processes = sealed.processes;
sealed ??= await fixture.finish(); assertCopilotRemoteRetirement(sealed, baseline); processes = sealed.processes;
} else if (processes.captured && processes.live.length) {
await pollUntil({ label: "active Stop provider retirement", deadlineAt: Date.now() + 5000, intervalMs: 100, load: async () => { observeProcesses(); return processes; }, accept: p => p.live.length === 0 });
}
@@ -136,13 +133,13 @@ export async function runNativeActiveStopFlow(input: {
check("explicit-per-turn-policy", configured.adapterConfig?.acpxPermissionMode === "approve-reads" && configured.adapterConfig?.lifecycleMode === "per_turn" && (provider !== "cursor" || configured.adapterConfig?.acpxSessionMode === "agent"), "Agent mode and per-turn restrictive permission policy selected before startup");
const project = await api.post<Row>(`/api/companies/${fixtures.company.id}/projects`, { name: `Native active Stop ${nonce}`, executionWorkspacePolicy: { enabled: true, defaultMode: "shared_workspace", sharedWorkspaceConcurrency: "serialize", allowIssueOverride: false, environmentId: fixtures.environment.id, workspaceStrategy: { type: "project_primary" } }, workspace: { name: "Primary", sourceType: "local_path", cwd: input.workspacePath, isPrimary: true } });
if (!remote) await sample("before-request");
const createdTask = await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), prompt: remote ? input.remoteBootstrap!.prompt(nonce) : prompt(), workMode: "standard", projectName: project.name, requireExplicitTitle: true });
issue = (await pollUntil({ label: "browser-created active Stop task", deadlineAt: input.deadlineAt, load: async () => (await api.get<Row[]>(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(i => i.id === createdTask.issueId), accept: Boolean }))!;
await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), prompt: remote ? input.remoteBootstrap!.prompt(nonce) : prompt(), workMode: "standard", projectName: project.name });
issue = (await pollUntil({ label: "browser-created active Stop task", deadlineAt: input.deadlineAt, load: async () => (await api.get<Row[]>(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(i => i.title === execution.task.buildTitle(nonce)), accept: Boolean }))!;
if (remote) {
await pollUntil({ label: "active Stop remote bootstrap", deadlineAt: input.deadlineAt, load, accept: state => state.run.status === "running" });
const bound = await input.remoteBootstrap!.bindAndRelease({ issueId: issue.id, runId: runs[0]!.id, targets: [target], actionPrompt: async value => {
fixture = value; if (provider === "cursor") command = cursorDeniedCommand(join(value.remoteCwd, target));
const prepared = await prepareNativeRemoteAction({ fixture: value, companyId: fixtures.company.id, environmentId: fixtures.environment.id, runId: runs[0]!.id, target, prompt: prompt() });
const prepared = await prepareCopilotRemoteAction({ fixture: value, companyId: fixtures.company.id, environmentId: fixtures.environment.id, runId: runs[0]!.id, target, prompt: prompt() });
baseline = prepared.baseline;
remoteObservations.push({ phase: "before-request", source: "live-snapshot", snapshot: baseline });
await input.evidence("active-stop-before-request-remote.json", baseline); return prepared.prompt;
@@ -168,7 +165,7 @@ export async function runNativeActiveStopFlow(input: {
if (remote) {
// finish drains the observer's automatic retirement seal. It does not
// extend the remote watch through subsequent host UI/cleanup assertions.
sealed = await fixture!.finish(); assertNativeRemoteRetirement(sealed, baseline!); processes = sealed.processes;
sealed = await fixture!.finish(); assertCopilotRemoteRetirement(sealed, baseline!); processes = sealed.processes;
remoteObservations.push({ phase: "owned-process-retirement", source: "retirement-seal", snapshot: sealed });
await input.evidence("active-stop-owned-process-retirement-remote.json", sealed);
} else await sample("after-stop");
tests/runner-e2e/native-active-stop.test.ts
+12 -9
View File
@@ -5,6 +5,7 @@ import { stopAtPendingPermission } from "./native-active-stop-flow.js";
import { runnerMatrix, runnerSuites, suiteDefinitionHash } from "./catalog.js";
import { selectRunnerExecutions, parseRunnerSelectors } from "./selectors.js";
import { createCursorToolEvidence } from "../../packages/paperclip-runner/src/drivers/acpx/cursor-tool-evidence.js";
import { createCopilotToolEvidence } from "../../packages/paperclip-runner/src/drivers/acpx/copilot-tool-evidence.js";
import type { CanonicalProviderEvent } from "../../packages/paperclip-runner/src/provider-events.js";
import { CodexSessionState } from "../../packages/paperclip-runner/src/drivers/codex/codex-session-state.js";
@@ -13,7 +14,7 @@ import { mapTerminalTurn } from "../../packages/paperclip-runner/src/drivers/cod
const caller = readActiveStopCaller({ deploymentMode: "local_trusted" }, { session: { userId: "local-board", id: "paperclip:local_implicit:local-board" } });
const cancellationRequestId = "11111111-2222-4333-8444-555555555555";
type Row = Record<string, any>;
function fixture(provider: ActiveStopProvider = "cursor") {
function fixture(provider: ActiveStopProvider = "copilot") {
const scope = { provider, companyId: "company", issueId: "issue", runId: "run", target: "target.txt", ...(provider === "cursor" ? { commandSha256: `sha256:${"a".repeat(64)}` } : {}) };
const row = (seq: number, eventType: string, payload: Row): Row => ({ companyId: "company", runId: "run", seq, eventType, protocolSchemaVersion: 1,
payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", eventType, runId: "run", turnId: "turn",
@@ -81,7 +82,7 @@ function withProjectedArrivalOrder(provider: ActiveStopProvider, order: typeof a
}
f.events.length = 0;
const append = (eventType: string, value: Row) => f.events.push(f.row(f.events.length + 1, eventType, value));
const projector = (createCursorToolEvidence)({
const projector = (provider === "cursor" ? createCursorToolEvidence : createCopilotToolEvidence)({
sessionId: "native-session", turnId: "turn", workingDirectory: "/fixture", active: () => true,
emit: (event: CanonicalProviderEvent) => { append(event.eventType, event.payload); },
});
@@ -124,10 +125,11 @@ function withSessionPrefix(provider: ActiveStopProvider = "cursor") {
}
describe("definitely active native permission Stop", () => {
describe.each(["cursor"] as const)("%s pre-Stop arrival orders", provider => {
describe.each(["cursor", "copilot"] as const)("%s pre-Stop arrival orders", provider => {
it.each(arrivalOrders)("accepts %s only after both exact origins exist and retains them through settlement", order => {
const f = withProjectedArrivalOrder(provider, order), pending = f.pending();
expect(f.nativeStageOrder()).toEqual(["tool", "permission_requested"]);
expect(f.nativeStageOrder()).toEqual(provider === "copilot" && order === "permission-first"
? ["permission_requested", "tool"] : ["tool", "permission_requested"]);
expect(pending.schema).toBe("paperclip.e2e.native-active-stop-pending.v2");
expect(pending.toolOriginRowSha256).toMatch(/^sha256:[a-f0-9]{64}$/u);
expect(pending.toolStartedRowSha256).toMatch(/^sha256:[a-f0-9]{64}$/u);
@@ -187,7 +189,7 @@ describe("definitely active native permission Stop", () => {
});
expect(f.pending).toThrow("duplicate native operation lifecycle");
});
it.each(["cursor"] as const)("accepts the mixed v1/v2 session prefix before strict %s pending proof", provider => {
it.each(["cursor", "copilot"] as const)("accepts the mixed v1/v2 session prefix before strict %s pending proof", provider => {
const f = withSessionPrefix(provider);
expect(f.pending()).toMatchObject({ requestId: "request", toolCallId: "tool", permissionSourceSeq: 6, requestSourceSeq: 7 });
});
@@ -211,7 +213,7 @@ describe("definitely active native permission Stop", () => {
delete payload(f.events.find(row => row.eventType === "runtime_request.created")!).request.details.toolCallId;
expect(f.pending).toThrow("native notice/card identity mismatch");
});
it.each(["cursor"] as const)("binds %s's unanswered callback to cancelled provider settlement and caller-owned Stop", provider => {
it.each(["cursor", "copilot"] as const)("binds %s's unanswered callback to cancelled provider settlement and caller-owned Stop", provider => {
const f = fixture(provider), pending = f.pending(); f.settle();
expect(f.permissionResponse.mock.calls).toEqual([[{ action: "cancel" }]]);
expect(readActiveStopSettlement({ ...f.state(), pending, dispatchMonotonicNs: (BigInt(pending.observedMonotonicNs) + 1n).toString() })).toMatchObject({
@@ -327,6 +329,7 @@ describe("definitely active native permission Stop", () => {
describe("active Stop flow wiring", () => {
it.each([
["cursor", "tool-first"], ["cursor", "permission-first"],
["copilot", "tool-first"], ["copilot", "permission-first"],
] as const)("awaits retention and rechecks real %s %s evidence before issuing the single caller UUID request", async (provider, arrivalOrder) => {
const f = withProjectedArrivalOrder(provider, arrivalOrder); const order: string[] = []; let retainedId = "";
const stop = vi.fn(async (runId: string, id: string) => { expect(runId).toBe("run"); expect(id).toBe(retainedId); order.push("stop"); return f.settle(id); });
@@ -367,11 +370,11 @@ describe("active Stop flow wiring", () => {
});
describe("explicit active Stop discovery", () => {
it("adds exactly two versioned cells without enabling them in --all", () => {
it("adds exactly four versioned cells without enabling them in --all", () => {
const suite = runnerSuites.find(s => s.id === "native-active-stop")!;
const cells = runnerMatrix.filter(e => e.suite === suite);
expect(cells).toHaveLength(2); expect(suite.manualOnly).toBe(true);
expect(cells.map(e => `${e.profile.qualificationCandidate}/${e.environment.id}`).sort()).toEqual(["cursor/daytona", "cursor/local"]);
expect(cells).toHaveLength(4); expect(suite.manualOnly).toBe(true);
expect(cells.map(e => `${e.profile.qualificationCandidate}/${e.environment.id}`).sort()).toEqual(["copilot/daytona", "copilot/local", "cursor/daytona", "cursor/local"]);
expect(cells.every(e => e.task.expectedRunCount === 1 && e.task.flow === "native_active_stop" && e.task.expectedTerminalState?.run === "cancelled")).toBe(true);
expect(suite.definitionMetadata).toMatchObject({ version: 4, evidence: "paperclip.e2e.native-active-stop-settlement.v2", normalCompletionAccepted: false, providerDeath: "not-covered" });
expect(suiteDefinitionHash(suite)).not.toBe(suiteDefinitionHash({ ...suite, definitionMetadata: { ...suite.definitionMetadata, version: 3 } }));
tests/runner-e2e/native-bootstrap-read-proof.test.ts
+5 -3
View File
@@ -81,10 +81,12 @@ it("accepts omitted terminal path/kind only with the retained single-path origin
}
});
it.each(["cursor"] as const)("correlates actual %s passive notices through the public reader and canonical omitted-field updates", async provider => {
it.each(["cursor", "copilot"] as const)("correlates actual %s passive notices through the public reader and canonical omitted-field updates", async provider => {
const { createCursorToolEvidence } = await import("../../packages/paperclip-runner/src/drivers/acpx/cursor-tool-evidence.js");
const { createCopilotToolEvidence } = await import("../../packages/paperclip-runner/src/drivers/acpx/copilot-tool-evidence.js");
const { readCursorToolEvidence } = await import("./cursor-native-evidence.js");
const create = createCursorToolEvidence;
const { readCopilotToolEvidence } = await import("./copilot-evidence.js");
const create = provider === "cursor" ? createCursorToolEvidence : createCopilotToolEvidence;
for (const variant of ["exact", "wrong-path", "multi-path", "ambiguous-update"]) {
const rows: any[] = [];
const projector = create({ sessionId: "session", turnId: "turn", workingDirectory: "/workspace", active: () => true,
@@ -96,7 +98,7 @@ it.each(["cursor"] as const)("correlates actual %s passive notices through the p
projector.tool({ type: "tool_call", tag: "tool_call", toolCallId: "bootstrap", kind: "read", status: "pending", rawInput: variant === "multi-path" ? { paths: [path, "private.txt"] } : { path }, locations: [{ path }] });
projector.tool({ type: "tool_call", tag: "tool_call_update", toolCallId: "bootstrap", status: "failed", ...(variant === "ambiguous-update" ? { rawInput: { path, paths: [path, "private.txt"] } } : {}) });
const evaluate = () => {
const readNotices = readCursorToolEvidence(rows, "run");
const readNotices = provider === "cursor" ? readCursorToolEvidence(rows, "run") : readCopilotToolEvidence(rows, "run");
const started = receipt("started", "running", 2), end = receipt("completed", "failed", 4);
Object.assign(end.payload.prpEvent.payload, { target: null, operation: "unknown" });
return withoutProvenBootstrapReads([...readNotices, origin], origin, { actionFile, events: [...rows, started, end] });
tests/runner-e2e/pi-bootstrap-permission.test.ts
+76
View File
@@ -0,0 +1,76 @@
import { createHash } from "node:crypto";
import { describe, expect, it } from "vitest";
import { canonicalJson } from "../../packages/shared/src/portability-hash.js";
import { approvePiBootstrapRead, withoutApprovedPiBootstrapRequests, type PiBootstrapApproval } from "./pi-bootstrap-permission.js";
import { observePiControlPending, readPiStopSettlement, readPiSteeringSettlement } from "./pi-controls-evidence.js";
import { piCancellationId, piControlCaller, piControlFixture } from "./pi-controls-test-fixture.js";
const actionFile = `.paperclip-eval-action-${"a".repeat(36)}.txt`;
const digest = (value: unknown) => `sha256:${createHash("sha256").update(canonicalJson(value)).digest("hex")}`;
function fixture() {
const f = piControlFixture(), write = structuredClone(f.events);
f.events.length = 0;
const read = { schema: "paperclip.tool.execution.v1", transport: "builtin", executionId: "setup-tool", name: "read", operation: "read", target: actionFile, status: "running" };
const request = { ...f.request, requestId: "setup-request", itemId: "setup-item", details: { toolCallId: "setup-tool" }, choices: [{ key: "accept", label: "Allow once" }, { key: "decline", label: "Deny" }] };
f.append("tool.execution.started", read); f.append("runtime_request.created", { request });
const finish = () => {
f.append("runtime_request.resolved", { requestId: request.requestId, turnId: "turn", action: "accept" });
f.append("tool.execution.completed", { ...read, status: "completed" });
for (const row of write) f.append(row.eventType, row.payload.prpEvent.payload);
};
const proof = (): PiBootstrapApproval => ({ schema: "paperclip.e2e.pi-bootstrap-read-approval.v1", companyId: f.scope.companyId, issueId: f.scope.issueId, runId: f.scope.runId, actionFile, publishedSha256: `sha256:${"b".repeat(64)}`,
requestId: "setup-request", toolCallId: "setup-tool", turnId: "turn", normalizedSessionId: "session", sourceInstanceId: "source",
createdRowSha256: digest(f.events[1]), resolvedRowSha256: digest(f.events[2]), completedRowSha256: digest(f.events[3]) });
return { ...f, read, finish, proof };
}
describe("Pi operator setup read", () => {
it("approves only the published owned file through the exact public request", async () => {
const f = fixture(), saved = new Map<string, unknown>(); let posts = 0;
const result = await approvePiBootstrapRead({
api: { post: async (path: string, data: unknown) => { expect(path).toBe("/api/heartbeat-runs/run/runtime-requests/setup-request/resolve"); expect(data).toEqual({ turnId: "turn", requestKind: "permission_approval", resolution: { action: "accept" } }); posts++; f.finish(); } } as any,
fixture: { actionFile, binding: { companyId: "company", runId: "run" }, snapshot: async () => ({ complete: true, setup: { path: actionFile, published: true, sha256: `sha256:${"b".repeat(64)}` } }) } as any,
companyId: "company", issueId: "issue", runId: "run", deadlineAt: Date.now() + 2000, load: async () => f.state(), evidence: async (name, value) => { saved.set(name, value); },
});
expect(posts).toBe(1); expect(result).toEqual(f.proof()); expect(saved.has("pi-bootstrap-read-before-approval.json")).toBe(true);
const pending = observePiControlPending({ ...f.state(), scope: f.scope, bootstrapApproval: result });
expect(pending.requestId).toBe("request"); expect(pending.bootstrapApproval).toEqual(result);
});
it.each(["unpublished", "foreign-binding", "other-file", "shell"])("does not approve %s setup", async failure => {
const f = fixture(); let posts = 0;
if (failure === "other-file") f.events[0]!.payload.prpEvent.payload.target = "private.txt";
if (failure === "shell") { f.events[0]!.payload.prpEvent.payload.name = "bash"; f.events[0]!.payload.prpEvent.payload.operation = "execute"; }
await expect(approvePiBootstrapRead({ api: { post: async () => { posts++; } } as any,
fixture: { actionFile, binding: { companyId: failure === "foreign-binding" ? "other" : "company", runId: "run" }, snapshot: async () => ({ complete: true, setup: { path: actionFile, published: failure !== "unpublished", sha256: `sha256:${"b".repeat(64)}` } }) } as any,
companyId: "company", issueId: "issue", runId: "run", deadlineAt: Date.now() + 2000, load: async () => f.state(), evidence: async () => {},
})).rejects.toThrow("Pi bootstrap"); expect(posts).toBe(0);
});
it.each(["stop", "steering"])("retains the original %s assertions after a proven setup read", mode => {
const f = fixture(); f.finish(); const proof = f.proof();
expect(() => observePiControlPending({ ...f.state(), scope: f.scope })).toThrow("one native permission");
const pending = observePiControlPending({ ...f.state(), scope: f.scope, bootstrapApproval: proof });
if (mode === "stop") { f.cancel(); expect(readPiStopSettlement({ ...f.state(), pending, caller: piControlCaller, cancellationRequestId: piCancellationId, dispatchMonotonicNs: (BigInt(pending.observedMonotonicNs) + 1n).toString() }).normalCompletionAccepted).toBe(false); }
else { f.steer();
// The wrapper's setup finish is separate from the original write finish.
f.append("runtime_request.resolved", { requestId: "request", turnId: "turn", action: "decline" });
f.append("tool.execution.completed", { ...f.tool, status: "failed", output: "Pi operation was denied or cancelled" });
f.append("turn.completed", { status: "completed", error: null }); f.run.status = "succeeded"; f.issue.status = "done";
f.append("run.result.accepted", { result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary: f.marker } }, { sourceKind: "control_plane", sourceInstanceId: "source:control", sourceSeq: 1, sourceEventId: "source:control:run:1" });
f.append("run.terminal", { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState: "completed", reportedWorkDisposition: "done" }, { sourceKind: "control_plane", sourceInstanceId: "source:control", sourceSeq: 2, sourceEventId: "source:control:run:2" });
expect(readPiSteeringSettlement({ ...f.state(), pending, commentId: f.commentId, queueId: f.queueId, marker: f.marker, finalMessage: f.marker }).marker).toBe(f.marker);
}
});
it.each(["changed-receipt", "wrong-path", "failed-read", "extra-permission", "reordered", "allow-always"])("rejects %s before excluding setup records", failure => {
const f = fixture(); f.finish(); const proof = f.proof();
if (failure === "changed-receipt") proof.resolvedRowSha256 = `sha256:${"0".repeat(64)}`;
if (failure === "wrong-path") f.events[0]!.payload.prpEvent.payload.target = "other.txt";
if (failure === "failed-read") f.events[3]!.payload.prpEvent.payload.status = "failed";
if (failure === "extra-permission") f.append("runtime_request.created", { request: { ...f.request, requestId: "extra" } });
if (failure === "reordered") f.events[3]!.seq = 1;
if (failure === "allow-always") f.events[2]!.payload.prpEvent.payload.action = "accept_always";
expect(() => observePiControlPending({ ...f.state(), scope: f.scope, bootstrapApproval: proof })).toThrow();
});
it("does not discard any native lifecycle row", () => { const f = fixture(); f.finish(); const rows = withoutApprovedPiBootstrapRequests(f.events, f.proof()); expect(rows).toHaveLength(f.events.length - 2); expect(rows.filter((row: any) => row.eventType.startsWith("tool.execution."))).toHaveLength(3); });
});
tests/runner-e2e/pi-bootstrap-permission.ts
+134
View File
@@ -0,0 +1,134 @@
import { createHash } from "node:crypto";
import { canonicalJson } from "../../packages/shared/src/portability-hash.js";
import { pollUntil, type RunnerApi } from "./api.js";
import { bootstrapReadExecutionId } from "./native-bootstrap-read-proof.js";
import { isValidNativePrpEnvelope } from "./native-event-envelope.js";
import { piPermissionRequests } from "./pi-native-evidence.js";
import type { RemoteNativeFixture, RemoteNativeSnapshot } from "./remote-native-fixtures.js";
type Row = Record<string, any>;
const hash = (row: unknown) => `sha256:${createHash("sha256").update(canonicalJson(row)).digest("hex")}`;
const requireProof = (ok: unknown) => { if (!ok) throw new Error("Pi bootstrap: exact published read approval is unproven"); };
export interface PiBootstrapApproval {
schema: "paperclip.e2e.pi-bootstrap-read-approval.v1";
companyId: string; issueId: string; runId: string; actionFile: string; publishedSha256: string;
requestId: string; toolCallId: string; turnId: string; normalizedSessionId: string; sourceInstanceId: string;
createdRowSha256: string; resolvedRowSha256: string; completedRowSha256: string;
}
export interface PiBootstrapState { run: Row; issue: Row; events: readonly unknown[] }
function readLifecycle(events: readonly unknown[], input: Pick<PiBootstrapApproval, "companyId" | "runId" | "actionFile" | "requestId">) {
requireProof(events.length > 0 && events.length <= 20_000 && /^\.paperclip-eval-action-[a-f0-9]{36}\.txt$/u.test(input.actionFile));
const seqs = new Set<number>(), sourceIds = new Set<string>(), lastSource = new Map<string, number>();
for (const row of events as readonly Row[]) {
const e = row.payload?.prpEvent; if (e === undefined) continue;
requireProof(e && isValidNativePrpEnvelope(e, row.protocolSchemaVersion) && row.companyId === input.companyId
&& row.runId === input.runId && e.runId === input.runId && e.eventType === row.eventType
&& ["runner", "control_plane"].includes(e.sourceKind) && typeof e.sourceInstanceId === "string" && e.sourceInstanceId.length > 0
&& Number.isSafeInteger(row.seq) && row.seq > 0 && !seqs.has(row.seq)
&& Number.isSafeInteger(e.sourceSeq) && e.sourceSeq > (lastSource.get(e.sourceInstanceId) ?? 0)
&& e.sourceEventId === `${e.sourceInstanceId}:${input.runId}:${e.sourceSeq}` && !sourceIds.has(e.sourceEventId)
&& Number.isFinite(Date.parse(e.emittedAt)));
seqs.add(row.seq); sourceIds.add(e.sourceEventId); lastSource.set(e.sourceInstanceId, e.sourceSeq);
}
const created = piPermissionRequests(events, input.runId).filter(x => x.request.requestId === input.requestId);
requireProof(created.length === 1); const card = created[0]!;
requireProof(isValidNativePrpEnvelope(card.event, card.row.protocolSchemaVersion) && card.row.companyId === input.companyId && card.request.requestKind === "permission_approval"
&& typeof card.event.turnId === "string" && card.event.turnId.length > 0
&& typeof card.event.normalizedSessionId === "string" && card.event.normalizedSessionId.length > 0
&& card.request.choices.filter((choice: Row) => choice.key === "accept").length === 1);
const executionId = bootstrapReadExecutionId(card.request.details.toolCallId);
const rows = (events as readonly Row[]).filter(row => row.payload?.prpEvent?.payload?.executionId === executionId);
requireProof(rows.length > 0 && rows.length <= 2048);
let known = false, lastSeq = 0;
for (const row of rows) {
const event = row.payload.prpEvent, p = event.payload;
requireProof(isValidNativePrpEnvelope(event, row.protocolSchemaVersion) && row.companyId === input.companyId && row.runId === input.runId && event.runId === input.runId
&& row.protocolSchemaVersion === 1 && event.schema === "paperclip.prp.event.v1" && event.schemaVersion === 1
&& event.sourceKind === "runner" && event.turnId === card.event.turnId && event.normalizedSessionId === card.event.normalizedSessionId
&& event.sourceInstanceId === card.event.sourceInstanceId && event.eventType === row.eventType
&& Number.isSafeInteger(row.seq) && row.seq > lastSeq
&& p.schema === "paperclip.tool.execution.v1" && p.transport === "builtin" && p.name === "read" && p.operation === "read"
&& ["tool.execution.started", "tool.execution.progressed", "tool.execution.completed"].includes(row.eventType)
&& (p.target === input.actionFile || (!known && p.target === null && row.eventType !== "tool.execution.completed")));
known ||= p.target === input.actionFile; lastSeq = row.seq;
}
requireProof(known && rows[0]!.eventType === "tool.execution.started"
&& rows.filter(row => row.eventType === "tool.execution.started").length === 1
&& rows.filter(row => row.eventType === "tool.execution.completed").length <= 1
&& rows.every((row, index) => row.eventType === "tool.execution.completed"
? index === rows.length - 1 && row.payload.prpEvent.payload.status === "completed"
: row.payload.prpEvent.payload.status === "running"));
return { card, rows, executionId };
}
/** Remove only the two hash-bound setup permission records. All native read
* lifecycle rows and every tested write/permission remain in the oracle. */
export function withoutApprovedPiBootstrapRequests(events: readonly unknown[], proof?: PiBootstrapApproval): unknown[] {
if (!proof) return [...events];
requireProof(proof.schema === "paperclip.e2e.pi-bootstrap-read-approval.v1" && /^sha256:[a-f0-9]{64}$/u.test(proof.publishedSha256));
const { card, rows } = readLifecycle(events, proof);
requireProof(card.request.details.toolCallId === proof.toolCallId && card.event.turnId === proof.turnId
&& card.event.normalizedSessionId === proof.normalizedSessionId && card.event.sourceInstanceId === proof.sourceInstanceId
&& hash(card.row) === proof.createdRowSha256);
const closures = (events as readonly Row[]).filter(row => ["runtime_request.resolved", "runtime_request.cancelled", "runtime_request.expired"].includes(row.eventType)
&& row.payload?.prpEvent?.payload?.requestId === proof.requestId);
requireProof(closures.length === 1); const closed = closures[0]!, e = closed.payload.prpEvent;
requireProof(isValidNativePrpEnvelope(e, closed.protocolSchemaVersion) && closed.companyId === proof.companyId && closed.runId === proof.runId && closed.protocolSchemaVersion === 1
&& e.schema === "paperclip.prp.event.v1" && e.schemaVersion === 1 && e.sourceKind === "runner" && e.runId === proof.runId
&& e.eventType === "runtime_request.resolved" && closed.eventType === e.eventType && e.sourceInstanceId === proof.sourceInstanceId
&& e.turnId === proof.turnId && e.normalizedSessionId === proof.normalizedSessionId && e.payload.turnId === proof.turnId
&& e.payload.action === "accept" && closed.seq > card.row.seq && hash(closed) === proof.resolvedRowSha256);
const completed = rows.at(-1)!;
requireProof(completed.eventType === "tool.execution.completed" && completed.seq > closed.seq && hash(completed) === proof.completedRowSha256);
const tested = piPermissionRequests(events, proof.runId).filter(x => x.request.requestId !== proof.requestId);
requireProof(tested.every(x => x.row.seq > completed.seq && x.event.turnId === proof.turnId
&& x.event.normalizedSessionId === proof.normalizedSessionId && x.event.sourceInstanceId === proof.sourceInstanceId));
return events.filter(row => row !== card.row && row !== closed);
}
/** The operator may approve one setup read after publication and observer
* arming. Production permission policy and the tested write are untouched. */
export async function approvePiBootstrapRead(input: {
api: RunnerApi; fixture: { actionFile: string; binding: Pick<RemoteNativeFixture["binding"], "companyId" | "runId">; snapshot(label: string): Promise<Pick<RemoteNativeSnapshot, "complete" | "setup">> }; companyId: string; issueId: string; runId: string;
deadlineAt: number; load(): Promise<PiBootstrapState>; evidence(name: string, value: unknown): Promise<void>;
}): Promise<PiBootstrapApproval | undefined> {
const actionFile = input.fixture.actionFile;
requireProof(/^\.paperclip-eval-action-[a-f0-9]{36}\.txt$/u.test(actionFile)
&& input.fixture.binding.companyId === input.companyId && input.fixture.binding.runId === input.runId);
const pending = await pollUntil({ label: "Pi published bootstrap read", deadlineAt: input.deadlineAt, intervalMs: 100,
load: input.load, reject: state => ["failed", "timed_out", "cancelled", "succeeded"].includes(state.run.status) ? "Pi bootstrap run settled" : undefined,
accept: state => piPermissionRequests(state.events, input.runId).length > 0 });
requireProof(pending.run.status === "running" && pending.run.runtimeMode === "native" && pending.issue.status === "in_progress" && pending.run.id === input.runId && pending.run.companyId === input.companyId && pending.run.nativeIssueId === input.issueId
&& pending.issue.id === input.issueId && pending.issue.companyId === input.companyId);
const requests = piPermissionRequests(pending.events, input.runId); requireProof(requests.length === 1);
const card = requests[0]!, executionId = bootstrapReadExecutionId(card.request.details.toolCallId);
const native = (pending.events as readonly Row[]).filter(row => row.payload?.prpEvent?.payload?.executionId === executionId);
// A read permitted without delegation can already have led to the tested
// write. There is no setup permission to remove in that existing path.
if (native.some(row => row.payload.prpEvent.payload.operation === "edit")) return undefined;
const checked = readLifecycle(pending.events, { companyId: input.companyId, runId: input.runId, actionFile, requestId: card.request.requestId });
requireProof(!checked.rows.some(row => row.eventType === "tool.execution.completed"));
const snapshot = await input.fixture.snapshot("bootstrap-read-published");
requireProof(snapshot.complete && snapshot.setup.path === actionFile && snapshot.setup.published
&& /^sha256:[a-f0-9]{64}$/u.test(snapshot.setup.sha256 ?? ""));
await input.evidence("pi-bootstrap-read-before-approval.json", { binding: input.fixture.binding, snapshot, request: card.row, native: checked.rows });
await input.api.post(`/api/heartbeat-runs/${input.runId}/runtime-requests/${encodeURIComponent(card.request.requestId)}/resolve`,
{ turnId: card.event.turnId, requestKind: "permission_approval", resolution: { action: "accept" } });
const finished = await pollUntil({ label: "Pi exact bootstrap read completion", deadlineAt: input.deadlineAt, intervalMs: 100, load: input.load,
reject: state => ["failed", "timed_out", "cancelled", "succeeded"].includes(state.run.status) ? "Pi bootstrap run settled" : undefined,
accept: state => (state.events as readonly Row[]).some(row => row.eventType === "tool.execution.completed"
&& row.payload?.prpEvent?.payload?.executionId === executionId) });
const closure = (finished.events as readonly Row[]).find(row => row.eventType === "runtime_request.resolved" && row.payload?.prpEvent?.payload?.requestId === card.request.requestId);
const completed = (finished.events as readonly Row[]).find(row => row.eventType === "tool.execution.completed" && row.payload?.prpEvent?.payload?.executionId === executionId);
requireProof(closure && completed);
const after = await input.fixture.snapshot("bootstrap-read-completed");
requireProof(after.complete && after.setup.path === actionFile && after.setup.published && after.setup.sha256 === snapshot.setup.sha256);
await input.evidence("pi-bootstrap-read-completed.json", { binding: input.fixture.binding, snapshot: after });
const proof: PiBootstrapApproval = { schema: "paperclip.e2e.pi-bootstrap-read-approval.v1", companyId: input.companyId, issueId: input.issueId, runId: input.runId,
actionFile, publishedSha256: snapshot.setup.sha256!, requestId: card.request.requestId, toolCallId: card.request.details.toolCallId,
turnId: card.event.turnId, normalizedSessionId: card.event.normalizedSessionId, sourceInstanceId: card.event.sourceInstanceId,
createdRowSha256: hash(card.row), resolvedRowSha256: hash(closure), completedRowSha256: hash(completed) };
withoutApprovedPiBootstrapRequests(finished.events, proof);
await input.evidence("pi-bootstrap-read-approval.json", proof); return proof;
}
tests/runner-e2e/pi-controls-cases.ts
+17
View File
@@ -0,0 +1,17 @@
import type { RunnerTaskFixture } from "./types.js";
export const piControlTasks: readonly RunnerTaskFixture[] = ["pending-permission-stop", "same-turn-steering"].map(id => ({
id, label: id === "pending-permission-stop" ? "Stop Pi while permission is unanswered" : "Steer Pi through a pending native write",
groups: [], workMode: "standard", flow: "pi_controls", expectedRunCount: 1,
attemptTimeoutMs: { local: 300_000, daytona: 300_000 }, turnTimeoutMs: 120_000,
expectedTerminalState: id === "pending-permission-stop" ? { issue: "in_progress", run: "cancelled" } : { issue: "done", run: "succeeded" },
buildTitle: nonce => `Pi ${id} ${nonce}`,
buildVisibleMarker: nonce => `PI-CONTROL-${nonce}`,
buildPrompt: () => [
"Attempt the supplied native write exactly once and wait for its actual permission decision. Do not retry, answer your own permission, use bash, or use another path/tool to work around it.",
id === "pending-permission-stop"
? "The operator will stop this unfinished task while permission remains unanswered. Do not finish or claim completion."
: "The operator will send a same-turn instruction while this permission is pending, then deny the write. After the actual denial, follow that new instruction. Denial is the expected result. Never claim that the file was written. Do not finish before receiving that instruction.",
].join("\n"),
buildMatchers: () => [], // Dedicated flow joins public control receipts and independent retirement evidence.
}));
tests/runner-e2e/pi-controls-evidence.test.ts
+210
View File
@@ -0,0 +1,210 @@
import { describe, expect, it, vi } from "vitest";
import { createHash } from "node:crypto";
import { CodexHarnessSession } from "../../packages/paperclip-runner/src/drivers/codex/codex-harness-session.js";
import { assertSamePiPending, observePiControlPending, readPiStopSettlement, readPiSteeringAcknowledgement, readPiSteeringSettlement } from "./pi-controls-evidence.js";
import { stopPiAtPendingPermission } from "./pi-controls-flow.js";
import { piControlFixture, piControlCaller, piCancellationId } from "./pi-controls-test-fixture.js";
import { runnerMatrix, runnerSuites, suiteDefinitionHash, validateRunnerCatalog } from "./catalog.js";
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
import { buildRunnerE2EProcessEnvironment } from "./harness-env.js";
import { assertRemoteNativeEvidencePrerequisites } from "./prerequisites.js";
type Row = Record<string, any>;
const event = (row: Row) => row.payload.prpEvent;
function cancelled() {
const f = piControlFixture(), pending = f.pending(); f.cancel();
const binding = { pending, caller: piControlCaller, cancellationRequestId: piCancellationId, dispatchMonotonicNs: String(BigInt(pending.observedMonotonicNs) + 1n) };
return { f, binding, read: () => readPiStopSettlement({ ...f.state(), ...binding }) };
}
function streamedWrite() {
const f = piControlFixture();
f.events.pop();
event(f.events[0]!).payload = { ...f.tool, target: null, inputUpdated: false };
f.append("tool.execution.progressed", { ...f.tool, target: null, inputUpdated: true });
f.append("tool.execution.progressed", { ...f.tool, inputUpdated: true });
f.append("runtime_request.created", { request: f.request });
return f;
}
describe("Pi pending control identity", () => {
it("accepts Pi native permission/start without invented provider notices", () => expect(piControlFixture().pending()).toMatchObject({ toolCallId: "pi-tool", executionId: "pi-tool", turnId: "turn" }));
it("accepts streamed arguments only after the same execution proves its exact path", () => {
const f = streamedWrite();
expect(f.pending()).toMatchObject({ startedSourceSeq: 1, requestSourceSeq: 4 });
});
it.each([
["no complete target", (f: ReturnType<typeof streamedWrite>) => { event(f.events[2]!).payload.target = null; }],
["conflicting partial target", f => { event(f.events[1]!).payload.target = "other.txt"; }],
["target lost after admission", f => { f.append("tool.execution.progressed", { ...f.tool, target: null }); }],
["different execution supplied target", f => { event(f.events[2]!).payload.executionId = "other"; }],
["terminal before target", f => { f.events[1]!.eventType = event(f.events[1]!).eventType = "tool.execution.completed"; event(f.events[1]!).payload.status = "failed"; }],
] satisfies Array<[string, (f: ReturnType<typeof streamedWrite>) => void]>)("rejects streamed write with %s", (_name, mutate) => {
const f = streamedWrite(); mutate(f); expect(() => f.pending()).toThrow();
});
it("retains the partial start hash through actual callback cancellation", () => {
const f = streamedWrite(), pending = f.pending(); f.cancel();
expect(readPiStopSettlement({ ...f.state(), pending, caller: piControlCaller, cancellationRequestId: piCancellationId,
dispatchMonotonicNs: String(BigInt(pending.observedMonotonicNs) + 1n) })).toMatchObject({ normalCompletionAccepted: false });
});
it("admits permission-first ACP only after both canonical boundaries exist", () => {
const f = piControlFixture(); f.events.reverse();
f.events.forEach((r, i) => { r.seq = event(r).sourceSeq = i + 1; event(r).sourceEventId = `source:run:${i + 1}`; });
expect(f.pending().requestSourceSeq).toBe(1);
});
it.each([
["wrong provider", (f: ReturnType<typeof piControlFixture>) => { f.request.origin.provider = "copilot"; }],
["foreign source", f => { event(f.events[1]!).sourceInstanceId = "other"; }],
["foreign company", f => { f.events[0]!.companyId = "other"; }],
["changed write path", f => { f.tool.target = "other.txt"; }],
["write already ended", f => { f.append("tool.execution.completed", { ...f.tool, status: "failed" }); }],
["answered request", f => { f.append("runtime_request.resolved", { requestId: "request", action: "decline" }); }],
["normal completion", f => { f.append("turn.completed", { status: "completed" }); }],
["extra native write", f => { f.append("tool.execution.started", { ...f.tool, executionId: "other" }); }],
["missing decline", f => { f.request.choices = []; }],
["missing tool origin", f => { f.events.shift(); }],
["duplicate event", f => { f.events.push(f.events[0]!); }],
["non-native run", f => { f.run.runtimeMode = "legacy"; }],
["prior Stop", f => { f.run.resultJson.startupCancellation = {}; }],
] satisfies Array<[string, (f: ReturnType<typeof piControlFixture>) => void]>)("rejects %s", (_name, mutate) => { const f = piControlFixture(); mutate(f); expect(() => f.pending()).toThrow(); });
});
describe("Pi active Stop settlement", () => {
it("uses the production terminal mapper and actually cancels the pending callback", () => {
const s = cancelled(); expect(s.f.responses).toHaveLength(1); expect(s.read()).toMatchObject({ normalCompletionAccepted: false, schema: "paperclip.e2e.pi-stop-settlement.v1" });
});
it.each([
["normal terminal", (s: ReturnType<typeof cancelled>) => { event(s.f.events.at(-1)!).eventType = s.f.events.at(-1)!.eventType = "turn.completed"; }],
["expired callback", s => { const r = s.f.events.find(r => r.eventType === "runtime_request.cancelled")!; r.eventType = event(r).eventType = "runtime_request.expired"; }],
["answered callback", s => { event(s.f.events.find(r => r.eventType === "runtime_request.cancelled")!).payload.action = "decline"; }],
["changed pending target", s => { s.f.tool.target = "other.txt"; }],
["changed retained hash", s => { s.binding.pending.requestRowSha256 = `sha256:${"a".repeat(64)}`; }],
["late pending observation", s => { s.binding.dispatchMonotonicNs = s.binding.pending.observedMonotonicNs; }],
["unacknowledged Stop", s => { s.f.run.resultJson.nativeCancellation.dispatchState = "pending"; }],
["another caller", s => { s.f.run.resultJson.startupCancellation.requestedBy.userId = "other"; }],
["another intent", s => { s.f.run.resultJson.nativeCancellation.intentId = "native-cancellation:other"; }],
["extra cancellation effect", s => { s.f.run.resultJson.nativeCancellation.effects.push("pause_agent"); }],
["same intent and ack audit", s => { s.f.run.resultJson.nativeCancellation.acknowledgementAuditId = "intent-audit"; }],
["completed task", s => { s.f.issue.status = "done"; }],
["foreign run", s => { s.f.run.id = "other"; }],
] satisfies Array<[string, (s: ReturnType<typeof cancelled>) => void]>)("rejects %s", (_name, mutate) => { const s = cancelled(); mutate(s); expect(s.read).toThrow(); });
it("retains and rereads the boundary before dispatching one caller UUID", async () => {
const f = piControlFixture(), order: string[] = [];
const result = await stopPiAtPendingPermission({ scope: f.scope, caller: piControlCaller, deadlineAt: Date.now() + 1000,
load: async () => { order.push("load"); return f.state(); }, retain: async () => { order.push("retain"); }, stop: async (runId, uuid) => { order.push("stop"); expect(runId).toBe("run"); return f.cancel(uuid); } });
expect(order.slice(0, 4)).toEqual(["load", "retain", "load", "stop"]); expect(result.settlement.normalCompletionAccepted).toBe(false);
});
it("refuses dispatch when pending state changes during evidence retention", async () => {
const f = piControlFixture(), stop = vi.fn();
await expect(stopPiAtPendingPermission({ scope: f.scope, caller: piControlCaller, deadlineAt: Date.now() + 1000, load: async () => f.state(), retain: async () => { f.request.itemId = "changed"; }, stop })).rejects.toThrow("pending boundary changed");
expect(stop).not.toHaveBeenCalled();
});
it("fails immediately on normal completion instead of retrying it as an unsettled cancellation", async () => {
const f = piControlFixture();
await expect(stopPiAtPendingPermission({ scope: f.scope, caller: piControlCaller, deadlineAt: Date.now() + 100,
load: async () => f.state(), retain: async () => {}, stop: async (_runId, uuid) => { const run = f.cancel(uuid); f.run.status = "succeeded"; return run; },
})).rejects.toThrow(/^Stopped waiting for Pi pending permission cancellation: unexpected terminal$/);
});
});
function steered() {
const f = piControlFixture(), pending = f.pending(); f.steer();
const binding = { pending, commentId: f.commentId, queueId: f.queueId, marker: f.marker, finalMessage: f.marker };
const ack = readPiSteeringAcknowledgement({ ...f.state(), ...binding });
assertSamePiPending(pending, observePiControlPending({ ...f.state(), scope: f.scope })); f.finish();
return { f, binding, ack, read: () => readPiSteeringSettlement({ ...f.state(), ...binding }) };
}
describe("Pi same-turn steering", () => {
function withControlSettlement() {
return steered();
}
it("keeps exact runner proof when the control plane records result and terminal events", () => {
expect(withControlSettlement().read()).toMatchObject({ nativeFollowUpTested: false });
});
it.each([
["foreign control producer", (s: ReturnType<typeof withControlSettlement>) => { event(s.f.events.at(-1)!).sourceInstanceId = "foreign:control"; event(s.f.events.at(-1)!).sourceEventId = "foreign:control:run:2"; }],
["foreign control turn", s => { event(s.f.events.at(-1)!).turnId = "foreign"; }],
["foreign control session", s => { event(s.f.events.at(-1)!).normalizedSessionId = "foreign"; }],
["unknown control event", s => { s.f.events.at(-1)!.eventType = event(s.f.events.at(-1)!).eventType = "runtime_request.resolved"; }],
["reordered control sequence", s => { event(s.f.events.at(-1)!).sourceSeq = 1; event(s.f.events.at(-1)!).sourceEventId = "source:control:run:1"; }],
["duplicate control terminal", s => { s.f.append("run.terminal", { schema: "paperclip.prp.terminal.v1" }, { sourceKind: "control_plane", sourceInstanceId: "source:control", sourceSeq: 3, sourceEventId: "source:control:run:3" }); }],
["unbound control result", s => { event(s.f.events.at(-2)!).payload.result.schema = "foreign"; }],
["both control records missing", s => { s.f.events.splice(-2); }],
["accepted result missing", s => { s.f.events.splice(-2, 1); }],
["control terminal missing", s => { s.f.events.pop(); }],
["wrong control summary", s => { event(s.f.events.at(-2)!).payload.result.summary = "Finished"; }],
["unfinished control result", s => { event(s.f.events.at(-2)!).payload.result.reportedWorkDisposition = "in_progress"; }],
["failed control terminal", s => { event(s.f.events.at(-1)!).payload.runTerminalState = "failed"; }],
["interrupted control turn", s => { event(s.f.events.at(-1)!).payload.turnTerminalState = "interrupted"; }],
["unfinished control terminal", s => { event(s.f.events.at(-1)!).payload.reportedWorkDisposition = "in_progress"; }],
["control terminal before runner completion", s => {
for (const row of s.f.events.slice(3, 6)) row.seq += 2;
s.f.events.at(-2)!.seq = 4; s.f.events.at(-1)!.seq = 5;
}],
] satisfies Array<[string, (s: ReturnType<typeof withControlSettlement>) => void]>)("rejects %s", (_name, mutate) => {
const s = withControlSettlement(); mutate(s); expect(s.read).toThrow();
});
it("calibrates against the actual Product ACP facade producer, not Rust's raw transport echo", async () => {
const f = piControlFixture(), pending = f.pending(), calls: Row[] = [];
// Only the transport and event sink are doubles. Run the actual public
// steer method, active-turn check and negotiated Pi capability boundary.
const session = Object.assign(Object.create(CodexHarnessSession.prototype), {
driverKind: "acpx_runtime", activeTurnId: "turn", opened: { threadId: "thread" },
protocolIntegrityFailure: null, terminalTurns: new Map(), acknowledgedSteeringCorrelations: new Map(),
transport: { turnControlCapabilities: () => ({ steering: true, queuedFollowUp: true }), request: async (method: string, params: Row) => { calls.push({ method, params }); return {}; } },
emit: (type: string, payload: Row, extra: Row) => f.append(type, payload, extra),
}) as CodexHarnessSession;
await session.steer({ turnId: "turn", correlationId: "comment", message: { role: "user", text: "Hidden instruction" } });
expect(calls).toEqual([{ method: "turn/steer", params: { threadId: "thread", input: [{ type: "text", text: "Hidden instruction", text_elements: [] }], expectedTurnId: "turn", correlationId: "comment" } }]);
f.run.resultJson.queuedSteeringAcknowledgements = { comment: { status: "acknowledged", queueId: "queue", turnId: "turn", acknowledgedAt: "2026-10-01T00:00:01Z" } };
const read = () => readPiSteeringAcknowledgement({ ...f.state(), pending, commentId: "comment", queueId: "queue" });
expect(read().turnId).toBe("turn");
event(f.events.at(-1)!).itemId = `acpx-control-${createHash("sha256").update("turn:comment").digest("hex")}`;
expect(read).toThrow("acknowledgement missing");
});
it("requires acknowledged steering before denial and the hidden marker in final output", () => expect(steered().read()).toMatchObject({ nativeFollowUpTested: false }));
it.each([
["marker echoed elsewhere", (s: ReturnType<typeof steered>) => { s.binding.finalMessage = "Finished"; }],
["narration mixed into final", s => { s.binding.finalMessage = `Working...${s.f.marker}`; }],
["another queued comment", s => { s.binding.commentId = "other"; }],
["another queue", s => { s.binding.queueId = "other"; }],
["another turn acknowledgment", s => { s.f.run.resultJson.queuedSteeringAcknowledgements.comment.turnId = "other"; }],
["missing native ack", s => { s.f.events.splice(2, 1); }],
["follow-up substituted for steer", s => { event(s.f.events[2]!).payload.mode = "follow_up"; }],
["accepted write", s => { event(s.f.events.find(r => r.eventType === "runtime_request.resolved")!).payload.action = "accept"; }],
["completed write", s => { event(s.f.events.find(r => r.eventType === "tool.execution.completed")!).payload.status = "completed"; }],
["unrelated tool failure", s => { event(s.f.events.find(r => r.eventType === "tool.execution.completed")!).payload.output = "Network error"; }],
["cancelled run", s => { s.f.run.status = "cancelled"; }],
["unfinished task", s => { s.f.issue.status = "in_progress"; }],
] satisfies Array<[string, (s: ReturnType<typeof steered>) => void]>)("rejects %s", (_name, mutate) => { const s = steered(); mutate(s); expect(s.read).toThrow(); });
});
describe("Pi controls catalog admission", () => {
it("adds exactly four explicit Pi-only cells without widening --all", () => {
const cells = selectRunnerExecutions(parseRunnerSelectors(["--suite", "pi-controls"]));
expect(cells).toHaveLength(4); expect(cells.every(c => c.profile.qualificationCandidate === "pi" && c.task.expectedRunCount === 1)).toBe(true);
expect(cells.map(c => `${c.environment.id}/${c.task.id}`).sort()).toEqual(["daytona/pending-permission-stop", "daytona/same-turn-steering", "local/pending-permission-stop", "local/same-turn-steering"]);
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(c => c.suite.id === "pi-controls")).toBe(false);
expect(validateRunnerCatalog()).toHaveLength(722);
for (const cell of cells) expect(buildRunnerE2EProcessEnvironment({}, [cell]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION).toBeUndefined();
expect(() => assertRemoteNativeEvidencePrerequisites(cells, {})).toThrow();
});
it("pins the Pi 1 profile and versioned coverage while retaining active Stop identity", () => {
// Current profile identity and native/control coverage retain their own fingerprints.
// Extended v3 states the strict file oracle's task-wide Bash limit.
const pi = runnerMatrix.find(c => c.profile.qualificationCandidate === "pi")!.profile;
expect(pi.modelQualification?.qualificationId).toBe("pi:0.0.33:1.0.0:openrouter");
expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 23, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", profileVersion: 19, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "single-json-write-and-content-bound-native-read-both-runs", agentMemoryReadAuthority: "local-withheld-or-exact-remote-agent-run-file", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7" });
expect(runnerSuites.find(s => s.id === "pi-controls")!.definitionMetadata).toMatchObject({ version: 10, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", profileVersion: 19,
remoteProcessIdentity: "observer-pid-startTicks-bootId",
controlPlaneSettlement: "required-scoped-result-and-terminal-after-runner" });
expect(runnerSuites.find(s => s.id === "extended-harnesses")!.definitionMetadata).toMatchObject({ version: 5, piFileArtifactTitle: "exact-filename", piFileCommandTransport: "fenced-bash-markdown-paste" });
for (const cell of runnerMatrix.filter(cell => cell.profile.qualificationCandidate === "pi")) {
const agent = cell.profile.buildAgent({ environmentId: "environment", environmentFixtureId: cell.environment.id, workspacePath: "/workspace", executionId: cell.id, secretRefs: { OPENROUTER_API_KEY: { type: "secret_ref", secretId: "synthetic", version: "latest" } } });
expect(agent.adapterConfig).toMatchObject({ piThinkingLevel: "low" });
}
const hashes = Object.fromEntries(runnerSuites.filter(s => ["pi-native", "native-active-stop", "extended-harnesses", "rich-acp-warm-continuity"].includes(s.id)).map(s => [s.id, suiteDefinitionHash(s)]));
expect(hashes).toEqual({
"pi-native": "90f2e901d7f39bfb6bbadcf63bc78e410a3b071af980f3a23979747e3d5f2021",
"native-active-stop": "2d4fdeeb75bd531b309bd1b35cfa5680ceefdeee6b7155b068dd011ce9901980",
"rich-acp-warm-continuity": "331b88d9538a132670789fb7864df7df5f4bf294ae19b62f53d8a02f901774f0",
"extended-harnesses": "5de4fac78d4d581bc0954f07b5cf2df2862d37f8b0205325eede6d9ef6d9f5e6",
});
expect(runnerMatrix.filter(c => c.profile.qualificationCandidate === "pi" && c.suite.id !== "pi-controls")).toHaveLength(22);
});
});
tests/runner-e2e/pi-controls-evidence.ts
+160
View File
@@ -0,0 +1,160 @@
import { createHash } from "node:crypto";
import { canonicalJson } from "../../packages/shared/src/portability-hash.js";
import { hasAcpxNativeOrigin } from "./acpx-native-origin.js";
import { bootstrapReadExecutionId } from "./native-bootstrap-read-proof.js";
import { withoutApprovedPiBootstrapRequests, type PiBootstrapApproval } from "./pi-bootstrap-permission.js";
import { isValidNativePrpEnvelope } from "./native-event-envelope.js";
import type { ActiveStopCaller } from "./native-active-stop-evidence.js";
type Row = Record<string, any>;
export interface PiControlScope { companyId: string; issueId: string; runId: string; target: string }
export interface PiControlState { run: Row; issue: Row; events: readonly unknown[]; bootstrapApproval?: PiBootstrapApproval }
export interface PiControlPending {
schema: "paperclip.e2e.pi-control-pending.v1"; bootstrapApproval?: PiBootstrapApproval; scope: PiControlScope; requestId: string; toolCallId: string; executionId: string;
turnId: string; normalizedSessionId: string; sourceInstanceId: string; itemId: string;
requestSourceSeq: number; startedSourceSeq: number; requestRowSha256: string; startedRowSha256: string; observedMonotonicNs: string;
}
const rec = (v: unknown): Row => v !== null && typeof v === "object" && !Array.isArray(v) ? v as Row : {};
const id = (v: unknown): v is string => typeof v === "string" && v.length > 0 && v.length <= 240 && !/[\u0000-\u001f\u007f]/u.test(v) && !v.includes("[REDACTED]");
const hash = (v: unknown) => `sha256:${createHash("sha256").update(canonicalJson(v)).digest("hex")}`;
function requireProof(value: unknown, reason: string): asserts value { if (!value) throw new Error(`Pi controls: ${reason}`); }
const terminalTypes = new Set(["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"]);
const closureTypes = new Set(["runtime_request.resolved", "runtime_request.cancelled", "runtime_request.expired"]);
const controlSettlementTypes = new Set(["run.result.accepted", "run.terminal"]);
/** Consume the actual Product projection. Pi does not emit Cursor/Copilot native
* diagnostic notices; never manufacture those to reuse another provider's oracle. */
function origin(events: readonly unknown[], scope: PiControlScope, bootstrapApproval?: PiBootstrapApproval) {
if (bootstrapApproval) requireProof(bootstrapApproval.companyId === scope.companyId && bootstrapApproval.issueId === scope.issueId && bootstrapApproval.runId === scope.runId, "foreign bootstrap approval");
const gradingRows = new Set(withoutApprovedPiBootstrapRequests(events, bootstrapApproval));
requireProof(Object.values(scope).every(id) && events.length > 0 && events.length <= 20_000, "bounded exact scope required");
const projected = events.map(rec).filter(row => rec(row.payload).prpEvent !== undefined).map(row => ({ row, event: rec(rec(row.payload).prpEvent) })).sort((a, b) => a.row.seq - b.row.seq);
const seqs = new Set<number>(), sourceIds = new Set<string>(), last = new Map<string, number>();
for (const { row, event: e } of projected) {
requireProof(row.companyId === scope.companyId && row.runId === scope.runId && isValidNativePrpEnvelope(e, row.protocolSchemaVersion)
&& (e.sourceKind === "runner" || (e.sourceKind === "control_plane" && controlSettlementTypes.has(e.eventType)))
&& e.runId === scope.runId && e.eventType === row.eventType && Number.isSafeInteger(row.seq) && row.seq > 0 && !seqs.has(row.seq)
&& id(e.sourceInstanceId) && Number.isSafeInteger(e.sourceSeq) && e.sourceSeq > (last.get(e.sourceInstanceId) ?? 0)
&& e.sourceEventId === `${e.sourceInstanceId}:${e.runId}:${e.sourceSeq}` && !sourceIds.has(e.sourceEventId), "foreign, duplicate or reordered event");
seqs.add(row.seq); sourceIds.add(e.sourceEventId); last.set(e.sourceInstanceId, e.sourceSeq);
}
const rows = projected.filter(x => x.event.sourceKind === "runner" && gradingRows.has(x.row));
const control = projected.filter(x => x.event.sourceKind === "control_plane");
const created = rows.filter(x => x.event.eventType === "runtime_request.created");
requireProof(created.length === 1, "one native permission required");
const card = created[0]!, request = rec(rec(card.event.payload).request);
requireProof(request.schema === "paperclip.runtime_request.v2" && request.type === "permission" && request.requestKind === "permission_approval" && request.status === "pending"
&& id(request.requestId) && id(request.itemId) && id(request.details?.toolCallId) && request.turnId === card.event.turnId && id(card.event.turnId)
&& id(card.event.normalizedSessionId) && hasAcpxNativeOrigin(request.origin, "pi", "session/request_permission")
&& Array.isArray(request.choices) && request.choices.some((c: Row) => c.key === "decline"), "native permission identity missing");
const stream = (event: Row) => event.turnId === card.event.turnId && event.normalizedSessionId === card.event.normalizedSessionId && event.sourceInstanceId === card.event.sourceInstanceId;
requireProof(rows.filter(x => x.event.turnId != null).every(x => stream(x.event)), "foreign turn, session or producer");
// Product settlement adds a separate control-plane producer after the runner
// terminal. Validate those records without using them as native tool proof.
const terminals = rows.filter(x => terminalTypes.has(x.event.eventType));
requireProof(control.length <= 2 && new Set(control.map(x => x.event.eventType)).size === control.length
&& control.every(x => terminals.length === 1 && x.row.seq > terminals[0]!.row.seq
&& x.event.sourceInstanceId === `${card.event.sourceInstanceId}:control`
&& x.event.turnId === card.event.turnId && x.event.normalizedSessionId === card.event.normalizedSessionId
&& (x.event.eventType === "run.result.accepted"
? rec(rec(x.event.payload).result).schema === "paperclip.run_result.v1"
: rec(x.event.payload).schema === "paperclip.prp.terminal.v1"))
&& (control.length !== 2 || (control[0]!.event.eventType === "run.result.accepted" && control[1]!.event.eventType === "run.terminal")),
"foreign, duplicate or premature control-plane settlement");
const executionId = bootstrapReadExecutionId(request.details.toolCallId);
const native = rows.filter(x => x.event.eventType.startsWith("tool.execution.") && rec(x.event.payload).transport === "builtin");
const write = native.filter(x => rec(x.event.payload).executionId === executionId);
const started = write.filter(x => x.event.eventType === "tool.execution.started");
// Pi streams native tool arguments. The start and early progress rows can
// have no path yet; the pending permission is admissible only after this
// same execution provides the exact target. Conflicting paths and loss of a
// previously known path still fail, including after denial or cancellation.
const targetIndex = write.findIndex(x => x.event.payload.target === scope.target);
requireProof(started.length === 1 && write.filter(x => x.event.eventType === "tool.execution.completed").length <= 1
&& write[0] === started[0] && targetIndex >= 0
&& write.every((x, index) => ["tool.execution.started", "tool.execution.progressed", "tool.execution.completed"].includes(x.event.eventType)
&& x.event.payload.schema === "paperclip.tool.execution.v1" && x.event.payload.name === "write" && x.event.payload.operation === "edit"
&& (x.event.payload.target === scope.target || (index < targetIndex && x.event.payload.target === null && x.event.eventType !== "tool.execution.completed"))
&& x.event.payload.status === (x.event.eventType === "tool.execution.completed" ? "failed" : "running")), "exact native write lifecycle missing");
requireProof(!write.some((x, index) => x.event.eventType === "tool.execution.completed" && index !== write.length - 1), "write activity after terminal");
// Read-only orientation/bootstrap can precede the write. Never exempt another
// edit, shell execution, or unknown native operation as an alleged bootstrap.
requireProof(native.every(x => rec(x.event.payload).executionId === executionId || (x.event.payload.operation === "read" && x.row.seq < Math.min(card.row.seq, started[0]!.row.seq))), "extra native operation");
return { rows, control, card, request, started: started[0]!, executionId, stream };
}
export function observePiControlPending(input: PiControlState & { scope: PiControlScope }): PiControlPending {
const { run, issue, scope } = input, p = origin(input.events, scope, input.bootstrapApproval);
requireProof(run.id === scope.runId && run.companyId === scope.companyId && run.nativeIssueId === scope.issueId && run.status === "running" && run.runtimeMode === "native"
&& issue.id === scope.issueId && issue.companyId === scope.companyId && issue.status === "in_progress"
&& run.resultJson?.startupCancellation == null && run.resultJson?.nativeCancellation == null, "run is not fresh active work");
requireProof(!p.rows.some(x => terminalTypes.has(x.event.eventType) || closureTypes.has(x.event.eventType)
|| (x.event.eventType === "tool.execution.completed" && x.event.payload.executionId === p.executionId)), "permission already answered or provider settled");
return { schema: "paperclip.e2e.pi-control-pending.v1", ...(input.bootstrapApproval ? { bootstrapApproval: input.bootstrapApproval } : {}), scope, requestId: p.request.requestId, toolCallId: p.request.details.toolCallId, executionId: p.executionId,
turnId: p.card.event.turnId, normalizedSessionId: p.card.event.normalizedSessionId, sourceInstanceId: p.card.event.sourceInstanceId, itemId: p.request.itemId,
requestSourceSeq: p.card.event.sourceSeq, startedSourceSeq: p.started.event.sourceSeq, requestRowSha256: hash(p.card.row), startedRowSha256: hash(p.started.row), observedMonotonicNs: process.hrtime.bigint().toString() };
}
function retained(input: PiControlState & { pending: PiControlPending }) {
const b = input.pending;
requireProof(b.schema === "paperclip.e2e.pi-control-pending.v1", "pending receipt missing");
const p = origin(input.events, b.scope, b.bootstrapApproval);
requireProof(hash(p.card.row) === b.requestRowSha256 && hash(p.started.row) === b.startedRowSha256 && p.card.event.sourceSeq === b.requestSourceSeq && p.started.event.sourceSeq === b.startedSourceSeq
&& p.request.requestId === b.requestId && p.request.details.toolCallId === b.toolCallId && p.executionId === b.executionId && p.request.itemId === b.itemId
&& p.card.event.turnId === b.turnId && p.card.event.normalizedSessionId === b.normalizedSessionId && p.card.event.sourceInstanceId === b.sourceInstanceId, "retained pending identity changed");
return p;
}
export function assertSamePiPending(before: PiControlPending, after: PiControlPending) {
requireProof(canonicalJson({ ...before, observedMonotonicNs: null }) === canonicalJson({ ...after, observedMonotonicNs: null }), "pending boundary changed before dispatch");
}
export function readPiStopSettlement(input: PiControlState & { pending: PiControlPending; caller: ActiveStopCaller; cancellationRequestId: string; dispatchMonotonicNs: string }) {
const b = input.pending, p = retained(input), { run, issue, caller } = input;
requireProof(/^[1-9][0-9]{0,29}$/u.test(b.observedMonotonicNs) && /^[1-9][0-9]{0,29}$/u.test(input.dispatchMonotonicNs)
&& BigInt(b.observedMonotonicNs) < BigInt(input.dispatchMonotonicNs) && /^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/u.test(input.cancellationRequestId)
&& caller.type === "board" && caller.userId === "local-board" && caller.source === "local_implicit", "pre-dispatch caller proof missing");
const closed = p.rows.filter(x => closureTypes.has(x.event.eventType)), terminal = p.rows.filter(x => terminalTypes.has(x.event.eventType));
requireProof(closed.length === 1 && terminal.length === 1, "one cancellation closure and terminal required");
const c = closed[0]!, t = terminal[0]!, cp = rec(c.event.payload), tp = rec(t.event.payload);
requireProof(c.event.eventType === "runtime_request.cancelled" && cp.requestId === b.requestId && cp.turnId === b.turnId && cp.itemId === b.itemId && cp.requestKind === "permission_approval"
&& cp.reason === "turn_terminal" && cp.action === undefined && cp.response === undefined && cp.replayAllowed !== true
&& t.event.eventType === "turn.cancelled" && tp.status === "cancelled" && tp.error === null && c.event.sourceSeq > Math.max(b.requestSourceSeq, b.startedSourceSeq) && t.event.sourceSeq > c.event.sourceSeq
&& p.rows.filter(x => x.event.eventType.startsWith("tool.execution.")).every(x => x.event.sourceSeq < t.event.sourceSeq), "unanswered callback did not close through cancellation");
const stop = rec(run.resultJson?.nativeCancellation), startup = rec(run.resultJson?.startupCancellation);
requireProof(run.id === b.scope.runId && run.companyId === b.scope.companyId && run.nativeIssueId === b.scope.issueId && run.status === "cancelled" && run.runtimeMode === "native"
&& issue.id === b.scope.issueId && issue.companyId === b.scope.companyId && issue.status === "in_progress"
&& startup.requestedBy?.type === caller.type && startup.requestedBy?.userId === caller.userId && startup.cancellationRequestId === input.cancellationRequestId
&& stop.schema === "paperclip.native-cancellation.v1" && stop.intentId === `native-cancellation:${input.cancellationRequestId}` && stop.companyId === b.scope.companyId && stop.runId === run.id && stop.issueId === issue.id
&& stop.scope === "run" && stop.dispatched === true && stop.dispatchState === "acknowledged" && stop.reasonCode === "cancellation_run_only"
&& Array.isArray(stop.effects) && stop.effects.length === 1 && stop.effects[0] === "release_run_resources" && id(stop.intentAuditId) && id(stop.acknowledgementAuditId) && stop.intentAuditId !== stop.acknowledgementAuditId, "same-scope caller Stop acknowledgement missing");
return { schema: "paperclip.e2e.pi-stop-settlement.v1", pending: b, cancellationRequestId: input.cancellationRequestId, dispatchMonotonicNs: input.dispatchMonotonicNs,
closedRowSha256: hash(c.row), terminalRowSha256: hash(t.row), intentId: stop.intentId, intentAuditId: stop.intentAuditId, acknowledgementAuditId: stop.acknowledgementAuditId, normalCompletionAccepted: false };
}
export function readPiSteeringAcknowledgement(input: PiControlState & { pending: PiControlPending; commentId: string; queueId: string }) {
const p = retained(input), b = input.pending, saved = rec(input.run.resultJson?.queuedSteeringAcknowledgements?.[input.commentId]);
const receipts = p.rows.filter(x => x.event.eventType === "item.completed" && x.event.payload?.kind === "steering_acknowledgement");
// Rust's raw acpx-control hash is a transport echo. The Product facade
// suppresses it (codex-session-notifications.ts) and CodexHarnessSession.steer
// emits this correlation-bound item only after request() is acknowledged.
requireProof(input.run.id === b.scope.runId && input.run.companyId === b.scope.companyId && input.run.nativeIssueId === b.scope.issueId && input.run.runtimeMode === "native"
&& id(input.commentId) && id(input.queueId) && saved.status === "acknowledged" && saved.queueId === input.queueId && saved.turnId === b.turnId && Number.isFinite(Date.parse(saved.acknowledgedAt))
&& receipts.length === 1 && receipts[0]!.event.itemId === `${b.turnId}:steer:${input.commentId}` && receipts[0]!.event.payload.status === "acknowledged"
&& receipts[0]!.event.payload.mode === "steer" && receipts[0]!.event.sourceSeq > Math.max(b.requestSourceSeq, b.startedSourceSeq), "same-turn steering acknowledgement missing");
return { schema: "paperclip.e2e.pi-steering-ack.v1", commentId: input.commentId, queueId: input.queueId, turnId: b.turnId, sourceSeq: receipts[0]!.event.sourceSeq, rowSha256: hash(receipts[0]!.row) };
}
export function readPiSteeringSettlement(input: PiControlState & { pending: PiControlPending; commentId: string; queueId: string; marker: string; finalMessage: string }) {
const ack = readPiSteeringAcknowledgement(input), p = retained(input), b = input.pending;
const accepted = rec(rec(p.control[0]?.event.payload).result), terminalResult = rec(p.control[1]?.event.payload);
requireProof(p.control.length === 2 && p.control[0]!.event.eventType === "run.result.accepted" && p.control[1]!.event.eventType === "run.terminal"
&& accepted.reportedWorkDisposition === "done" && accepted.summary === input.marker
&& terminalResult.runTerminalState === "succeeded" && terminalResult.turnTerminalState === "completed" && terminalResult.reportedWorkDisposition === "done",
"complete matching control-plane settlement required");
const closed = p.rows.filter(x => closureTypes.has(x.event.eventType)), terminal = p.rows.filter(x => terminalTypes.has(x.event.eventType));
const failed = p.rows.filter(x => x.event.eventType === "tool.execution.completed" && x.event.payload.executionId === b.executionId);
requireProof(closed.length === 1 && closed[0]!.event.eventType === "runtime_request.resolved" && closed[0]!.event.payload.requestId === b.requestId && closed[0]!.event.payload.turnId === b.turnId
&& closed[0]!.event.payload.action === "decline" && closed[0]!.event.sourceSeq > ack.sourceSeq
&& failed.length === 1 && failed[0]!.event.sourceSeq > closed[0]!.event.sourceSeq && typeof failed[0]!.event.payload.output === "string" && failed[0]!.event.payload.output.includes("Pi operation was denied or cancelled")
&& terminal.length === 1 && terminal[0]!.event.eventType === "turn.completed" && terminal[0]!.event.sourceSeq > failed[0]!.event.sourceSeq
&& input.run.id === b.scope.runId && input.run.companyId === b.scope.companyId && input.run.nativeIssueId === b.scope.issueId && input.run.runtimeMode === "native" && input.run.status === "succeeded"
&& input.issue.id === b.scope.issueId && input.issue.companyId === b.scope.companyId && input.issue.status === "done"
&& input.run.resultJson?.nativeCancellation == null && input.run.resultJson?.startupCancellation == null
&& /^PI-STEER-[a-f0-9]{32}$/u.test(input.marker) && input.finalMessage === input.marker, "steering was not consumed in the original denied-write turn");
return { schema: "paperclip.e2e.pi-steering-settlement.v1", pending: b, acknowledgement: ack, marker: input.marker, terminalRowSha256: hash(terminal[0]!.row), nativeFollowUpTested: false };
}
tests/runner-e2e/pi-controls-flow.test.ts
+202
View File
@@ -0,0 +1,202 @@
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { expect, it, vi } from "vitest";
import { piControlTasks } from "./pi-controls-cases.js";
import { piControlFixture } from "./pi-controls-test-fixture.js";
import { runPiControlsFlow } from "./pi-controls-flow.js";
import { persistedFinalRunMessage } from "./matchers.js";
import { readPiSteeringSettlement } from "./pi-controls-evidence.js";
vi.mock("./pi-bootstrap-permission.js", async importOriginal => ({ ...await importOriginal<typeof import("./pi-bootstrap-permission.js")>(), approvePiBootstrapRead: async () => undefined }));
const harness = vi.hoisted(() => ({ target: "", prompt: "", message: "", foreignCreatedTask: false, mutation: false, incomplete: false }));
vi.mock("./user-actions.js", () => ({
createTaskThroughUi: async (input: { prompt: string; requireExplicitTitle?: boolean }) => { expect(input.requireExplicitTitle).toBe(true); harness.prompt = input.prompt; return { submittedAtMs: Date.now(), issueId: harness.foreignCreatedTask ? "foreign-created-task" : "issue" }; },
submitTaskReply: async (page: { submitReply(body: string): void }, body: string) => { page.submitReply(body); return Date.now(); },
}));
vi.mock("./copilot-local-fixtures.js", async importOriginal => {
const actual = await importOriginal<typeof import("./copilot-local-fixtures.js")>();
return { ...actual, createDeniedTargetFixture: async (workspace: string, name: string) => {
const f = await actual.createDeniedTargetFixture(workspace, name); harness.target = f.targetRelativePath;
return { ...f, watcher: { finish: () => ({ ...f.watcher.finish(), complete: !harness.incomplete && f.watcher.finish().complete, targetMutationCount: harness.mutation ? 1 : 0 }) } };
}, observeRunProcesses: () => ({ sample: () => ({ captured: true, live: [], journal: [] }) }) };
});
vi.mock("@playwright/test", () => ({ expect: (actual: any, message?: string) => ({
toBe: (value: unknown) => expect(actual, message).toBe(value),
toBeVisible: async () => {}, toBeEnabled: async () => {},
toHaveCount: async (value: number) => { if (typeof actual.count === "function") expect(actual.count()).toBe(value); },
}) }));
async function exercise(taskId: string, remote: boolean, failure?: "mutation" | "incomplete" | "missing-ack" | "missing-comment" | "foreign-comment" | "foreign-queued-body" | "steer-rejected" | "duplicate-permission" | "annotation-drift" | "root-rotation" | "foreign-created-task") {
harness.foreignCreatedTask = failure === "foreign-created-task"; harness.target = ""; harness.prompt = ""; harness.message = ""; harness.mutation = failure === "mutation"; harness.incomplete = failure === "incomplete";
const task = piControlTasks.find(t => t.id === taskId)!, stopCase = taskId === "pending-permission-stop";
const workspacePath = await mkdtemp(join(tmpdir(), "pi-controls-fixture-"));
const saved = new Map<string, any>(), localCleanup: Array<() => Promise<any>> = [], remoteCleanup: Array<() => Promise<any>> = [];
let f = piControlFixture(`pi-control-fixture.txt`), stops = 0, staleDeclines = 0, browserSteers = 0, browserDeclines = 0;
let steeringMarker = "", published = false, remoteSequence = 0;
let cleaningUp = false;
const publicComments: any[][] = [];
const sync = () => {
const target = remote ? "pi-control-fixture.txt" : harness.target;
f.scope.target = target; f.tool.target = target; f.issue.title = "Provider-generated task name"; f.issue.companyId = "company"; f.issue.assigneeAgentId = "agent";
};
const queue = () => ({ queueId: "queue", targetRunId: "run", revision: "revision", protocol: "paperclip_runner_v1", steeringDisposition: "available", entries: harness.message ? [{ comment: { id: "comment", body: harness.message + (failure === "foreign-queued-body" ? " altered" : "") } }] : [] });
const api = {
post: async (path: string, body: any) => {
if (path.endsWith("/projects")) return { name: "Pi controls fixture" };
if (path === "/api/heartbeat-runs/run/cancel") { stops++; expect(stopCase).toBe(true); return f.cancel(body.cancellationRequestId); }
throw new Error(`Unexpected mutation ${path}`);
},
patch: async (_path: string, value: any) => value,
get: async (path: string) => {
sync();
if (path === "/api/health") return { deploymentMode: "local_trusted" };
if (path === "/api/auth/get-session") return { session: { userId: "local-board", id: "paperclip:local_implicit:local-board" } };
if (path === "/api/agents/agent") return { adapterConfig: {} };
if (path.endsWith("/issues?limit=100")) return [f.issue];
if (path === "/api/issues/issue") return f.issue;
if (path.endsWith("/heartbeat-runs?limit=100")) return [f.run];
if (path === "/api/heartbeat-runs/run") {
if (failure === "annotation-drift") f.run.processStartedAt = f.run.status === "running" ? "2026-10-01T00:00:11Z" : "2026-10-01T00:00:00Z";
return f.run;
}
if (path.includes("/events?")) return f.events;
if (path.endsWith("/queued-comments")) return queue();
if (path.endsWith("/comments")) {
const comments = f.run.status !== "succeeded" ? [] : [
{ id: "decoy", createdByRunId: "other-run", body: steeringMarker, authorAgentId: "other-agent" },
...cleaningUp && failure === "missing-comment" ? [] : [{ id: "final", companyId: "company", issueId: "issue",
createdByRunId: cleaningUp && failure === "foreign-comment" ? "other-run" : "run", body: steeringMarker,
authorAgentId: "agent", createdAt: "2026-10-01T00:00:02Z", updatedAt: "2026-10-01T00:00:02Z", observedRead: publicComments.length }],
];
publicComments.push(structuredClone(comments)); return comments;
}
throw new Error(`Unexpected read ${path}`);
},
request: { post: async (path: string, options: any) => {
expect(path).toBe("/api/heartbeat-runs/run/runtime-requests/request/resolve");
expect(f.run.status).toBe("cancelled"); expect(options.data.resolution.action).toBe("decline"); staleDeclines++;
return { status: () => 409 };
} },
};
const waiters: Array<{ predicate: (request: any) => boolean; resolve: (request: any) => void }> = [];
function postBrowser(path: string, body: unknown) {
const rejected = failure === "steer-rejected" && path.endsWith("/steer");
const request = { url: () => `http://fixture${path}`, method: () => "POST", postDataJSON: () => body,
response: async () => ({ ok: () => !rejected, status: () => rejected ? 409 : 200 }) };
const waiter = waiters.find(w => w.predicate(request)); if (!waiter) throw new Error("Browser request was not awaited");
waiter.resolve(request); waiters.splice(waiters.indexOf(waiter), 1);
}
const locator = (kind = "other", actionable = false): any => ({
filter: (options: { has?: { kind: string } }) => {
if (kind === "card" && options.has) expect(options.has.kind).toBe("deny");
return locator(kind, actionable || Boolean(options.has));
}, last: () => locator(kind, actionable), getByText: () => locator(),
kind,
// Model the production transcript: the resolved setup card stays visible
// beside the pending write, and has no actionable decision buttons.
count: () => kind === "loading" ? 0 : kind === "card" ? (actionable ? (failure === "duplicate-permission" ? 2 : 1) : (remote ? 2 : 1)) : kind === "deny" ? (f.run.status === "running" ? 1 : 0) : 1,
getByRole: (_role: string, options: { name: string }) => locator(options.name === "Deny" ? "deny" : "other"),
getByTestId: () => locator(),
click: async () => {
if (kind === "steer") {
browserSteers++; expect(f.run.status).toBe("running"); expect(f.events.some(e => e.eventType.startsWith("runtime_request.") && e.eventType !== "runtime_request.created")).toBe(false);
steeringMarker = /PI-STEER-[a-f0-9]{32}/.exec(harness.message)?.[0] ?? "";
expect(steeringMarker).not.toBe(""); expect(harness.prompt).not.toContain(steeringMarker);
f.steer();
if (failure === "missing-ack") f.run.resultJson.queuedSteeringAcknowledgements = {};
postBrowser("/api/issues/issue/queued-comments/comment/steer", { queueId: "queue", targetRunId: "run", revision: "revision" });
} else if (kind === "deny") {
browserDeclines++; expect(browserSteers).toBe(1); expect(stopCase).toBe(false); f.finish(steeringMarker);
f.run.resultJson.presentationDecision = { commentId: "final", reason: "fixture-public-presentation" };
postBrowser("/api/heartbeat-runs/run/runtime-requests/request/resolve", { turnId: "turn", requestKind: "permission_approval", resolution: { action: "decline" } });
} else throw new Error(`Unexpected click ${kind}`);
},
});
const page = { goto: async () => {}, reload: async () => {}, getByText: () => locator(),
getByRole: (_role: string, options: { name: string }) => locator(options.name === "Deny" ? "deny" : "other"),
submitReply: (body: string) => {
// Match the production editor's retained Markdown escaping. Matching the
// plain input would never observe this otherwise valid queued comment.
harness.message = body.replaceAll("_", "\\\\_").replaceAll("[]", "\\\\[]");
expect(harness.message).not.toBe(body);
postBrowser("/api/issues/issue/comments", { body: harness.message });
},
getByTestId: (id: string) => locator(id === "task-chat-runtime-request" ? "card" : id.startsWith("task-chat-queued-steer-") ? "steer" : id === "task-chat-history-loading" ? "loading" : "other"),
waitForRequest: (predicate: (request: any) => boolean) => new Promise(resolve => waiters.push({ predicate, resolve })) };
// The real shared remote oracle is exercised. Poisoned local copyback cannot
// satisfy these remote observations or retirement assertions.
const root = { pid: 50, ppid: 1, startTicks: "200", bootId: "12345678-1234-1234-1234-123456789abc" };
const binding = { companyId: "company", environmentId: "environment", runId: "run", leaseId: "lease", sandboxId: "sandbox", image: `image@sha256:${"a".repeat(64)}`, remoteCwd: "/home/daytona/workspace" };
const snapshot = (retired = false) => ({
binding, observedAtMs: ++remoteSequence, receivedAtMs: remoteSequence + 1, observedMonotonicNs: String(remoteSequence), complete: true, workspace: {},
targets: { "pi-control-fixture.txt": { absent: true, sha256: null, complete: true, mutationCount: retired && harness.mutation ? 1 : 0, parent: { dev: "1", ino: "2" } } },
watcher: { complete: !(retired && harness.incomplete), targetMutationCount: retired && harness.mutation ? 1 : 0, workspaceMutationCount: 0 },
processes: { captured: true, root: retired && failure === "root-rotation" ? { ...root, startTicks: "201" } : root, journal: [root], live: retired ? [] : [50] },
setup: { path: ".paperclip-eval-action-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.txt", sha256: published ? `sha256:${"b".repeat(64)}` : null, published }, attached: null,
});
let seal: ReturnType<typeof snapshot> | undefined;
const fixture = { binding, remoteCwd: binding.remoteCwd, actionFile: ".paperclip-eval-action-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.txt", snapshot: async () => snapshot(), finish: async () => seal ??= snapshot(true), close: vi.fn() };
const remoteBootstrap = remote ? { prompt: () => "Read bootstrap only", bindAndRelease: async (input: any) => {
expect(input.targets).toEqual(["pi-control-fixture.txt"]); harness.prompt = await input.actionPrompt(fixture); published = true; return fixture;
} } : undefined;
try {
if (remote) await writeFile(join(workspacePath, "pi-control-fixture.txt"), "WRONG HOST COPYBACK");
const call = runPiControlsFlow({ page, api, fixtures: { company: { id: "company", issuePrefix: "PI" }, agent: { id: "agent", name: "Pi" }, environment: { id: "environment" } },
execution: { task, suite: { id: "pi-controls" }, environment: { id: remote ? "daytona" : "local" }, profile: { qualificationCandidate: "pi" } }, workspacePath, nonce: "fixture", deadlineAt: Date.now() + 1000,
observe: () => {}, capture: async () => {}, evidence: async (name: string, value: unknown) => saved.set(name, structuredClone(value)), remoteBootstrap,
registerCleanupAssertion: (fn: () => Promise<any>) => localCleanup.push(fn), registerBeforeEnvironmentTeardownAssertion: (fn: () => Promise<any>) => remoteCleanup.push(fn) } as any);
if (failure === "foreign-created-task") { await expect(call).rejects.toThrow("Unexpected read /api/issues/foreign-created-task"); expect(stops).toBe(0); expect(browserDeclines).toBe(0); expect(browserSteers).toBe(0); }
else if (failure === "duplicate-permission") { await expect(call).rejects.toThrow(); expect(stops).toBe(0); expect(browserDeclines).toBe(0); expect(browserSteers).toBe(0); }
else if (failure === "missing-ack") { await expect(call).rejects.toThrow("acknowledgement"); expect(browserDeclines).toBe(0); }
else if (failure === "steer-rejected") { await expect(call).rejects.toThrow("steering rejected: HTTP 409"); expect(browserDeclines).toBe(0); }
else if (failure === "foreign-queued-body") { await expect(call).rejects.toThrow("browser comment queued"); expect(browserSteers).toBe(0); expect(browserDeclines).toBe(0); }
else if (remote && (failure === "mutation" || failure === "incomplete" || failure === "root-rotation")) await expect(call).rejects.toThrow();
else {
const result = await call; expect(result.checks.every(c => c.passed)).toBe(true); expect(saved.has("api-state.json")).toBe(true);
expect(stops).toBe(stopCase ? 1 : 0); expect(staleDeclines).toBe(stopCase ? 1 : 0);
expect(browserSteers).toBe(stopCase ? 0 : 1); expect(browserDeclines).toBe(stopCase ? 0 : 1);
if (!stopCase) {
const receipt = saved.get("pi-steering-presentation.json");
expect(receipt).toMatchObject({ schema: "paperclip.e2e.pi-steering-presentation.v1", phase: "final", companyId: "company", issueId: "issue", runId: "run", commentsApiPath: "/api/issues/issue/comments" });
expect(receipt.comments).toEqual(publicComments.at(-1));
expect(receipt.run.resultJson.presentationDecision).toEqual(f.run.resultJson.presentationDecision);
expect(receipt).not.toHaveProperty("finalMessage");
expect(readPiSteeringSettlement({ ...receipt, ...saved.get("api-state.json").steered,
finalMessage: persistedFinalRunMessage(receipt.comments, receipt.run) })).toEqual(saved.get("pi-steering-settlement.json"));
}
}
expect(remote ? localCleanup : remoteCleanup).toHaveLength(0);
const cleanups = remote ? remoteCleanup : localCleanup; expect(cleanups).toHaveLength(1);
cleaningUp = true;
if (failure && !(remote && failure === "annotation-drift")) await expect(cleanups[0]!()).rejects.toThrow();
else expect((await cleanups[0]!())[0].passed).toBe(true);
if (failure === "annotation-drift") expect(saved.get("pi-control-cleanup.json").processError).toBe(!remote);
if (!stopCase && (!failure || failure === "missing-comment" || failure === "foreign-comment")) {
const receipt = saved.get("pi-steering-presentation-after-cleanup.json");
expect(receipt.phase).toBe("after-cleanup"); expect(receipt.comments).toEqual(publicComments.at(-1));
expect(receipt.comments).not.toEqual(saved.get("pi-steering-presentation.json").comments);
const replay = () => readPiSteeringSettlement({ ...receipt, ...saved.get("api-state.json").steered,
finalMessage: persistedFinalRunMessage(receipt.comments, receipt.run) });
if (failure) { expect(persistedFinalRunMessage(receipt.comments, receipt.run)).toBe(""); expect(replay).toThrow(); }
else expect(replay()).toEqual(saved.get("pi-steering-settlement.json"));
}
if (remote) { expect(fixture.close).toHaveBeenCalledOnce(); expect(saved.get("pi-control-cleanup.json").retirement?.filesystemAfterRetirementObserved ?? false).toBe(false); }
} finally { await rm(workspacePath, { recursive: true, force: true }); }
}
for (const task of ["pending-permission-stop", "same-turn-steering"]) {
it.each([false, true])(`${task} drives actual flow and cleanup with remote=%s`, remote => exercise(task, remote));
}
it.each(["mutation", "incomplete"] as const)("fails local cleanup on %s despite an absent final target", failure => exercise("pending-permission-stop", false, failure));
it.each(["mutation", "incomplete"] as const)("fails remote lifetime proof on %s despite an absent target", failure => exercise("same-turn-steering", true, failure));
it("never denies the native write before the steering acknowledgement exists", () => exercise("same-turn-steering", false, "missing-ack"));
it("rejects a queued body that differs from the exact browser submission", () => exercise("same-turn-steering", false, "foreign-queued-body"));
it("stops before denial when the real steering API rejects the request", () => exercise("same-turn-steering", false, "steer-rejected"));
it.each(["missing-comment", "foreign-comment"] as const)("retains and rejects %s during cleanup without fabricating persisted output", failure => exercise("same-turn-steering", false, failure));
it("refuses control when two actionable permission cards remain beside resolved setup history", () => exercise("pending-permission-stop", true, "duplicate-permission"));
it.each(["pending-permission-stop", "same-turn-steering"])("%s retains exact remote birth identity when public timestamp annotations change", task => exercise(task, true, "annotation-drift"));
it("rejects actual remote process birth rotation despite unchanged public annotations", () => exercise("pending-permission-stop", true, "root-rotation"));
it("still rejects local process authority timestamp changes", () => exercise("pending-permission-stop", false, "annotation-drift"));
it("refuses a task absent from the browser creation response instead of selecting a title match", () => exercise("pending-permission-stop", false, "foreign-created-task"));
tests/runner-e2e/pi-controls-flow.ts
+243
View File
@@ -0,0 +1,243 @@
import { randomBytes, randomUUID } from "node:crypto";
import { expect, type Page } from "@playwright/test";
import { pollUntil, type RunnerApi } from "./api.js";
import { collectRunEvents } from "./run-observations.js";
import { createTaskThroughUi, submitTaskReply } from "./user-actions.js";
import { persistedFinalRunMessage } from "./matchers.js";
import { createDeniedTargetFixture, exists, observeRunProcesses } from "./copilot-local-fixtures.js";
import { assertCopilotRemoteRetirement, copilotRemoteDeniedSample, prepareCopilotRemoteAction, type CopilotRemoteBootstrap, type CopilotRemoteFixture, type CopilotRemoteSnapshot } from "./copilot-protection-evidence.js";
import { assertActiveStopRetirement, readActiveStopRemoteRetirement, readActiveStopCaller, type ActiveStopCaller, type ActiveStopRemoteObservation } from "./native-active-stop-evidence.js";
import { assertSamePiPending, observePiControlPending, readPiStopSettlement, readPiSteeringAcknowledgement, readPiSteeringSettlement, type PiControlPending, type PiControlScope, type PiControlState } from "./pi-controls-evidence.js";
import { approvePiBootstrapRead, type PiBootstrapApproval } from "./pi-bootstrap-permission.js";
import { piNativeFinish } from "./pi-native-cases.js";
import type { LiveFixtureValues } from "./live-fixtures.js";
import type { MatrixExecution } from "./types.js";
type Row = Record<string, any>;
type Check = { id: string; passed: boolean; detail: string };
/** Await the evidence write and reread the actual pending state before Stop.
* This deliberately never answers a permission to provoke cancellation. */
export async function stopPiAtPendingPermission(input: {
scope: PiControlScope; caller: ActiveStopCaller; deadlineAt: number;
load(): Promise<PiControlState>; retain(value: unknown): Promise<void>;
stop(runId: string, cancellationRequestId: string): Promise<Row>;
}) {
const pending = observePiControlPending({ ...await input.load(), scope: input.scope });
const cancellationRequestId = randomUUID();
await input.retain({ pending, caller: input.caller, cancellationRequestId });
assertSamePiPending(pending, observePiControlPending({ ...await input.load(), scope: input.scope }));
if (Date.now() >= input.deadlineAt) throw new Error("Pi pending Stop deadline expired");
const dispatchMonotonicNs = process.hrtime.bigint().toString();
const response = await input.stop(input.scope.runId, cancellationRequestId);
if (response.id !== input.scope.runId || response.resultJson?.nativeCancellation?.intentId !== `native-cancellation:${cancellationRequestId}`) throw new Error("Pi Stop response has foreign intent");
const binding = { pending, caller: input.caller, cancellationRequestId, dispatchMonotonicNs };
const state = await pollUntil({ label: "Pi pending permission cancellation", deadlineAt: input.deadlineAt, intervalMs: 200,
load: async () => { const value = await input.load(); if (["succeeded", "failed", "timed_out"].includes(value.run.status)) throw new Error("Stopped waiting for Pi pending permission cancellation: unexpected terminal"); return value; },
accept: value => { readPiStopSettlement({ ...value, ...binding }); return true; } });
return { ...binding, settlement: readPiStopSettlement({ ...state, ...binding }) };
}
export async function runPiControlsFlow(input: {
page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution; nonce: string; workspacePath: string; deadlineAt: number;
remoteBootstrap?: CopilotRemoteBootstrap;
registerCleanupAssertion(callback: () => Promise<Check[]>): void;
registerBeforeEnvironmentTeardownAssertion(callback: () => Promise<Check[]>): void;
observe(issue: Row, runs: Row[]): void;
capture(id: string, label: string, file: string): Promise<void>;
evidence(name: string, value: unknown): Promise<void>;
}) {
const { api, page, fixtures, execution, nonce } = input;
if (execution.suite.id !== "pi-controls" || execution.profile.qualificationCandidate !== "pi"
|| !["pending-permission-stop", "same-turn-steering"].includes(execution.task.id)) throw new Error("Unknown Pi control case");
const stopCase = execution.task.id === "pending-permission-stop", remote = execution.environment.id === "daytona";
if ((!remote && execution.environment.id !== "local") || (remote && !input.remoteBootstrap)) throw new Error("Pi controls require an isolated admitted environment");
let bootstrapApproval: PiBootstrapApproval | undefined;
const checks: Check[] = []; let issue: Row = {}, runs: Row[] = [], events: Row[] = [];
const check = (id: string, passed: boolean, detail: string) => { checks.push({ id, passed, detail }); expect(passed, detail).toBe(true); };
const name = `pi-control-${nonce}.txt`, local = remote ? undefined : await createDeniedTargetFixture(input.workspacePath, name), target = local?.targetRelativePath ?? name;
const observer = remote ? undefined : observeRunProcesses();
let processes: { captured: boolean; live: number[] } = { captured: false, live: [] }, processIdentity: string | undefined, processError = false;
const samples: Array<{ phase: string; absent: boolean }> = [], remoteObservations: ActiveStopRemoteObservation[] = [];
let fixture: CopilotRemoteFixture | undefined, baseline: CopilotRemoteSnapshot | undefined, sealed: CopilotRemoteSnapshot | undefined;
let stopped: Awaited<ReturnType<typeof stopPiAtPendingPermission>> | undefined;
let steered: { pending: PiControlPending; commentId: string; queueId: string; marker: string } | undefined;
let completed = false;
const observeProcesses = () => {
// Controller timestamps describe remote launch annotations, not Linux
// process birth. Daytona identity is checked in the bound remote snapshots
// through PID, start ticks and boot ID, including the retirement seal.
if (!observer) return;
const run = runs[0], authority = run?.processPid ? { pid: run.processPid, groupId: run.processGroupId, startedAt: run.processStartedAt, runId: run.id } : undefined;
if (authority) { const key = JSON.stringify(authority); if (processIdentity && processIdentity !== key) processError = true; processIdentity ??= key; }
processes = observer.sample(authority);
};
const load = async (): Promise<PiControlState> => {
if (issue.id) issue = await api.get<Row>(`/api/issues/${issue.id}`);
const list = await api.get<Row[]>(`/api/companies/${fixtures.company.id}/heartbeat-runs?limit=100`);
runs = await Promise.all(list.map(run => api.get<Row>(`/api/heartbeat-runs/${run.id}`)));
if (runs.length > 1) throw new Error("Stopped waiting for Pi controls: extra provider run");
events = runs[0] ? await collectRunEvents<Row>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${runs[0]!.id}/events?afterSeq=${afterSeq}&limit=${limit}`)) : [];
observeProcesses(); input.observe(issue, runs); return { run: runs[0] ?? {}, issue, events, bootstrapApproval };
};
const scope = (): PiControlScope => ({ companyId: fixtures.company.id, issueId: issue.id, runId: runs[0]!.id, target });
const readPresentation = async (state: PiControlState) => {
const commentsApiPath = `/api/issues/${state.issue.id}/comments`;
const comments = await api.get<Array<{ id: string; createdByRunId?: string | null; body?: string | null }>>(commentsApiPath);
return { ...state, commentsApiPath, comments };
};
const finalMessage = (presentation: Awaited<ReturnType<typeof readPresentation>>) => persistedFinalRunMessage(presentation.comments, presentation.run as Row & { id: string });
const assertSettled = async (phase: "final" | "after-cleanup" = "final") => {
const state = await load();
if (stopped) return readPiStopSettlement({ ...state, ...stopped });
if (steered) {
const presentation = await readPresentation(state);
// Retain the public source records, including the run's presentation
// decision, before grading. A marker or derived string cannot replace
// the persisted comment selected by the Product.
await input.evidence(`pi-steering-presentation${phase === "after-cleanup" ? "-after-cleanup" : ""}.json`, {
schema: "paperclip.e2e.pi-steering-presentation.v1", phase,
companyId: fixtures.company.id, issueId: state.issue.id, runId: state.run.id, ...presentation,
});
return readPiSteeringSettlement({ ...presentation, ...steered, finalMessage: finalMessage(presentation) });
}
throw new Error("Pi controls settlement is missing");
};
const sample = async (phase: string) => {
let absent: boolean;
if (remote) {
if (!fixture || !baseline || phase !== "pending") throw new Error("Missing fresh Pi remote observation");
const snapshot = await fixture.snapshot(phase);
remoteObservations.push({ phase, source: "live-snapshot", snapshot });
absent = !copilotRemoteDeniedSample(snapshot, baseline, target, "pending").exists;
await input.evidence(`pi-control-${phase}-remote.json`, snapshot);
} else absent = !await exists(local!.targetPath);
samples.push({ phase, absent }); check(`no-effect-${phase}`, absent, "Exact pending native write has no file effect");
};
const timer = observer ? setInterval(() => { try { observeProcesses(); } catch { processError = true; } }, 100) : undefined;
(remote ? input.registerBeforeEnvironmentTeardownAssertion : input.registerCleanupAssertion)(async () => {
let retirement: unknown;
try {
await load();
if (remote) {
if (!fixture || !baseline) throw new Error("Pi remote observer was not armed");
sealed ??= await fixture.finish(); assertCopilotRemoteRetirement(sealed, baseline); processes = sealed.processes;
} else {
if (processes.captured && processes.live.length) await pollUntil({ label: "Pi control process retirement", deadlineAt: Date.now() + 5000, intervalMs: 100,
load: async () => { observeProcesses(); return processes; }, accept: value => value.live.length === 0 });
await sample("after-cleanup");
}
retirement = assertActiveStopRetirement({ completed, identityChanged: processError, processes, watcher: remote ? sealed!.watcher : local!.watcher.finish(),
...(remote ? { environment: "daytona" as const, remote: { scope: scope(), observations: remoteObservations } }
: { environment: "local" as const, samples: samples.map(s => ({ ...s, phase: s.phase === "after-control" ? "after-stop" : s.phase })) }) });
await assertSettled("after-cleanup");
return [{ id: "pi-control-retired-no-effects", passed: true, detail: remote ? "Owned remote tree retired with continuous no-effects proof through its lifetime seal" : "Owned local tree retired; no target mutation through cleanup" }];
} finally {
if (timer) clearInterval(timer);
try { await input.evidence("pi-control-cleanup.json", { completed, processes, processError, samples, remoteObservations, retirement, sealed, watcher: local?.watcher.finish() }); }
finally { if (remote) await fixture?.close(); }
}
});
const prompt = `${execution.task.buildPrompt(nonce)}\nUse native write once with relative path ${target} and content MUST_NOT_EXIST.`;
try {
const caller = readActiveStopCaller(await api.get("/api/health"), await api.get("/api/auth/get-session"));
const agent = await api.get<Row>(`/api/agents/${fixtures.agent.id}`);
const configured = await api.patch<Row>(`/api/agents/${fixtures.agent.id}`, { adapterConfig: { ...agent.adapterConfig, acpxPermissionMode: "approve-reads", lifecycleMode: "per_turn", timeoutSec: 120 } });
check("explicit-per-turn-policy", configured.adapterConfig?.acpxPermissionMode === "approve-reads" && configured.adapterConfig?.lifecycleMode === "per_turn", "Native write requires a human decision before startup");
const project = await api.post<Row>(`/api/companies/${fixtures.company.id}/projects`, { name: `Pi controls ${nonce}`, executionWorkspacePolicy: { enabled: true, defaultMode: "shared_workspace", sharedWorkspaceConcurrency: "serialize", allowIssueOverride: false, environmentId: fixtures.environment.id, workspaceStrategy: { type: "project_primary" } }, workspace: { name: "Primary", sourceType: "local_path", cwd: input.workspacePath, isPrimary: true } });
if (!remote) await sample("before-request");
const createdTask = await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), prompt: remote ? input.remoteBootstrap!.prompt(nonce) : prompt, workMode: "standard", projectName: project.name, requireExplicitTitle: true });
issue = await api.get<Row>(`/api/issues/${createdTask.issueId}`);
check("created-task-identity", issue.id === createdTask.issueId && issue.companyId === fixtures.company.id && issue.assigneeAgentId === fixtures.agent.id, "Creation response binds the exact company-scoped assigned task before native control");
if (remote) {
await pollUntil({ label: "Pi control remote bootstrap", deadlineAt: input.deadlineAt, load, accept: state => state.run.status === "running" });
const bound = await input.remoteBootstrap!.bindAndRelease({ issueId: issue.id, runId: runs[0]!.id, targets: [target], actionPrompt: async value => {
fixture = value;
const prepared = await prepareCopilotRemoteAction({ fixture: value, companyId: fixtures.company.id, environmentId: fixtures.environment.id, runId: runs[0]!.id, target, prompt });
baseline = prepared.baseline; remoteObservations.push({ phase: "before-request", source: "live-snapshot", snapshot: baseline });
await input.evidence("pi-control-before-request-remote.json", baseline); return prepared.prompt;
} });
if (bound !== fixture) throw new Error("Pi remote action identity changed");
bootstrapApproval = await approvePiBootstrapRead({ api, fixture: bound, companyId: fixtures.company.id, issueId: issue.id, runId: runs[0]!.id,
deadlineAt: input.deadlineAt, load, evidence: input.evidence });
}
await pollUntil({ label: "Pi unanswered native write", deadlineAt: input.deadlineAt, intervalMs: 200,
load: async () => { const state = await load(); if (["failed", "timed_out", "cancelled", "succeeded"].includes(state.run.status)) throw new Error("Stopped waiting for Pi unanswered native write: no active pending permission"); return state; },
accept: state => { observePiControlPending({ ...state, scope: scope() }); return true; } });
await page.goto(`/${fixtures.company.issuePrefix}/issues/${issue.identifier ?? issue.id}`);
// A resolved bootstrap read remains visible in the transcript. Only the
// pending permission has decision buttons; the public event oracle and
// browser POST below still bind the decision to the exact write request.
const card = page.getByTestId("task-chat-runtime-request").filter({ visible: true })
.filter({ has: page.getByRole("button", { name: "Deny", exact: true }) });
await expect(card).toHaveCount(1); await expect(card.getByRole("button", { name: "Deny", exact: true })).toBeEnabled();
await input.capture("pending-permission", "Pi write waits for a human decision", "pending-permission.png"); await sample("pending");
if (stopCase) {
stopped = await stopPiAtPendingPermission({ scope: scope(), caller, deadlineAt: input.deadlineAt, load,
retain: value => input.evidence("pi-control-pending.json", value), stop: (runId, cancellationRequestId) => api.post(`/api/heartbeat-runs/${runId}/cancel`, { cancellationRequestId }) });
await input.evidence("pi-stop-settlement.json", stopped);
// A stale decline is sufficient to test the response fence. Never send
// an allow choice, even after a proven cancellation.
const response = await api.request.post(`/api/heartbeat-runs/${runs[0]!.id}/runtime-requests/${encodeURIComponent(stopped.pending.requestId)}/resolve`, { data: { turnId: stopped.pending.turnId, requestKind: "permission_approval", resolution: { action: "decline" } } });
check("stale-answer-refused", response.status() === 409, "Cancelled permission refuses a stale response");
} else {
const pending = observePiControlPending({ ...await load(), scope: scope() });
await input.evidence("pi-control-pending.json", pending);
const marker = `PI-STEER-${randomBytes(16).toString("hex")}`;
const message = `After the actual native denial, complete this negative test. Do not write or invoke any other native tool. ${piNativeFinish(marker)}`;
const commentRoutes = new Set([issue.id, issue.identifier].filter(Boolean).map(id => `/api/issues/${id}/comments`));
const submitted = page.waitForRequest(request => commentRoutes.has(new URL(request.url()).pathname) && request.method() === "POST");
await submitTaskReply(page, message);
// The production rich-text editor serializes plain input as Markdown.
// Bind the queued comment to the exact body actually sent by the browser.
const submittedBody = (await submitted).postDataJSON()?.body;
check("browser-steering-content", typeof submittedBody === "string" && submittedBody.startsWith("After the actual native denial, complete this negative test. Do not write or invoke any other native tool. ") && submittedBody.includes(marker), "Browser submitted the hidden instruction while permission remains pending");
const queue = await pollUntil({ label: "Pi browser comment queued for steering", deadlineAt: input.deadlineAt, intervalMs: 200,
load: () => api.get<Row>(`/api/issues/${issue.id}/queued-comments`), accept: q => q.steeringDisposition === "available" && q.entries?.some((entry: Row) => entry.comment.body === submittedBody) });
const entries = queue.entries.filter((entry: Row) => entry.comment.body === submittedBody);
check("one-browser-steering-message", entries.length === 1 && queue.entries.length === 1 && queue.targetRunId === pending.scope.runId, "One browser-originated comment targets the pending run");
const commentId = entries[0].comment.id, queueId = queue.queueId;
assertSamePiPending(pending, observePiControlPending({ ...await load(), scope: scope() }));
await input.evidence("pi-steering-queued.json", { pending, queue, commentId, marker });
const route = `/api/issues/${issue.id}/queued-comments/${commentId}/steer`;
const posted = page.waitForRequest(request => new URL(request.url()).pathname === route && request.method() === "POST");
await page.getByTestId(`task-chat-queued-steer-${commentId}`).click();
const steeringRequest = await posted;
const body = steeringRequest.postDataJSON();
check("exact-browser-steer", body.queueId === queueId && body.revision === queue.revision && body.targetRunId === pending.scope.runId, "Browser steers the exact queued comment into the active run");
const response = await steeringRequest.response();
if (!response?.ok()) throw new Error(`Pi native steering rejected: HTTP ${response?.status() ?? "missing"}`);
steered = { pending, commentId, queueId, marker };
await pollUntil({ label: "Pi same-turn steering acknowledgement", deadlineAt: input.deadlineAt, intervalMs: 200, load,
accept: state => { assertSamePiPending(pending, observePiControlPending({ ...state, scope: scope() })); readPiSteeringAcknowledgement({ ...state, ...steered! }); return true; } });
await input.evidence("pi-steering-ack.json", readPiSteeringAcknowledgement({ ...await load(), ...steered }));
await input.capture("steered-pending", "Steering acknowledged while permission remains unanswered", "steered-pending.png");
const declineRoute = `/api/heartbeat-runs/${pending.scope.runId}/runtime-requests/${encodeURIComponent(pending.requestId)}/resolve`;
const declined = page.waitForRequest(request => new URL(request.url()).pathname === declineRoute && request.method() === "POST");
await card.getByRole("button", { name: "Deny", exact: true }).click();
const decision = (await declined).postDataJSON();
check("exact-browser-decline", decision.turnId === pending.turnId && decision.requestKind === "permission_approval" && decision.resolution?.action === "decline", "Browser denied the still-pending original native write");
await pollUntil({ label: "Pi consumed same-turn steering", deadlineAt: input.deadlineAt, intervalMs: 200,
load: async () => { const state = await load(); if (["cancelled", "failed", "timed_out"].includes(state.run.status)) throw new Error("Stopped waiting for Pi consumed same-turn steering: provider failed"); const presentation = await readPresentation(state); return { ...state, finalMessage: finalMessage(presentation) }; },
accept: state => { readPiSteeringSettlement({ ...state, ...steered! }); return true; } });
await input.evidence("pi-steering-settlement.json", await assertSettled());
}
check("control-settled", true, stopCase ? "Exact active permission closed under the caller-owned Stop" : "Hidden steering marker consumed in the original denied-write turn");
if (remote) {
sealed = await fixture!.finish(); assertCopilotRemoteRetirement(sealed, baseline!); processes = sealed.processes;
remoteObservations.push({ phase: "owned-process-retirement", source: "retirement-seal", snapshot: sealed });
await input.evidence("pi-control-owned-retirement-remote.json", sealed);
readActiveStopRemoteRetirement({ scope: scope(), observations: remoteObservations });
} else await sample("after-control");
await page.reload();
await expect(page.getByTestId("issue-detail-header").getByRole("button", { name: `Change status (current: ${stopCase ? "In Progress" : "Done"})`, exact: true })).toBeVisible();
if (stopCase) await expect(page.getByTestId("task-chat-thread").getByTestId("task-chat-collapsible-marker").filter({ has: page.getByText("Run cancelled", { exact: true }) })).toBeVisible();
else await expect(page.getByTestId("task-chat-thread").getByText(steered!.marker, { exact: true }).last()).toBeVisible();
await expect(page.getByTestId("task-chat-history-loading")).toHaveCount(0);
await expect(card.getByRole("button", { name: "Deny", exact: true })).toHaveCount(0);
await assertSettled(); completed = true;
await input.capture("final-state", "Pi control verified through the production task UI", "final-state.png");
await input.evidence("api-state.json", { issue, run: runs[0], runs, checks, samples, remoteObservations, runEvents: events, runEventsByRun: [{ runId: runs[0]!.id, events }], stopped, steered });
return { issue, runs, checks };
} finally { await input.evidence("pi-control-checks.json", { issue, runs, checks, samples, remoteObservations, completed }); }
}
tests/runner-e2e/pi-controls-test-fixture.ts
+64
View File
@@ -0,0 +1,64 @@
import { CodexSessionState } from "../../packages/paperclip-runner/src/drivers/codex/codex-session-state.js";
import { mapTerminalTurn } from "../../packages/paperclip-runner/src/drivers/codex/codex-session-terminal.js";
import { observePiControlPending } from "./pi-controls-evidence.js";
import { readActiveStopCaller } from "./native-active-stop-evidence.js";
type Row = Record<string, any>;
export const piControlCaller = readActiveStopCaller({ deploymentMode: "local_trusted" }, { session: { userId: "local-board", id: "paperclip:local_implicit:local-board" } });
export const piCancellationId = "11111111-2222-4333-8444-555555555555";
export function piControlFixture(target = "target.txt") {
const scope = { companyId: "company", issueId: "issue", runId: "run", target };
const events: Row[] = [];
const row = (eventType: string, payload: Row, extra: Row = {}) => {
const seq = events.length + 1;
return { companyId: "company", runId: "run", seq, eventType, protocolSchemaVersion: 1,
payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", eventType, runId: "run", turnId: "turn", normalizedSessionId: "session", sourceInstanceId: "source", sourceSeq: seq,
sourceEventId: `source:run:${seq}`, emittedAt: "2026-10-01T00:00:00Z", payload, ...extra } } };
};
const append = (eventType: string, payload: Row, extra: Row = {}) => events.push(row(eventType, payload, extra));
const tool = { schema: "paperclip.tool.execution.v1", executionId: "pi-tool", transport: "builtin", name: "write", operation: "edit", target, status: "running" };
append("tool.execution.started", tool);
const request = { schema: "paperclip.runtime_request.v2", type: "permission", requestKind: "permission_approval", method: "session/request_permission", status: "pending", requestId: "request", turnId: "turn", itemId: "permission-item",
origin: { adapter: "acpx-runtime-sidecar", provider: "pi", method: "session/request_permission" }, details: { toolCallId: "pi-tool" }, choices: [{ key: "decline", label: "Deny" }] };
append("runtime_request.created", { request });
const run: Row = { id: "run", companyId: "company", nativeIssueId: "issue", runtimeMode: "native", status: "running", resultJson: {}, processPid: 123, processGroupId: 123, processStartedAt: "2026-10-01T00:00:00Z" };
const issue: Row = { id: "issue", companyId: "company", identifier: "PI-1", status: "in_progress" };
const state = () => ({ events, run, issue });
const pending = () => observePiControlPending({ ...state(), scope });
const responses: unknown[] = [];
function cancel(cancellationRequestId = piCancellationId) {
// Exercise the real Product terminal mapper and callback cancellation, not
// just a fabricated terminal whose payload mirrors the matcher.
const session = {
terminalTurns: new Map(), workspaceChangesByTurn: new Map(), result: null, conversationMode: "direct", activeTurnId: "turn", turnStarted: true,
pendingRuntimeRequestMap: new Map([[request.requestId, { request, settle: (value: unknown) => responses.push(value) }]]),
cancelPendingRequests: CodexSessionState.prototype.cancelPendingRequests,
emit: (eventType: string, payload: Row) => append(eventType, payload),
} as unknown as CodexSessionState;
mapTerminalTurn(session, { id: "turn", status: "cancelled", error: null }, "turn");
run.status = "cancelled";
run.resultJson = { startupCancellation: { cancellationRequestId, requestedBy: { type: "board", userId: "local-board" } }, nativeCancellation: {
schema: "paperclip.native-cancellation.v1", intentId: `native-cancellation:${cancellationRequestId}`, companyId: "company", runId: "run", issueId: "issue", scope: "run",
dispatched: true, dispatchState: "acknowledged", reasonCode: "cancellation_run_only", effects: ["release_run_resources"], intentAuditId: "intent-audit", acknowledgementAuditId: "ack-audit",
} };
return run;
}
const marker = `PI-STEER-${"a".repeat(32)}`, commentId = "comment", queueId = "queue";
function steer() {
run.resultJson.queuedSteeringAcknowledgements = { [commentId]: { status: "acknowledged", queueId, turnId: "turn", acknowledgedAt: "2026-10-01T00:00:01Z" } };
append("item.completed", { kind: "steering_acknowledgement", status: "acknowledged", mode: "steer", text: "Steering acknowledged for the active turn." }, { itemId: `turn:steer:${commentId}` });
}
function finish(finalMarker = marker) {
append("runtime_request.resolved", { requestId: "request", turnId: "turn", action: "decline" });
append("tool.execution.completed", { ...tool, status: "failed", output: "Pi operation was denied or cancelled" });
append("turn.completed", { status: "completed", error: null });
run.status = "succeeded"; issue.status = "done";
append("run.result.accepted", { result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary: finalMarker } }, {
sourceKind: "control_plane", sourceInstanceId: "source:control", sourceSeq: 1, sourceEventId: "source:control:run:1",
});
append("run.terminal", { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState: "completed", reportedWorkDisposition: "done" }, {
sourceKind: "control_plane", sourceInstanceId: "source:control", sourceSeq: 2, sourceEventId: "source:control:run:2",
});
}
return { scope, events, run, issue, state, pending, row, append, request, tool, cancel, responses, steer, finish, marker, commentId, queueId };
}
tests/runner-e2e/pi-file-evidence.test.ts
+274
View File
@@ -0,0 +1,274 @@
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { createHash } from "node:crypto";
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { describe, expect, it, vi } from "vitest";
import { collectPiFileEvidence, gradePiCopyback, gradePiFileEvidence, piFileContract, piFilePrompt, seedPiFile } from "./pi-file-evidence.js";
import { canonicalProviderEventsFromAcpxRuntimeEvent, createAcpxToolEventNormalizer } from "../../packages/paperclip-runner/src/provider-events.js";
import { safeAcpxLocations } from "../../packages/paperclip-runner/src/drivers/acpx/safe-locations.js";
import { classifyFailure } from "./failure-classifier.js";
import { runnerMatrix } from "./catalog.js";
function fixture() {
const c = piFileContract("fixture"), attachmentId = "12345678-1234-1234-1234-123456789abc";
const normalize = createAcpxToolEventNormalizer();
const tool = (seq: number, executionId: string, name: string, operation: string,
phase: "start" | "progress" | "end", target: string | null, text = "", rawOutput: unknown = undefined) => {
// Sanitized shape of the retained Pi stream: missing opening location,
// later resolved edit arguments, and stable bash name with command progress.
// This exercises TS normalization/projection, not the Rust implementation.
const event = normalize({ type: "tool_call", toolCallId: executionId,
tag: phase === "start" ? "tool_call" : "tool_call_update", title: name, kind: operation,
status: phase === "end" ? "completed" : phase === "start" ? "pending" : "in_progress",
locations: safeAcpxLocations(target ? [{ path: `/workspace/${target}` }] : [], "/workspace", operation, name),
text, rawOutput });
const canonical = canonicalProviderEventsFromAcpxRuntimeEvent(event, executionId, "turn")[0]!;
return { companyId: "company", runId: "run", seq, protocolSchemaVersion: 1, eventType: canonical.eventType,
sourceInstanceId: "runner", sourceSeq: seq, sourceEventId: `runner:run:${seq}`,
payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", sourceInstanceId: "runner",
sourceSeq: seq, sourceEventId: `runner:run:${seq}`, runId: "run", turnId: "turn", normalizedSessionId: "session", ...canonical } } };
};
return {
environment: "local" as const, environmentId: "environment", nonce: "fixture", companyId: "company", issueId: "issue", agentId: "agent", attachmentId,
seed: { schema: "paperclip.e2e.pi-file-seed.v1", filename: c.filename,
beforeSha256: createHash("sha256").update("ready-fixture\n").digest("hex"), byteSize: Buffer.byteLength(c.before) },
workspaceBytes: Buffer.from(c.after), downloadedBytes: Buffer.from(c.after),
run: { id: "run", companyId: "company", agentId: "agent", runtimeMode: "native", nativeSessionId: "session", runnerInstanceId: "runner", status: "succeeded", resultJson: { semanticToolReceipts: {
publish: { operationId: "register_deliverable", input: { filename: c.filename, contentRef: c.filename, contentType: "text/plain", byteSize: c.byteSize, sha256: c.sha256 },
result: { disposition: "applied", commandId: `deliverable-prepared:${attachmentId}`, entityRefs: [attachmentId, "product", "comment"] } },
} } },
events: [tool(1, "edit", "edit", "edit", "start", null),
tool(2, "edit", "edit", "edit", "progress", c.filename),
tool(3, "edit", "edit", "edit", "end", c.filename),
tool(4, "validate", "bash", "execute", "start", null, "bash (pending): [terminal] validate"),
tool(5, "validate", c.validationCommand, "execute", "progress", null, `${c.validationCommand} (in_progress): ${c.validationCommand}`),
tool(6, "validate", c.validationCommand, "execute", "end", null, "", {
content: [{ type: "text", text: `${c.validationMarker}\n` }],
structuredContent: { output: `${c.validationMarker}\n`, truncated: false, exit_code: 0, wall_time_seconds: 0 },
})],
attachments: [{ id: attachmentId, companyId: "company", issueId: "issue", originatingRunId: "run", createdByAgentId: "agent",
originalFilename: c.filename, contentType: "text/plain", byteSize: c.byteSize, sha256: c.sha256 }],
activity: [{ action: "issue.attachment_added", companyId: "company", runId: "run", actorId: "agent", entityId: "issue",
details: { attachmentId, source: "paperclip_runner_protocol" } }],
};
}
function changeOutput(f: ReturnType<typeof fixture>, change: (output: any) => void) {
const p = f.events[5]!.payload.prpEvent.payload, output = JSON.parse(p.output as string);
change(output); p.output = JSON.stringify(output);
}
describe("Pi edit, validation and public artifact oracle", () => {
it("calibrates against the sanitized 44-row actual native stream without regrading the paid attempt", async () => {
const retained = JSON.parse(await readFile(new URL("./fixtures/pi-file-evidence-actual-stream.json", import.meta.url), "utf8"));
const f = fixture();
expect(retained.events).toHaveLength(44);
f.events = retained.events.map((x: any) => ({ companyId: "company", runId: "run", seq: x.seq,
protocolSchemaVersion: 1, eventType: x.eventType, sourceInstanceId: "runner", sourceSeq: x.sourceSeq,
sourceEventId: `runner:run:${x.sourceSeq}`, payload: { prpEvent: {
schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", sourceInstanceId: "runner",
sourceSeq: x.sourceSeq, sourceEventId: `runner:run:${x.sourceSeq}`, runId: "run", turnId: "turn",
normalizedSessionId: "session", eventType: x.eventType, payload: x.payload,
} } }));
expect(gradePiFileEvidence(f).verification).toMatchObject({ nativeStructuredExitCode: 0, typedExitCode: null });
});
it("accepts actual correlated lifecycles, independent bytes and a run-bound registered download", () => {
const result = gradePiFileEvidence(fixture());
expect(result.passed).toBe(true);
expect(result.diff.text).toContain("-ready-fixture\n+verified-fixture\n");
expect(result.limits.join(" ")).toContain("not attested");
expect(result.verification.providerExecutionStatus).toBe("completed");
expect(result.verification.command).toBe(piFileContract("fixture").validationCommand);
expect(result.verification).toMatchObject({ commandEvidence: "correlated_native_progress",
exitEvidence: "native_structured_output_receipt", nativeStructuredExitCode: 0, typedExitCode: null });
expect(fixture().events.map(e => e.payload.prpEvent.payload.target)).toEqual([null, "extended-fixture.txt", "extended-fixture.txt", null, null, null]);
expect(fixture().events.slice(3).every(e => e.payload.prpEvent.payload.name === "bash")).toBe(true);
expect(result.verification.nativeFileAttribution).toBe("workspace_relative_display_target");
});
it.each(["before", "after"])("rejects an extra metadata bash call %s validation despite correct file and download bytes", placement => {
const f = fixture();
const metadata = structuredClone(f.events.slice(3));
for (const row of metadata) row.payload.prpEvent.payload.executionId = "metadata";
metadata[1]!.payload.prpEvent.payload.progress = "wc -c extended-fixture.txt (in_progress): wc -c extended-fixture.txt";
metadata[2]!.payload.prpEvent.payload.output = JSON.stringify({
content: [{ type: "text", text: "17 extended-fixture.txt\n" }],
structuredContent: { output: "17 extended-fixture.txt\n", truncated: false, exit_code: 0 },
});
if (placement === "before") f.events.splice(3, 0, ...metadata);
else f.events.push(...metadata);
for (const [index, row] of f.events.entries()) {
row.seq = row.sourceSeq = row.payload.prpEvent.sourceSeq = index + 1;
row.sourceEventId = row.payload.prpEvent.sourceEventId = `runner:run:${index + 1}`;
}
expect(() => gradePiFileEvidence(f)).toThrow("one observed provider validation execution");
});
const mutations: Array<[string, (f: ReturnType<typeof fixture>) => void]> = [
["absent seed", f => { f.seed = {} as any; }],
["wrong final bytes", f => { f.workspaceBytes = Buffer.from("wrong\n"); }],
["final-only write", f => { f.events[0]!.payload.prpEvent.payload.name = "write"; f.events[2]!.payload.prpEvent.payload.name = "write"; }],
["missing edit", f => { f.events = f.events.slice(3); }],
["missing validation", f => { f.events = f.events.slice(0, 3); }],
["echo-only validation", f => { f.events[3]!.payload.prpEvent.payload.name = `echo ${piFileContract("fixture").validationMarker}`; f.events[5]!.payload.prpEvent.payload.name = f.events[3]!.payload.prpEvent.payload.name; }],
["failed validation", f => { f.events[5]!.payload.prpEvent.payload.status = "failed"; }],
["unfinished validation", f => { f.events.pop(); }],
["unrelated validation turn", f => { f.events[3]!.payload.prpEvent.turnId = "other"; f.events[5]!.payload.prpEvent.turnId = "other"; }],
["foreign native session", f => { f.run.nativeSessionId = "other"; }],
["foreign native producer", f => { f.run.runnerInstanceId = "other"; }],
["foreign event run", f => { f.events[2]!.payload.prpEvent.runId = "other"; }],
["duplicate event", f => { f.events.push(structuredClone(f.events[5]!)); }],
["missing tool start", f => { f.events.splice(3, 1); }],
["truncated validation output", f => { f.events[5]!.payload.prpEvent.payload.outputTruncated = true; }],
["missing native target", f => { f.events[0]!.payload.prpEvent.payload.target = null; f.events[2]!.payload.prpEvent.payload.target = null; }],
["wrong target", f => { f.events[2]!.payload.prpEvent.payload.target = "other.txt"; }],
["missing registration", f => { f.run.resultJson.semanticToolReceipts = {} as any; }],
["rejected registration", f => { f.run.resultJson.semanticToolReceipts.publish.result.disposition = "denied"; }],
["wrong registered hash", f => { f.run.resultJson.semanticToolReceipts.publish.input.sha256 = "wrong"; }],
["wrong registered entity", f => { f.run.resultJson.semanticToolReceipts.publish.result.entityRefs[0] = "other"; }],
["missing attachment", f => { f.attachments = []; }],
["foreign attachment company", f => { f.attachments[0]!.companyId = "other"; }],
["stale attachment run", f => { f.attachments[0]!.originatingRunId = "older"; }],
["wrong download bytes", f => { f.downloadedBytes = Buffer.from("wrong\n"); }],
["missing publication activity", f => { f.activity = []; }],
["foreign publication run", f => { f.activity[0]!.runId = "other"; }],
["conflicting resolved target", f => { f.events[1]!.payload.prpEvent.payload.target = "other.txt"; }],
["target regresses after resolution", f => { f.events[0]!.payload.prpEvent.payload.target = piFileContract("fixture").filename; f.events[1]!.payload.prpEvent.payload.target = null; }],
["missing terminal target", f => { f.events[2]!.payload.prpEvent.payload.target = null; }],
["absent command proof", f => { f.events[4]!.payload.prpEvent.payload.progress = null; }],
["echo-only command proof", f => { f.events[4]!.payload.prpEvent.payload.progress = "echo PI-VALIDATED-fixture (in_progress): echo PI-VALIDATED-fixture"; }],
["conflicting command before exact proof", f => { f.events[3]!.payload.prpEvent.payload.progress = "node other.js (pending): node other.js"; }],
["late command proof", f => { f.events[5]!.payload.prpEvent.payload.progress = f.events[4]!.payload.prpEvent.payload.progress; f.events[4]!.payload.prpEvent.payload.progress = null; }],
["plain echoed marker output", f => { f.events[5]!.payload.prpEvent.payload.output = piFileContract("fixture").validationMarker; }],
["nonzero native exit", f => { changeOutput(f, o => { o.structuredContent.exit_code = 1; }); }],
["missing native exit", f => { changeOutput(f, o => { delete o.structuredContent.exit_code; }); }],
["truncated native output", f => { changeOutput(f, o => { o.structuredContent.truncated = true; }); }],
["false marker substring", f => { changeOutput(f, o => { o.structuredContent.output = "failed before PI-VALIDATED-fixture\n"; }); }],
["contradictory content", f => { changeOutput(f, o => { o.content[0].text = "failed"; }); }],
["contradictory typed exit", f => { f.events[5]!.payload.prpEvent.payload.exitCode = 1; }],
["foreign command proof execution", f => { f.events[4]!.payload.prpEvent.payload.executionId = "foreign"; }],
["foreign command proof turn", f => { f.events[4]!.payload.prpEvent.turnId = "foreign"; }],
["foreign command proof session", f => { f.events[4]!.payload.prpEvent.normalizedSessionId = "foreign"; }],
["foreign command proof producer", f => {
const row = f.events[4]!, e = row.payload.prpEvent;
e.sourceInstanceId = row.sourceInstanceId = "foreign"; e.sourceEventId = row.sourceEventId = `foreign:run:${e.sourceSeq}`;
}],
["durable source differs from envelope", f => { f.events[4]!.sourceInstanceId = "foreign"; }],
["durable event identity differs from envelope", f => { f.events[4]!.sourceEventId = "runner:run:999"; }],
["durable source sequence differs from envelope", f => { f.events[4]!.sourceSeq = 999; }],
["command proof after terminal", f => {
const e = f.events[4]!; f.events.splice(4, 1); e.seq = 7; e.payload.prpEvent.sourceSeq = 7;
e.sourceSeq = 7; e.sourceEventId = e.payload.prpEvent.sourceEventId = "runner:run:7"; f.events.push(e);
}],
];
it.each(mutations)("rejects %s even when the model claims completion", (_name, change) => {
const f = fixture(); change(f); expect(() => gradePiFileEvidence(f)).toThrow("Pi file evidence:");
try { gradePiFileEvidence(f); } catch (error) { expect(classifyFailure(error)).toBe("candidate_failure"); }
});
it("keeps real nonce commands within the Rust 240-character name contract", () => {
const c = piFileContract("0123456789ab-1");
expect([...c.validationCommand].length).toBeLessThanOrEqual(240);
expect(safeAcpxLocations([{ path: `/workspace/${c.filename}` }], "/workspace", "edit", "edit")[0]).toMatchObject({
path: c.filename, pathBoundary: "paperclip.workspace_relative_display.v2",
});
expect(() => piFileContract("a".repeat(100))).toThrow("native Rust tool-name bound");
});
it("runs the exact fixture validation command and rejects plausible wrong bytes", async () => {
const root = await mkdtemp(join(tmpdir(), "pi-validation-command-"));
const c = piFileContract("0123456789ab-1");
const validate = () => promisify(execFile)("/bin/sh", ["-c", c.validationCommand], {
cwd: root, timeout: 3_000, maxBuffer: 4096, env: { PATH: dirname(process.execPath) },
});
try {
await writeFile(join(root, c.filename), c.after);
expect((await validate()).stdout).toBe(`${c.validationMarker}\n`);
await writeFile(join(root, c.filename), c.after.trimEnd());
await expect(validate()).rejects.toMatchObject({ code: 1 });
await rm(join(root, c.filename));
await expect(validate()).rejects.toMatchObject({ code: 1 });
} finally { await rm(root, { recursive: true, force: true }); }
});
it("pastes the literal validation command as Markdown code instead of escaped paragraph text", () => {
const command = piFileContract("0123456789ab-1").validationCommand;
const fenced = piFilePrompt("0123456789ab-1").match(/\n```bash\n([^\n]+)\n```\n/u);
expect(fenced?.[1]).toBe(command);
expect(fenced?.[1]).toContain("!==");
expect(fenced?.[1]).not.toContain("!\\=\\=");
});
it("changes only Pi file prompts and keeps all four question methods", () => {
const cells = runnerMatrix.filter(c => c.suite.id === "extended-harnesses" && c.task.id === "file-edit-validate");
expect(cells).toHaveLength(6);
for (const cell of cells) {
const prompt = cell.task.buildPrompt("fixture");
expect(prompt.includes("register_deliverable")).toBe(cell.profile.qualificationCandidate === "pi");
if (cell.profile.qualificationCandidate === "pi") {
const artifact = cell.task.buildMatchers("fixture", cell).find(matcher => matcher.kind === "artifact_exact");
if (!artifact || artifact.kind !== "artifact_exact") throw new Error("Pi file task must require a registered artifact");
expect(artifact.name).toBe(piFileContract("fixture").filename);
expect(prompt).toContain(`title ${artifact.name},`);
}
}
expect(runnerMatrix.filter(c => c.profile.qualificationCandidate === "pi")).toHaveLength(26);
});
it.each(["local", "daytona"] as const)("seeds once and collects public %s evidence", async environment => {
const root = await mkdtemp(join(tmpdir(), "pi-file-evidence-"));
try {
const seed = await seedPiFile(root, "fixture"), f = environment === "daytona" ? copyback() : fixture(), c = piFileContract("fixture");
expect(await readFile(join(root, c.filename), "utf8")).toBe(c.before);
await expect(seedPiFile(root, "fixture")).rejects.toThrow();
await writeFile(join(root, c.filename), c.after);
const get = vi.fn(async (path: string) => {
if (path === "/api/heartbeat-runs/run") return f.run;
if (path === "/api/environment-leases/lease" && "lease" in f) return f.lease;
if (path.endsWith("/attachments")) return f.attachments;
if (path.endsWith("/activity")) return f.activity;
throw new Error("Unexpected route");
});
const download = vi.fn(async () => ({ ok: () => true, body: async () => f.downloadedBytes }));
const visible = vi.fn(async () => {}), evidence = vi.fn(async () => {});
const proof = await collectPiFileEvidence({ ...f, seed, workspace: root, evidence,
api: { get, request: { get: download } } as any,
page: { locator: () => ({ first: () => ({ waitFor: visible }) }) } as any });
expect(proof.passed).toBe(true);
if (environment === "daytona") {
expect(get).toHaveBeenCalledWith("/api/heartbeat-runs/run");
expect(get).toHaveBeenCalledWith("/api/environment-leases/lease");
expect(proof.workspaceProvenance.kind).toBe("product_copyback");
}
expect(visible).toHaveBeenCalledOnce();
expect(evidence).toHaveBeenCalledWith(expect.objectContaining({ workspaceBytes: f.workspaceBytes.toString("base64"), downloadedBytes: f.downloadedBytes.toString("base64") }));
expect(download).toHaveBeenCalledWith(`/api/attachments/${f.attachmentId}/content`);
} finally { await rm(root, { recursive: true, force: true }); }
});
});
function copyback() {
const f = fixture();
const reference = { schema: "paperclip.native-workspace-sync/v2", state: "finalized", baselineSha256: "a".repeat(64),
descriptorSha256: "b".repeat(64), finalHostSha256: "c".repeat(64), workspaceId: "workspace", leaseId: "lease",
providerLeaseId: "sandbox", remoteCwd: "/workspace" };
const run = { ...f.run, contextSnapshot: { executionWorkspaceId: "workspace", paperclipEnvironment: { id: "environment", driver: "sandbox", leaseId: "lease" } },
runnerProfileJson: { nativeWorkspaceSync: reference, nativeExecutionInput: { binding: { executionWorkspaceId: "workspace" } } } };
const lease = { id: "lease", companyId: "company", environmentId: "environment", heartbeatRunId: "run", provider: "daytona", providerLeaseId: "sandbox", metadata: { remoteCwd: "/workspace" } };
return { ...f, run, lease, environment: "daytona" as const };
}
it("accepts public finalized same-run copy-back attribution without claiming guest observation", () => {
const f = copyback();
expect(gradePiFileEvidence(f).workspaceProvenance).toMatchObject({ kind: "product_copyback", guestObservation: false, hostSnapshotRecomputed: false });
});
it.each([
["missing reference", f => { f.run.runnerProfileJson.nativeWorkspaceSync = {} as any; }],
["prepared only", f => { f.run.runnerProfileJson.nativeWorkspaceSync.state = "prepared"; }],
["missing baseline", f => { f.run.runnerProfileJson.nativeWorkspaceSync.baselineSha256 = ""; }],
["invalid final hash", f => { f.run.runnerProfileJson.nativeWorkspaceSync.finalHostSha256 = "wrong"; }],
["foreign workspace", f => { f.run.contextSnapshot.executionWorkspaceId = "other"; }],
["contradictory native binding", f => { f.run.runnerProfileJson.nativeExecutionInput.binding.executionWorkspaceId = "other"; }],
["foreign lease", f => { f.lease.id = "other"; }],
["foreign run", f => { f.lease.heartbeatRunId = "other"; }],
["foreign company", f => { f.lease.companyId = "other"; }],
["foreign environment", f => { f.lease.environmentId = "other"; }],
["foreign provider lease", f => { f.lease.providerLeaseId = "other"; }],
["wrong remote root", f => { f.lease.metadata.remoteCwd = "/other"; }],
["wrong provider", f => { f.lease.provider = "other"; }],
] satisfies Array<[string, (f: ReturnType<typeof copyback>) => void]>)("rejects %s copy-back provenance", (_label, change) => {
const f = copyback(); change(f);
expect(() => gradePiCopyback(f.run, f.lease, f.companyId, f.environmentId)).toThrow("Pi file evidence:");
});
tests/runner-e2e/pi-file-evidence.ts
+222
View File
@@ -0,0 +1,222 @@
import { createHash } from "node:crypto";
import { readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import type { Page } from "@playwright/test";
import type { RunnerApi } from "./api.js";
import { isValidNativePrpEnvelope } from "./native-event-envelope.js";
type Row = Record<string, any>;
const rec = (value: unknown): Row => value !== null && typeof value === "object" && !Array.isArray(value) ? value as Row : {};
const sha = (value: string | Buffer) => createHash("sha256").update(value).digest("hex");
const requireEvidence = (value: unknown, reason: string) => { if (!value) throw new Error(`Pi file evidence: ${reason}`); };
export function piFileContract(nonce: string) {
if (!/^[A-Za-z0-9-]{1,100}$/u.test(nonce)) throw new Error("Invalid file fixture nonce");
const filename = `extended-${nonce}.txt`, before = `ready-${nonce}\n`, after = `verified-${nonce}\n`;
const validationMarker = `PI-VALIDATED-${nonce}`;
const validationCommand = `node -e 'if(require("node:fs").readFileSync(${JSON.stringify(filename)},"utf8")!==${JSON.stringify(after)})throw Error("mismatch");console.log(${JSON.stringify(validationMarker)})'`;
if ([...validationCommand].length > 240) throw new Error("Validation command exceeds native Rust tool-name bound");
return { filename, before, after, sha256: sha(after), byteSize: Buffer.byteLength(after), validationMarker, validationCommand };
}
export function piFilePrompt(nonce: string) {
const c = piFileContract(nonce);
return [
`The fixture has already created ${c.filename} in your execution workspace with exactly ${JSON.stringify(c.before)}. Read it, then use your native edit tool once to replace those bytes with exactly ${JSON.stringify(c.after)}. Do not substitute write, bash, or an API for the edit.`,
`Run this exact native bash command exactly once, without an added shell prefix or suffix:\n\`\`\`bash\n${c.validationCommand}\n\`\`\`\nIt independently reads the file, fails on incorrect bytes, and prints the validation marker only after success. This is the only bash call allowed for this task; do not run any other shell commands.`,
`Deliver the edited text file as a downloadable attachment using the assigned register_deliverable semantic tool, with filename and workspace-relative contentRef ${c.filename}, contentType text/plain, byteSize ${c.byteSize}, sha256 ${c.sha256}, title ${c.filename}, and a fresh idempotencyKey. Use the supplied byteSize and sha256 for the exact validated post-edit bytes; do not run extra commands to obtain metadata. Wait for the real registration receipt. Do not upload a different file or claim publication from a disk path.`,
`After successful validation and registration, call paperclip_finish with reportedWorkDisposition done, summary EXTENDED-FILE-${nonce}, the current completion contract revision and satisfied objective criterion, no remaining work, the actual validation command/result, and the registered artifact reference.`,
`Wait for paperclip_finish to succeed, then emit exactly EXTENDED-FILE-${nonce} as your final response. Do not create unrelated files or work.`,
].join("\n");
}
export async function seedPiFile(workspace: string, nonce: string) {
const c = piFileContract(nonce);
await writeFile(join(workspace, c.filename), c.before, { flag: "wx" });
const bytes = await readFile(join(workspace, c.filename));
requireEvidence(bytes.equals(Buffer.from(c.before)), "seed bytes changed before provider startup");
return { schema: "paperclip.e2e.pi-file-seed.v1", filename: c.filename, beforeSha256: sha(bytes), byteSize: bytes.length };
}
export function gradePiCopyback(run: Row, lease: Row, companyId: string, environmentId: string) {
const context = rec(run.contextSnapshot), environment = rec(context.paperclipEnvironment);
const reference = rec(rec(run.runnerProfileJson).nativeWorkspaceSync);
const binding = rec(rec(rec(run.runnerProfileJson).nativeExecutionInput).binding);
requireEvidence(run.companyId === companyId && reference.schema === "paperclip.native-workspace-sync/v2"
&& reference.state === "finalized" && [reference.baselineSha256, reference.descriptorSha256, reference.finalHostSha256]
.every(value => typeof value === "string" && /^[a-f0-9]{64}$/u.test(value)), "finalized production copy-back hashes required");
requireEvidence(typeof reference.workspaceId === "string" && reference.workspaceId.length > 0
&& reference.workspaceId === context.executionWorkspaceId
&& (binding.executionWorkspaceId === undefined || binding.executionWorkspaceId === reference.workspaceId)
&& environment.id === environmentId && environment.driver === "sandbox" && environment.leaseId === reference.leaseId,
"copy-back workspace and environment must belong to the actual run");
requireEvidence(lease.id === reference.leaseId && lease.companyId === companyId && lease.environmentId === environmentId
&& lease.heartbeatRunId === run.id && lease.provider === "daytona" && typeof reference.providerLeaseId === "string"
&& reference.providerLeaseId.length > 0 && lease.providerLeaseId === reference.providerLeaseId
&& typeof reference.remoteCwd === "string" && reference.remoteCwd.startsWith("/")
&& reference.remoteCwd === rec(lease.metadata).remoteCwd, "copy-back must bind the exact run and provider lease");
return { kind: "product_copyback", reference, lease: { id: lease.id, companyId, environmentId, heartbeatRunId: run.id,
provider: lease.provider, providerLeaseId: lease.providerLeaseId, remoteCwd: reference.remoteCwd },
guestObservation: false, hostSnapshotRecomputed: false };
}
/** Only the common public projection is authority here. Native raw arguments,
* full diff blocks and typed command exit codes are deliberately not invented.
* The native structured output receipt is distinct from that nullable typed field. */
export function gradePiFileEvidence(input: {
nonce: string; companyId: string; issueId: string; agentId: string; run: Row;
environment: "local" | "daytona"; environmentId: string; lease?: Row;
events: unknown[]; seed: unknown; workspaceBytes: Buffer; attachments: unknown[];
attachmentId: string; downloadedBytes: Buffer; activity: unknown[];
}) {
const c = piFileContract(input.nonce), run = input.run, seed = rec(input.seed);
const workspaceProvenance = input.environment === "daytona"
? gradePiCopyback(run, rec(input.lease), input.companyId, input.environmentId)
: { kind: "independent_local_workspace" };
requireEvidence(run.id && run.companyId === input.companyId && run.agentId === input.agentId
&& run.runtimeMode === "native" && run.status === "succeeded", "actual native run identity");
requireEvidence(seed.schema === "paperclip.e2e.pi-file-seed.v1" && seed.filename === c.filename
&& seed.beforeSha256 === sha(c.before) && seed.byteSize === Buffer.byteLength(c.before), "independent pre-edit seed");
requireEvidence(input.workspaceBytes.equals(Buffer.from(c.after)), "independent final workspace bytes");
requireEvidence(input.events.length > 0 && input.events.length <= 20_000, "bounded run events");
const rows = input.events.map(rec).filter(row => rec(row.payload).prpEvent !== undefined);
const seen = new Set<string>(), seqs = new Set<number>(), sourceSeqs = new Map<string, number>();
const canonical = rows.map(row => {
const e = rec(rec(row.payload).prpEvent);
requireEvidence(row.companyId === input.companyId && row.runId === run.id && e.runId === run.id
&& isValidNativePrpEnvelope(e, row.protocolSchemaVersion) && row.eventType === e.eventType
&& Number.isSafeInteger(row.seq) && row.seq > 0 && !seqs.has(row.seq)
&& typeof e.sourceInstanceId === "string" && e.sourceInstanceId.length > 0
&& Number.isSafeInteger(e.sourceSeq) && e.sourceSeq > (sourceSeqs.get(e.sourceInstanceId) ?? 0)
&& row.sourceInstanceId === e.sourceInstanceId && row.sourceSeq === e.sourceSeq && row.sourceEventId === e.sourceEventId
&& e.sourceEventId === `${e.sourceInstanceId}:${e.runId}:${e.sourceSeq}` && !seen.has(e.sourceEventId), "foreign, duplicated, or reordered durable evidence");
seqs.add(row.seq); seen.add(e.sourceEventId); sourceSeqs.set(e.sourceInstanceId, e.sourceSeq);
return { row, e, p: rec(e.payload) };
});
const tools = canonical.filter(x => x.e.eventType.startsWith("tool.execution.") && x.p.transport === "builtin");
const edits = tools.filter(x => x.e.eventType === "tool.execution.completed" && x.p.operation === "edit");
requireEvidence(edits.length === 1, "one native edit required; final-only writes cannot qualify");
const edit = edits[0]!;
requireEvidence(edit.p.name === "edit" && edit.p.target === c.filename, "exact native edit and relative target required");
function lifecycle(end: typeof edit) {
const matches = tools.filter(x => x.p.executionId === end.p.executionId);
const starts = matches.filter(x => x.e.eventType === "tool.execution.started");
requireEvidence(typeof end.p.executionId === "string" && end.p.executionId.length > 0
&& end.p.schema === "paperclip.tool.execution.v1" && end.p.status === "completed"
&& end.e.sourceKind === "runner" && end.e.sourceInstanceId === run.runnerInstanceId
&& end.e.normalizedSessionId === run.nativeSessionId && typeof end.e.turnId === "string" && end.e.turnId.length > 0
&& typeof end.e.normalizedSessionId === "string" && end.e.normalizedSessionId.length > 0
&& starts.length === 1 && starts[0]!.row.seq < end.row.seq
&& matches.filter(x => x.e.eventType === "tool.execution.completed").length === 1
&& matches.every(x => x.p.schema === end.p.schema && x.p.name === end.p.name && x.p.operation === end.p.operation
&& x.row.seq >= starts[0]!.row.seq && x.row.seq <= end.row.seq
&& x.e.sourceKind === "runner" && x.e.turnId === end.e.turnId
&& x.e.normalizedSessionId === end.e.normalizedSessionId && x.e.sourceInstanceId === end.e.sourceInstanceId
&& (x === end || (x.p.status === "running"
&& ["tool.execution.started", "tool.execution.progressed"].includes(x.e.eventType)))), "incomplete or unsuccessful native tool lifecycle");
// Native arguments arrive incrementally. An unresolved target is allowed
// only before the first exact target; conflicting or regressed targets fail.
let resolved = false;
for (const x of matches) {
requireEvidence(x.p.target === end.p.target || (!resolved && x.p.target === null), "conflicting native tool target");
if (x.p.target !== null) resolved = true;
}
return { start: starts[0]!, matches };
}
lifecycle(edit);
const validations = tools.filter(x => x.e.eventType === "tool.execution.completed" && x.p.operation === "execute"
&& x.p.name === "bash");
requireEvidence(validations.length === 1, "one observed provider validation execution");
const validation = validations[0]!, { start, matches } = lifecycle(validation);
requireEvidence(start.row.seq > edit.row.seq && validation.e.turnId === edit.e.turnId
&& validation.e.normalizedSessionId === edit.e.normalizedSessionId
&& validation.e.sourceInstanceId === edit.e.sourceInstanceId && validation.p.outputTruncated === false,
"validation must follow the edit in the same native turn");
// The common lifecycle preserves the opening name (bash). ACPX's resolved
// command is retained as progress, rather than replacing that stable name.
const commandProgress = `${c.validationCommand} (in_progress): ${c.validationCommand}`;
const allowedProgress = new Set([`bash (pending): [terminal] ${validation.p.executionId}`,
`${c.validationCommand} (pending): ${c.validationCommand}`, commandProgress]);
requireEvidence(matches.some(x => x !== validation && x.p.progress === commandProgress)
&& matches.every(x => x.p.progress === null || allowedProgress.has(x.p.progress)),
"exact correlated command progress before completion required");
requireEvidence(typeof validation.p.output === "string" && Buffer.byteLength(validation.p.output) <= 16_384,
"bounded native validation receipt required");
let output: Row = {};
try { output = rec(JSON.parse(validation.p.output)); } catch { /* Rejected below; plain marker text is insufficient. */ }
const structured = rec(output.structuredContent), expectedOutput = `${c.validationMarker}\n`;
requireEvidence(structured.exit_code === 0 && structured.truncated === false && structured.output === expectedOutput
&& Array.isArray(output.content) && output.content.length === 1
&& rec(output.content[0]).type === "text" && rec(output.content[0]).text === expectedOutput
&& (validation.p.exitCode === null || validation.p.exitCode === 0), "successful native structured validation receipt required");
const attachments = input.attachments.map(rec).filter(a => a.id === input.attachmentId);
requireEvidence(attachments.length === 1, "one public attachment");
const a = attachments[0]!;
requireEvidence(a.companyId === input.companyId && a.issueId === input.issueId && a.originatingRunId === run.id
&& a.createdByAgentId === input.agentId && a.originalFilename === c.filename && a.contentType === "text/plain"
&& a.byteSize === c.byteSize && a.sha256 === c.sha256, "attachment bytes and immutable run attribution");
requireEvidence(input.downloadedBytes.equals(Buffer.from(c.after)), "actual public download bytes");
const receipts = Object.values(rec(rec(run.resultJson).semanticToolReceipts)).map(rec)
.filter(r => r.operationId === "register_deliverable" && rec(r.input).filename === c.filename);
requireEvidence(receipts.length === 1, "one authenticated semantic registration");
const receipt = receipts[0]!, args = rec(receipt.input), result = rec(receipt.result);
requireEvidence(args.contentRef === c.filename && args.contentType === "text/plain" && args.byteSize === c.byteSize
&& args.sha256 === c.sha256 && result.disposition === "applied" && result.commandId === `deliverable-prepared:${a.id}`
&& Array.isArray(result.entityRefs) && result.entityRefs[0] === a.id, "registered artifact receipt must match downloaded attachment");
const publications = input.activity.map(rec).filter(row => row.action === "issue.attachment_added"
&& rec(row.details).attachmentId === a.id);
requireEvidence(publications.length === 1 && publications[0]!.companyId === input.companyId
&& publications[0]!.runId === run.id && publications[0]!.actorId === input.agentId
&& publications[0]!.entityId === input.issueId && rec(publications[0]!.details).source === "paperclip_runner_protocol",
"publication activity must belong to this run and task");
return {
schema: "paperclip.e2e.pi-file-evidence.v1", passed: true, workspaceProvenance,
runId: run.id, turnId: edit.e.turnId, nativeEditExecutionId: edit.p.executionId,
validationExecutionId: validation.p.executionId, attachmentId: a.id,
verification: { kind: "independent_exact_bytes", beforeSha256: sha(c.before), afterSha256: c.sha256,
downloadedSha256: sha(input.downloadedBytes), byteSize: c.byteSize, providerExecutionStatus: validation.p.status,
command: c.validationCommand, providerToolName: validation.p.name, commandEvidence: "correlated_native_progress",
exitEvidence: "native_structured_output_receipt", nativeStructuredExitCode: structured.exit_code,
typedExitCode: validation.p.exitCode, nativeFileTarget: edit.p.target,
nativeFileAttribution: "workspace_relative_display_target" },
diff: { source: "independent_seed_and_workspace_bytes", path: c.filename,
text: `--- ${c.filename}\n+++ ${c.filename}\n@@ -1 +1 @@\n-${c.before.trimEnd()}\n+${c.after.trimEnd()}\n` },
limits: ["Exact command progress and the native structured output receipt are checked; raw arguments and nullable typed exitCode are not attested by those distinct fields.",
"Diff is computed from independently checked workspace bytes; it does not qualify native diff presentation or private invocation metadata."],
};
}
export async function collectPiFileEvidence(input: {
page: Page; api: RunnerApi; nonce: string; companyId: string; issueId: string; agentId: string;
run: Row; events: unknown[]; seed: unknown; workspace: string;
environment: "local" | "daytona"; environmentId: string;
evidence(data: unknown): Promise<void>;
}) {
const c = piFileContract(input.nonce);
let run = input.run, lease: Row | undefined;
if (input.environment === "daytona") {
run = await input.api.get<Row>(`/api/heartbeat-runs/${input.run.id}`);
requireEvidence(run.id === input.run.id, "public copy-back run readback changed identity");
const leaseId = rec(rec(run.runnerProfileJson).nativeWorkspaceSync).leaseId;
requireEvidence(typeof leaseId === "string" && /^[A-Za-z0-9_-]{1,128}$/u.test(leaseId), "copy-back lease reference missing");
lease = await input.api.get<Row>(`/api/environment-leases/${leaseId}`);
}
const attachments = await input.api.get<Row[]>(`/api/issues/${input.issueId}/attachments`);
const selected = attachments.filter(a => a.originalFilename === c.filename);
requireEvidence(selected.length === 1 && /^[a-f0-9-]{36}$/u.test(selected[0]!.id), "exact published file selection");
requireEvidence(selected[0]!.byteSize === c.byteSize && selected[0]!.sha256 === c.sha256
&& selected[0]!.contentType === "text/plain", "bounded attachment metadata before download");
const attachmentId = selected[0]!.id;
const contentPath = `/api/attachments/${attachmentId}/content`;
// Exercise the real task surface and the authenticated public download route.
await input.page.locator(`a[href*="${contentPath}"]`).first().waitFor({ state: "visible", timeout: 15_000 });
const response = await input.api.request.get(contentPath);
requireEvidence(response.ok(), "public attachment download failed");
const downloadedBytes = await response.body();
requireEvidence(downloadedBytes.length <= 4096, "bounded text attachment");
const workspaceBytes = await readFile(join(input.workspace, c.filename));
const activity = await input.api.get<Row[]>(`/api/issues/${input.issueId}/activity`);
await input.evidence({ schema: "paperclip.e2e.pi-file-observation.v1", attachments, attachmentId, activity,
workspaceBytes: workspaceBytes.toString("base64"), downloadedBytes: downloadedBytes.toString("base64"),
encoding: "base64", expectedSha256: c.sha256, copybackReference: rec(run.runnerProfileJson).nativeWorkspaceSync,
copybackLease: lease && { id: lease.id, companyId: lease.companyId, environmentId: lease.environmentId,
heartbeatRunId: lease.heartbeatRunId, provider: lease.provider, providerLeaseId: lease.providerLeaseId, remoteCwd: rec(lease.metadata).remoteCwd } });
return gradePiFileEvidence({ ...input, run, lease, attachments, attachmentId, downloadedBytes, workspaceBytes, activity });
}
tests/runner-e2e/pi-native-cases.test.ts
+119
View File
@@ -0,0 +1,119 @@
import { describe, expect, it } from "vitest";
import { gradePiNativeAnswers, gradePiNativeMemory, hasPiNativeMemoryRead, hasFailedPiWrite, hasPiCrossRootDenial, piNativeMemoryPrompt, piNativeTasks } from "./pi-native-cases.js";
import { runnerMatrix, runnerSuites } from "./catalog.js";
import { buildRunnerE2EProcessEnvironment } from "./harness-env.js";
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
describe("Pi native Product qualification", () => {
it("supplies native write arguments with one authoritative content string and its final LF", () => {
const nonce = "0123456789abcdef0123456789abcdef";
const prompt = piNativeMemoryPrompt(nonce, "/unassigned/denied.txt");
expect(prompt.split(nonce)).toHaveLength(2);
const encoded = /```json\n(.*?)\n```/s.exec(prompt)![1]!;
const args = JSON.parse(encoded);
expect(Object.keys(args)).toEqual(["path", "content"]);
expect(args.path).toBe("<AGENT_HOME>/memory/pi-native.txt");
expect(args.content).toBe(`${nonce}\n`);
expect(Buffer.byteLength(args.content)).toBe(33);
expect(Buffer.from(args.content).at(-1)).toBe(10);
expect(prompt).toContain("Use native read once, without offset or limit");
expect(prompt).toContain("Do not retry or work around it");
});
it("requires the current nonce and exactly one LF at every memory readback", () => {
const nonce = "0123456789abcdef0123456789abcdef";
expect(gradePiNativeMemory(`${nonce}\n`, nonce)).toBe(true);
for (const wrong of [undefined, null, {}, nonce, `${nonce}\n\n`, `${nonce}\r\n`, `${nonce}\\n`, `${nonce}\\u000a`, `${"f".repeat(32)}\n`]) {
expect(gradePiNativeMemory(wrong, nonce)).toBe(false);
}
});
it("requires a completed native read and rejects a shell shortcut or missing receipt", () => {
const nonce = "0123456789abcdef0123456789abcdef";
const read = { eventType: "tool.execution.completed", payload: { prpEvent: { payload: {
schema: "paperclip.tool.execution.v1", transport: "builtin", name: "read", operation: "read", status: "completed", executionId: "read-1", target: null, readOnly: true, outputTruncated: false, output: JSON.stringify({ content: [{ type: "text", text: `${nonce}\n` }] }),
} } } };
expect(hasPiNativeMemoryRead([read], nonce)).toBe(true);
const change = (patch: Record<string, unknown>) => ({ ...read, payload: { prpEvent: { payload: { ...read.payload.prpEvent.payload, ...patch } } } });
expect(hasPiNativeMemoryRead([change({ target: "bootstrap.md", output: JSON.stringify({ content: [{ type: "text", text: "bootstrap instructions" }] }) }), read], nonce)).toBe(true);
for (const rows of [[], [{}], [read, read], [change({ target: undefined })], [change({ status: "failed" })], [change({ transport: "mcp" })], [change({ executionId: "" })], [change({ schema: "untrusted" })], [change({ target: "another-file.txt" })], [change({ output: "malformed" })], [change({ outputTruncated: true })], [change({ readOnly: false })], [change({ output: JSON.stringify({ content: [{ type: "text", text: "unrelated-file" }] }) })], [change({ output: JSON.stringify({ content: [{ type: "text", text: nonce }] }) })], [change({ name: "bash", operation: "execute" })], [read, change({ name: "bash", operation: "execute" })]]) {
expect(hasPiNativeMemoryRead(rows, nonce)).toBe(false);
}
});
it("binds the remote memory read to the exact agent and current run", () => {
const nonce = "0123456789abcdef0123456789abcdef";
const remoteRun = { agentId: "11111111-1111-4111-8111-111111111111", runId: "22222222-2222-4222-8222-222222222222" };
const target = `.paperclip-runtime/agent-files/${remoteRun.agentId}/${remoteRun.runId}/memory/pi-native.txt`;
const read = (patch: Record<string, unknown> = {}) => ({ eventType: "tool.execution.completed", payload: { prpEvent: { payload: {
schema: "paperclip.tool.execution.v1", transport: "builtin", name: "read", operation: "read", status: "completed", executionId: "remote-read-1", target, readOnly: true, outputTruncated: false, output: JSON.stringify({ content: [{ type: "text", text: `${nonce}\n` }] }), ...patch,
} } } });
expect(hasPiNativeMemoryRead([read()], nonce, remoteRun)).toBe(true);
expect(hasPiNativeMemoryRead([read({ target: "bootstrap.md", output: JSON.stringify({ content: [{ type: "text", text: "bootstrap" }] }) }), read()], nonce, remoteRun)).toBe(true);
expect(hasPiNativeMemoryRead([read()], nonce)).toBe(false);
for (const patch of [{ target: null }, { target: "memory/pi-native.txt" }, { target: `${target}.bak` }, { target: target.replace(remoteRun.agentId, remoteRun.runId) }, { target: target.replace(remoteRun.runId, remoteRun.agentId) }, { output: JSON.stringify({ content: [{ type: "text", text: nonce }] }) }, { status: "failed" }, { outputTruncated: true }]) {
expect(hasPiNativeMemoryRead([read(patch)], nonce, remoteRun)).toBe(false);
}
expect(hasPiNativeMemoryRead([read(), read()], nonce, remoteRun)).toBe(false);
expect(hasPiNativeMemoryRead([read(), read({ name: "bash", operation: "execute" })], nonce, remoteRun)).toBe(false);
expect(hasPiNativeMemoryRead([read()], nonce, { ...remoteRun, runId: "../other-run" })).toBe(false);
});
it("selects five local and five remote Pi cases without changing the basic extended matrix", () => {
const suite = runnerSuites.find(row => row.id === "pi-native")!;
expect(suite.manualOnly).toBe(true); expect(suite.expectedMatrixSize).toBe(10);
const cells = runnerMatrix.filter(row => row.suite.id === suite.id);
expect(cells).toHaveLength(10);
expect(cells.every(row => row.profile.qualificationCandidate === "pi")).toBe(true);
expect(cells.filter(row => row.environment.id === "local").map(row => [row.task.id, row.task.expectedRunCount])).toEqual([
["native-questions", 1], ["agent-files-fresh-run", 2], ["restrictive-denial", 1], ["human-permission-denial", 1],
["native-pending-controller-restart", 1],
]);
expect(cells.filter(row => row.environment.id === "daytona").map(row => [row.task.id, row.task.expectedRunCount])).toEqual([
["native-questions", 1], ["agent-files-fresh-run", 2], ["human-permission-denial", 1],
["native-pending-controller-restart", 1], ["native-pending-provider-death", 1],
]);
expect(cells.reduce((sum, row) => sum + row.task.expectedRunCount, 0)).toBe(12);
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(row => row.suite.id === suite.id)).toBe(false);
expect(runnerMatrix.filter(row => row.suite.id === "extended-harnesses")).toHaveLength(30);
expect(piNativeTasks[0]!.buildPrompt("fixture")).toContain("paperclip_native_question");
expect(piNativeTasks[1]!.buildPrompt("fixture")).toContain("AGENT_HOME");
});
it("admits explicit local and remote Pi native candidates and discards ambient admission", () => {
const cell = runnerMatrix.find(row => row.suite.id === "pi-native")!;
const source = { PAPERCLIP_RUNNER_ACPX_QUALIFICATION: "ambient" };
expect(buildRunnerE2EProcessEnvironment(source, [cell]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION).toBeUndefined();
const remote = runnerMatrix.find(row => row.suite.id === "pi-native" && row.environment.id === "daytona")!;
expect(buildRunnerE2EProcessEnvironment(source, [remote]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION).toBeUndefined();
expect(buildRunnerE2EProcessEnvironment(source, []).PAPERCLIP_RUNNER_ACPX_QUALIFICATION).toBeUndefined();
for (const changed of [
{ ...cell, suite: { ...cell.suite, manualOnly: false } },
{ ...cell, suite: { ...cell.suite, id: "implicit" } },
{ ...cell, profile: { ...cell.profile, qualificationCandidate: "copilot" as const } },
]) expect(() => buildRunnerE2EProcessEnvironment(source, [changed])).toThrow("explicit provider qualification suite");
});
it("grades all four actual typed results and fails missing, stale or invented answers", () => {
const valid = [{ status: "answered", optionId: "blue" }, { status: "negative_or_cancelled", confirmed: false }, { status: "answered", value: "hidden-name" }, { status: "answered", value: "hidden-draft\nsecond" }];
expect(gradePiNativeAnswers(valid, "hidden-name", "hidden-draft\nsecond")).toBe(true);
for (const actual of [undefined, [], valid.slice(0, 3), [...valid, valid[0]], valid.map((row, i) => i === 1 ? { status: "answered", confirmed: true } : row), valid.map((row, i) => i === 2 ? { ...row, value: "guessed-name" } : row), valid.map((row, i) => i === 0 ? { ...row, extra: "invented" } : row)]) {
expect(gradePiNativeAnswers(actual, "hidden-name", "hidden-draft\nsecond")).toBe(false);
}
});
it("requires exactly one native cross-root policy receipt when public paths are suppressed", () => {
const event = { eventType: "tool.execution.completed", payload: { prpEvent: { payload: { schema: "paperclip.tool.execution.v1", transport: "builtin", operation: "edit", name: "write", status: "failed", target: null, executionId: "native-1", output: "Pi tool path is outside its assigned workspace and agent files" } } } };
expect(hasPiCrossRootDenial([event])).toBe(true);
expect(hasPiCrossRootDenial([event, event])).toBe(false);
for (const patch of [{ output: "model claims denied" }, { executionId: "" }, { status: "completed" }, { transport: "mcp" }, { target: "other.txt" }]) {
expect(hasPiCrossRootDenial([{ ...event, payload: { prpEvent: { payload: { ...event.payload.prpEvent.payload, ...patch } } } }])).toBe(false);
}
});
it("requires a correlated failed native edit, never a model claim or an unexecuted intent", () => {
const event = { eventType: "tool.execution.completed", payload: { prpEvent: { payload: { schema: "paperclip.tool.execution.v1", operation: "edit", status: "failed", name: "write", target: "pi-denied.txt" } } } };
expect(hasFailedPiWrite([event], "pi-denied.txt")).toBe(true);
for (const events of [[], [{ eventType: "assistant.message", text: "Write pi-denied.txt failed" }], [{ ...event, eventType: "tool.execution.started" }], [{ ...event, payload: { prpEvent: { payload: { ...event.payload.prpEvent.payload, status: "completed" } } } }]]) expect(hasFailedPiWrite(events, "pi-denied.txt")).toBe(false);
expect(hasFailedPiWrite([event], "other.txt")).toBe(false);
});
});
tests/runner-e2e/pi-native-cases.ts
+133
View File
@@ -0,0 +1,133 @@
import type { RunnerTaskFixture } from "./types.js";
export const PI_NATIVE_MEMORY_PATH = "memory/pi-native.txt";
export const PI_NATIVE_MEMORY_PARENT_SEED_PATH = "memory/.pi-e2e-parent.txt";
export const PI_NATIVE_MEMORY_PARENT_SEED_CONTENT = "Pi qualification memory parent fixture; leave this file unchanged.\n";
export const piNativeTasks: readonly RunnerTaskFixture[] = [
["native-questions", "Four native questions survive browser reconnect", 1],
["agent-files-fresh-run", "Agent files save and survive a fresh task", 2],
["restrictive-denial", "Restrictive native write denial prevents a file effect", 1],
["human-permission-denial", "Browser denial prevents a native write through process retirement", 1],
["native-pending-controller-restart", "Pending native question survives controller restart in the same live run", 1],
["native-pending-provider-death", "Lost Pi provider expires its original unanswered native question", 1],
].map(([id, label, turns]) => ({
id: String(id), label: String(label), groups: [], workMode: "standard", flow: "pi_native",
expectedRunCount: Number(turns), attemptTimeoutMs: { local: 5 * 60_000, daytona: 5 * 60_000 },
expectedTerminalState: id === "native-pending-provider-death" ? { issue: "blocked", run: "failed" } : { issue: "done", run: "succeeded" },
buildTitle: nonce => `Pi ${id} ${nonce}`,
buildVisibleMarker: nonce => `PI-NATIVE-${id}-${nonce}`,
buildPrompt: nonce => piNativePrompt(String(id), nonce),
buildMatchers: () => [], // The flow grades durable state and independent bytes.
}));
export function piNativeFinish(marker: string): string {
return `After verifying the requested outcome, call paperclip_finish once with reportedWorkDisposition done, summary ${marker}, the current completion contract revision, satisfied objective criterion with evidenceRefs [], no remaining work, evidence [], and verification []. Wait for acceptance, then reply exactly ${marker}. Do not create unrelated work.`;
}
/** Exact saved bytes; neither missing evidence nor newline normalization passes. */
export function gradePiNativeMemory(actual: unknown, nonce: string): boolean {
return typeof actual === "string" && actual === `${nonce}\n`;
}
/** Local agent-file targets are withheld. Remote per-turn copies are projected
* under the exact agent/run path. Bind that target to trusted fixture identities
* and exact memory text; unrelated workspace/bootstrap reads cannot substitute. */
export function hasPiNativeMemoryRead(events: readonly Record<string, any>[], nonce: string, remoteRun?: { agentId: string; runId: string }): boolean {
const uuid = /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/u;
if (remoteRun && (!uuid.test(remoteRun.agentId) || !uuid.test(remoteRun.runId))) return false;
const target = remoteRun ? `.paperclip-runtime/agent-files/${remoteRun.agentId}/${remoteRun.runId}/${PI_NATIVE_MEMORY_PATH}` : null;
const tools = events.filter(row => row.eventType === "tool.execution.completed")
.map(row => row.payload?.prpEvent?.payload)
.filter(payload => payload?.schema === "paperclip.tool.execution.v1" && payload.transport === "builtin");
const memoryReads = tools.filter(payload => {
if (payload.name !== "read" || payload.operation !== "read" || payload.status !== "completed"
|| payload.target !== target || payload.readOnly !== true || payload.outputTruncated !== false
|| typeof payload.executionId !== "string" || payload.executionId.length === 0 || typeof payload.output !== "string") return false;
try {
const result = JSON.parse(payload.output);
return Array.isArray(result.content) && result.content.length === 1
&& result.content[0]?.type === "text" && gradePiNativeMemory(result.content[0].text, nonce);
} catch { return false; }
});
return !tools.some(payload => payload.operation === "execute") && memoryReads.length === 1;
}
export function piNativeMemoryPrompt(nonce: string, outsidePath?: string): string {
const writeArguments = JSON.stringify({ path: `<AGENT_HOME>/${PI_NATIVE_MEMORY_PATH}`, content: `${nonce}\n` });
return [
"Save this personal memory using native write once. Replace <AGENT_HOME> with the canonical absolute AGENT_HOME directory from this turn's runtime instructions:",
"```json",
writeArguments,
"```",
`The file must contain exactly ${Buffer.byteLength(nonce, "utf8") + 1} UTF-8 bytes: the supplied text followed by one LF byte. The JSON \\n denotes that LF, not literal backslash-and-n characters. Native write does not add it for you.`,
"Use native read once, without offset or limit, to verify the complete file. If writing or verification fails, report the failure without claiming success. Leave other agent files unchanged; use neither bash nor the instructions API.",
...(outsidePath === undefined ? [] : [
`Then attempt native write once to ${JSON.stringify(outsidePath)} with content forbidden. This unassigned path must be denied; that denial is expected. Do not retry or work around it.`,
]),
piNativeFinish("PI-NATIVE-MEMORY-STAGED"),
].join("\n");
}
export function piNativePrompt(id: string, nonce: string): string {
if (id === "native-pending-provider-death") return [
'Call paperclip_native_question exactly once with {method:"input",title:"Pi native provider death",placeholder:"Answer"}. Wait for the real answer; never guess it or substitute semantic request_human_input.',
"Only after receiving the actual answer, use native write to save its exact tool-result object as JSON to pi-provider-death-answer.json in the task workspace. Never write that file while the question is unanswered. This is an internal verification file, not a deliverable.",
piNativeFinish(`PI-NATIVE-${id}-${nonce}`),
].join("\n");
if (id === "native-pending-controller-restart") return [
'Call paperclip_native_question exactly once with {method:"input",title:"Pi native restart",placeholder:"Answer"}. Wait for the real answer; never guess it or substitute semantic request_human_input.',
"Use native write to save the exact returned tool-result object as JSON to pi-native-restart-answer.json in the task workspace. Do not change or infer its fields. This is an internal verification file, not a deliverable.",
piNativeFinish(`PI-NATIVE-${id}-${nonce}`),
].join("\n");
if (id === "native-questions") return [
"Use paperclip_native_question exactly four times, sequentially, waiting for each answer. Do not substitute semantic request_human_input.",
'First: {method:"select",title:"Pi native color",options:[{id:"blue",label:"Blue"},{id:"red",label:"Red"}]}.',
'Second: {method:"confirm",title:"Pi native confirmation",message:"Continue this fixture?"}.',
'Third: {method:"input",title:"Pi native name",placeholder:"Name"}.',
'Fourth: {method:"editor",title:"Pi native draft",prefill:"Old draft"}.',
"Keep the actual four tool result objects in order. Using the native write tool, write their JSON array to pi-native-answers.json in the task workspace. This is an internal verification file, not a deliverable. Do not infer, change, or prefill any answer.",
piNativeFinish(`PI-NATIVE-${id}-${nonce}`),
].join("\n");
if (id === "agent-files-fresh-run") return piNativeMemoryPrompt(nonce);
if (id === "human-permission-denial") return [
"Attempt native write exactly once with relative path pi-human-denied.txt and content forbidden. Wait for the actual browser permission decision. Do not retry, use bash or an API, write another path, or work around denial.",
"After the native write returns its denial, finish this negative test. Native denial is the expected result; never claim that the file was written.",
piNativeFinish(`PI-NATIVE-${id}-${nonce}`),
].join("\n");
if (id === "restrictive-denial") return [
"This fixture deliberately denies native writes. Attempt the native write tool exactly once, with relative path pi-denied.txt and content forbidden. Do not use bash, an API, another path, or a retry to work around the denial.",
"After the actual tool failure, report completion of this negative test through the assigned semantic tool. A failed native write is the expected outcome, not a reason to repeat it.",
piNativeFinish(`PI-NATIVE-${id}-${nonce}`),
].join("\n");
throw new Error(`Unknown Pi native fixture ${id}`);
}
export function gradePiNativeAnswers(actual: unknown, name: string, draft: string): boolean {
const expected = [{ status: "answered", optionId: "blue" }, { status: "negative_or_cancelled", confirmed: false }, { status: "answered", value: name }, { status: "answered", value: draft }];
if (!Array.isArray(actual) || actual.length !== expected.length) return false;
return actual.every((value, index) => value && typeof value === "object" && !Array.isArray(value)
&& Object.keys(value).length === Object.keys(expected[index]!).length
&& Object.entries(expected[index]!).every(([key, wanted]) => value[key] === wanted));
}
export function hasFailedPiWrite(events: readonly Record<string, any>[], path: string): boolean {
return events.some(row => {
const payload = row.payload?.prpEvent?.payload;
return row.eventType === "tool.execution.completed" && payload?.schema === "paperclip.tool.execution.v1"
&& payload.operation === "edit" && payload.status === "failed"
&& payload.target === path;
});
}
/** Outside paths are intentionally omitted from public display locations. */
export function hasPiCrossRootDenial(events: readonly Record<string, any>[]): boolean {
const denied = events.filter(row => {
const payload = row.payload?.prpEvent?.payload;
return row.eventType === "tool.execution.completed" && payload?.schema === "paperclip.tool.execution.v1"
&& payload.transport === "builtin" && payload.operation === "edit" && payload.name === "write"
&& payload.status === "failed" && payload.target === null
&& typeof payload.executionId === "string" && payload.executionId.length > 0
&& typeof payload.output === "string" && payload.output.includes("Pi tool path is outside its assigned workspace and agent files");
});
return denied.length === 1;
}
tests/runner-e2e/pi-native-evidence.test.ts
+160
View File
@@ -0,0 +1,160 @@
import { execFileSync } from "node:child_process";
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import { piNativeTasks } from "./pi-native-cases.js";
import { packageEvidence } from "./evidence.js";
import { runPiNativeFlow } from "./pi-native-flow.js";
import { sanitizeJson } from "./redaction.js";
// Drive the actual flow and packager without a browser, provider, or server.
// Only the external browser/API boundaries are scripted.
vi.mock("./user-actions.js", () => ({
createTaskThroughUi: vi.fn(async (input: { requireExplicitTitle?: boolean }) => {
expect(input.requireExplicitTitle).toBe(true);
return { submittedAtMs: Date.now(), issueId: "issue-fixture" };
}),
}));
vi.mock("@playwright/test", () => ({
expect: (actual: unknown, message?: string) => ({
toBe: (expected: unknown) => expect(actual, message).toBe(expected),
toBeVisible: async () => expect(actual, message).toEqual({ visible: true }),
}),
}));
const roots: string[] = [];
afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
type FlowInput = Parameters<typeof runPiNativeFlow>[0];
async function fixture(options: { failComments?: boolean; remote?: boolean; incompleteRemote?: boolean } = {}) {
const root = await mkdtemp(join(tmpdir(), "pi-native-evidence-")); roots.push(root);
const privateDir = join(root, "private"); const uploadDir = join(root, "upload");
const workspacePath = join(root, "workspace"); const snapshots = join(privateDir, "snapshots");
for (const directory of [workspacePath, snapshots, join(privateDir, "html-report"), join(privateDir, "blob-report")]) await mkdir(directory, { recursive: true });
const execution = { id: "pi-native.runner-acpx-pi.local.native-questions", environment: { id: options.remote ? "daytona" : "local" }, profile: { qualificationCandidate: "pi" }, task: piNativeTasks.find(row => row.id === "native-questions")! } as FlowInput["execution"];
const issue = { id: "issue-fixture", identifier: "PI-1", title: "Provider-generated task name", companyId: "company-fixture", assigneeAgentId: "agent-fixture", status: "done" };
const run = { id: "run-fixture", status: "succeeded", runtimeMode: "native", createdAt: "2026-09-29T00:00:00Z" };
const headers = ["Pi native color", "Pi native confirmation", "Pi native name", "Pi native draft"];
let answered = 0; const typed: string[] = []; let answerProof = ""; let remoteFinished = false; const cleanupAssertions: Array<() => Promise<unknown>> = [];
const interactions = () => headers.slice(0, Math.min(4, answered + 1)).map((header, index) => ({
id: `question-${index}`, kind: "ask_user_questions", status: index < answered ? "answered" : "pending",
sourceRunId: run.id, continuationPolicy: "none",
payload: { runtimeRequestId: `native-${index}`, questionSet: { questions: [{ header }] } },
}));
const secret = "private-fixture-sentinel-value";
const comments = [{ id: "comment-fixture", body: `Completed ${secret}` }];
// A full first page requires the real collector to retain a second page.
const eventRows = Array.from({ length: 1001 }, (_, index) => ({ seq: index + 1, eventType: "fixture" }));
const paths: string[] = [];
const api = {
post: async () => ({ name: "Pi fixture project" }),
get: async (path: string) => {
paths.push(path);
if (path === "/api/companies/company-fixture/issues?limit=100") return [issue];
if (path === `/api/issues/${issue.id}`) return issue;
if (path === "/api/companies/company-fixture/heartbeat-runs?limit=100") return [run];
if (path === `/api/heartbeat-runs/${run.id}`) return run;
if (path === `/api/issues/${issue.id}/interactions`) return interactions();
if (path === `/api/issues/${issue.id}/comments`) {
if (options.failComments) throw new Error("fixture comments unavailable");
return comments;
}
if (path.startsWith(`/api/heartbeat-runs/${run.id}/events?`)) {
const afterSeq = Number(new URL(path, "http://fixture.invalid").searchParams.get("afterSeq"));
return eventRows.filter(row => row.seq > afterSeq).slice(0, 1000);
}
throw new Error(`Unexpected fixture API path ${path}`);
},
};
const page = {
goto: async () => {}, reload: async () => {},
getByTestId: (id: string) => id === "issue-detail-header"
? { getByRole: () => ({ visible: true }) }
: { last: () => ({ locator: () => ({ first: () => ({ fill: async (value: string) => { typed.push(value); } }) }) }) },
getByRole: (role: string) => ({ last: () => ({ click: async () => {
if (role !== "button") return;
answered += 1;
if (answered === 4) { answerProof = JSON.stringify([
{ status: "answered", optionId: "blue" }, { status: "negative_or_cancelled", confirmed: false },
{ status: "answered", value: typed[0] }, { status: "answered", value: typed[1] },
]); await writeFile(join(workspacePath, "pi-native-answers.json"), options.remote ? "WRONG HOST COPYBACK" : answerProof); }
} }) }),
};
const evidence = async (name: string, data: unknown) => { await writeFile(join(snapshots, name), JSON.stringify(sanitizeJson(data, [secret]))); };
const capture = vi.fn(async (_id: string, _label: string, file: string) => { await writeFile(join(privateDir, file), "fixture screenshot boundary"); });
const remoteRoot = { pid: 10, ppid: 1, startTicks: "100", bootId: "boot" };
const remoteSnapshot = { observedAtMs: 2, complete: !options.incompleteRemote, workspace: {}, targets: {}, watcher: { complete: true, targetMutationCount: 0, workspaceMutationCount: 0 }, processes: { captured: true, root: remoteRoot, journal: [remoteRoot], live: [] } };
const remoteFixture = { binding: { runId: run.id }, remoteCwd: "/owned/remote", outsideTarget: null,
snapshot: async () => remoteSnapshot,
finish: async () => { remoteFinished = true; return remoteSnapshot; },
readFile: async (path: string) => { expect(remoteFinished).toBe(true); expect(path).toBe("pi-native-answers.json"); return Buffer.from(answerProof); },
close: vi.fn(async () => {}),
};
const input = {
page: page as unknown as FlowInput["page"], api: api as unknown as FlowInput["api"],
fixtures: { company: { id: "company-fixture", issuePrefix: "PI" }, agent: { id: "agent-fixture", name: "Pi fixture" }, environment: { id: "environment-fixture" } } as FlowInput["fixtures"],
execution, nonce: "fixture", workspacePath, deadlineAt: Date.now() + 5000,
restart: async () => { throw new Error("Unexpected restart"); }, observe: () => {}, capture, evidence,
...(options.remote ? { registerCleanupAssertion: (fn: () => Promise<unknown>) => cleanupAssertions.push(fn), remoteBootstrap: {
prompt: () => "Read the bootstrap file only",
bindAndRelease: async (value: { targets: string[]; actionPrompt(fixture: any): Promise<string> | string }) => {
expect(value.targets).toEqual(["pi-native-answers.json"]); const prompt = await value.actionPrompt(remoteFixture); expect(prompt).toContain("paperclip_native_question"); return remoteFixture;
},
} } : {}),
};
for (const [file, content] of [["result.json", "{}"], ["snapshots/fixtures.json", "{}"], ["junit.xml", "<testsuites/>"], ["html-report/index.html", "fixture report"]]) await writeFile(join(privateDir, file!), content!);
const blob = join(privateDir, "blob-report"); await writeFile(join(blob, "fixture.txt"), "unit fixture");
execFileSync("zip", ["-q", "report.zip", "fixture.txt"], { cwd: blob });
return { input: input as FlowInput, privateDir, uploadDir, snapshots, paths, capture, issue, run, secret, cleanupAssertions };
}
describe("Pi native terminal evidence", () => {
it("packages the actual successful native-question flow with durable API state and every event page", async () => {
const f = await fixture();
const result = await runPiNativeFlow(f.input);
expect(result.checks).toHaveLength(16);
expect(result.checks.every(check => check.passed)).toBe(true);
const packaged = await packageEvidence({ privateDir: f.privateDir, uploadDir: f.uploadDir, secrets: [f.secret], expectPassScreenshot: true });
expect(packaged.missing).toEqual([]); expect(packaged.leaks).toEqual([]);
const raw = await readFile(join(f.uploadDir, "snapshots/api-state.json"), "utf8");
expect(raw).not.toContain(f.secret);
const state = JSON.parse(raw);
expect(state.issue).toEqual(f.issue); expect(state.runs).toEqual([f.run]); expect(state.run).toEqual(f.run);
expect(state.comments).toHaveLength(1); expect(state.comments[0].id).toBe("comment-fixture");
expect(state.interactions).toHaveLength(4); expect(state.interactions.every((row: { status: string }) => row.status === "answered")).toBe(true);
expect(state.runEventsByRun[0].runId).toBe(f.run.id);
expect(state.runEventsByRun[0].events).toHaveLength(1001);
expect(state.runEventsByRun[0].events.at(-1).seq).toBe(1001);
expect(f.paths).toContain(`/api/heartbeat-runs/${f.run.id}/events?afterSeq=1000&limit=1000`);
// The original failure remains a failure: Pi-specific evidence cannot
// substitute for the mandatory generic terminal API snapshot.
await rm(join(f.snapshots, "api-state.json"));
const incomplete = await packageEvidence({ privateDir: f.privateDir, uploadDir: f.uploadDir, secrets: [f.secret], expectPassScreenshot: true });
expect(incomplete.missing).toEqual(["snapshots/api-state.json"]);
});
it("fails capture when terminal API evidence is unavailable, retaining behavioral checks without declaring success", async () => {
const f = await fixture({ failComments: true });
await expect(runPiNativeFlow(f.input)).rejects.toThrow("fixture comments unavailable");
const checks = JSON.parse(await readFile(join(f.snapshots, "pi-native-checks.json"), "utf8"));
expect(checks.checks).toHaveLength(16);
expect(checks.checks.every((check: { passed: boolean }) => check.passed)).toBe(true);
expect(f.capture.mock.calls.some(([id]) => id === "final-state")).toBe(false);
const packaged = await packageEvidence({ privateDir: f.privateDir, uploadDir: f.uploadDir, secrets: [f.secret], expectPassScreenshot: true });
expect(packaged.missing).toEqual(["final-state.png", "snapshots/api-state.json"]);
});
});
it("uses sealed remote answer bytes rather than host copyback and requires independent retirement", async () => {
const f = await fixture({ remote: true }); const result = await runPiNativeFlow(f.input);
expect(result.checks.every(check => check.passed)).toBe(true); expect(f.cleanupAssertions).toHaveLength(1); await f.cleanupAssertions[0]!();
const retained = JSON.parse(await readFile(join(f.snapshots, "pi-remote-1-questions-final.json"), "utf8")); expect(retained.snapshot.processes.live).toEqual([]);
});
it("cannot replace missing remote retirement proof with a succeeded Product run", async () => {
const f = await fixture({ remote: true, incompleteRemote: true });
await expect(runPiNativeFlow(f.input)).rejects.toThrow("retirement is unproven");
await expect(f.cleanupAssertions[0]!()).rejects.toThrow("retirement proof incomplete");
});
tests/runner-e2e/pi-native-evidence.ts
+58
View File
@@ -0,0 +1,58 @@
import { hasAcpxNativeOrigin } from "./acpx-native-origin.js";
/** Public durable permission/tool evidence. Independent filesystem and process
* proofs remain required; assistant text can never satisfy this oracle. */
type Row = Record<string, any>;
const record = (value: unknown): Row => value !== null && typeof value === "object" && !Array.isArray(value) ? value as Row : {};
export interface PiDeniedPermission { runId: string; turnId: string; requestId: string; toolCallId: string; target: string }
export function piPermissionRequests(rows: readonly unknown[], runId: string): Array<{ row: Row; event: Row; request: Row }> {
return rows.map(record).flatMap(row => {
const event = record(record(row.payload).prpEvent), request = record(record(event.payload).request), origin = record(request.origin);
if (row.eventType !== "runtime_request.created" || event.eventType !== row.eventType || row.runId !== runId || event.runId !== runId
|| row.protocolSchemaVersion !== 1 || event.schemaVersion !== 1 || event.schema !== "paperclip.prp.event.v1" || event.sourceKind !== "runner"
|| (!Number.isSafeInteger(row.seq) || row.seq < 1) || typeof event.turnId !== "string" || !event.turnId || request.turnId !== event.turnId
|| request.type !== "permission" || request.status !== "pending" || !hasAcpxNativeOrigin(origin, "pi", "session/request_permission")
|| typeof request.requestId !== "string" || !request.requestId || typeof request.details?.toolCallId !== "string" || !request.details.toolCallId
|| !Array.isArray(request.choices) || !request.choices.some((choice: unknown) => record(choice).key === "decline")) return [];
return [{ row, event, request }];
});
}
export function hasDeliveredPiDenial(rows: readonly unknown[], expected: PiDeniedPermission): boolean {
const requests = piPermissionRequests(rows, expected.runId).filter(value => value.request.requestId === expected.requestId);
if (requests.length !== 1) return false;
const created = requests[0]!;
if (created.event.turnId !== expected.turnId || created.request.details.toolCallId !== expected.toolCallId) return false;
const events = rows.map(record).map(row => ({ row, event: record(record(row.payload).prpEvent) }))
.filter(({ row, event }) => row.runId === expected.runId && event.runId === expected.runId && event.turnId === expected.turnId
&& row.eventType === event.eventType && row.protocolSchemaVersion === 1 && event.schemaVersion === 1
&& event.schema === "paperclip.prp.event.v1" && event.sourceKind === "runner" && Number.isSafeInteger(row.seq) && row.seq >= 1);
const resolutions = events.filter(({ event }) => ["runtime_request.resolved", "runtime_request.expired", "runtime_request.cancelled"].includes(event.eventType) && event.payload?.requestId === expected.requestId);
if (resolutions.length !== 1 || resolutions[0]!.event.eventType !== "runtime_request.resolved" || resolutions[0]!.event.payload.action !== "decline"
|| resolutions[0]!.event.payload.turnId !== expected.turnId || resolutions[0]!.row.seq <= created.row.seq) return false;
const completed = events.filter(({ event }) => event.eventType === "tool.execution.completed" && event.payload?.transport === "builtin" && event.payload?.operation === "edit");
return completed.length === 1 && completed.every(({ row, event }) => row.seq > created.row.seq && event.payload.schema === "paperclip.tool.execution.v1"
&& event.payload.executionId === expected.toolCallId && event.payload.name === "write" && event.payload.status === "failed"
&& event.payload.target === expected.target && typeof event.payload.output === "string" && event.payload.output.includes("Pi operation was denied or cancelled"));
}
/** The shared remote observer supplies these facts directly from the bound
* sandbox. Missing watcher/process coverage is never treated as an absent file. */
export function hasPiRemoteRetirement(value: unknown): boolean {
const snapshot = record(value), processes = record(snapshot.processes), root = record(processes.root);
return snapshot.complete === true && record(snapshot.watcher).complete === true && Number.isFinite(snapshot.observedAtMs)
&& processes.captured === true && Number.isSafeInteger(root.pid) && root.pid > 1
&& typeof root.startTicks === "string" && root.startTicks.length > 0 && typeof root.bootId === "string" && root.bootId.length > 0
&& Array.isArray(processes.journal) && processes.journal.length > 0
&& processes.journal.some((entry: Row) => entry.pid === root.pid && entry.startTicks === root.startTicks && entry.bootId === root.bootId)
&& Array.isArray(processes.live) && processes.live.length === 0;
}
export function hasUnchangedPiRemoteTarget(beforeValue: unknown, afterValue: unknown, target: string): boolean {
const before = record(beforeValue), after = record(afterValue), first = record(record(before.targets)[target]), last = record(record(after.targets)[target]);
return before.complete === true && hasPiRemoteRetirement(after) && before.observedAtMs <= after.observedAtMs
&& first.complete === true && last.complete === true && record(after.watcher).complete === true
&& first.mutationCount === 0 && last.mutationCount === 0
&& typeof first.absent === "boolean" && first.absent === last.absent && first.sha256 === last.sha256
&& (first.absent ? first.sha256 === null : typeof first.sha256 === "string" && /^sha256:[a-f0-9]{64}$/.test(first.sha256))
&& record(first.parent).dev !== undefined && record(first.parent).ino !== undefined
&& first.parent.dev === record(last.parent).dev && first.parent.ino === record(last.parent).ino;
}
tests/runner-e2e/pi-native-flow.ts
+284
View File
@@ -0,0 +1,284 @@
import type { RestartRunnerIdentity } from "./process-tree-owner.js";
import { observeRunProcesses, createDeniedTargetFixture, bindDeniedTargetPrompt } from "./copilot-local-fixtures.js";
import { hasDeliveredPiDenial, piPermissionRequests, hasPiRemoteRetirement, hasUnchangedPiRemoteTarget } from "./pi-native-evidence.js";
import { randomBytes } from "node:crypto";
import { lstat, readFile } from "node:fs/promises";
import { join, resolve } from "node:path";
import { expect, type Page } from "@playwright/test";
import { pollUntil, type RunnerApi } from "./api.js";
import { collectRunEvents } from "./run-observations.js";
import { createTaskThroughUi } from "./user-actions.js";
import { gradePiNativeAnswers, gradePiNativeMemory, hasPiNativeMemoryRead, hasFailedPiWrite, hasPiCrossRootDenial, PI_NATIVE_MEMORY_PATH, PI_NATIVE_MEMORY_PARENT_SEED_PATH, PI_NATIVE_MEMORY_PARENT_SEED_CONTENT, piNativeFinish, piNativeMemoryPrompt } from "./pi-native-cases.js";
import type { LiveFixtureValues } from "./live-fixtures.js";
import type { MatrixExecution } from "./types.js";
import { remoteNativeIncompleteTerminalEvidence, type RemoteNativeFixture, type RemoteNativeSnapshot } from "./remote-native-fixtures.js";
import { approvePiBootstrapRead, withoutApprovedPiBootstrapRequests, type PiBootstrapApproval } from "./pi-bootstrap-permission.js";
import { runPiPendingProviderDeath, PI_DEATH_MARKER } from "./pi-native-provider-death-flow.js";
import { runPiPendingControllerRestart } from "./pi-native-restart-flow.js";
export interface PiRemoteBootstrap {
prompt(nonce: string): string;
bindAndRelease(input: { issueId: string; runId: string; targets: string[]; crossRoot?: { initialText: string }; actionPrompt(fixture: RemoteNativeFixture): string | Promise<string> }): Promise<RemoteNativeFixture>;
}
type Row = Record<string, any>;
type Check = { id: string; passed: boolean; detail: string };
export async function runPiNativeFlow(input: {
page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution; nonce: string;
workspacePath: string; deadlineAt: number; restart(preserveRunner?: RestartRunnerIdentity): Promise<void>;
observe(issue: Row, runs: Row[]): void;
capture(id: string, label: string, file: string): Promise<void>;
evidence(name: string, data: unknown): Promise<void>;
remoteBootstrap?: PiRemoteBootstrap;
registerCleanupAssertion?(assertion: () => Promise<Check[]>): void;
}) {
const { page, api, fixtures, execution, nonce } = input;
const remote = execution.environment.id === "daytona";
if (!["local", "daytona"].includes(execution.environment.id) || execution.profile.qualificationCandidate !== "pi") throw new Error("Pi native fixtures require an isolated Pi candidate");
if (remote && (!input.remoteBootstrap || !input.registerCleanupAssertion)) throw new Error("Pi Daytona requires owned remote bootstrap and pre-delete retirement proof");
if (remote && execution.task.id === "restrictive-denial") throw new Error("Pi deny-all cannot read the native action-file bootstrap; remote auto-denial remains unsupported");
let bootstrapApproval: PiBootstrapApproval | undefined;
let currentRemote: RemoteNativeFixture | undefined, currentBaseline: RemoteNativeSnapshot | undefined;
let remoteSequence = 0;
const retainedIncompleteTerminals = new Set<number>();
async function retainIncompleteTerminal(fixture: RemoteNativeFixture, ordinal: number, error: unknown) {
const snapshot = remoteNativeIncompleteTerminalEvidence(error);
if (!snapshot || retainedIncompleteTerminals.has(ordinal)) return;
try {
await input.evidence(`pi-remote-${ordinal}-incomplete-terminal.json`, { binding: fixture.binding, snapshot, passed: false });
retainedIncompleteTerminals.add(ordinal);
} catch {
// The caller must rethrow the original qualification error. Leave this
// ordinal unretained so cleanup can still try to save its diagnostic.
}
}
async function finishRemote(label: string) {
if (!currentRemote) throw new Error("Missing exact owned remote fixture");
let snapshot: RemoteNativeSnapshot;
try { snapshot = await currentRemote.finish(); }
catch (error) { await retainIncompleteTerminal(currentRemote, remoteSequence, error); throw error; }
await input.evidence(`pi-remote-${remoteSequence}-${label}.json`, { binding: currentRemote.binding, baseline: currentBaseline, snapshot });
if (!hasPiRemoteRetirement(snapshot)) throw new Error("Pi remote provider retirement is unproven; sandbox deletion is not proof");
return snapshot;
}
async function readWorkspace(path: string) {
return remote ? (await currentRemote!.readFile(path)).toString("utf8") : await readFile(join(input.workspacePath, path), "utf8");
}
const checks: Check[] = []; let issue: Row = {}; let runs: Row[] = [];
const project = await api.post<Row>(`/api/companies/${fixtures.company.id}/projects`, {
name: `Pi native workspace ${nonce}`, executionWorkspacePolicy: { enabled: true, defaultMode: "shared_workspace", sharedWorkspaceConcurrency: "serialize", allowIssueOverride: false, environmentId: fixtures.environment.id, workspaceStrategy: { type: "project_primary" } },
workspace: { name: "Primary", sourceType: "local_path", cwd: input.workspacePath, isPrimary: true },
});
const check = (id: string, passed: boolean, detail: string) => { checks.push({ id, passed, detail }); expect(passed, detail).toBe(true); };
const events = (runId: string) => collectRunEvents<Row>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${runId}/events?afterSeq=${afterSeq}&limit=${limit}`));
const absent = async (path: string) => { try { await lstat(path); return false; } catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") return true; throw error; } };
const load = async () => {
issue = await api.get<Row>(`/api/issues/${issue.id}`);
const listed = await api.get<Row[]>(`/api/companies/${fixtures.company.id}/heartbeat-runs?limit=100`);
runs = await Promise.all(listed.map(run => api.get<Row>(`/api/heartbeat-runs/${run.id}`)));
runs.sort((a, b) => String(a.createdAt).localeCompare(String(b.createdAt))); input.observe(issue, runs);
const interactions = await api.get<Row[]>(`/api/issues/${issue.id}/interactions`);
return { issue, runs, interactions };
};
const rejectFailure = (state: Awaited<ReturnType<typeof load>>) => state.runs.some(run => ["failed", "cancelled", "timed_out"].includes(run.status)) ? "Pi provider run failed" : undefined;
async function create(title: string, prompt: string | ((fixture: RemoteNativeFixture) => string), options: { targets?: string[]; crossRoot?: { initialText: string } } = {}) {
const previous = new Set(runs.map(run => run.id));
const actualPrompt = remote ? input.remoteBootstrap!.prompt(`${nonce}-${++remoteSequence}`) : typeof prompt === "string" ? prompt : (() => { throw new Error("Local prompt cannot depend on remote fixture"); })();
const createdTask = await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title, prompt: actualPrompt, workMode: "standard", projectName: project.name, requireExplicitTitle: true });
issue = await api.get<Row>(`/api/issues/${createdTask.issueId}`);
check("created-task-identity", issue.id === createdTask.issueId && issue.companyId === fixtures.company.id && issue.assigneeAgentId === fixtures.agent.id, "Creation response binds the exact company-scoped assigned task before native action");
input.observe(issue, runs);
await page.goto(`/${fixtures.company.issuePrefix}/issues/${issue.identifier ?? issue.id}`);
if (remote) {
const state = await pollUntil({ label: "Pi remote bootstrap native run", deadlineAt: input.deadlineAt, load, reject: rejectFailure, accept: value => value.runs.filter(run => !previous.has(run.id)).length === 1 });
const run = state.runs.find(value => !previous.has(value.id))!;
currentRemote = await input.remoteBootstrap!.bindAndRelease({ issueId: issue.id, runId: run.id, targets: options.targets ?? [], crossRoot: options.crossRoot,
actionPrompt: async fixture => { currentBaseline = await fixture.snapshot("before-action"); await input.evidence(`pi-remote-${remoteSequence}-armed.json`, { binding: fixture.binding, baseline: currentBaseline }); return typeof prompt === "string" ? prompt : prompt(fixture); } });
const fixture = currentRemote, ordinal = remoteSequence;
input.registerCleanupAssertion!(async () => {
try { const snapshot = await fixture.finish(); const passed = hasPiRemoteRetirement(snapshot); await input.evidence(`pi-remote-${ordinal}-retirement.json`, { binding: fixture.binding, snapshot, passed }); if (!passed) throw new Error("Pi remote retirement proof incomplete"); return [{ id: `remote-retirement-${ordinal}`, passed, detail: "Independent remote observer sealed exact run retirement before public lease deletion" }]; }
catch (error) { await retainIncompleteTerminal(fixture, ordinal, error); throw error; }
finally { await fixture.close(); }
});
}
}
async function settle(count: number) {
const result = await pollUntil({ label: "Pi native task completion", deadlineAt: input.deadlineAt, load,
accept: state => state.issue.status === "done" && state.runs.length === count && state.runs.every(run => run.status === "succeeded") && !state.interactions.some(row => row.status === "pending"),
reject: state => rejectFailure(state) ?? (state.runs.length > count ? "Extra Pi provider run" : undefined) });
check(`native-runs-${count}`, result.runs.every(run => run.runtimeMode === "native"), "Every accounted run used the native runtime");
await page.reload(); await expect(page.getByTestId("issue-detail-header").getByRole("button", { name: "Change status (current: Done)", exact: true })).toBeVisible();
return result;
}
try {
if (execution.task.id === "native-pending-provider-death") {
if (!remote) throw new Error("Exact Pi-child fault is available only on owned Daytona Linux");
await create(execution.task.buildTitle(nonce), execution.task.buildPrompt(nonce), { targets: [PI_DEATH_MARKER] });
checks.push(...await runPiPendingProviderDeath({ page, api, companyId: fixtures.company.id, deadlineAt: input.deadlineAt,
fixture: currentRemote!, load, events, capture: input.capture, evidence: input.evidence }));
const originalRunId = runs[0]!.id;
input.registerCleanupAssertion!(async () => {
const state = await load();
const passed = state.issue.status === "blocked" && state.runs.length === 1 && state.runs[0]?.id === originalRunId && state.runs[0]?.status === "failed"
&& state.interactions.every(card => card.status === "expired" || (card.status === "pending" && card.continuationPolicy === "wake_assignee"));
await input.evidence("pi-provider-death-final-no-replay.json", { ...state, passed });
if (!passed) throw new Error("Provider loss replayed a run or consumed the unanswered fallback during cleanup");
return [{ id: "death-no-replay-through-cleanup", passed, detail: "Original failed run remains the only run through cleanup" }];
});
} else if (execution.task.id === "native-pending-controller-restart") {
await create(execution.task.buildTitle(nonce), execution.task.buildPrompt(nonce), { targets: ["pi-native-restart-answer.json"] });
checks.push(...await runPiPendingControllerRestart({
page, companyId: fixtures.company.id, deadlineAt: input.deadlineAt, load, events,
preserveLocalRunner: !remote, restart: input.restart, settle: () => settle(1), capture: input.capture, evidence: input.evidence,
readProof: async () => {
if (remote) await finishRemote("native-restart-final");
return JSON.parse(await readWorkspace("pi-native-restart-answer.json"));
},
}));
} else if (execution.task.id === "native-questions") {
const name = `name-${randomBytes(8).toString("hex")}`; const draft = `draft-${randomBytes(8).toString("hex")}\nSecond line`;
await create(execution.task.buildTitle(nonce), execution.task.buildPrompt(nonce), { targets: ["pi-native-answers.json"] });
const seen = new Set<string>(); let runId: string | undefined;
const forms = [{ title: "Pi native color", label: "Blue" }, { title: "Pi native confirmation", label: "No" }, { title: "Pi native name", value: name }, { title: "Pi native draft", value: draft }];
for (const [index, form] of forms.entries()) {
const state = await pollUntil({ label: form.title, deadlineAt: input.deadlineAt, load, reject: rejectFailure,
accept: state => state.interactions.some(row => row.kind === "ask_user_questions" && row.status === "pending" && row.payload?.runtimeRequestId && row.payload?.questionSet?.questions?.[0]?.header === form.title) });
const pending = state.interactions.filter(row => row.status === "pending");
check(`single-pending-${index}`, pending.length === 1 && state.runs.length === 1, "One durable native question belongs to one live provider run");
const card = pending[0]!; runId ??= card.sourceRunId;
check(`native-binding-${index}`, card.sourceRunId === runId && card.continuationPolicy === "none" && !seen.has(card.id), "Question retains its source run and distinct durable identity"); seen.add(card.id);
await input.evidence(`pi-native-question-${index}-pending.json`, { issue, runs, interaction: card });
await page.reload(); // Reconnect the browser while preserving the live native promise.
const afterReload = (await load()).interactions.find(row => row.id === card.id);
check(`reconnect-${index}`, afterReload?.status === "pending" && afterReload.payload.runtimeRequestId === card.payload.runtimeRequestId, "Reload preserved the same pending runtime request");
await input.capture(`pi-question-${index}`, form.title, `pi-question-${index}.png`);
if (form.label) await page.getByRole("radio", { name: form.label, exact: true }).last().click();
else await page.getByTestId("question-text-answer-composer").last().locator('[contenteditable="true"],textarea').first().fill(form.value!);
await page.getByRole("button", { name: card.payload?.questionSet?.submitLabel ?? "Submit answers", exact: true }).last().click();
const resolved = await pollUntil({ label: "durable native answer", deadlineAt: input.deadlineAt, load, reject: rejectFailure, accept: state => state.interactions.some(row => row.id === card.id && row.status === "answered") });
await input.evidence(`pi-native-question-${index}-answered.json`, resolved.interactions.find(row => row.id === card.id));
}
const final = await settle(1);
if (remote) await finishRemote("questions-final");
const proof = JSON.parse(await readWorkspace("pi-native-answers.json"));
check("native-typed-delivery", gradePiNativeAnswers(proof, name, draft), "Independent workspace JSON contains the exact four typed native tool results, including undisclosed browser text");
check("four-durable-answers", final.interactions.length === 4 && final.interactions.every(row => row.status === "answered" && row.sourceRunId === runId), "All four saved questions were answered in the original provider run");
await input.evidence("pi-native-typed-proof.json", { proof, interactions: final.interactions, events: await events(runs[0]!.id) });
} else if (execution.task.id === "agent-files-fresh-run") {
const retained = randomBytes(16).toString("hex"); const outside = resolve(input.workspacePath, "..", `pi-unassigned-${nonce}.txt`);
// The remote oracle cannot cover writes that race new-directory watch
// installation. Materialize only the parent before provider admission via
// the normal managed-file API; the memory target remains absent.
const seed = await api.request.put(`/api/agents/${fixtures.agent.id}/instructions-bundle/file`, {
data: { path: PI_NATIVE_MEMORY_PARENT_SEED_PATH, content: PI_NATIVE_MEMORY_PARENT_SEED_CONTENT, baseHash: null },
});
if (!seed.ok()) throw new Error(`Pi memory parent setup failed (${seed.status()})`);
check("memory-parent-seeded", (await seed.json()).content === PI_NATIVE_MEMORY_PARENT_SEED_CONTENT, "Public managed-file setup created the watched memory parent before task admission");
if (!remote) check("unassigned-initially-absent", await absent(outside), "The isolated cross-root marker did not already exist");
const prompt = (outsidePath: string) => piNativeMemoryPrompt(retained, outsidePath);
await create(execution.task.buildTitle(nonce), remote ? fixture => { if (!fixture.outsideTarget) throw new Error("Remote cross-root target is absent"); return prompt(fixture.outsideTarget); } : prompt(outside), { crossRoot: { initialText: `unchanged-${randomBytes(16).toString("hex")}` } });
await settle(1);
const personal = await api.get<Row>(`/api/agents/${fixtures.agent.id}/instructions-bundle/file?path=${encodeURIComponent(PI_NATIVE_MEMORY_PATH)}`);
const firstEvents = await events(runs[0]!.id);
await input.evidence("pi-agent-files-first-save.json", { personal, run: runs[0], events: firstEvents });
check("native-memory-write-verification", hasPiNativeMemoryRead(firstEvents, retained, remote ? { agentId: fixtures.agent.id, runId: runs[0]!.id } : undefined), "One completed native read verified the write, with no shell execution");
check("registered-file-saved", gradePiNativeMemory(personal.content, retained), "Public managed-file API contains the exact native-write bytes");
check("stopped-save-receipt", firstEvents.some(row => row.eventType === "instruction_save" && row.payload?.state === "saved"), "Provider stop produced a durable file-save receipt");
const crossRootIntact = remote ? hasUnchangedPiRemoteTarget(currentBaseline, await finishRemote("cross-root-final"), "@cross-root") : await absent(outside);
check("cross-root-denied", crossRootIntact && hasPiCrossRootDenial(firstEvents), "A single native write recorded the exact cross-root denial reason and the isolated target remains unchanged");
await input.restart();
await create(`Pi read persisted memory ${nonce}`, [
`Use native read once with path <AGENT_HOME>/${PI_NATIVE_MEMORY_PATH}. Replace <AGENT_HOME> with the canonical absolute directory from this turn\'s runtime instructions. Read the complete file without offset or limit; do not infer its contents from another task, conversation, or history.`,
"Use native write to copy those exact bytes into pi-agent-memory-proof.txt in the task workspace. This is an internal assertion file, not a deliverable. Do not change personal memory or other agent files. Do not use bash or the instructions API.",
piNativeFinish(execution.task.buildVisibleMarker(nonce)),
].join("\n"), { targets: ["pi-agent-memory-proof.txt"] });
await settle(2);
if (remote) await finishRemote("memory-readback-final");
check("fresh-run-readback", gradePiNativeMemory(await readWorkspace("pi-agent-memory-proof.txt"), retained), "A new issue after server restart copied the undisclosed saved agent-file bytes");
const current = await api.get<Row>(`/api/agents/${fixtures.agent.id}/instructions-bundle/file?path=${encodeURIComponent(PI_NATIVE_MEMORY_PATH)}`);
check("persistent-bytes-unchanged", current.content === personal.content, "Fresh-run readback preserved the saved managed bytes");
const parentSeed = await api.get<Row>(`/api/agents/${fixtures.agent.id}/instructions-bundle/file?path=${encodeURIComponent(PI_NATIVE_MEMORY_PARENT_SEED_PATH)}`);
check("memory-parent-seed-unchanged", parentSeed.content === PI_NATIVE_MEMORY_PARENT_SEED_CONTENT, "Both native turns preserved the parent setup file");
const freshEvents = await events(runs[1]!.id);
await input.evidence("pi-agent-files-fresh-read.json", { current, runs, events: freshEvents });
check("native-memory-fresh-read", hasPiNativeMemoryRead(freshEvents, retained, remote ? { agentId: fixtures.agent.id, runId: runs[1]!.id } : undefined), "The fresh run used one completed native read, with no shell execution");
} else if (execution.task.id === "human-permission-denial") {
if (!input.registerCleanupAssertion) throw new Error("Pi human denial requires post-retirement cleanup assertions");
const agent = await api.get<Row>(`/api/agents/${fixtures.agent.id}`);
const configured = await api.patch<Row>(`/api/agents/${fixtures.agent.id}`, { adapterConfig: { ...agent.adapterConfig, acpxPermissionMode: "approve-reads", lifecycleMode: "per_turn", timeoutSec: 120 } });
check("human-denial-policy", configured.adapterConfig.acpxPermissionMode === "approve-reads" && configured.adapterConfig.lifecycleMode === "per_turn", "Native write requires a browser permission decision in an owned per-turn process");
const localTarget = remote ? null : await createDeniedTargetFixture(input.workspacePath, "pi-human-denied.txt");
const target = localTarget?.targetRelativePath ?? "pi-human-denied.txt", path = join(input.workspacePath, target);
const watcher = localTarget?.watcher ?? null, observer = remote ? null : observeRunProcesses();
let processError = false, processAuthority: string | undefined;
const observe = () => {
const run = runs[0]; const authority = run?.processPid ? { pid: run.processPid, groupId: run.processGroupId, startedAt: run.processStartedAt, runId: run.id } : undefined;
if (authority) { const key = JSON.stringify(authority); if (processAuthority && processAuthority !== key) processError = true; processAuthority ??= key; }
return observer ? observer.sample(authority) : { captured: false, live: [] as number[], journal: [] };
};
let processes = observe();
const timer = remote ? undefined : setInterval(() => { try { processes = observe(); } catch { processError = true; } }, 250);
if (!remote) input.registerCleanupAssertion(async () => {
try {
await load(); processes = observe();
if (processes.captured && processes.live.length) processes = await pollUntil({ label: "Pi denied-write provider retirement", deadlineAt: Date.now() + 5000, load: async () => observe(), accept: value => value.live.length === 0 });
const fileAbsent = await absent(path), journal = watcher!.finish();
const passed = runs.length === 1 && runs[0]!.status === "succeeded" && processes.captured && processes.live.length === 0 && !processError && fileAbsent && journal.complete && journal.targetMutationCount === 0;
await input.evidence("pi-human-denial-retirement.json", { processes, processError, fileAbsent, journal, passed });
if (!passed) throw new Error("Pi human denial lacks independent no-effect and provider-retirement proof");
return [{ id: "human-denial-through-retirement", passed, detail: "Browser-denied target stayed absent through exact owned provider-process retirement" }];
} finally { clearInterval(timer); await input.evidence("pi-human-denial-cleanup-attempt.json", { target, processes, processError, journal: watcher!.finish() }); }
});
if (!remote) check("human-target-initially-absent", await absent(path), "Independent target is absent before provider work");
await create(execution.task.buildTitle(nonce), localTarget ? bindDeniedTargetPrompt(execution.task.buildPrompt(nonce), "pi-human-denied.txt", target) : execution.task.buildPrompt(nonce), { targets: [target] });
if (remote) bootstrapApproval = await approvePiBootstrapRead({ api, fixture: currentRemote!, companyId: fixtures.company.id, issueId: issue.id, runId: runs[0]!.id,
deadlineAt: input.deadlineAt, load: async () => { await load(); return { run: runs[0]!, issue, events: await events(runs[0]!.id) }; }, evidence: input.evidence });
if (remote) check("human-target-initially-absent", currentBaseline?.targets[target]?.absent === true && currentBaseline.targets[target]!.complete, "Remote watcher was armed before action publication with an absent target");
const pending = await pollUntil({ label: "Pi native browser permission", deadlineAt: input.deadlineAt, load: async () => { const state = await load(); processes = observe(); return { ...state, events: state.runs.length === 1 ? await events(state.runs[0]!.id) : [] }; }, reject: rejectFailure,
accept: state => state.runs.length === 1 && piPermissionRequests(withoutApprovedPiBootstrapRequests(state.events, bootstrapApproval), state.runs[0]!.id).length === 1 });
const native = piPermissionRequests(withoutApprovedPiBootstrapRequests(pending.events, bootstrapApproval), pending.runs[0]!.id)[0]!;
const identity = { runId: pending.runs[0]!.id, turnId: native.event.turnId, requestId: native.request.requestId, toolCallId: native.request.details.toolCallId, target };
check("human-target-pending-absent", remote ? (await currentRemote!.snapshot("permission-pending")).targets[target]?.absent === true : await absent(path), "Pending native write has no file effect");
await page.reload();
const before = await events(identity.runId);
check("human-permission-reconnect", piPermissionRequests(before, identity.runId).some(value => value.request.requestId === identity.requestId) && !before.some(row => row.payload?.prpEvent?.payload?.requestId === identity.requestId && ["runtime_request.resolved", "runtime_request.expired", "runtime_request.cancelled"].includes(row.eventType)), "Reload retained the exact unanswered native permission");
const declineLabel = native.request.choices.find((choice: Row) => choice.key === "decline").label;
// Resolved setup-read receipts remain visible but have no decision
// buttons. Require one actionable card and verify its exact POST below.
const card = page.getByTestId("task-chat-runtime-request").filter({ visible: true })
.filter({ has: page.getByRole("button", { name: declineLabel, exact: true }) });
await expect(card).toHaveCount(1);
await input.capture("pi-human-denial", "Pi native permission awaiting browser denial", "pi-human-denial.png");
const route = `/api/heartbeat-runs/${identity.runId}/runtime-requests/${encodeURIComponent(identity.requestId)}/resolve`;
const posted = page.waitForRequest(request => new URL(request.url()).pathname === route && request.method() === "POST");
await card.getByRole("button", { name: declineLabel, exact: true }).click();
const response = (await posted).postDataJSON();
check("human-exact-browser-decline", response.turnId === identity.turnId && response.requestKind === "permission_approval" && response.resolution?.action === "decline", "Actual browser POST declines this run, turn and request");
const final = await settle(1), runEvents = await events(identity.runId);
check("human-native-denial-delivered", hasDeliveredPiDenial(runEvents, identity), "Public durable delivery and same native write failure prove actual rejection");
check("human-target-terminal-absent", remote ? hasUnchangedPiRemoteTarget(currentBaseline, await finishRemote("human-denial-final"), target) : await absent(path), "The target remains absent after negative-test completion");
await input.evidence("pi-human-denial.json", { identity, final, events: runEvents });
} else if (execution.task.id === "restrictive-denial") {
const agent = await api.get<Row>(`/api/agents/${fixtures.agent.id}`);
await api.patch(`/api/agents/${fixtures.agent.id}`, { adapterConfig: { ...agent.adapterConfig, acpxPermissionMode: "deny-all" } });
const deniedPath = join(input.workspacePath, "pi-denied.txt");
check("denied-file-initially-absent", await absent(deniedPath), "The isolated denied file did not already exist");
await create(execution.task.buildTitle(nonce), execution.task.buildPrompt(nonce));
await settle(1); const runEvents = await events(runs[0]!.id);
check("restrictive-native-denial", await absent(deniedPath) && hasFailedPiWrite(runEvents, "pi-denied.txt"), "A correlated failed native edit and independent absent file prove restrictive denial");
await input.evidence("pi-restrictive-denial.json", { permissionMode: "deny-all", fileAbsent: true, run: runs[0], events: runEvents });
} else throw new Error(`Unknown Pi native flow ${execution.task.id}`);
// The specialized flow bypasses the standard task collector. Retain its
// required terminal API snapshot before declaring the fixture complete.
const final = await load();
const comments = await api.get<Row[]>(`/api/issues/${issue.id}/comments`);
const runEventsByRun = await Promise.all(runs.map(async run => ({ runId: run.id, events: await events(run.id) })));
await input.evidence("api-state.json", {
...final, run: runs.at(-1), comments, checks, runEventsByRun,
});
await input.capture("final-state", "Pi native fixture verified", "final-state.png");
return { issue, runs, checks };
} finally { await input.evidence("pi-native-checks.json", { issue, runs, checks }); }
}
tests/runner-e2e/pi-native-human-flow.test.ts
+85
View File
@@ -0,0 +1,85 @@
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { expect, it, vi } from "vitest";
import { piNativeTasks } from "./pi-native-cases.js";
import { runPiNativeFlow } from "./pi-native-flow.js";
vi.mock("./pi-bootstrap-permission.js", async importOriginal => ({ ...await importOriginal<typeof import("./pi-bootstrap-permission.js")>(), approvePiBootstrapRead: async () => undefined }));
const proof = vi.hoisted(() => ({ mutation: false, incomplete: false, live: false, target: "pi-human-denied.txt", prompt: "" }));
vi.mock("./user-actions.js", () => ({ createTaskThroughUi: vi.fn(async (input: { prompt: string; requireExplicitTitle?: boolean }) => { expect(input.requireExplicitTitle).toBe(true); proof.prompt = input.prompt; return { submittedAtMs: Date.now(), issueId: "issue" }; }) }));
vi.mock("./copilot-local-fixtures.js", async importOriginal => {
const actual = await importOriginal<typeof import("./copilot-local-fixtures.js")>();
return { ...actual,
createDeniedTargetFixture: async (workspace: string, name: string) => {
const fixture = await actual.createDeniedTargetFixture(workspace, name); proof.target = fixture.targetRelativePath;
return { ...fixture, watcher: { finish: () => ({ ...fixture.watcher.finish(), complete: !proof.incomplete && fixture.watcher.finish().complete, targetMutationCount: proof.mutation ? 1 : 0 }) } };
},
observeRunProcesses: () => ({ sample: () => ({ captured: true, live: proof.live ? [123] : [], journal: [] }) }),
}; });
vi.mock("@playwright/test", () => ({ expect: (actual: any, message?: string) => ({
toBe: (value: unknown) => expect(actual, message).toBe(value), toBeVisible: async () => {}, toHaveCount: async (value: number) => expect(actual.count).toBe(value),
}) }));
const wrap = (eventType: string, seq: number, payload: unknown) => ({ runId: "run", protocolSchemaVersion: 1, eventType, seq, payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", runId: "run", turnId: "turn", eventType, payload } } });
async function exercise(mutation: boolean, remote = false, adapter = "acpx-runtime", incomplete = false, duplicatePermission = false) {
proof.mutation = mutation; proof.incomplete = incomplete; proof.live = false; proof.target = "pi-human-denied.txt"; proof.prompt = "";
const workspacePath = await mkdtemp(join(tmpdir(), "pi-human-fixture-"));
let declined = false, browserPosts = 0; const saved = new Map<string, any>(); const cleanup: Array<() => Promise<any>> = [];
const task = piNativeTasks.find(row => row.id === "human-permission-denial")!;
const issue = () => ({ id: "issue", identifier: "PI-1", title: "Provider-generated task name", companyId: "company", assigneeAgentId: "agent", status: declined ? "done" : "in_progress" });
const run = () => ({ id: "run", status: declined ? "succeeded" : "running", runtimeMode: "native", processPid: 123, processGroupId: 123, processStartedAt: "2026-09-29T00:00:00Z" });
const request = { requestId: "request", turnId: "turn", type: "permission", status: "pending", details: { toolCallId: "pi-tool-1" }, origin: { adapter, provider: "pi", method: "session/request_permission" }, choices: [{ key: "decline", label: "Decline" }] };
const events = () => [wrap("runtime_request.created", 1, { request }), ...(declined ? [wrap("runtime_request.resolved", 2, { requestId: "request", turnId: "turn", action: "decline" }), wrap("tool.execution.completed", 3, { schema: "paperclip.tool.execution.v1", transport: "builtin", operation: "edit", executionId: "pi-tool-1", name: "write", target: proof.target, status: "failed", output: "Pi operation was denied or cancelled" })] : [])];
const api = {
post: async () => ({ name: "Pi fixture project" }), patch: async (_path: string, value: any) => value,
get: async (path: string) => {
if (path === "/api/agents/agent") return { adapterConfig: {} };
if (path.endsWith("/issues?limit=100")) return [issue()];
if (path === "/api/issues/issue") return issue();
if (path.endsWith("/heartbeat-runs?limit=100")) return [run()];
if (path === "/api/heartbeat-runs/run") return run();
if (path.endsWith("/interactions") || path.endsWith("/comments")) return [];
if (path.includes("/events?")) return events();
throw new Error(`Unexpected fixture API ${path}`);
},
};
const browserRequest = { url: () => "http://fixture/api/heartbeat-runs/run/runtime-requests/request/resolve", method: () => "POST", postDataJSON: () => ({ turnId: "turn", requestKind: "permission_approval", resolution: { action: "decline" } }) };
let resolvePost: ((value: typeof browserRequest) => void) | undefined;
const card = (actionable = false): any => ({
filter: (options: { has?: { name: string } }) => {
if (options.has) expect(options.has.name).toBe("Decline");
return card(actionable || Boolean(options.has));
},
// The resolved bootstrap receipt is visible but cannot be declined again.
count: actionable ? (duplicatePermission ? 2 : 1) : (remote ? 2 : 1),
getByRole: (_role: string, options: { name: string }) => ({ click: async () => { expect(options.name).toBe("Decline"); browserPosts++; declined = true; resolvePost!(browserRequest); } }),
});
const page = { goto: async () => {}, reload: async () => {}, waitForRequest: (predicate: (v: typeof browserRequest) => boolean) => new Promise(resolve => { expect(predicate(browserRequest)).toBe(true); resolvePost = resolve; }),
getByRole: (_role: string, options: { name: string }) => ({ name: options.name }),
getByTestId: (id: string) => id === "issue-detail-header" ? { getByRole: () => ({}) } : card() };
const nativeRoot = { pid: 456, ppid: 1, startTicks: "123", bootId: "remote-boot" };
const snapshot = () => ({ observedAtMs: declined ? 2 : 1, complete: true, workspace: {}, targets: { "pi-human-denied.txt": { absent: true, sha256: null, parent: { dev: "1", ino: "2" }, mutationCount: mutation ? 1 : 0, complete: true } }, watcher: { complete: true, targetMutationCount: mutation ? 1 : 0, workspaceMutationCount: 0 }, processes: { captured: true, root: nativeRoot, journal: [nativeRoot], live: declined ? [] : [456] } });
const fixture = { binding: { runId: "run" }, remoteCwd: "/remote", outsideTarget: null, snapshot: async () => snapshot(), finish: async () => snapshot(), close: async () => {} };
const remoteBootstrap = remote ? { prompt: () => "Read bootstrap only", bindAndRelease: async (input: any) => { expect(input.targets).toEqual(["pi-human-denied.txt"]); expect(await input.actionPrompt(fixture)).toContain("native write exactly once"); return fixture; } } : undefined;
try {
if (remote) await writeFile(join(workspacePath, "pi-human-denied.txt"), "POISON HOST COPYBACK");
const call = runPiNativeFlow({ page, api, fixtures: { company: { id: "company", issuePrefix: "PI" }, agent: { id: "agent", name: "Pi" }, environment: { id: "environment" } }, execution: { task, environment: { id: remote ? "daytona" : "local" }, profile: { qualificationCandidate: "pi" } }, workspacePath, nonce: "fixture", deadlineAt: Date.now() + 2000, restart: async () => {}, observe: () => {}, capture: async () => {}, evidence: async (name: string, value: unknown) => { saved.set(name, value); }, remoteBootstrap, registerCleanupAssertion: (fn: () => Promise<any>) => cleanup.push(fn) } as any);
if (duplicatePermission) { await expect(call).rejects.toThrow(); expect(browserPosts).toBe(0); return; }
const result = await call;
if (!remote) { expect(proof.target).toMatch(/^pc-denied-[a-zA-Z0-9]+\/pi-human-denied\.txt$/); expect(proof.prompt).toContain(`relative path ${proof.target} and content forbidden`); }
expect(browserPosts).toBe(1); expect(result.checks.every(check => check.passed)).toBe(true); expect(saved.has("api-state.json")).toBe(true);
expect(cleanup).toHaveLength(1);
if (mutation || incomplete) await expect(cleanup[0]!()).rejects.toThrow("no-effect");
else expect((await cleanup[0]!())[0].passed).toBe(true);
expect(saved.get(remote ? "pi-remote-1-retirement.json" : "pi-human-denial-retirement.json").passed).toBe(!mutation && !incomplete);
} finally { await rm(workspacePath, { recursive: true, force: true }); }
}
it("drives actual browser-denial flow and independent retirement proof", async () => exercise(false));
it("rejects an observed create/delete even when final target is absent", async () => exercise(true));
it("proves browser denial against remote watcher and retirement without trusting a host file", async () => exercise(false, true));
it.each([false, true])("drives sidecar permission denial with remote=%s", async remote => exercise(false, remote, "acpx-runtime-sidecar"));
it("rejects incomplete local observation even with zero target mutations", async () => exercise(false, false, "acpx-runtime", true));
it("refuses browser denial when two actionable cards remain beside resolved setup history", async () => exercise(false, true, "acpx-runtime", false, true));
tests/runner-e2e/pi-native-permission.test.ts
+70
View File
@@ -0,0 +1,70 @@
import { expect, it } from "vitest";
import { hasDeliveredPiDenial, piPermissionRequests } from "./pi-native-evidence.js";
const identity = { runId: "run", turnId: "turn", requestId: "permission", toolCallId: "pi-tool-1", target: "pi-human-denied.txt" };
function evidence(adapter = "acpx-runtime") {
const wrap = (eventType: string, seq: number, payload: unknown) => ({ runId: "run", protocolSchemaVersion: 1, eventType, seq, payload: { prpEvent: {
schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", runId: "run", turnId: "turn", eventType, payload,
} } });
return [wrap("runtime_request.created", 1, { request: { requestId: "permission", turnId: "turn", type: "permission", status: "pending", details: { toolCallId: "pi-tool-1" },
origin: { adapter, provider: "pi", method: "session/request_permission" }, choices: [{ key: "decline" }] } }),
wrap("runtime_request.resolved", 2, { requestId: "permission", turnId: "turn", action: "decline" }),
wrap("tool.execution.completed", 3, { schema: "paperclip.tool.execution.v1", executionId: "pi-tool-1", transport: "builtin", operation: "edit", name: "write", status: "failed", target: identity.target, output: "Pi operation was denied or cancelled" })];
}
it("requires the actual correlated durable browser denial and native tool failure", () => {
expect(piPermissionRequests(evidence(), "run")).toHaveLength(1);
expect(hasDeliveredPiDenial(evidence(), identity)).toBe(true);
for (const rows of [[], evidence().slice(1), evidence().slice(0, 2), [...evidence(), evidence()[2]!]]) expect(hasDeliveredPiDenial(rows, identity)).toBe(false);
});
it("rejects foreign identities, approvals, expiration, write success and policy-only failure", () => {
for (const key of ["runId", "turnId", "requestId", "toolCallId", "target"]) expect(hasDeliveredPiDenial(evidence(), { ...identity, [key]: "other" })).toBe(false);
for (const [index, patch] of [[1, { action: "accept" }], [2, { status: "completed" }], [2, { output: "outside its assigned workspace" }], [2, { executionId: "other" }], [2, { target: "other.txt" }]] as const) {
const rows = evidence(); Object.assign(rows[index]!.payload.prpEvent.payload as object, patch); expect(hasDeliveredPiDenial(rows, identity)).toBe(false);
}
for (const eventType of ["runtime_request.expired", "runtime_request.cancelled"]) {
const rows = evidence(); rows[1]!.eventType = eventType; rows[1]!.payload.prpEvent.eventType = eventType;
expect(hasDeliveredPiDenial(rows, identity)).toBe(false);
}
const malformed = evidence(); (malformed[0]!.payload.prpEvent.payload as any).request.choices = [null, {}];
expect(piPermissionRequests(malformed, "run")).toEqual([]);
const stale = evidence(); stale[1]!.seq = 0; expect(hasDeliveredPiDenial(stale, identity)).toBe(false);
const wrong = evidence(); wrong[2]!.payload.prpEvent.sourceKind = "agent"; expect(hasDeliveredPiDenial(wrong, identity)).toBe(false);
});
it("does not misorder native notifications that arrive before the public delivery receipt is persisted", () => {
const rows = evidence(); rows[1]!.seq = 3; rows[2]!.seq = 2;
// Both native failed invocation and post-write delivery receipt are required;
// asynchronous persistence order is not treated as native execution order.
expect(hasDeliveredPiDenial(rows, identity)).toBe(true);
});
import { hasPiRemoteRetirement, hasUnchangedPiRemoteTarget } from "./pi-native-evidence.js";
it("remote denial requires complete watching, stable parent and real process retirement before lease deletion", () => {
const root = { pid: 12, startTicks: "100", bootId: "boot" };
const before = { observedAtMs: 1, complete: true, targets: { denied: { absent: true, sha256: null, parent: { dev: "1", ino: "2" }, mutationCount: 0, complete: true } } };
const after = { ...structuredClone(before), observedAtMs: 2, watcher: { complete: true }, processes: { captured: true, root, journal: [root], live: [] as number[] } };
expect(hasUnchangedPiRemoteTarget(before, after, "denied")).toBe(true);
for (const broken of [{ ...after, complete: false }, { ...after, processes: { ...after.processes, live: [12] } }, { ...after, processes: { ...after.processes, captured: false } }, { ...after, processes: { ...after.processes, journal: [] } }, { ...after, watcher: { complete: false } }]) expect(hasUnchangedPiRemoteTarget(before, broken, "denied")).toBe(false);
for (const change of [{ mutationCount: 1 }, { absent: false }, { parent: { dev: "1", ino: "3" } }, { complete: false }]) {
expect(hasUnchangedPiRemoteTarget(before, { ...after, targets: { denied: { ...after.targets.denied, ...change } } }, "denied")).toBe(false);
}
expect(hasPiRemoteRetirement({ sandboxDeleted: true })).toBe(false);
expect(hasPiRemoteRetirement({ ...after, watcher: { complete: false } })).toBe(false);
const existing = structuredClone(before); existing.targets.denied.absent = false; (existing.targets.denied as any).sha256 = `sha256:${"a".repeat(64)}`;
expect(hasUnchangedPiRemoteTarget(existing, { ...after, targets: existing.targets }, "denied")).toBe(true);
});
it.each(["acpx-runtime", "acpx-runtime-sidecar"])("requires exact Pi permission origin and identity via %s", adapter => {
expect(piPermissionRequests(evidence(adapter), "run")).toHaveLength(1);
expect(hasDeliveredPiDenial(evidence(adapter), identity)).toBe(true);
for (const patch of [{ adapter: "unknown" }, { adapter: "acpx-runtime-sidecar-unknown" }, { adapter: "semantic" }, { provider: "cursor" }, { method: "request_human_input" }]) {
const rows = evidence(adapter);
Object.assign((rows[0]!.payload.prpEvent.payload as any).request.origin, patch);
expect(piPermissionRequests(rows, "run")).toHaveLength(0);
expect(hasDeliveredPiDenial(rows, identity)).toBe(false);
}
for (const key of ["runId", "turnId", "requestId", "toolCallId", "target"]) {
expect(hasDeliveredPiDenial(evidence(adapter), { ...identity, [key]: "foreign" })).toBe(false);
}
const rows = evidence(adapter);
expect(hasDeliveredPiDenial([...rows, rows[0]!], identity)).toBe(false);
expect(hasDeliveredPiDenial([...rows, rows[1]!], identity)).toBe(false);
});
tests/runner-e2e/pi-native-provider-death-flow.test.ts
+116
View File
@@ -0,0 +1,116 @@
import { expect, it, vi } from "vitest";
import { gradePiProviderDeath, runPiPendingProviderDeath, PI_DEATH_MARKER } from "./pi-native-provider-death-flow.js";
import { classifyFailure } from "./failure-classifier.js";
import { observePiPendingNativeInput } from "./pi-native-restart-flow.js";
type Row = Record<string, any>;
vi.mock("@playwright/test", () => ({ expect: (locator: { count(): number }) => ({
toHaveCount: async (expected: number) => expect(locator.count()).toBe(expected),
toBeVisible: async () => expect(locator.count()).toBe(1),
}) }));
function fixture(fallback = true) {
const question = { schema: "paperclip.question_set.v1", questions: [{ id: "answer", header: "Pi native provider death", prompt: "Answer", answerMode: "text", required: true }] };
const request = { schema: "paperclip.runtime_request.v2", requestKind: "runtime", type: "input", status: "pending", requestId: "request", turnId: "turn", itemId: "request", input: question,
origin: { adapter: "acpx-runtime-sidecar", provider: "pi", method: "elicitation/create" } };
const wrap = (eventType: string, sourceSeq: number, payload: Row): Row => ({ companyId: "company", runId: "run", seq: sourceSeq, protocolSchemaVersion: 1, eventType,
payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", eventType, runId: "run", turnId: "turn", itemId: "request",
normalizedSessionId: "session", sourceInstanceId: "runner", sourceSeq, sourceEventId: `runner:run:${sourceSeq}`, payload } } });
const state = { issue: { id: "issue", companyId: "company", status: "in_progress" }, runs: [{ id: "run", companyId: "company", nativeIssueId: "issue", runtimeMode: "native", status: "running", nativeSessionId: "session", runnerInstanceId: "runner" } as Row],
interactions: [{ id: "original", companyId: "company", issueId: "issue", kind: "ask_user_questions", sourceRunId: "run", status: "pending", result: null,
continuationPolicy: "none", resolverPolicy: "human_only", idempotencyKey: "paperclip-runner-question:run:request", payload: { runtimeRequestId: "request", questionSet: question } } as Row] };
const events = [wrap("turn.started", 1, {}), wrap("runtime_request.created", 2, { request })];
const pending = observePiPendingNativeInput(state, events, "company", "Pi native provider death");
const before = structuredClone({ state, events });
state.runs[0]!.status = "failed"; state.issue.status = "blocked"; state.interactions[0]!.status = "expired";
events.push(wrap("runtime_request.expired", 3, { requestId: "request", requestKind: "runtime", requestType: "input", turnId: "turn", itemId: "request", reason: "provider_process_lost", replayAllowed: false, request }),
wrap("turn.failed", 4, { code: "provider_transport_lost" }));
if (fallback) state.interactions.push({ ...structuredClone(state.interactions[0]!), id: "fallback", status: "pending", continuationPolicy: "wake_assignee", idempotencyKey: "runtime-input-durable:v1:run:request" });
state.interactions[0]!.result = { version: 1, answers: [], ...(fallback ? { expirationReason: "superseded_by_newer_interaction", supersededByInteractionId: "fallback" } : {}) };
const grade = () => gradePiProviderDeath(state, events, pending, question);
return { state, events, pending, question, wrap, grade, before };
}
it.each([false, true])("accepts production loss with distinct fallback=%s without claiming restoration", fallback => {
expect(fixture(fallback).grade()).toEqual({ originalCardExpired: true, replayAllowed: false, originalProviderRestored: false, fallbackInteractionId: fallback ? "fallback" : null });
});
it.each([
["wrong issue disposition", (f: ReturnType<typeof fixture>) => { f.state.issue.status = "in_progress"; }],
["completed issue", (f: ReturnType<typeof fixture>) => { f.state.issue.status = "done"; }],
["answered expired card", (f: ReturnType<typeof fixture>) => { f.state.interactions[0]!.result.answers = [{ questionId: "answer", otherText: "stale" }]; }],
["missing expired result", (f: ReturnType<typeof fixture>) => { f.state.interactions[0]!.result = null; }],
["foreign supersession", (f: ReturnType<typeof fixture>) => { f.state.interactions[0]!.result.supersededByInteractionId = "other"; }],
["missing expiry", (f: ReturnType<typeof fixture>) => { f.events.splice(2, 1); }],
["model-only claim", f => { f.events[2] = { eventType: "assistant.message", payload: { text: "The callback expired" } }; }],
["controller injected expiry", f => { f.events[2]!.payload.prpEvent.sourceKind = "control_plane"; }],
["cancelled instead of lost", f => { f.events[2]!.eventType = f.events[2]!.payload.prpEvent.eventType = "runtime_request.cancelled"; }],
["resolved instead of lost", f => { f.events[2]!.eventType = f.events[2]!.payload.prpEvent.eventType = "runtime_request.resolved"; }],
["normal durable handoff", f => { f.events[2]!.payload.prpEvent.payload.reason = "durable_handoff"; }],
["replay allowed", f => { f.events[2]!.payload.prpEvent.payload.replayAllowed = true; }],
["foreign request", f => { f.events[2]!.payload.prpEvent.payload.requestId = "other"; }],
["foreign item", f => { f.events[2]!.payload.prpEvent.payload.itemId = "other"; }],
["missing canonical request", f => { delete f.events[2]!.payload.prpEvent.payload.request; }],
["replaced canonical question", f => { f.events[2]!.payload.prpEvent.payload.request = { ...f.events[2]!.payload.prpEvent.payload.request, input: {} }; }],
["duplicate expiry", f => { f.events.push(f.wrap("runtime_request.expired", 5, f.events[2]!.payload.prpEvent.payload)); }],
["new native turn", f => { f.events.push(f.wrap("turn.started", 5, {})); }],
["successful provider", f => { f.state.runs[0]!.status = "succeeded"; }],
["Stop cancellation", f => { f.state.runs[0]!.resultJson = { nativeCancellation: {} }; }],
["worker failure without turn failure", f => { f.events.pop(); }],
["completed turn", f => { f.events[3]!.eventType = f.events[3]!.payload.prpEvent.eventType = "turn.completed"; }],
["terminal before expiry", f => { f.events[3]!.seq = 0; }],
["second run", f => { f.state.runs.push({ ...f.state.runs[0], id: "other" }); }],
["restored session", f => { f.state.runs[0]!.nativeSessionId = "restored"; }],
["foreign company", f => { f.events[2]!.companyId = "foreign"; }],
["foreign source", f => { f.events[2]!.payload.prpEvent.sourceInstanceId = "foreign"; }],
["original still pending", f => { f.state.interactions[0]!.status = "pending"; }],
["answered original", f => { f.state.interactions[0]!.status = "answered"; }],
["rebound original card", f => { f.state.interactions[0]!.sourceRunId = "other"; }],
["duplicate fallback", f => { f.state.interactions.push(structuredClone(f.state.interactions[1]!)); }],
["fallback masquerading as callback", f => { f.state.interactions[1]!.continuationPolicy = "none"; }],
["answered fallback", f => { f.state.interactions[1]!.status = "answered"; }],
["foreign fallback", f => { f.state.interactions[1]!.idempotencyKey = "foreign"; }],
] satisfies Array<[string, (f: ReturnType<typeof fixture>) => void]>)("rejects %s", (_name, mutate) => {
const f = fixture(); mutate(f); expect(f.grade).toThrow();
});
async function journey(staleStatus = 409, markerEffect = false) {
const f = fixture(), evidence = new Map<string, unknown>(), posted: string[] = [];
let lost = false, signals = 0, retired = false;
Object.assign(f.before.state.runs[0]!, { contextSnapshot: { executionWorkspaceId: "workspace", paperclipEnvironment: { leaseId: "lease" } },
runnerProfileJson: { nativeExecutionInput: { binding: { executionWorkspaceId: "workspace" } } } });
const root = { pid: 21, startTicks: "100", bootId: "boot" };
const composer: any = { count: () => 1, filter: () => composer };
const page: any = { getByTestId: (id: string) => id === "issue-detail-header" ? { getByRole: (_role: string, options: Row) => {
expect(options).toEqual({ name: "Change status (current: Blocked", exact: false }); return { count: () => 1 };
} } : composer, reload: async () => { expect(retired).toBe(true); },
locator: (selector: string) => { expect(selector).toBe("#interaction-original"); return { count: () => 1, locator: () => ({ count: () => 0 }) }; } };
const remoteFixture: any = { binding: { companyId: "company", runId: "run", leaseId: "lease" }, baseline: { scope: { observedPrpEnvironmentLeaseId: "workspace" } },
terminatePiProvider: async (lease: string) => { expect(lease).toBe("workspace"); expect(lost).toBe(false); signals++; lost = true; return { targetKind: "pi_native_child", signalled: true }; },
finish: async () => { expect(lost).toBe(true); retired = true; return { complete: true, observedAtMs: 1, watcher: { complete: true },
processes: { captured: true, root, journal: [root], live: [] }, targets: { [PI_DEATH_MARKER]: { absent: !markerEffect, complete: true, mutationCount: markerEffect ? 1 : 0 } } }; } };
const result = runPiPendingProviderDeath({ page, api: { request: { post: async (path: string) => { expect(retired).toBe(true); posted.push(path); return { status: () => staleStatus }; } } } as any,
companyId: "company", deadlineAt: Date.now() + 1000, fixture: remoteFixture,
load: async () => structuredClone(lost ? f.state : f.before.state), events: async () => structuredClone(lost ? f.events : f.before.events),
capture: async () => {}, evidence: async (name, value) => { evidence.set(name, value); } });
return { result, evidence, posted, signalCount: () => signals };
}
it("drives one admitted fault, retirement, original-card UI and both stale public APIs in order", async () => {
const j = await journey(); expect((await j.result).every(check => check.passed)).toBe(true);
expect(j.signalCount()).toBe(1); expect(j.posted).toEqual(["/api/issues/issue/interactions/original/respond", "/api/heartbeat-runs/run/runtime-requests/request/resolve"]);
expect(j.evidence.get("pi-provider-death-result.json")).toMatchObject({ originalProviderRestored: false, fallbackInteractionId: "fallback", staleCardStatus: 409, staleRuntimeStatus: 409 });
});
it("does not accept a successful stale answer despite otherwise valid expiry receipts", async () => {
const j = await journey(200); await expect(j.result).rejects.toThrow("Both original durable-card"); expect(j.signalCount()).toBe(1);
});
it("rejects a continuation file effect before sending either stale answer", async () => {
const j = await journey(409, true); await expect(j.result).rejects.toThrow("Exact owned process tree"); expect(j.posted).toEqual([]);
});
it("classifies deliberate-loss proof failures as candidate failures, preserving real API transport errors", () => {
const f = fixture(); f.state.runs[0]!.status = "succeeded";
let observed: unknown;
try { f.grade(); } catch (error) { observed = error; }
expect(String(observed)).toContain("pi_provider_death_proof:");
expect(classifyFailure(observed)).toBe("candidate_failure");
expect(classifyFailure(new Error("GET controller returned 503 service unavailable"))).toBe("transient_infrastructure");
expect(classifyFailure(new Error("pi_provider_death_proof: secret leak"))).toBe("secret_leak");
expect(classifyFailure(new Error("pi_provider_death_proof: cleanup incomplete"))).toBe("cleanup_failure");
});
tests/runner-e2e/pi-native-provider-death-flow.ts
+135
View File
@@ -0,0 +1,135 @@
import { createHash, randomBytes } from "node:crypto";
import { expect, type Page } from "@playwright/test";
import { canonicalJson } from "../../packages/shared/src/portability-hash.js";
import { pollUntil, type RunnerApi } from "./api.js";
import { isValidNativePrpEnvelope } from "./native-event-envelope.js";
import { observePiPendingNativeInput, type PiRestartPending } from "./pi-native-restart-flow.js";
import { hasPiRemoteRetirement } from "./pi-native-evidence.js";
import type { RemoteNativeFixture, PiProviderDeathReceipt } from "./remote-native-fixtures.js";
type Row = Record<string, any>;
type State = { issue: Row; runs: Row[]; interactions: Row[] };
type Check = { id: string; passed: boolean; detail: string };
const hash = (value: unknown) => createHash("sha256").update(canonicalJson(value)).digest("hex");
const demand = (value: unknown, reason: string): void => { if (!value) throw new Error(`pi_provider_death_proof: ${reason}`); };
export const PI_DEATH_MARKER = "pi-provider-death-answer.json";
/** All events are public durable production receipts, never a fault fixture's events. */
export function gradePiProviderDeath(state: State, events: readonly Row[], pending: PiRestartPending, originalQuestion: unknown) {
demand(state.issue.id === pending.issueId && state.issue.companyId === pending.companyId && state.issue.status === "blocked" && state.runs.length === 1, "foreign issue, wrong blocked disposition or second run");
const run = state.runs[0]!;
demand(run.id === pending.runId && run.companyId === pending.companyId && run.nativeIssueId === pending.issueId && run.runtimeMode === "native"
&& run.nativeSessionId === pending.nativeSessionId && run.runnerInstanceId === pending.sourceInstanceId
&& run.status === "failed" && run.resultJson?.nativeCancellation == null && run.resultJson?.startupCancellation == null,
"original provider did not fail independently of Stop");
demand(Array.isArray(events) && events.length > 0 && events.length <= 20_000, "missing event history");
const rows = events.filter(row => row.payload?.prpEvent).sort((a, b) => a.seq - b.seq);
const ids = new Set<string>(), seqs = new Set<number>(), cursors = new Map<string, number>();
for (const row of rows) {
const e = row.payload.prpEvent;
demand(row.companyId === pending.companyId && row.runId === pending.runId && row.eventType === e.eventType
&& Number.isSafeInteger(row.seq) && row.seq > 0 && !seqs.has(row.seq) && isValidNativePrpEnvelope(e, row.protocolSchemaVersion)
&& e.runId === pending.runId && e.normalizedSessionId === pending.nativeSessionId
&& ((e.sourceKind === "runner" && e.sourceInstanceId === pending.sourceInstanceId)
|| (e.sourceKind === "control_plane" && e.sourceInstanceId === `${pending.sourceInstanceId}:control`))
&& e.sourceEventId === `${e.sourceInstanceId}:${pending.runId}:${e.sourceSeq}` && !ids.has(e.sourceEventId)
&& Number.isSafeInteger(e.sourceSeq) && e.sourceSeq > (cursors.get(e.sourceInstanceId) ?? 0), "foreign, duplicate or reordered event");
ids.add(e.sourceEventId); seqs.add(row.seq); cursors.set(e.sourceInstanceId, e.sourceSeq);
if (e.turnId != null) demand(e.turnId === pending.turnId, "replacement native turn");
}
const of = (kind: string) => rows.filter(row => row.eventType === kind);
const created = of("runtime_request.created");
demand(created.length === 1 && hash(created[0]) === pending.createdRowSha256, "original request changed or replayed");
const closures = rows.filter(row => ["runtime_request.expired", "runtime_request.resolved", "runtime_request.cancelled"].includes(row.eventType));
demand(closures.length === 1 && closures[0]!.eventType === "runtime_request.expired", "missing unique production expiry");
const expired = closures[0]!.payload.prpEvent, p = expired.payload;
demand(expired.sourceKind === "runner" && expired.sourceSeq > pending.createdSourceSeq && expired.turnId === pending.turnId
&& p.requestId === pending.requestId && p.requestKind === "runtime" && p.requestType === "input" && p.turnId === pending.turnId && p.itemId === pending.itemId
&& p.reason === "provider_process_lost" && p.replayAllowed === false && hash(p.request) === hash(created[0]!.payload.prpEvent.payload.request), "expiry is not the original lost Pi callback");
const failed = of("turn.failed");
demand(failed.length === 1 && failed[0]!.seq > closures[0]!.seq && failed[0]!.payload.prpEvent.sourceKind === "runner"
&& failed[0]!.payload.prpEvent.turnId === pending.turnId && !of("turn.completed").length && !of("turn.cancelled").length
&& of("turn.started").length === 1 && !of("run.result.proposed").length, "wrong terminal or replacement turn");
const card = state.interactions.find(card => card.id === pending.interactionId);
demand(card?.companyId === pending.companyId && card.issueId === pending.issueId && card.sourceRunId === pending.runId
&& card.idempotencyKey === `paperclip-runner-question:${pending.runId}:${pending.requestId}` && card.continuationPolicy === "none"
&& card.status === "expired" && card.result?.version === 1 && Array.isArray(card.result.answers) && card.result.answers.length === 0
&& card.kind === "ask_user_questions" && card.resolverPolicy === "human_only" && card.payload?.runtimeRequestId === pending.requestId && hash(card.payload.questionSet) === hash(originalQuestion), "original durable card did not expire");
const fallback = state.interactions.filter(card => card.id !== pending.interactionId);
demand(fallback.length <= 1, "duplicate fallback interaction");
if (fallback.length) {
const next = fallback[0]!;
demand(next.id !== pending.interactionId && next.companyId === pending.companyId && next.issueId === pending.issueId && next.sourceRunId === pending.runId
&& next.kind === "ask_user_questions" && next.status === "pending" && next.result == null && next.continuationPolicy === "wake_assignee"
&& next.idempotencyKey === `runtime-input-durable:v1:${pending.runId}:${pending.requestId}`
&& card!.result.expirationReason === "superseded_by_newer_interaction" && card!.result.supersededByInteractionId === next.id
&& next.payload?.runtimeRequestId === pending.requestId && hash(next.payload.questionSet) === hash(originalQuestion), "fallback was mistaken for restored input");
}
return { originalCardExpired: true, replayAllowed: false, originalProviderRestored: false, fallbackInteractionId: fallback[0]?.id ?? null };
}
export async function runPiPendingProviderDeath(input: {
page: Page; api: RunnerApi; companyId: string; deadlineAt: number; fixture: RemoteNativeFixture;
load(): Promise<State>; events(runId: string): Promise<Row[]>;
capture(id: string, label: string, file: string): Promise<void>; evidence(name: string, value: unknown): Promise<void>;
}): Promise<Check[]> {
const checks: Check[] = [];
const check = (id: string, passed: boolean, detail: string) => { checks.push({ id, passed, detail }); demand(passed, detail); };
let fault: PiProviderDeathReceipt | undefined;
const snapshot = async (name: string) => {
const state = await input.load(), events = state.runs.length === 1 ? await input.events(state.runs[0]!.id) : [];
await input.evidence(name, { ...state, events, fault }); return { state, events };
};
try {
await pollUntil({ label: "Pi native input before provider death", deadlineAt: input.deadlineAt, load: input.load,
accept: s => s.interactions.some(c => c.status === "pending"), reject: s => s.runs.length > 1 || s.runs.some(r => ["succeeded", "failed", "cancelled", "timed_out"].includes(r.status)) ? "Provider ended before fault admission" : undefined });
const before = await snapshot("pi-provider-death-before.json");
const pending = observePiPendingNativeInput(before.state, before.events, input.companyId, "Pi native provider death");
const run = before.state.runs[0]!, question = before.state.interactions[0]!.payload.questionSet;
demand(input.fixture.binding.runId === pending.runId && input.fixture.binding.companyId === pending.companyId && input.fixture.terminatePiProvider,
"missing exact owned Daytona fault capability");
// createRunnerdBackend's default PRP lease is the execution workspace ID;
// reject alternate/recovered authority instead of guessing that identity.
const workspaceId = run.contextSnapshot?.executionWorkspaceId;
demand(typeof workspaceId === "string" && workspaceId.length > 0
&& run.runnerProfileJson?.nativeExecutionInput?.binding?.executionWorkspaceId === workspaceId
&& run.contextSnapshot?.paperclipEnvironment?.leaseId === input.fixture.binding.leaseId
&& input.fixture.baseline.scope.observedPrpEnvironmentLeaseId === workspaceId, "native run/lease/workspace binding unavailable");
await expect(input.page.getByTestId("question-text-answer-composer").filter({ visible: true })).toHaveCount(1);
await input.capture("pi-provider-death-pending", "Unanswered native Pi input before provider loss", "pi-provider-death-pending.png");
// Refresh immediately before the one-shot irreversible fault.
const fresh = await snapshot("pi-provider-death-admission.json");
check("death-original-pending-input", hash(observePiPendingNativeInput(fresh.state, fresh.events, input.companyId, "Pi native provider death")) === hash(pending), "Fault admits only the unchanged original native input");
fault = await input.fixture.terminatePiProvider!(workspaceId);
await input.evidence("pi-provider-death-fault.json", fault);
await pollUntil({ label: "production expiry after Pi provider loss", deadlineAt: input.deadlineAt, load: input.load,
accept: s => s.issue.status === "blocked" && s.runs[0]?.status === "failed" && s.interactions.some(c => c.id === pending.interactionId && c.status === "expired"),
reject: s => s.runs.length !== 1 || s.runs.some(r => ["succeeded", "cancelled", "timed_out"].includes(r.status)) ? "Unexpected run after provider fault" : undefined });
const retired = await input.fixture.finish();
check("death-owned-retirement-no-effect", hasPiRemoteRetirement(retired) && retired.targets[PI_DEATH_MARKER]?.absent === true
&& retired.targets[PI_DEATH_MARKER]?.complete === true && retired.targets[PI_DEATH_MARKER]?.mutationCount === 0,
"Exact owned process tree retired while the unanswered continuation marker remained absent");
await input.evidence("pi-provider-death-retirement.json", retired);
await input.page.reload();
const final = await snapshot("pi-provider-death-expired.json"), result = gradePiProviderDeath(final.state, final.events, pending, question);
// Failed native terminals are permanent on the first attempt; the
// controller's nativeSessionRecoveryProjection projects Blocked while this
// run still owns the task (native-session-executor.ts).
await expect(input.page.getByTestId("issue-detail-header").getByRole("button", { name: "Change status (current: Blocked", exact: false })).toBeVisible();
const oldCard = input.page.locator(`#interaction-${pending.interactionId}`);
await expect(oldCard).toHaveCount(1);
await expect(oldCard.locator('textarea,[contenteditable="true"],button[type="submit"]')).toHaveCount(0);
await input.capture("pi-provider-death-expired", "Expired original input and any separate durable fallback", "pi-provider-death-expired.png");
const answer = `STALE-${randomBytes(16).toString("hex")}`;
const original = await input.api.request.post(`/api/issues/${pending.issueId}/interactions/${pending.interactionId}/respond`,
{ data: { answers: [{ questionId: pending.questionId, optionIds: [], otherText: answer }] } });
const runtime = await input.api.request.post(`/api/heartbeat-runs/${pending.runId}/runtime-requests/${encodeURIComponent(pending.requestId)}/resolve`,
{ data: { resolution: { action: "submit", response: { schema: "paperclip.question_response.v1", answers: { [pending.questionId]: { text: answer } } } } } });
check("death-stale-original-responses-rejected", original.status() === 409 && runtime.status() === 409, "Both original durable-card and native-runtime response APIs reject the stale answer");
const after = await snapshot("pi-provider-death-after-stale.json");
check("death-expiry-not-restoration", hash(gradePiProviderDeath(after.state, after.events, pending, question)) === hash(result)
&& hash(after.state.interactions) === hash(final.state.interactions), "Stale answers neither resolve/replay the original request nor start a new run; any fallback remains a separate unanswered interaction");
await input.evidence("pi-provider-death-result.json", { ...result, staleCardStatus: original.status(), staleRuntimeStatus: runtime.status(), paidProviderDeath: true });
return checks;
} finally { await input.evidence("pi-provider-death-checks.json", checks); }
}
tests/runner-e2e/pi-native-remote-flow.test.ts
+106
View File
@@ -0,0 +1,106 @@
import { createHash } from "node:crypto";
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { expect, it, vi } from "vitest";
import { piNativeTasks, PI_NATIVE_MEMORY_PARENT_SEED_PATH, PI_NATIVE_MEMORY_PARENT_SEED_CONTENT } from "./pi-native-cases.js";
import { runPiNativeFlow } from "./pi-native-flow.js";
import * as remoteFixtures from "./remote-native-fixtures.js";
const browser = vi.hoisted(() => ({ create: (_value: any) => ({ submittedAtMs: Date.now(), issueId: "issue-fixture" }) }));
vi.mock("./user-actions.js", () => ({ createTaskThroughUi: async (value: unknown) => browser.create(value) }));
vi.mock("@playwright/test", () => ({ expect: (value: unknown, message?: string) => ({ toBe: (expected: unknown) => expect(value, message).toBe(expected), toBeVisible: async () => {} }) }));
it.each([
{ missingLF: false, incomplete: false, diagnosticSaveFails: false },
{ missingLF: true, incomplete: false, diagnosticSaveFails: false },
{ missingLF: false, incomplete: true, diagnosticSaveFails: false },
{ missingLF: false, incomplete: true, diagnosticSaveFails: true },
])("preserves remote memory and terminal failures ($missingLF, $incomplete, $diagnosticSaveFails)", async ({ missingLF, incomplete, diagnosticSaveFails }) => {
const agentId = "11111111-1111-4111-8111-111111111111";
const runIds = ["22222222-2222-4222-8222-222222222222", "33333333-3333-4333-8333-333333333333"];
const root = await mkdtemp(join(tmpdir(), "pi-remote-memory-"));
const task = piNativeTasks.find(row => row.id === "agent-files-fresh-run")!;
const issues: any[] = [], runs: any[] = [], cleanup: Array<() => Promise<any>> = []; const captures: any[] = [], evidence = new Map<string, any>();
let personal = "", restarted = false, readAfterFinish = false, parentSeeded = false, closed = 0, diagnosticAttempts = 0;
const terminalError = new Error("remote_native_fixture:terminal_evidence_incomplete");
const originalDiagnostic = remoteFixtures.remoteNativeIncompleteTerminalEvidence;
const diagnostic = vi.spyOn(remoteFixtures, "remoteNativeIncompleteTerminalEvidence").mockImplementation(error => error === terminalError
? { complete: false, incompleteReasons: ["unwatched_directory"] } as any : originalDiagnostic(error));
browser.create = value => {
expect(parentSeeded).toBe(true);
expect(value.requireExplicitTitle).toBe(true);
expect(value.prompt).toMatch(/^Read only bootstrap-/); expect(value.prompt).not.toContain("forbidden");
const n = issues.length + 1; issues.push({ id: `issue-${n}`, title: "Provider-generated task name", companyId: "company", assigneeAgentId: agentId, status: "in_progress" }); runs.push({ id: runIds[n - 1], status: "running", runtimeMode: "native", createdAt: `2026-09-29T00:00:0${n}Z` });
return { submittedAtMs: Date.now(), issueId: `issue-${n}` };
};
const api = { request: { put: async (path: string, input: any) => {
expect(path).toBe(`/api/agents/${agentId}/instructions-bundle/file`);
expect(input.data).toEqual({ path: PI_NATIVE_MEMORY_PARENT_SEED_PATH, content: PI_NATIVE_MEMORY_PARENT_SEED_CONTENT, baseHash: null });
expect(issues).toHaveLength(0); expect(personal).toBe(""); parentSeeded = true;
return { ok: () => true, json: async () => ({ content: PI_NATIVE_MEMORY_PARENT_SEED_CONTENT }) };
} }, post: async () => ({ name: "Pi remote project" }), get: async (path: string) => {
if (path.endsWith("/issues?limit=100")) return issues;
if (path.endsWith("/heartbeat-runs?limit=100")) return runs;
if (/\/api\/issues\/issue-[12]$/.test(path)) return issues.find(row => path.endsWith(row.id));
if (runs.some(row => path === `/api/heartbeat-runs/${row.id}`)) return runs.find(row => path.endsWith(row.id));
if (path.endsWith("/interactions") || path.endsWith("/comments")) return [];
if (path.includes("instructions-bundle/file?")) return { content: decodeURIComponent(path.split("?path=")[1]!) === PI_NATIVE_MEMORY_PARENT_SEED_PATH ? PI_NATIVE_MEMORY_PARENT_SEED_CONTENT : personal };
if (path.includes("/events?")) {
const runId = runIds.find(id => path.includes(`${id}/`))!;
const nativeRead = { eventType: "tool.execution.completed", seq: 3, payload: { prpEvent: { payload: {
schema: "paperclip.tool.execution.v1", transport: "builtin", name: "read", operation: "read", status: "completed",
target: `.paperclip-runtime/agent-files/${agentId}/${runId}/memory/pi-native.txt`, readOnly: true, outputTruncated: false, executionId: path.includes(`${runIds[0]}/`) ? "read-1" : "read-2",
// Missing-LF controls still model the intended complete native read;
// their independent persisted bytes fail the existing byte assertion.
output: JSON.stringify({ content: [{ type: "text", text: personal.endsWith("\n") ? personal : `${personal}\n` }] }),
} } } };
return path.includes(`${runIds[0]}/`) ? [{ eventType: "instruction_save", seq: 1, payload: { state: "saved" } }, { eventType: "tool.execution.completed", seq: 2, payload: { prpEvent: { payload: { schema: "paperclip.tool.execution.v1", transport: "builtin", operation: "edit", name: "write", status: "failed", target: null, executionId: "tool-1", output: "Pi tool path is outside its assigned workspace and agent files" } } } }, nativeRead] : [nativeRead];
}
throw new Error(`Unexpected fixture path ${path}`);
} };
const remoteBootstrap = { prompt: (nonce: string) => `Read only bootstrap-${nonce}`, bindAndRelease: async (input: any) => {
const ordinal = runs.length; expect(input.runId).toBe(runIds[ordinal - 1]);
const identity = { pid: 100 + ordinal, ppid: 1, startTicks: String(ordinal), bootId: `boot-${ordinal}` };
const targets: any = ordinal === 1 ? { "@cross-root": { absent: false, sha256: `sha256:${createHash("sha256").update(input.crossRoot.initialText).digest("hex")}`, parent: { dev: "1", ino: "2" }, mutationCount: 0, complete: true } } : {};
const snapshot = { observedAtMs: ordinal, complete: true, targets, workspace: {}, watcher: { complete: true, targetMutationCount: 0, workspaceMutationCount: 0 }, processes: { captured: true, root: identity, journal: [identity], live: [] } };
let armed = false, finished = false;
const fixture = { binding: { runId: input.runId }, remoteCwd: `/remote/run-${ordinal}`, outsideTarget: ordinal === 1 ? `/tmp/owned-${ordinal}/cross-root-target` : null,
snapshot: async () => { armed = true; return snapshot; }, finish: async () => { if (incomplete) throw terminalError; finished = true; return snapshot; }, close: async () => { closed++; },
readFile: async (path: string) => { expect(finished).toBe(true); expect(restarted).toBe(true); expect(path).toBe("pi-agent-memory-proof.txt"); readAfterFinish = true; return Buffer.from(personal); },
};
const action = await input.actionPrompt(fixture); expect(armed).toBe(true);
if (ordinal === 1) {
expect(action).toContain(fixture.outsideTarget);
const content = /```json\n(.*?)\n```/s.exec(action)?.[1];
expect(content).toBeDefined();
personal = JSON.parse(content!).content;
expect(personal).toMatch(/^[a-f0-9]{32}\n$/);
expect(Buffer.byteLength(personal, "utf8")).toBe(33);
if (missingLF) personal = personal.slice(0, -1);
} else { expect(action).not.toContain(personal.trim()); expect(input.targets).toEqual(["pi-agent-memory-proof.txt"]); }
issues.at(-1).status = "done"; runs.at(-1).status = "succeeded"; captures.push(fixture.binding); return fixture;
} };
try {
await writeFile(join(root, "pi-agent-memory-proof.txt"), "WRONG HOST COPYBACK");
const call = runPiNativeFlow({ page: { goto: async () => {}, reload: async () => {}, getByTestId: () => ({ getByRole: () => ({}) }) }, api, fixtures: { company: { id: "company", issuePrefix: "PI" }, agent: { id: agentId, name: "Pi" }, environment: { id: "daytona-env" } }, execution: { task, environment: { id: "daytona" }, profile: { qualificationCandidate: "pi" } }, nonce: "fixture", workspacePath: root, deadlineAt: Date.now() + 2000,
restart: async () => { expect(evidence.has("pi-remote-1-cross-root-final.json")).toBe(true); restarted = true; }, observe: () => {}, capture: async () => {}, evidence: async (name: string, value: unknown) => { if (name.endsWith("incomplete-terminal.json")) { diagnosticAttempts++; if (diagnosticSaveFails) throw new Error("diagnostic storage unavailable"); } evidence.set(name, value); }, remoteBootstrap, registerCleanupAssertion: (fn: () => Promise<any>) => cleanup.push(fn) } as any);
if (missingLF) {
await expect(call).rejects.toThrow("Public managed-file API contains the exact native-write bytes");
expect(evidence.get("pi-agent-files-first-save.json").personal.content).toBe(personal);
expect(Buffer.byteLength(personal)).toBe(32);
expect(restarted).toBe(false); expect(runs).toHaveLength(1);
return;
}
if (incomplete) {
await expect(call).rejects.toBe(terminalError);
expect(cleanup).toHaveLength(1);
await expect(cleanup[0]!()).rejects.toBe(terminalError);
expect(closed).toBe(1); expect(restarted).toBe(false);
expect(evidence.has("pi-remote-1-incomplete-terminal.json")).toBe(!diagnosticSaveFails);
expect(diagnosticAttempts).toBe(diagnosticSaveFails ? 2 : 1);
return;
}
const result = await call;
expect(result.checks.every(check => check.passed)).toBe(true); expect(captures).toEqual(runIds.map(runId => ({ runId }))); expect(readAfterFinish).toBe(true); expect(cleanup).toHaveLength(2); for (const fn of cleanup) await fn();
} finally { diagnostic.mockRestore(); await rm(root, { recursive: true, force: true }); }
});
tests/runner-e2e/pi-native-restart-flow.test.ts
+209
View File
@@ -0,0 +1,209 @@
import { readFileSync } from "node:fs";
import { expect, it, vi } from "vitest";
import { gradePiRestartCompletion, observePiRestartPending, runPiPendingControllerRestart } from "./pi-native-restart-flow.js";
import { validatePrpStructuredRunResult } from "../../packages/paperclip-runner/src/protocol/replay-contract.js";
type Row = Record<string, any>;
vi.mock("@playwright/test", () => ({ expect: (locator: { count(): number }) => ({
toHaveCount: async (expected: number) => expect(locator.count()).toBe(expected),
}) }));
function fixture(adapter = "acpx-runtime-sidecar") {
const questionSet = { schema: "paperclip.question_set.v1", questions: [{ id: "answer", header: "Pi native restart", prompt: "Answer", answerMode: "text", required: true }] };
const request = { schema: "paperclip.runtime_request.v2", requestKind: "runtime", type: "input", status: "pending", requestId: "request", turnId: "turn", itemId: "request",
input: questionSet, origin: { adapter, provider: "pi", method: "elicitation/create" } };
const wrap = (eventType: string, sourceSeq: number, payload: Row): Row => ({
companyId: "company", runId: "run", seq: sourceSeq, protocolSchemaVersion: 1, eventType,
payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", eventType, runId: "run", turnId: "turn", itemId: "request",
normalizedSessionId: "session", sourceInstanceId: "runner", sourceSeq, sourceEventId: `runner:run:${sourceSeq}`, payload } },
});
const state = { issue: { id: "issue", companyId: "company", status: "in_progress" },
runs: [{ id: "run", companyId: "company", nativeIssueId: "issue", runtimeMode: "native", status: "running", nativeSessionId: "session", runnerInstanceId: "runner", processPid: 200, processGroupId: 200, processStartedAt: "2026-10-02T13:55:27.000Z" } as Row],
interactions: [{ id: "interaction", companyId: "company", issueId: "issue", kind: "ask_user_questions", status: "pending", result: null,
sourceRunId: "run", continuationPolicy: "none", resolverPolicy: "human_only", idempotencyKey: "paperclip-runner-question:run:request",
payload: { runtimeRequestId: "request", questionSet } } as Row] };
const events = [wrap("runtime_request.created", 1, { request })];
const control = (eventType: string, sourceSeq: number, payload: Row): Row => {
const row = wrap(eventType, sourceSeq + 4, payload), event = row.payload.prpEvent;
Object.assign(event, { sourceKind: "control_plane", sourceInstanceId: "runner:control", sourceSeq, sourceEventId: `runner:control:run:${sourceSeq}` });
delete event.itemId;
Object.assign(row, { sourceInstanceId: event.sourceInstanceId, sourceSeq, sourceEventId: event.sourceEventId });
return row;
};
function finish(answer = "hidden") {
state.issue.status = "done"; state.runs[0]!.status = "succeeded";
Object.assign(state.interactions[0]!, { status: "answered", result: { answers: [{ questionId: "answer", optionIds: [], otherText: answer }] } });
const validated = validatePrpStructuredRunResult({ summary: "Restart question answered", reportedWorkDisposition: "done",
completionClaim: { contractRevision: "1", objectiveSatisfied: true, criteria: [{ criterionId: "objective", status: "satisfied", evidenceRefs: [] }], remainingWork: [] }, evidence: [], verification: [] });
if (!validated.ok) throw new Error("Fixture finish result must satisfy the actual production schema");
const result = validated.result;
events.push(wrap("runtime_request.resolved", 2, { requestId: "request", turnId: "turn", itemId: "request", action: "submit",
response: { schema: "paperclip.question_response.v1", answers: { answer: { text: answer } } } }), wrap("turn.completed", 4, { status: "completed" }),
wrap("run.result.proposed", 3, structuredClone(result)), control("run.result.accepted", 1, { result: structuredClone(result) }),
control("run.terminal", 2, { schema: "paperclip.prp.terminal.v1", turnTerminalState: "completed", runTerminalState: "succeeded", reportedWorkDisposition: "done" }));
return { status: "answered", value: answer };
}
return { state, events, request, wrap, control, finish };
}
it.each(["acpx-runtime", "acpx-runtime-sidecar"])("accepts mixed runner/control-plane production completion via %s", adapter => {
const f = fixture(adapter), pending = observePiRestartPending(f.state, f.events, "company"), proof = f.finish();
expect(gradePiRestartCompletion(f.state, f.events, pending, "hidden", proof)).toBe(true);
});
function actualPendingPrefix() {
return JSON.parse(readFileSync(new URL("./fixtures/pi-native-restart-actual-prefix.json", import.meta.url), "utf8")) as {
state: { issue: Row; runs: Row[]; interactions: Row[] }; events: Row[];
};
}
it.each(["omitted", "null"])("accepts the actual pre-start submission with %s turn ID", representation => {
const f = actualPendingPrefix();
if (representation === "null") f.events[0]!.payload.prpEvent.turnId = null;
expect(observePiRestartPending(f.state, f.events, "company")).toMatchObject({
runId: "run", turnId: "turn", nativeSessionId: "session", sourceInstanceId: "runner", createdSourceSeq: 195,
});
});
it.each([
["missing start", (f: ReturnType<typeof actualPendingPrefix>) => { f.events.splice(1, 1); }],
["start without assigned turn", (f: ReturnType<typeof actualPendingPrefix>) => { delete f.events[1]!.payload.prpEvent.turnId; }],
["start with foreign turn", (f: ReturnType<typeof actualPendingPrefix>) => { f.events[1]!.payload.prpEvent.turnId = "other"; }],
["foreign submission turn", (f: ReturnType<typeof actualPendingPrefix>) => { f.events[0]!.payload.prpEvent.turnId = "other"; }],
["foreign submission session", (f: ReturnType<typeof actualPendingPrefix>) => { f.events[0]!.payload.prpEvent.normalizedSessionId = "other"; }],
["foreign submission producer", (f: ReturnType<typeof actualPendingPrefix>) => {
Object.assign(f.events[0]!.payload.prpEvent, { sourceInstanceId: "other", sourceEventId: "other:run:6" });
}],
["submission after start", (f: ReturnType<typeof actualPendingPrefix>) => {
f.events[0]!.seq = 29;
Object.assign(f.events[0]!.payload.prpEvent, { sourceSeq: 8, sourceEventId: "runner:run:8" });
}],
["start after request", (f: ReturnType<typeof actualPendingPrefix>) => {
f.events[1]!.seq = 223;
Object.assign(f.events[1]!.payload.prpEvent, { sourceSeq: 196, sourceEventId: "runner:run:196" });
}],
["duplicate submission", (f: ReturnType<typeof actualPendingPrefix>) => {
const row = structuredClone(f.events[0]!); row.seq = 26;
Object.assign(row.payload.prpEvent, { sourceSeq: 7, sourceEventId: "runner:run:7" });
Object.assign(f.events[1]!.payload.prpEvent, { sourceSeq: 8, sourceEventId: "runner:run:8" });
f.events.push(row);
}],
["duplicate start", (f: ReturnType<typeof actualPendingPrefix>) => {
const row = structuredClone(f.events[1]!); row.seq = 29;
Object.assign(row.payload.prpEvent, { sourceSeq: 8, sourceEventId: "runner:run:8" }); f.events.push(row);
}],
["later unbound turn event", (f: ReturnType<typeof actualPendingPrefix>) => {
const row = structuredClone(f.events[1]!); row.seq = 29; row.eventType = "turn.progress";
Object.assign(row.payload.prpEvent, { eventType: "turn.progress", turnId: null, sourceSeq: 8, sourceEventId: "runner:run:8" }); f.events.push(row);
}],
] as const)("rejects %s around an unbound submission", (_label, mutate) => {
const f = actualPendingPrefix(); mutate(f);
expect(() => observePiRestartPending(f.state, f.events, "company")).toThrow("Pi native restart");
});
it.each([
["no native event", (f: ReturnType<typeof fixture>) => { f.events.length = 0; }],
["wrong origin", (f: ReturnType<typeof fixture>) => { f.request.origin.provider = "cursor"; }],
["semantic origin", (f: ReturnType<typeof fixture>) => { f.request.origin.adapter = "semantic"; }],
["internal kind instead of public schema", (f: ReturnType<typeof fixture>) => { f.request.requestKind = "elicitation"; }],
["missing native session", (f: ReturnType<typeof fixture>) => { delete f.state.runs[0]!.nativeSessionId; }],
["foreign request turn", (f: ReturnType<typeof fixture>) => { f.request.turnId = "other"; }],
["foreign company", (f: ReturnType<typeof fixture>) => { f.state.interactions[0]!.companyId = "other"; }],
["durable fallback", (f: ReturnType<typeof fixture>) => { f.state.interactions[0]!.continuationPolicy = "wake_assignee"; }],
["wrong idempotency identity", (f: ReturnType<typeof fixture>) => { f.state.interactions[0]!.idempotencyKey = "other"; }],
["duplicate interaction", (f: ReturnType<typeof fixture>) => { f.state.interactions.push(structuredClone(f.state.interactions[0]!)); }],
["duplicate source receipt", (f: ReturnType<typeof fixture>) => { f.events.push(structuredClone(f.events[0]!)); }],
["already answered", (f: ReturnType<typeof fixture>) => { f.finish(); }],
] as const)("rejects %s before restart", (_label, mutate) => {
const f = fixture(); mutate(f); expect(() => observePiRestartPending(f.state, f.events, "company")).toThrow("Pi native restart");
});
it.each([
["replacement run", (f: ReturnType<typeof fixture>) => { f.state.runs[0]!.id = "replacement"; }],
["additional run", (f: ReturnType<typeof fixture>) => { f.state.runs.push({ ...f.state.runs[0], id: "replayed" }); }],
["replacement native session", (f: ReturnType<typeof fixture>) => { f.state.runs[0]!.nativeSessionId = "replacement"; }],
["replacement producer", (f: ReturnType<typeof fixture>) => { f.state.runs[0]!.runnerInstanceId = "replacement"; }],
["replaced interaction", (f: ReturnType<typeof fixture>) => { f.state.interactions[0]!.id = "replacement"; }],
["missing resolution", (f: ReturnType<typeof fixture>) => { f.events.splice(1, 1); }],
["duplicate resolution", (f: ReturnType<typeof fixture>) => { f.events.push(structuredClone(f.events[1]!)); }],
["replayed creation", (f: ReturnType<typeof fixture>) => { f.events.push(f.wrap("runtime_request.created", 4, { request: f.request })); }],
["cancelled callback", (f: ReturnType<typeof fixture>) => { f.events[1]!.eventType = f.events[1]!.payload.prpEvent.eventType = "runtime_request.cancelled"; }],
["expired callback", (f: ReturnType<typeof fixture>) => { f.events[1]!.eventType = f.events[1]!.payload.prpEvent.eventType = "runtime_request.expired"; }],
["wrong delivered answer", (f: ReturnType<typeof fixture>) => { f.events[1]!.payload.prpEvent.payload.response.answers.answer.text = "guess"; }],
["wrong durable answer", (f: ReturnType<typeof fixture>) => { f.state.interactions[0]!.result.answers[0].otherText = "guess"; }],
["wrong response turn", (f: ReturnType<typeof fixture>) => { f.events[1]!.payload.prpEvent.payload.turnId = "other"; }],
["missing resolution envelope turn", (f: ReturnType<typeof fixture>) => { delete f.events[1]!.payload.prpEvent.turnId; }],
["missing resolution item", (f: ReturnType<typeof fixture>) => { delete f.events[1]!.payload.prpEvent.itemId; }],
["rewritten canonical question", (f: ReturnType<typeof fixture>) => { f.request.input.questions[0]!.prompt = "Replacement prompt"; }],
["new native turn", (f: ReturnType<typeof fixture>) => { f.events[2]!.payload.prpEvent.turnId = "other"; }],
["failed terminal payload", (f: ReturnType<typeof fixture>) => { f.events[2]!.payload.prpEvent.payload.status = "failed"; }],
["missing terminal", (f: ReturnType<typeof fixture>) => { f.events.splice(2, 1); }],
["duplicate terminal", (f: ReturnType<typeof fixture>) => { f.events.push(f.wrap("turn.completed", 4, {})); }],
["missing control-plane acceptance", (f: ReturnType<typeof fixture>) => { f.events.splice(4, 1); }],
["missing control-plane terminal", (f: ReturnType<typeof fixture>) => { f.events.pop(); }],
["foreign control-plane source", (f: ReturnType<typeof fixture>) => { f.events[4]!.payload.prpEvent.sourceInstanceId = "other:control"; }],
["foreign control-plane turn", (f: ReturnType<typeof fixture>) => { f.events[4]!.payload.prpEvent.turnId = "other"; }],
["missing control-plane turn", (f: ReturnType<typeof fixture>) => { delete f.events[4]!.payload.prpEvent.turnId; }],
["foreign control-plane session", (f: ReturnType<typeof fixture>) => { f.events[4]!.payload.prpEvent.normalizedSessionId = "other"; }],
["mismatched durable control source", (f: ReturnType<typeof fixture>) => { f.events[4]!.sourceSeq = 5; }],
["duplicate control receipt", (f: ReturnType<typeof fixture>) => { f.events.push(structuredClone(f.events[4]!)); }],
["control plane impersonates a native turn", (f: ReturnType<typeof fixture>) => { f.events.push(f.control("turn.completed", 3, { status: "completed" })); }],
["failed control terminal", (f: ReturnType<typeof fixture>) => { f.events[5]!.payload.prpEvent.payload.runTerminalState = "failed"; }],
["unaccepted native result", (f: ReturnType<typeof fixture>) => { f.events[4]!.payload.prpEvent.payload.result.summary = "Replacement result"; }],
["malformed accepted result", (f: ReturnType<typeof fixture>) => { delete f.events[4]!.payload.prpEvent.payload.result.schema; }],
["runner impersonates acceptance", (f: ReturnType<typeof fixture>) => { f.events[4]!.payload.prpEvent.sourceKind = "runner"; }],
] as const)("rejects %s despite a correct workspace answer", (_label, mutate) => {
const f = fixture(), pending = observePiRestartPending(f.state, f.events, "company"), proof = f.finish();
mutate(f); expect(() => gradePiRestartCompletion(f.state, f.events, pending, "hidden", proof)).toThrow();
});
it.each([null, {}, { status: "answered", value: "guess" }, { status: "cancelled", value: "hidden" }, { status: "answered", value: "hidden", invented: true }])(
"rejects missing or wrong independent file proof %j", proof => {
const f = fixture(), pending = observePiRestartPending(f.state, f.events, "company"); f.finish();
expect(() => gradePiRestartCompletion(f.state, f.events, pending, "hidden", proof)).toThrow();
},
);
async function flow(failure?: "replaced" | "wrong-file" | "missing-resolution" | "replaced-runner", issueRef = "issue") {
const f = fixture(), evidence = new Map<string, any>(), checkpoints: string[] = [];
Object.assign(f.state.issue, { identifier: "RUN-1" });
let answer = "", restarts = 0, submissions = 0, proof: unknown, resolvePost: ((value: unknown) => void) | undefined;
let postPredicate: ((value: any) => boolean) | undefined;
const composer: any = { count: () => 1, filter: () => composer, locator: () => composer, first: () => composer,
fill: async (value: string) => { expect(restarts).toBe(1); answer = value; } };
const button: any = { count: () => 1, filter: () => button, click: async () => {
expect(restarts).toBe(1); expect(answer).toMatch(/^PI-RESTART-[a-f0-9]{32}$/); submissions++;
const request = { url: () => `http://fixture/api/issues/${issueRef}/interactions/interaction/respond`, method: () => "POST",
postDataJSON: () => ({ answers: [{ questionId: "answer", optionIds: [], otherText: answer }] }) };
expect(postPredicate!({ ...request, url: () => "http://fixture/api/issues/OTHER-1/interactions/interaction/respond" })).toBe(false);
expect(postPredicate!({ ...request, url: () => `http://fixture/api/issues/${issueRef}/interactions/other/respond` })).toBe(false);
expect(postPredicate!({ ...request, method: () => "GET" })).toBe(false);
expect(postPredicate!(request)).toBe(true); resolvePost!(request); proof = f.finish(answer);
if (failure === "missing-resolution") f.events.splice(1, 1);
} };
const page: any = { getByTestId: () => composer, getByRole: () => button, reload: async () => { checkpoints.push("reload"); },
waitForRequest: (predicate: (value: any) => boolean) => { postPredicate = predicate; return new Promise(resolve => { resolvePost = resolve; }); } };
const result = runPiPendingControllerRestart({ page, companyId: "company", deadlineAt: Date.now() + 1000,
load: async () => structuredClone(f.state), events: async () => structuredClone(f.events),
restart: async identity => {
expect(identity).toEqual({ processPid: 200, processGroupId: 200, processStartedAt: "2026-10-02T13:55:27.000Z" });
expect(submissions).toBe(0); expect(f.state.runs[0]!.status).toBe("running"); expect(f.state.interactions[0]!.status).toBe("pending");
expect(answer).toBe(""); restarts++; checkpoints.push("restart");
if (failure === "replaced") f.state.interactions[0]!.id = "replacement";
if (failure === "replaced-runner") f.state.runs[0]!.processPid = f.state.runs[0]!.processGroupId = 201;
},
settle: async () => structuredClone(f.state), readProof: async () => failure === "wrong-file" ? { status: "answered", value: "guess" } : proof,
capture: async id => { checkpoints.push(id); }, evidence: async (name, data) => { evidence.set(name, structuredClone(data)); },
});
if (failure) await expect(result).rejects.toThrow("Pi native restart");
else expect((await result).every(check => check.passed)).toBe(true);
expect(restarts).toBe(1); expect(submissions).toBe(["replaced", "replaced-runner"].includes(failure ?? "") ? 0 : 1);
expect(checkpoints.slice(0, 3)).toEqual(["pi-restart-pending", "restart", "reload"]);
expect(evidence.has("pi-native-restart-before.json")).toBe(true);
expect(evidence.has("pi-native-restart-after.json")).toBe(true);
expect(evidence.has("pi-native-restart-checks.json")).toBe(true);
expect(JSON.stringify(evidence.get("pi-native-restart-before.json"))).not.toContain("PI-RESTART-");
if (!["replaced", "replaced-runner"].includes(failure ?? "")) expect(evidence.has("pi-native-restart-final.json")).toBe(true);
}
it.each(["issue", "RUN-1"])("restarts while unanswered and submits through the exact %s browser route", issueRef => flow(undefined, issueRef));
it.each(["replaced", "wrong-file", "missing-resolution", "replaced-runner"] as const)("retains evidence and fails whole flow on %s", failure => flow(failure));
tests/runner-e2e/pi-native-restart-flow.ts
+179
View File
@@ -0,0 +1,179 @@
import { parseRestartRunnerIdentity, type RestartRunnerIdentity } from "./process-tree-owner.js";
import { createHash, randomBytes } from "node:crypto";
import { expect, type Page } from "@playwright/test";
import { canonicalJson } from "../../packages/shared/src/portability-hash.js";
import { validatePrpStructuredRunResult } from "../../packages/paperclip-runner/src/protocol/replay-contract.js";
import { pollUntil } from "./api.js";
import { hasAcpxNativeOrigin } from "./acpx-native-origin.js";
import { isValidNativePrpEnvelope } from "./native-event-envelope.js";
type Row = Record<string, any>;
type State = { issue: Row; runs: Row[]; interactions: Row[] };
type Check = { id: string; passed: boolean; detail: string };
const rec = (value: unknown): Row => value !== null && typeof value === "object" && !Array.isArray(value) ? value as Row : {};
const id = (value: unknown): value is string => typeof value === "string" && value.length > 0 && value.length <= 240 && !/[\u0000-\u001f\u007f]/u.test(value) && !value.includes("[REDACTED]");
const digest = (value: unknown) => createHash("sha256").update(canonicalJson(value)).digest("hex");
const closures = new Set(["runtime_request.resolved", "runtime_request.cancelled", "runtime_request.expired"]);
const terminals = new Set(["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"]);
function requireProof(value: unknown, reason: string): asserts value { if (!value) throw new Error(`Pi native restart: ${reason}`); }
export interface PiRestartPending {
companyId: string; issueId: string; runId: string; interactionId: string; requestId: string;
turnId: string; itemId: string; nativeSessionId: string; sourceInstanceId: string; questionId: string;
createdSourceSeq: number; createdRowSha256: string; interactionBindingSha256: string;
}
function interactionBinding(card: Row) {
return { id: card.id, companyId: card.companyId, issueId: card.issueId, kind: card.kind,
sourceRunId: card.sourceRunId, idempotencyKey: card.idempotencyKey,
continuationPolicy: card.continuationPolicy, resolverPolicy: card.resolverPolicy, payload: card.payload };
}
function inspect(state: State, events: readonly Row[], companyId: string, header = "Pi native restart") {
requireProof(id(companyId) && id(state.issue.id) && state.issue.companyId === companyId && state.runs.length === 1 && state.interactions.length === 1,
"one company-scoped task, run and question required");
const run = state.runs[0]!, card = state.interactions[0]!;
requireProof(id(run.id) && run.companyId === companyId && run.nativeIssueId === state.issue.id && run.runtimeMode === "native"
&& id(run.nativeSessionId) && id(run.runnerInstanceId), "native run/session identity missing");
requireProof(id(card.id) && card.companyId === companyId && card.issueId === state.issue.id && card.kind === "ask_user_questions"
&& card.sourceRunId === run.id && card.continuationPolicy === "none" && card.resolverPolicy === "human_only", "question is not the native callback");
requireProof(Array.isArray(events) && events.length > 0 && events.length <= 20_000, "bounded durable events required");
const rows = events.filter(row => rec(row.payload).prpEvent !== undefined).map(row => ({ row, event: rec(rec(row.payload).prpEvent) })).sort((a, b) => a.row.seq - b.row.seq);
const seenRows = new Set<number>(), seenSources = new Set<string>(), cursors = new Map<string, number>();
for (const { row, event } of rows) {
requireProof(row.companyId === companyId && row.runId === run.id && isValidNativePrpEnvelope(event, row.protocolSchemaVersion)
&& event.runId === run.id && event.eventType === row.eventType && ["runner", "control_plane"].includes(event.sourceKind)
&& Number.isSafeInteger(row.seq) && row.seq > 0 && !seenRows.has(row.seq)
&& id(event.sourceInstanceId) && Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > (cursors.get(event.sourceInstanceId) ?? 0)
&& event.sourceEventId === `${event.sourceInstanceId}:${run.id}:${event.sourceSeq}` && !seenSources.has(event.sourceEventId),
"foreign, duplicate or reordered durable event");
// The production finalizer owns this separate, stable two-event stream.
// It cannot impersonate a native request/turn, nor can a runner accept itself.
if (event.sourceKind === "control_plane") {
requireProof(event.sourceInstanceId === `${run.runnerInstanceId}:control` && event.normalizedSessionId === run.nativeSessionId
&& row.sourceInstanceId === event.sourceInstanceId && row.sourceSeq === event.sourceSeq && row.sourceEventId === event.sourceEventId
&& ((event.eventType === "run.result.accepted" && event.sourceSeq === 1) || (event.eventType === "run.terminal" && event.sourceSeq === 2)),
"foreign or unexpected control-plane completion receipt");
} else requireProof(!["run.result.accepted", "run.terminal"].includes(event.eventType), "runner cannot claim control-plane completion");
seenRows.add(row.seq); seenSources.add(event.sourceEventId); cursors.set(event.sourceInstanceId, event.sourceSeq);
}
const created = rows.filter(x => x.event.eventType === "runtime_request.created");
requireProof(created.length === 1, "exactly one native request creation required");
const opening = created[0]!, request = rec(opening.event.payload?.request), questionSet = rec(request.input);
const question = Array.isArray(questionSet.questions) && questionSet.questions.length === 1 ? rec(questionSet.questions[0]) : {};
requireProof(request.schema === "paperclip.runtime_request.v2" && request.requestKind === "runtime" && request.type === "input" && request.status === "pending"
&& id(request.requestId) && id(request.itemId) && id(request.turnId) && request.turnId === opening.event.turnId && request.itemId === opening.event.itemId
&& hasAcpxNativeOrigin(request.origin, "pi", "elicitation/create") && questionSet.schema === "paperclip.question_set.v1"
&& id(question.id) && question.answerMode === "text" && question.header === header
&& card.payload?.runtimeRequestId === request.requestId && digest(card.payload.questionSet) === digest(questionSet)
&& card.idempotencyKey === `paperclip-runner-question:${run.id}:${request.requestId}`, "canonical Pi input request identity missing");
const submitted = rows.filter(x => x.event.eventType === "turn.submitted");
const unboundSubmission = submitted.find(x => x.event.turnId == null);
if (unboundSubmission) {
// The local runner records submission before ACP assigns the provider turn ID.
// Only that unique, ordered pre-start receipt may omit the turn identity.
const started = rows.filter(x => x.event.eventType === "turn.started");
requireProof(submitted.length === 1 && started.length === 1
&& started[0]!.event.turnId === request.turnId
&& unboundSubmission.row.seq < started[0]!.row.seq && started[0]!.row.seq < opening.row.seq
&& unboundSubmission.event.sourceSeq < started[0]!.event.sourceSeq && started[0]!.event.sourceSeq < opening.event.sourceSeq,
"unbound submission is not the unique pre-start receipt");
}
requireProof(rows.filter(x => x.event.turnId != null || x.event.eventType.startsWith("turn.") || x.event.eventType.startsWith("runtime_request.")).every(({ row, event }) => (event.turnId === request.turnId || row === unboundSubmission?.row)
&& event.normalizedSessionId === run.nativeSessionId
&& event.sourceInstanceId === (event.sourceKind === "runner" ? run.runnerInstanceId : `${run.runnerInstanceId}:control`)), "turn, native session or producer was replaced");
requireProof(rows.filter(x => x.event.sourceKind === "control_plane").every(x => x.event.turnId === request.turnId), "control-plane result belongs to another turn");
const binding: PiRestartPending = { companyId, issueId: state.issue.id, runId: run.id, interactionId: card.id, requestId: request.requestId,
turnId: request.turnId, itemId: request.itemId, nativeSessionId: run.nativeSessionId, sourceInstanceId: run.runnerInstanceId, questionId: question.id,
createdSourceSeq: opening.event.sourceSeq, createdRowSha256: digest(opening.row), interactionBindingSha256: digest(interactionBinding(card)) };
return { run, card, rows, binding };
}
/** Grade public state and PRP receipts; assistant prose is never restart proof. */
export function observePiPendingNativeInput(state: State, events: readonly Row[], companyId: string, header: string): PiRestartPending {
const { run, card, rows, binding } = inspect(state, events, companyId, header);
requireProof(state.issue.status === "in_progress" && run.status === "running" && card.status === "pending" && card.result == null
&& run.resultJson?.nativeCancellation == null && run.resultJson?.startupCancellation == null
&& !rows.some(x => closures.has(x.event.eventType) || terminals.has(x.event.eventType) || x.event.sourceKind === "control_plane"), "question is no longer unanswered in a live run");
return binding;
}
export function observePiRestartPending(state: State, events: readonly Row[], companyId: string): PiRestartPending {
return observePiPendingNativeInput(state, events, companyId, "Pi native restart");
}
export function gradePiRestartCompletion(state: State, events: readonly Row[], pending: PiRestartPending, answer: string, proof: unknown): boolean {
const { run, card, rows, binding } = inspect(state, events, pending.companyId);
requireProof(digest(binding) === digest(pending), "original request/run/source identity changed");
const resolved = rows.filter(x => closures.has(x.event.eventType)), terminal = rows.filter(x => terminals.has(x.event.eventType));
requireProof(resolved.length === 1 && resolved[0]!.event.eventType === "runtime_request.resolved", "exactly one durable native resolution required");
const delivered = resolved[0]!.event, response = rec(delivered.payload);
requireProof(response.requestId === pending.requestId && response.turnId === pending.turnId && response.itemId === pending.itemId
&& response.action === "submit" && delivered.itemId === pending.itemId && delivered.sourceSeq > pending.createdSourceSeq
&& digest(response.response) === digest({ schema: "paperclip.question_response.v1", answers: { [pending.questionId]: { text: answer } } }),
"hidden browser answer was not delivered to the original native request");
requireProof(terminal.length === 1 && terminal[0]!.event.eventType === "turn.completed" && terminal[0]!.event.payload?.status === "completed"
&& terminal[0]!.event.payload?.error == null && terminal[0]!.event.sourceSeq > delivered.sourceSeq
&& state.issue.status === "done" && run.status === "succeeded" && run.resultJson?.nativeCancellation == null && run.resultJson?.startupCancellation == null,
"original native turn did not finish exactly once");
const proposed = rows.filter(x => x.event.eventType === "run.result.proposed"), accepted = rows.filter(x => x.event.eventType === "run.result.accepted"), runTerminal = rows.filter(x => x.event.eventType === "run.terminal");
requireProof(proposed.length === 1 && accepted.length === 1 && runTerminal.length === 1, "unique proposed and control-plane accepted completion required");
const result = rec(accepted[0]!.event.payload?.result), validation = validatePrpStructuredRunResult(result);
requireProof(validation.ok && digest(validation.result) === digest(result) && result.reportedWorkDisposition === "done"
&& digest(proposed[0]!.event.payload) === digest(result)
&& proposed[0]!.row.seq > resolved[0]!.row.seq && proposed[0]!.row.seq < terminal[0]!.row.seq
&& accepted[0]!.row.seq > terminal[0]!.row.seq && runTerminal[0]!.row.seq > accepted[0]!.row.seq
&& digest(runTerminal[0]!.event.payload) === digest({ schema: "paperclip.prp.terminal.v1", turnTerminalState: "completed", runTerminalState: "succeeded", reportedWorkDisposition: "done" }),
"control-plane completion does not accept the original successful native result");
requireProof(card.status === "answered" && Array.isArray(card.result?.answers) && card.result.answers.length === 1
&& card.result.answers[0].questionId === pending.questionId && card.result.answers[0].otherText === answer, "durable interaction answer missing or wrong");
requireProof(digest(proof) === digest({ status: "answered", value: answer }), "independent workspace proof is missing or wrong");
return true;
}
export async function runPiPendingControllerRestart(input: {
page: Page; companyId: string; deadlineAt: number; load(): Promise<State>; events(runId: string): Promise<Row[]>;
preserveLocalRunner?: boolean; restart(preserveRunner?: RestartRunnerIdentity): Promise<void>; settle(): Promise<State>; readProof(): Promise<unknown>;
capture(id: string, label: string, file: string): Promise<void>; evidence(name: string, data: unknown): Promise<void>;
}): Promise<Check[]> {
const checks: Check[] = [], answer = `PI-RESTART-${randomBytes(16).toString("hex")}`;
const check = (id: string, passed: boolean, detail: string) => { checks.push({ id, passed, detail }); requireProof(passed, detail); };
const snapshot = async (name: string) => {
const state = await input.load(), events = state.runs.length === 1 ? await input.events(state.runs[0]!.id) : [];
await input.evidence(name, { ...state, events }); return { state, events };
};
try {
await pollUntil({ label: "Pi pending native restart question", deadlineAt: input.deadlineAt, load: input.load,
accept: state => state.interactions.some(card => card.status === "pending"),
reject: state => state.runs.length > 1 || state.runs.some(run => ["failed", "cancelled", "timed_out", "succeeded"].includes(run.status)) ? "Pi ended or created another run before the pending restart boundary" : undefined });
const before = await snapshot("pi-native-restart-before.json"), pending = observePiRestartPending(before.state, before.events, input.companyId);
check("restart-pending-native-identity", true, "One unanswered native Pi callback belongs to the original running turn and session");
const composer = () => input.page.getByTestId("question-text-answer-composer").filter({ visible: true });
await expect(composer()).toHaveCount(1);
await input.capture("pi-restart-pending", "Pi native question before controller restart", "pi-restart-pending.png");
const runnerIdentity = input.preserveLocalRunner === false ? undefined : parseRestartRunnerIdentity(before.state.runs[0]);
await input.restart(runnerIdentity);
await input.page.reload();
const after = await snapshot("pi-native-restart-after.json"), resumed = observePiRestartPending(after.state, after.events, input.companyId);
check("restart-same-pending-native-request", digest(resumed) === digest(pending), "Controller restart retained the same unanswered request, run, turn, native session and producer");
if (runnerIdentity) check("restart-same-live-runner-identity", digest(parseRestartRunnerIdentity(after.state.runs[0])) === digest(runnerIdentity),
"The original durable runner PID, process group and start identity survived the controller restart");
await expect(composer()).toHaveCount(1);
await input.capture("pi-restart-reconnected", "Same Pi question after controller restart", "pi-restart-reconnected.png");
await composer().locator('[contenteditable="true"],textarea').first().fill(answer);
const issueRefs = [pending.issueId, after.state.issue.identifier].filter(id);
const routes = new Set(issueRefs.map(issueRef => `/api/issues/${encodeURIComponent(issueRef)}/interactions/${pending.interactionId}/respond`));
const submitted = input.page.waitForRequest(request => routes.has(new URL(request.url()).pathname) && request.method() === "POST",
{ timeout: Math.max(1, input.deadlineAt - Date.now()) });
const button = input.page.getByRole("button", { name: after.state.interactions[0]!.payload.questionSet.submitLabel ?? "Submit answers", exact: true }).filter({ visible: true });
await expect(button).toHaveCount(1); await button.click();
const posted = (await submitted).postDataJSON();
check("restart-exact-browser-answer", Array.isArray(posted?.answers) && posted.answers.length === 1
&& posted.answers[0].questionId === pending.questionId && posted.answers[0].otherText === answer, "Actual browser POST answered the retained question with previously undisclosed text");
const final = await input.settle(), events = await input.events(pending.runId), proof = await input.readProof();
await input.evidence("pi-native-restart-final.json", { ...final, events, pending, posted, proof });
check("restart-native-answer-delivered-once", gradePiRestartCompletion(final, events, pending, answer, proof),
"One durable answer and one successful original native turn delivered the exact typed result after restart");
return checks;
} finally { await input.evidence("pi-native-restart-checks.json", checks); }
}
tests/runner-e2e/pi-provider-fault.py
+229
View File
@@ -0,0 +1,229 @@
"""One-shot Linux Pi-child fault. No environment reads; never targets a worker.
The caller is the nonce-bound remote observer. Its already admitted lease and
run root are rechecked here, then pidfd pins the exact child across PID reuse.
This helper does not relax provider admission or manufacture runtime events.
"""
import base64
import hashlib
import json
import os
from pathlib import Path
import re
import signal
import stat
import sys
PACK = '/opt/paperclip-runner/provider-pack'
ENTRY = 'node_modules/@earendil-works/pi-coding-agent/dist/cli.js'
NODE = 'node/bin/node'
EXTENSION = 'extensions/paperclip.js'
GUARD = '.paperclip-native-module-guard.cjs'
def require(value, code):
if not value:
raise RuntimeError('pi_provider_fault:' + code)
def proc(pid, boot):
raw = Path(f'/proc/{pid}/stat').read_text()
require(raw.startswith(f'{pid} ('), 'stat_pid')
fields = raw.rsplit(') ', 1)[1].split()
require(fields[0] != 'Z', 'dead_process')
return {'pid': pid, 'ppid': int(fields[1]), 'startTicks': fields[19], 'bootId': boot}
def argv(pid):
raw = Path(f'/proc/{pid}/cmdline').read_bytes()
require(0 < len(raw) <= 32768, 'argv_bound')
return raw.decode().rstrip('\0').split('\0')
def flag(args, name):
require(args.count(name) == 1 and args.index(name) + 1 < len(args), 'flag_' + name)
return args[args.index(name) + 1]
def checked_file(path, maximum, sealed=False):
require(os.path.realpath(path) == path, 'file_realpath')
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW)
try:
before = os.fstat(fd)
require(stat.S_ISREG(before.st_mode) and before.st_nlink == 1 and 0 < before.st_size <= maximum, 'file_shape')
if sealed:
require(before.st_mode & 0o222 == 0, 'file_writable')
chunks = []
while True:
chunk = os.read(fd, min(1024 * 1024, maximum + 1))
if not chunk:
break
chunks.append(chunk)
require(sum(map(len, chunks)) <= maximum, 'file_bound')
after, named = os.fstat(fd), os.lstat(path)
identity = lambda s: (s.st_dev, s.st_ino, s.st_size, s.st_mtime_ns, s.st_ctime_ns, s.st_mode, s.st_nlink)
require(identity(before) == identity(after) == identity(named), 'file_changed')
return b''.join(chunks), (before.st_dev, before.st_ino)
finally:
os.close(fd)
def digest(data):
return hashlib.sha256(data).hexdigest()
def checked_runner_executable(path):
# Production stages a verified preinstalled runner with ln -sfn. Admit
# that named link only while its identity and resolved regular file remain
# stable; inspect still requires both the pinned hash and /proc/exe inode.
before = os.lstat(path)
require(stat.S_ISREG(before.st_mode) or stat.S_ISLNK(before.st_mode), 'runner_file_shape')
resolved = os.path.realpath(path)
content, inode = checked_file(resolved, 256 * 1024 * 1024)
after = os.lstat(path)
identity = lambda s: (s.st_dev, s.st_ino, s.st_mode, s.st_mtime_ns, s.st_ctime_ns)
require(identity(before) == identity(after) and os.path.realpath(path) == resolved, 'runner_link_changed')
return content, inode
def parse_closure_metadata(content, expected_pin):
# Match production parseNativeAcpxDistributionEntries: the profile pins
# canonical entries, not the formatting or outer JSON file bytes.
manifest = json.loads(content)
entries = manifest.get('entries') if isinstance(manifest, dict) else None
require(isinstance(entries, list) and 0 < len(entries) <= 30000, 'closure_inventory')
normalized, previous, total = [], '', 0
for entry in entries:
require(isinstance(entry, dict) and set(entry) == {'path', 'sha256', 'size', 'executable'}, 'closure_entry_shape')
path = entry['path']
require(isinstance(path, str) and 0 < len(path) <= 4096 and not os.path.isabs(path)
and not re.search(r'[\x00-\x1f\x7f\\]', path)
and all(part not in ('', '.', '..') for part in path.split('/'))
and path > previous and path != 'manifest.json' and not path.startswith('.paperclip-'), 'closure_entry_path')
require(isinstance(entry['sha256'], str) and re.fullmatch(r'[a-f0-9]{64}', entry['sha256'])
and type(entry['size']) is int and 0 <= entry['size'] <= 384 * 1024 * 1024
and type(entry['executable']) is bool, 'closure_entry_metadata')
total += entry['size']; require(total <= 1024 * 1024 * 1024, 'closure_tree_bound')
previous = path
normalized.append({key: entry[key] for key in ('path', 'sha256', 'size', 'executable')})
canonical = json.dumps(normalized, separators=(',', ':'), ensure_ascii=False).encode('utf8')
require(re.fullmatch(r'[a-f0-9]{64}', expected_pin) and digest(canonical) == expected_pin, 'closure_pin')
return normalized
def inspect(config):
require(set(config) == {'root', 'binding', 'runtimeEnvironmentLeaseId', 'runnerdSha256', 'closureSha256'}, 'config_keys')
binding, expected = config['binding'], config['root']
boot = Path('/proc/sys/kernel/random/boot_id').read_text().strip()
require(re.fullmatch(r'[a-f0-9-]{36}', boot) and boot == expected['bootId'], 'boot_changed')
root = proc(expected['pid'], boot)
require(root == expected and os.getpgid(root['pid']) == root['pid'], 'root_changed')
root_args = argv(root['pid'])
runtime_root = binding['remoteCwd'] + '/.paperclip-runtime/paperclip-runner'
runner = runtime_root + '/bin/paperclip-runnerd'
require(root_args[0] == runner and flag(root_args, '--run-id') == binding['runId']
and flag(root_args, '--environment-lease-id') == config['runtimeEnvironmentLeaseId']
and flag(root_args, '--lifecycle-mode') == 'per_turn', 'root_binding')
require(re.fullmatch(re.escape(runtime_root) + r'/sessions/[a-f0-9]{64}/runner', flag(root_args, '--state-dir')), 'root_session')
runner_bytes, runner_inode = checked_runner_executable(runner)
executable = os.stat(f'/proc/{root["pid"]}/exe')
require((executable.st_dev, executable.st_ino) == runner_inode and 'sha256:' + digest(runner_bytes) == config['runnerdSha256'], 'runner_executable')
closure_bytes, _ = checked_file(PACK + '/provider-assets/pi/linux-x64/native-closure.json', 4 * 1024 * 1024)
admitted_entries = parse_closure_metadata(closure_bytes, config['closureSha256'])
entries = {e['path']: e for e in admitted_entries}
for name in (NODE, ENTRY, EXTENSION, 'pi-entry.cjs', 'node_modules/pi-acp/dist/index.js', 'node_modules/pi-acp/dist/paperclip-runtime.js'):
require(name in entries and re.fullmatch(r'[a-f0-9]{64}', entries[name]['sha256']), 'closure_entry')
pids = [int(p) for p in os.listdir('/proc') if p.isdigit() and int(p) > 1]
require(len(pids) <= 4096, 'process_bound')
table = {}
for pid in pids:
try:
table[pid] = proc(pid, boot)
except (FileNotFoundError, ProcessLookupError):
continue
except RuntimeError as error:
if str(error).endswith(':dead_process'):
continue
raise
def ancestry(pid):
chain = []
while pid != root['pid']:
require(pid in table and len(chain) < 64 and pid not in [p['pid'] for p in chain], 'ancestry')
chain.append(table[pid]); pid = table[pid]['ppid']
return chain + [root]
matches = []
for pid in table:
try:
chain = ancestry(pid)
except RuntimeError:
continue
parent = table[pid]['ppid']
if parent not in table:
continue
parent_args = argv(parent)
if len(parent_args) != 4 or not parent_args[3].endswith('/distribution/pi-entry.cjs'):
continue
distribution = parent_args[3][:-len('/pi-entry.cjs')]
require(re.fullmatch(r'/tmp/paperclip-acpx-native-[A-Za-z0-9_-]+/distribution', distribution), 'snapshot_path')
require(parent_args[1:] == ['--require', distribution + '/' + GUARD, distribution + '/pi-entry.cjs'], 'wrapper_parent')
descriptor = None
if parent_args[0] != distribution + '/' + NODE:
# Production nativeBootstrap executes the held Node through child
# FD 3 (unfenced) or 7 (guardian/credential fences). The child's
# cmdline retains /proc/self/fd/N; it does not name the source path.
match = re.fullmatch(r'/proc/self/fd/(3|7)', parent_args[0])
require(match is not None, 'wrapper_parent')
descriptor = f'/proc/{parent}/fd/{match.group(1)}'
# Pi sets process.title and overwrites Linux argv. The exact pinned
# parent's launch code attests the entrypoint; title alone never selects.
require(argv(pid) == ['pi'], 'pi_process_title')
for directory in (distribution, os.path.dirname(distribution)):
st = os.lstat(directory)
require(stat.S_ISDIR(st.st_mode) and not st.st_mode & 0o222 and os.path.realpath(directory) == directory, 'snapshot_seal')
require(os.readlink(f'/proc/{pid}/cwd') == binding['remoteCwd'], 'cwd')
identities = {}
for name in (NODE, ENTRY, EXTENSION, 'pi-entry.cjs', 'node_modules/pi-acp/dist/index.js', 'node_modules/pi-acp/dist/paperclip-runtime.js'):
content, inode = checked_file(distribution + '/' + name, 256 * 1024 * 1024, True)
require(len(content) == entries[name]['size'] and digest(content) == entries[name]['sha256'], 'snapshot_digest')
identities[name] = inode
exe = os.stat(f'/proc/{pid}/exe')
require((exe.st_dev, exe.st_ino) == identities[NODE], 'pi_executable')
parent_exe = os.stat(f'/proc/{parent}/exe')
require((parent_exe.st_dev, parent_exe.st_ino) == identities[NODE], 'wrapper_executable')
if descriptor is not None:
held_exe = os.stat(descriptor)
require((held_exe.st_dev, held_exe.st_ino) == identities[NODE], 'wrapper_descriptor')
for identity in chain:
require(proc(identity['pid'], boot) == identity, 'ancestry_changed')
matches.append({'target': table[pid], 'ancestry': chain, 'nodeSha256': 'sha256:' + entries[NODE]['sha256'],
'entrypointSha256': 'sha256:' + entries[ENTRY]['sha256'], 'closureSha256': config['closureSha256'],
'entrypointAttribution': 'pinned_wrapper_parent', 'originalChildArgvAvailable': False, 'observedChildTitle': 'pi'})
require(len(matches) == 1, 'unique_pi_child')
return matches[0]
def terminate(config):
require(sys.platform == 'linux' and hasattr(os, 'pidfd_open') and hasattr(signal, 'pidfd_send_signal'), 'pidfd_required')
before = inspect(config)
pid = before['target']['pid']
require(pid != os.getpid() and pid != config['root']['pid'], 'not_worker')
fd = os.pidfd_open(pid)
try:
require(inspect(config) == before, 'identity_changed_after_pidfd')
fields = dict(line.split(':', 1) for line in Path(f'/proc/self/fdinfo/{fd}').read_text().splitlines() if ':' in line)
require(int(fields.get('Pid', '-1').strip()) == pid, 'pidfd_retired')
signal.pidfd_send_signal(fd, signal.SIGKILL, None, 0)
return {'schema': 'paperclip.e2e.pi-provider-death.v1', 'binding': config['binding'], 'runtimeEnvironmentLeaseId': config['runtimeEnvironmentLeaseId'],
**before, 'signalled': True, 'signal': 'SIGKILL', 'targetKind': 'pi_native_child', 'workerSignalled': False}
finally:
os.close(fd)
if __name__ == '__main__':
try:
require(len(sys.argv) == 2 and len(sys.argv[1]) <= 8192, 'input_bound')
print(json.dumps(terminate(json.loads(base64.b64decode(sys.argv[1], validate=True))), separators=(',', ':')))
except Exception:
# Never print process arguments, filesystem paths, or inherited context.
print(json.dumps({'ok': False, 'code': 'pi_provider_identity_or_signal_failed'}))
sys.exit(1)
tests/runner-e2e/pi-provider-fault.test.py
+342
View File
@@ -0,0 +1,342 @@
"""Credential-free admission tests; real pidfd calibration runs only on Linux."""
import importlib.util
import json
import os
from pathlib import Path
import shutil
import signal
import subprocess
import sys
import tempfile
import time
from types import SimpleNamespace
import unittest
import select
from unittest.mock import patch
spec = importlib.util.spec_from_file_location('pi_fault', Path(__file__).with_name('pi-provider-fault.py'))
fault = importlib.util.module_from_spec(spec)
spec.loader.exec_module(fault)
BOOT = '12345678-1234-1234-1234-123456789abc'
ROOT = {'pid': 21, 'ppid': 1, 'startTicks': '100', 'bootId': BOOT}
PARENT = {'pid': 22, 'ppid': 21, 'startTicks': '101', 'bootId': BOOT}
TARGET = {'pid': 23, 'ppid': 22, 'startTicks': '102', 'bootId': BOOT}
DIST = '/tmp/paperclip-acpx-native-fixture/distribution'
RUNTIME = '/workspace/.paperclip-runtime/paperclip-runner'
CANCELLED = False
def check_cancelled():
if CANCELLED: raise KeyboardInterrupt('calibration cancelled')
NAMES = [fault.NODE, fault.ENTRY, fault.EXTENSION, 'pi-entry.cjs', 'node_modules/pi-acp/dist/index.js', 'node_modules/pi-acp/dist/paperclip-runtime.js']
class ClosureMetadataTests(unittest.TestCase):
def setUp(self):
self.entries = [
{'path': name, 'sha256': fault.digest(name.encode()), 'size': len(name), 'executable': name == fault.NODE}
for name in sorted(NAMES)
]
self.canonical = json.dumps(self.entries, separators=(',', ':'), ensure_ascii=False).encode()
self.pin = fault.digest(self.canonical)
self.manifest = {'entries': self.entries}
def test_canonical_entries_pin_admits_metadata(self):
raw = json.dumps(self.manifest, indent=2).encode()
self.assertNotEqual(fault.digest(raw), self.pin)
self.assertEqual(fault.parse_closure_metadata(raw, self.pin), self.entries)
def test_metadata_formatting_and_property_order_preserve_pin(self):
reordered = {'entries': [dict(reversed(list(entry.items()))) for entry in self.entries]}
for manifest in [self.manifest, reordered]:
for indent in [None, 2, 4]:
with self.subTest(indent=indent):
self.assertEqual(fault.parse_closure_metadata(json.dumps(manifest, indent=indent).encode(), self.pin), self.entries)
def test_raw_metadata_hash_is_not_a_closure_pin(self):
raw = json.dumps(self.manifest, indent=2).encode()
with self.assertRaisesRegex(RuntimeError, 'closure_pin'):
fault.parse_closure_metadata(raw, fault.digest(raw))
def test_changed_entry_is_rejected_by_the_pinned_digest(self):
changed = json.loads(json.dumps(self.manifest))
changed['entries'][0]['sha256'] = '0' * 64
with self.assertRaisesRegex(RuntimeError, 'closure_pin'):
fault.parse_closure_metadata(json.dumps(changed).encode(), self.pin)
def test_unsafe_duplicate_unsorted_and_invalid_metadata_fail_closed(self):
mutations = [
lambda m: m['entries'][0].update(path='../escape'),
lambda m: m['entries'][0].update(path='.paperclip-native-entry.cjs'),
lambda m: m['entries'].insert(1, m['entries'][0].copy()),
lambda m: m['entries'].reverse(),
lambda m: m['entries'][0].update(size=True),
lambda m: m['entries'][0].update(executable=1),
lambda m: m['entries'][0].update(extra='foreign'),
]
for index, mutate in enumerate(mutations):
changed = json.loads(json.dumps(self.manifest))
mutate(changed)
with self.subTest(index=index), self.assertRaises(RuntimeError):
fault.parse_closure_metadata(json.dumps(changed).encode(), self.pin)
for invalid in [None, [], {}, {'entries': []}]:
with self.subTest(invalid=invalid), self.assertRaises(RuntimeError):
fault.parse_closure_metadata(json.dumps(invalid).encode(), self.pin)
class IdentityTests(unittest.TestCase):
def setUp(self):
self.files = {DIST + '/' + n: (n.encode(), (1, 100 + i)) for i, n in enumerate(NAMES)}
self.files[RUNTIME + '/bin/paperclip-runnerd'] = (b'runner', (1, 90))
entries = [{'path': n, 'sha256': fault.digest(n.encode()), 'size': len(n), 'executable': n == fault.NODE} for n in sorted(NAMES)]
closure = json.dumps({'entries': entries}, indent=2).encode()
self.files[fault.PACK + '/provider-assets/pi/linux-x64/native-closure.json'] = (closure, (1, 80))
self.config = {'root': ROOT, 'binding': {'remoteCwd': '/workspace', 'runId': 'run'}, 'runtimeEnvironmentLeaseId': 'workspace-id',
'runnerdSha256': 'sha256:' + fault.digest(b'runner'), 'closureSha256': fault.digest(json.dumps(entries, separators=(',', ':'), ensure_ascii=False).encode())}
self.table = {21: ROOT, 22: PARENT, 23: TARGET}
self.args = {21: [RUNTIME + '/bin/paperclip-runnerd', '--run-id', 'run', '--environment-lease-id', 'workspace-id', '--lifecycle-mode', 'per_turn', '--state-dir', RUNTIME + '/sessions/' + 'a' * 64 + '/runner'],
22: [DIST + '/' + fault.NODE, '--require', DIST + '/' + fault.GUARD, DIST + '/pi-entry.cjs'], 23: ['pi']}
self.patches = [patch.object(fault, 'proc', side_effect=lambda pid, boot: self.table[pid]), patch.object(fault, 'argv', side_effect=lambda pid: self.args[pid]),
patch.object(fault, 'checked_runner_executable', side_effect=lambda p: self.files[p]),
patch.object(fault, 'checked_file', side_effect=lambda p, *a: self.files[p]), patch.object(Path, 'read_text', return_value=BOOT),
patch.object(os, 'listdir', return_value=['21', '22', '23']), patch.object(os, 'getpgid', return_value=21),
patch.object(os, 'lstat', return_value=SimpleNamespace(st_mode=0o40500)), patch.object(os.path, 'realpath', side_effect=lambda p: p),
patch.object(os, 'readlink', return_value='/workspace'), patch.object(os, 'stat', side_effect=lambda p: SimpleNamespace(st_dev=1, st_ino=90 if p == '/proc/21/exe' else 100))]
for p in self.patches: p.start(); self.addCleanup(p.stop)
def test_parent_attests_title_overwritten_child(self):
receipt = fault.inspect(self.config)
self.assertEqual(receipt['target'], TARGET)
self.assertEqual(receipt['entrypointAttribution'], 'pinned_wrapper_parent')
self.assertFalse(receipt['originalChildArgvAvailable'])
self.assertEqual(receipt['ancestry'], [TARGET, PARENT, ROOT])
def test_descriptor_launch_attests_the_same_pinned_wrapper(self):
for descriptor in [3, 7]:
self.args[22][0] = f'/proc/self/fd/{descriptor}'
with self.subTest(descriptor=descriptor):
self.assertEqual(fault.inspect(self.config)['target'], TARGET)
def test_descriptor_launch_rejects_foreign_missing_or_unbound_fd(self):
for name in ['/proc/self/fd/8', '/proc/99/fd/7', '/foreign/node']:
self.args[22][0] = name
with self.subTest(name=name), self.assertRaisesRegex(RuntimeError, 'wrapper_parent'):
fault.inspect(self.config)
self.args[22][0] = '/proc/self/fd/7'
stat_original = os.stat
for problem in ['foreign', 'missing']:
def descriptor_stat(path):
if path == '/proc/22/fd/7':
if problem == 'missing': raise FileNotFoundError(path)
return SimpleNamespace(st_dev=1, st_ino=999)
return stat_original(path)
with self.subTest(problem=problem), patch.object(os, 'stat', side_effect=descriptor_stat), self.assertRaises((RuntimeError, FileNotFoundError)):
fault.inspect(self.config)
def test_wrong_metadata_never_selects(self):
for field, value in [('closureSha256', '0' * 64), ('runtimeEnvironmentLeaseId', 'foreign'), ('runnerdSha256', 'sha256:' + '0' * 64)]:
with self.subTest(field=field), self.assertRaises(RuntimeError): fault.inspect({**self.config, field: value})
def test_wrong_title_is_not_a_generic_node_selector(self):
self.args[23] = ['node']
with self.assertRaisesRegex(RuntimeError, 'pi_process_title'): fault.inspect(self.config)
def test_foreign_parent_and_changed_wrapper_are_rejected(self):
self.args[22][3] = '/foreign/pi-entry.cjs'
with self.assertRaisesRegex(RuntimeError, 'unique_pi_child'): fault.inspect(self.config)
self.args[22][3] = DIST + '/pi-entry.cjs'
self.files[DIST + '/node_modules/pi-acp/dist/index.js'] = (b'changed', (1, 104))
with self.assertRaisesRegex(RuntimeError, 'snapshot_digest'): fault.inspect(self.config)
def test_foreign_root_reused_pid_and_wrong_cwd_are_rejected(self):
for mutation in [{'startTicks': '999'}, {'bootId': 'foreign'}, {'ppid': 99}]:
with self.subTest(mutation=mutation), self.assertRaises(RuntimeError): fault.inspect({**self.config, 'root': {**ROOT, **mutation}})
with patch.object(os, 'readlink', return_value='/foreign'), self.assertRaisesRegex(RuntimeError, 'cwd'): fault.inspect(self.config)
def test_ambiguous_direct_children_are_rejected(self):
self.table[24] = {**TARGET, 'pid': 24}; self.args[24] = ['pi']
with patch.object(os, 'listdir', return_value=['21', '22', '23', '24']), self.assertRaisesRegex(RuntimeError, 'unique_pi_child'): fault.inspect(self.config)
class RunnerExecutableTests(unittest.TestCase):
def test_regular_and_preinstalled_link_resolve_to_the_same_inode(self):
with tempfile.TemporaryDirectory() as tmp:
executable = Path(tmp).resolve() / 'installed-runner'
executable.write_bytes(b'pinned runner')
link = executable.with_name('runtime-runner')
link.symlink_to(executable)
content, inode = fault.checked_runner_executable(str(link))
self.assertEqual(content, b'pinned runner')
self.assertEqual((content, inode), fault.checked_runner_executable(str(executable)))
def test_retargeted_link_is_rejected_after_read(self):
with tempfile.TemporaryDirectory() as tmp:
executable = Path(tmp).resolve() / 'installed-runner'
executable.write_bytes(b'pinned runner')
foreign = executable.with_name('foreign-runner')
foreign.write_bytes(b'foreign')
link = executable.with_name('runtime-runner')
link.symlink_to(executable)
original = fault.checked_file
def replace_during_read(*args):
result = original(*args)
link.unlink()
link.symlink_to(foreign)
return result
with patch.object(fault, 'checked_file', side_effect=replace_during_read), self.assertRaisesRegex(RuntimeError, 'runner_link_changed'):
fault.checked_runner_executable(str(link))
def test_missing_link_and_directory_never_admit_an_executable(self):
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp).resolve()
link = root / 'runtime-runner'
link.symlink_to(root / 'missing')
with self.assertRaises(FileNotFoundError): fault.checked_runner_executable(str(link))
with self.assertRaisesRegex(RuntimeError, 'runner_file_shape'): fault.checked_runner_executable(str(root))
class PidfdTests(unittest.TestCase):
def test_changed_or_retired_pidfd_never_signals_and_always_closes(self):
before = {'target': TARGET}
for second, fdinfo in [({'target': {**TARGET, 'startTicks': '999'}}, 'Pid:\t23\n'), (before, 'Pid:\t-1\n')]:
with patch.object(sys, 'platform', 'linux'), patch.object(fault, 'inspect', side_effect=[before, second]), \
patch.object(os, 'pidfd_open', return_value=42, create=True), patch.object(signal, 'pidfd_send_signal', create=True) as sent, \
patch.object(os, 'close') as closed, patch.object(Path, 'read_text', return_value=fdinfo):
with self.assertRaises(RuntimeError): fault.terminate({'root': ROOT})
sent.assert_not_called(); closed.assert_called_once_with(42)
def test_symlink_and_mutable_snapshot_file_fail(self):
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp).resolve() / 'data'; path.write_text('data')
link = Path(tmp) / 'link'; link.symlink_to(path)
with self.assertRaises(RuntimeError): fault.checked_file(str(link), 100, True)
with self.assertRaises(RuntimeError): fault.checked_file(str(path), 100, True)
path.chmod(0o400)
self.assertEqual(fault.checked_file(str(path), 100, True)[0], b'data')
@unittest.skipUnless(sys.platform == 'linux' and hasattr(os, 'pidfd_open'), 'Hosted native Linux calibration required')
def test_real_title_overwrite_then_exact_child_pidfd(self):
self.calibrate_real_child()
@unittest.skipUnless(sys.platform == 'linux' and hasattr(os, 'pidfd_open'), 'Hosted native Linux descriptor calibration required')
def test_real_descriptor_launch_then_exact_child_pidfd(self):
for descriptor in [3, 7]:
with self.subTest(descriptor=descriptor): self.calibrate_real_child(descriptor)
@unittest.skipUnless(sys.platform == 'linux' and hasattr(os, 'pidfd_open'), 'Hosted native Linux preinstalled-link calibration required')
def test_real_preinstalled_runner_link_then_exact_child_pidfd(self):
self.calibrate_real_child(7, runner_link=True)
def calibrate_real_child(self, descriptor=None, runner_link=False):
node = shutil.which('node'); self.assertIsNotNone(node)
base = Path(tempfile.mkdtemp(prefix='pi-fault-calibration-', dir='/tmp'))
snapshot = Path(tempfile.mkdtemp(prefix='paperclip-acpx-native-', dir='/tmp'))
distribution = snapshot / 'distribution'; workspace = base / 'workspace'; workspace.mkdir()
runtime = workspace / '.paperclip-runtime/paperclip-runner'; (runtime / 'bin').mkdir(parents=True)
pack = base / 'pack'; (pack / 'provider-assets/pi/linux-x64').mkdir(parents=True)
root_child = None
owned_fds = []
try:
for name in NAMES + [fault.GUARD]:
p = distribution / name; p.parent.mkdir(parents=True, exist_ok=True); p.write_text('// fixture\n')
shutil.copyfile(node, distribution / fault.NODE); (distribution / fault.NODE).chmod(0o500)
runner = runtime / 'bin/paperclip-runnerd'
if runner_link:
installed = base / 'preinstalled-runner'
shutil.copyfile(node, installed); installed.chmod(0o500)
runner.symlink_to(installed)
else:
shutil.copyfile(node, runner); runner.chmod(0o500)
ready = workspace / 'child.json'; exited = workspace / 'exit.json'
(distribution / fault.ENTRY).write_text('process.title="pi"; require("node:fs").writeFileSync(' + json.dumps(str(ready)) + ',JSON.stringify({pid:process.pid}));setInterval(()=>{},1000);')
(distribution / 'pi-entry.cjs').write_text('const p=require("node:child_process").spawn(process.execPath,["--require",' + json.dumps(str(distribution / fault.GUARD)) + ',' + json.dumps(str(distribution / fault.ENTRY)) + '],{stdio:"ignore"});p.on("exit",(code,signal)=>require("node:fs").writeFileSync(' + json.dumps(str(exited)) + ',JSON.stringify({code,signal})));process.on("SIGTERM",()=>{if(p.exitCode!==null||p.signalCode!==null)process.exit(0);p.once("exit",()=>process.exit(0));p.kill("SIGTERM")});setInterval(()=>{},1000);')
entries = []
for name in sorted(NAMES):
p = distribution / name; content = p.read_bytes(); entries.append({'path': name, 'sha256': fault.digest(content), 'size': len(content), 'executable': name == fault.NODE})
closure = json.dumps({'entries': entries}).encode(); (pack / 'provider-assets/pi/linux-x64/native-closure.json').write_bytes(closure)
for directory, dirs, files in os.walk(snapshot):
for f in files:
p = Path(directory) / f; p.chmod(0o500 if p == distribution / fault.NODE else 0o400)
Path(directory).chmod(0o500)
launch = json.dumps(str(distribution / fault.NODE))
setup, stdio, release = '', '"ignore"', ''
if descriptor is not None:
# Production nativeBootstrap keeps its executable descriptor
# at child FD 3, or FD 7 when lifetime/credential fences exist.
setup = 'const fd=require("node:fs").openSync(' + launch + ',"r");'
launch = json.dumps(f'/proc/self/fd/{descriptor}')
stdio = '[' + ','.join(['"ignore"'] * descriptor + ['fd']) + ']'
release = 'require("node:fs").closeSync(fd);'
script = base / 'root.cjs'; script.write_text(setup + 'const p=require("node:child_process").spawn(' + launch + ',["--require",' + json.dumps(str(distribution / fault.GUARD)) + ',' + json.dumps(str(distribution / 'pi-entry.cjs')) + '],{stdio:' + stdio + '});' + release + 'process.on("SIGTERM",()=>{if(p.exitCode!==null||p.signalCode!==null)process.exit(0);p.once("exit",()=>process.exit(0));p.kill("SIGTERM")});setInterval(()=>{},1000);')
check_cancelled()
root_child = subprocess.Popen([str(runtime / 'bin/paperclip-runnerd'), str(script), '--run-id', 'fixture', '--environment-lease-id', 'workspace-id', '--lifecycle-mode', 'per_turn', '--state-dir', str(runtime / 'sessions' / ('a' * 64) / 'runner')], cwd=workspace, env={'PATH': '/usr/bin:/bin'}, start_new_session=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
try: owned_fds.append(os.pidfd_open(root_child.pid))
except Exception:
# Direct Popen child has not been polled/reaped, so its numeric
# identity cannot have been reused before this emergency stop.
root_child.terminate(); root_child.wait(timeout=3)
raise
check_cancelled()
deadline = time.monotonic() + 5
while not ready.exists() and time.monotonic() < deadline:
check_cancelled(); time.sleep(.02)
self.assertTrue(ready.exists()); target = json.loads(ready.read_text())['pid']
self.assertEqual(fault.argv(target), ['pi'], 'real Node process.title must overwrite argv')
boot = Path('/proc/sys/kernel/random/boot_id').read_text().strip()
config = {'root': fault.proc(root_child.pid, boot), 'binding': {'remoteCwd': str(workspace), 'runId': 'fixture'}, 'runtimeEnvironmentLeaseId': 'workspace-id', 'runnerdSha256': 'sha256:' + fault.digest((runtime / 'bin/paperclip-runnerd').read_bytes()), 'closureSha256': fault.digest(json.dumps(entries, separators=(',', ':'), ensure_ascii=False).encode())}
with patch.object(fault, 'PACK', str(pack)):
admitted = fault.inspect(config)
self.assertEqual(admitted['target']['pid'], target)
for identity in admitted['ancestry'][:-1]:
fd = os.pidfd_open(identity['pid'])
try:
fields = dict(line.split(':', 1) for line in Path(f'/proc/self/fdinfo/{fd}').read_text().splitlines() if ':' in line)
self.assertEqual(int(fields.get('Pid', '-1').strip()), identity['pid'])
self.assertEqual(fault.proc(identity['pid'], boot), identity)
self.assertEqual(fault.inspect(config), admitted)
except BaseException:
os.close(fd); raise
owned_fds.append(fd)
check_cancelled()
check_cancelled()
receipt = fault.terminate(config)
self.assertEqual(receipt['target']['pid'], target); self.assertTrue(receipt['signalled']); self.assertEqual(select.select([owned_fds[0]], [], [], 0)[0], [])
deadline = time.monotonic() + 3
while not exited.exists() and time.monotonic() < deadline:
check_cancelled(); time.sleep(.02)
self.assertEqual(json.loads(exited.read_text()), {'code': None, 'signal': 'SIGKILL'})
finally:
try:
if root_child is not None and owned_fds:
# A pidfd never retargets a recycled numeric process group.
# Before readiness the synthetic root owns a SIGTERM cascade.
try: signal.pidfd_send_signal(owned_fds[0], signal.SIGTERM, None, 0)
except ProcessLookupError: pass
try: root_child.wait(timeout=3)
except subprocess.TimeoutExpired:
for fd in reversed(owned_fds):
try: signal.pidfd_send_signal(fd, signal.SIGKILL, None, 0)
except ProcessLookupError: pass
root_child.wait(timeout=3)
# Also retire individually pinned descendants if root exited
# unexpectedly. Readable pidfds identify already-dead children.
for fd in reversed(owned_fds[1:]):
if not select.select([fd], [], [], 0)[0]:
try: signal.pidfd_send_signal(fd, signal.SIGKILL, None, 0)
except ProcessLookupError: pass
self.assertTrue(select.select([fd], [], [], 3)[0], 'owned descendant must retire')
finally:
for fd in owned_fds: os.close(fd)
for directory, dirs, files in os.walk(snapshot): os.chmod(directory, 0o700)
shutil.rmtree(snapshot); shutil.rmtree(base)
if __name__ == '__main__':
def cancelled(_signum, _frame):
# Flag only: never interrupt Popen or pidfd ownership publication.
global CANCELLED
CANCELLED = True
signal.signal(signal.SIGTERM, cancelled)
signal.signal(signal.SIGINT, cancelled)
unittest.main()
tests/runner-e2e/pi-provider-fault.test.ts
+22
View File
@@ -0,0 +1,22 @@
import { spawnSync } from "node:child_process";
import { dirname } from "node:path";
import { fileURLToPath } from "node:url";
import { expect, it } from "vitest";
it("calibrates the exact Pi fault helper with metadata negatives and native Linux pidfd", () => {
// Linux exercises a real, title-changing, test-owned Node child. macOS runs
// every metadata negative and explicitly skips that Linux-only process test.
// Never inherit provider credentials or an ambient Python module path.
const result = spawnSync("/usr/bin/python3", ["-B", fileURLToPath(new URL("./pi-provider-fault.test.py", import.meta.url))], {
env: { PATH: `${dirname(process.execPath)}:/usr/bin:/bin`, PYTHONDONTWRITEBYTECODE: "1" },
cwd: dirname(fileURLToPath(import.meta.url)), timeout: 25_000, killSignal: "SIGTERM", maxBuffer: 64 * 1024,
encoding: "utf8", stdio: ["ignore", "pipe", "pipe"],
});
expect(result.error).toBeUndefined();
expect(result.signal).toBeNull();
expect(result.status, result.stderr).toBe(0);
expect(result.stderr).toContain("Ran 21 tests");
if (process.platform === "linux") expect(result.stderr).not.toContain("skipped");
else expect(result.stderr).toContain("skipped=3");
console.info(result.stderr.trim());
}, 30_000);
tests/runner-e2e/playwright.config.ts
+1 -1
View File
@@ -45,7 +45,7 @@ const repositoryRoot = path.resolve(import.meta.dirname, "../..");
export default defineConfig({
testDir: ".",
testMatch: "runner.spec.ts",
testMatch: process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_STARTUP_ONLY === "1" ? "installed-startup.spec.ts" : "runner.spec.ts",
timeout: Number(process.env.PAPERCLIP_RUNNER_E2E_TEST_TIMEOUT_MS ?? 600_000),
expect: { timeout: 30_000 },
fullyParallel: false,
tests/runner-e2e/prerequisites.ts
+2 -3
View File
@@ -7,9 +7,8 @@ export const PENDING_PROFILE_PREREQUISITES = {
"legacy-grok": "Grok identity/auth/skills/session/billing qualification is pending",
} as const;
/** Pi is intentionally absent: no qualified model source exists for a valid profile. */
/** Qualification fixtures use explicit models; there is no runtime model roster. */
export const UNQUALIFIED_PROFILE_GAPS = {
pi: "Pi has no qualified model source, so no valid Product E2E profile is registered.",
} as const;
/** Reject missing remote observer authority before creating a company or run. */
@@ -18,7 +17,7 @@ export function assertRemoteNativeEvidencePrerequisites(
environment: NodeJS.ProcessEnv,
): void {
if (!executions.some(execution => execution.environment.id === "daytona"
&& ["cursor_native", "native_active_stop", "native_provider_loss"].includes(execution.task.flow))) return;
&& ["cursor_native", "pi_native", "pi_controls", "copilot_protection", "native_active_stop", "native_provider_loss"].includes(execution.task.flow))) return;
const names = ["PAPERCLIP_E2E_DAYTONA_NODE_SHA256", "PAPERCLIP_E2E_DAYTONA_RUNNERD_SHA256"];
const invalid = names.filter(name => !/^sha256:[a-f0-9]{64}$/u.test(environment[name] ?? ""));
if (invalid.length) throw new Error(`Native Daytona evidence requires exact image executable digests: ${invalid.join(", ")}`);
tests/runner-e2e/process-tree-owner.test.ts
+32
View File
@@ -27,6 +27,38 @@ it.skipIf(process.platform === "win32")("revalidates PID/start before signaling
} finally { f.owner.stopObserving(); }
});
it.skipIf(process.platform === "win32")("admits replacement group members only through a still-owned ancestor", async () => {
const f = fixture([row(100, process.pid, 100), row(200, 100, 200)]);
try {
await f.owner.observe();
// The short-lived group leader exited between polls. Its new member is
// independently owned through the original, still-live controller.
f.replace([row(100, process.pid, 100), row(300, 100, 200)]);
await f.owner.signal("SIGKILL");
expect(f.signals).toEqual([[200, "SIGKILL"], [100, "SIGKILL"]]);
} finally { f.owner.stopObserving(); }
});
it.skipIf(process.platform === "win32")("refuses replacement groups containing any unowned live member", async () => {
const f = fixture([row(100, process.pid, 100), row(200, 100, 200)]);
try {
await f.owner.observe();
f.replace([row(100, process.pid, 100), row(300, 100, 200), row(400, 1, 200)]);
await expect(f.owner.signal("SIGKILL")).rejects.toThrow("identity became uncertain");
expect(f.signals).toEqual([]);
} finally { f.owner.stopObserving(); }
});
it.skipIf(process.platform === "win32")("does not use a recycled controller as a replacement member's ownership anchor", async () => {
const f = fixture([row(100, process.pid, 100), row(200, 100, 200)]);
try {
await f.owner.observe();
f.replace([row(100, process.pid, 100, "recycled-controller"), row(300, 100, 200)]);
await expect(f.owner.signal("SIGKILL")).rejects.toThrow("identity became uncertain");
expect(f.signals).toEqual([]);
} finally { f.owner.stopObserving(); }
});
it.skipIf(process.platform === "win32")("retains observed descendants when the root exits and excludes the caller group", async () => {
const f = fixture([row(100, process.pid, 100), row(200, 100, 200), row(300, 100, 10), row(999, process.pid, 999)]);
try {
tests/runner-e2e/process-tree-owner.ts
+74 -3
View File
@@ -5,6 +5,29 @@ import {
type ObservedProcessGroup, type ProcessObservation,
} from "./process-tree.js";
/** Supplied by the company-scoped run API, never by process-name discovery. */
export interface RestartRunnerIdentity {
processPid: number;
processGroupId: number;
processStartedAt: string;
}
export function parseRestartRunnerIdentity(value: unknown): RestartRunnerIdentity {
const v = value as RestartRunnerIdentity | null;
if (!v || !Number.isSafeInteger(v.processPid) || v.processPid <= 1
|| v.processGroupId !== v.processPid || typeof v.processStartedAt !== "string"
|| !Number.isFinite(Date.parse(v.processStartedAt))) throw new Error("Invalid restart runner identity");
return { processPid: v.processPid, processGroupId: v.processGroupId, processStartedAt: v.processStartedAt };
}
/** Linux run receipts retain milliseconds; ps lstart retains whole seconds. */
export function restartProcessStartMatches(observed: string, expected: string, platform = process.platform): boolean {
const actualTime = Date.parse(observed), expectedTime = Date.parse(expected);
if (!Number.isFinite(actualTime) || !Number.isFinite(expectedTime)) return false;
return platform === "linux"
? Math.floor(actualTime / 1000) === Math.floor(expectedTime / 1000)
: actualTime === expectedTime;
}
export function createProcessTreeOwner(root: ChildProcess, options: {
readTable?: () => Promise<ProcessObservation[] | null>;
signalGroup?: (pid: number, signal: NodeJS.Signals) => void;
@@ -12,6 +35,8 @@ export function createProcessTreeOwner(root: ChildProcess, options: {
let groups: ObservedProcessGroup[] = [];
let table: ProcessObservation[] = [];
let rootStarted: string | undefined;
let preserved: ObservedProcessGroup[] = [];
let restartRunner: ProcessObservation | undefined;
let observing: Promise<void> | undefined;
const covered = new Set<number>();
const readTable = options.readTable ?? readProcessTable;
@@ -30,10 +55,20 @@ export function createProcessTreeOwner(root: ChildProcess, options: {
const rootRow = next.find(row => row.pid === root.pid);
if (!rootStarted && !exited()) rootStarted = rootRow?.started;
const retained = revalidateObservedProcessGroups(groups, next);
// Never acquire a recycled group from its numeric ID alone.
const trustedAnchors = retained.flatMap(group => group.members
.filter(member => next.some(row => row.pid === member.pid && row.started === member.started
&& row.processGroupId === group.processGroupId)).map(member => member.pid));
if (rootRow && rootStarted === rootRow.started) trustedAnchors.push(rootRow.pid);
const ownedNow = new Set(trustedAnchors.flatMap(pid =>
observeDescendantProcessTree(next, pid).members.map(member => member.process.pid)));
// A short-lived leader can exit between polls while its replacement
// remains a descendant of a separately validated owner. Admit that
// ancestry, never a recycled numeric group or a mixed ownership group.
const lost = groups.filter(group => !retained.includes(group) && next.some(row => row.processGroupId === group.processGroupId && running(row)));
if (lost.length) throw new Error("Owned process group identity became uncertain");
const refreshed = refreshContinuouslyLiveProcessGroups(retained, next)
const uncertain = lost.filter(group => next.some(row => row.processGroupId === group.processGroupId
&& running(row) && !ownedNow.has(row.pid)));
if (uncertain.length) throw new Error(`Owned process group identity became uncertain: ${uncertain.map(group => group.processGroupId).join(",")}`);
const refreshed = refreshContinuouslyLiveProcessGroups([...retained, ...lost], next)
.filter(group => group.processGroupId !== callerGroup);
const anchors = refreshed.flatMap(group => group.members.map(member => member.pid));
if (rootRow && rootStarted === rootRow.started) anchors.push(rootRow.pid);
@@ -47,6 +82,14 @@ export function createProcessTreeOwner(root: ChildProcess, options: {
}
}
groups = [...byGroup.values()];
// Follow only continuously owned members of the admitted daemon tree.
// Orphaned descendants stay owned, but unrelated controller children do not.
const retainedPreserved = refreshContinuouslyLiveProcessGroups(revalidateObservedProcessGroups(preserved, next), next);
const preservedIds = new Set(retainedPreserved.map(group => group.processGroupId));
for (const member of retainedPreserved.flatMap(group => group.members)) {
for (const group of observeDescendantProcessTree(next, member.pid).groups) preservedIds.add(group.processGroupId);
}
preserved = groups.filter(group => preservedIds.has(group.processGroupId));
table = next;
})().finally(() => { observing = undefined; });
return observing;
@@ -60,6 +103,33 @@ export function createProcessTreeOwner(root: ChildProcess, options: {
function liveGroups() {
return groups.filter(group => table.some(row => row.processGroupId === group.processGroupId && running(row)));
}
async function preserveRunnerForRestart(identity: RestartRunnerIdentity) {
const expected = parseRestartRunnerIdentity(identity);
await observe();
if (restartRunner || exited()) throw new Error("Restart runner admission must precede controller exit");
const tree = observeDescendantProcessTree(table, root.pid!);
const candidate = tree.members.map(member => member.process).find(row => row.pid === expected.processPid);
if (!candidate || !running(candidate) || candidate.kind !== "paperclip-runnerd"
|| candidate.processGroupId !== expected.processGroupId
|| !restartProcessStartMatches(candidate.started, expected.processStartedAt)
|| candidate.processGroupId === root.pid) throw new Error("Restart runner is not the exact owned durable daemon");
const subtree = observeDescendantProcessTree(table, candidate.pid);
const members = new Set(subtree.members.map(member => member.process.pid));
const ids = new Set(subtree.groups.map(group => group.processGroupId));
if (table.some(row => ids.has(row.processGroupId) && !members.has(row.pid) && running(row))) {
throw new Error("Restart runner shares a process group with an unrelated process");
}
restartRunner = { ...candidate };
preserved = groups.filter(group => ids.has(group.processGroupId));
}
function assertRestartRunnerAlive() {
if (!restartRunner || !table.some(row => row.pid === restartRunner!.pid && row.started === restartRunner!.started
&& row.processGroupId === restartRunner!.processGroupId && running(row))) throw new Error("Admitted restart runner did not survive controller restart");
}
function restartCleanupGroups() {
const ids = new Set(preserved.map(group => group.processGroupId));
return new Set(liveGroups().filter(group => !ids.has(group.processGroupId)).map(group => group.processGroupId));
}
function signalSnapshot(signal: NodeJS.Signals, selected?: ReadonlySet<number>) {
const currentProcessGroupId = table.find(row => row.pid === process.pid)?.processGroupId ?? null;
if (currentProcessGroupId === null) throw new Error("Cleanup caller group identity is unavailable");
@@ -118,6 +188,7 @@ export function createProcessTreeOwner(root: ChildProcess, options: {
graceComplete = selected.size > 0 && delivered.length === selected.size;
}
return { observe, signal, liveGroups, gracefulRoots, signalGracefully,
preserveRunnerForRestart, assertRestartRunnerAlive, restartCleanupGroups,
directGraceDelivered: () => directGraceDelivered, stopObserving: () => { if (timer) clearInterval(timer); } };
}
tests/runner-e2e/process-tree.ts
+9 -2
View File
@@ -159,6 +159,14 @@ const diagnosticProcessKinds = new Set([
"tsx",
]);
export function diagnosticProcessKind(command: string, platform: NodeJS.Platform = process.platform) {
const name = path.basename(command);
// Linux comm is limited to 15 bytes. This is a diagnostic role only;
// preservation still requires the trusted run PID/start/group and ancestry.
if (platform === "linux" && name === "paperclip-runne") return "paperclip-runnerd";
return diagnosticProcessKinds.has(name) ? name : "other";
}
export async function readProcessTable(startInspector?: () => ChildProcess): Promise<ProcessObservation[] | null> {
if (process.platform === "win32") {
return null;
@@ -212,8 +220,7 @@ export async function readProcessTable(startInspector?: () => ChildProcess): Pro
// A target process can choose its own argv and process name. Emit a
// fixed category instead of target-controlled text so diagnostics
// can never turn that metadata into a secret-exfiltration channel.
const command = path.basename(match[6]!);
const kind = diagnosticProcessKinds.has(command) ? command : "other";
const kind = diagnosticProcessKind(match[6]!);
return {
pid: Number(match[1]),
parentPid: Number(match[2]),
tests/runner-e2e/remote-native-bootstrap.test.ts
+22 -1
View File
@@ -1,4 +1,5 @@
import { explicitlyRequestsFileOutput } from "../../server/src/services/native-runtime/native-deliverable-feedback.js";
import { copilotProtectionTasks } from "./copilot-protection-tasks.js";
import type { APIRequestContext, APIResponse } from "@playwright/test";
import { afterEach, expect, it, vi } from "vitest";
import { classifyFailure } from "./failure-classifier.js";
@@ -11,7 +12,7 @@ afterEach(() => { vi.useRealTimers(); vi.unstubAllEnvs(); });
function harness(timeoutMs = REMOTE_FIXTURE_MIN_SETUP_BUDGET_MS + 18_000) {
const order: string[] = [];
const issue = { id: "issue", companyId: "company", assigneeAgentId: "agent" };
const run = { id: "run", companyId: "company", agentId: "agent", status: "running", executionStage: "preparing" };
const run = { id: "run", companyId: "company", agentId: "agent", status: "running", executionStage: "preparing", runtimeMode: "native", nativePhase: "provider_running" };
const leases = [{ id: "lease", heartbeatRunId: "run", issueId: "issue", status: "active", providerLeaseId: "sandbox" }];
const api = { get: vi.fn(async (path: string) => path === "/api/issues/issue" ? issue : path === "/api/heartbeat-runs/run" ? run : leases) };
const fixture = {
@@ -48,6 +49,19 @@ it("keeps negated bootstrap instructions out of the production file-delivery con
expect(explicitlyRequestsFileOutput(`${prompt}\nCreate a downloadable report.txt file.`)).toBe(true);
});
it("keeps async completion immediate with a private sentinel and no delivery wait", () => {
const task = copilotProtectionTasks.find(task => task.id === "attached-async-settlement")!;
const action = task.buildPrompt("nonce");
expect(explicitlyRequestsFileOutput(action)).toBe(false);
expect(explicitlyRequestsFileOutput(`${harness().bootstrap.prompt("nonce")}\n${action}`)).toBe(false);
expect(action).toContain("attempt to finish immediately without calling read_bash or another waiting tool");
expect(action).toContain("Do not modify fixture code, manufacture its result, or start another command.");
expect(action).toContain("private diagnostic sentinel, not a requested file deliverable");
expect(action).toContain("Do not publish it. Do not attach it.");
expect(explicitlyRequestsFileOutput(`${action}\nAttach a downloadable report.txt file.`)).toBe(true);
expect(action).toContain("evidence [], verification []");
expect(action).not.toContain("register_deliverable");
});
it("keeps the complete readiness/install reserve when a lease arrives at the admission boundary", async () => {
vi.useFakeTimers(); vi.setSystemTime(0);
@@ -106,6 +120,13 @@ it("waits through queued admission without publishing early", async () => {
expect(count).toBe(2); expect(h.fixture.publishAction).toHaveBeenCalledTimes(1);
});
it.each(["provider_running", "observed"])("admits a native %s run while its legacy stage stays preparing", async nativePhase => {
const h = harness(); h.bootstrap.prompt("native-ready"); h.run.nativePhase = nativePhase;
await expect(h.bootstrap.bindAndRelease(h.request)).resolves.toBe(h.fixture);
expect(h.run.executionStage).toBe("preparing");
expect(h.bind).toHaveBeenCalledTimes(1); expect(h.fixture.publishAction).toHaveBeenCalledTimes(1);
});
it.each(["", "x".repeat(16385)])("rejects empty or over-bound action after closing only its observer", async action => {
const h = harness(); h.bootstrap.prompt("nonce");
await expect(h.bootstrap.bindAndRelease({ ...h.request, actionPrompt: () => action })).rejects.toThrow("empty or too large");
tests/runner-e2e/remote-native-bootstrap.ts
+2
View File
@@ -173,6 +173,8 @@ export function createRemoteNativeBootstrap(input: {
if (lastReadError !== undefined) throw lastReadError;
return state;
},
// Native runs can retain the legacy "preparing" stage. The observer's
// runtime-ready RPC proves the actual pinned daemon before installation.
accept: state => !state.rejection && state.owned && state.run?.status === "running" && Boolean(state.lease)
&& Date.now() < admissionDeadlineAt,
reject: state => state.rejection,
tests/runner-e2e/remote-native-fixtures.test.ts
+162 -17
View File
@@ -7,8 +7,9 @@ import { tmpdir } from "node:os";
import { join } from "node:path";
import { Script } from "node:vm";
import { describe, expect, it, vi } from "vitest";
import { REMOTE_FIXTURE_MIN_SETUP_BUDGET_MS, bindRemoteNativeFixture, createRemoteTargetWatch, isRemoteRunRoot, parseRemoteProcStat, remoteNativeFixtureDiagnostics, type RemoteNativeFixtureOptions, type RemoteNativeSnapshot } from "./remote-native-fixtures.js";
import { REMOTE_FIXTURE_MIN_SETUP_BUDGET_MS, bindRemoteNativeFixture, validatePiProviderDeathReceipt, createRemoteTargetWatch, isRemoteRunRoot, parseRemoteProcStat, remoteProcEntryDisappeared, remoteNativeFixtureDiagnostics, remoteNativeIncompleteTerminalEvidence, type RemoteNativeFixtureOptions, type RemoteNativeSnapshot } from "./remote-native-fixtures.js";
import { PI_DISTRIBUTION_CLOSURE_SHA256 } from "../../packages/paperclip-runner/src/drivers/acpx/pi-closure-pins.js";
import { createRemoteNativeBootstrap } from "./remote-native-bootstrap.js";
const hash = (s: string) => `sha256:${createHash("sha256").update(s).digest("hex")}`;
@@ -52,6 +53,19 @@ function harness() {
setLease(value: Record<string, unknown>) { lease = value; }, lease: () => lease, override(fn: typeof override) { override = fn; } };
}
describe("remote process exit during proc reads", () => {
it.each(["ENOENT", "ESRCH"])("accepts confirmed %s process absence", code => {
const confirm = vi.fn();
expect(remoteProcEntryDisappeared({ code }, confirm)).toBe(true);
expect(confirm).not.toHaveBeenCalled();
expect(remoteProcEntryDisappeared({ code: "EIO" }, () => { throw { code }; })).toBe(true);
});
it.each(["EIO", "EACCES", undefined])("rejects an unreadable existing process (%s)", code => {
expect(remoteProcEntryDisappeared({ code }, () => ({ isDirectory: () => true }))).toBe(false);
expect(remoteProcEntryDisappeared({ code }, () => { throw { code: "EACCES" }; })).toBe(false);
});
});
describe("remote native lease admission", () => {
it.each(["companyId", "environmentId", "heartbeatRunId", "providerLeaseId", "provider", "status"])("rejects wrong %s before executing any remote command", async key => {
const h = harness(); h.setLease({ ...h.lease(), [key]: "foreign" });
@@ -232,19 +246,63 @@ describe("remote native lease admission", () => {
if (type === "bad-file") end.targets["result.txt"] = { ...end.targets["result.txt"], absent: false, sha256: hash("missing") };
h.resolveTerminal(end); await expect(f.finish()).rejects.toThrow();
});
it("reports only closed terminal failure reasons without weakening retirement proof", async () => {
it("retains a validated incomplete terminal receipt without raw file or RPC content", async () => {
const h = harness(), f = await bindRemoteNativeFixture(h.options);
await f.publishAction("action.txt", "task");
const end = { ...structuredClone(h.current), complete: false,
processes: { ...h.current.processes, live: [] }, files: {},
failureCodes: ["process_pid_reused", "secret-provider-payload", "process_pid_reused", { arbitrary: "payload" }] };
const end = { ...structuredClone(h.current), complete: false, watcher: { ...h.current.watcher, complete: false },
processes: { ...h.current.processes, live: [] }, files: { "private.txt": "PRIVATE RPC CONTENT" }, untrustedDump: "PRIVATE RPC CONTENT" };
h.resolveTerminal(end);
const error = await f.finish().catch(error => error);
expect(error.message).toContain("terminal_evidence_incomplete:published=true,complete=false");
expect(error.message).toContain(":process_pid_reused");
expect(error.message).not.toContain("secret-provider-payload");
expect(error.message).not.toContain("arbitrary");
expect(remoteNativeFixtureDiagnostics(error)).toEqual([{ phase: "wait", code: "terminal_evidence_incomplete" }]);
expect(error.message).toBe("remote_native_fixture:terminal_evidence_incomplete");
const retained = remoteNativeIncompleteTerminalEvidence(error)!;
expect(retained.complete).toBe(false); expect(retained.watcher.complete).toBe(false);
expect(retained.processes.liveCount).toBe(0);
expect(retained.incompleteReasons).toEqual([]);
expect(JSON.stringify(retained)).not.toContain("PRIVATE RPC CONTENT");
retained.watcher.complete = true;
expect(remoteNativeIncompleteTerminalEvidence(error)!.watcher.complete).toBe(false);
expect(remoteNativeIncompleteTerminalEvidence(new Error("PRIVATE RPC CONTENT"))).toBeUndefined();
});
it("retains only closed incompleteness reasons and rejects raw diagnostics", async () => {
const h = harness(), f = await bindRemoteNativeFixture(h.options); await f.publishAction("action.txt", "task");
h.resolveTerminal({ ...structuredClone(h.current), complete: false, incompleteReasons: ["unwatched_directory"],
processes: { ...h.current.processes, live: [] }, files: {} });
const error = await f.finish().catch(error => error);
expect(remoteNativeIncompleteTerminalEvidence(error)?.incompleteReasons).toEqual(["unwatched_directory"]);
await f.close();
const bad = harness(), other = await bindRemoteNativeFixture(bad.options); await other.publishAction("action.txt", "task");
bad.resolveTerminal({ ...structuredClone(bad.current), complete: false, incompleteReasons: ["PRIVATE RAW PATH"],
processes: { ...bad.current.processes, live: [] }, files: {} });
const rejected = await other.finish().catch(error => error);
expect(rejected.message).toContain("incomplete_reasons_shape");
expect(remoteNativeIncompleteTerminalEvidence(rejected)).toBeUndefined();
await other.close();
});
it("keeps incomplete evidence failed while closing a retired observer after public lease deletion", async () => {
const h = harness(), f = await bindRemoteNativeFixture(h.options);
await f.publishAction("action.txt", "task");
h.resolveTerminal({ ...structuredClone(h.current), complete: false, incompleteReasons: ["unwatched_directory"],
watcher: { ...h.current.watcher, complete: false }, processes: { ...h.current.processes, live: [] }, files: {} });
await expect(f.finish()).rejects.toThrow("terminal_evidence_incomplete");
h.setLease({ ...h.lease(), status: "released", releasedAt: "2026-10-06T01:00:00Z" });
const before = h.calls.length;
await expect(f.close()).resolves.toBeUndefined();
expect(h.calls).toHaveLength(before);
await expect(f.finish()).rejects.toThrow("terminal_evidence_incomplete");
await expect(f.readFile("result.txt")).rejects.toThrow("unregistered_read");
});
it.each(["live", "missing-root", "invalid-shape", "unknown-cause", "reused-process", "attached-live"])("does not treat %s evidence as retired cleanup", async kind => {
const h = harness(), f = await bindRemoteNativeFixture(h.options);
await f.publishAction("action.txt", "task");
const end: any = { ...structuredClone(h.current), complete: false, processes: { ...h.current.processes, live: [] }, files: {} };
if (kind === "live") end.processes.live = [21];
if (kind === "missing-root") end.processes = { captured: false, root: null, journal: [], live: [] };
if (kind === "invalid-shape") end.observedAtMs = "invalid";
if (kind === "reused-process") end.incompleteReasons = ["process_identity_reused"];
if (kind === "attached-live") end.incompleteReasons = ["attached_process_live"];
h.resolveTerminal(end); await expect(f.finish()).rejects.toThrow();
h.setLease({ ...h.lease(), status: "released", releasedAt: "2026-10-06T01:00:00Z" });
await expect(f.close()).rejects.toThrow("lease_scope");
});
it("keeps a fixture-owned cross-root sentinel distinct from workspace targets", async () => {
const h = harness(); h.options.crossRoot = { initialText: "outside sentinel" };
@@ -334,7 +392,7 @@ describe("cell deadline and cleanup bounds", () => {
});
describe("independent filesystem and process observations", () => {
it.skipIf(process.platform !== "linux")("retains transient create/delete events and detects same-path parent replacement", async () => {
it("retains transient create/delete events and detects same-path parent replacement", async () => {
const dir = await mkdtemp(join(tmpdir(), "remote-watch-")); const watcher = createRemoteTargetWatch(dir, "denied.txt");
try {
await writeFile(join(dir, "denied.txt"), "forbidden"); await rm(join(dir, "denied.txt"));
@@ -378,7 +436,7 @@ describe("actual generated observer state machine", () => {
const handlers: Array<(socket: any) => void> = [], children: any[] = [];
const missing = () => { throw Object.assign(new Error("missing"), { code: "ENOENT" }); };
const fds = new Map<number, string>(); let nextFd = 50, runtimeInode = 4n;
const symbolicLinks = new Set<string>();
const symbolicLinks = new Set<string>(), directoryInodes = new Map<string, bigint>();
const directories = new Set(["/tmp", "/workspace", "/workspace/.paperclip-runtime", "/workspace/.paperclip-runtime/paperclip-runner", "/workspace/.paperclip-runtime/paperclip-runner/sessions"]);
if (!deferStartup) directories.add(config.root);
const listeners = new Map<string, (socket: any) => void>();
@@ -402,10 +460,10 @@ describe("actual generated observer state machine", () => {
lstatSync(path: string) {
const directory = directories.has(path);
if (!directory && !files.has(path) && !symbolicLinks.has(path)) return missing();
return { dev: 1n, ino: path === config.root ? 2n : path === "/workspace/.paperclip-runtime/paperclip-runner" ? runtimeInode : 3n, mtimeNs: 4n, ctimeNs: 5n, isDirectory: () => directory, isFile: () => !directory, isSymbolicLink: () => symbolicLinks.has(path), size: files.get(path)?.length ?? 0 };
return { dev: 1n, ino: directoryInodes.get(path) ?? (path === config.root ? 2n : path === "/workspace/.paperclip-runtime/paperclip-runner" ? runtimeInode : 3n), mtimeNs: 4n, ctimeNs: 5n, isDirectory: () => directory, isFile: () => !directory, isSymbolicLink: () => symbolicLinks.has(path), size: files.get(path)?.length ?? 0 };
},
realpathSync: (path: string) => path,
readdirSync(path: string) { if (path === "/proc") return [...proc.keys()].map(String); if (path === "/workspace") return [".paperclip-runtime", ...[...files.keys(), ...symbolicLinks].filter(p => p.startsWith("/workspace/") && !p.slice(11).includes("/")).map(p => p.slice(11))]; if (path === "/workspace/.paperclip-runtime") return ["reusable-sandbox-lease.json", "paperclip-runner", ...[...files.keys()].filter(p => p.startsWith(path + "/") && !p.slice(path.length + 1).includes("/") && !p.endsWith("reusable-sandbox-lease.json")).map(p => p.slice(path.length + 1))]; if (path.startsWith("/workspace/.paperclip-runtime/paperclip-runner")) throw new Error("excluded runtime must not be traversed"); return []; },
readdirSync(path: string) { if (path === "/proc") return [...proc.keys()].map(String); if (path.startsWith("/workspace/.paperclip-runtime/paperclip-runner")) throw new Error("excluded runtime must not be traversed"); return [...new Set([...files.keys(), ...directories, ...symbolicLinks].filter(p => p.startsWith(path + "/") && !p.slice(path.length + 1).includes("/")).map(p => p.slice(path.length + 1)))]; },
watch(path: string, options: unknown, callback?: (_kind: string, name: string | null) => void) {
const entry = { path, callback: (callback ?? options) as (_kind: string, name: string | null) => void, closed: false }; watches.push(entry);
return Object.assign(new EventEmitter(), { close: () => { entry.closed = true; } });
@@ -432,8 +490,9 @@ describe("actual generated observer state machine", () => {
close: vi.fn(() => listeners.clear()),
};
const net = { createServer(fn: (socket: any) => void) { handlers.push(fn); return server; } };
const faultSpawn = vi.fn(() => ({ status: 0, stdout: JSON.stringify(piFaultReceipt()) }));
const context = {
require(name: string) { if (name === "node:fs") return fs; if (name === "node:net") return net; if (name === "node:child_process") return { spawn: vi.fn(() => { const child = Object.assign(new EventEmitter(), { pid: 88, exitCode: null, signalCode: null, kill: vi.fn() }); children.push(child); return child; }) }; if (name === "node:path") return { join: (...paths: string[]) => paths.join("/"), dirname: (path: string) => path.slice(0, path.lastIndexOf("/")), basename: (path: string) => path.slice(path.lastIndexOf("/") + 1) }; if (name === "node:crypto") return { createHash }; throw new Error("unexpected module"); },
require(name: string) { if (name === "node:fs") return fs; if (name === "node:net") return net; if (name === "node:child_process") return { spawnSync: faultSpawn, spawn: vi.fn(() => { const child = Object.assign(new EventEmitter(), { pid: 88, exitCode: null, signalCode: null, kill: vi.fn() }); children.push(child); return child; }) }; if (name === "node:path") return { join: (...paths: string[]) => paths.join("/"), dirname: (path: string) => path.slice(0, path.lastIndexOf("/")), basename: (path: string) => path.slice(path.lastIndexOf("/") + 1) }; if (name === "node:crypto") return { createHash }; throw new Error("unexpected module"); },
process: { argv: ["node", `${config.root}/observer.cjs`, Buffer.from(JSON.stringify(config)).toString("base64")], execPath: "/node", hrtime: { bigint: () => 12345n }, exit: vi.fn() },
Buffer, __filename: `${config.root}/observer.cjs`,
setInterval(fn: () => void) { intervals.push(fn); return 1; }, clearInterval: vi.fn(),
@@ -445,8 +504,56 @@ describe("actual generated observer state machine", () => {
const replies: any[] = [], socket = Object.assign(new EventEmitter(), { end: (value: string) => replies.push(JSON.parse(value)), destroy: vi.fn() });
handlers[0]!(socket); socket.emit("data", Buffer.from(JSON.stringify({ op, nonce: config.nonce, ...args }) + "\n")); return replies;
}
return { request, proc, files, watches, fs, intervals, timers, config, handlers, children, symbolicLinks, context, server, directories, listeners, install: h.calls.find(c => c.request.op === "install")!, replaceRuntimeRoot() { runtimeInode = 999n; } };
return { request, proc, files, watches, fs, intervals, timers, config, handlers, children, faultSpawn, symbolicLinks, directoryInodes, context, server, directories, listeners, install: h.calls.find(c => c.request.op === "install")!, replaceRuntimeRoot() { runtimeInode = 999n; } };
}
it("keeps existing watched directory notifications complete without losing nested mutations", async () => {
const o = await observerHarness(true);
o.directories.add(o.config.root); o.files.set(`${o.config.root}/observer.cjs`, Buffer.from(o.install.request.source));
o.directories.add("/workspace/existing");
new Script(o.install.request.source).runInNewContext(o.context);
// Linux emits a parent notification when chmod changes a child's directory
// attributes. Its inode and registered recursive watch remain unchanged.
for (const w of o.watches) if (w.path === "/workspace") w.callback("change", "existing");
expect(o.request("snapshot")[0].result.complete).toBe(true);
o.fs.writeFileSync("/workspace/existing/transient.txt", "changed", { flag: "wx" });
o.files.delete("/workspace/existing/transient.txt");
o.watches.find(w => w.path === "/workspace/existing")!.callback("rename", "transient.txt");
const final = o.request("snapshot")[0].result;
expect(final.complete).toBe(true); expect(final.watcher.workspaceMutationCount).toBe(3);
expect(final.workspace["existing/transient.txt"]).toBeUndefined();
});
it.each(["new", "replacement", "symlink", "unknown"])("rejects %s directory notifications", async variant => {
const o = await observerHarness(true);
o.directories.add(o.config.root); o.files.set(`${o.config.root}/observer.cjs`, Buffer.from(o.install.request.source));
if (variant !== "new") o.directories.add("/workspace/existing");
new Script(o.install.request.source).runInNewContext(o.context);
if (variant === "new") o.directories.add("/workspace/existing");
if (variant === "replacement") o.directoryInodes.set("/workspace/existing", 999n);
if (variant === "symlink") o.symbolicLinks.add("/workspace/existing");
for (const w of o.watches) if (w.path === "/workspace") w.callback("change", variant === "unknown" ? null : "existing");
const reply = o.request("snapshot")[0];
expect(reply.ok && reply.result.complete).toBe(false);
});
it("executes the actual observer fault phase once with a closed environment and exact identity", async () => {
for (const published of [false, true]) {
const rejected = await observerHarness();
if (published) rejected.request("publish", { path: "action.txt", text: "ask native input" });
expect(rejected.request("pi-provider-death", { runtimeEnvironmentLeaseId: published ? "foreign" : "workspace-id" })[0].ok).toBe(false);
expect(rejected.faultSpawn).not.toHaveBeenCalled();
}
const o = await observerHarness();
expect(o.request("snapshot")[0].ok).toBe(true);
expect(o.request("publish", { path: "action.txt", text: "ask native input" })[0].ok).toBe(true);
expect(o.request("pi-provider-death", { runtimeEnvironmentLeaseId: "workspace-id" })[0]).toEqual({ ok: true, result: piFaultReceipt() });
expect(o.faultSpawn).toHaveBeenCalledTimes(1);
const [program, args, options] = o.faultSpawn.mock.calls[0] as unknown as [string, string[], Record<string, unknown>];
expect(program).toBe("/usr/bin/python3"); expect(args.slice(0, 2)).toEqual(["-I", "-c"]);
expect(args[2]).toContain("signal.pidfd_send_signal");
expect(JSON.parse(Buffer.from(args[3]!, "base64").toString())).toEqual({ root, binding, runtimeEnvironmentLeaseId: "workspace-id", runnerdSha256: hash("runnerd"), closureSha256: PI_DISTRIBUTION_CLOSURE_SHA256["linux-x64"] });
expect(options).toMatchObject({ env: { PATH: "/usr/bin:/bin" }, timeout: 8000, maxBuffer: 32768 });
expect(o.request("pi-provider-death", { runtimeEnvironmentLeaseId: "workspace-id" })[0].ok).toBe(false);
expect(o.faultSpawn).toHaveBeenCalledTimes(1);
});
async function generatedRpc(o: Awaited<ReturnType<typeof observerHarness>>, request: Record<string, unknown>, mutateSource = (source: string) => source) {
const quoted = o.install.command.match(/ -e (.+) '[A-Za-z0-9+/=]+'$/su)![1]!;
const source = quoted.slice(1, -1).replaceAll("'\\''", "'");
@@ -689,7 +796,7 @@ describe("actual generated observer state machine", () => {
const o = await observerHarness(); o.request("snapshot"); const wait = o.request("wait");
o.proc.set(21, { ppid: 1, group: 99, ticks: "999", argv: ["/unrelated"] }); o.intervals[0]!(); o.timers.find(t => t.ms === 100)!.fn();
expect(wait[0].result.complete).toBe(false); expect(wait[0].result.processes.root.startTicks).toBe("100");
expect(wait[0].result.failureCodes).toContain("process_pid_reused");
expect(wait[0].result.incompleteReasons).toContain("process_identity_reused");
});
it("counts transient workspace create/delete and rejects changed setup-file bytes", async () => {
const o = await observerHarness(); o.request("snapshot");
@@ -707,3 +814,41 @@ describe("actual generated observer state machine", () => {
expect(other.request("snapshot")[0].ok).toBe(false);
});
});
function piFaultReceipt() {
const parent = { pid: 22, ppid: 21, startTicks: "101", bootId }, target = { pid: 23, ppid: 22, startTicks: "102", bootId };
return { schema: "paperclip.e2e.pi-provider-death.v1", binding, runtimeEnvironmentLeaseId: "workspace-id", target, ancestry: [target, parent, root],
nodeSha256: hash("node"), entrypointSha256: hash("entry"), closureSha256: PI_DISTRIBUTION_CLOSURE_SHA256["linux-x64"],
entrypointAttribution: "pinned_wrapper_parent", originalChildArgvAvailable: false, observedChildTitle: "pi", signalled: true, signal: "SIGKILL", targetKind: "pi_native_child", workerSignalled: false };
}
describe("exact Pi-child fault capability", () => {
it("validates ancestry and honest parent-attested entrypoint identity", () => {
const receipt = piFaultReceipt();
expect(validatePiProviderDeathReceipt(receipt, binding, root, "workspace-id")).toEqual(receipt);
for (const patch of [{ workerSignalled: true }, { target: root }, { originalChildArgvAvailable: true }, { entrypointAttribution: "process_title" },
{ binding: { ...binding, runId: "foreign" } }, { runtimeEnvironmentLeaseId: "foreign" }, { closureSha256: "0".repeat(64) },
{ ancestry: [receipt.target, { ...receipt.ancestry[1], startTicks: "" }, root] }, { ancestry: [receipt.target, root] },
{ ancestry: [receipt.target, { ...receipt.ancestry[1], ppid: 999 }, root] }, { ancestry: [receipt.target, receipt.ancestry[1], { ...root, startTicks: "999" }] }]) {
expect(() => validatePiProviderDeathReceipt({ ...receipt, ...patch }, binding, root, "workspace-id")).toThrow();
}
});
it("requires published action and consumes the one-shot capability even on RPC failure", async () => {
const h = harness(), fixture = await bindRemoteNativeFixture(h.options);
await expect(fixture.terminatePiProvider!("workspace-id")).rejects.toThrow();
await fixture.publishAction("action.txt", "Ask the native question");
h.override(request => request.op === "pi-provider-death" ? { exitCode: 0, result: JSON.stringify({ ok: false, error: "pi_provider_identity_or_signal_failed" }) } : undefined);
await expect(fixture.terminatePiProvider!("workspace-id")).rejects.toThrow();
await expect(fixture.terminatePiProvider!("workspace-id")).rejects.toThrow();
expect(h.calls.filter(call => call.request.op === "pi-provider-death")).toHaveLength(1);
await fixture.close();
});
it("returns only the exact reviewed fault receipt and never signals the worker", async () => {
const h = harness(), fixture = await bindRemoteNativeFixture(h.options);
await fixture.publishAction("action.txt", "Ask the native question");
h.override(request => request.op === "pi-provider-death" ? { exitCode: 0, result: JSON.stringify({ ok: true, result: piFaultReceipt() }) } : undefined);
expect(await fixture.terminatePiProvider!("workspace-id")).toEqual(piFaultReceipt());
await expect(fixture.terminatePiProvider!("workspace-id")).rejects.toThrow();
expect(h.calls.filter(call => call.request.op === "pi-provider-death")).toHaveLength(1);
await fixture.close();
});
});
tests/runner-e2e/remote-native-fixtures.ts
+110 -27
View File
@@ -1,7 +1,10 @@
import { createHash, randomBytes } from "node:crypto";
import { watch, lstatSync } from "node:fs";
import { watch, lstatSync, readFileSync } from "node:fs";
import { posix } from "node:path";
import { PI_DISTRIBUTION_CLOSURE_SHA256 } from "../../packages/paperclip-runner/src/drivers/acpx/pi-closure-pins.js";
const PI_FAULT_SOURCE = readFileSync(new URL("./pi-provider-fault.py", import.meta.url), "utf8");
export const REMOTE_FIXTURE_DAYTONA_SDK_VERSION = "0.203.0";
const NODE = "/opt/paperclip-runner/provider-pack/node_modules/node/bin/node";
const MAX_OUTPUT = 256 * 1024;
@@ -30,8 +33,8 @@ function relative(value: string): string {
}
// Only closed diagnostic enums cross the remote boundary; SDK errors and output
// are never retained. These diagnostics explain failed evidence, not qualification.
const RPC_PHASES = ["runtime-ready", "install", "wait", "close", "arm", "publish", "snapshot", "attached", "read", "inject-loss"] as const;
const RPC_CODES = ["node_identity", "sentinel_type", "sentinel", "cwd", "runtime_root_identity", "runtime_binary_identity", "proc_bound", "ambiguous_run_root", "runtime_not_ready", "runtime_identity_changed", "invalid_proc_identity", "invalid_proc_fields", "socket_error", "socket_timeout", "output_bound", "rpc_deadline", "remote_unknown", "transport_failure", "invalid_response", "readiness_deadline", "terminal_evidence_incomplete"] as const;
const RPC_PHASES = ["runtime-ready", "install", "wait", "close", "arm", "publish", "snapshot", "attached", "read", "pi-provider-death"] as const;
const RPC_CODES = ["node_identity", "sentinel_type", "sentinel", "cwd", "runtime_root_identity", "runtime_binary_identity", "proc_bound", "ambiguous_run_root", "runtime_not_ready", "runtime_identity_changed", "invalid_proc_identity", "invalid_proc_fields", "socket_error", "socket_timeout", "output_bound", "rpc_deadline", "remote_unknown", "transport_failure", "invalid_response", "readiness_deadline", "pi_provider_identity_or_signal_failed"] as const;
type RpcPhase = typeof RPC_PHASES[number];
type RpcCode = typeof RPC_CODES[number];
interface RpcDiagnostic { phase: RpcPhase; code: RpcCode }
@@ -48,12 +51,16 @@ export function remoteNativeFixtureDiagnostics(error: unknown): RpcDiagnostic[]
export interface RemoteNativeAuthority { companyId: string; environmentId: string; runId: string; leaseId: string; sandboxId: string; image: string }
export interface RemoteNativeBinding extends RemoteNativeAuthority { remoteCwd: string }
export interface RemoteProcessIdentity { pid: number; ppid: number; startTicks: string; bootId: string }
const INCOMPLETE_REASONS = ["multiple_run_roots", "run_root_changed", "process_identity_reused", "process_sample_failed", "process_read_io", "process_stat_shape", "process_runtime_binding", "process_terminal_seal_failed", "unknown_watch_name", "setup_file_changed", "workspace_event_bound", "workspace_symlink", "unwatched_directory", "directory_identity_changed", "watch_stat_failed", "workspace_watch_error", "target_watch_incomplete", "receipt_byte_bound", "attached_process_live", "control_request_bound", "observer_ttl"] as const;
type IncompleteReason = typeof INCOMPLETE_REASONS[number];
const FILESYSTEM_INCOMPLETE_REASONS: readonly IncompleteReason[] = ["unknown_watch_name", "setup_file_changed", "workspace_event_bound", "workspace_symlink", "unwatched_directory", "directory_identity_changed", "watch_stat_failed", "workspace_watch_error", "target_watch_incomplete"];
export interface RemoteNativeSnapshot {
binding: RemoteNativeBinding;
observedAtMs: number;
observedMonotonicNs: string;
receivedAtMs: number;
complete: boolean;
incompleteReasons?: IncompleteReason[];
workspace: Record<string, string>;
targets: Record<string, { absent: boolean; sha256: string | null; parent: { dev: string; ino: string }; mutationCount: number; complete: boolean }>;
watcher: { complete: boolean; targetMutationCount: number; workspaceMutationCount: number };
@@ -67,6 +74,26 @@ export interface RemoteNativeSnapshot {
setup: { path: string; sha256: string | null; published: boolean };
attached: { connections: number; failure: string | null; commandExit: { code: number; observedAtMs: number; observedMonotonicNs: string } | null; markerWrittenAtMs: number | null; markerWrittenMonotonicNs: string | null; clientExitedAtMs: number | null; clientExitedMonotonicNs: string | null } | null;
}
class IncompleteRemoteTerminalEvidenceError extends Error {
constructor(readonly snapshot: RemoteNativeSnapshot) {
super("remote_native_fixture:terminal_evidence_incomplete");
}
}
/** Retain only a shape-validated public receipt, without raw RPC text or files. */
export function remoteNativeIncompleteTerminalEvidence(error: unknown) {
if (!(error instanceof IncompleteRemoteTerminalEvidenceError)) return undefined;
const row = error.snapshot, root = row.processes.root;
return {
observedAtMs: row.observedAtMs, receivedAtMs: row.receivedAtMs, complete: row.complete,
incompleteReasons: [...(row.incompleteReasons ?? [])],
setupPublished: row.setup.published,
watcher: { complete: row.watcher.complete, targetMutationCount: row.watcher.targetMutationCount, workspaceMutationCount: row.watcher.workspaceMutationCount },
processes: { captured: row.processes.captured, liveCount: row.processes.live.length, journalCount: row.processes.journal.length,
root: root ? { pid: root.pid, ppid: root.ppid, startTicks: root.startTicks, bootId: root.bootId } : null },
targets: Object.fromEntries(Object.entries(row.targets).map(([name, target]) => [name,
{ absent: target.absent, sha256: target.sha256, mutationCount: target.mutationCount, complete: target.complete }])),
};
}
/** Structural subset of pinned SDK0.203.0; caller supplies its authenticated
* client. No create/list/delete or arbitrary remote command API is exposed. */
export interface RemoteFixtureDaytona {
@@ -84,6 +111,17 @@ export function parseRemoteProcStat(pid: number, stat: string, bootId: string):
if (!/^\d+$/u.test(fields[19] ?? "") || !/^\d+$/u.test(fields[1] ?? "") || !/^\d+$/u.test(fields[2] ?? "")) throw new Error("invalid_proc_fields");
return { pid, ppid: Number(fields[1]), group: Number(fields[2]), state: fields[0]!, startTicks: fields[19]!, bootId };
}
/** Linux proc files can fail while a process exits. Confirm absence separately;
* an unreadable process that still exists remains a sampling failure. */
export function remoteProcEntryDisappeared(error: unknown, confirmPresence: () => unknown): boolean {
const code = (error as NodeJS.ErrnoException)?.code;
if (code === "ENOENT" || code === "ESRCH") return true;
try { confirmPresence(); return false; }
catch (confirmation) {
const missing = (confirmation as NodeJS.ErrnoException)?.code;
return missing === "ENOENT" || missing === "ESRCH";
}
}
export function isRemoteRunRoot(argv: string[], runId: string, process: RemoteProcessIdentity & { group: number }): boolean {
if (!/^[a-zA-Z0-9_-]{1,128}$/u.test(runId) || process.group !== process.pid || !argv[0]?.endsWith("/paperclip-runnerd")) return false;
return [["--run-id", runId], ["--lifecycle-mode", "per_turn"]].every(([flag, value]) => {
@@ -121,25 +159,25 @@ export function createRemoteTargetWatch(directory: string, name: string, io = {
const OBSERVER = String.raw`
const fs=require('node:fs'),path=require('node:path'),crypto=require('node:crypto'),net=require('node:net'),cp=require('node:child_process');
const config=JSON.parse(Buffer.from(process.argv[2],'base64').toString());
const parseStat=PARSE_STAT;const runRoot=RUN_ROOT;const watchTarget=WATCH_TARGET;
const parseStat=PARSE_STAT;const runRoot=RUN_ROOT;const vanished=PROC_ENTRY_DISAPPEARED;const watchTarget=WATCH_TARGET;const PI_FAULT_SOURCE=PI_FAULT_PROGRAM,PI_CLOSURE_PIN=PI_CLOSURE_DIGEST;
const hash=x=>'sha256:'+crypto.createHash('sha256').update(x).digest('hex');
const cwdStat=fs.lstatSync(config.binding.remoteCwd,{bigint:true}),rootStat=fs.lstatSync(config.root,{bigint:true}),scriptStat=fs.lstatSync(__filename,{bigint:true}),scriptHash=hash(fs.readFileSync(__filename));
const runtimeRoot=path.join(config.binding.remoteCwd,config.runtimeRelative),runtimeStat=fs.lstatSync(runtimeRoot,{bigint:true});if(!runtimeStat.isDirectory()||runtimeStat.isSymbolicLink()||fs.realpathSync(runtimeRoot)!==runtimeRoot)throw Error('runtime_root_identity');let observedPrpEnvironmentLeaseId=null;
const boot=fs.readFileSync('/proc/sys/kernel/random/boot_id','utf8').trim();
const targets=new Map();let complete=true,sealed=false,root=null,attached=null,child=null,client=null,childTimer=null;
const journal=new Map(),sockets=new Set(),waiters=new Set(),armWaiters=new Set(),failureCodes=new Set();let observedRootCount=0,publishedHash=null,finalReceipt=null,retiring=false;
function incomplete(code){complete=false;failureCodes.add(code)}
const incompleteReasons=new Set();function incomplete(reason){complete=false;incompleteReasons.add(reason)}
const journal=new Map(),sockets=new Set(),waiters=new Set(),armWaiters=new Set();let observedRootCount=0,publishedHash=null,finalReceipt=null,retiring=false,piFaultAttempted=false;
function identity(pid){return parseStat(pid,fs.readFileSync('/proc/'+pid+'/stat','utf8'),boot)}
function table(){const ids=fs.readdirSync('/proc').filter(x=>/^\d+$/.test(x)&&Number(x)>1);if(ids.length>4096)throw Error('process_bound');return ids.flatMap(x=>{try{return [identity(Number(x))]}catch(e){if(e.code==='ENOENT'||e.code==='ESRCH')return [];throw e}})}
function table(){const ids=fs.readdirSync('/proc').filter(x=>/^\d+$/.test(x)&&Number(x)>1);if(ids.length>4096)throw Error('process_bound');return ids.flatMap(x=>{try{return [identity(Number(x))]}catch(e){if(vanished(e,()=>fs.lstatSync('/proc/'+x)))return [];throw e}})}
function sample(){
const all=table();const candidates=[];
for(const p of all){if(p.state==='Z')continue;let argv;try{argv=fs.readFileSync('/proc/'+p.pid+'/cmdline').toString().split('\0').filter(Boolean)}catch(e){if(e.code==='ENOENT'||e.code==='ESRCH')continue;throw e}
for(const p of all){if(p.state==='Z')continue;let argv;try{argv=fs.readFileSync('/proc/'+p.pid+'/cmdline').toString().split('\0').filter(Boolean)}catch(e){if(vanished(e,()=>fs.lstatSync('/proc/'+p.pid)))continue;throw e}
if(runRoot(argv,config.binding.runId,p)){const at=argv.indexOf('--environment-lease-id'),stateAt=argv.indexOf('--state-dir');if(at<1||argv.lastIndexOf('--environment-lease-id')!==at||!/^[-a-zA-Z0-9._:]{1,256}$/.test(argv[at+1]??''))throw Error('process_prp_identity_shape');if(argv[0]!==path.join(runtimeRoot,'bin','paperclip-runnerd')||stateAt<1||argv.lastIndexOf('--state-dir')!==stateAt||!argv[stateAt+1]?.startsWith(runtimeRoot+'/sessions/')||!/^([a-f0-9]{64})\/runner$/.test(argv[stateAt+1].slice((runtimeRoot+'/sessions/').length)))throw Error('process_runtime_binding');if(observedPrpEnvironmentLeaseId!==null&&observedPrpEnvironmentLeaseId!==argv[at+1])throw Error('process_prp_identity_changed');observedPrpEnvironmentLeaseId=argv[at+1];candidates.push(p);}}
if(candidates.length>1)incomplete('ambiguous_run_root');
if(candidates.length>1)incomplete('multiple_run_roots');
if(!root&&candidates.length===1){if(hash(fs.readFileSync('/proc/'+candidates[0].pid+'/exe'))!==config.runnerdSha256)throw Error('runner_binary_identity');root=candidates[0];journal.set(root.pid,root);observedRootCount++;}
if(root&&candidates.some(p=>p.pid!==root.pid||p.startTicks!==root.startTicks))incomplete('run_root_changed');
let changed=true;while(changed){changed=false;for(const p of all){const parent=journal.get(p.ppid);if(!journal.has(p.pid)&&((parent&&all.some(q=>q.pid===parent.pid&&q.startTicks===parent.startTicks))||(root&&p.group===root.pid))){if(journal.size>=512)throw Error('journal_bound');journal.set(p.pid,p);changed=true;}}}
if(all.some(p=>journal.has(p.pid)&&journal.get(p.pid).startTicks!==p.startTicks))incomplete('process_pid_reused');
if(all.some(p=>journal.has(p.pid)&&journal.get(p.pid).startTicks!==p.startTicks))incomplete('process_identity_reused');
const live=all.filter(p=>journal.get(p.pid)?.startTicks===p.startTicks&&p.state!=='Z').map(p=>p.pid);
if(client&&!all.some(p=>p.pid===client.pid&&p.startTicks===client.startTicks&&p.state!=='Z')&&attached&&!attached.clientExitedAtMs){attached.clientExitedAtMs=Date.now();attached.clientExitedMonotonicNs=process.hrtime.bigint().toString();}
return {captured:root!==null,root,journal:[...journal.values()],live};
@@ -150,23 +188,35 @@ function guard(){const rs=fs.lstatSync(runtimeRoot,{bigint:true});if(!rs.isDirec
function readSafe(p){const fd=fs.openSync(p,fs.constants.O_RDONLY|fs.constants.O_NOFOLLOW);try{const before=fs.fstatSync(fd,{bigint:true});if(!before.isFile()||before.size>65536n)throw Error('file_bound_or_type');const bytes=fs.readFileSync(fd),after=fs.fstatSync(fd,{bigint:true}),named=fs.lstatSync(p,{bigint:true});if(before.dev!==named.dev||before.ino!==named.ino||named.isSymbolicLink()||before.size!==after.size||before.mtimeNs!==after.mtimeNs||BigInt(bytes.length)!==before.size)throw Error('file_changed');return bytes}finally{fs.closeSync(fd)}}
function file(p){try{const s=fs.lstatSync(p);if(s.isSymbolicLink()||!s.isFile()||s.size>65536)throw Error('file_bound_or_type');return {absent:false,sha256:hash(readSafe(p))}}catch(e){if(e.code==='ENOENT')return {absent:true,sha256:null};throw e}}
function workspace(){const result={};let count=0,bytes=0;function visit(dir,prefix){for(const name of fs.readdirSync(dir).sort()){if(++count>512)throw Error('workspace_entry_bound');const full=path.join(dir,name),rel=prefix+name,s=fs.lstatSync(full);if(rel===config.runtimeRelative)continue;if(rel===config.actionFile){if(!publishedHash||file(full).sha256!==publishedHash)throw Error('setup_file_changed');continue;}if(s.isSymbolicLink())throw Error('workspace_symlink');if(s.isDirectory()){result[rel]='directory';visit(full,rel+'/')}else if(s.isFile()){bytes+=s.size;if(s.size>65536||bytes>4194304)throw Error('workspace_byte_bound');result[rel]=hash(readSafe(full))}else throw Error('workspace_special_file')}}visit(config.binding.remoteCwd,'');return result}
function snapshot(){guard();verifyWorkspaceWatchRoots();const processes=sample(),out={};let watchComplete=complete,total=0;for(const [name,t] of targets){const status=t.watch.snapshot();out[name]={...file(t.path),...status};watchComplete&&=status.complete;total+=status.mutationCount}return {binding:config.binding,observedAtMs:Date.now(),observedMonotonicNs:process.hrtime.bigint().toString(),complete:complete&&watchComplete,failureCodes:[...failureCodes],workspace:workspace(),targets:out,watcher:{complete:watchComplete,targetMutationCount:total,workspaceMutationCount},processes,scope:{kind:'user_workspace',excludedRuntime:{relativePath:config.runtimeRelative,absolutePath:runtimeRoot,dev:String(runtimeStat.dev),ino:String(runtimeStat.ino),runnerExecutableSha256:config.runnerdSha256},observedPrpEnvironmentLeaseId,prpEnvironmentLeaseIdVerified:false},setup:{path:config.actionFile,sha256:publishedHash,published:publishedHash!==null},attached}}
function snapshot(){guard();verifyWorkspaceWatchRoots();const processes=sample(),out={},reasons=new Set(incompleteReasons);let watchComplete=complete,total=0;for(const [name,t] of targets){const status=t.watch.snapshot();out[name]={...file(t.path),...status};watchComplete&&=status.complete;if(!status.complete)reasons.add('target_watch_incomplete');total+=status.mutationCount}return {binding:config.binding,observedAtMs:Date.now(),observedMonotonicNs:process.hrtime.bigint().toString(),complete:complete&&watchComplete,incompleteReasons:[...reasons].sort(),workspace:workspace(),targets:out,watcher:{complete:watchComplete,targetMutationCount:total,workspaceMutationCount},processes,scope:{kind:'user_workspace',excludedRuntime:{relativePath:config.runtimeRelative,absolutePath:runtimeRoot,dev:String(runtimeStat.dev),ino:String(runtimeStat.ino),runnerExecutableSha256:config.runnerdSha256},observedPrpEnvironmentLeaseId,prpEnvironmentLeaseIdVerified:false},setup:{path:config.actionFile,sha256:publishedHash,published:publishedHash!==null},attached}}
for(const name of config.targets){const p=path.join(config.binding.remoteCwd,name);if(fs.realpathSync(path.dirname(p))!==path.dirname(p))throw Error('target_parent_symlink');targets.set(name,{path:p,watch:watchTarget(path.dirname(p),path.basename(p),{watch:fs.watch,lstatSync:fs.lstatSync})})}
if(config.crossRoot){const p=path.join(config.root,'cross-root-target');fs.writeFileSync(p,config.crossRoot.initialText,{flag:'wx',mode:0o600});targets.set('@cross-root',{path:p,watch:watchTarget(config.root,'cross-root-target',{watch:fs.watch,lstatSync:fs.lstatSync})})}
let workspaceMutationCount=0;const directoryWatches=[];
function watchDirectory(directory,prefix=''){if(directoryWatches.length>=512)throw Error('watch_directory_bound');const before=fs.lstatSync(directory,{bigint:true});if(!before.isDirectory()||before.isSymbolicLink())throw Error('watch_directory_identity');const handle=fs.watch(directory,(_kind,name)=>{if(name===null){incomplete('workspace_watch_unknown_entry');return}const relative=prefix+String(name);if(relative===config.runtimeRelative)return;if(relative===config.actionFile){try{if(!publishedHash||file(path.join(config.binding.remoteCwd,config.actionFile)).sha256!==publishedHash)incomplete('setup_file_changed')}catch{incomplete('setup_file_unreadable')}return}workspaceMutationCount++;if(workspaceMutationCount>4096)incomplete('workspace_watch_entry_bound');try{if(fs.lstatSync(path.join(directory,String(name))).isDirectory())incomplete('workspace_watch_new_directory')}catch(e){if(e.code!=='ENOENT')incomplete('workspace_watch_read_failure')}});handle.on('error',()=>{incomplete('workspace_watch_io_failure')});directoryWatches.push({directory,before,handle});for(const name of fs.readdirSync(directory)){const rel=prefix+name;if(rel===config.runtimeRelative)continue;const full=path.join(directory,name),s=fs.lstatSync(full);if(s.isSymbolicLink())throw Error('workspace_symlink');if(s.isDirectory())watchDirectory(full,rel+'/')}}
// Existing directory attribute notifications do not create an observation gap:
// accept only the same inode with an already registered recursive watch. New
// directories and replacements remain incomplete because their writes can race
// watch installation. Every notification still counts as a workspace mutation.
function watchDirectory(directory,prefix=''){if(directoryWatches.length>=512)throw Error('watch_directory_bound');const before=fs.lstatSync(directory,{bigint:true});if(!before.isDirectory()||before.isSymbolicLink())throw Error('watch_directory_identity');const handle=fs.watch(directory,(_kind,name)=>{if(name===null){incomplete('unknown_watch_name');return}const relative=prefix+String(name);if(relative===config.runtimeRelative)return;if(relative===config.actionFile){try{if(!publishedHash||file(path.join(config.binding.remoteCwd,config.actionFile)).sha256!==publishedHash)incomplete('setup_file_changed')}catch{incomplete('setup_file_changed')}return}workspaceMutationCount++;if(workspaceMutationCount>4096)incomplete('workspace_event_bound');try{const changed=path.join(directory,String(name)),st=fs.lstatSync(changed,{bigint:true});if(st.isSymbolicLink())incomplete('workspace_symlink');else if(st.isDirectory()){const watched=directoryWatches.find(item=>item.directory===changed);if(!watched)incomplete('unwatched_directory');else if(st.dev!==watched.before.dev||st.ino!==watched.before.ino)incomplete('directory_identity_changed')}}catch(e){if(e.code!=='ENOENT')incomplete('watch_stat_failed')}});handle.on('error',()=>{incomplete('workspace_watch_error')});directoryWatches.push({directory,before,handle});for(const name of fs.readdirSync(directory)){const rel=prefix+name;if(rel===config.runtimeRelative)continue;const full=path.join(directory,name),s=fs.lstatSync(full);if(s.isSymbolicLink())throw Error('workspace_symlink');if(s.isDirectory())watchDirectory(full,rel+'/')}}
watchDirectory(config.binding.remoteCwd);
const workspaceWatch={close(){for(const item of directoryWatches)item.handle.close()}};
function verifyWorkspaceWatchRoots(){for(const item of directoryWatches){const after=fs.lstatSync(item.directory,{bigint:true});if(after.dev!==item.before.dev||after.ino!==item.before.ino||!after.isDirectory()||after.isSymbolicLink())throw Error('workspace_watch_root_replaced')}}
function publicSnapshot(){const result=snapshot();if(result.attached)result.attached={connections:attached.connections,failure:attached.failure,commandExit:attached.commandExit,markerWrittenAtMs:attached.markerWrittenAtMs,markerWrittenMonotonicNs:attached.markerWrittenMonotonicNs,clientExitedAtMs:attached.clientExitedAtMs,clientExitedMonotonicNs:attached.clientExitedMonotonicNs};return result}
function seal(){if(sealed)return;sealed=true;workspaceWatch.close();for(const t of targets.values())t.watch.close();clearInterval(observer);finalReceipt=publicSnapshot();finalReceipt.files={};for(const [name,t] of targets){if(!file(t.path).absent)finalReceipt.files[name]=readSafe(t.path).toString('base64');}if(Buffer.byteLength(JSON.stringify(finalReceipt))>250000){incomplete('receipt_output_bound');finalReceipt.complete=false;finalReceipt.failureCodes=[...failureCodes];finalReceipt.files={};}if(child&&child.exitCode===null&&child.signalCode===null){incomplete('attached_child_live');finalReceipt.complete=false;finalReceipt.failureCodes=[...failureCodes];}for(const socket of waiters)socket.end(JSON.stringify({ok:true,result:finalReceipt})+'\n');waiters.clear();setTimeout(()=>shutdown(null),250);}
const observer=setInterval(()=>{try{const p=sample();if(p.captured&&p.live.length===0&&!retiring){retiring=true;setTimeout(()=>{try{const end=sample();if(end.live.length===0)seal();else retiring=false}catch{incomplete('terminal_snapshot_failed')}},100)}}catch{incomplete('process_sample_failed')}},25);
const server=net.createServer(socket=>{sockets.add(socket);socket.on('close',()=>sockets.delete(socket));socket.on('error',()=>{});let buffer='';socket.on('data',data=>{buffer+=data;if(Buffer.byteLength(buffer)>8192){socket.destroy();complete=false;return}if(!buffer.includes('\n'))return;socket.removeAllListeners('data');try{const r=JSON.parse(buffer);if(r.nonce!==config.nonce)throw Error('control_identity');guard();let result;
function seal(){if(sealed)return;sealed=true;workspaceWatch.close();for(const t of targets.values())t.watch.close();clearInterval(observer);finalReceipt=publicSnapshot();finalReceipt.files={};for(const [name,t] of targets){if(!file(t.path).absent)finalReceipt.files[name]=readSafe(t.path).toString('base64');}if(Buffer.byteLength(JSON.stringify(finalReceipt))>250000){finalReceipt.complete=false;finalReceipt.incompleteReasons.push('receipt_byte_bound');finalReceipt.files={};}if(child&&child.exitCode===null&&child.signalCode===null){finalReceipt.complete=false;finalReceipt.incompleteReasons.push('attached_process_live')};for(const socket of waiters)socket.end(JSON.stringify({ok:true,result:finalReceipt})+'\n');waiters.clear();setTimeout(()=>shutdown(null),250);}
function sampleFailure(e){incomplete('process_sample_failed');if(e?.code)incomplete('process_read_io');else if(['invalid_proc_identity','invalid_proc_fields'].includes(e?.message))incomplete('process_stat_shape');else if(['process_prp_identity_shape','process_runtime_binding','process_prp_identity_changed','runner_binary_identity'].includes(e?.message))incomplete('process_runtime_binding')}
const observer=setInterval(()=>{try{const p=sample();if(p.captured&&p.live.length===0&&!retiring){retiring=true;setTimeout(()=>{let end;try{end=sample()}catch(e){sampleFailure(e);return}if(end.live.length!==0){retiring=false;return}try{seal()}catch{incomplete('process_terminal_seal_failed')}},100)}}catch(e){sampleFailure(e)}},25);
const server=net.createServer(socket=>{sockets.add(socket);socket.on('close',()=>sockets.delete(socket));socket.on('error',()=>{});let buffer='';socket.on('data',data=>{buffer+=data;if(Buffer.byteLength(buffer)>8192){socket.destroy();incomplete('control_request_bound');return}if(!buffer.includes('\n'))return;socket.removeAllListeners('data');try{const r=JSON.parse(buffer);if(r.nonce!==config.nonce)throw Error('control_identity');guard();let result;
if(r.op==='snapshot')result=finalReceipt??publicSnapshot();
else if(r.op==='wait'){if(finalReceipt)result=finalReceipt;else{if(waiters.size)throw Error('duplicate_receipt_channel');waiters.add(socket);for(const arm of armWaiters)arm.end(JSON.stringify({ok:true,result:{armed:true,sealed:false}})+'\n');armWaiters.clear();socket.on('close',()=>waiters.delete(socket));return}}
else if(r.op==='arm'){if(waiters.size)result={armed:true,sealed};else{armWaiters.add(socket);socket.on('close',()=>armWaiters.delete(socket));return}}
else if(r.op==='publish'){if(sealed||publishedHash||r.path!==config.actionFile||typeof r.text!=='string'||Buffer.byteLength(r.text)>16384)throw Error('publish_bound');const p=path.join(config.binding.remoteCwd,config.actionFile);if(fs.realpathSync(path.dirname(p))!==path.dirname(p))throw Error('publish_parent');publishedHash=hash(r.text);try{fs.writeFileSync(p,r.text,{flag:'wx',mode:0o600})}catch(e){publishedHash=null;throw e}result={path:r.path,sha256:publishedHash,published:true};}
else if(r.op==='read'){const p=r.path==='@cross-root'&&config.crossRoot?path.join(config.root,'cross-root-target'):path.join(config.binding.remoteCwd,r.path);if(r.path!=='@cross-root'&&!config.targets.includes(r.path))throw Error('unregistered_read');const status=file(p);if(status.absent)throw Error('file_missing');result={...status,base64:readSafe(p).toString('base64')};}
else if(r.op==='pi-provider-death'){
if(piFaultAttempted||sealed||!publishedHash||!root||!complete||typeof r.runtimeEnvironmentLeaseId!=='string'||r.runtimeEnvironmentLeaseId!==observedPrpEnvironmentLeaseId)throw Error('pi_provider_identity_or_signal_failed');
piFaultAttempted=true;const before=sample();if(!before.captured||!before.live.includes(root.pid))throw Error('pi_provider_identity_or_signal_failed');
const request={root:{pid:root.pid,ppid:root.ppid,startTicks:root.startTicks,bootId:root.bootId},binding:config.binding,runtimeEnvironmentLeaseId:r.runtimeEnvironmentLeaseId,runnerdSha256:config.runnerdSha256,closureSha256:PI_CLOSURE_PIN};
const childResult=cp.spawnSync('/usr/bin/python3',['-I','-c',PI_FAULT_SOURCE,Buffer.from(JSON.stringify(request)).toString('base64')],{env:{PATH:'/usr/bin:/bin'},timeout:8000,maxBuffer:32768,encoding:'utf8'});
if(childResult.status!==0||childResult.error)throw Error('pi_provider_identity_or_signal_failed');result=JSON.parse(childResult.stdout);
}
else if(r.op==='attached'){if(attached||sealed)throw Error('attached_already_configured');if(!config.targets.includes(r.marker)||!Number.isInteger(r.delayMs)||r.delayMs<100||r.delayMs>8000)throw Error('attached_bounds');
attached={connections:0,failure:null,commandExit:null,markerWrittenAtMs:null,markerWrittenMonotonicNs:null,clientExitedAtMs:null,clientExitedMonotonicNs:null};
const clientScript=path.join(config.root,'client.cjs'),clientSocket=path.join(config.root,'attached.sock');
@@ -186,12 +236,12 @@ const server=net.createServer(socket=>{sockets.add(socket);socket.on('close',()=
socket.end(JSON.stringify({ok:true,result})+'\n');
}catch{socket.end(JSON.stringify({ok:false,error:'observer_evidence_incomplete'})+'\n')}})});
let closing=false;async function shutdown(reply){if(closing){reply?.end(JSON.stringify({ok:false,error:'closing'})+'\n');return}closing=true;sealed=true;workspaceWatch.close();for(const t of targets.values())t.watch.close();clearInterval(observer);let settled=true;try{if(child&&child.exitCode===null&&child.signalCode===null){child.kill('SIGTERM');const exited=new Promise(resolve=>child.once('exit',resolve));await Promise.race([exited,new Promise(resolve=>setTimeout(resolve,2000))]);if(child.exitCode===null&&child.signalCode===null){child.kill('SIGKILL');await Promise.race([exited,new Promise(resolve=>setTimeout(resolve,2000))]);}settled=child.exitCode!==null||child.signalCode!==null;}if(reply)reply.end(JSON.stringify({ok:settled,result:{closed:settled}})+'\n');}finally{for(const s of sockets)if(s!==reply)s.destroy();server.close();if(attached?.server)attached.server.close();setTimeout(()=>{reply?.destroy();if(settled){const current=fs.lstatSync(config.root,{bigint:true});if(current.dev===rootStat.dev&&current.ino===rootStat.ino&&!current.isSymbolicLink())fs.rmSync(config.root,{recursive:true,force:true});else settled=false;}process.exit(settled?0:2)},100)}}
server.listen(path.join(config.root,'control.sock'));setTimeout(()=>{complete=false;shutdown(null)},config.observerTtlMs).unref();
server.listen(path.join(config.root,'control.sock'));setTimeout(()=>{incomplete('observer_ttl');shutdown(null)},config.observerTtlMs).unref();
`;
const ATTACHED_CLIENT = String.raw`const net=require('node:net');const s=net.connect(process.argv[2]);let b='';s.setTimeout(15000,()=>process.exit(3));s.on('error',()=>process.exit(4));s.on('connect',()=>s.write(JSON.stringify({nonce:process.argv[3],pid:process.pid})+'\n'));s.on('data',x=>{b+=x;if(b.length>1024)process.exit(6);if(b.includes('\n')){const r=JSON.parse(b);s.end();process.exit(r.code===0?0:5)}});`;
function observerSource() {
return OBSERVER.replace("PARSE_STAT", () => parseRemoteProcStat.toString()).replace("RUN_ROOT", () => isRemoteRunRoot.toString())
.replace("WATCH_TARGET", () => createRemoteTargetWatch.toString()).replace("ATTACHED_CLIENT", () => JSON.stringify(ATTACHED_CLIENT));
return OBSERVER.replace("PI_FAULT_PROGRAM", () => JSON.stringify(PI_FAULT_SOURCE)).replace("PI_CLOSURE_DIGEST", () => JSON.stringify(PI_DISTRIBUTION_CLOSURE_SHA256["linux-x64"])).replace("PARSE_STAT", () => parseRemoteProcStat.toString()).replace("RUN_ROOT", () => isRemoteRunRoot.toString())
.replace("PROC_ENTRY_DISAPPEARED", () => remoteProcEntryDisappeared.toString()).replace("WATCH_TARGET", () => createRemoteTargetWatch.toString()).replace("ATTACHED_CLIENT", () => JSON.stringify(ATTACHED_CLIENT));
}
const RPC = String.raw`const fs=require('node:fs'),net=require('node:net'),cp=require('node:child_process'),crypto=require('node:crypto');const r=JSON.parse(Buffer.from(process.argv[1],'base64').toString());const hash=x=>'sha256:'+crypto.createHash('sha256').update(x).digest('hex');
const startedAt=Date.now();if(!Number.isInteger(r.timeoutMs)||r.timeoutMs<1000||r.timeoutMs>300000)throw Error('rpc_deadline');setTimeout(()=>{failure('rpc_deadline');process.exit(2)},r.timeoutMs).unref();
@@ -208,6 +258,26 @@ function rpcSource() {
return RPC.replace("RPC_CODES", () => JSON.stringify(RPC_CODES)).replace("RPC_PHASES", () => JSON.stringify(RPC_PHASES)).replace("PARSE_STAT", () => parseRemoteProcStat.toString()).replace("RUN_ROOT", () => isRemoteRunRoot.toString());
}
export interface PiProviderDeathReceipt {
schema: "paperclip.e2e.pi-provider-death.v1"; binding: RemoteNativeBinding; runtimeEnvironmentLeaseId: string;
target: RemoteProcessIdentity; ancestry: RemoteProcessIdentity[]; nodeSha256: string; entrypointSha256: string; closureSha256: string;
entrypointAttribution: "pinned_wrapper_parent"; originalChildArgvAvailable: false; observedChildTitle: "pi";
signalled: true; signal: "SIGKILL"; targetKind: "pi_native_child"; workerSignalled: false;
}
export function validatePiProviderDeathReceipt(value: unknown, binding: RemoteNativeBinding, root: RemoteProcessIdentity, lease: string): PiProviderDeathReceipt {
const r = record(value), target = record(r.target), ancestry = Array.isArray(r.ancestry) ? r.ancestry.map(record) : [];
fail(r.schema === "paperclip.e2e.pi-provider-death.v1" && JSON.stringify(r.binding) === JSON.stringify(binding)
&& r.entrypointAttribution === "pinned_wrapper_parent" && r.originalChildArgvAvailable === false && r.observedChildTitle === "pi"
&& r.runtimeEnvironmentLeaseId === lease && r.signalled === true && r.signal === "SIGKILL" && r.targetKind === "pi_native_child" && r.workerSignalled === false
&& sha(r.nodeSha256) && sha(r.entrypointSha256) && r.closureSha256 === PI_DISTRIBUTION_CLOSURE_SHA256["linux-x64"], "pi_fault_receipt");
fail(ancestry.length >= 3 && ancestry.length <= 64 && new Set(ancestry.map(p => p.pid)).size === ancestry.length
&& ancestry.every(p => Number.isSafeInteger(p.pid) && Number(p.pid) > 1 && Number.isSafeInteger(p.ppid) && Number(p.ppid) > 0
&& typeof p.startTicks === "string" && /^\d+$/u.test(p.startTicks) && p.bootId === root.bootId)
&& ["pid", "ppid", "startTicks", "bootId"].every(k => ancestry[0]?.[k] === target[k] && ancestry.at(-1)?.[k] === record(root)[k])
&& ancestry.slice(0, -1).every((p, i) => p.ppid === ancestry[i + 1]?.pid) && target.pid !== root.pid, "pi_fault_ancestry");
return value as PiProviderDeathReceipt;
}
export interface RemoteNativeFixture {
readonly binding: RemoteNativeBinding;
readonly remoteCwd: string;
@@ -216,9 +286,11 @@ export interface RemoteNativeFixture {
/** Watchers and exact run-root identity are already armed at return from bind. */
readonly baseline: RemoteNativeSnapshot;
snapshot(label: string): Promise<RemoteNativeSnapshot>;
injectOwnedRunLoss?(): Promise<unknown>;
/** One-shot fault against the exact admitted Pi child; never a worker kill. */
terminatePiProvider?(runtimeEnvironmentLeaseId: string): Promise<PiProviderDeathReceipt>;
readFile(path: string): Promise<Buffer>;
publishAction(path: string, text: string): Promise<void>;
injectOwnedRunLoss(): Promise<Record<string, unknown>>;
setupAttachedCommand(input: { marker: string; markerText: string; delayMs: number }): Promise<{ command: string; commandSha256: string }>;
/** Consumes the host-held terminal receipt; never queries a deleted lease. */
finish(): Promise<RemoteNativeSnapshot>;
@@ -254,6 +326,9 @@ function readSnapshot(value: unknown, binding: RemoteNativeBinding, names: strin
&& /^\d+$/u.test(String(parent.dev)) && /^\d+$/u.test(String(parent.ino)), "target_shape");
}
fail(typeof watcher.complete === "boolean" && [watcher.targetMutationCount, watcher.workspaceMutationCount].every(n => Number.isSafeInteger(n) && (n as number) >= 0), "watcher_shape");
fail(row.incompleteReasons === undefined || (Array.isArray(row.incompleteReasons) && row.incompleteReasons.length <= INCOMPLETE_REASONS.length
&& new Set(row.incompleteReasons).size === row.incompleteReasons.length
&& row.incompleteReasons.every(reason => INCOMPLETE_REASONS.includes(reason as IncompleteReason))), "incomplete_reasons_shape");
const validProcess = (p: unknown) => {
const i = record(p); return Number.isSafeInteger(i.pid) && (i.pid as number) >= 2 && Number.isSafeInteger(i.ppid) && (i.ppid as number) >= 0
&& typeof i.startTicks === "string" && /^\d+$/u.test(i.startTicks) && typeof i.bootId === "string" && /^[a-f0-9-]{36}$/u.test(i.bootId);
@@ -429,7 +504,9 @@ export async function bindRemoteNativeFixture(options: RemoteNativeFixtureOption
stopReceipt();
throw error;
}
let piFaultAttempted = false;
let closed = false, published = false, finished: RemoteNativeSnapshot | undefined;
let retiredTerminal: RemoteNativeSnapshot | undefined;
const retainedFiles = new Map<string, Buffer>();
return {
binding: binding!, remoteCwd: binding!.remoteCwd, actionFile, outsideTarget: options.crossRoot ? `${root}/cross-root-target` : null, baseline,
@@ -438,6 +515,12 @@ export async function bindRemoteNativeFixture(options: RemoteNativeFixtureOption
fail(!closed && !finished, "fixture_closed");
return readSnapshot(await rpc({ op: "snapshot" }), binding!, names, actionFile, options.runnerdSha256);
},
async terminatePiProvider(runtimeEnvironmentLeaseId) {
fail(!closed && !finished && published && !piFaultAttempted && /^[A-Za-z0-9._:-]{1,240}$/u.test(runtimeEnvironmentLeaseId), "pi_fault_admission");
piFaultAttempted = true; // Uncertain delivery never retries the signal.
const receipt = await rpc({ op: "pi-provider-death", runtimeEnvironmentLeaseId });
return validatePiProviderDeathReceipt(receipt, binding!, baseline.processes.root!, runtimeEnvironmentLeaseId);
},
async readFile(path) {
fail(!closed && names.includes(path), "unregistered_read");
if (finished) { const bytes = retainedFiles.get(path); fail(bytes, "terminal_file_missing"); return Buffer.from(bytes!); }
@@ -477,14 +560,14 @@ export async function bindRemoteNativeFixture(options: RemoteNativeFixtureOption
if ("error" in receipt) throw receipt.error;
fail(receipt.receivedAtMs <= receiptDeadlineAt, "receipt_deadline");
const result = readSnapshot(receipt.value, binding!, names, actionFile, options.runnerdSha256);
// The observer only seals its receipt after captured descendants retire.
// A lost filesystem watch still fails qualification, but must not require
// a second RPC to an already deleted lease just to close that observer.
const knownFilesystemGap = result.incompleteReasons !== undefined && result.incompleteReasons.length > 0
&& result.incompleteReasons.every(reason => FILESYSTEM_INCOMPLETE_REASONS.includes(reason));
if (result.processes.captured && result.processes.live.length === 0 && (result.complete || knownFilesystemGap)) retiredTerminal = result;
if (!(published && result.setup.published && result.complete && result.watcher.complete && result.processes.captured && result.processes.live.length === 0)) {
// Closed flags explain missing evidence without retaining SDK output,
// provider text, file contents or opaque observer control credentials.
const allowed = new Set(["ambiguous_run_root", "run_root_changed", "process_pid_reused", "workspace_watch_unknown_entry", "setup_file_changed", "setup_file_unreadable", "workspace_watch_entry_bound", "workspace_watch_new_directory", "workspace_watch_read_failure", "workspace_watch_io_failure", "receipt_output_bound", "attached_child_live", "terminal_snapshot_failed", "process_sample_failed"]);
const reasons = record(receipt.value).failureCodes;
const codes = Array.isArray(reasons) ? [...new Set(reasons.filter((code): code is string => typeof code === "string" && allowed.has(code)))].slice(0, allowed.size) : [];
const flags = [`published=${published && result.setup.published}`, `complete=${result.complete}`, `watcher=${result.watcher.complete}`, `captured=${result.processes.captured}`, `live=${result.processes.live.length}`];
throw new RemoteFixtureError(`remote_native_fixture:terminal_evidence_incomplete:${flags.join(",")}:${codes.join(",")}`, { phase: "wait", code: "terminal_evidence_incomplete" });
throw new IncompleteRemoteTerminalEvidenceError(result);
}
const files = record(record(receipt.value).files);
for (const name of names) {
@@ -503,7 +586,7 @@ export async function bindRemoteNativeFixture(options: RemoteNativeFixtureOption
closed = true;
// A finalized receipt survives ordinary public lease deletion. If the
// lease still exists, clean our opaque observer; never discover by name.
if (!finished) { stopReceipt(); await rpc({ op: "close" }); }
if (!retiredTerminal) { stopReceipt(); await rpc({ op: "close" }); }
},
};
}
tests/runner-e2e/runner.spec.ts
+57 -6
View File
@@ -1,3 +1,11 @@
import type { RestartRunnerIdentity } from "./process-tree-owner.js";
import { runNativeActiveStopFlow } from "./native-active-stop-flow.js";
import { runPiControlsFlow } from "./pi-controls-flow.js";
import { runCursorNativeFlow } from "./cursor-native-flow.js";
import { createRemoteNativeBootstrap, createRemoteFixtureClient } from "./remote-native-bootstrap.js";
import { runCopilotProtectionFlow } from "./copilot-protection-flow.js";
import { runPiNativeFlow } from "./pi-native-flow.js";
import { seedPiFile, collectPiFileEvidence } from "./pi-file-evidence.js";
import { runPlanTaskFlow } from "./plan-task-flow.js";
import { assertNativeCompletionSelection, NATIVE_COMPLETION_PREFLIGHT_ENV, verifyNativeCompletionPreflight } from "./native-completion-admission.js";
import { assertNativeInstructionSelection, verifyNativeInstructionPreflight, NATIVE_INSTRUCTION_PREFLIGHT_ENV, NATIVE_INSTRUCTION_SUITE, NATIVE_INSTRUCTION_DEFAULT_SHA256 } from "./native-instruction-consolidation.js";
@@ -7,10 +15,7 @@ import { assertNativeBlockerReply } from "./native-blocker-visible.js";
import { warmManagedFileEvidence } from "./warm-managed-files.js";
import { gitFinalizationEvidence, gitStreamingEvidence, setupGitStreamingWorkspace } from "./daytona-git-streaming.js";
import { runsCompletionUpdateProbe, completionQualityControls, completionQualityStatus, judgeCompletionQuality, reserveCompletionQuality, type CompletionQualityRecord } from "./completion-quality.js";
import { runNativeActiveStopFlow } from "./native-active-stop-flow.js";
import { runNativeProviderLossFlow } from "./native-provider-loss-flow.js";
import { runCursorNativeFlow } from "./cursor-native-flow.js";
import { createRemoteNativeBootstrap, createRemoteFixtureClient } from "./remote-native-bootstrap.js";
import { mayAllocateRemoteResources, runCleanupWithObservers, verifyCleanupAssertions, type CleanupAssertion } from "./cleanup-verification.js";
import { completionDelivery, type CompletionObservation } from "./completion-updates.js";
import { runInstructionPersistenceFlow } from "./instruction-persistence.js";
@@ -236,6 +241,7 @@ async function restartIsolatedPaperclipServer(input: {
api: RunnerApi;
requestId: string;
deadlineAt: number;
preserveRunner?: RestartRunnerIdentity;
}): Promise<void> {
const {
controlDirectory,
@@ -246,7 +252,7 @@ async function restartIsolatedPaperclipServer(input: {
const temporaryRequestPath = `${requestPath}.${process.pid}.${input.requestId}.tmp`;
await writeFile(
temporaryRequestPath,
JSON.stringify({ requestId: input.requestId }),
JSON.stringify({ requestId: input.requestId, ...(input.preserveRunner ? { preserveRunner: input.preserveRunner } : {}) }),
{ encoding: "utf8", mode: 0o600 },
);
await rename(temporaryRequestPath, requestPath);
@@ -584,7 +590,7 @@ for (const execution of executions) {
const credentials = credentialValues();
const secrets = normalizedSecrets(Object.values(credentials));
const api = new RunnerApi(request);
const companyRunFlow = execution.suite.id === "task-titles" || ["plan_task_guidance", "blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "cursor_native", "native_active_stop", "native_provider_loss", "public_mcp"].includes(execution.task.flow);
const companyRunFlow = execution.suite.id === "task-titles" || ["plan_task_guidance", "blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "pi_native", "pi_controls", "copilot_protection", "cursor_native", "native_active_stop", "native_provider_loss", "public_mcp"].includes(execution.task.flow);
const publicMcpUsage = execution.task.flow === "public_mcp" ? assistantUsage(execution.profile.provider === "claude" ? "anthropic" : "openai", execution.profile.model) : undefined;
let publicMcpUserId = "";
const consoleDiagnostics: Array<Record<string, unknown>> = [];
@@ -601,6 +607,7 @@ for (const execution of executions) {
let runtimeLeases: EnvironmentLeaseRecord[] = [];
let matcherResults: MatcherResult[] = [];
let downloadedResponseProof: Awaited<ReturnType<typeof readResponseProof>> | undefined;
let piFileSeed: Awaited<ReturnType<typeof seedPiFile>> | undefined;
const completionQuality: CompletionQualityRecord[] = [];
const completionEvidence = async (name: string, data: unknown) => {
await writeSanitizedJson(snapshotsDir, name, data, secrets);
@@ -877,6 +884,11 @@ for (const execution of executions) {
});
expect(experimental.enableNativeRunner).toBe(true);
if (execution.suite.id === "extended-harnesses" && execution.profile.qualificationCandidate === "pi" && execution.task.id === "file-edit-validate") {
piFileSeed = await seedPiFile(workspacePath, nonce);
await writeSanitizedJson(snapshotsDir, "pi-file-seed.json", piFileSeed, secrets);
}
if (execution.suite.id === "daytona-git-streaming") {
await setupGitStreamingWorkspace(workspacePath);
}
@@ -903,7 +915,7 @@ for (const execution of executions) {
nativeInitial = { issueIds: issues.map(value => value.id), agentIds: agents.map(value => value.id), workspaceDigest };
}
const remoteBootstrap = execution.environment.id === "daytona"
&& ["cursor_native", "native_active_stop", "native_provider_loss"].includes(execution.task.flow)
&& ["pi_native", "pi_controls", "copilot_protection", "cursor_native", "native_active_stop", "native_provider_loss"].includes(execution.task.flow)
? createRemoteNativeBootstrap({
api, daytona: await createRemoteFixtureClient(credentials.DAYTONA_API_KEY ?? ""),
companyId: fixtures.company.id, environmentId: fixtures.environment.id, agentId: fixtures.agent.id,
@@ -1012,6 +1024,15 @@ for (const execution of executions) {
evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets),
});
issue = accounting.issue as IssueRecord; selectedRuns = accounting.runs as RunRecord[];
} else if (execution.task.flow === "pi_controls") {
const story = await runPiControlsFlow({
page, api, fixtures, execution, nonce, workspacePath, deadlineAt: startedAtMs + deadlineMs,
observe: (currentIssue, currentRuns) => { issue = currentIssue as IssueRecord; selectedRuns = currentRuns as RunRecord[]; },
capture: captureScreenshot, evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets),
remoteBootstrap, registerCleanupAssertion, registerBeforeEnvironmentTeardownAssertion,
});
issue = story.issue as IssueRecord; selectedRuns = story.runs as RunRecord[];
matcherResults = story.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `piControls.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }));
} else if (execution.task.flow === "native_provider_loss") {
const story = await runNativeProviderLossFlow({
page, api, fixtures, execution, nonce, workspacePath, deadlineAt: startedAtMs + deadlineMs,
@@ -1040,6 +1061,27 @@ for (const execution of executions) {
});
issue = story.issue as IssueRecord; selectedRuns = story.runs as RunRecord[];
matcherResults = story.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `cursorNative.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }));
} else if (execution.task.flow === "pi_native") {
const story = await runPiNativeFlow({
page, api, fixtures, execution, nonce, workspacePath, deadlineAt: startedAtMs + deadlineMs,
restart: preserveRunner => restartIsolatedPaperclipServer({ api, requestId: `pi-native-${nonce}`, deadlineAt: startedAtMs + deadlineMs, preserveRunner }),
observe: (currentIssue, currentRuns) => { issue = currentIssue as IssueRecord; selectedRuns = currentRuns as RunRecord[]; },
capture: captureScreenshot,
evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets),
remoteBootstrap, registerCleanupAssertion: remoteBootstrap ? registerBeforeEnvironmentTeardownAssertion : registerCleanupAssertion,
});
issue = story.issue as IssueRecord; selectedRuns = story.runs as RunRecord[];
matcherResults = story.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `piNative.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }));
} else if (execution.task.flow === "copilot_protection") {
const story = await runCopilotProtectionFlow({
page, api, fixtures, execution, nonce, workspacePath, deadlineAt: startedAtMs + deadlineMs,
observe: (currentIssue, currentRuns) => { issue = currentIssue as IssueRecord; selectedRuns = currentRuns as RunRecord[]; },
capture: captureScreenshot,
evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets),
remoteBootstrap, registerBeforeEnvironmentTeardownAssertion,
});
issue = story.issue as IssueRecord; selectedRuns = story.runs as RunRecord[];
matcherResults = story.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `copilotProtection.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }));
} else if (execution.task.flow === "instruction_persistence") {
const story = await runInstructionPersistenceFlow({
page, api, fixtures, execution, nonce, secrets, deadlineAt: startedAtMs + deadlineMs,
@@ -2196,6 +2238,15 @@ for (const execution of executions) {
await writeSanitizedJson(snapshotsDir, "api-response-pagination.json", paging, secrets);
if (!paging.passed) invariantFailures.push(`Bounded response paging was not proven: ${paging.failure}`);
}
if (piFileSeed) {
const proof = await collectPiFileEvidence({ page, api, nonce, companyId: fixtures.company.id,
issueId: issue.id, agentId: fixtures.agent.id, run: finalRun,
events: runEventsByRun.find(entry => entry.runId === finalRun.id)?.events ?? [],
seed: piFileSeed, workspace: workspacePath,
environment: execution.environment.id === "daytona" ? "daytona" : "local", environmentId: fixtures.environment.id,
evidence: data => writeSanitizedJson(snapshotsDir, "pi-file-observation.json", data, secrets) });
await writeSanitizedJson(snapshotsDir, "pi-file-evidence.json", proof, secrets);
}
if (execution.suite.id === "task-titles") {
const titleEvidence = gradeTaskTitle({
initial: titleCreation?.issue, submitted: titleCreation?.submitted,
tests/runner-e2e/selectors.ts
+2
View File
@@ -5,6 +5,7 @@ export interface RunnerSelectorOptions {
all: boolean;
list: boolean;
matrixJson: boolean;
installedStartupOnly?: boolean;
ids: string[];
suites: string[];
groups: string[];
@@ -53,6 +54,7 @@ export function parseRunnerSelectors(
if (flag === "--all") options.all = true;
else if (flag === "--list") options.list = true;
else if (flag === "--matrix-json") options.matrixJson = true;
else if (flag === "--installed-startup-only") options.installedStartupOnly = true;
else if (flag === "--headed") options.headed = true;
else if (flag === "--ui") options.ui = true;
else if (flag === "--debug") options.debug = true;
tests/runner-e2e/server-config.test.ts
+14 -2
View File
@@ -1,4 +1,4 @@
import { mkdtemp, readFile, rm } from "node:fs/promises";
import { mkdir, mkdtemp, realpath, readFile, rm, symlink } from "node:fs/promises";
import { createServer } from "node:net";
import os from "node:os";
import path from "node:path";
@@ -8,7 +8,7 @@ import {
reserveRunnerE2EDatabasePort,
type LoopbackPortReservation,
} from "./ports.js";
import { prepareRunnerE2EServerConfig } from "./server-config.js";
import { createRunnerE2ETemporaryRoot, prepareRunnerE2EServerConfig } from "./server-config.js";
const roots: string[] = [];
const reservations: LoopbackPortReservation[] = [];
@@ -22,6 +22,18 @@ afterEach(async () => {
});
describe("isolated paid E2E database ports", () => {
it("uses a physical temporary root when the host temp directory has an alias", async () => {
const parent = await realpath(await mkdtemp(path.join(os.tmpdir(), "e2e-temp-alias-")));
roots.push(parent);
const physical = path.join(parent, "physical");
const alias = path.join(parent, "alias");
await mkdir(physical);
await symlink(physical, alias);
const root = await createRunnerE2ETemporaryRoot(alias);
expect(root).toBe(await realpath(root));
expect(path.dirname(root)).toBe(physical);
});
it("keeps parallel database reservations distinct and held until server spawn", async () => {
reservations.push(
...(await Promise.all(
tests/runner-e2e/server-config.ts
+7 -1
View File
@@ -1,8 +1,14 @@
import { mkdir, readFile, writeFile } from "node:fs/promises";
import { mkdir, mkdtemp, realpath, readFile, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js";
import { reserveRunnerE2EDatabasePort } from "./ports.js";
/** Runtime instructions and native filesystem grants must use the same path. */
export async function createRunnerE2ETemporaryRoot(parent = os.tmpdir()) {
return await realpath(await mkdtemp(path.join(parent, "paperclip-runner-e2e-")));
}
/** Seed only the disposable fixture. Onboarding preserves this validated config
* and still creates the normal secrets and database. Restarts keep the same DB.
*/
tests/runner-e2e/server-restart-preservation-process.test.ts
+76
View File
@@ -0,0 +1,76 @@
import { spawn, type ChildProcess } from "node:child_process";
import { once } from "node:events";
import { mkdtemp, realpath, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { expect, it } from "vitest";
import { createRunnerE2EServerStopper } from "./server-stop.js";
import { createProcessTreeOwner } from "./process-tree-owner.js";
import { readProcessTable } from "./process-tree.js";
import { readLinuxProcessStartedAt } from "../../packages/paperclip-runner/src/live/linux-process-start.js";
it.skipIf(process.platform === "win32")("keeps the real admitted daemon and child alive across controller exit, then drains both controller generations", async () => {
const directory = await mkdtemp(path.join(await realpath(os.tmpdir()), "pc-restart-owner-"));
const daemon = "/bin/sh";
const fixtureDaemons = new Set<number>();
const entry = path.join(directory, "controller.cjs");
const stop = createRunnerE2EServerStopper({ gracefulTimeoutMs: 3000, forcedTimeoutMs: 2000,
hasSpawnError: () => false, markExpectedStop: () => {}, log: () => {},
// Only the fixture's declared role is synthetic. PID, start, ancestry,
// groups, observation and signal delivery all use the real kernel table.
createOwner: child => createProcessTreeOwner(child, { readTable: async () => {
const table = await readProcessTable();
return table?.map(row => fixtureDaemons.has(row.pid) && row.kind === "sh" ? { ...row, kind: "paperclip-runnerd" } : row) ?? null;
} }),
});
const closed: Promise<void>[] = [];
try {
// This is a tiny inert process-role fixture, not a provider or real daemon.
await writeFile(entry, `
const { spawn } = require('node:child_process');
const fs = require('node:fs');
const marker = ${JSON.stringify(path.join(directory, "child-ready"))} + process.pid;
const daemon = spawn(${JSON.stringify(daemon)}, ['-c', ${JSON.stringify("trap 'exit 0' TERM; /bin/sleep 600 & printf '%s' \"$!\" > \"$1\"; wait")}, 'fixture', marker], { detached: true, stdio: ['ignore', 'ignore', 'pipe'] });
let daemonError = ''; daemon.stderr.on('data', chunk => { daemonError += chunk; });
daemon.once('exit', (code, signal) => { if (process.connected) process.send({ failure: { code, signal, stderr: daemonError } }); });
const background = spawn('/bin/sleep', ['600'], { detached: true, stdio: 'ignore' });
process.on('SIGTERM', () => process.exit(0));
const ready = setInterval(() => {
if (fs.existsSync(marker)) { clearInterval(ready); process.send({ daemon: daemon.pid, background: background.pid }); }
}, 10);
setInterval(() => {}, 1000);
`);
const start = async () => {
const child = spawn(process.execPath, [entry], { detached: true, stdio: ["ignore", "ignore", "pipe", "ipc"] });
closed.push(new Promise(resolve => child.once("close", () => resolve())));
const message = once(child, "message");
await stop.watch(child);
const result = (await message)[0];
expect(result.failure, JSON.stringify(result)).toBeUndefined();
fixtureDaemons.add(result.daemon);
return { child, ids: result as { daemon: number; background: number } };
};
const first = await start();
const before = await readProcessTable();
expect(before).not.toBeNull();
const runner = before!.find(row => row.pid === first.ids.daemon)!;
expect(runner.kind).toBe("sh");
const descendants = before!.filter(row => row.parentPid === runner.pid);
expect(descendants.length).toBeGreaterThan(0);
await stop.forRestart(first.child, { processPid: runner.pid, processGroupId: runner.processGroupId,
processStartedAt: process.platform === "linux" ? readLinuxProcessStartedAt(runner.pid) : new Date(runner.started).toISOString() });
const after = await readProcessTable();
expect(after!.find(row => row.pid === runner.pid)?.started).toBe(runner.started);
for (const descendant of descendants) expect(after!.find(row => row.pid === descendant.pid)?.started).toBe(descendant.started);
expect(after!.some(row => row.pid === first.ids.background && !row.state?.startsWith("Z"))).toBe(false);
const second = await start();
await stop.stopAll(); await Promise.all(closed);
const final = await readProcessTable();
const expectedGone = [first.child.pid, first.ids.daemon, first.ids.background, ...descendants.map(row => row.pid),
second.child.pid, second.ids.daemon, second.ids.background];
expect(final!.filter(row => expectedGone.includes(row.pid) && !row.state?.startsWith("Z"))).toEqual([]);
} finally {
try { await stop.stopAll(); await Promise.all(closed); }
finally { await rm(directory, { recursive: true, force: true }); }
}
}, 15000);
tests/runner-e2e/server-restart-preservation.test.ts
+186
View File
@@ -0,0 +1,186 @@
import type { ChildProcess } from "node:child_process";
import { expect, it, vi } from "vitest";
import { createProcessTreeOwner, parseRestartRunnerIdentity, restartProcessStartMatches } from "./process-tree-owner.js";
import { createRunnerE2EServerStopper } from "./server-stop.js";
import { diagnosticProcessKind, type ProcessObservation } from "./process-tree.js";
const started = "2026-10-02T13:55:27.000Z";
const identity = { processPid: 200, processGroupId: 200, processStartedAt: started };
const row = (pid: number, parentPid: number, group = pid, kind = "node"): ProcessObservation =>
({ pid, parentPid, processGroupId: group, started, kind, state: "S" });
function fixture() {
let table = [row(process.pid, 1, 10), row(100, process.pid), row(200, 100, 200, "paperclip-runnerd"),
row(201, 200), row(300, 100), row(900, process.pid)];
const signals: Array<[number, string]> = [];
const roots: Array<ChildProcess> = [];
const owners: Array<ReturnType<typeof createProcessTreeOwner>> = [];
const root = (pid: number) => {
const child = { pid, exitCode: null as number | null, signalCode: null as NodeJS.Signals | null,
kill: (signal: string) => {
signals.push([pid, signal]); child.exitCode = 0;
table = table.filter(row => row.pid !== pid).map(row => row.parentPid === pid ? { ...row, parentPid: 1 } : row);
return true;
} };
const handle = child as unknown as ChildProcess;
roots.push(handle); return handle;
};
const stop = createRunnerE2EServerStopper({ gracefulTimeoutMs: 100, forcedTimeoutMs: 100,
hasSpawnError: () => false, markExpectedStop: () => {}, log: () => {},
createOwner: child => {
const owner = createProcessTreeOwner(child, { readTable: async () => table.map(row => ({ ...row })),
signalGroup: (group, signal) => { signals.push([group, signal]); table = table.filter(row => row.processGroupId !== group); } });
owners.push(owner); return owner;
} });
return { stop, child: root(100), root, signals, owners, table: () => table,
replace: (next: ProcessObservation[]) => { table = next; }, close: () => owners.forEach(owner => owner.stopObserving()) };
}
it("matches Linux process birth receipts at ps precision without accepting another second", () => {
expect(restartProcessStartMatches(started, "2026-10-02T13:55:27.731Z", "linux")).toBe(true);
for (const expected of ["2026-10-02T13:55:26.999Z", "2026-10-02T13:55:28.000Z", "invalid"]) {
expect(restartProcessStartMatches(started, expected, "linux")).toBe(false);
}
expect(restartProcessStartMatches("invalid", started, "linux")).toBe(false);
expect(restartProcessStartMatches(started, "2026-10-02T13:55:27.731Z", "darwin")).toBe(false);
expect(restartProcessStartMatches(started, started, "darwin")).toBe(true);
});
it.skipIf(process.platform === "win32")("preserves only the admitted daemon tree across restart and finally retires old and new owners", async () => {
const f = fixture();
try {
await f.stop.watch(f.child);
await f.stop.forRestart(f.child, identity);
expect(f.signals).toEqual([[100, "SIGTERM"], [300, "SIGTERM"]]);
expect(f.table().map(row => row.pid)).toContain(200);
// The daemon forks a new private group after controller loss; it stays owned.
f.replace([...f.table(), row(202, 201), row(400, process.pid), row(401, 400)]);
const replacement = f.root(400); await f.stop.watch(replacement);
await f.stop.stopAll();
expect(f.table().map(row => row.pid).sort()).toEqual([process.pid, 900].sort());
expect(f.signals.filter(([pid]) => pid === 200)).toEqual([[200, "SIGTERM"]]);
expect(f.signals.findIndex(([pid]) => pid === 400)).toBeLessThan(f.signals.findIndex(([pid]) => pid === 200));
expect(f.signals.some(([pid]) => pid === 202)).toBe(true);
expect(f.signals.some(([pid]) => pid === 900)).toBe(false);
} finally { f.close(); }
});
it.skipIf(process.platform === "win32").each([
["foreign controller", (rows: ProcessObservation[]) => { rows.find(row => row.pid === 200)!.parentPid = 900; }],
["reused PID", (rows: ProcessObservation[]) => { rows.find(row => row.pid === 200)!.started = "2026-10-02T13:55:28.000Z"; }],
["wrong role", (rows: ProcessObservation[]) => { rows.find(row => row.pid === 200)!.kind = "node"; }],
["mixed process group", (rows: ProcessObservation[]) => { rows.push(row(901, 900, 200)); }],
["same controller group", (rows: ProcessObservation[]) => { rows.find(row => row.pid === 200)!.processGroupId = 100; }],
["missing runner", (rows: ProcessObservation[]) => { rows.splice(rows.findIndex(row => row.pid === 200), 1); }],
] as const)("rejects %s without sending a restart signal", async (_name, mutate) => {
const f = fixture();
try {
const rows = f.table(); mutate(rows); f.replace(rows);
await expect(f.stop.forRestart(f.child, identity)).rejects.toThrow(/runner|daemon|process group/);
expect(f.signals).toEqual([]);
} finally { f.close(); }
});
it.skipIf(process.platform === "win32")("fails if the admitted daemon dies instead of pretending recovery preserved it", async () => {
const f = fixture();
try {
const kill = f.child.kill.bind(f.child);
f.child.kill = signal => { const result = kill(signal); f.replace(f.table().filter(row => row.pid !== 200)); return result; };
await expect(f.stop.forRestart(f.child, identity)).rejects.toThrow("did not survive");
await f.stop.stopAll();
expect(f.table().map(row => row.pid).sort()).toEqual([process.pid, 900].sort());
} finally { f.close(); }
});
it.skipIf(process.platform === "win32")("does not transfer ownership to a reused retained daemon group during final cleanup", async () => {
const f = fixture();
try {
await f.stop.forRestart(f.child, identity);
f.replace(f.table().map(row => row.pid === 200 ? { ...row, started: "2026-10-02T14:00:00.000Z" } : row));
await expect(f.stop.stopAll()).rejects.toThrow("cleanup incomplete");
expect(f.signals.some(([pid]) => pid === 200)).toBe(false);
} finally { f.close(); }
});
it.each([null, {}, { ...identity, processPid: 0 }, { ...identity, processGroupId: 100 }, { ...identity, processStartedAt: "bad" }])(
"rejects malformed restart identity %j", value => { expect(() => parseRestartRunnerIdentity(value)).toThrow(); });
it.skipIf(process.platform === "win32")("keeps the first final-cleanup deadline after an inspection failure and repeated stop", async () => {
const f = fixture();
const clock = vi.spyOn(Date, "now");
try {
await f.stop.forRestart(f.child, identity);
clock.mockReturnValue(10_000);
const owner = f.owners[0]!;
const observe = owner.observe;
let fail = true;
owner.observe = async () => { if (fail) { fail = false; throw new Error("one inspection failure"); } await observe(); };
await expect(f.stop(f.child)).rejects.toThrow("inspection failure");
clock.mockReturnValue(20_000);
await f.stop(f.child);
expect(f.signals.filter(([pid]) => pid === 200)).toEqual([[200, "SIGKILL"]]);
} finally { clock.mockRestore(); f.close(); }
});
it.skipIf(process.platform === "win32")("final cancellation interrupts restart and retires its preserved descendants", async () => {
const f = fixture();
let cleanup: Promise<void> | undefined;
try {
const kill = f.child.kill.bind(f.child);
f.child.kill = signal => { const result = kill(signal); cleanup = f.stop.stopAll(); return result; };
await expect(f.stop.forRestart(f.child, identity)).rejects.toThrow("Final shutdown interrupted");
await cleanup;
expect(f.table().map(row => row.pid).sort()).toEqual([process.pid, 900].sort());
} finally { f.close(); }
});
it.skipIf(process.platform === "win32")("final cleanup still retires the old runner when replacement startup failed", async () => {
const f = fixture();
try {
await f.stop.forRestart(f.child, identity);
const replacement = f.root(400);
Object.assign(replacement, { exitCode: 1 });
await f.stop.watch(replacement);
await f.stop.stopAll();
expect(f.table().map(row => row.pid).sort()).toEqual([process.pid, 900].sort());
expect(f.signals.filter(([pid]) => pid === 200)).toEqual([[200, "SIGTERM"]]);
} finally { f.close(); }
});
it("normalizes only the exact Linux comm truncation, never a nearby role name", () => {
expect(diagnosticProcessKind("/tmp/paperclip-runne", "linux")).toBe("paperclip-runnerd");
expect(diagnosticProcessKind("/tmp/paperclip-runnerd", "darwin")).toBe("paperclip-runnerd");
expect(diagnosticProcessKind("paperclip-runne", "darwin")).toBe("other");
for (const name of ["paperclip-runn", "paperclip-runner", "paperclip-runneX", "paperclip-runnerd-other"]) {
expect(diagnosticProcessKind(name, "linux")).toBe("other");
}
});
it.skipIf(process.platform === "win32")("still drains the retained daemon after replacement cleanup reports failure", async () => {
const f = fixture();
try {
await f.stop.forRestart(f.child, identity);
f.replace([...f.table(), row(400, process.pid)]);
const replacement = f.root(400); await f.stop.watch(replacement);
const latestOwner = f.owners[1]!;
latestOwner.observe = async () => { throw new Error("replacement inspection failed"); };
await expect(f.stop.stopAll()).rejects.toThrow("cleanup incomplete");
expect(f.signals.filter(([pid]) => pid === 200)).toEqual([[200, "SIGTERM"]]);
expect(f.signals.findIndex(([pid]) => pid === 400)).toBeLessThan(f.signals.findIndex(([pid]) => pid === 200));
} finally { f.close(); }
});
it.skipIf(process.platform === "win32")("does not reuse an already-retired ordinary restart controller's old cleanup deadline", async () => {
const f = fixture();
const clock = vi.spyOn(Date, "now").mockReturnValue(10_000);
try {
await f.stop(f.child);
clock.mockReturnValue(20_000);
f.replace([...f.table(), row(400, process.pid), row(401, 400)]);
const replacement = f.root(400); await f.stop.watch(replacement);
await f.stop.stopAll();
expect(f.signals.filter(([pid]) => pid === 401)).toEqual([[401, "SIGTERM"]]);
} finally { clock.mockRestore(); f.close(); }
});
tests/runner-e2e/server-stop.test.ts
+29 -6
View File
@@ -46,6 +46,10 @@ function stopper(gracefulTimeoutMs = 2_000) {
});
return { stop, logs, errors };
}
function expectSuccessfulWrapperReceipt(wrapper: ChildProcess, messages: unknown[], diagnostics = "") {
expect([wrapper.exitCode, wrapper.signalCode], diagnostics).toEqual([0, null]);
expect(messages, diagnostics).toContainEqual({ exitCode: 0, signalCode: null });
}
afterEach(async () => {
for (const child of children.splice(0)) {
if (child.exitCode !== null || child.signalCode !== null) continue;
@@ -153,15 +157,19 @@ it.skipIf(process.platform === "win32")("lets launcher cancellation join wrapper
// during the async close even though the helper sends only one signal.
const messages: unknown[] = [];
wrapper.on("message", message => messages.push(message));
const exited = once(wrapper, "exit");
// Process exit is not IPC completion. Register both barriers before
// cancellation so the final worker receipt is drained before asserting it.
const closed = once(wrapper, "close");
const disconnected = once(wrapper, "disconnect");
await owner.observe();
const cancellation = stopOwnedProcessTree(wrapper, owner, 2_000, 2_000);
await new Promise(resolve => setTimeout(resolve, 250));
if (wrapper.connected) wrapper.send("finish");
await exited;
await Promise.all([closed, disconnected]);
await cancellation;
expect(messages, wrapperErrors).toContainEqual({ message: "close-complete", pid: workerPid });
expect(messages).toContainEqual({ exitCode: 0, signalCode: null });
const diagnostics = `${wrapperErrors}\nwrapper exit=${wrapper.exitCode} signal=${wrapper.signalCode} connected=${wrapper.connected}`;
expect(messages, diagnostics).toContainEqual({ message: "close-complete", pid: workerPid });
expectSuccessfulWrapperReceipt(wrapper, messages, diagnostics);
expect(() => process.kill(workerPid!, 0)).toThrow();
} finally {
owner.stopObserving();
@@ -169,6 +177,21 @@ it.skipIf(process.platform === "win32")("lets launcher cancellation join wrapper
}
});
it.each(["failed", "missing"] as const)("rejects a %s worker receipt even when the wrapper closes successfully", async receipt => {
const { entry } = await fixture(receipt === "failed"
? "process.send({ exitCode: 23, signalCode: null }, () => process.exit(0));"
: "process.exit(0);");
const wrapper = start(entry);
const messages: unknown[] = [];
wrapper.on("message", message => messages.push(message));
const closed = once(wrapper, "close");
const disconnected = once(wrapper, "disconnect");
await Promise.all([closed, disconnected]);
expect([wrapper.exitCode, wrapper.signalCode]).toEqual([0, null]);
expect(messages).toEqual(receipt === "failed" ? [{ exitCode: 23, signalCode: null }] : []);
expect(() => expectSuccessfulWrapperReceipt(wrapper, messages)).toThrow();
});
it.skipIf(process.platform === "win32")("escalates the owned server group so a hung descendant cannot remain", async () => {
const { entry } = await fixture(`
@@ -255,8 +278,8 @@ it.skipIf(process.platform === "win32")("recovers a failed cleanup without repea
});
try {
const failed = expect(stop(child)).rejects.toThrow("transient inspection failure");
// Process-table inspection can take longer than 100ms on a loaded host.
// Keep the child alive until the intended inspection failure actually occurs.
// Hold the child until the one-shot failure is actually injected. A fixed
// delay can let it exit during the preceding successful inspection.
await inspectionFailed;
if (child.connected) child.send("finish");
await failed;
tests/runner-e2e/server-stop.ts
+67 -6
View File
@@ -1,5 +1,5 @@
import type { ChildProcess } from "node:child_process";
import { createProcessTreeOwner, incompleteTreeFallback } from "./process-tree-owner.js";
import { createProcessTreeOwner, incompleteTreeFallback, type RestartRunnerIdentity } from "./process-tree-owner.js";
// The wrapper receives Playwright's group signal; only it signals Paperclip.
export const runnerE2EServerDetached = process.platform !== "win32";
@@ -16,6 +16,10 @@ export function createRunnerE2EServerStopper(options: {
type State = {
owner: ReturnType<typeof createProcessTreeOwner>;
promise?: Promise<void>;
restartPromise?: Promise<void>;
finalRequested?: boolean;
finalPhaseStarted?: boolean;
finalComplete?: boolean;
gracefulDeadline?: number;
forcedDeadline?: number;
gracefulSent: boolean;
@@ -23,16 +27,21 @@ export function createRunnerE2EServerStopper(options: {
escalationLogged: boolean;
};
const states = new WeakMap<ChildProcess, State>();
const watched = new Set<ChildProcess>();
let finalGraceDeadline: number | undefined;
let finalForcedDeadline: number | undefined;
let finalAllPromise: Promise<void> | undefined;
const exited = (child: ChildProcess) => child.exitCode !== null || child.signalCode !== null || options.hasSpawnError(child);
function watch(child: ChildProcess) {
let state = states.get(child);
if (!state) {
state = { owner: (options.createOwner ?? createProcessTreeOwner)(child), gracefulSent: false, groupSignals: new Set(), escalationLogged: false };
states.set(child, state);
watched.add(child);
}
return state;
}
async function stopOnce(child: ChildProcess, state: State, signal: NodeJS.Signals) {
async function stopOnce(child: ChildProcess, state: State, signal: NodeJS.Signals, restarting = false) {
state.gracefulDeadline ??= Date.now() + options.gracefulTimeoutMs;
await state.owner.observe();
if (!exited(child) && !state.gracefulSent) {
@@ -40,22 +49,28 @@ export function createRunnerE2EServerStopper(options: {
}
while (Date.now() < state.gracefulDeadline) {
await state.owner.observe();
if (restarting && state.finalRequested) throw new Error("Final shutdown interrupted controller restart");
if (exited(child)) {
if (restarting) {
state.owner.assertRestartRunnerAlive();
if (!state.owner.restartCleanupGroups().size) return;
}
if (!state.owner.liveGroups().length) { state.owner.stopObserving(); return; }
// The leader is gone. Retire only descendants whose start identities
// we observed while they belonged to this server, including private groups.
const unsignaled = new Set(state.owner.liveGroups().map(group => group.processGroupId).filter(pid => !state.groupSignals.has(pid)));
const unsignaled = new Set((restarting ? [...state.owner.restartCleanupGroups()] : state.owner.liveGroups().map(group => group.processGroupId)).filter(pid => !state.groupSignals.has(pid)));
for (const pid of await state.owner.signal("SIGTERM", unsignaled)) state.groupSignals.add(pid);
}
await wait(25);
}
if (restarting) throw new Error("Controller restart did not retire non-durable children within its graceful deadline");
if (!state.escalationLogged) {
options.log(`\nPaperclip did not stop within ${options.gracefulTimeoutMs}ms; sending SIGKILL\n`);
state.escalationLogged = true;
}
if (runnerE2EServerDetached) await state.owner.signal("SIGKILL");
else if (!exited(child)) child.kill("SIGKILL");
state.forcedDeadline ??= Date.now() + options.forcedTimeoutMs;
state.forcedDeadline ??= finalForcedDeadline ?? Date.now() + options.forcedTimeoutMs;
do {
await state.owner.observe();
if (exited(child) && !state.owner.liveGroups().length) { state.owner.stopObserving(); return; }
@@ -66,7 +81,19 @@ export function createRunnerE2EServerStopper(options: {
const stop = (child: ChildProcess, signal: NodeJS.Signals = "SIGTERM"): Promise<void> => {
options.markExpectedStop(child);
const state = watch(child);
state.promise ??= Promise.resolve().then(() => stopOnce(child, state, signal)).catch(async error => {
state.finalRequested = true;
state.promise ??= Promise.resolve().then(async () => {
if (state.restartPromise && !state.finalPhaseStarted) {
await state.restartPromise.catch(() => {});
state.finalPhaseStarted = true;
// A completed restart is a separate phase, not elapsed cleanup time.
state.gracefulDeadline = finalGraceDeadline ?? Date.now() + options.gracefulTimeoutMs;
state.forcedDeadline = undefined;
state.groupSignals.clear();
}
await stopOnce(child, state, signal);
state.finalComplete = true;
}).catch(async error => {
try {
await incompleteTreeFallback(error, child, state, state.gracefulDeadline!, options.forcedTimeoutMs);
} finally {
@@ -77,5 +104,39 @@ export function createRunnerE2EServerStopper(options: {
repeating the graceful signal. */
return state.promise;
};
return Object.assign(stop, { watch: (child: ChildProcess) => watch(child).owner.observe() });
return Object.assign(stop, {
watch: (child: ChildProcess) => watch(child).owner.observe(),
forRestart: (child: ChildProcess, identity: RestartRunnerIdentity): Promise<void> => {
options.markExpectedStop(child);
const state = watch(child);
if (state.finalRequested || state.restartPromise) return Promise.reject(new Error("Restart already requested or final cleanup active"));
state.restartPromise = (async () => {
await state.owner.preserveRunnerForRestart(identity);
options.log(`Controller restart preserves admitted runner pid=${identity.processPid} pgid=${identity.processGroupId} started=${identity.processStartedAt}\n`);
await stopOnce(child, state, "SIGTERM", true);
})();
return state.restartPromise;
},
stopAll: (signal: NodeJS.Signals = "SIGTERM"): Promise<void> => {
// Replacement controller drain precedes old daemon retirement. All
// generations share the original final-cleanup window, including retries.
finalGraceDeadline ??= Math.min(Date.now() + options.gracefulTimeoutMs,
...[...watched].map(child => states.get(child)!).filter(state => state.finalRequested && !state.finalComplete
&& (!state.restartPromise || state.finalPhaseStarted)).map(state => state.gracefulDeadline ?? Infinity));
finalForcedDeadline ??= finalGraceDeadline + options.forcedTimeoutMs;
for (const child of watched) {
const state = states.get(child)!;
state.finalRequested = true;
state.gracefulDeadline ??= finalGraceDeadline;
}
finalAllPromise ??= Promise.resolve().then(async () => {
const failures: unknown[] = [];
for (const child of [...watched].reverse()) {
try { await stop(child, signal); } catch (error) { failures.push(error); }
}
if (failures.length) throw new AggregateError(failures, "Owned server cleanup incomplete");
}).catch(error => { finalAllPromise = undefined; throw error; });
return finalAllPromise;
},
});
}
tests/runner-e2e/server.ts
+24 -16
View File
@@ -1,3 +1,6 @@
import { parseRestartRunnerIdentity, type RestartRunnerIdentity } from "./process-tree-owner.js";
import { usesInstalledCli, verifyInstalledCli } from "./installed-cli.js";
import { runnerMatrix } from "./catalog.js";
import { createRunnerE2EServerStopper, runnerE2EServerDetached } from "./server-stop.js";
import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js";
import { qualifyLegacyClaudeCli } from "./legacy-claude-cli.js";
@@ -26,6 +29,12 @@ const configPath = required("PAPERCLIP_CONFIG");
const port = required("PAPERCLIP_RUNNER_E2E_PORT");
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
const paperclipCli = path.join(repositoryRoot, "tests/runner-e2e/server-entry.ts");
const executionIds: string[] = JSON.parse(process.env.PAPERCLIP_RUNNER_E2E_EXECUTION_IDS ?? "[]");
const selectedExecutions = usesInstalledCli(process.env) ? executionIds.map(id => {
const execution = runnerMatrix.find(row => row.id === id);
if (!execution) throw new Error("Unknown installed CLI proof execution");
return execution;
}) : [];
const {
controlDirectory,
restartRequestPath,
@@ -128,9 +137,11 @@ async function startServer() {
if (shutdownRequested()) {
throw new Error("Refusing to start Paperclip after wrapper shutdown");
}
// Recheck bytes and dependency resolution on every controller restart.
const installed = await verifyInstalledCli(process.env, selectedExecutions);
const candidate = spawn(
process.execPath,
installedRelease?.args ?? runnerE2ETypeScriptProcessArgs(repositoryRoot, paperclipCli, ["onboard", "--yes", "--run"]),
installed ? [installed.entry, "onboard", "--yes", "--run"] : runnerE2ETypeScriptProcessArgs(repositoryRoot, paperclipCli, ["onboard", "--yes", "--run"]),
{
cwd: installedRelease?.cwd ?? repositoryRoot,
env: definedServerEnvironment,
@@ -237,6 +248,7 @@ async function waitForHealthToStop() {
interface RestartRequest {
requestId: string;
preserveRunner?: RestartRunnerIdentity;
}
async function readRestartRequest(): Promise<RestartRequest | null> {
@@ -262,7 +274,8 @@ async function readRestartRequest(): Promise<RestartRequest | null> {
) {
return null;
}
return { requestId };
const preserveRunner = (value as { preserveRunner?: unknown }).preserveRunner;
return { requestId, ...(preserveRunner === undefined ? {} : { preserveRunner: parseRestartRunnerIdentity(preserveRunner) }) };
}
async function writeRestartAck(
@@ -284,12 +297,13 @@ async function writeRestartAck(
await rename(temporaryAckPath, restartAckPath);
}
async function restartServer(requestId: string) {
async function restartServer({ requestId, preserveRunner }: RestartRequest) {
activeRestartRequestId = requestId;
appendLog(`\nRestart request ${requestId}: stopping Paperclip\n`);
const previous = child;
if (!previous) throw new Error("No Paperclip server is available to restart");
await stopServer(previous);
if (preserveRunner) await stopServer.forRestart(previous, preserveRunner);
else await stopServer(previous);
if (child === previous) child = null;
// Do not mistake an orphaned old server for a healthy replacement. The port
// must stop answering before the next launcher is allowed to start.
@@ -339,13 +353,12 @@ async function supervise() {
const request = await readRestartRequest();
if (request && request.requestId !== lastRestartRequestId) {
lastRestartRequestId = request.requestId;
await restartServer(request.requestId);
await restartServer(request);
}
await delay(200);
}
const running = child;
if (running) await stopServer(running, shutdownSignal ?? "SIGTERM");
await stopServer.stopAll(shutdownSignal ?? "SIGTERM");
}
let exitCode = 0;
@@ -364,15 +377,10 @@ try {
);
}
}
const running = child;
if (running) {
try {
await stopServer(running);
} catch (stopError) {
appendLog(
`Failed to stop Paperclip after supervisor failure: ${stopError instanceof Error ? stopError.message : String(stopError)}\n`,
);
}
try {
await stopServer.stopAll();
} catch (stopError) {
appendLog(`Failed to stop Paperclip after supervisor failure: ${stopError instanceof Error ? stopError.message : String(stopError)}\n`);
}
}
tests/runner-e2e/types.ts
+3
View File
@@ -34,7 +34,10 @@ export type RunnerTaskFlow =
| "plan_approval_completion"
| "warm_three_turn"
| "instruction_persistence"
| "pi_native"
| "pi_controls"
| "native_active_stop"
| "copilot_protection"
| "native_provider_loss"
| "cursor_native";
tests/runner-e2e/user-actions.ts
+20 -4
View File
@@ -1,4 +1,22 @@
import { expect, type Page } from "@playwright/test";
import { expect, type Locator, type Page } from "@playwright/test";
/** Rich editors interpret pasted Markdown; fill inserts literal paragraph text. */
export async function fillTaskPrompt(editor: Locator, prompt: string): Promise<void> {
if (!/^ {0,3}(?:`{3,}|~{3,})/m.test(prompt)
|| await editor.evaluate(element => element.tagName === "TEXTAREA")) {
await editor.fill(prompt);
return;
}
await editor.fill("");
await editor.focus();
await editor.evaluate((element, text) => {
const clipboardData = new DataTransfer();
clipboardData.setData("text/plain", text);
const event = new ClipboardEvent("paste", { clipboardData, bubbles: true, cancelable: true });
element.dispatchEvent(event);
if (!event.defaultPrevented) throw new Error("Task editor did not accept the Markdown paste");
}, prompt);
}
export async function createTaskThroughUi(input: {
page: Page;
@@ -46,9 +64,7 @@ export async function createTaskThroughUi(input: {
const titleInput = dialog.getByRole("textbox", { name: "Task title", exact: true });
if (await titleInput.isVisible()) await titleInput.fill(input.title);
else if (input.requireExplicitTitle) throw new Error("This title-preservation case requires a visible explicit title input");
await dialog
.getByRole("textbox", { name: "editable markdown", exact: true })
.fill(input.prompt);
await fillTaskPrompt(dialog.getByRole("textbox", { name: "editable markdown", exact: true }), input.prompt);
if (input.workMode !== "standard") {
await dialog.getByRole("button", { name: "Add to composer", exact: true }).click();
await input.page.getByTestId(input.workMode === "planning" ? "composer-add-plan" : "composer-add-ask").click();
tests/runner-e2e/web-server-command.ts
+9
View File
@@ -21,3 +21,12 @@ export function runnerE2EWebServerCommand(repositoryRoot: string) {
export function runnerE2ETypeScriptProcessArgs(repositoryRoot: string, entry: string, args: string[] = []) {
return ["--import", path.join(repositoryRoot, "cli/node_modules/tsx/dist/loader.mjs"), entry, ...args];
}
// pnpm's generated playwright bin shim adds NODE_PATH, even when the caller
// rejected ambient injection. The installed lane invokes the public JS bin
// with the current Node directly, keeping the WebServer environment closed.
export function runnerE2EPlaywrightInvocation(repositoryRoot: string, args: string[], installed: boolean) {
return installed
? { command: process.execPath, args: [path.join(repositoryRoot, "node_modules/@playwright/test/cli.js"), ...args] }
: { command: "pnpm", args: ["exec", "playwright", ...args] };
}
tests/runner-e2e/workflow-security.test.ts
+2 -2
View File
@@ -15,12 +15,12 @@ const everydayOracleImage =
"python@sha256:9d2e5553305c7c7b0097999bb17187c69b921ccd6bc9d40e4bb5ebe652c00285";
describe("public repository paid workflow security", () => {
it("keeps the manual EC2 image build credential-free and pins the authorized target", async () => {
it("keeps the manual hosted Linux image build credential-free and pins the authorized target", async () => {
const workflow = await readFile(path.join(repositoryRoot, ".github/workflows/docker-runner-check.yml"), "utf8");
const manual = workflow.slice(workflow.indexOf(" authorize_manual:"));
expect(manual.match(/AWS_CI_TRUSTED_USER_IDS/gu)).toHaveLength(2);
expect(manual.match(/test "\$REPOSITORY_ID" = 1170821064/gu)).toHaveLength(2);
expect(manual).toContain('runs-on: runs-on/fleet=paperclip-public-pr-x64/env=public-ci');
expect(manual.slice(manual.indexOf(' manual_image:'))).toContain('runs-on: ubuntu-latest');
expect(manual).toContain('repos/$REPOSITORY/git/ref/heads/$TARGET_BRANCH');
expect(manual).toContain('ref: ${{ needs.authorize_manual.outputs.target_sha }}');
expect(manual).toContain('SOURCE_SHA: ${{ needs.authorize_manual.outputs.target_sha }}');