Merge origin/master into task workspace ownership

Preserve owned-home cleanup safety while integrating upstream runtime-context cleanup, current Runner build caching, release crate caching and verification docs.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-12 13:40:12 -05:00
commit 650198cc04
65 files changed
+3843 -58

No files matched your search

@@ -0,0 +1,46 @@
# create-verification-skill attribution
This skill comes from Paperclip's portable pstack fork in
`paperclipai/agent-plugins`, commit
`410257663e0b344df656e604f1601aa1db95f7e4`, under `vendor/pstack/`.
The fork imported pstack 0.15.15 by Lauren Tan (poteto) from
[cursor/plugins](https://github.com/cursor/plugins/tree/ccb5507cec1546dc88135c1139c811e6c59115ba/pstack),
upstream revision `ccb5507cec1546dc88135c1139c811e6c59115ba`.
Paperclip adapts the workflow to the existing repo-root `feature-map/`.
It preserves the invocation policy in `agents/openai.yaml`, removes the
Cursor-specific `disable-model-invocation` frontmatter field, and omits the
separate runtime skill dependency because Paperclip supplies execution.
No upstream Cursor built-in skills are redistributed.
The imported resources include `SKILL.md`, `agents/openai.yaml`, and all three
files in `references/feature-map-example/`. Keep those examples with the skill
when copying or refreshing it.
## License
The original license is reproduced verbatim below.
```text
MIT License
Copyright (c) 2026 Lauren Tan
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```
@@ -0,0 +1,45 @@
---
name: create-verification-skill
description: "Generate a project-local verification skill that drives your app the way a user does — any language, framework, or platform. Use for /create-verification-skill, \"make a control skill for this repo\", or when a project has no scripted way to prove UI/CLI/service behavior."
---
# Create a verification skill
Every serious project needs a scripted way to drive the real app and prove behavior: launch it, exercise a feature the way a user would, and capture evidence. This skill generates that as a project-local skill (`.agents/skills/verify-<app>/`) tailored to the repo. You write the generator's output for the next agent, not for a human: it will be read cold, mid-task, by an agent that has never seen the app.
## 1. Interview the repo, not the user
Answer these from the codebase and only ask the user what you cannot observe:
- **Surface:** what does a user actually touch? A web UI, a CLI/TUI, a desktop app, an API, a mobile app, a library? A repo can have several; pick the primary one and note the rest.
- **Run:** how does the app start locally? Prefer the repo's own documented dev command (package scripts, Makefile, README quickstart). Note ports, env vars, seed data, auth.
- **Drive:** how can an agent interact with it programmatically? Existing harnesses first — Playwright/Cypress specs, expect scripts, PTY helpers, curl-able endpoints, a debug port. Only then pick a generic recipe: browser/CDP for web and Electron, a tmux/PTY harness for CLI/TUI, plain HTTP for services.
- **Observe:** what evidence can be captured? Screenshots, terminal transcripts, response bodies, logs, exit codes, DB state.
- **Isolate:** can two instances run side by side (ports, data dirs, profiles)? If not, say so in the generated skill: refusing to double-drive a shared instance beats corrupting the user's session.
If the checkout doesn't build or start as-is, fix that first (or report it precisely) before generating; a skill written against a broken base teaches wrong steps. When an irrelevant missing asset blocks startup (a static dir the API never serves, a sample config), the generated skill may create it, clearly marked as verification scaffolding, and remove it in cleanup.
## 2. Generate the skill
Write `.agents/skills/verify-<app>/SKILL.md` with YAML frontmatter (`name: verify-<app>` and a `description` that names the app, the surface, and when to reach for it — without frontmatter the skill never registers) and these sections, each grounded in what the interview actually found (no placeholders left):
- **Launch:** the exact command that starts the app for verification, and how to tell it's ready (a log line, a port answering, a prompt). Include teardown. For a short-lived CLI or TUI there is no server to keep alive: launch means build the binary (or install deps) once, then start each drive in its own isolated PTY or tmux session.
- **Doctor:** one read-only check that answers "is this instance worth driving?" — process up, right version/build, port owned by us, auth valid. An agent runs this first whenever anything looks off.
- **Drive:** the harness recipe with real selectors/commands from this repo, not examples. Prefer stable handles (ARIA labels, data attributes, prompt strings, route paths) over coordinates and tab order.
- **Evidence:** what to capture for a proof and where it goes. State the proof standards: exercise the real user path, not internal setters or test-only endpoints; capture the action and the resulting state, not just the final screen; verify side effects (files written, rows inserted, messages sent) alongside what's visible; mocks only where a production boundary already isolates the external system. When the safe path is a dry-run or test mode, verify what it actually skips by observing (files, network, git refs) rather than trusting its name: some dry-runs still touch the network or open a browser.
- **Cleanup:** how to tear down instances the run created. Never kill by process name; kill what you started. Cleanup removes instances and scratch state, never the evidence: proof artifacts survive the teardown, in a location the skill names.
- **Helpers:** any script the skill ships is executable and its invocation is shown in the skill body. A helper the reader has to reverse-engineer is not a helper.
## 3. Seed the feature map
For Paperclip, use the existing repo-root `feature-map/` directory and its `README.md`; have the generated verification skill reference that map instead of creating a second map inside the skill directory. Reuse existing feature files and use Poteto's format below for new or updated entries. For other projects, create `.agents/skills/verify-<app>/features/README.md` plus one file per user-facing feature you can identify (aim for the top 3-5 to start, from routes, commands, menus, or docs). Follow the shape in [`references/feature-map-example/`](references/feature-map-example/), with a README index and one file per feature. Each file answers, from the user's point of view: what the feature is, how to reach it, how to drive it with the harness, and what observable end state proves it works. The four H2s are `Sub-features`, `How to get to it (user POV)`, `Driving it with <harness>`, and `Gotchas`. The map is the repo's maintained verification source; a proof that drives one convenient entry point is incomplete when the map lists others.
## 4. Prove the generated skill before handing it over
Run its own instructions end to end once: launch, doctor, drive ONE mapped feature (one is enough; the map exists so later runs can cover the rest), capture evidence, clean up. After cleanup, confirm the evidence still exists at the named location — a cleanup that eats the proof fails this step. Fix what fails, and run the generated cleanup after every failed iteration too, so broken attempts don't strand processes and ports. A generated skill that was never executed is a draft, not a deliverable.
## 5. Offer the maintenance loop
Point the user at `/maintain-verification-skill` for keeping the map honest as the app changes. Suggest a cadence only if they ask.
Origin and license: [NOTICE.md](NOTICE.md).
@@ -0,0 +1,2 @@
policy:
allow_implicit_invocation: false
@@ -0,0 +1,47 @@
# Notes verification map
This directory is the maintained source for verifying the user-facing behavior of Notes. Read the index before driving the app, then use the matching feature file as the recipe.
## Baseline preconditions
- Launch Notes at `http://127.0.0.1:4173` with a disposable data directory.
- Set `NOTES_DATA_DIR=/tmp/notes-verify-$RUN_ID` so concurrent runs do not share state.
- Seed notes titled `Quarterly plan` and `Grocery list`.
- Put `control-notes` and the `notes` CLI on `PATH`.
- Run `control-notes doctor` and require the expected URL, data directory, and build revision.
- Never drive an instance that was not started by this verification run.
## Driving conventions
- Start every recipe from the baseline state unless its preconditions say otherwise.
- Prefer ARIA roles and accessible names over CSS selectors or DOM position.
- Treat every command as literal. Keep quoted names and flags unchanged.
- Run browser actions through `control-notes browser`.
- Run terminal actions through `control-notes cli -- <command>`.
- Restore seeded data after a mutation. Do not remove proof artifacts during cleanup.
## Proof and skip reporting
- Capture the user action and the resulting state, not only the final screen.
- UI proof includes an ARIA snapshot and a screenshot with the app identity visible.
- CLI proof includes the command, stdout, stderr, and exit code.
- Mutation proof includes a read-only second view of the stored value.
- Record the feature ID and entry point used with every artifact.
- Report an unreachable path with the attempted command and the unmet precondition.
- Do not report a skipped entry point as verified through a different path.
## Feature entry contract
Each feature file starts with an H1 title and one paragraph describing the user-visible behavior. It then uses exactly four H2 sections in this order.
1. `Sub-features` lists short IDs with one line for each behavior.
2. `How to get to it (user POV)` lists every user entry point.
3. `Driving it with <harness>` starts with `Preconditions:` and uses labeled bullets that pair each user action with an exact command and observable result.
4. `Gotchas` lists traps that can waste or invalidate a verification run.
Keep implementation details out of the map. Name only user paths, stable handles, required state, commands, and observable proof.
## Features
- [Create a note](./create-note.md) covers browser and CLI creation, cancellation, persistence, and cleanup.
- [Search notes](./search.md) covers toolbar, keyboard, and CLI search with matching, empty, and clear states.
@@ -0,0 +1,39 @@
# Create a note
Create note lets a user save a titled note from the browser or CLI, cancel an unfinished draft, and confirm the saved note from a second user-facing view.
## Sub-features
- `create-open` opens a blank editor from each browser entry point.
- `create-save` persists a title and body.
- `create-cancel` discards an unfinished browser draft.
- `create-cli` creates the same note shape from the terminal.
## How to get to it (user POV)
- Choose the `New note` button in the browser toolbar.
- Press `n` in the browser while focus is outside an editable field.
- Run `notes create --title <title> --body <body>` in a terminal.
## Driving it with control-notes
Preconditions:
- Notes is healthy at `http://127.0.0.1:4173`.
- No note is titled `Release checklist`.
- `control-notes doctor` reports the expected URL and disposable data directory.
- **Open editor.** Choose `New note`. Run `control-notes browser click --role button --name "New note"`. A form named `Note editor` appears with focus in the `Title` textbox.
- **Enter content.** Type the title and body. Run `control-notes browser fill --role textbox --name "Title" --value "Release checklist"` and `control-notes browser fill --role textbox --name "Body" --value "Tag and publish"`. The `Save note` button becomes enabled.
- **Save note.** Choose `Save note`. Run `control-notes browser click --role button --name "Save note"`. A status named `Note saved` appears and the heading reads `Release checklist`.
- **Confirm persistence.** Return to the note list and reopen the note. Run `control-notes browser click --role link --name "All notes"` and `control-notes browser click --role link --name "Release checklist"`. The editor shows both saved values.
- **Cancel draft.** Open a new note, enter `Discard me`, and choose `Cancel`. Run `control-notes browser click --role button --name "New note"`, `control-notes browser fill --role textbox --name "Title" --value "Discard me"`, and `control-notes browser click --role button --name "Cancel"`. The note list returns and has no `Discard me` link.
- **CLI entry.** Create a second note. Run `control-notes cli -- notes create --title "CLI note" --body "Created from terminal" --format json`. Exit code `0` and stdout contain the new note ID and title.
- **Proof.** Reopen both saved notes from `All notes`. Run `control-notes browser snapshot --aria --path artifacts/create-note/list.aria.txt` and `control-notes browser screenshot --path artifacts/create-note/list.png`. The artifacts show `Release checklist` and `CLI note`.
## Gotchas
- Pressing `n` while a textbox has focus types the character instead of opening a new editor.
- Titles are trimmed on save. Assert the rendered title, not the draft input value.
- A save status alone is insufficient proof. Reopen the note from the list.
- Remove `Release checklist` and `CLI note` during fixture cleanup, but retain their proof artifacts.
@@ -0,0 +1,45 @@
# Search notes
Search lets a user find notes by title or body text, inspect a matching note, and distinguish no matches from an unavailable search.
## Sub-features
- `search-open` opens search from each supported browser entry point.
- `search-match` returns title and body matches without changing note data.
- `search-open-result` opens a result in the note editor.
- `search-empty` shows a complete empty state for a query with no matches.
- `search-clear` removes the query and restores the recent-notes view.
- `search-cli` returns the same matching notes from the terminal.
## How to get to it (user POV)
- Choose the `Search` button in the browser toolbar.
- Press `/` in the browser while focus is outside an editable field.
- Run `notes search <query>` in a terminal.
## Driving it with control-notes
Preconditions:
- Notes is healthy at `http://127.0.0.1:4173`.
- The disposable data directory contains `Quarterly plan` with body text `Draft budget`.
- `control-notes doctor` reports the expected URL and data directory.
- **Toolbar entry.** Choose the `Search` button. Run `control-notes browser click --role button --name "Search"`. A dialog named `Search notes` appears with focus in its searchbox.
- **Keyboard entry.** Close the dialog, focus the page, and press `/`. Run `control-notes browser press --key "/"`. The same dialog appears and the page does not insert a slash.
- **Title match.** Type `quarterly`. Run `control-notes browser fill --role searchbox --name "Search notes" --value "quarterly"`. The `Search results` list contains `Quarterly plan` and does not contain `Grocery list`.
- **Body match.** Replace the query with `budget`. Run `control-notes browser fill --role searchbox --name "Search notes" --value "budget"`. The result `Quarterly plan` remains visible with a body-match excerpt.
- **Open result.** Choose `Quarterly plan`. Run `control-notes browser click --role link --name "Quarterly plan"`. The dialog closes and the editor heading reads `Quarterly plan`.
- **Empty state.** Reopen search and enter `volcano`. Run `control-notes browser fill --role searchbox --name "Search notes" --value "volcano"`. A status named `No matching notes` appears after search completes.
- **Clear query.** Choose `Clear search`. Run `control-notes browser click --role button --name "Clear search"`. The searchbox is empty and the `Recent notes` region replaces the result list.
- **CLI match.** Search from the terminal. Run `control-notes cli -- notes search "quarterly" --format json`. Exit code `0` and stdout contain one object whose title is `Quarterly plan`.
- **CLI miss.** Search for an absent value. Run `control-notes cli -- notes search "volcano" --format json`. Exit code `0` and stdout are `[]`.
- **Proof.** Capture the populated result state. Run `control-notes browser snapshot --aria --path artifacts/search/results.aria.txt` and `control-notes browser screenshot --path artifacts/search/results.png`. Both artifacts identify Notes, the query, and `Quarterly plan`.
## Gotchas
- Pressing `/` while the editor or searchbox has focus inserts text instead of opening search.
- Results update after a short debounce. Wait for the results list or empty status, not a fixed sleep.
- Archived notes are excluded unless the user enables `Include archived`.
- The CLI defaults to human-readable output. Use `--format json` for stable assertions.
- Opening a result changes browser state. Reopen search before proving another query.
@@ -0,0 +1,46 @@
# maintain-verification-skill attribution
This skill comes from Paperclip's portable pstack fork in
`paperclipai/agent-plugins`, commit
`410257663e0b344df656e604f1601aa1db95f7e4`, under `vendor/pstack/`.
The fork imported pstack 0.15.15 by Lauren Tan (poteto) from
[cursor/plugins](https://github.com/cursor/plugins/tree/ccb5507cec1546dc88135c1139c811e6c59115ba/pstack),
upstream revision `ccb5507cec1546dc88135c1139c811e6c59115ba`.
Paperclip adapts the workflow to the existing repo-root `feature-map/`.
It preserves the invocation policy in `agents/openai.yaml`, removes the
Cursor-specific `disable-model-invocation` frontmatter field, and omits the
separate runtime skill dependency because Paperclip supplies execution.
No upstream Cursor built-in skills are redistributed.
The imported resources include `SKILL.md` and `agents/openai.yaml`.
The workflow uses the examples shipped with
[`create-verification-skill`](../create-verification-skill/references/feature-map-example/).
## License
The original license is reproduced verbatim below.
```text
MIT License
Copyright (c) 2026 Lauren Tan
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```
@@ -0,0 +1,40 @@
---
name: maintain-verification-skill
description: "Periodic pass that keeps a project's verification skill and feature map honest: parallel source readers per feature, one live session driving every feature, at most one PR of proven corrections. Use for /maintain-verification-skill or \"audit the verify skill\"."
---
# Maintain a verification skill
A feature map rots the moment the app changes. This skill is the upkeep loop for a skill generated by `/create-verification-skill` (or any project-local verification skill with a feature map). The unit of rigor is the feature, not every sentence: cover every feature file from source and exercise every feature live, without terminalising every bullet.
## Outcomes
Pick one, and say which:
- **clean** — every feature got source and live coverage; nothing worth shipping. No branch, no PR.
- **changed** — one PR ships proven doc, harness, or map corrections.
- **blocked** — coverage could not finish or a proven fix could not ship safely. Say exactly what blocked it.
## Edit scope
Only edit the verification skill's own directory (its SKILL.md and any harness scripts it owns) and its feature-map directory. For Paperclip, the map is the existing repo-root `feature-map/`; maintain it in place instead of creating or moving it into a skill directory. Use [Poteto's feature-map format](../create-verification-skill/references/feature-map-example/) for new or updated entries. Never edit product code during a run: a behavior the map describes that the app no longer does is either doc drift (fix the map) or a product regression (report it, don't paper over it in docs).
## Pass
0. **Locate the target.** Find the verification skill to maintain: the project-local skill whose body has launch/drive sections and a feature map (usually `.agents/skills/verify-*/`). Several candidates → ask which one; none → stop and point at `/create-verification-skill` instead of inventing a target.
1. **Index hygiene.** Read the feature map README and glob its sibling files. Fix missing, extra, duplicate, or dead entries. Lightweight; no generated inventory.
2. **Source wave.** One read-only subagent per feature file, launched concurrently. Each explains "how does this user-facing feature work?" from source, flags likely doc drift with citations, and returns one concise live-verification recipe. Children never drive the app and never edit files. Return shape: feature summary / source entry points / likely drift or none / one recipe.
3. **Reconcile.** Every feature file has a returned summary. Merge overlapping recipes into as few app states as practical. Spot-check cited drift; don't re-prove clean claims. Sweep recent churn for user-facing surfaces missing from the map — require a concrete source path before calling one missing.
4. **Live pass.** Required even when source looks clean. The coordinator owns all driving; follow the verification skill's own launch model — one long-lived instance driven serially for servers and UIs, or a fresh isolated session per drive for short-lived CLIs (the skill's Launch section decides, not this one). Exercise every feature at least once, and hold three invariants the whole pass, whatever the failure: (1) never drive an instance you haven't health-checked since it last did something surprising — doctor before first drive, doctor on each fresh session where sessions are the unit, doctor again after any failed drive, and where doctor can't see the failure (a wedged UI state on a healthy process), reset to a known state or relaunch rather than hoping; (2) evidence captured so far survives every cleanup, checked at its named location, not assumed; (3) nothing a drive started outlives that drive's usefulness — failed-iteration residue is cleaned whether the session is stuck, exited, or shared (for a shared instance, clean the residue, not the instance). A doctor failure caused by skill drift is drift: fix it under edit scope and retry once — restart whatever the fix invalidated, nothing more — before calling the pass `blocked`. A feature that can't be reached is `verified-unreachable` only with the concrete prerequisite (auth, entitlement, OS, external state) and the route attempted; if the map omits that prerequisite, that's drift. Any harness fix from triage gets re-driven live before it ships. Final teardown happens after the last drive of the run — including those re-proofs — so nothing outlives the run (evidence stays, per the skill).
5. **Triage.** Wrong or missing user-POV description → doc drift, fix it. Working behavior the harness can't drive → harness gap, fix it; a harness fix follows the same helpers rule as generation (scripts executable, invocation documented in the skill body). App behavior that's actually broken → product gap; record it for the user, keep it out of this PR.
6. **Ship or stop.** For changed: one PR of proven corrections, re-read every changed file first. For clean or blocked: no PR, report the outcome and the coverage honestly.
Keep concise run notes (features covered, unreachable prerequisites, confirmed drift, outcome) in a scratch location; don't commit them.
Origin and license: [NOTICE.md](NOTICE.md).
@@ -0,0 +1,2 @@
policy:
allow_implicit_invocation: false
+166
View File
@@ -0,0 +1,166 @@
---
name: verify-paperclip
description: Launch an isolated Paperclip instance, drive its real web UI or CLI using the existing feature-map recipes, capture evidence, and clean up owned processes. Use to verify user-facing Paperclip behavior.
---
# Verify Paperclip
Use the repo-root [feature map](../../../feature-map/README.md) to select
verification recipes. Read the relevant feature file before driving it. Report
any recipe drift with evidence; map updates belong to
[maintain-verification-skill](../maintain-verification-skill/SKILL.md).
Read [development instructions](../../../doc/DEVELOPING.md) for current commands.
## Launch
Run from the repository root with Node 24.11+, Python 3, `lsof`, and dependencies
installed with `pnpm install --frozen-lockfile`. Browser driving uses the existing
`@playwright/test` dependency. If Chromium is unavailable, run
`node node_modules/@playwright/test/cli.js install chromium`, or select an installed
browser with `PAPERCLIP_PLAYWRIGHT_CHANNEL=chrome`.
Create separate, unique data and evidence directories. Keep the evidence outside
the data directory:
```sh
VERIFY_DATA_DIR=$(mktemp -d "${TMPDIR:-/tmp}/paperclip-verify-XXXXXX")
VERIFY_EVIDENCE_DIR=$(mktemp -d "${TMPDIR:-/tmp}/paperclip-proof-XXXXXX")
python3 .agents/skills/verify-paperclip/scripts/launch.py \
--data-dir "$VERIFY_DATA_DIR" --evidence-dir "$VERIFY_EVIDENCE_DIR"
```
Keep this command in a foreground terminal/session. The helper runs the supported
`onboard --yes --bind loopback --no-install-service --run` command, with isolated
embedded PostgreSQL, a free loopback port, no inherited instance routing or cwd
`.env`, and no automatic browser opening. It records `launch.json` in the evidence
directory. Source startup builds its required shared/plugin packages and migrates
the empty database; readiness can take longer on the first run.
This path starts an empty instance without model credentials. Create company and
agent state through the journey being checked. A UI check that stops before model
execution does not prove an agent run. For a preconfigured company/CEO and live
model work, use the documented [test-drive](../../../doc/DEVELOPING.md#one-command-isolated-manual-test-drive)
command with an authorized environment-backed credential and `--no-browser`.
Record its printed data directory, URL, runtime PID and commit before driving;
never use a fake credential to claim provider verification.
For Paperclip-managed reusable services, follow
[dev-workspace run/verify/fix](../paperclip-dev-workspace-run-verify-fix/SKILL.md)
instead of launching a second server. Its cloned-data/login requirements differ
from this empty local instance.
## Doctor
After startup, and again after any surprising behavior, run:
```sh
python3 .agents/skills/verify-paperclip/scripts/doctor.py \
--evidence-dir "$VERIFY_EVIDENCE_DIR"
```
This read-only check compares the recorded launcher, runtime PID, port owner,
process cwd, configuration, served Git commit, health/bootstrap readiness, root
HTML response, and company-list access. It writes `doctor.json` or exits nonzero.
Do not trust a root 200 or CLI doctor output alone. The CLI doctor checks whether
the port is available before startup; it is not proof of a running server's
ownership. Browser readiness still requires the expected visible page. Reset or
relaunch a wedged browser even when the process health passes.
## Drive
Use accessible roles, labels, and placeholders from the latest page state.
Existing Playwright specs in `tests/e2e/` supply recipes and selectors; their
configured servers and fixtures are separate from this live instance. Run a named
suite through its own configuration only when that is the evidence needed.
The executable onboarding smoke drives `onboarding.md` / `web-wizard`: open
`/onboarding` in the disposable self-hosted instance,
enter a unique organization name, choose Continue, reach the agent-name field,
reopen `/<company-prefix>/onboarding`, reload, and corroborate the saved company
through read-only API checks:
```sh
node .agents/skills/verify-paperclip/scripts/onboarding-smoke.mjs \
--evidence-dir "$VERIFY_EVIDENCE_DIR"
```
This covers company creation and setup resumption, with no mocked responses or
provider calls. Hiring, first task execution, and the alternate CLI entry remain
separate journeys. The script requires an empty instance and refuses a non-owned
or non-loopback target. For other features, reuse the owned URL and follow the
map's entry points. Company routes normally require the selected company's
`issuePrefix`, for example `/<PREFIX>/goals`; use current navigation rather than
guessing resource URLs. The unprefixed `/onboarding` route starts naming; use the
company-prefixed onboarding route to resume its agent setup. Check relevant
experimental gates and actor permissions.
For CLI checks, discover exact flags with
`node cli/node_modules/tsx/dist/cli.mjs cli/src/index.ts <command> --help`, then
select the disposable data directory/API/company explicitly. API reads can
corroborate UI writes; API-created state does not prove a creation UI.
## Evidence
Record feature file, sub-feature and entry-point ID, commit/dirty state, owned URL,
company/user, flags, runtime/harness, live versus simulated dependencies, command,
exit code, expected and observed behavior, and evidence links. Report each entry
as passed, failed, blocked with its prerequisite, or not run.
The smoke saves before/after/reload screenshots, visible-page text, a Playwright
trace, and `result.json` with company ID and API corroboration. On failure it
retains the trace, visible state, and failure result; rerun doctor before retrying.
Preserve launch output and relevant application logs without secrets. Reopen or
reload changed state and verify persistence/continuation, not just a success toast.
One passing sub-feature does not certify the entire feature family or other
surfaces. A full maintenance pass must exercise every mapped feature or report
incomplete coverage.
For Paperclip-assigned work, attach important proof through
[the artifact workflow](../../../doc/AGENT-ARTIFACTS.md). Keep runtime results in
the evidence directory or issue, not in the map; `coverage.json` describes
documentation coverage rather than live test health.
## Cleanup
Close the browser context you created (the smoke does this in `finally`). Stop the owned runtime with:
```sh
python3 .agents/skills/verify-paperclip/scripts/stop.py \
--evidence-dir "$VERIFY_EVIDENCE_DIR"
```
The helper verifies the runtime's port, cwd, and launch ancestry before sending
SIGTERM to that PID, then waits for runtime exit and checks that its HTTP
listener and embedded database have stopped. Signaling the runtime PID lets the server drain before stopping
PostgreSQL; a terminal group interrupt can reach the database first. Do this after
failed drives as well as successful runs. If startup fails before a runtime record
exists, stop only the foreground session you created and inspect its recorded data
directory for residue. Never kill by process name or stop another checkout's server.
Verify the recorded HTTP port has no listener and the owned runtime PID has
exited. Embedded PostgreSQL should stop with the server; inspect only the recorded
data directory if shutdown is incomplete. Retain the data directory for diagnosis
until no owned process uses it; remove only that exact temporary directory when
it is no longer useful. Never delete the evidence directory during teardown.
Confirm screenshots, trace, `result.json`, `launch.json`, and `doctor.json` still
exist after shutdown. A failed launch leaves its evidence intact.
## Helpers
All helper invocations are shown above. `launch.py`, `doctor.py`, and `stop.py` use Python's
standard library and the repo CLI; `onboarding-smoke.mjs` uses the repo's existing
Playwright dependency. They introduce no new package dependency or test framework.
## Helper safety check
Run the standard-library ownership test before changing launch/doctor/stop helpers:
```sh
python3 -m unittest discover -s .agents/skills/verify-paperclip/tests -v
```
It starts an owned fixture in another temporary checkout and confirms that both
doctor and stop reject its claimed port/PID without stopping the server. A second
fixture closes its listener before exiting and proves that cleanup waits for
runtime exit. These checks do
not qualify Paperclip startup, database readiness, or onboarding.
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env python3
"""Read-only identity and readiness check for an owned verification launch."""
import argparse
import json
import os
from pathlib import Path
import subprocess
import sys
import urllib.request
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--evidence-dir', required=True, type=Path)
args = parser.parse_args()
evidence = args.evidence_dir.resolve()
try:
launch = json.loads((evidence / 'launch.json').read_text())
repo = Path(launch['repo']).resolve()
assert repo == Path(__file__).resolve().parents[4], 'Launch belongs to another checkout.'
runtime = json.loads((Path(launch['dataDir']) / 'instances/default/runtime-info.json').read_text())
assert runtime['dashboardUrl'] == launch['url'], 'Runtime URL changed.'
assert runtime['port'] == launch['port'], 'Runtime port changed.'
assert runtime['host'] == '127.0.0.1', 'Expected trusted loopback.'
pid = runtime['pid']
os.kill(pid, 0)
listeners = subprocess.check_output(['lsof', '-nP', f'-iTCP:{launch["port"]}',
'-sTCP:LISTEN', '-t'], text=True).split()
assert set(listeners) == {str(pid)}, 'Port owner does not match runtime PID.'
if sys.platform == 'darwin':
rows = subprocess.check_output(['lsof', '-a', '-p', str(pid), '-d', 'cwd', '-Fn'], text=True).splitlines()
cwd = Path(next(row[1:] for row in rows if row.startswith('n'))).resolve()
else:
cwd = Path(f'/proc/{pid}/cwd').resolve(strict=True)
assert cwd == repo or repo in cwd.parents, 'Port owner cwd belongs to another checkout.'
ancestor = pid
seen = set()
while ancestor != launch['launcherPid']:
assert ancestor > 1 and ancestor not in seen, 'Runtime is not descended from this launch.'
seen.add(ancestor)
ancestor = int(subprocess.check_output(['ps', '-p', str(ancestor), '-o', 'ppid='], text=True).strip())
config = json.loads((Path(launch['dataDir']) / 'instances/default/config.json').read_text())
assert config['database']['mode'] == 'embedded-postgres', 'Expected isolated embedded database.'
assert Path(config['database']['embeddedPostgresDataDir']).resolve().is_relative_to(Path(launch['dataDir'])), 'Database path escaped isolation.'
with urllib.request.urlopen(launch['url'] + '/api/health', timeout=15) as response:
health = json.load(response)
assert health['status'] == 'ok' and health['bootstrapStatus'] == 'ready', 'Instance is not ready.'
assert health['deploymentMode'] == 'local_trusted' and health['authReady'], 'Unexpected auth mode.'
commit = subprocess.check_output(['git', '-C', str(repo), 'rev-parse', 'HEAD'], text=True).strip()
assert health['commit'] == launch['commit'] == commit, 'Served commit differs from checkout/launch.'
with urllib.request.urlopen(launch['url'], timeout=15) as response:
assert response.status == 200 and 'text/html' in response.headers.get('Content-Type', ''), 'Root UI is unavailable.'
with urllib.request.urlopen(launch['url'] + '/api/companies', timeout=15) as response:
companies = json.load(response)
assert isinstance(companies, list), 'Board company access failed.'
result = {'status': 'passed', 'pid': pid, 'cwd': str(cwd), 'url': launch['url'],
'health': health, 'companyIds': [company['id'] for company in companies]}
(evidence / 'doctor.json').write_text(json.dumps(result, indent=2) + '\n')
print(f'PASS: owned PID {pid}, {launch["url"]}, commit {commit}, ready board UI/API.')
except Exception as error:
(evidence / 'doctor-failure.json').write_text(json.dumps({'status': 'failed', 'error': str(error)}, indent=2) + '\n')
raise SystemExit(f'Doctor failed: {error}')
+52
View File
@@ -0,0 +1,52 @@
#!/usr/bin/env python3
"""Run the repository's credential-free isolated foreground onboarding path."""
import argparse
import json
import os
from pathlib import Path
import socket
import subprocess
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--data-dir', required=True, type=Path)
parser.add_argument('--evidence-dir', required=True, type=Path)
args = parser.parse_args()
repo = Path(__file__).resolve().parents[4]
data = args.data_dir.resolve()
evidence = args.evidence_dir.resolve()
if data == evidence or data in evidence.parents or evidence in data.parents:
parser.error('Data and evidence directories must be separate.')
data.mkdir(parents=True, exist_ok=True)
evidence.mkdir(parents=True, exist_ok=True)
if any(data.iterdir()) or (evidence / 'launch.json').exists():
parser.error('Use a fresh empty data directory and a new evidence directory.')
cli = repo / 'cli/node_modules/tsx/dist/cli.mjs'
if not cli.is_file():
parser.error('Install checkout dependencies first: pnpm install --frozen-lockfile')
for port in range(3100, 4100):
try:
with socket.socket() as probe:
probe.bind(('127.0.0.1', port))
break
except OSError:
continue
else:
raise RuntimeError('No available verification port between 3100 and 4099.')
excluded = {'DATABASE_URL', 'DATABASE_MIGRATION_URL', 'HOST', 'PORT', 'SERVE_UI',
'BETTER_AUTH_URL', 'BETTER_AUTH_BASE_URL', 'NODE_OPTIONS'}
env = {key: value for key, value in os.environ.items()
if not key.startswith('PAPERCLIP_') and key not in excluded}
env.update({'PAPERCLIP_HOME': str(data), 'PAPERCLIP_INSTANCE_ID': 'default',
'PAPERCLIP_DISABLE_CWD_ENV_FILE': 'true', 'PAPERCLIP_NO_BROWSER': 'true',
'PAPERCLIP_OPEN_ON_LISTEN': 'false', 'PAPERCLIP_UPDATE_CHECK': '0',
'PAPERCLIP_TELEMETRY_DISABLED': '1', 'PAPERCLIP_DEPLOYMENT_MODE': 'local_trusted',
'PAPERCLIP_DEPLOYMENT_EXPOSURE': 'private', 'PAPERCLIP_BIND': 'loopback',
'HOST': '127.0.0.1', 'PORT': str(port)})
record = {'dataDir': str(data), 'evidenceDir': str(evidence), 'repo': str(repo),
'launcherPid': os.getpid(), 'port': port, 'url': f'http://127.0.0.1:{port}',
'commit': subprocess.check_output(['git', '-C', str(repo), 'rev-parse', 'HEAD'], text=True).strip()}
(evidence / 'launch.json').write_text(json.dumps(record, indent=2) + '\n')
print(json.dumps(record), flush=True)
os.chdir(repo)
os.execvpe('node', ['node', str(cli), 'cli/src/index.ts', 'onboard', '--data-dir',
str(data), '--yes', '--bind', 'loopback', '--no-install-service', '--run'], env)
@@ -0,0 +1,79 @@
#!/usr/bin/env node
import assert from 'node:assert/strict';
import fs from 'node:fs/promises';
import path from 'node:path';
import { createRequire } from 'node:module';
import { fileURLToPath } from 'node:url';
import { execFileSync } from 'node:child_process';
const repo = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');
const require = createRequire(path.join(repo, 'package.json'));
const { chromium, expect } = require('@playwright/test');
assert.equal(process.argv[2], '--evidence-dir', 'Usage: onboarding-smoke.mjs --evidence-dir <path>');
assert.ok(process.argv[3]);
const evidence = path.resolve(process.argv[3]);
execFileSync('python3', [path.join(repo, '.agents/skills/verify-paperclip/scripts/doctor.py'), '--evidence-dir', evidence], { stdio: 'inherit' });
const launch = JSON.parse(await fs.readFile(path.join(evidence, 'launch.json'), 'utf8'));
assert.equal(new URL(launch.url).hostname, '127.0.0.1');
const browser = await chromium.launch({ headless: true, ...(process.env.PAPERCLIP_PLAYWRIGHT_CHANNEL ? { channel: process.env.PAPERCLIP_PLAYWRIGHT_CHANNEL } : {}) });
const context = await browser.newContext({ viewport: { width: 1440, height: 1000 }, serviceWorkers: 'block' });
const page = await context.newPage();
const errors = [];
page.on('pageerror', error => errors.push(error.message));
const companyName = `Verification ${Date.now()}`;
const result = { feature: 'onboarding.md', subFeatures: ['company', 'resume'], entryPoint: 'web-wizard', commit: launch.commit, url: launch.url, actor: 'local_trusted board', providerCalls: false, mockedResponses: false, companyName, status: 'failed' };
await context.tracing.start({ screenshots: true, snapshots: true, sources: true });
async function capture(name) {
await page.screenshot({ path: path.join(evidence, `${name}.png`), fullPage: true });
await fs.writeFile(path.join(evidence, `${name}.txt`), await page.locator('body').innerText());
}
try {
const before = await page.request.get(`${launch.url}/api/companies`);
assert.equal(before.ok(), true);
assert.deepEqual(await before.json(), [], 'Smoke requires a fresh instance.');
await page.goto(`${launch.url}/onboarding`);
const start = page.getByRole('button', { name: /Start Onboarding|New Organization|Add Agent/ });
if (await start.count()) await start.first().click();
const create = page.getByRole('button', { name: /Build a new organization/ });
if (await create.count()) await create.first().click();
await expect(page.getByRole('heading', { name: 'What is the name of your organization?' })).toBeVisible({ timeout: 60_000 });
await page.getByPlaceholder('e.g. Northwind Labs').fill(companyName);
await capture('before-create');
await page.getByRole('button', { name: /^Continue/ }).click();
await expect(page.locator('#onboarding-agent-name')).toBeVisible({ timeout: 30_000 });
await capture('after-create');
const companiesResponse = await page.request.get(`${launch.url}/api/companies`);
assert.equal(companiesResponse.ok(), true);
const companies = await companiesResponse.json();
const company = companies.find(item => item.name === companyName);
assert.ok(company, 'UI-created company must be persisted.');
result.companyId = company.id;
const goalsResponse = await page.request.get(`${launch.url}/api/companies/${company.id}/goals`);
assert.equal(goalsResponse.ok(), true);
assert.deepEqual(await goalsResponse.json(), [], 'Naming must not invent a mission.');
await page.goto(`${launch.url}/${company.issuePrefix}/onboarding`);
await expect(page.locator('#onboarding-agent-name')).toBeVisible({ timeout: 30_000 });
await capture('after-reopen');
await page.reload();
await expect(page.locator('#onboarding-agent-name')).toBeVisible({ timeout: 30_000 });
await capture('after-reload');
const persistedResponse = await page.request.get(`${launch.url}/api/companies`);
assert.equal(persistedResponse.ok(), true);
const persisted = await persistedResponse.json();
assert.equal(persisted.length, 1, 'Resume must not create another company.');
assert.equal(persisted[0].id, company.id);
assert.deepEqual(errors, [], 'No browser exceptions.');
result.status = 'passed';
result.observed = 'Company persisted; onboarding resumed at the agent step after reload; no mission invented.';
result.notRun = ['agent hiring', 'first task/model execution', 'cli-setup entry point'];
} catch (error) {
result.error = error.message;
await capture('failure').catch(() => {});
process.exitCode = 1;
} finally {
result.browserErrors = errors;
await context.tracing.stop({ path: path.join(evidence, 'trace.zip') });
await browser.close();
await fs.writeFile(path.join(evidence, 'result.json'), `${JSON.stringify(result, null, 2)}\n`);
console.log(JSON.stringify(result));
}
+65
View File
@@ -0,0 +1,65 @@
#!/usr/bin/env python3
"""Stop only an owned verification runtime and check that its database exits."""
import argparse
import json
import os
from pathlib import Path
import signal
import subprocess
import sys
import time
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--evidence-dir', required=True, type=Path)
args = parser.parse_args()
evidence = args.evidence_dir.resolve()
launch = json.loads((evidence / 'launch.json').read_text())
repo = Path(launch['repo']).resolve()
assert repo == Path(__file__).resolve().parents[4], 'Launch belongs to another checkout.'
instance = Path(launch['dataDir']) / 'instances/default'
runtime_file = instance / 'runtime-info.json'
runtime = json.loads(runtime_file.read_text()) if runtime_file.exists() else None
pid = None
if runtime:
assert runtime['port'] == launch['port'] and runtime['dashboardUrl'] == launch['url']
pid = runtime['pid']
owners = subprocess.run(['lsof', '-nP', f'-iTCP:{launch["port"]}', '-sTCP:LISTEN', '-t'], capture_output=True, text=True).stdout.split()
assert set(owners) == {str(pid)}, 'Refusing to signal a different or unverified port owner.'
if sys.platform == 'darwin':
rows = subprocess.check_output(['lsof', '-a', '-p', str(pid), '-d', 'cwd', '-Fn'], text=True).splitlines()
cwd = Path(next(row[1:] for row in rows if row.startswith('n'))).resolve()
else:
cwd = Path(f'/proc/{pid}/cwd').resolve(strict=True)
assert cwd == repo or repo in cwd.parents, 'Refusing to signal another checkout.'
ancestor = pid
seen = set()
while ancestor != launch['launcherPid']:
assert ancestor > 1 and ancestor not in seen, 'Runtime is not descended from this launch.'
seen.add(ancestor)
ancestor = int(subprocess.check_output(['ps', '-p', str(ancestor), '-o', 'ppid='], text=True).strip())
os.kill(pid, signal.SIGTERM)
def runtime_exited():
if pid is None:
return True
try:
os.kill(pid, 0)
except ProcessLookupError:
return True
# An exited child can remain a zombie until its launcher reaps it.
state = subprocess.run(['ps', '-p', str(pid), '-o', 'stat='],
capture_output=True, text=True)
return state.returncode == 0 and state.stdout.strip().startswith('Z')
deadline = time.monotonic() + 45
while time.monotonic() < deadline:
owners = subprocess.run(['lsof', '-nP', f'-iTCP:{launch["port"]}', '-sTCP:LISTEN', '-t'], capture_output=True, text=True).stdout.split()
if not owners and not (instance / 'db/postmaster.pid').exists() and runtime_exited():
result = {'status': 'passed', 'port': launch['port'], 'serverStopped': True,
'databaseStopped': True, 'runtimeExited': True, 'evidenceRetained': sorted(p.name for p in evidence.iterdir())}
(evidence / 'cleanup.json').write_text(json.dumps(result, indent=2) + '\n')
print(f'PASS: owned runtime/database stopped; evidence retained at {evidence}.')
break
time.sleep(0.5)
else:
raise SystemExit('Shutdown incomplete; retain data and evidence and inspect only this owned instance.')
@@ -0,0 +1,124 @@
"""Check that verification helpers cannot take ownership of another checkout."""
import json
import os
import time
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest
import urllib.request
class OwnershipSafety(unittest.TestCase):
def test_foreign_checkout_server_remains_alive(self):
repo = Path(__file__).resolve().parents[4]
helpers = repo / '.agents/skills/verify-paperclip/scripts'
with tempfile.TemporaryDirectory(prefix='paperclip-ownership-test-') as temp:
root = Path(temp)
foreign = root / 'another-checkout'
foreign.mkdir()
server = subprocess.Popen(
[sys.executable, '-u', '-c',
"from http.server import HTTPServer, BaseHTTPRequestHandler\n"
"class Handler(BaseHTTPRequestHandler):\n"
" def do_GET(self):\n"
" self.send_response(200); self.end_headers(); self.wfile.write(b'foreign fixture alive')\n"
" def log_message(self, *args): pass\n"
"server = HTTPServer(('127.0.0.1', 0), Handler)\n"
"print(server.server_port, flush=True)\n"
"server.serve_forever()\n"],
cwd=foreign, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
text=True,
)
try:
port = int(server.stdout.readline().strip())
url = f'http://127.0.0.1:{port}'
data = root / 'verification-data'
instance = data / 'instances/default'
instance.mkdir(parents=True)
evidence = root / 'evidence'
evidence.mkdir()
# The claimed PID really owns this port, but its cwd is foreign.
(evidence / 'launch.json').write_text(json.dumps({
'repo': str(repo), 'dataDir': str(data), 'port': port,
'url': url, 'launcherPid': server.pid,
}))
(instance / 'runtime-info.json').write_text(json.dumps({
'pid': server.pid, 'port': port, 'dashboardUrl': url,
'host': '127.0.0.1',
}))
for helper in ['doctor.py', 'stop.py']:
with self.subTest(helper=helper):
result = subprocess.run(
[sys.executable, str(helpers / helper),
'--evidence-dir', str(evidence)],
capture_output=True, text=True, timeout=10,
)
self.assertNotEqual(result.returncode, 0)
self.assertIn('checkout', result.stdout + result.stderr)
self.assertIsNone(server.poll(), 'Helper killed foreign server')
with urllib.request.urlopen(url, timeout=5) as response:
self.assertEqual(response.read(), b'foreign fixture alive')
self.assertFalse((evidence / 'cleanup.json').exists())
finally:
server.terminate()
try:
server.communicate(timeout=5)
except subprocess.TimeoutExpired:
server.kill()
server.communicate()
def test_cleanup_waits_for_exit_after_listener_closes(self):
repo = Path(__file__).resolve().parents[4]
with tempfile.TemporaryDirectory(prefix='paperclip-delayed-exit-test-') as temp:
root = Path(temp)
instance = root / 'data/instances/default'
instance.mkdir(parents=True)
evidence = root / 'evidence'
evidence.mkdir()
marker = root / 'listener-closed'
server = subprocess.Popen(
[sys.executable, '-u', '-c',
"import signal, socket, sys, time\n"
"from pathlib import Path\n"
"listener = socket.socket(); listener.bind(('127.0.0.1', 0)); listener.listen()\n"
"def stop(signum, frame):\n"
" listener.close(); Path(sys.argv[1]).write_text('closed')\n"
" time.sleep(2); sys.exit(0)\n"
"signal.signal(signal.SIGTERM, stop)\n"
"print(listener.getsockname()[1], flush=True)\n"
"signal.pause()\n", str(marker)],
cwd=repo, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
)
try:
port = int(server.stdout.readline().strip())
url = f'http://127.0.0.1:{port}'
(evidence / 'launch.json').write_text(json.dumps({
'repo': str(repo), 'dataDir': str(root / 'data'),
'port': port, 'url': url, 'launcherPid': os.getpid(),
}))
(instance / 'runtime-info.json').write_text(json.dumps({
'pid': server.pid, 'port': port, 'dashboardUrl': url,
}))
started = time.monotonic()
result = subprocess.run(
[sys.executable, str(repo / '.agents/skills/verify-paperclip/scripts/stop.py'),
'--evidence-dir', str(evidence)],
capture_output=True, text=True, timeout=12,
)
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
self.assertTrue(marker.exists(), 'Fixture did not close its listener')
self.assertGreaterEqual(time.monotonic() - started, 2)
self.assertEqual(server.poll(), 0, 'Cleanup returned before runtime exit')
cleanup = json.loads((evidence / 'cleanup.json').read_text())
self.assertTrue(cleanup['runtimeExited'])
finally:
if server.poll() is None:
server.kill()
server.communicate(timeout=5)
if __name__ == '__main__':
unittest.main()
+153
View File
@@ -475,6 +475,42 @@ jobs:
# Restore only. Never let a pull request evict master's entry.
save-if: false
# The Runner's own crate is the one thing the entry above leaves out:
# rust-cache drops workspace crates when it saves, and a fresh checkout
# would invalidate cargo's mtime fingerprints for them anyway. So this
# job compiled paperclip-runner-core in release mode on every run,
# 28-49s (runs 38169436532 to 38183149920, 2026-10-11), while 221 of
# the 239 master commits of the preceding week left its inputs alone.
# The master Rust lane in release-verify.yml saves that crate's release
# artifacts (about 70 MB) under a key made from the compiler, the
# checkout root and a hash of every input the build reads. A pull
# request whose merge tree carries byte-identical inputs restores them
# and marks them newer than the checkout, so cargo keeps them; any
# other pull request misses and compiles exactly as before. Exact key
# only: a prefix match would hand cargo another revision's output. The
# path list and the key live in scripts/runner-release-crate-cache.mjs
# and must match the writer's; a guard test holds them together.
- name: Compute the Runner release crate cache key
id: runner_release_crate_key
run: echo "scope=$(node scripts/runner-release-crate-cache.mjs scope)" >> "$GITHUB_OUTPUT"
- name: Restore the Runner release crate (read only)
id: runner_release_crate
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-runner-core-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_runner_core-*
packages/paperclip-runner/runner/target/release/deps/paperclip_runner_core-*.d
packages/paperclip-runner/runner/target/release/deps/paperclip_runnerd-*
packages/paperclip-runner/runner/target/release/paperclip-runnerd
packages/paperclip-runner/runner/target/release/paperclip-runnerd.d
key: runner-release-crate-v1-${{ steps.runner_release_crate_key.outputs.scope }}-${{ hashFiles('packages/paperclip-runner/rust-toolchain.toml', 'packages/paperclip-runner/runner/Cargo.toml', 'packages/paperclip-runner/runner/Cargo.lock', 'packages/paperclip-runner/runner/crates/**', 'packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/test-fixtures/**', 'packages/paperclip-runner/test/fixtures/**', 'tests/runner-e2e/fixtures/**') }}
- name: Mark the restored Runner release crate current
if: ${{ steps.runner_release_crate.outputs.cache-hit == 'true' }}
run: node scripts/runner-release-crate-cache.mjs mark-current
- name: Typecheck workspaces whose build scripts skip TypeScript
run: pnpm run typecheck:build-gaps
@@ -749,10 +785,12 @@ jobs:
command: check:static
- lane_label: rust
command: check:runner
release_crate: true
- lane_label: vitest 1/2
command: test:typescript:vitest --shard=1/2
- lane_label: vitest 2/2
command: test:typescript:vitest --shard=2/2
release_crate: true
steps:
- name: Checkout repository
@@ -878,6 +916,49 @@ jobs:
# Restore only. Never let a pull request evict master's entry.
save-if: false
# The Runner's own crate is the one thing the entry above leaves out:
# rust-cache drops workspace crates when it saves, and a fresh checkout
# would invalidate cargo's mtime fingerprints for them anyway. So this
# job compiled paperclip-runner-core in release mode on every run,
# 28-49s (runs 38169436532 to 38183149920, 2026-10-11), while 221 of
# the 239 master commits of the preceding week left its inputs alone.
# The master Rust lane in release-verify.yml saves that crate's release
# artifacts (about 70 MB) under a key made from the compiler, the
# checkout root and a hash of every input the build reads. A pull
# request whose merge tree carries byte-identical inputs restores them
# and marks them newer than the checkout, so cargo keeps them; any
# other pull request misses and compiles exactly as before. Exact key
# only: a prefix match would hand cargo another revision's output. The
# path list and the key live in scripts/runner-release-crate-cache.mjs
# and must match the writer's; a guard test holds them together.
#
# Only the lanes that build the release crate restore it: the Rust
# lane's release tests link the lib, and the final vitest shard hosts
# the server's native-runner suite, which builds the binary in its
# beforeAll. The static and first vitest lanes run no cargo at all.
- name: Compute the Runner release crate cache key
id: runner_release_crate_key
if: ${{ matrix.release_crate == true }}
run: echo "scope=$(node scripts/runner-release-crate-cache.mjs scope)" >> "$GITHUB_OUTPUT"
- name: Restore the Runner release crate (read only)
id: runner_release_crate
if: ${{ matrix.release_crate == true }}
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-runner-core-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_runner_core-*
packages/paperclip-runner/runner/target/release/deps/paperclip_runner_core-*.d
packages/paperclip-runner/runner/target/release/deps/paperclip_runnerd-*
packages/paperclip-runner/runner/target/release/paperclip-runnerd
packages/paperclip-runner/runner/target/release/paperclip-runnerd.d
key: runner-release-crate-v1-${{ steps.runner_release_crate_key.outputs.scope }}-${{ hashFiles('packages/paperclip-runner/rust-toolchain.toml', 'packages/paperclip-runner/runner/Cargo.toml', 'packages/paperclip-runner/runner/Cargo.lock', 'packages/paperclip-runner/runner/crates/**', 'packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/test-fixtures/**', 'packages/paperclip-runner/test/fixtures/**', 'tests/runner-e2e/fixtures/**') }}
- name: Mark the restored Runner release crate current
if: ${{ steps.runner_release_crate.outputs.cache-hit == 'true' }}
run: node scripts/runner-release-crate-cache.mjs mark-current
- name: Verify Paperclip Runner
# pnpm appends trailing args to the end of the script's shell chain,
# so the vitest lanes' --shard lands on `vitest run`. Do not add a
@@ -1013,6 +1094,42 @@ jobs:
# Restore only. Never let a pull request evict master's entry.
save-if: false
# The Runner's own crate is the one thing the entry above leaves out:
# rust-cache drops workspace crates when it saves, and a fresh checkout
# would invalidate cargo's mtime fingerprints for them anyway. So this
# job compiled paperclip-runner-core in release mode on every run,
# 28-49s (runs 38169436532 to 38183149920, 2026-10-11), while 221 of
# the 239 master commits of the preceding week left its inputs alone.
# The master Rust lane in release-verify.yml saves that crate's release
# artifacts (about 70 MB) under a key made from the compiler, the
# checkout root and a hash of every input the build reads. A pull
# request whose merge tree carries byte-identical inputs restores them
# and marks them newer than the checkout, so cargo keeps them; any
# other pull request misses and compiles exactly as before. Exact key
# only: a prefix match would hand cargo another revision's output. The
# path list and the key live in scripts/runner-release-crate-cache.mjs
# and must match the writer's; a guard test holds them together.
- name: Compute the Runner release crate cache key
id: runner_release_crate_key
run: echo "scope=$(node scripts/runner-release-crate-cache.mjs scope)" >> "$GITHUB_OUTPUT"
- name: Restore the Runner release crate (read only)
id: runner_release_crate
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-runner-core-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_runner_core-*
packages/paperclip-runner/runner/target/release/deps/paperclip_runner_core-*.d
packages/paperclip-runner/runner/target/release/deps/paperclip_runnerd-*
packages/paperclip-runner/runner/target/release/paperclip-runnerd
packages/paperclip-runner/runner/target/release/paperclip-runnerd.d
key: runner-release-crate-v1-${{ steps.runner_release_crate_key.outputs.scope }}-${{ hashFiles('packages/paperclip-runner/rust-toolchain.toml', 'packages/paperclip-runner/runner/Cargo.toml', 'packages/paperclip-runner/runner/Cargo.lock', 'packages/paperclip-runner/runner/crates/**', 'packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/test-fixtures/**', 'packages/paperclip-runner/test/fixtures/**', 'tests/runner-e2e/fixtures/**') }}
- name: Mark the restored Runner release crate current
if: ${{ steps.runner_release_crate.outputs.cache-hit == 'true' }}
run: node scripts/runner-release-crate-cache.mjs mark-current
- name: Build Runner Evalbook viewer
run: pnpm --filter @paperclipai/paperclip-runner build:issue-thread
@@ -1259,6 +1376,42 @@ jobs:
# Restore only. Never let a pull request evict master's entry.
save-if: false
# The Runner's own crate is the one thing the entry above leaves out:
# rust-cache drops workspace crates when it saves, and a fresh checkout
# would invalidate cargo's mtime fingerprints for them anyway. So this
# job compiled paperclip-runner-core in release mode on every run,
# 28-49s (runs 38169436532 to 38183149920, 2026-10-11), while 221 of
# the 239 master commits of the preceding week left its inputs alone.
# The master Rust lane in release-verify.yml saves that crate's release
# artifacts (about 70 MB) under a key made from the compiler, the
# checkout root and a hash of every input the build reads. A pull
# request whose merge tree carries byte-identical inputs restores them
# and marks them newer than the checkout, so cargo keeps them; any
# other pull request misses and compiles exactly as before. Exact key
# only: a prefix match would hand cargo another revision's output. The
# path list and the key live in scripts/runner-release-crate-cache.mjs
# and must match the writer's; a guard test holds them together.
- name: Compute the Runner release crate cache key
id: runner_release_crate_key
run: echo "scope=$(node scripts/runner-release-crate-cache.mjs scope)" >> "$GITHUB_OUTPUT"
- name: Restore the Runner release crate (read only)
id: runner_release_crate
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-runner-core-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_runner_core-*
packages/paperclip-runner/runner/target/release/deps/paperclip_runner_core-*.d
packages/paperclip-runner/runner/target/release/deps/paperclip_runnerd-*
packages/paperclip-runner/runner/target/release/paperclip-runnerd
packages/paperclip-runner/runner/target/release/paperclip-runnerd.d
key: runner-release-crate-v1-${{ steps.runner_release_crate_key.outputs.scope }}-${{ hashFiles('packages/paperclip-runner/rust-toolchain.toml', 'packages/paperclip-runner/runner/Cargo.toml', 'packages/paperclip-runner/runner/Cargo.lock', 'packages/paperclip-runner/runner/crates/**', 'packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/test-fixtures/**', 'packages/paperclip-runner/test/fixtures/**', 'tests/runner-e2e/fixtures/**') }}
- name: Mark the restored Runner release crate current
if: ${{ steps.runner_release_crate.outputs.cache-hit == 'true' }}
run: node scripts/runner-release-crate-cache.mjs mark-current
# `release.sh` always executes its Step 2/7 workspace build, even when
# `--skip-verify` bypasses the initial verification gate. release.sh
# also requires a clean working tree, and the install step may have
+55
View File
@@ -414,6 +414,61 @@ jobs:
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
run: pnpm --filter @paperclipai/paperclip-runner build:rust
# The dependency cache above deliberately leaves the Runner's own crate
# out, so every pull request build compiled paperclip-runner-core in
# release mode (28-49s) although most pushes leave its inputs alone.
# This master push saves that crate's release artifacts for the readers
# in pr-trusted.yml ("Restore the Runner release crate") under a key
# made from the compiler, the checkout root and a hash of every input
# the build reads, so a reader only ever restores output of inputs
# byte-identical to its own. Same writer gate as the dependency cache:
# the Rust lane of an own-repository master push. The entry is looked
# up first; a push that left the inputs alone already has it. On a
# miss the crate is cleaned and built with the exact command the
# readers run, so the artifacts carry no test-profile variants. The
# path list and the key live in scripts/runner-release-crate-cache.mjs
# and must match the readers'; a guard test holds them together.
- name: Compute the Runner release crate cache key
id: runner_release_crate_key
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
run: echo "scope=$(node scripts/runner-release-crate-cache.mjs scope)" >> "$GITHUB_OUTPUT"
- name: Check for this push's Runner release crate
id: runner_release_crate
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
lookup-only: true
path: |
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-runner-core-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_runner_core-*
packages/paperclip-runner/runner/target/release/deps/paperclip_runner_core-*.d
packages/paperclip-runner/runner/target/release/deps/paperclip_runnerd-*
packages/paperclip-runner/runner/target/release/paperclip-runnerd
packages/paperclip-runner/runner/target/release/paperclip-runnerd.d
key: runner-release-crate-v1-${{ steps.runner_release_crate_key.outputs.scope }}-${{ hashFiles('packages/paperclip-runner/rust-toolchain.toml', 'packages/paperclip-runner/runner/Cargo.toml', 'packages/paperclip-runner/runner/Cargo.lock', 'packages/paperclip-runner/runner/crates/**', 'packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/test-fixtures/**', 'packages/paperclip-runner/test/fixtures/**', 'tests/runner-e2e/fixtures/**') }}
- name: Build the Runner release crate for its cache
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha && steps.runner_release_crate.outputs.cache-hit != 'true' }}
working-directory: packages/paperclip-runner
run: |
set -euo pipefail
cargo clean --release --manifest-path runner/Cargo.toml -p paperclip-runner-core
pnpm build:binary
- name: Save the Runner release crate
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha && steps.runner_release_crate.outputs.cache-hit != 'true' }}
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-runner-core-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_runner_core-*
packages/paperclip-runner/runner/target/release/deps/paperclip_runner_core-*.d
packages/paperclip-runner/runner/target/release/deps/paperclip_runnerd-*
packages/paperclip-runner/runner/target/release/paperclip-runnerd
packages/paperclip-runner/runner/target/release/paperclip-runnerd.d
key: runner-release-crate-v1-${{ steps.runner_release_crate_key.outputs.scope }}-${{ hashFiles('packages/paperclip-runner/rust-toolchain.toml', 'packages/paperclip-runner/runner/Cargo.toml', 'packages/paperclip-runner/runner/Cargo.lock', 'packages/paperclip-runner/runner/crates/**', 'packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/test-fixtures/**', 'packages/paperclip-runner/test/fixtures/**', 'tests/runner-e2e/fixtures/**') }}
build:
name: Build
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
+35
View File
@@ -560,6 +560,41 @@ falls back to the fresh path automatically, so the cache can make a suite
slower but never fail it or skip it: a migration that cannot apply still fails
the suite on the fresh path.
#### Reused Runner build in the server build
The server build vendors `packages/paperclip-runner/dist`, so its
`prepare:runner-vendor` step has to make sure that directory is a current
build. It used to run the Runner's full `build` every time. Now the Runner's
`build` brackets itself with `scripts/record-build-inputs.mjs`: `--begin`, its
first step, forgets the previous stamp and records a digest of the build's
inputs (every file under `packages/paperclip-runner` except `dist`,
`dist-browser`, `runner/target`, `node_modules` and dot-directories; every
file under `packages/paperclip-eval-kernel` except `dist` and `node_modules`;
every file under the root `patches` directory, which the acpx profile check
hashes and the provider entrypoint bundler copies into `dist`; and the root
`package.json`, `pnpm-lock.yaml`, `pnpm-workspace.yaml` and
`tsconfig.base.json`). `--finish`, its last step, recomputes the digest and
writes the stamp
`packages/paperclip-runner/node_modules/.cache/paperclip-runner-build-inputs.json`
only if the inputs are still the ones the build began with, together with the
path and size of every file under `dist`. An edit that lands while the build
runs leaves no stamp, so the next consumer rebuilds.
`node server/scripts/prepare-runner-vendor.mjs` reuses the build when the
stamp is for this platform, every recorded output is still there at its
recorded size, and the current digest matches. Anything else (no stamp, a
changed input, a deleted or truncated output, a stamp from another platform)
runs `pnpm --filter @paperclipai/paperclip-runner build` exactly as before, so
a standalone `pnpm --filter @paperclipai/server build` on a fresh checkout
still builds the Runner itself. Inside `pnpm build` the Runner has just been
built by the same command, and the step skips a 12-16s rebuild. The step
prints one `paperclip.runner-vendor.timing.v1` line saying which it did.
Set `PAPERCLIP_RUNNER_VENDOR_REBUILD=1` to rebuild regardless of the stamp. A
stamp is only written after a Runner build that passed every step, and both
the Runner's first build step and the prepare step remove any existing stamp
before a rebuild starts, so a build that stops partway never leaves a stamp
that vouches for it.
### Task search evaluation
The task search relevance rubric and regression corpus are documented in
+53 -2
View File
@@ -363,14 +363,65 @@ paths and the PR lanes can restore master's entry. The guard tests under
`.github/scripts/tests/` require this step, with identical text, before every
Rust cache step in both workflows.
Workspace crates and installed Cargo binaries are excluded. Every run rebuilds
workspace code and runs all assigned checks, including on a cache hit. Only an
Workspace crates and installed Cargo binaries are excluded from this entry.
Every run runs all assigned checks, including on a cache hit. Only an
own-repository master-push run verifying that push's exact SHA can restore the
cache, and only a successful Rust lane saves it. PR, tag, and manual candidate
verification compile without this cache. A miss or eviction costs compilation
time but does not change the checks. To discard old dependency caches, increment
the shared-key version and let the next successful master verification warm it.
### Runner release crate cache
The dependency entry leaves the Runner's own crate out, and a fresh checkout
invalidates cargo's mtime fingerprints for workspace crates anyway, so every
pull request build compiled `paperclip-runner-core` in release mode from
scratch (28-49s per job on 2026-10-11) although most master commits leave its
inputs untouched. A second, content-addressed entry holds only that crate's
release artifacts: its fingerprints, the lib's rlib and rmeta, the
`paperclip-runnerd` binary and their dep-info, about 70 MB. The path list and
the key inputs live in `scripts/runner-release-crate-cache.mjs`, which both
workflows call, and `scripts/__tests__/runner-release-crate-cache.test.mjs`
holds the readers and the writer to them.
The key is `runner-release-crate-v1-<scope>-<hashFiles>`. The scope names the
platform, the pinned compiler (`rustc --version --verbose`: release, commit and
host) and a hash of the checkout root, because cargo's dep-info records the
protocol files the crate includes by relative path as absolute paths. The
`hashFiles` part covers every input the build reads: the toolchain pin, the
manifests and lockfile, the crate sources, the protocol schemas and fixtures,
and the three fixture trees the test modules include (two under the Runner
package and `tests/runner-e2e/fixtures` at the repository root). Any change
to any of them selects a new key. A guard test resolves every `include_str!`
and `include_bytes!` in the crate, whether its argument is a literal or a
`concat!` of literals, `env!("CARGO_MANIFEST_DIR")` and macro metavariables,
and fails on an include that lands outside these patterns or uses a form it
cannot resolve. Readers pass no `restore-keys`: a prefix match would hand
cargo another revision's output.
The writer is the Rust lane of the same master-push verification, after its
checks and the debug warm-up. It looks the key up first and, on a miss, runs
`cargo clean --release -p paperclip-runner-core` and then the package's
`build:binary` (the exact command the readers run) before saving, so the entry
carries no test-profile variants. In the trusted PR workflow, the Typecheck,
Build and Canary Dry Run jobs and the two Verify Paperclip Runner lanes that
build the crate (`rust` and the final vitest shard) restore the entry
read-only right after the dependency restore. On a hit, `mark-current` sets
the mtime of every restored artifact to now, so cargo's freshness check sees
every source as older than the build and keeps the artifacts. A restore that
lacks any file cargo reads or checks for either unit (the fingerprint hash
and dep-info of each unit, the lib's rlib and rmeta, the binary under
`deps`; measured with cargo 1.97.1, which recompiles without any one of
them and recreates the uplifted binary without any of the rest) is removed
instead, the job log names the missing file, and cargo compiles as on a
miss. `pnpm build`
then prints `Finished release profile` in well under a second for that crate.
On a miss nothing changes: cargo compiles the crate as before.
The trust boundary is unchanged: only master writes, a pull request can only
restore master's entry, and it restores only artifacts built from inputs
byte-identical to its own merge tree.
The trust boundary is the protected master branch, not the cache-key text.
GitHub does not let master restore caches created by a child branch, sibling
branch, tag, or PR merge ref. Both permitted restore scopes (current branch and
+10 -4
View File
@@ -9,7 +9,7 @@ Product behavior is still governed by [the implementation spec](../doc/SPEC-impl
The map is organized by user capability, not by source file. It covers **35 feature
families** across the product, CLI, and operator workflows, checked against source
on 2026-10-05. Experimental and developer-only surfaces are labeled explicitly.
on 2026-10-12. Experimental and developer-only surfaces are labeled explicitly.
Each recipe includes sub-features, current entry points, automated evidence,
manual verification steps, and gotchas.
@@ -113,7 +113,10 @@ fixture integration. Record executed results separately from the map.
persisted state or continuation. Use read-only API checks to corroborate UI
evidence; creating state through the API does not prove the creation UI.
For agent-driven acceptance work, the existing
For agent-driven acceptance work, start with the
[Paperclip verification skill](../.agents/skills/verify-paperclip/SKILL.md) for an
isolated launch, ownership/readiness checks, browser driving, evidence, and cleanup.
It uses this directory as its feature map. The existing
[dev-workspace run/verify skill](../.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md)
and [evaluation skill](../.agents/skills/paperclip-evals/SKILL.md) describe runtime
ownership and evidence handling. Reuse them; this map introduces no environment
@@ -168,10 +171,13 @@ Each recipe starts with an H1 and a user-visible description, then exactly:
1. **Sub-features** — stable backticked IDs and observable behavior/states.
2. **How to get to it (user POV)** — one H3 backticked entry-point ID per surface.
3. **Driving it** — starts with `Preconditions:`; repeat each entry-point H3,
3. **Driving it with <harness>** — starts with `Preconditions:`; repeat each entry-point H3,
with `Automated:` and `Manual:` paragraphs. Name test scope and gaps honestly.
4. **Gotchas** — misleading look-alikes, feature gates, and invalid evidence.
The format is inspired by
New or updated entries follow
[Poteto's feature-map examples](../.agents/skills/create-verification-skill/references/feature-map-example/).
Existing entries with a `Driving it` heading can adopt the named harness when
their recipes are updated. The original format was inspired by
[Omnigent's feature map](https://github.com/omnigent-ai/omnigent/tree/91acfbbb59f6fc210ff95a9e9428aadd62e06582/feature-map).
Paperclip's recipes and checks follow its own task model and test infrastructure.
+8 -1
View File
@@ -21,12 +21,15 @@ Open `/auth`; first-owner bootstrap uses `/board-claim/:token` when configured.
### `invite-members`
Open `/invite/:token`; operators manage members in company settings and join requests at `/inbox/requests`.
Create/revoke human links at Members → Invites
(`/company/settings/members?tab=invites`). Human invitees receive the chosen
role after sign-in; agent invitations still use join-request approval.
### `cli-authorization`
Start the CLI auth/connect flow and approve its `/cli-auth/:id` browser page; instance access is under Settings when visible.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). For login tests use authenticated mode and two test users; local-trusted mode does not prove session enforcement. Use two companies for isolation checks.
@@ -48,6 +51,10 @@ Automated: [board auth client](../cli/src/__tests__/board-auth.test.ts) and [tok
Manual: Authorize the intended CLI context, make a read-only company request, then revoke its token and retry. Verify failure is explicit and does not silently fall back to another identity.
Use `auth login`, `auth whoami`, and `auth revoke-current` with explicit context.
Retry with the same bearer to prove revocation; deleting local credentials with
logout alone does not establish that server revocation succeeded.
## Gotchas
- Settings visibility is not authorization; direct requests still need permission checks.
+6 -1
View File
@@ -22,7 +22,7 @@ Open `/dashboard` or `/dashboard/live`.
Open `/activity`, `/activity/runs`, or `/activity/timeline` in the streamlined shell; legacy routes may redirect.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Prepare known task mutations and completed/failed runs in a disposable company, recording their IDs and actors.
@@ -38,6 +38,11 @@ Automated: [activity service](../server/src/__tests__/activity-service.test.ts),
Manual: Find a known mutation by actor/resource and a known run by status. Change filters, load more, and follow the source links. Reload the chosen view and verify no company leakage or duplicate events. Inspect a routine run from its audit entry.
Audit-event Load more and run listings differ: Runs currently loads at most 200
entries without pagination. The tested RoutineAuditActivity component has no
registered activity-route host; inspect execution links actually offered by
the routine rather than treating that component test as a reachable journey.
## Gotchas
- Audit/activity events, run-log rows, first-party Telemetry, and operator OpenTelemetry traces are separate data paths.
+9 -2
View File
@@ -25,7 +25,7 @@ Ask for project work in Agent Chat and follow its created project/task links.
Open `/board-chat` only when its conference-room gate is enabled.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Enable the applicable chat feature, use a configured test agent, and distinguish model replies from a mocked conversation fixture.
@@ -33,7 +33,10 @@ Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Enab
Automated: [chat landing](../ui/src/pages/AgentChats.test.tsx) checks recent-chat routing; [conversation service](../server/src/__tests__/agent-conversations.test.ts) covers persisted conversation behavior.
Manual: Start a conversation, send a recognizable request, and inspect its reply. Navigate away and reopen the same history. Test a new conversation, unavailable historical agent, and an inaccessible saved conversation; verify no cross-user or cross-company history appears.
Manual: Start a conversation, send a recognizable request, and inspect its reply.
Navigate away and reopen the same history. Use `/new` for a fresh session within
the same standing conversation. Test an unavailable historical agent and an
inaccessible saved conversation; verify no cross-user or cross-company history appears.
### `project-handoff`
@@ -45,6 +48,10 @@ Manual: Ask the configured agent to create bounded project work, inspect the pro
Automated: [board chat gate](../server/src/__tests__/board-chat-route-feature-flag.test.ts) proves route gating, not a live conversation.
Preconditions for this entry point: `local_trusted` mode and an installed,
authenticated `claude` CLI, in addition to the Conference Room flag. These are
separate from Agent Chat's configured-agent prerequisites.
Manual: Check disabled access first, enable in the disposable instance, then send and reload a conversation. Inspect the actual responding agent and persisted history; report this result separately from Agent Chat.
## Gotchas
+8 -2
View File
@@ -6,7 +6,8 @@ Implementation: [hiring](../ui/src/pages/NewAgent.tsx), [configuration](../ui/sr
## Sub-features
- `hire`: choose role, manager, adapter, model, and required account settings.
- `hire`: choose identity, adapter, model, and required account settings; edit
reporting relationships through the agent configuration afterward.
- `instructions`: edit agent configuration and instruction files with durable revisions.
- `organization`: inspect and change reporting relationships through supported controls.
- `lifecycle`: pause/resume or retire an agent while preserving history.
@@ -26,7 +27,7 @@ Open `/agents/:agentId` and the configuration/instruction surfaces it exposes.
Use the roster’s organization view; `/org` redirects to the roster in the streamlined shell.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a disposable company and supported adapter; hiring may require a formal approval.
@@ -36,6 +37,11 @@ Automated: [new agent page](../ui/src/pages/NewAgent.test.tsx) and [configuratio
Manual: Hire an agent with a distinct name and role. Complete any required approval, reload the roster, and open its configuration. Confirm manager, adapter, and model persisted; run a small task to prove execution readiness.
The current hiring wizard assigns the first hire CEO and subsequent hires
General reporting to the CEO. Change manager/role afterward through supported
configuration. Process/HTTP agents can be created through API/CLI but are excluded
from the hiring picker; API fixture creation does not prove this UI journey.
### `agent-configuration`
Automated: [instruction service](../server/src/__tests__/agent-instructions-service.test.ts) and [instruction revisions](../server/src/__tests__/agent-instruction-revisions.test.ts) cover file/revision contracts.
+5 -2
View File
@@ -21,7 +21,7 @@ Use Activity → Costs/Budgets in the streamlined shell, `/costs` in the product
Open the governing budget configuration and any resulting budget incident card.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use synthetic low-cost events and small disposable budgets. Do not buy model usage merely to hit a threshold.
@@ -29,7 +29,10 @@ Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use
Automated: [costs service](../server/src/__tests__/costs-service.test.ts) covers cost contracts; [embedded costs UI](../ui/src/pages/Costs.test.tsx) has narrow Audit-host coverage.
Manual: Inspect a known fixture run and reconcile its recorded spend with the selected report period and resource. Switch company/project filters and reload. Check empty and unavailable cost data without treating missing data as zero spend.
Manual: Inspect a known fixture run and reconcile recorded spend with the chosen
period and resource. Switch the report period/company and inspect the **By
project** breakdown; it is not a project-selector filter. Reload and check empty
or unavailable cost data without treating missing data as zero spend.
### `budget-enforcement`
+9 -3
View File
@@ -21,7 +21,7 @@ Open `/cases` and switch between available grouped/tree views.
Open `/cases/:caseIdentifier` or a task’s linked case.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Enable cases and prepare related records with different types/statuses plus an associated task.
@@ -35,9 +35,15 @@ Manual: Search for a child, verify its parent context remains understandable, ch
Automated: [case routes](../server/src/__tests__/cases-routes.test.ts) covers API behavior; [case fields](../ui/src/components/CaseFieldsPanel.test.tsx) covers field UI separately.
Manual: Edit an offered field, reload, inspect its revision/activity, and follow child and task links. Verify cross-company and invalid field changes fail clearly. Distinguish a pipeline-owned item from a stand-alone case.
Manual: Change status or labels, reload, inspect activity, and follow child/task
links. Edit a case document and inspect its document revisions separately.
Arbitrary JSON fields are currently displayed without a general field editor.
Verify cross-company and invalid changes fail clearly. Distinguish a
pipeline-owned item from a stand-alone case.
## Gotchas
- A case is not a task execution; inspect linked tasks to prove work occurred.
- Feature gating and case-type schema determine which actions are available.
- Feature gating and offered controls determine available actions. The API
accepts a string case type and arbitrary JSON fields; do not infer typed
case-field validation or a field-editing UI from those values.
+9 -2
View File
@@ -22,9 +22,13 @@ Message the configured bot/address from its external provider and follow the cor
Open `/apps/chat/:endpointId/settings` and the endpoint’s other available tabs; identity confirmation uses `/chat-identity/confirm`.
## Driving it
## Driving it with Playwright and provider clients
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Enable chat connectors and use a test provider account/channel with permission to send messages. Setup is covered in [connection setup](./connection-setup.md).
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a
test provider account/channel with permission to send messages. Gates differ:
AgentMail bypasses the chat-connectors UI gate, GitHub uses
`enableGitHubReviewBots`, and other chat providers use `enableChatConnectors`.
Setup is covered in [connection setup](./connection-setup.md).
### `external-thread`
@@ -40,6 +44,9 @@ Manual: Inspect the bound agent and identity, recent activity, and delivery fail
## Gotchas
- Visibility flags do not pause existing delivery. Disable the disposable
endpoint itself to test stopped traffic.
- One provider’s passing fixture does not qualify another provider or live delivery.
- An accepted send request does not prove the recipient received the message or file.
- Use [steering](./steering.md) and [questions](./questions-and-approvals.md) for their channel-specific interaction checks.
+6 -1
View File
@@ -25,10 +25,15 @@ Use issue subresource, asset/skill, and run/heartbeat commands exposed by the CL
### `worktree-instance`
Use the documented worktree/test-drive commands in an owned checkout; inspect `paperclipai worktree --help` first.
Inspect root help too: `worktree:make/list/merge-history/cleanup` are top-level
commands rather than subcommands of `worktree`.
## Driving it
## Driving it with the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use `--help` from the checked-out CLI to discover exact arguments. Select a disposable company and isolated context; do not inherit an unknown API target.
Pass the server origin to `--api-base` (without appending `/api`); the commands
already include the API path. `issue create` requires explicit `--company-id`
even when a profile stores a company. A persona label is not authentication proof.
### `client-context`
+5 -1
View File
@@ -21,7 +21,7 @@ Use the company switcher or `/companies`, then enter company Settings.
Use `/company/export` and `/company/import` where enabled, or the corresponding CLI company commands.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use two disposable companies and a small package fixture. Record cloud-managed restrictions and visible settings gates.
@@ -39,6 +39,10 @@ Manual: Export a small company, inspect the package preview, and import into a d
## Gotchas
- Restore an archived company through `/companies` → its menu → **Unarchive**.
Archiving pauses eligible agents and cancels active runs; restoration handles
agents paused specifically for company archival.
- Package presence does not prove a runnable imported company; credentials and environment bindings need separate verification.
- Company deletion is a distinct gated operation; do not use it as the archive test.
- Settings hidden in a managed deployment should be recorded as unavailable, not bypassed.
+5 -1
View File
@@ -57,7 +57,7 @@ or open `/apps/chat/connect`. Complete that provider's endpoint setup, then
open its settings (`/apps/chat/:endpointId/settings`). Follow identity linking
as the intended sender before messaging the agent externally.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey).
Choose a disposable provider account/resource and record its auth method,
@@ -138,6 +138,10 @@ Slack setup does not qualify Discord, Telegram, GitHub, email, or iMessage.
## Gotchas
- Private endpoint rejection depends on deployment: owned loopback MCP servers
are supported in local-trusted/private instances. Public authenticated
deployments apply the private-network restriction.
- A successful OAuth callback is not proof of saved connection access or task
continuation. Verify all three checkpoints.
- Personal identity, shared account access, and per-agent access are distinct.
+7 -2
View File
@@ -21,7 +21,7 @@ Use `/design-guide`, registered `/ux-lab/...` routes, or Storybook; consult `ui/
Open `/tests/perf/long-thread` only in an appropriate development environment.
## Driving it
## Driving it with Playwright
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a development/test environment. Confirm the route is registered for the current build before claiming the page is available.
@@ -35,9 +35,14 @@ Manual: Open the specific fixture state, inspect controls and layout, then repea
Automated: [task thread component](../ui/src/components/IssueChatThread.test.tsx) does not establish a performance budget. Timing/scroll measurements need a dedicated browser run.
Manual: Record browser/build/data size, exercise scrolling and composing in the fixture, and retain timing or trace evidence. Confirm a representative real task does not regress before extrapolating fixture measurements.
Manual: Record browser/build/data size, exercise scrolling and rendering in the
long-thread fixture, and retain timing or trace evidence. Its composer is
disabled; test composing separately in a real task. Confirm representative
real-task behavior before extrapolating fixture measurements.
## Gotchas
- An unmounted or obsolete lab file remains in the conservative page inventory but is not a shipped entry point.
- Intended developer use is not a universal runtime gate: Design Guide and
global lab/performance routes are also registered in production builds.
- Screenshots of a lab do not prove live provider behavior, persistence, or authorization.
+12 -3
View File
@@ -21,12 +21,15 @@ Open a document from the task thread or document side panel.
### `task-output`
Open an attachment, output card, or file link from a task.
For direct uploads, use the task side panel's **Open a new tab → Artifacts →
Upload files** or drop files anywhere in that panel. On mobile, open
**More actions → Properties** first. Agent conversations do not expose this upload path.
### `artifact-library`
Open `/artifacts` and its grouping, search, and media controls.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a task with a text document, a small attachment, and a generated work product. Use nonsensitive fixture files.
@@ -38,9 +41,15 @@ Manual: Edit a paragraph, save, reload, and compare revisions. Add an annotation
### `task-output`
Automated: [attachment routes](../server/src/__tests__/issue-attachment-routes.test.ts) and [work products](../server/src/__tests__/work-products.test.ts) cover data/access behavior; [artifact arrival browser suite](../tests/e2e/artifact-tab-arrival.spec.ts) targets the arrival experience.
Automated: [attachment routes](../server/src/__tests__/issue-attachment-routes.test.ts) and [work products](../server/src/__tests__/work-products.test.ts) cover data/access behavior; [artifact arrival browser suite](../tests/e2e/artifact-tab-arrival.spec.ts) targets arrival. [Artifact uploads](../tests/e2e/artifact-uploads.spec.ts) drives file selection/drop, filename collisions, retry, and preservation of paused/unassigned tasks.
Manual: Upload a fixture and have the test task produce an output. Preview and download each, compare the contents, and follow the source task link. Reload and test an inaccessible user/company; verify missing workspace-only resources have an explicit state.
Manual: Upload a small Markdown file and CSV through Artifacts. Markdown becomes
a versioned task document; other files become attachments. Read the document,
download the CSV and compare bytes, then reload. Upload the same Markdown filename
again and verify a separate suffixed document without overwriting the original.
On a paused unassigned task, confirm upload does not start work or add comments.
Verify agent-produced work products separately using an eligible test run.
Test an inaccessible user/company and explicit missing workspace-only resources.
### `artifact-library`
+9 -1
View File
@@ -17,12 +17,16 @@ Implementation: [environment settings](../ui/src/pages/CompanyEnvironments.tsx),
### `environment-settings`
Open Settings → Environments, then create/edit a supported target.
The canonical URL is `/company/settings/instance/environments`. Enable
`enableEnvironments` first; keep `enableManagedSandboxOnly` off to inspect local
targets. The catalog is instance-scoped and mutation requires a local operator
or instance administrator.
### `remote-execution`
Choose the environment in agent/project/workspace configuration, then inspect the resulting task/run and workspace.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use an owned disposable target and a supported provider plugin. Record provider, target identity, account, and spending constraints before a live remote run.
@@ -32,6 +36,10 @@ Automated: [environment UI](../ui/src/pages/CompanyEnvironments.test.tsx), [driv
Manual: Save a disposable target, reload it, and run its available probe. Verify both success and missing-credential/reachability errors identify the target. Confirm unavailable providers are not offered as working run targets.
The local row has no UI Test button. Its API/CLI probe reports host/cwd
availability only; it does not validate an adapter binary, workspace, or task
execution. Remote-driver probes and a real bounded run need their own evidence.
### `remote-execution`
Automated: [execution target](../server/src/__tests__/environment-execution-target.test.ts) and [custom images](../server/src/__tests__/environment-custom-images-service.test.ts) cover contracts; [SSH live test](../server/src/__tests__/environment-live-ssh.test.ts) requires its own external setup.
+7 -2
View File
@@ -6,7 +6,7 @@ Implementation: [gateways](../ui/src/pages/apps/gateways/GatewaysList.tsx), [gat
## Sub-features
- `gateway`: create/edit an endpoint and select its exposed apps/tools.
- `gateway`: create/edit an endpoint backed by a tool access profile.
- `profiles`: define reusable tool access configuration and inspect draft/detail state.
- `client-connect`: copy the intended endpoint/configuration to an authorized test client.
- `tokens`: create/revoke client credentials and verify their scope.
@@ -22,10 +22,15 @@ Open `/apps/gateways`, then a gateway’s Overview, Apps/Tools, Tokens, and Acti
Open `/apps/advanced/profiles`, create a profile, or follow its detail/edit route.
## Driving it
## Driving it with Playwright and an MCP client
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a fixture tool connection and test client. Record gateway/profile, token owner, transport, and selected company.
Gateway creation selects a profile. Maintain that profile's app/action selection
through the Apps/Tools links; the gateway form is not a separate tool picker.
Use an active organization member for human-scoped access, and check the actual
endpoint authorization independently of whether the admin UI is visible.
### `gateway-client`
Automated: [gateway service](../server/src/__tests__/tool-gateway-service.test.ts), [connect client dialog](../ui/src/pages/apps/gateways/ConnectClientDialog.test.tsx), and [token panel](../ui/src/pages/apps/gateways/panels/TokensPanel.test.tsx) cover service/UI contracts.
+7 -2
View File
@@ -20,7 +20,7 @@ Open `/goals`, create a goal, and enter `/goals/:goalId`.
Inspect the goal context from a project or task and compare explicit versus inherited context.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Prepare two goals and a disposable project/task in the selected company.
@@ -28,7 +28,12 @@ Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Prep
Automated: [project-goal validation](../server/src/__tests__/project-goal-validation.test.ts) covers relationship validation; [goal detail control](../ui/src/pages/GoalDetail.test.tsx) only tests the properties toggle, not the whole creation journey.
Manual: Create a parent and child goal, save owner/status changes, and reload. Link a project and task through the offered controls, follow their goal context, and verify a cross-company or invalid relationship is rejected.
Manual: Choose **Add Goal/New Goal**, enter **Goal title**, select the level, and
create a parent. Open it and choose **Sub Goal** to create a child with the
parent preselected. Change status in Properties and reload; follow the Parent
Goal link and inspect Sub-Goals. Owner is currently display-only in this UI.
Link a project and task through the offered controls, follow their goal context,
and verify a cross-company or invalid relationship is rejected.
### `task-goal-context`
+5 -1
View File
@@ -25,7 +25,7 @@ Open `/decisions`, then a queue at `/decisions/queues/:key` and its source item.
Open `/search` or the navigation search entry.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Seed assigned, unread, blocked, and resolved tasks plus a pending decision. Record the combined Inbox/Tasks experimental setting.
@@ -49,6 +49,10 @@ Manual: Search a unique task title and content phrase, open the result, and comp
## Gotchas
- Combined Inbox/Tasks requires both `enableCombinedInboxTasks` and
`enableStreamlinedUi`. [Standalone Inbox tests](../ui/src/pages/Inbox.test.tsx)
render archive/read controls and toolbars; this is not live multi-user evidence.
- Personal archival is not task cancellation or completion.
- Legacy Inbox routes can redirect to the combined task view; record which host actually rendered.
- Search service tests do not establish the relevance quality of every query.
+6 -1
View File
@@ -22,7 +22,7 @@ Use `paperclipai install`, `run`, `service`, `doctor`, `configure`, `channels`,
Use currently visible instance settings for experiments/access and `paperclipai db:backup --help` for one-off backup. Some legacy general-settings URLs redirect.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a disposable data directory/service identity. Do not update, restart, or restore over an unrelated live instance as a documentation test.
@@ -40,6 +40,11 @@ Manual: Change a harmless experimental setting, reload, and confirm the affected
## Gotchas
- Source-checkout update/check commands report manual update guidance; they do
not qualify a managed installation. Rollback restores retained CLI payloads,
not database migrations. Service status applies to its selected service
identity, not automatically to an owned foreground dev server.
- A successful backup write does not prove restoration works.
- Settings visibility and mutability differ for cloud-managed and self-hosted instances.
- Keep the three data paths distinct: first-party Telemetry, operator-configured OpenTelemetry, and local run logs.
+6 -1
View File
@@ -22,7 +22,7 @@ Use starred/recent/sidebar links and direct company-prefixed URLs.
Use Settings → Profile, `/u/:userSlug`, and the announcement well when a publication is available.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use two test users and companies; record the selected UI shell and any client-local preference storage.
@@ -32,6 +32,11 @@ Automated: [sidebar](../ui/src/components/Sidebar.test.tsx) and [sidebar prefere
Manual: Star and reorder supported resources, collapse a section, navigate and reload. Switch company/user and inspect the intended preference scope. Open an unavailable resource and a legacy route; verify clear fallback or correct redirect without leaking another company’s state.
Record persistence separately: section collapse is in-memory and reopens after
reload; global sidebar collapse is disabled. Streamlined starred projects sort
alphabetically. Recents are browser-local by user/company; stars and supported
ordering are saved server-side. Local-trusted mode does not prove two-user isolation.
### `profile-announcements`
Automated: [profile page](../ui/src/pages/ProfileSettings.test.tsx), [announcement hook](../ui/src/hooks/useAnnouncement.test.tsx), and [announcement routes](../server/src/__tests__/announcements-routes.test.ts) cover separate UI/persistence layers.
+18 -2
View File
@@ -17,12 +17,15 @@ Implementation: [web wizard](../ui/src/components/OnboardingWizard.tsx), [CLI se
### `web-wizard`
Open `/onboarding` or the onboarding launcher in an empty company context.
Use `/<company-prefix>/onboarding` to resume an existing company's agent setup;
the unprefixed route starts at organization naming on self-hosted instances.
Cloud-managed instances reuse their existing company for agent setup.
### `cli-setup`
Use `paperclipai onboard --help` or `paperclipai run --help` to select an isolated data directory before setup.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a new isolated data directory and record deployment mode. A configured adapter may still need account authentication.
@@ -30,7 +33,20 @@ Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use
Automated: [wizard tests](../ui/src/components/OnboardingWizard.test.tsx) cover mocked wizard behavior; [onboarding browser suite](../tests/e2e/onboarding.spec.ts) supplies its own test environment. Live model execution is a separate check.
Manual: Create a disposable company and first agent. Save and leave midway, reopen, and finish. Follow the created first task, verify the selected agent and company, and inspect its output or explicit authentication gate. Reload both company and task.
- **Create company and resume.** Start a fresh empty instance and run doctor using
the [verification skill](../.agents/skills/verify-paperclip/SKILL.md). Run
`node .agents/skills/verify-paperclip/scripts/onboarding-smoke.mjs --evidence-dir "$VERIFY_EVIDENCE_DIR"`.
The helper opens `/onboarding` in a fresh self-hosted instance, enters a unique
name in `e.g. Northwind Labs`, and chooses `Continue`. The agent-name field must
appear and the saved company must have no invented goal. Reopen the
company-prefixed onboarding route and reload; the agent step must appear
without creating another company. Screenshots and a trace capture the real UI action and result.
This proves company creation/resumption, not hiring or a first model run.
Manual: Create a disposable company and first agent. Leave midway, reopen the
company's onboarding route, and finish; drafts save automatically. Follow the
created first task, verify the selected agent and company, and inspect its output
or explicit authentication gate. Reload both company and task.
### `cli-setup`
+5 -1
View File
@@ -21,7 +21,7 @@ Open `/pipelines`, `/pipelines/:pipelineId`, and its Settings route.
Open `/pipelines/:pipelineId/items/:caseId`, `/review-queue`, or `/learnings`.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Enable pipelines and prepare a disposable pipeline with at least two stages and a fixture agent. Record any associated case type.
@@ -39,6 +39,10 @@ Manual: Open a pending review, inspect its source output and revision, accept or
## Gotchas
- Learnings is filtered case-event history, such as review decisions and forced
moves, not a separate authored learning-record system. The UI visibility flag
is separate from API authorization.
- These routes are experimental and may redirect or deny access when disabled.
- ReviewQueue and Learnings are hosted from the pipeline module; a page-file inventory alone misses them.
- Pipeline items and free-standing [cases](./cases.md) share records but have different navigation contexts.
+6 -1
View File
@@ -21,7 +21,7 @@ Open Settings → Plugins where permitted, or the CLI plugin commands.
Open `/plugins/:pluginId`, a registered plugin route, or its contributed company/project settings surface.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a known local fixture plugin with bounded capabilities. Record installed version and any declared external dependencies.
@@ -39,6 +39,11 @@ Manual: Open the contributed page by navigation and cold deep link, perform a ha
## Gotchas
- Installation/lifecycle is instance-level; configuration can be company-scoped.
Settings lives at `/company/settings/instance/plugins`. npm installation and
bundled-local installation are different controls. Developer file-watcher
reload is not a lifecycle action in the plugin manager.
- Installing a plugin does not authorize every host capability or external action.
- Plugin-provided routes are dynamic; this map covers the host lifecycle, not every third-party feature.
- Environment-driver plugins also need the [environment](./execution-environments.md) checks.
+8 -1
View File
@@ -21,7 +21,7 @@ Open `/projects`, create a project, and enter its overview or Issues view.
Open `/projects/:projectId/configuration` and its repository/workspace settings.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a disposable project and a small repository you can access; record checkout and credential ownership.
@@ -39,6 +39,13 @@ Manual: Attach a small repository, save, and create a project task. Inspect the
## Gotchas
- `/overview` now selects Configuration, not a distinct overview page. Personal
join/leave and star controls have their own persistence and should be checked
when affected; **Star/Unstar {project name}** is the visible star action.
- Project name slugs change on rename and have no historical alias lookup.
Reopen a renamed project through its stable UUID route, wait for the current
Configuration form, and record the newly normalized URL before reloading.
- Project repository configuration and a provisioned execution workspace are different records.
- See [workspaces](./workspaces.md), [goals](./goals.md), and [budgets](./budgets-costs.md) for those features.
- Do not assume a deleted or archived project should erase its historical task evidence.
+8 -2
View File
@@ -20,9 +20,10 @@ Open `/routines`, create one, and inspect `/routines/:routineId` and its edit se
### `trigger-history`
Use the routine’s trigger controls and History/Audit links; lists also offer Run now.
Use the routine's trigger controls and Runs/Recent Runs/Audit links; lists also
offer Run now. **History** records configuration versions, not executions.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a disposable routine, deterministic test agent, and harmless target. Record schedule timezone and enabled state.
@@ -32,6 +33,11 @@ Automated: [routine detail helper](../ui/src/pages/RoutineDetail.test.tsx) tests
Manual: Save an agent, request, project, and a required variable. Reload and inspect the saved definition. Run now with a distinguishable input, follow the resulting task, and verify its output and history entry. Check a missing required input.
Variables are detected from `{{name}}` placeholders; mark a detected variable
Required and verify missing input prevents Run routine. Configuration can be
checked without model credentials. A successful subprocess run alone does not
complete the generated task or establish a useful task output.
### `trigger-history`
Automated: [trigger wizard](../ui/src/components/routine-triggers/TriggerWizard.test.tsx) is component coverage; [routine integration](../server/src/__tests__/routines-e2e.test.ts) is server integration rather than a provider-wide browser proof.
+5 -1
View File
@@ -21,7 +21,7 @@ Open an agent’s model/adapter configuration or the instance adapter settings p
Open an agent run at `/agents/:agentId/runs/:runId` from the agent or task activity.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Record adapter, model, account, native/legacy runtime, and environment. Capabilities vary; one harness result is not a matrix-wide qualification.
@@ -39,6 +39,10 @@ Manual: Start a disposable task, open its running transcript, and follow it to a
## Gotchas
- Model accounts are managed in Apps and inline agent account controls.
`/company/settings/connections` configures decision/fast-response models,
not the general model-account library.
- The run log is instance database data; it is not the opt-out first-party Telemetry system or operator-configured OpenTelemetry tracing.
- Authentication/model discovery does not establish tool, image, steering, or resumption capability parity.
- Use [recovery](./recovery.md) for interrupted runs and [execution environments](./execution-environments.md) for local/SSH/sandbox differences.
+5 -1
View File
@@ -22,7 +22,7 @@ Open company Settings → Secrets and the company/personal secret views.
Use the Proposals tab or Import from vault where available.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use dummy credentials in an isolated company, two user identities, and a test agent. Never print actual secret values in evidence.
@@ -40,6 +40,10 @@ Manual: Review a dummy proposal, reject one and accept another, then inspect the
## Gotchas
- Vault import currently targets eligible AWS Secrets Manager providers.
Organization-secret checks in local-trusted mode do not establish two-user
personal-secret ownership or actual external-vault import.
- A displayed reference name is not proof that the actor can resolve its value at runtime.
- Company secrets and user-specific values have different ownership.
- Redaction needs verification in errors, activity, and run logs as well as the form.
+7 -1
View File
@@ -21,12 +21,15 @@ Open `/skills`, Discover, or `/skills/sources`; project import is available thro
### `studio`
Use New skill or `/skills/studio/:skillId`; writable copies and source-backed originals differ.
Direct authoring starts at `/skills/studio/new`: fill **Name** and **Slug**, then
**Create skill**. Authoring/save/reload requires no model credential; running a
Studio test is separate.
### `agent-enablement`
Use skill policy/agent selection in the library or an agent’s Skills tab.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a harmless skill fixture and test agent with skill support. Record whether the source is local, bundled, project, or remote.
@@ -50,6 +53,9 @@ Manual: Enable the skill for one test agent and leave another without access. Ru
## Gotchas
- The release picker requires `enableBetaSkills`. Do not invent an Instructions
accessibility label for the Studio body editor; inspect its current handles.
- Installed does not mean enabled for every agent.
- Read-only GitHub/bundled skills may require a writable copy rather than an in-place edit.
- Studio interaction coverage is in [questions and approvals](./questions-and-approvals.md); a passing mock is not a live harness test.
+5 -1
View File
@@ -21,10 +21,14 @@ Open the task’s parent/child and dependency links in task detail.
Inspect the task’s execution policy and available review, pause, resume, or stop controls.
## Driving it
## Driving it with Playwright and the Paperclip API
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Prepare a parent, child, and dependent task plus implementer/reviewer agents. Record execution policy and use a disposable tree.
Parent/dependency navigation and inherited tree holds can be checked on unassigned
tasks. Agents are required for the continuation and signoff portion. A tree pause
creates a hold; it does not change task statuses to a paused status.
### `task-relationships`
Automated: [dependency wakeups](../server/src/__tests__/issue-dependency-wakeups-routes.test.ts) checks server behavior; visual relationship navigation remains a manual check.
+11 -2
View File
@@ -26,9 +26,12 @@ Open `/issues` or a project’s Issues view; legacy `/tasks` links redirect.
Open `/issues/:issueId` from a row, mention, or direct link.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use an invokable test agent and a project; record whether a real or simulated harness handles the request.
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a
disposable project. Persistence checks can assign work to Me or leave it
unassigned; execution checks additionally need an invokable test agent. Record
whether a real or simulated harness handles execution.
### `new-task`
@@ -36,6 +39,10 @@ Automated: [creation dialog](../ui/src/components/NewIssueDialog.test.tsx) is co
Manual: Create a uniquely named task with an agent and project. Open its resulting link and reload. Confirm text, attachment, assignment, and project persisted, and verify execution started only when the task and agent are eligible.
Creation is request-first: enter **Describe a task…**, choose the assignee and
project, and submit **Create task**. A standalone title field is not always shown.
Use a human/unassigned task to verify persistence without invoking a model.
### `task-list`
Automated: [task-list helpers](../ui/src/pages/Issues.test.tsx) cover search URLs, pagination, deduplication, and presentation constants. The suite does not render the page or exercise filtering, grouping, opening a task, and returning; those interactions still need a running-browser check.
@@ -50,6 +57,8 @@ Manual: Add a comment and edit authorized properties. Reload and verify history,
## Gotchas
- Priority remains an API field, but ordinary priority UI editing is hidden.
- A status label alone does not prove that the requested work exists.
- Single-assignee and atomic checkout rules still apply when tasks are created from different hosts.
- Use [coordination](./task-coordination.md), [steering](./steering.md), and [documents](./documents-artifacts.md) for their distinct workflows.
+5 -1
View File
@@ -21,7 +21,7 @@ Use the CLI teams catalog/preview/install operations in the selected context.
The catalog components and install wizard are available in developer tests/design surfaces. No top-level `/teams` route is registered; use the CLI/API for the current installation workflow.
## Driving it
## Driving it with the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a disposable company, a small catalog entry, and explicit company context. Check `paperclipai teams --help` for current subcommands.
@@ -39,6 +39,10 @@ Manual: When changing or mounting these components, exercise preview, manager/so
## Gotchas
- Catalog preview records an audit event but installs no agents/projects. Use
`teams browse/inspect/preview` with explicit company and origin API context;
the current product has no `/teams` route.
- Source files do not prove that a screen is reachable in the current app.
- Installed team configuration may still need credentials or environment setup.
- Company package import is a separate [portability](./companies.md) workflow.
+6 -1
View File
@@ -22,7 +22,7 @@ Use the task’s workspace card or Files panel and follow its workspace link.
Open `/workspaces`, a project workspace, or `/execution-workspaces/:workspaceId` and its Services/Configuration/Runtime logs views.
## Driving it
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Enable isolated-workspace UI where needed. Use an owned disposable repository and environment; record workspace ID and branch.
@@ -40,6 +40,11 @@ Manual: Provision a disposable workspace, wait for readiness, start an available
## Gotchas
- UI visibility, default isolation policy, the local environment row, a workspace
record, and a running service are distinct prerequisites. A fresh instance
still needs an owned cwd/repository and runtime commands; `/workspaces` lists
existing work. Reopening depends on lifecycle, not a generic detail action.
- A link or running badge does not prove the service is reachable from the user’s browser.
- Workspace IDs, runtime leases, and repository paths are not interchangeable.
- Use [recovery](./recovery.md) for divergence/export repair and [CLI operations](./cli-operations.md) for local worktree tooling.
+1 -1
View File
@@ -61,7 +61,7 @@
"smoke:posthog-live": "node scripts/smoke/posthog-live.mjs",
"smoke:pipelines-tutorial": "./scripts/smoke/pipelines-tutorial-smoke.sh",
"smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs",
"test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs",
"test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs",
"storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs",
"test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts",
"test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack",
+2 -2
View File
@@ -63,7 +63,7 @@
},
"sideEffects": false,
"scripts": {
"build": "pnpm run check:protocol-manifest && pnpm run build:typescript && pnpm run check:capability-contract && pnpm run check:capability-inventory && pnpm run check:protocol-coverage && node scripts/generate-semantic-contracts.mjs --check && node scripts/check-runner-workflow-traceability.mjs && pnpm run build:binary && node scripts/generate-replay-goldens.mjs --check && node scripts/generate-semantic-action-catalog.mjs --check",
"build": "node scripts/record-build-inputs.mjs --begin && pnpm run check:protocol-manifest && pnpm run build:typescript && pnpm run check:capability-contract && pnpm run check:capability-inventory && pnpm run check:protocol-coverage && node scripts/generate-semantic-contracts.mjs --check && node scripts/check-runner-workflow-traceability.mjs && pnpm run build:binary && node scripts/generate-replay-goldens.mjs --check && node scripts/generate-semantic-action-catalog.mjs --check && node scripts/record-build-inputs.mjs --finish",
"build:typescript": "pnpm run check:acpx-profiles && pnpm run ensure:eval-build-deps && pnpm run check:protocol-types && node ./node_modules/typescript/bin/tsc --version && node ./node_modules/typescript/bin/tsc -p tsconfig.json && node ./node_modules/typescript/bin/tsc -p tsconfig.surfaces.json && node scripts/build-verified-provider-entrypoints.mjs",
"build:rust": "cargo build --manifest-path runner/Cargo.toml --locked --workspace --bins",
"build:binary": "cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd && node scripts/stage-runner-binary.mjs",
@@ -81,7 +81,7 @@
"typecheck:browser": "tsc -p tsconfig.browser.json --noEmit",
"test": "pnpm run test:typescript && pnpm run test:rust",
"test:typescript": "pnpm run test:typescript:prep && vitest run",
"test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node --test test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs",
"test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node --test test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs",
"test:typescript:vitest": "node ./scripts/run-pr-vitest-lane.mjs",
"test:rust": "cargo test --release --manifest-path runner/Cargo.toml --locked --workspace",
"test:codex": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test codex_provider",
@@ -0,0 +1,444 @@
// Records which inputs produced the Runner package's current build output, so
// a consumer that would otherwise rebuild the Runner can reuse the build when
// nothing it depends on has changed.
//
// server's `prepare:runner-vendor` runs the Runner's full `build`, because a
// standalone `pnpm --filter @paperclipai/server build` (the Dockerfile's path)
// must produce a current vendored Runner on its own. Inside `pnpm -r build`
// the same command has just built the Runner a few seconds earlier, and the
// rebuild cost 12-16s on the CI Build job's critical path (six master runs,
// 2026-10-11), the same again in every other `pnpm build`.
//
// The Runner's `build` brackets itself with scripts/record-build-inputs.mjs:
// `--begin` removes any previous stamp and writes a pending record of the
// input digest before the first step reads anything; `--finish` recomputes
// the digest after the last check and writes the stamp under
// node_modules/.cache only if the inputs are still the ones the build started
// from, together with a manifest of every file the build left under dist. The
// server's prepare step (server/scripts/prepare-runner-vendor.mjs) reuses the
// build only when the stamp matches, every recorded output is still there at
// its recorded size, and the current input digest matches. Every other state
// rebuilds: a missing stamp, a changed input, a deleted or truncated output, a
// different platform, an edit that landed while the build was running. The
// slow direction is always the safe one.
//
// Each build chain has its own pending record, so two Runner builds running
// at once in one checkout (server `build` and `typecheck` side by side, say)
// do not consume each other's: `--begin` and `--finish` are separate
// processes, and what they share is the shell pnpm runs the chain in, so the
// record is keyed by that shell's pid. Both builds write the same dist, which
// the stamp cannot make safe; what it does guarantee is that a stamp is
// written only by a build whose own inputs did not move, and that the other
// build's `--finish` drops the stamp if its inputs did.
//
// Inputs: every file under packages/paperclip-runner except its build outputs
// (dist, dist-browser, runner/target), node_modules and dot-directories (tool
// caches); every file under packages/paperclip-eval-kernel except dist and
// node_modules, because build:typescript builds that package first; every
// file under the root patches directory, because check:acpx-profiles hashes
// the acpx patch and build-verified-provider-entrypoints.mjs copies the pi-acp
// and brace-expansion patches into dist; and the root files the Runner's
// tsconfig and dependency graph read. The toolchain pins live inside the
// package (rust-toolchain.toml, runner/Cargo.lock), and the TypeScript version
// comes from pnpm-lock.yaml, so both are covered.
//
// Outputs: the server build vendors packages/paperclip-runner/dist wholesale
// (`cp -Rf ../packages/paperclip-runner/dist/. dist/vendor/paperclip-runner/`),
// so the stamp lists every file under dist with its size and reuse needs all
// of them present; a helper such as dist/cli/acpx-runtime-sidecar.cjs is
// opened by path at runtime, so a tree that lost it must not count as current.
import { createHash } from "node:crypto";
import {
existsSync,
lstatSync,
mkdirSync,
readdirSync,
readFileSync,
readlinkSync,
renameSync,
rmSync,
writeFileSync,
} from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
export const BUILD_INPUTS_SCHEMA = "paperclip.runner-build-inputs.v2";
const RUNNER_PACKAGE_RELATIVE = "packages/paperclip-runner";
const EVAL_KERNEL_PACKAGE_RELATIVE = "packages/paperclip-eval-kernel";
const ROOT_INPUT_DIRECTORIES = ["patches"];
const ROOT_INPUT_FILES = [
"package.json",
"pnpm-lock.yaml",
"pnpm-workspace.yaml",
"tsconfig.base.json",
];
const OUTPUT_DIRECTORY_RELATIVE = "dist";
const SKIPPED_DIRECTORY_NAMES = new Set(["node_modules", "dist", "dist-browser"]);
const RUNNER_SKIPPED_DIRECTORIES = new Set([`${RUNNER_PACKAGE_RELATIVE}/runner/target`]);
export function runnerPackageRootFromScripts() {
return path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
}
export function repoRootFromRunnerPackage(packageRoot) {
return path.resolve(packageRoot, "../..");
}
export function buildInputsStampPath(packageRoot) {
return path.join(packageRoot, "node_modules", ".cache", "paperclip-runner-build-inputs.json");
}
/** One record per build in progress, written by `--begin` and consumed by that build's `--finish`. */
export function buildInputsPendingDirectory(packageRoot) {
return path.join(packageRoot, "node_modules", ".cache", "paperclip-runner-build-inputs.pending");
}
export function buildInputsPendingPath(packageRoot, buildId) {
return path.join(buildInputsPendingDirectory(packageRoot), `${buildId}.json`);
}
/**
* A pending record older than this belongs to a build that never reached
* `--finish` (a failed step, an interrupted build); `--begin` removes such
* records so they cannot pile up. No Runner build runs anywhere near this long.
*/
export const PENDING_RECORD_MAX_AGE_MS = 24 * 60 * 60 * 1000;
/**
* Identifies the build chain a `--begin` or `--finish` process belongs to.
* The two are separate processes; what they share is the shell pnpm runs the
* chain in, so the chain is named after its pid, which `--begin` sees as its
* parent. A shell that execs the last command of a `-c` string (bash can)
* turns `--finish` into the shell itself, so `--finish` also accepts a record
* named after its own pid.
*/
export function currentBuildId() {
return String(process.ppid);
}
function finishingBuildIds() {
return [String(process.ppid), String(process.pid)];
}
function readPendingRecord(pendingPath) {
const pending = readBuildInputsStamp(pendingPath);
if (!pending || pending.schema !== BUILD_INPUTS_SCHEMA || typeof pending.digest !== "string") return null;
return pending;
}
/** Drops pending records of builds that never finished, leaving the one for `buildId`. */
function prunePendingRecords(packageRoot, buildId, now) {
const directory = buildInputsPendingDirectory(packageRoot);
let entries;
try {
entries = readdirSync(directory);
} catch {
return;
}
const cutoff = now().getTime() - PENDING_RECORD_MAX_AGE_MS;
for (const name of entries) {
if (name === `${buildId}.json`) continue;
const absolute = path.join(directory, name);
let stat;
try {
stat = lstatSync(absolute);
} catch {
continue;
}
if (stat.mtimeMs < cutoff) rmSync(absolute, { force: true });
}
}
/**
* The outputs every consumer of the vendored Runner reads; a build that did
* not produce them is not recorded, and a tree missing any of them is not
* current whatever the stamp says.
*/
export function expectedBuildOutputs(packageRoot, platform = process.platform) {
const executable = platform === "win32" ? "paperclip-runnerd.exe" : "paperclip-runnerd";
return ["dist/index.js", "dist/live/index.js", "dist/testing.js", `dist/bin/${executable}`]
.map((relative) => path.join(packageRoot, ...relative.split("/")));
}
function walk(root, relativeDirectory, { skipNames, skippedDirectories, skipDotDirectories }, out) {
const absolute = path.join(root, ...relativeDirectory.split("/"));
if (!existsSync(absolute)) return;
const entries = readdirSync(absolute, { withFileTypes: true })
.sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0));
for (const entry of entries) {
const relative = `${relativeDirectory}/${entry.name}`;
if (entry.isDirectory()) {
if (skipDotDirectories && entry.name.startsWith(".")) continue;
if (skipNames.has(entry.name)) continue;
if (skippedDirectories.has(relative)) continue;
walk(root, relative, { skipNames, skippedDirectories, skipDotDirectories }, out);
continue;
}
// Files and symlinks; a symlinked directory is hashed as its link target
// rather than followed, so a link into node_modules cannot pull a
// dependency tree into the input set.
out.push(relative);
}
}
const INPUT_WALK = {
skipNames: SKIPPED_DIRECTORY_NAMES,
skippedDirectories: new Set(),
skipDotDirectories: true,
};
/** Repo-relative POSIX paths of every build input, sorted. */
export function collectBuildInputFiles(repoRoot) {
const files = [];
walk(repoRoot, RUNNER_PACKAGE_RELATIVE, { ...INPUT_WALK, skippedDirectories: RUNNER_SKIPPED_DIRECTORIES }, files);
walk(repoRoot, EVAL_KERNEL_PACKAGE_RELATIVE, INPUT_WALK, files);
for (const directory of ROOT_INPUT_DIRECTORIES) walk(repoRoot, directory, INPUT_WALK, files);
for (const name of ROOT_INPUT_FILES) {
if (existsSync(path.join(repoRoot, name))) files.push(name);
}
return files.sort();
}
export function computeBuildInputsDigest(repoRoot) {
const files = collectBuildInputFiles(repoRoot);
const hash = createHash("sha256");
let byteCount = 0;
for (const relative of files) {
const absolute = path.join(repoRoot, ...relative.split("/"));
hash.update(relative);
hash.update("\0");
if (lstatSync(absolute).isSymbolicLink()) {
hash.update("link\0");
hash.update(readlinkSync(absolute));
} else {
const content = readFileSync(absolute);
byteCount += content.length;
hash.update(`file ${content.length}\0`);
hash.update(content);
}
hash.update("\0");
}
return {
schema: BUILD_INPUTS_SCHEMA,
digest: hash.digest("hex"),
fileCount: files.length,
byteCount,
};
}
/**
* Every file the build left under dist (the tree the server vendors), as
* package-relative POSIX paths with the size lstat reports, sorted. Nothing
* under dist is skipped: the server copies all of it.
*/
export function collectBuildOutputFiles(packageRoot) {
const files = [];
walk(packageRoot, OUTPUT_DIRECTORY_RELATIVE, { skipNames: new Set(), skippedDirectories: new Set(), skipDotDirectories: false }, files);
return files.sort().map((relative) => ({
path: relative,
size: lstatSync(path.join(packageRoot, ...relative.split("/"))).size,
}));
}
/** The first recorded output that is gone or has changed size, or null when all are intact. */
export function findChangedBuildOutput(packageRoot, outputs) {
for (const output of outputs) {
const absolute = path.join(packageRoot, ...output.path.split("/"));
let stat;
try {
stat = lstatSync(absolute);
} catch {
return { ...output, state: "missing" };
}
if (stat.size !== output.size) return { ...output, state: "changed size", size: stat.size, recordedSize: output.size };
}
return null;
}
export function readBuildInputsStamp(stampPath) {
try {
const parsed = JSON.parse(readFileSync(stampPath, "utf8"));
return parsed && typeof parsed === "object" ? parsed : null;
} catch {
return null;
}
}
export function writeBuildInputsStamp(stampPath, record) {
mkdirSync(path.dirname(stampPath), { recursive: true });
const staged = `${stampPath}.${process.pid}.tmp`;
writeFileSync(staged, `${JSON.stringify(record, null, 2)}\n`);
renameSync(staged, stampPath);
}
export function removeBuildInputsStamp(stampPath) {
rmSync(stampPath, { force: true });
}
/**
* First step of the Runner's build: forgets the previous build (a build in
* progress can no longer vouch for its outputs) and records the inputs the
* build is about to read, so that `recordRunnerBuild` can tell whether they
* changed while it ran.
*/
export function beginRunnerBuild({
repoRoot,
packageRoot,
platform = process.platform,
arch = process.arch,
buildId = currentBuildId(),
now = () => new Date(),
}) {
removeBuildInputsStamp(buildInputsStampPath(packageRoot));
prunePendingRecords(packageRoot, buildId, now);
const inputs = computeBuildInputsDigest(repoRoot);
const pending = { ...inputs, platform, arch, buildId, startedAt: now().toISOString() };
const pendingPath = buildInputsPendingPath(packageRoot, buildId);
writeBuildInputsStamp(pendingPath, pending);
return { ...pending, pendingPath };
}
/**
* Last step of the Runner's build, after every step and check has passed:
* records the inputs captured by `beginRunnerBuild` as the ones that produced
* the outputs now under dist. An edit that landed while the build ran means
* the outputs may not reflect the current inputs, so nothing is recorded and
* the next consumer rebuilds. A build that did not begin with
* `beginRunnerBuild` (no pending record for this build chain), or that left
* a required output missing, is an error.
*/
export function recordRunnerBuild({
repoRoot,
packageRoot,
platform = process.platform,
arch = process.arch,
buildId = null,
now = () => new Date(),
}) {
const stampPath = buildInputsStampPath(packageRoot);
const candidates = buildId === null ? finishingBuildIds() : [String(buildId)];
let pending = null;
let pendingPath = null;
for (const candidate of candidates) {
pendingPath = buildInputsPendingPath(packageRoot, candidate);
pending = readPendingRecord(pendingPath);
if (pending) break;
}
if (!pending) {
removeBuildInputsStamp(stampPath);
throw new Error(
`no pending build inputs for build ${candidates.join(" or ")} under ` +
`${buildInputsPendingDirectory(packageRoot)}: the Runner build must start with ` +
"`node scripts/record-build-inputs.mjs --begin` before it can record its result",
);
}
rmSync(pendingPath, { force: true });
const missing = expectedBuildOutputs(packageRoot, platform).filter((file) => !existsSync(file));
if (missing.length > 0) {
removeBuildInputsStamp(stampPath);
throw new Error(`the Runner build finished without producing ${path.relative(packageRoot, missing[0])}`);
}
const inputs = computeBuildInputsDigest(repoRoot);
if (inputs.digest !== pending.digest) {
removeBuildInputsStamp(stampPath);
return {
recorded: false,
code: "inputs_changed_during_build",
detail:
`inputs changed while the build ran (digest ${pending.digest.slice(0, 12)} at start, ` +
`${inputs.digest.slice(0, 12)} now); the next consumer rebuilds`,
startedAt: pending.startedAt,
stampPath,
};
}
const outputs = collectBuildOutputFiles(packageRoot);
const record = {
...inputs,
platform,
arch,
startedAt: pending.startedAt,
recordedAt: now().toISOString(),
outputCount: outputs.length,
outputs,
};
writeBuildInputsStamp(stampPath, record);
return { recorded: true, code: "recorded", ...record, stampPath };
}
/**
* Decides whether the Runner's outputs are current for the present inputs.
* `code` is a stable slug for timing lines; `detail` is for humans.
*/
export function assessRunnerBuild({
repoRoot,
packageRoot,
platform = process.platform,
arch = process.arch,
}) {
const stampPath = buildInputsStampPath(packageRoot);
const stamp = readBuildInputsStamp(stampPath);
if (!stamp) {
return { current: false, code: "no_stamp", detail: `no build stamp at ${stampPath}`, stampPath };
}
if (stamp.schema !== BUILD_INPUTS_SCHEMA || !Array.isArray(stamp.outputs)) {
return {
current: false,
code: "schema_mismatch",
detail: `stamp schema ${String(stamp.schema)} is not ${BUILD_INPUTS_SCHEMA}`,
stampPath,
};
}
if (stamp.platform !== platform || stamp.arch !== arch) {
return {
current: false,
code: "platform_mismatch",
detail: `stamp recorded on ${stamp.platform}/${stamp.arch}, this is ${platform}/${arch}`,
stampPath,
};
}
const missing = expectedBuildOutputs(packageRoot, platform).filter((file) => !existsSync(file));
if (missing.length > 0) {
return {
current: false,
code: "missing_output",
detail: `missing build output ${path.relative(packageRoot, missing[0])}`,
stampPath,
};
}
const changedOutput = findChangedBuildOutput(packageRoot, stamp.outputs);
if (changedOutput) {
return {
current: false,
code: "outputs_changed",
detail:
`build output ${changedOutput.path} is ${changedOutput.state} since the build recorded at ` +
`${stamp.recordedAt}; ${stamp.outputs.length} outputs checked`,
stampPath,
};
}
const inputs = computeBuildInputsDigest(repoRoot);
if (inputs.digest !== stamp.digest) {
return {
current: false,
code: "inputs_changed",
detail: `inputs changed since the recorded build; ${inputs.fileCount} files checked`,
stampPath,
inputs,
};
}
return {
current: true,
code: "inputs_unchanged",
detail:
`inputs unchanged since the build recorded at ${stamp.recordedAt}; ${inputs.fileCount} files ` +
`and ${stamp.outputs.length} outputs checked`,
stampPath,
inputs,
stamp,
};
}
@@ -0,0 +1,533 @@
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { existsSync, readdirSync } from "node:fs";
import { mkdtemp, mkdir, readdir, readFile, rm, stat, symlink, truncate, utimes, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { test } from "node:test";
import { promisify } from "node:util";
import { fileURLToPath } from "node:url";
import {
BUILD_INPUTS_SCHEMA,
PENDING_RECORD_MAX_AGE_MS,
assessRunnerBuild,
beginRunnerBuild,
buildInputsPendingDirectory,
buildInputsPendingPath,
buildInputsStampPath,
collectBuildInputFiles,
collectBuildOutputFiles,
computeBuildInputsDigest,
currentBuildId,
expectedBuildOutputs,
readBuildInputsStamp,
recordRunnerBuild,
} from "./build-inputs.mjs";
const execFileAsync = promisify(execFile);
const recordScript = fileURLToPath(new URL("./record-build-inputs.mjs", import.meta.url));
/** A child of this process names its build after this process, as `--begin` names it after the shell. */
const childBuildId = String(process.pid);
async function waitFor(condition, label, timeoutMs = 10_000) {
const deadline = Date.now() + timeoutMs;
while (!condition()) {
if (Date.now() > deadline) throw new Error(`timed out waiting for ${label}`);
await new Promise((resolve) => setTimeout(resolve, 20));
}
}
/**
* The whole build chain the way pnpm runs it: one shell, `--begin` first,
* `--finish` last, with a stand-in for the build steps that waits for a
* marker file so two chains can be held open at the same time.
*/
function runChainInShell(shell, packageRoot, marker) {
const quote = (value) => `'${value.replace(/'/g, `'\\''`)}'`;
const wait =
`${quote(process.execPath)} -e ${quote(
"const fs=require('node:fs');const [marker]=process.argv.slice(1);const deadline=Date.now()+10000;" +
"(function poll(){if(fs.existsSync(marker))return;if(Date.now()>deadline)process.exit(3);setTimeout(poll,20)})()",
)} ${quote(marker)}`;
const phase = (flag) => `${quote(process.execPath)} ${quote(recordScript)} ${flag} --package-root ${quote(packageRoot)}`;
return execFileAsync(shell, ["-c", `${phase("--begin")} && ${wait} && ${phase("--finish")}`]);
}
async function writeTree(root, files) {
for (const [relative, content] of Object.entries(files)) {
const absolute = path.join(root, ...relative.split("/"));
await mkdir(path.dirname(absolute), { recursive: true });
await writeFile(absolute, content);
}
}
/**
* A throwaway repo with one Runner source file, one Rust source, two root
* patches, built outputs (the four every consumer reads plus one helper that
* is only opened by path), and the usual noise.
*/
async function makeRepo() {
const repoRoot = await mkdtemp(path.join(tmpdir(), "runner-build-inputs-"));
await writeTree(repoRoot, {
"package.json": "{}\n",
"pnpm-lock.yaml": "lockfileVersion: 9\n",
"pnpm-workspace.yaml": "packages:\n - packages/*\n",
"tsconfig.base.json": "{}\n",
"patches/acpx@0.13.1.patch": "--- a\n+++ b\n",
"patches/pi-acp@0.0.33.patch": "--- c\n+++ d\n",
"packages/paperclip-runner/package.json": '{"name":"@paperclipai/paperclip-runner"}\n',
"packages/paperclip-runner/src/index.ts": "export const one = 1;\n",
"packages/paperclip-runner/runner/src/main.rs": "fn main() {}\n",
"packages/paperclip-runner/runner/target/release/paperclip-runnerd": "binary\n",
"packages/paperclip-runner/dist/index.js": "export const one = 1;\n",
"packages/paperclip-runner/dist/live/index.js": "export {};\n",
"packages/paperclip-runner/dist/testing.js": "export {};\n",
"packages/paperclip-runner/dist/bin/paperclip-runnerd": "binary\n",
"packages/paperclip-runner/dist/cli/acpx-runtime-sidecar.cjs": "module.exports = {};\n",
"packages/paperclip-runner/dist-browser/index.js": "browser\n",
"packages/paperclip-runner/node_modules/dep/index.js": "dep\n",
"packages/paperclip-runner/.vite/cache.json": "{}\n",
"packages/paperclip-eval-kernel/src/index.ts": "export const kernel = 1;\n",
"packages/paperclip-eval-kernel/dist/index.js": "export const kernel = 1;\n",
});
const packageRoot = path.join(repoRoot, "packages", "paperclip-runner");
return { repoRoot, packageRoot };
}
const INPUT_FILE_COUNT = 10;
const OUTPUT_FILE_COUNT = 5;
async function withRepo(action) {
const repo = await makeRepo();
try {
await action(repo);
} finally {
await rm(repo.repoRoot, { recursive: true, force: true });
}
}
/** A complete, undisturbed build: begin, (nothing changes), finish. */
function recordCurrentBuild({ repoRoot, packageRoot, platform, arch, now }) {
beginRunnerBuild({ repoRoot, packageRoot, platform, arch, now });
return recordRunnerBuild({ repoRoot, packageRoot, platform, arch, now });
}
test("the input set covers sources, the eval kernel, root patches and root config, and skips outputs and caches", async () => {
await withRepo(async ({ repoRoot }) => {
const files = collectBuildInputFiles(repoRoot);
assert.deepEqual(files, [
"package.json",
"packages/paperclip-eval-kernel/src/index.ts",
"packages/paperclip-runner/package.json",
"packages/paperclip-runner/runner/src/main.rs",
"packages/paperclip-runner/src/index.ts",
"patches/acpx@0.13.1.patch",
"patches/pi-acp@0.0.33.patch",
"pnpm-lock.yaml",
"pnpm-workspace.yaml",
"tsconfig.base.json",
]);
assert.equal(files.length, INPUT_FILE_COUNT);
});
});
test("the output manifest lists every file under dist with its size, and nothing outside it", async () => {
await withRepo(async ({ packageRoot }) => {
const outputs = collectBuildOutputFiles(packageRoot);
assert.deepEqual(outputs, [
{ path: "dist/bin/paperclip-runnerd", size: 7 },
{ path: "dist/cli/acpx-runtime-sidecar.cjs", size: 21 },
{ path: "dist/index.js", size: 22 },
{ path: "dist/live/index.js", size: 11 },
{ path: "dist/testing.js", size: 11 },
]);
assert.equal(outputs.length, OUTPUT_FILE_COUNT);
});
});
test("the digest is stable, ignores outputs and caches, and tracks every kind of input", async () => {
await withRepo(async ({ repoRoot }) => {
const before = computeBuildInputsDigest(repoRoot);
assert.equal(before.schema, BUILD_INPUTS_SCHEMA);
assert.equal(before.fileCount, INPUT_FILE_COUNT);
assert.equal(computeBuildInputsDigest(repoRoot).digest, before.digest, "two reads disagree");
// Outputs, dependencies, caches and the Rust target directory may change freely.
await writeTree(repoRoot, {
"packages/paperclip-runner/dist/index.js": "export const one = 2;\n",
"packages/paperclip-runner/dist/cli/acpx-runtime-sidecar.cjs": "changed\n",
"packages/paperclip-runner/dist-browser/index.js": "changed\n",
"packages/paperclip-runner/node_modules/dep/index.js": "changed\n",
"packages/paperclip-runner/.vite/cache.json": '{"changed":true}\n',
"packages/paperclip-runner/runner/target/release/paperclip-runnerd": "rebuilt\n",
"packages/paperclip-eval-kernel/dist/index.js": "changed\n",
});
assert.equal(computeBuildInputsDigest(repoRoot).digest, before.digest, "an output changed the digest");
// Every input kind changes it: TypeScript, Rust, the eval kernel, a root
// patch, root config, a new file, a rename.
const seen = new Set([before.digest]);
const expectChange = async (label, files) => {
await writeTree(repoRoot, files);
const next = computeBuildInputsDigest(repoRoot).digest;
assert.ok(!seen.has(next), `${label} did not change the digest`);
seen.add(next);
};
await expectChange("a Runner source", { "packages/paperclip-runner/src/index.ts": "export const one = 2;\n" });
await expectChange("a Rust source", { "packages/paperclip-runner/runner/src/main.rs": "fn main() { }\n" });
await expectChange("an eval-kernel source", { "packages/paperclip-eval-kernel/src/index.ts": "export const kernel = 2;\n" });
await expectChange("the pi-acp patch the build copies into dist", { "patches/pi-acp@0.0.33.patch": "--- c\n+++ d\n+e\n" });
await expectChange("the acpx patch the profile check hashes", { "patches/acpx@0.13.1.patch": "--- a\n+++ b\n+c\n" });
await expectChange("a new patch", { "patches/brace-expansion@5.0.9.patch": "--- x\n+++ y\n" });
await expectChange("the lockfile", { "pnpm-lock.yaml": "lockfileVersion: 9\nchanged: true\n" });
await expectChange("the base tsconfig", { "tsconfig.base.json": '{"compilerOptions":{}}\n' });
await expectChange("a new file", { "packages/paperclip-runner/protocol/new.json": "{}\n" });
await rm(path.join(repoRoot, "packages/paperclip-runner/protocol/new.json"));
await expectChange("a renamed file", { "packages/paperclip-runner/protocol/renamed.json": "{}\n" });
});
});
test("a symlink is hashed as its target path, not followed", async () => {
await withRepo(async ({ repoRoot, packageRoot }) => {
const before = computeBuildInputsDigest(repoRoot);
await symlink("../node_modules/dep", path.join(packageRoot, "src", "linked"), "dir");
const linked = computeBuildInputsDigest(repoRoot);
assert.notEqual(linked.digest, before.digest, "the link itself is an input");
assert.equal(linked.fileCount, before.fileCount + 1, "the link target's files were followed");
await writeFile(path.join(packageRoot, "node_modules/dep/index.js"), "changed\n");
assert.equal(computeBuildInputsDigest(repoRoot).digest, linked.digest, "a change behind the link changed the digest");
});
});
test("a build is current only with a matching stamp and every recorded output intact", async () => {
await withRepo(async ({ repoRoot, packageRoot }) => {
const platform = "linux";
const arch = "x64";
const assess = () => assessRunnerBuild({ repoRoot, packageRoot, platform, arch });
assert.equal(assess().current, false);
assert.equal(assess().code, "no_stamp");
const record = recordCurrentBuild({ repoRoot, packageRoot, platform, arch, now: () => new Date(0) });
assert.equal(record.recorded, true);
assert.equal(record.stampPath, buildInputsStampPath(packageRoot));
const stamp = readBuildInputsStamp(record.stampPath);
assert.equal(stamp.schema, BUILD_INPUTS_SCHEMA);
assert.equal(stamp.digest, record.digest);
assert.equal(stamp.fileCount, INPUT_FILE_COUNT);
assert.equal(stamp.byteCount, record.byteCount);
assert.equal(stamp.platform, platform);
assert.equal(stamp.arch, arch);
assert.equal(stamp.startedAt, "1970-01-01T00:00:00.000Z");
assert.equal(stamp.recordedAt, "1970-01-01T00:00:00.000Z");
assert.equal(stamp.outputCount, OUTPUT_FILE_COUNT);
assert.deepEqual(stamp.outputs, collectBuildOutputFiles(packageRoot));
const current = assess();
assert.equal(current.current, true, current.detail);
assert.equal(current.code, "inputs_unchanged");
assert.match(current.detail, /10 files and 5 outputs checked/);
assert.equal(assessRunnerBuild({ repoRoot, packageRoot, platform: "win32", arch }).code, "platform_mismatch");
assert.equal(assessRunnerBuild({ repoRoot, packageRoot, platform, arch: "arm64" }).code, "platform_mismatch");
// The four outputs every consumer reads are checked by name.
for (const output of expectedBuildOutputs(packageRoot, platform)) {
const content = await readFile(output);
await rm(output);
const missing = assess();
assert.equal(missing.current, false, `${output} missing still counted as current`);
assert.equal(missing.code, "missing_output");
assert.match(missing.detail, new RegExp(path.basename(output)));
await writeFile(output, content);
}
assert.equal(assess().current, true);
// So is every other file the build left under dist: a helper that is only
// opened by path must be there, at the size the build wrote.
const helper = path.join(packageRoot, "dist", "cli", "acpx-runtime-sidecar.cjs");
const helperContent = await readFile(helper);
await rm(helper);
const gone = assess();
assert.equal(gone.current, false, "a missing helper still counted as current");
assert.equal(gone.code, "outputs_changed");
assert.match(gone.detail, /^build output dist\/cli\/acpx-runtime-sidecar\.cjs is missing since the build recorded at /);
await writeFile(helper, helperContent);
assert.equal(assess().current, true);
await truncate(helper, 3);
const truncated = assess();
assert.equal(truncated.current, false, "a truncated helper still counted as current");
assert.equal(truncated.code, "outputs_changed");
assert.match(truncated.detail, /^build output dist\/cli\/acpx-runtime-sidecar\.cjs is changed size since /);
await writeFile(helper, helperContent);
assert.equal(assess().current, true);
// A file the build did not produce may appear without invalidating it.
await writeFile(path.join(packageRoot, "dist", "extra.txt"), "later\n");
assert.equal(assess().current, true);
await writeFile(path.join(packageRoot, "src", "index.ts"), "export const one = 3;\n");
const changed = assess();
assert.equal(changed.current, false);
assert.equal(changed.code, "inputs_changed");
await writeFile(record.stampPath, JSON.stringify({ ...readBuildInputsStamp(record.stampPath), schema: "other" }));
assert.equal(assess().code, "schema_mismatch");
await writeFile(record.stampPath, JSON.stringify({ ...readBuildInputsStamp(record.stampPath), schema: BUILD_INPUTS_SCHEMA, outputs: undefined }));
assert.equal(assess().code, "schema_mismatch", "a stamp without an output manifest was trusted");
await writeFile(record.stampPath, "not json");
assert.equal(assess().code, "no_stamp");
});
});
test("an input edited while the build runs leaves no stamp, so the next consumer rebuilds", async () => {
await withRepo(async ({ repoRoot, packageRoot }) => {
const stampPath = buildInputsStampPath(packageRoot);
const pendingPath = buildInputsPendingPath(packageRoot, currentBuildId());
const assess = () => assessRunnerBuild({ repoRoot, packageRoot });
// A previous good build, then a new build begins: the old stamp is gone at
// once, because the outputs are about to be replaced.
recordCurrentBuild({ repoRoot, packageRoot });
assert.equal(assess().current, true);
const pending = beginRunnerBuild({ repoRoot, packageRoot, now: () => new Date(1000) });
assert.equal(pending.pendingPath, pendingPath);
assert.equal(pending.buildId, String(process.ppid), "a build is named after the shell that runs its chain");
assert.equal(readBuildInputsStamp(stampPath), null, "the old stamp survived the start of a new build");
assert.equal(readBuildInputsStamp(pendingPath).digest, pending.digest);
assert.equal(readBuildInputsStamp(pendingPath).buildId, pending.buildId);
assert.equal(assess().code, "no_stamp");
// The build "compiles" the old source, then an edit lands before it ends.
await writeFile(path.join(packageRoot, "src", "index.ts"), "export const one = 2;\n");
const result = recordRunnerBuild({ repoRoot, packageRoot });
assert.equal(result.recorded, false);
assert.equal(result.code, "inputs_changed_during_build");
assert.match(result.detail, /^inputs changed while the build ran \(digest [0-9a-f]{12} at start, [0-9a-f]{12} now\); the next consumer rebuilds$/);
assert.equal(result.startedAt, "1970-01-01T00:00:01.000Z");
assert.equal(readBuildInputsStamp(stampPath), null, "a stamp was written for a build whose inputs moved");
assert.equal(readBuildInputsStamp(pendingPath), null, "the pending record was left behind");
assert.equal(assess().code, "no_stamp");
// An undisturbed build records normally, and only the pending record's
// start time is kept.
const record = recordCurrentBuild({ repoRoot, packageRoot });
assert.equal(record.recorded, true);
assert.equal(readBuildInputsStamp(pendingPath), null);
assert.equal(assess().current, true);
});
});
test("a build that did not begin, or that left a required output missing, cannot be recorded", async () => {
await withRepo(async ({ repoRoot, packageRoot }) => {
const stampPath = buildInputsStampPath(packageRoot);
recordCurrentBuild({ repoRoot, packageRoot });
assert.throws(
() => recordRunnerBuild({ repoRoot, packageRoot }),
new RegExp(
`no pending build inputs for build ${process.ppid} or ${process.pid} under .*paperclip-runner-build-inputs\\.pending: ` +
"the Runner build must start with `node scripts/record-build-inputs\\.mjs --begin`",
),
);
assert.equal(readBuildInputsStamp(stampPath), null, "the previous stamp outlived a build that did not begin");
// Another build's record is not this build's: a chain that did not begin
// cannot finish on the strength of one that is still running.
beginRunnerBuild({ repoRoot, packageRoot, buildId: "other-shell" });
assert.throws(() => recordRunnerBuild({ repoRoot, packageRoot }), /no pending build inputs for build /);
assert.equal(
readBuildInputsStamp(buildInputsPendingPath(packageRoot, "other-shell")).digest,
computeBuildInputsDigest(repoRoot).digest,
"a build that did not begin consumed another build's record",
);
await rm(buildInputsPendingPath(packageRoot, "other-shell"));
beginRunnerBuild({ repoRoot, packageRoot });
await rm(path.join(packageRoot, "dist", "testing.js"));
assert.throws(
() => recordRunnerBuild({ repoRoot, packageRoot }),
/the Runner build finished without producing dist\/testing\.js/,
);
assert.equal(readBuildInputsStamp(stampPath), null);
assert.equal(assessRunnerBuild({ repoRoot, packageRoot }).code, "no_stamp");
});
});
test("record-build-inputs.mjs brackets the build for the package it is pointed at", async () => {
await withRepo(async ({ repoRoot, packageRoot }) => {
const run = (...args) => execFileAsync(process.execPath, [recordScript, ...args, "--package-root", packageRoot]);
const expected = computeBuildInputsDigest(repoRoot);
const short = expected.digest.slice(0, 12);
// --finish before any --begin is a wiring error, not a silent stamp.
await assert.rejects(run("--finish"), /no pending build inputs/);
assert.equal(readBuildInputsStamp(buildInputsStampPath(packageRoot)), null);
const begin = await run("--begin");
assert.match(begin.stdout, new RegExp(`^${BUILD_INPUTS_SCHEMA} begin: digest=${short} files=${INPUT_FILE_COUNT} bytes=\\d+\n$`));
assert.equal(readBuildInputsStamp(buildInputsPendingPath(packageRoot, childBuildId)).digest, expected.digest);
const finish = await run("--finish");
assert.match(
finish.stdout,
new RegExp(`^${BUILD_INPUTS_SCHEMA} recorded: digest=${short} files=${INPUT_FILE_COUNT} bytes=\\d+ outputs=${OUTPUT_FILE_COUNT} stamp=node_modules/.cache/paperclip-runner-build-inputs.json\n$`),
);
const stamp = readBuildInputsStamp(buildInputsStampPath(packageRoot));
assert.equal(stamp.digest, expected.digest);
assert.equal(stamp.platform, process.platform);
assert.equal(stamp.arch, process.arch);
assert.equal(stamp.outputCount, OUTPUT_FILE_COUNT);
assert.equal(assessRunnerBuild({ repoRoot, packageRoot }).current, true);
assert.equal((await stat(buildInputsStampPath(packageRoot))).isFile(), true);
// An edit between the two phases is reported and leaves nothing behind.
await run("--begin");
await writeFile(path.join(packageRoot, "src", "index.ts"), "export const one = 2;\n");
const moved = await run("--finish");
assert.match(moved.stdout, /^paperclip\.runner-build-inputs not recorded: inputs changed while the build ran /);
assert.equal(readBuildInputsStamp(buildInputsStampPath(packageRoot)), null);
await assert.rejects(run(), /one of --begin or --finish is required/);
await assert.rejects(run("--begin", "--finish"), /--finish cannot be combined with --begin/);
await assert.rejects(
execFileAsync(process.execPath, [recordScript, "--begin", "--bogus"]),
/unknown argument --bogus/,
);
});
});
test("two builds running at once in one checkout each keep their own pending record", async () => {
await withRepo(async ({ repoRoot, packageRoot }) => {
const stampPath = buildInputsStampPath(packageRoot);
const pendingDirectory = buildInputsPendingDirectory(packageRoot);
const first = { repoRoot, packageRoot, buildId: "shell-1" };
const second = { repoRoot, packageRoot, buildId: "shell-2" };
const digestNow = () => computeBuildInputsDigest(repoRoot).digest;
// Nothing changes: both builds record, in either finishing order, and the
// second to finish does not fail for want of a record.
beginRunnerBuild(first);
beginRunnerBuild(second);
assert.deepEqual((await readdir(pendingDirectory)).sort(), ["shell-1.json", "shell-2.json"]);
assert.equal(recordRunnerBuild(first).recorded, true);
assert.deepEqual(await readdir(pendingDirectory), ["shell-2.json"], "the first finish consumed the wrong record");
assert.equal(readBuildInputsStamp(stampPath).digest, digestNow());
assert.equal(recordRunnerBuild(second).recorded, true);
assert.deepEqual(await readdir(pendingDirectory), []);
assert.equal(assessRunnerBuild({ repoRoot, packageRoot }).current, true);
beginRunnerBuild(first);
beginRunnerBuild(second);
assert.equal(recordRunnerBuild(second).recorded, true);
assert.equal(recordRunnerBuild(first).recorded, true);
assert.equal(assessRunnerBuild({ repoRoot, packageRoot }).current, true);
// An edit lands between the two starts. The build that began before it
// must not leave a stamp whichever of the two finishes last: a stamp means
// every build that wrote dist read the inputs it names.
const source = path.join(packageRoot, "src", "index.ts");
beginRunnerBuild(first);
await writeFile(source, "export const one = 2;\n");
beginRunnerBuild(second);
const staleFirst = recordRunnerBuild(first);
assert.equal(staleFirst.recorded, false);
assert.equal(staleFirst.code, "inputs_changed_during_build");
assert.equal(readBuildInputsStamp(stampPath), null);
assert.equal(recordRunnerBuild(second).recorded, true, "the build that read the edited inputs was not recorded");
assert.equal(assessRunnerBuild({ repoRoot, packageRoot }).current, true);
beginRunnerBuild(first);
await writeFile(source, "export const one = 3;\n");
beginRunnerBuild(second);
assert.equal(recordRunnerBuild(second).recorded, true);
assert.equal(readBuildInputsStamp(stampPath).digest, digestNow());
const staleLast = recordRunnerBuild(first);
assert.equal(staleLast.recorded, false);
assert.equal(staleLast.code, "inputs_changed_during_build");
assert.equal(readBuildInputsStamp(stampPath), null, "a build that began before the edit left the other build's stamp standing");
assert.equal(assessRunnerBuild({ repoRoot, packageRoot }).code, "no_stamp");
assert.deepEqual(await readdir(pendingDirectory), []);
});
});
test("a build's start drops the pending records of builds that never finished, and no others", async () => {
await withRepo(async ({ repoRoot, packageRoot }) => {
const pendingDirectory = buildInputsPendingDirectory(packageRoot);
const now = new Date("2026-10-11T22:00:00.000Z");
const age = async (buildId, ageMs) => {
const moment = new Date(now.getTime() - ageMs);
await utimes(buildInputsPendingPath(packageRoot, buildId), moment, moment);
};
// A build that failed a step yesterday, one that failed a minute ago, and
// one still running, all left records; a new build starts.
beginRunnerBuild({ repoRoot, packageRoot, buildId: "crashed-yesterday", now: () => now });
await age("crashed-yesterday", PENDING_RECORD_MAX_AGE_MS + 60_000);
beginRunnerBuild({ repoRoot, packageRoot, buildId: "crashed-just-now", now: () => now });
await age("crashed-just-now", 60_000);
beginRunnerBuild({ repoRoot, packageRoot, buildId: "still-running", now: () => now });
await writeFile(path.join(pendingDirectory, "notes.txt"), "not a record\n");
await age("still-running", 0);
beginRunnerBuild({ repoRoot, packageRoot, buildId: "new-build", now: () => now });
assert.deepEqual(
(await readdir(pendingDirectory)).sort(),
["crashed-just-now.json", "new-build.json", "notes.txt", "still-running.json"],
"pruning removed the wrong records",
);
// The running build and the new one still finish on their own records.
assert.equal(recordRunnerBuild({ repoRoot, packageRoot, buildId: "still-running" }).recorded, true);
assert.equal(recordRunnerBuild({ repoRoot, packageRoot, buildId: "new-build" }).recorded, true);
// A record that fell out of the directory is simply absent.
await rm(pendingDirectory, { recursive: true, force: true });
beginRunnerBuild({ repoRoot, packageRoot, buildId: "after-wipe" });
assert.deepEqual(await readdir(pendingDirectory), ["after-wipe.json"]);
});
});
test("two chains run by separate shells at the same time both record", { skip: process.platform === "win32" }, async () => {
await withRepo(async ({ repoRoot, packageRoot }) => {
const pendingDirectory = buildInputsPendingDirectory(packageRoot);
const marker = path.join(repoRoot, "release-the-chains");
const records = () => (existsSync(pendingDirectory) ? readdirSync(pendingDirectory).sort() : []);
const first = runChainInShell("sh", packageRoot, marker);
await waitFor(() => records().length === 1, "the first chain's record");
const second = runChainInShell("sh", packageRoot, marker);
await waitFor(() => records().length === 2, "the second chain's record");
const [firstId, secondId] = records().map((name) => name.replace(/\.json$/, ""));
assert.notEqual(firstId, secondId);
for (const id of [firstId, secondId]) {
assert.match(id, /^\d+$/, "a chain's record is named after its shell's pid");
assert.equal(readBuildInputsStamp(buildInputsPendingPath(packageRoot, id)).buildId, id);
}
await writeFile(marker, "go\n");
const results = await Promise.all([first, second]);
for (const result of results) {
assert.match(result.stdout, new RegExp(`${BUILD_INPUTS_SCHEMA} begin: `));
assert.match(result.stdout, new RegExp(`${BUILD_INPUTS_SCHEMA} recorded: `), "a chain finished without recording");
}
assert.deepEqual(records(), []);
assert.equal(assessRunnerBuild({ repoRoot, packageRoot }).current, true);
// The same chain under bash, which may exec its last command: `--finish`
// is then the shell itself and still finds the record `--begin` wrote.
let bash;
try {
bash = (await execFileAsync("bash", ["-c", "echo ok"])).stdout.trim() === "ok";
} catch {
bash = false;
}
if (bash) {
await rm(marker);
const chain = runChainInShell("bash", packageRoot, marker);
await waitFor(() => records().length === 1, "the bash chain's record");
await writeFile(marker, "go\n");
assert.match((await chain).stdout, new RegExp(`${BUILD_INPUTS_SCHEMA} recorded: `));
assert.deepEqual(records(), []);
}
});
});
@@ -0,0 +1,79 @@
// Brackets the Runner package's `build` so server's prepare:runner-vendor can
// reuse this build instead of repeating it when nothing has changed. See
// build-inputs.mjs for what counts as an input and an output.
//
// --begin first step of the build: forgets the previous stamp and records
// the inputs the build is about to read
// --finish last step of the build, after every check: writes the stamp if
// the inputs are still the ones the build began with, with a
// manifest of every file under dist; otherwise writes nothing, so
// the next consumer rebuilds
//
// Each build chain has its own pending record, named after the shell that
// runs the chain (the parent of both phases), so two Runner builds running at
// once in one checkout do not consume each other's record.
//
// Usage: node scripts/record-build-inputs.mjs (--begin | --finish) [--package-root <dir>]
// The package root defaults to this package; the flag exists for tests that
// build a throwaway package layout.
import path from "node:path";
import { fileURLToPath } from "node:url";
import {
beginRunnerBuild,
recordRunnerBuild,
repoRootFromRunnerPackage,
runnerPackageRootFromScripts,
} from "./build-inputs.mjs";
export function parseArguments(argv) {
let packageRoot = null;
let phase = null;
for (let index = 0; index < argv.length; index += 1) {
const argument = argv[index];
if (argument === "--begin" || argument === "--finish") {
if (phase) throw new Error(`${argument} cannot be combined with --${phase}`);
phase = argument.slice(2);
continue;
}
if (argument === "--package-root") {
packageRoot = argv[index + 1];
if (!packageRoot) throw new Error("--package-root needs a directory");
index += 1;
continue;
}
throw new Error(`unknown argument ${argument}`);
}
if (!phase) throw new Error("one of --begin or --finish is required");
return { phase, packageRoot };
}
export function main(argv = process.argv.slice(2), log = console.log) {
const { phase, packageRoot: requested } = parseArguments(argv);
const packageRoot = requested ? path.resolve(requested) : runnerPackageRootFromScripts();
const repoRoot = repoRootFromRunnerPackage(packageRoot);
if (phase === "begin") {
const pending = beginRunnerBuild({ repoRoot, packageRoot });
log(
`${pending.schema} begin: digest=${pending.digest.slice(0, 12)} files=${pending.fileCount} ` +
`bytes=${pending.byteCount}`,
);
return pending;
}
const record = recordRunnerBuild({ repoRoot, packageRoot });
if (!record.recorded) {
log(`${record.code === "inputs_changed_during_build" ? "paperclip.runner-build-inputs" : record.code} not recorded: ${record.detail}`);
return record;
}
log(
`${record.schema} recorded: digest=${record.digest.slice(0, 12)} files=${record.fileCount} ` +
`bytes=${record.byteCount} outputs=${record.outputCount} stamp=${path.relative(packageRoot, record.stampPath)}`,
);
return record;
}
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
main();
}
@@ -0,0 +1,558 @@
import assert from "node:assert/strict";
import { execFileSync, spawnSync } from "node:child_process";
import { chmodSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, utimesSync, writeFileSync } from "node:fs";
import os from "node:os";
import path from "node:path";
import test from "node:test";
import { fileURLToPath } from "node:url";
import { CACHED_PATHS, HASHED_INPUTS, REQUIRED, RUNNER_PACKAGE, computeScope, markCurrent } from "../runner-release-crate-cache.mjs";
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../..");
const script = path.join(repoRoot, "scripts/runner-release-crate-cache.mjs");
const readWorkflow = (name) => readFileSync(path.join(repoRoot, ".github/workflows", name), "utf8");
const prWorkflow = readWorkflow("pr-trusted.yml");
const releaseWorkflow = readWorkflow("release-verify.yml");
// Slice one job out of a workflow, from its two-space-indented key to the next.
function job(workflow, name) {
const start = workflow.indexOf(`\n ${name}:\n`);
assert.ok(start >= 0, `workflow is missing the ${name} job`);
const body = workflow.slice(start + 1);
const header = ` ${name}:\n`.length;
const next = body.slice(header).search(/\n [a-z0-9_-]+:\n/);
return next === -1 ? body : body.slice(0, header + next + 1);
}
// Slice one step out of a job by its name, up to the next step.
function step(body, name) {
const marker = ` - name: ${name}\n`;
const start = body.indexOf(marker);
assert.ok(start >= 0, `missing step: ${name}`);
const rest = body.slice(start + marker.length);
const next = rest.search(/\n - name: /);
return marker + (next === -1 ? rest : rest.slice(0, next + 1));
}
const RELEASE_DIR = `${RUNNER_PACKAGE}/runner/target/release`;
const KEY_STEP = "Compute the Runner release crate cache key";
const RESTORE_STEP = "Restore the Runner release crate (read only)";
const MARK_STEP = "Mark the restored Runner release crate current";
const LOOKUP_STEP = "Check for this push's Runner release crate";
const BUILD_STEP = "Build the Runner release crate for its cache";
const SAVE_STEP = "Save the Runner release crate";
const RUST_RESTORE_STEP = "Restore Runner Rust dependencies (read only)";
// Every PR job that builds the Runner release binary, and the command each
// one runs cargo through. The crate must be restored and marked before it.
const READERS = [
["typecheck_release_registry", "run: pnpm run typecheck:build-gaps"],
["verify_paperclip_runner", "run: pnpm --filter @paperclipai/paperclip-runner ${{ matrix.command }}"],
["build", "run: pnpm build"],
["canary_dry_run", "- name: Release canary dry run via release.sh internal build"],
];
const WRITER_GATE = "matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha";
const expectedPathBlock = ` path: |\n${CACHED_PATHS.map((entry) => ` ${entry}\n`).join("")}`;
const expectedKey = `key: runner-release-crate-v1-\${{ steps.runner_release_crate_key.outputs.scope }}-\${{ hashFiles(${HASHED_INPUTS.map((entry) => `'${entry}'`).join(", ")}) }}`;
function assertCacheStep(body, name, { lookupOnly = false } = {}) {
assert.match(body, /uses: actions\/cache\/(restore|save)@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6\.1\.0/, name);
assert.ok(body.includes(expectedPathBlock), `${name}: the cached path list must be exactly the script's CACHED_PATHS`);
assert.ok(body.includes(expectedKey), `${name}: the key must be built from the script's scope and HASHED_INPUTS`);
// An exact match or nothing: a prefix match would restore another
// revision's artifacts and the mark step would make cargo keep them.
assert.doesNotMatch(body, /restore-keys:/, name);
assert.equal(/lookup-only: true/.test(body), lookupOnly, name);
assert.doesNotMatch(body, /enableCrossOsArchive|fail-on-cache-miss: true/, name);
}
test("the script's path list and key inputs cover what the release build of the crate reads and writes", () => {
assert.deepEqual(CACHED_PATHS, [
`${RELEASE_DIR}/.fingerprint/paperclip-runner-core-*`,
`${RELEASE_DIR}/deps/libpaperclip_runner_core-*`,
`${RELEASE_DIR}/deps/paperclip_runner_core-*.d`,
`${RELEASE_DIR}/deps/paperclip_runnerd-*`,
`${RELEASE_DIR}/paperclip-runnerd`,
`${RELEASE_DIR}/paperclip-runnerd.d`,
]);
assert.deepEqual(HASHED_INPUTS, [
`${RUNNER_PACKAGE}/rust-toolchain.toml`,
`${RUNNER_PACKAGE}/runner/Cargo.toml`,
`${RUNNER_PACKAGE}/runner/Cargo.lock`,
`${RUNNER_PACKAGE}/runner/crates/**`,
`${RUNNER_PACKAGE}/protocol/**`,
`${RUNNER_PACKAGE}/test-fixtures/**`,
`${RUNNER_PACKAGE}/test/fixtures/**`,
"tests/runner-e2e/fixtures/**",
]);
for (const input of HASHED_INPUTS) {
const fixed = input.replace(/\/\*\*$/, "");
assert.ok(statSync(path.join(repoRoot, fixed)), `${input} must exist in the repository`);
}
// The stage step copies the uplifted binary; it has to be in the set.
const stage = readFileSync(path.join(repoRoot, RUNNER_PACKAGE, "scripts/stage-runner-binary.mjs"), "utf8");
assert.match(stage, /"runner", "target", "release", executable/);
assert.ok(CACHED_PATHS.includes(`${RELEASE_DIR}/paperclip-runnerd`));
});
// Every `include_str!` / `include_bytes!` in one Rust source, with the path
// each one names. Three argument forms are resolved: a string literal
// (relative to the source file), and `concat!(...)` of string literals,
// `env!("CARGO_MANIFEST_DIR")` (the directory of the nearest Cargo.toml) and
// macro metavariables (`$name`, which make the resolved path a directory
// prefix rather than a file). Anything else is reported as unsupported, so
// a new include form cannot slip past the guard by not matching a regex.
function crateIncludes(source, text) {
const includes = [];
for (const match of text.matchAll(/include_(?:str|bytes)!\s*\(/g)) {
const start = match.index + match[0].length;
let depth = 1;
let inString = false;
let end = start;
for (; end < text.length && depth > 0; end += 1) {
const ch = text[end];
if (inString) {
if (ch === "\\") end += 1;
else if (ch === '"') inString = false;
} else if (ch === '"') inString = true;
else if (ch === "(") depth += 1;
else if (ch === ")") depth -= 1;
}
assert.equal(depth, 0, `${source}: unbalanced include at offset ${match.index}`);
const argument = text.slice(start, end - 1).trim();
includes.push({ argument, ...resolveIncludeArgument(source, argument) });
}
return includes;
}
function manifestDir(source) {
let dir = path.dirname(source);
while (!statSync(path.join(dir, "Cargo.toml"), { throwIfNoEntry: false })?.isFile()) {
const parent = path.dirname(dir);
assert.notEqual(parent, dir, `${source}: no Cargo.toml above the source`);
dir = parent;
}
return dir;
}
function resolveIncludeArgument(source, argument) {
const literal = argument.match(/^"((?:[^"\\]|\\.)*)"$/);
if (literal) return { form: "literal", resolved: path.resolve(path.dirname(source), literal[1]), exact: true };
const concat = argument.match(/^concat!\s*\(([\s\S]*)\)$/);
assert.ok(concat, `${source}: unsupported include form: ${argument}`);
const parts = splitTopLevel(concat[1]);
// A relative path is joined to the source's directory; one that starts
// with the manifest directory is one string, since its next literal
// starts with the "/" that separates the two.
let base = `${path.dirname(source)}/`;
let text = "";
let exact = true;
let form = "concat";
for (const [index, part] of parts.entries()) {
const string = part.match(/^"((?:[^"\\]|\\.)*)"$/);
if (string) {
assert.ok(exact || !/(^|\/)\.\.(\/|$)/.test(string[1]), `${source}: a path segment after a metavariable must not climb out: ${argument}`);
if (exact) text += string[1];
continue;
}
if (part === 'env!("CARGO_MANIFEST_DIR")') {
assert.equal(index, 0, `${source}: env!("CARGO_MANIFEST_DIR") must be the first part of concat!: ${argument}`);
base = manifestDir(source);
form = "concat-manifest-dir";
continue;
}
if (/^\$[A-Za-z_][A-Za-z0-9_]*$/.test(part)) {
// The fixed prefix before the first metavariable is the directory the
// macro's callers can reach; the suffix cannot climb out of it.
exact = false;
form = form === "concat-manifest-dir" ? form : "concat-metavariable";
continue;
}
assert.fail(`${source}: unsupported concat! part in include: ${part}`);
}
const joined = base + text;
const resolved = path.resolve(exact || text.endsWith("/") ? joined : path.dirname(joined));
return { form, resolved, exact };
}
// Split concat!'s arguments on the commas outside string literals.
function splitTopLevel(text) {
const parts = [];
let current = "";
let inString = false;
for (let index = 0; index < text.length; index += 1) {
const ch = text[index];
if (inString) {
current += ch;
if (ch === "\\") current += text[++index] ?? "";
else if (ch === '"') inString = false;
} else if (ch === '"') {
inString = true;
current += ch;
} else if (ch === ",") {
parts.push(current);
current = "";
} else current += ch;
}
parts.push(current);
return parts.map((part) => part.trim()).filter(Boolean);
}
const underHashedInputs = (resolved) =>
HASHED_INPUTS.map((input) => path.join(repoRoot, input.replace(/\/\*\*$/, ""))).some((root) => resolved === root || resolved.startsWith(`${root}${path.sep}`));
test("the include parser resolves literals, concat! with the manifest directory and with metavariables, and rejects other forms", () => {
const root = mkdtempSync(path.join(os.tmpdir(), "runner-release-crate-includes-"));
try {
const crate = path.join(root, "crates/demo");
mkdirSync(path.join(crate, "src/nested"), { recursive: true });
mkdirSync(path.join(root, "protocol/fixtures"), { recursive: true });
writeFileSync(path.join(crate, "Cargo.toml"), '[package]\nname = "demo"\n');
writeFileSync(path.join(root, "protocol/schema.json"), "{}");
const source = path.join(crate, "src/nested/mod.rs");
const text = [
'let a = include_str!("../../../../protocol/schema.json");',
'let b = include_bytes!(\n "../../../../protocol/schema.json"\n);',
'let c = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../protocol/", "schema.json"));',
'let d = include_str!(concat!("../../../../protocol/fixtures/", $fixture, ".json"));',
'let e = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../protocol/fixtures/", $golden, ".summary.json"));',
'let f = include_str!("a (paren) in the name.json");',
].join("\n");
const includes = crateIncludes(source, text);
assert.deepEqual(
includes.map(({ form, resolved, exact }) => [form, path.relative(root, resolved), exact]),
[
["literal", "protocol/schema.json", true],
["literal", "protocol/schema.json", true],
["concat-manifest-dir", "protocol/schema.json", true],
["concat-metavariable", "protocol/fixtures", false],
["concat-manifest-dir", "protocol/fixtures", false],
["literal", "crates/demo/src/nested/a (paren) in the name.json", true],
],
);
const rejected = [
'include_str!(concat!("../protocol/", $x, "/../../../outside.json"))',
'include_str!(concat!("../protocol/", env!("CARGO_MANIFEST_DIR")))',
'include_str!(concat!(env!("OTHER_DIR"), "/schema.json"))',
'include_str!(r"../protocol/schema.json")',
'include_str!(path_from_a_function())',
'include_str!(concat!("../protocol/", some_macro!("x")))',
];
for (const argument of rejected) {
assert.throws(() => crateIncludes(source, argument), /unsupported|must not climb out|must be the first part/, argument);
}
} finally {
rmSync(root, { recursive: true, force: true });
}
});
test("every file the crate includes by path sits under a hashed input, so an edit to it changes the key", () => {
const crateRoot = path.join(repoRoot, RUNNER_PACKAGE, "runner/crates");
const sources = [];
(function walk(dir) {
for (const entry of readdirSync(dir, { withFileTypes: true })) {
const full = path.join(dir, entry.name);
if (entry.isDirectory()) walk(full);
else if (entry.name.endsWith(".rs")) sources.push(full);
}
})(crateRoot);
assert.ok(sources.length > 10, "the crate sources must be found");
const forms = new Set();
let includes = 0;
for (const source of sources) {
for (const { argument, form, resolved, exact } of crateIncludes(source, readFileSync(source, "utf8"))) {
includes += 1;
forms.add(form);
const where = `${path.relative(repoRoot, source)} includes ${argument.replace(/\s+/g, " ")}`;
assert.ok(underHashedInputs(resolved), `${where}, which resolves to ${path.relative(repoRoot, resolved)}, outside every HASHED_INPUTS pattern`);
const info = statSync(resolved, { throwIfNoEntry: false });
assert.ok(exact ? info?.isFile() : info?.isDirectory(), `${where}: ${path.relative(repoRoot, resolved)} must exist`);
}
}
assert.ok(includes > 0, "the crate includes protocol files by path; the guard must see them");
// The crate uses every supported form today, so none of the parser's
// branches is exercised only by the synthetic test above.
assert.deepEqual([...forms].sort(), ["concat-manifest-dir", "concat-metavariable", "literal"]);
});
test("every PR reader restores the crate after the dependency cache and marks it before building", () => {
for (const [name, buildCommand] of READERS) {
const body = job(prWorkflow, name);
const positions = [RUST_RESTORE_STEP, KEY_STEP, RESTORE_STEP, MARK_STEP].map((stepName) => body.indexOf(` - name: ${stepName}\n`));
assert.ok(positions.every((position) => position >= 0), `${name}: a crate cache step is missing`);
assert.deepEqual([...positions].sort((a, b) => a - b), positions, `${name}: the crate steps must follow the dependency restore in order`);
const build = body.indexOf(buildCommand);
assert.ok(build > positions.at(-1), `${name}: the crate must be marked before ${buildCommand}`);
const key = step(body, KEY_STEP);
assert.match(key, /id: runner_release_crate_key\n/, name);
assert.match(key, /run: echo "scope=\$\(node scripts\/runner-release-crate-cache\.mjs scope\)" >> "\$GITHUB_OUTPUT"/, name);
const restore = step(body, RESTORE_STEP);
assert.match(restore, /id: runner_release_crate\n/, name);
assertCacheStep(restore, `${name} restore`);
assert.doesNotMatch(restore, /actions\/cache\/save/, name);
const mark = step(body, MARK_STEP);
assert.match(mark, /if: \$\{\{ steps\.runner_release_crate\.outputs\.cache-hit == 'true' \}\}/, name);
assert.match(mark, /run: node scripts\/runner-release-crate-cache\.mjs mark-current/, name);
}
// No other job restores or saves the crate.
const total = prWorkflow.match(/- name: Restore the Runner release crate \(read only\)\n/g)?.length ?? 0;
assert.equal(total, READERS.length);
assert.doesNotMatch(prWorkflow, /actions\/cache\/save@/);
});
test("the Verify Paperclip Runner job restores the crate only in the lanes that build it", () => {
const body = job(prWorkflow, "verify_paperclip_runner");
const lanes = [...body.matchAll(/- lane_label: (.+)\n\s+command: (.+)\n(\s+release_crate: (true)\n)?/g)].map(([, label, , , flag]) => [label, flag === "true"]);
assert.deepEqual(lanes, [
["static checks", false],
["rust", true],
["vitest 1/2", false],
["vitest 2/2", true],
]);
for (const stepName of [KEY_STEP, RESTORE_STEP]) {
assert.match(step(body, stepName), /if: \$\{\{ matrix\.release_crate == true \}\}/, stepName);
}
// A lane that skipped the restore has no cache-hit output, so the mark
// step's own condition is enough there.
assert.match(step(body, MARK_STEP), /if: \$\{\{ steps\.runner_release_crate\.outputs\.cache-hit == 'true' \}\}/);
for (const [name] of READERS.filter(([name]) => name !== "verify_paperclip_runner")) {
for (const stepName of [KEY_STEP, RESTORE_STEP]) {
assert.doesNotMatch(step(job(prWorkflow, name), stepName), /^\s+if:/m, `${name}: ${stepName} is unconditional`);
}
}
});
test("only the master Rust lane writes the crate, after a lookup, with the readers' exact path list and key", () => {
const body = job(releaseWorkflow, "verify_paperclip_runner");
const positions = ["Verify Paperclip Runner", "Warm debug dependencies for the shared Runner cache", KEY_STEP, LOOKUP_STEP, BUILD_STEP, SAVE_STEP].map((stepName) => body.indexOf(` - name: ${stepName}\n`));
assert.ok(positions.every((position) => position >= 0), "a writer step is missing");
assert.deepEqual([...positions].sort((a, b) => a - b), positions, "the writer steps must follow the checks and the debug warm-up, in order");
for (const stepName of [KEY_STEP, LOOKUP_STEP]) {
assert.match(step(body, stepName), new RegExp(`if: \\$\\{\\{ ${WRITER_GATE.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")} \\}\\}`), stepName);
}
for (const stepName of [BUILD_STEP, SAVE_STEP]) {
assert.match(
step(body, stepName),
new RegExp(`if: \\$\\{\\{ ${WRITER_GATE.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")} && steps\\.runner_release_crate\\.outputs\\.cache-hit != 'true' \\}\\}`),
stepName,
);
}
assert.match(step(body, KEY_STEP), /run: echo "scope=\$\(node scripts\/runner-release-crate-cache\.mjs scope\)" >> "\$GITHUB_OUTPUT"/);
const lookup = step(body, LOOKUP_STEP);
assert.match(lookup, /id: runner_release_crate\n/);
assert.match(lookup, /uses: actions\/cache\/restore@/);
assertCacheStep(lookup, "writer lookup", { lookupOnly: true });
// The saved artifacts come from the readers' exact command, on a crate
// cleaned of the test-profile variants the lane's checks built.
const build = step(body, BUILD_STEP);
assert.match(build, /working-directory: packages\/paperclip-runner\n/);
assert.match(build, /set -euo pipefail\n\s+cargo clean --release --manifest-path runner\/Cargo\.toml -p paperclip-runner-core\n\s+pnpm build:binary\n/);
const runnerScripts = JSON.parse(readFileSync(path.join(repoRoot, RUNNER_PACKAGE, "package.json"), "utf8")).scripts;
assert.match(runnerScripts["build:binary"], /^cargo build --release --manifest-path runner\/Cargo\.toml --locked -p paperclip-runner-core --bin paperclip-runnerd && /);
const save = step(body, SAVE_STEP);
assert.match(save, /uses: actions\/cache\/save@/);
assertCacheStep(save, "writer save");
// The dependency cache's own writer gate is the same expression.
assert.match(body, new RegExp(`save-if: \\$\\{\\{ ${WRITER_GATE.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")} \\}\\}`));
assert.equal((releaseWorkflow.match(/actions\/cache\/save@/g) ?? []).length, 1);
});
test("the scope names the platform, the compiler and the checkout root, and survives a missing rustc", () => {
const a = computeScope({ workspace: "/home/runner/_work/paperclip/paperclip", rustcVersion: "1.97.1-abcdef123-x86_64-unknown-linux-gnu" });
const b = computeScope({ workspace: "/home/runner/work/paperclip/paperclip", rustcVersion: "1.97.1-abcdef123-x86_64-unknown-linux-gnu" });
const c = computeScope({ workspace: "/home/runner/_work/paperclip/paperclip", rustcVersion: "1.97.2-0123456ab-x86_64-unknown-linux-gnu" });
assert.notEqual(a, b, "a different checkout root must select a different entry");
assert.notEqual(a, c, "a different compiler must select a different entry");
assert.equal(a, computeScope({ workspace: "/home/runner/_work/paperclip/paperclip/", rustcVersion: "1.97.1-abcdef123-x86_64-unknown-linux-gnu" }));
assert.match(a, new RegExp(`^${os.platform()}-${os.arch()}-1\\.97\\.1-abcdef123-x86_64-unknown-linux-gnu-[0-9a-f]{12}$`));
assert.doesNotMatch(computeScope({ rustcVersion: "weird version/with spaces" }), /[^A-Za-z0-9._-]/);
// The CLI reads rustc from PATH; a stand-in proves which fields it reads,
// and an empty PATH proves a missing compiler still yields a usable key.
const bin = mkdtempSync(path.join(os.tmpdir(), "runner-release-crate-rustc-"));
try {
const stub = path.join(bin, "rustc");
writeFileSync(stub, '#!/bin/sh\nprintf "rustc 1.97.1 (8bab26f4f 2026-06-30)\\nbinary: rustc\\ncommit-hash: 8bab26f4f0123456789abcdef\\ncommit-date: 2026-06-30\\nhost: x86_64-unknown-linux-gnu\\nrelease: 1.97.1\\nLLVM version: 21.1.2\\n"\n');
chmodSync(stub, 0o755);
const withStub = execFileSync(process.execPath, [script, "scope"], { encoding: "utf8", env: { ...process.env, PATH: bin, GITHUB_WORKSPACE: "/home/runner/_work/paperclip/paperclip" } }).trim();
assert.equal(withStub, computeScope({ workspace: "/home/runner/_work/paperclip/paperclip", rustcVersion: "1.97.1-8bab26f4f-x86_64-unknown-linux-gnu" }));
const without = execFileSync(process.execPath, [script, "scope"], { encoding: "utf8", env: { ...process.env, PATH: mkdtempSync(path.join(os.tmpdir(), "empty-path-")), GITHUB_WORKSPACE: "/home/runner/_work/paperclip/paperclip" } }).trim();
assert.match(without, /-no-rustc-/);
} finally {
rmSync(bin, { recursive: true, force: true });
}
});
// The tree cargo leaves behind for the crate's release lib and bin units,
// with the mtimes GitHub's cache restores: older than a fresh checkout.
function layRestoredCrate(root) {
const release = path.join(root, RELEASE_DIR);
const lib = path.join(release, ".fingerprint/paperclip-runner-core-0b6c4654c2710743");
const bin = path.join(release, ".fingerprint/paperclip-runner-core-46cce78dbc4840c2");
mkdirSync(path.join(release, "deps"), { recursive: true });
mkdirSync(lib, { recursive: true });
mkdirSync(bin, { recursive: true });
const files = {
[path.join(lib, "lib-paperclip_runner_core")]: "0123456789abcdef",
[path.join(lib, "lib-paperclip_runner_core.json")]: "{}",
[path.join(lib, "dep-lib-paperclip_runner_core")]: "dep",
[path.join(lib, "invoked.timestamp")]: "t",
[path.join(bin, "bin-paperclip-runnerd")]: "fedcba9876543210",
[path.join(bin, "bin-paperclip-runnerd.json")]: "{}",
[path.join(bin, "dep-bin-paperclip-runnerd")]: "dep",
[path.join(release, "deps/libpaperclip_runner_core-0b6c4654c2710743.rlib")]: "rlib",
[path.join(release, "deps/libpaperclip_runner_core-0b6c4654c2710743.rmeta")]: "rmeta",
[path.join(release, "deps/paperclip_runner_core-0b6c4654c2710743.d")]: "d",
[path.join(release, "deps/paperclip_runnerd-46cce78dbc4840c2")]: "ELF",
[path.join(release, "deps/paperclip_runnerd-46cce78dbc4840c2.d")]: "d",
[path.join(release, "paperclip-runnerd")]: "ELF",
[path.join(release, "paperclip-runnerd.d")]: "d",
// A dependency's artifacts sit beside the crate's and must not be touched.
[path.join(release, "deps/libserde-1234567890abcdef.rlib")]: "dep rlib",
};
const old = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000);
for (const [file, content] of Object.entries(files)) {
writeFileSync(file, content);
utimesSync(file, old, old);
}
for (const dir of [lib, bin, path.join(release, "deps"), path.join(release, ".fingerprint"), release]) utimesSync(dir, old, old);
return { release, files: Object.keys(files), old };
}
test("mark-current makes every restored artifact newer than the checkout and leaves dependencies alone", () => {
const root = mkdtempSync(path.join(os.tmpdir(), "runner-release-crate-"));
try {
const { release, files, old } = layRestoredCrate(root);
const checkout = path.join(root, RUNNER_PACKAGE, "runner/crates/runner-core/src/lib.rs");
mkdirSync(path.dirname(checkout), { recursive: true });
writeFileSync(checkout, "// checked out after the artifacts were built\n");
const checkedOutAt = statSync(checkout).mtimeMs;
const result = markCurrent({ root, now: new Date(checkedOutAt + 5000) });
assert.equal(result.action, "marked");
assert.equal(result.files, 14);
assert.equal(result.bytes, files.filter((file) => !file.includes("libserde")).reduce((sum, file) => sum + statSync(file).size, 0));
for (const file of files) {
const mtime = statSync(file).mtimeMs;
if (file.includes("libserde")) assert.equal(Math.floor(mtime / 1000), Math.floor(old.getTime() / 1000), "a dependency artifact must keep its mtime");
else assert.ok(mtime > checkedOutAt, `${path.relative(release, file)} must be newer than the checkout`);
}
for (const dir of [".fingerprint/paperclip-runner-core-0b6c4654c2710743", ".fingerprint/paperclip-runner-core-46cce78dbc4840c2"]) {
assert.ok(statSync(path.join(release, dir)).mtimeMs > checkedOutAt, `${dir} must be newer than the checkout`);
}
assert.ok(statSync(checkout).mtimeMs === checkedOutAt, "sources are never touched");
// The CLI prints the same result as one line of JSON for the job log.
const printed = JSON.parse(execFileSync(process.execPath, [script, "mark-current"], { encoding: "utf8", env: { ...process.env, GITHUB_WORKSPACE: root } }));
assert.equal(printed.schema, "paperclip.runner-release-crate.v1");
assert.equal(printed.action, "marked");
assert.equal(printed.files, 14);
} finally {
rmSync(root, { recursive: true, force: true });
}
});
// One file per REQUIRED entry, as cargo 1.97.1 lays them out for the crate's
// release lib and bin units. Measured: without any one of these cargo
// recompiles the unit (the whole crate for a lib file, a relink for a bin
// file); without any of the files in NOT_NEEDED it reports the crate fresh
// and recreates the uplifted binary from the one under deps.
const REQUIRED_FILES = {
"the lib fingerprint hash": ".fingerprint/paperclip-runner-core-0b6c4654c2710743/lib-paperclip_runner_core",
"the lib dep-info": ".fingerprint/paperclip-runner-core-0b6c4654c2710743/dep-lib-paperclip_runner_core",
"the lib rlib": "deps/libpaperclip_runner_core-0b6c4654c2710743.rlib",
"the lib rmeta": "deps/libpaperclip_runner_core-0b6c4654c2710743.rmeta",
"the bin fingerprint hash": ".fingerprint/paperclip-runner-core-46cce78dbc4840c2/bin-paperclip-runnerd",
"the bin dep-info": ".fingerprint/paperclip-runner-core-46cce78dbc4840c2/dep-bin-paperclip-runnerd",
"the bin under deps": "deps/paperclip_runnerd-46cce78dbc4840c2",
};
const NOT_NEEDED = [
".fingerprint/paperclip-runner-core-0b6c4654c2710743/lib-paperclip_runner_core.json",
".fingerprint/paperclip-runner-core-0b6c4654c2710743/invoked.timestamp",
".fingerprint/paperclip-runner-core-46cce78dbc4840c2/bin-paperclip-runnerd.json",
"deps/paperclip_runner_core-0b6c4654c2710743.d",
"deps/paperclip_runnerd-46cce78dbc4840c2.d",
"paperclip-runnerd",
"paperclip-runnerd.d",
];
test("mark-current discards a restore that lacks any file cargo needs for either unit, naming the file, so cargo sees a plain miss", () => {
assert.deepEqual(REQUIRED.map(({ label }) => label), Object.keys(REQUIRED_FILES), "every REQUIRED entry has a missing-file case");
for (const [label, file] of Object.entries(REQUIRED_FILES)) {
const root = mkdtempSync(path.join(os.tmpdir(), "runner-release-crate-"));
try {
const { release } = layRestoredCrate(root);
rmSync(path.join(release, file));
const result = markCurrent({ root });
assert.equal(result.action, "discarded", `${file} missing must discard the restore`);
assert.equal(result.reason, `incomplete: ${label} is missing`);
assert.ok(result.removed > 0, label);
assert.deepEqual(readdirSync(path.join(release, ".fingerprint")), [], label);
assert.deepEqual(readdirSync(path.join(release, "deps")), ["libserde-1234567890abcdef.rlib"], `${label}: only the crate's own artifacts are removed`);
assert.deepEqual(readdirSync(release).sort(), [".fingerprint", "deps"], label);
} finally {
rmSync(root, { recursive: true, force: true });
}
}
// The bin's dep-info alone does not stand in for the bin: the glob that
// finds the binary under deps must not be satisfied by the `.d` beside it.
const root = mkdtempSync(path.join(os.tmpdir(), "runner-release-crate-"));
try {
const { release } = layRestoredCrate(root);
rmSync(path.join(release, "deps/paperclip_runnerd-46cce78dbc4840c2"));
assert.ok(statSync(path.join(release, "deps/paperclip_runnerd-46cce78dbc4840c2.d")).isFile());
assert.equal(markCurrent({ root }).reason, "incomplete: the bin under deps is missing");
} finally {
rmSync(root, { recursive: true, force: true });
}
// A file cargo neither reads nor checks for may be absent without costing
// a rebuild, so its absence must not cost one here either.
for (const file of NOT_NEEDED) {
const intact = mkdtempSync(path.join(os.tmpdir(), "runner-release-crate-"));
try {
const { release } = layRestoredCrate(intact);
rmSync(path.join(release, file));
const result = markCurrent({ root: intact });
assert.equal(result.action, "marked", `${file} missing must not discard the restore`);
assert.equal(result.files, 13, file);
} finally {
rmSync(intact, { recursive: true, force: true });
}
}
// Nothing restored at all is reported the same way and exits cleanly.
const empty = mkdtempSync(path.join(os.tmpdir(), "runner-release-crate-empty-"));
try {
const printed = JSON.parse(execFileSync(process.execPath, [script, "mark-current"], { encoding: "utf8", env: { ...process.env, GITHUB_WORKSPACE: empty } }));
assert.equal(printed.action, "discarded");
assert.equal(printed.reason, "incomplete: the lib fingerprint hash is missing");
assert.equal(printed.removed, 0);
} finally {
rmSync(empty, { recursive: true, force: true });
}
});
test("the CLI prints the lists the workflows embed", () => {
const paths = execFileSync(process.execPath, [script, "paths"], { encoding: "utf8" }).trim().split("\n");
assert.deepEqual(paths, CACHED_PATHS);
const inputs = execFileSync(process.execPath, [script, "hashed-inputs"], { encoding: "utf8" }).trim().split("\n");
assert.deepEqual(inputs, HASHED_INPUTS);
const usage = spawnSync(process.execPath, [script, "nonsense"], { encoding: "utf8" });
assert.equal(usage.status, 2);
assert.match(usage.stderr, /usage: runner-release-crate-cache\.mjs/);
});
+210
View File
@@ -0,0 +1,210 @@
#!/usr/bin/env node
// The Runner's own crate is the one thing the shared Rust dependency cache
// leaves out: rust-cache drops workspace crates when it saves, and a fresh
// checkout would invalidate cargo's mtime fingerprints for them anyway. So
// every CI build compiled `paperclip-runner-core` in release mode from
// scratch (28-49s per run on 2026-10-11) although 221 of the 239 master
// commits of the preceding week left its inputs untouched.
//
// This script backs a second, content-addressed cache that holds only that
// crate's release artifacts (about 70 MB). The master writer saves them
// under a key made from the compiler, the checkout root and a hash of every
// input the build reads; a pull request whose merge tree carries
// byte-identical inputs restores them. Cargo judges a workspace crate fresh
// only while no source is newer than its dep-info file, and GitHub's cache
// keeps the artifacts' original mtimes, so `mark-current` touches the
// restored set after the checkout. Any input change selects a different key,
// the restore misses, and cargo compiles exactly as it did before.
//
// Subcommands:
// paths print the cached paths, one per line (repo-relative globs)
// hashed-inputs print the hashFiles() patterns that make up the key
// scope print the compiler/checkout part of the key
// mark-current make a restored artifact set newer than the checkout, or
// discard an incomplete one so cargo rebuilds instead
import { execFileSync } from "node:child_process";
import { createHash } from "node:crypto";
import { existsSync, readdirSync, rmSync, statSync, utimesSync, lstatSync } from "node:fs";
import os from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
export const RUNNER_PACKAGE = "packages/paperclip-runner";
const RELEASE_DIR = `${RUNNER_PACKAGE}/runner/target/release`;
// Everything cargo writes for the crate's release lib and bin units:
// fingerprints (both units share the package-named directory prefix), the
// lib's rlib/rmeta and dep-info, the bin and its dep-info under deps, and the
// uplifted executable the staging step copies into dist/bin.
export const CACHED_PATHS = [
`${RELEASE_DIR}/.fingerprint/paperclip-runner-core-*`,
`${RELEASE_DIR}/deps/libpaperclip_runner_core-*`,
`${RELEASE_DIR}/deps/paperclip_runner_core-*.d`,
`${RELEASE_DIR}/deps/paperclip_runnerd-*`,
`${RELEASE_DIR}/paperclip-runnerd`,
`${RELEASE_DIR}/paperclip-runnerd.d`,
];
// Every file the release build of the crate reads, as hashFiles() patterns:
// the toolchain pin, the manifests and lockfile, the crate sources, and the
// protocol schemas and fixtures the sources include with include_str!. The
// three fixture trees are read only by test builds, but hashing them costs
// nothing and keeps the include_str! guard test simple: every include in the
// crate, whatever its form, must land under one of these patterns.
export const HASHED_INPUTS = [
`${RUNNER_PACKAGE}/rust-toolchain.toml`,
`${RUNNER_PACKAGE}/runner/Cargo.toml`,
`${RUNNER_PACKAGE}/runner/Cargo.lock`,
`${RUNNER_PACKAGE}/runner/crates/**`,
`${RUNNER_PACKAGE}/protocol/**`,
`${RUNNER_PACKAGE}/test-fixtures/**`,
`${RUNNER_PACKAGE}/test/fixtures/**`,
"tests/runner-e2e/fixtures/**",
];
// A restored set is usable only when both units are complete: everything
// cargo's freshness check reads for a unit (the fingerprint hash and the
// dep-info it compares source mtimes against) and every output it checks for
// (the lib's rlib and rmeta, the bin under deps). Measured with cargo 1.97.1:
// without any one of these it recompiles the unit; without the `.json`
// fingerprint records, rustc's `.d` files or the uplifted binary and its `.d`
// it reports the crate fresh and recreates the uplifted copy from deps.
// Anything less than this set is discarded, with the missing file named in
// the job log, so cargo sees a plain miss rather than a half-restored crate.
export const REQUIRED = [
{ label: "the lib fingerprint hash", glob: `${RELEASE_DIR}/.fingerprint/paperclip-runner-core-*/lib-paperclip_runner_core` },
{ label: "the lib dep-info", glob: `${RELEASE_DIR}/.fingerprint/paperclip-runner-core-*/dep-lib-paperclip_runner_core` },
{ label: "the lib rlib", glob: `${RELEASE_DIR}/deps/libpaperclip_runner_core-*.rlib` },
{ label: "the lib rmeta", glob: `${RELEASE_DIR}/deps/libpaperclip_runner_core-*.rmeta` },
{ label: "the bin fingerprint hash", glob: `${RELEASE_DIR}/.fingerprint/paperclip-runner-core-*/bin-paperclip-runnerd` },
{ label: "the bin dep-info", glob: `${RELEASE_DIR}/.fingerprint/paperclip-runner-core-*/dep-bin-paperclip-runnerd` },
{ label: "the bin under deps", glob: `${RELEASE_DIR}/deps/paperclip_runnerd-*`, exclude: /\.d$/ },
];
const SCHEMA = "paperclip.runner-release-crate.v1";
function keySafe(value) {
return String(value).replace(/[^A-Za-z0-9._-]+/g, "_");
}
/**
* The compiler and checkout part of the cache key. The compiler decides the
* artifacts' metadata hashes; the checkout root is baked into cargo's
* dep-info for the protocol files the sources include by relative path, so an
* entry written under one root is useless under another (the RunsOn fleets
* check out under /home/runner/_work, GitHub-hosted runners under
* /home/runner/work). Hashing the root into the key turns that into a plain
* miss instead of a restore cargo then throws away.
*/
export function computeScope({ workspace = process.env.GITHUB_WORKSPACE ?? process.cwd(), rustcVersion = readRustcVersion() } = {}) {
const root = createHash("sha256").update(path.resolve(workspace)).digest("hex").slice(0, 12);
return `${keySafe(os.platform())}-${keySafe(os.arch())}-${keySafe(rustcVersion)}-${root}`;
}
function readRustcVersion(env = process.env) {
let output;
try {
output = execFileSync("rustc", ["--version", "--verbose"], { encoding: "utf8", env });
} catch {
return "no-rustc";
}
const field = (name) => output.match(new RegExp(`^${name}: (.+)$`, "m"))?.[1]?.trim() ?? "unknown";
return `${field("release")}-${field("commit-hash").slice(0, 9)}-${field("host")}`;
}
function expandGlob(root, pattern) {
// Patterns only ever use `*` inside a single path segment.
const segments = pattern.split("/");
let current = [root];
for (const segment of segments) {
const next = [];
for (const base of current) {
if (!segment.includes("*")) {
const candidate = path.join(base, segment);
if (existsSync(candidate)) next.push(candidate);
continue;
}
let entries;
try {
entries = readdirSync(base);
} catch {
continue;
}
const matcher = new RegExp(`^${segment.split("*").map((part) => part.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")).join(".*")}$`);
for (const entry of entries) {
if (matcher.test(entry)) next.push(path.join(base, entry));
}
}
current = next;
}
return current.sort();
}
function walk(target, visit) {
const info = lstatSync(target);
visit(target, info);
if (info.isDirectory()) {
for (const entry of readdirSync(target)) walk(path.join(target, entry), visit);
}
}
/**
* Touch every restored artifact so cargo's freshness check, which compares
* each source's mtime against the unit's dep-info, sees the checkout as
* older than the build. When the set is incomplete, remove whatever was
* restored and report it: cargo then compiles the crate as on a miss.
*/
export function markCurrent({ root = process.env.GITHUB_WORKSPACE ?? process.cwd(), now = new Date() } = {}) {
const missing = REQUIRED.find(({ glob, exclude }) => expandGlob(root, glob).filter((entry) => !exclude?.test(entry)).length === 0);
const matched = CACHED_PATHS.flatMap((pattern) => expandGlob(root, pattern));
if (missing) {
for (const target of matched) rmSync(target, { recursive: true, force: true });
return { schema: SCHEMA, action: "discarded", reason: `incomplete: ${missing.label} is missing`, removed: matched.length };
}
let files = 0;
let bytes = 0;
const seen = new Set();
for (const target of matched) {
walk(target, (entry, info) => {
if (info.isSymbolicLink()) return;
utimesSync(entry, now, now);
if (info.isFile()) {
files += 1;
// The uplifted binary is a hard link to the one under deps.
const { ino, size } = statSync(entry);
if (!seen.has(ino)) {
seen.add(ino);
bytes += size;
}
}
});
}
return { schema: SCHEMA, action: "marked", files, bytes };
}
function main(argv) {
const [command] = argv;
switch (command) {
case "paths":
process.stdout.write(`${CACHED_PATHS.join("\n")}\n`);
return 0;
case "hashed-inputs":
process.stdout.write(`${HASHED_INPUTS.join("\n")}\n`);
return 0;
case "scope":
process.stdout.write(`${computeScope()}\n`);
return 0;
case "mark-current": {
const result = markCurrent();
process.stdout.write(`${JSON.stringify(result)}\n`);
return 0;
}
default:
process.stderr.write("usage: runner-release-crate-cache.mjs <paths|hashed-inputs|scope|mark-current>\n");
return 2;
}
}
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
process.exitCode = main(process.argv.slice(2));
}
+1 -1
View File
@@ -40,7 +40,7 @@
"postpack": "rm -rf ui-dist",
"clean": "rm -rf dist",
"start": "node dist/index.js",
"prepare:runner-vendor": "pnpm --filter @paperclipai/paperclip-runner build",
"prepare:runner-vendor": "node scripts/prepare-runner-vendor.mjs",
"typecheck": "pnpm run prepare:runner-vendor && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && tsc --noEmit"
},
"dependencies": {
+143
View File
@@ -0,0 +1,143 @@
// Makes sure packages/paperclip-runner/dist is a current build before the
// server build vendors it, rebuilding the Runner only when that is needed.
//
// The server build (and typecheck) used to run the Runner's full `build`
// unconditionally here. That is what lets a standalone
// `pnpm --filter @paperclipai/server build` produce a current vendored Runner
// on its own, which the Dockerfile relies on. But inside `pnpm -r build` the
// Runner has just been built by the same command, and repeating it cost
// 12-16s on the CI Build job's critical path (six master runs, 2026-10-11)
// and the same again in every other `pnpm build`.
//
// The Runner's `build` now records a digest of its inputs, captured before its
// first step and confirmed after its last, together with a manifest of every
// file it left under dist (packages/paperclip-runner/scripts/build-inputs.mjs).
// This step recomputes the digest and reuses the build when the stamp matches
// and every recorded output is still there at its recorded size; anything else
// runs the full Runner build exactly as before. The standalone guarantee
// therefore holds: a fresh checkout, a changed input, a deleted or truncated
// output, or an edit that landed while the build ran always rebuilds. Set
// PAPERCLIP_RUNNER_VENDOR_REBUILD=1 to rebuild regardless of the stamp.
//
// Usage: node scripts/prepare-runner-vendor.mjs [--repo-root <dir>]
// The repo root defaults to this checkout; the flag exists for tests that
// build a throwaway workspace layout.
import { spawnSync } from "node:child_process";
import { performance } from "node:perf_hooks";
import path from "node:path";
import { fileURLToPath } from "node:url";
import {
assessRunnerBuild,
removeBuildInputsStamp,
} from "../../packages/paperclip-runner/scripts/build-inputs.mjs";
/** The command the server build has always run here; the rebuild path is unchanged. */
export const RUNNER_VENDOR_BUILD_COMMAND = "pnpm --filter @paperclipai/paperclip-runner build";
export const FORCE_REBUILD_VARIABLE = "PAPERCLIP_RUNNER_VENDOR_REBUILD";
export const TIMING_SCHEMA = "paperclip.runner-vendor.timing.v1";
export function decideRunnerVendorAction({ env, assessment }) {
if (env[FORCE_REBUILD_VARIABLE] === "1") {
return { action: "rebuild", code: "forced", detail: `${FORCE_REBUILD_VARIABLE}=1` };
}
if (assessment.current) {
return { action: "reuse", code: assessment.code, detail: assessment.detail };
}
return { action: "rebuild", code: assessment.code, detail: assessment.detail };
}
/**
* The pnpm that is running this script, the same rule as the CLI's
* resolvePnpmInstallInvocation: pnpm sets npm_execpath to its own entry, which
* is a JavaScript file for a normal install (run through this Node) or a native
* executable for a standalone pnpm (run directly). Outside a pnpm script,
* `pnpm` on PATH is the only option. The pnpm entry is not always on PATH in a
* script (corepack, for one), so this cannot just spawn `pnpm`.
*/
export function resolvePnpmInvocation(env = process.env, nodeExecPath = process.execPath) {
const npmExecPath = env.npm_execpath?.trim();
if (npmExecPath && npmExecPath.toLowerCase().includes("pnpm")) {
if (/\.(cjs|mjs|js)$/i.test(npmExecPath)) {
return { command: nodeExecPath, argsPrefix: [npmExecPath] };
}
return { command: npmExecPath, argsPrefix: [] };
}
return { command: "pnpm", argsPrefix: [] };
}
export function runBuildCommand(command, { cwd, env = process.env }) {
const [tool, ...args] = command.split(" ");
if (tool !== "pnpm") throw new Error(`unexpected build command ${command}`);
const pnpm = resolvePnpmInvocation(env);
const result = spawnSync(pnpm.command, [...pnpm.argsPrefix, ...args], {
cwd,
env,
stdio: "inherit",
// A bare `pnpm` on Windows is pnpm.cmd, which needs the shell to resolve.
shell: pnpm.command === "pnpm" && process.platform === "win32",
});
if (result.error) throw result.error;
return result.status ?? 1;
}
/**
* Returns the exit status for the step. `run` is injectable so tests can
* observe the command and the stamp state at the moment it runs.
*/
export function prepareRunnerVendor({
repoRoot,
env = process.env,
run = runBuildCommand,
log = console.log,
now = () => performance.now(),
}) {
const packageRoot = path.join(repoRoot, "packages", "paperclip-runner");
const checkStart = now();
const assessment = assessRunnerBuild({ repoRoot, packageRoot });
const decision = decideRunnerVendorAction({ env, assessment });
const checkMs = Math.round(now() - checkStart);
if (decision.action === "reuse") {
log(`prepare:runner-vendor: reusing the current Runner build (${decision.detail})`);
log(`${TIMING_SCHEMA},action:reuse,reason:${decision.code},check_ms:${checkMs},build_ms:0`);
return 0;
}
log(`prepare:runner-vendor: rebuilding the Runner (${decision.detail})`);
// A build that stops partway must not leave a stamp that vouches for it. The
// Runner's own build removes the stamp in its first step and writes a fresh
// one in its last; removing it here as well covers a build command that
// fails before its first step runs.
removeBuildInputsStamp(assessment.stampPath);
const buildStart = now();
const status = run(RUNNER_VENDOR_BUILD_COMMAND, { cwd: path.join(repoRoot, "server"), env });
const buildMs = Math.round(now() - buildStart);
log(
`${TIMING_SCHEMA},action:rebuild,reason:${decision.code},check_ms:${checkMs},build_ms:${buildMs},status:${status}`,
);
return status;
}
export function parseArguments(argv) {
let repoRoot = null;
for (let index = 0; index < argv.length; index += 1) {
const argument = argv[index];
if (argument === "--repo-root") {
repoRoot = argv[index + 1];
if (!repoRoot) throw new Error("--repo-root needs a directory");
index += 1;
continue;
}
throw new Error(`unknown argument ${argument}`);
}
return { repoRoot };
}
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
const { repoRoot: requested } = parseArguments(process.argv.slice(2));
const repoRoot = requested
? path.resolve(requested)
: path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../..");
process.exitCode = prepareRunnerVendor({ repoRoot });
}
@@ -0,0 +1,312 @@
import { execFile } from "node:child_process";
import { chmodSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { promisify } from "node:util";
import { fileURLToPath } from "node:url";
import { afterEach, describe, expect, it } from "vitest";
import {
assessRunnerBuild,
beginRunnerBuild,
buildInputsStampPath,
readBuildInputsStamp,
recordRunnerBuild,
} from "../../packages/paperclip-runner/scripts/build-inputs.mjs";
import {
FORCE_REBUILD_VARIABLE,
RUNNER_VENDOR_BUILD_COMMAND,
TIMING_SCHEMA,
decideRunnerVendorAction,
prepareRunnerVendor,
resolvePnpmInvocation,
} from "./prepare-runner-vendor.mjs";
const execFileAsync = promisify(execFile);
const prepareScript = fileURLToPath(new URL("./prepare-runner-vendor.mjs", import.meta.url));
const recordScript = fileURLToPath(
new URL("../../packages/paperclip-runner/scripts/record-build-inputs.mjs", import.meta.url),
);
const executable = process.platform === "win32" ? "paperclip-runnerd.exe" : "paperclip-runnerd";
const roots = [];
function writeTree(root, files) {
for (const [relative, content] of Object.entries(files)) {
const absolute = path.join(root, ...relative.split("/"));
mkdirSync(path.dirname(absolute), { recursive: true });
writeFileSync(absolute, content);
}
}
/** A throwaway workspace whose Runner is already built, with no stamp yet. */
function makeRepo() {
const repoRoot = mkdtempSync(path.join(tmpdir(), "prepare-runner-vendor-"));
roots.push(repoRoot);
writeTree(repoRoot, {
"package.json": "{}\n",
"pnpm-lock.yaml": "lockfileVersion: 9\n",
"pnpm-workspace.yaml": "packages:\n - packages/*\n - server\n",
"tsconfig.base.json": "{}\n",
"patches/acpx@0.13.1.patch": "--- a\n+++ b\n",
"server/package.json": '{"name":"@paperclipai/server"}\n',
"packages/paperclip-runner/package.json": '{"name":"@paperclipai/paperclip-runner"}\n',
"packages/paperclip-runner/src/index.ts": "export const one = 1;\n",
"packages/paperclip-runner/dist/index.js": "export const one = 1;\n",
"packages/paperclip-runner/dist/live/index.js": "export {};\n",
"packages/paperclip-runner/dist/testing.js": "export {};\n",
"packages/paperclip-runner/dist/cli/acpx-runtime-sidecar.cjs": "module.exports = {};\n",
[`packages/paperclip-runner/dist/bin/${executable}`]: "binary\n",
"packages/paperclip-eval-kernel/src/index.ts": "export const kernel = 1;\n",
});
const packageRoot = path.join(repoRoot, "packages", "paperclip-runner");
return { repoRoot, packageRoot };
}
/** What the Runner's build does around its steps when nothing changes meanwhile. */
function recordCurrentBuild({ repoRoot, packageRoot }) {
beginRunnerBuild({ repoRoot, packageRoot });
return recordRunnerBuild({ repoRoot, packageRoot });
}
afterEach(() => {
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
});
describe("decideRunnerVendorAction", () => {
const current = { current: true, code: "inputs_unchanged", detail: "unchanged" };
const stale = { current: false, code: "inputs_changed", detail: "changed" };
it("reuses a current build and rebuilds a stale one, carrying the assessment's reason", () => {
expect(decideRunnerVendorAction({ env: {}, assessment: current })).toEqual({
action: "reuse",
code: "inputs_unchanged",
detail: "unchanged",
});
expect(decideRunnerVendorAction({ env: {}, assessment: stale })).toEqual({
action: "rebuild",
code: "inputs_changed",
detail: "changed",
});
});
it("rebuilds a current build when the operator forces it", () => {
expect(decideRunnerVendorAction({ env: { [FORCE_REBUILD_VARIABLE]: "1" }, assessment: current })).toEqual({
action: "rebuild",
code: "forced",
detail: `${FORCE_REBUILD_VARIABLE}=1`,
});
// Only the documented value forces; an unrelated value does not.
expect(decideRunnerVendorAction({ env: { [FORCE_REBUILD_VARIABLE]: "0" }, assessment: current }).action).toBe(
"reuse",
);
});
});
describe("resolvePnpmInvocation", () => {
it("runs the pnpm that is running the script, through Node for a JavaScript entry", () => {
expect(resolvePnpmInvocation({ npm_execpath: "/corepack/pnpm/9.15.4/bin/pnpm.cjs" }, "/usr/bin/node")).toEqual({
command: "/usr/bin/node",
argsPrefix: ["/corepack/pnpm/9.15.4/bin/pnpm.cjs"],
});
expect(resolvePnpmInvocation({ npm_execpath: "/opt/pnpm/pnpm" }, "/usr/bin/node")).toEqual({
command: "/opt/pnpm/pnpm",
argsPrefix: [],
});
expect(resolvePnpmInvocation({ npm_execpath: "C:\\pnpm\\pnpm.exe" }, "node.exe")).toEqual({
command: "C:\\pnpm\\pnpm.exe",
argsPrefix: [],
});
});
it("falls back to pnpm on PATH outside a pnpm script or under another package manager", () => {
expect(resolvePnpmInvocation({}, "/usr/bin/node")).toEqual({ command: "pnpm", argsPrefix: [] });
expect(resolvePnpmInvocation({ npm_execpath: "" }, "/usr/bin/node")).toEqual({ command: "pnpm", argsPrefix: [] });
expect(resolvePnpmInvocation({ npm_execpath: "/usr/lib/node_modules/npm/bin/npm-cli.js" }, "/usr/bin/node")).toEqual({
command: "pnpm",
argsPrefix: [],
});
});
});
describe("prepareRunnerVendor", () => {
it("still runs the Runner build the server build has always run", () => {
// The Dockerfile's standalone `pnpm --filter @paperclipai/server build`
// depends on this exact command producing a current vendored Runner.
expect(RUNNER_VENDOR_BUILD_COMMAND).toBe("pnpm --filter @paperclipai/paperclip-runner build");
});
it("skips the rebuild when the recorded build is current and reports the check time", () => {
const { repoRoot, packageRoot } = makeRepo();
recordCurrentBuild({ repoRoot, packageRoot });
const runs = [];
const lines = [];
const status = prepareRunnerVendor({
repoRoot,
env: {},
run: (...args) => {
runs.push(args);
return 0;
},
log: (line) => lines.push(line),
});
expect(status).toBe(0);
expect(runs).toEqual([]);
expect(lines[0]).toMatch(/^prepare:runner-vendor: reusing the current Runner build \(inputs unchanged/);
expect(lines[1]).toMatch(
new RegExp(`^${TIMING_SCHEMA},action:reuse,reason:inputs_unchanged,check_ms:\\d+,build_ms:0$`),
);
expect(readBuildInputsStamp(buildInputsStampPath(packageRoot))).not.toBeNull();
});
it("rebuilds when the inputs changed, removing the stale stamp before the build starts", () => {
const { repoRoot, packageRoot } = makeRepo();
recordCurrentBuild({ repoRoot, packageRoot });
writeFileSync(path.join(packageRoot, "src", "index.ts"), "export const one = 2;\n");
const stampPath = buildInputsStampPath(packageRoot);
const observed = [];
const lines = [];
const env = { npm_execpath: "/corepack/pnpm.cjs" };
const status = prepareRunnerVendor({
repoRoot,
env,
run: (command, options) => {
observed.push({ command, ...options, stampPresent: readBuildInputsStamp(stampPath) !== null });
return 0;
},
log: (line) => lines.push(line),
});
expect(status).toBe(0);
expect(observed).toEqual([
{ command: RUNNER_VENDOR_BUILD_COMMAND, cwd: path.join(repoRoot, "server"), env, stampPresent: false },
]);
expect(lines[0]).toMatch(/^prepare:runner-vendor: rebuilding the Runner \(inputs changed/);
expect(lines[1]).toMatch(
new RegExp(`^${TIMING_SCHEMA},action:rebuild,reason:inputs_changed,check_ms:\\d+,build_ms:\\d+,status:0$`),
);
});
it("rebuilds when a file the build left under dist is gone, even one no consumer imports", () => {
const { repoRoot, packageRoot } = makeRepo();
recordCurrentBuild({ repoRoot, packageRoot });
rmSync(path.join(packageRoot, "dist", "cli", "acpx-runtime-sidecar.cjs"));
let runs = 0;
const lines = [];
const status = prepareRunnerVendor({
repoRoot,
env: {},
run: () => {
runs += 1;
return 0;
},
log: (line) => lines.push(line),
});
expect(status).toBe(0);
expect(runs).toBe(1);
expect(lines[0]).toMatch(
/^prepare:runner-vendor: rebuilding the Runner \(build output dist\/cli\/acpx-runtime-sidecar\.cjs is missing since /,
);
expect(lines[1]).toMatch(/,action:rebuild,reason:outputs_changed,/);
});
it("returns the build's failure status and leaves no stamp behind", () => {
const { repoRoot, packageRoot } = makeRepo();
const lines = [];
const status = prepareRunnerVendor({
repoRoot,
env: {},
run: () => 3,
log: (line) => lines.push(line),
});
expect(status).toBe(3);
expect(lines[0]).toMatch(/^prepare:runner-vendor: rebuilding the Runner \(no build stamp/);
expect(lines[1]).toMatch(/,action:rebuild,reason:no_stamp,.*,status:3$/);
expect(readBuildInputsStamp(buildInputsStampPath(packageRoot))).toBeNull();
});
it("rebuilds a current build when PAPERCLIP_RUNNER_VENDOR_REBUILD=1", () => {
const { repoRoot, packageRoot } = makeRepo();
recordCurrentBuild({ repoRoot, packageRoot });
let runs = 0;
const lines = [];
const status = prepareRunnerVendor({
repoRoot,
env: { [FORCE_REBUILD_VARIABLE]: "1" },
run: () => {
runs += 1;
return 0;
},
log: (line) => lines.push(line),
});
expect(status).toBe(0);
expect(runs).toBe(1);
expect(lines[1]).toMatch(/,action:rebuild,reason:forced,/);
});
it.skipIf(process.platform === "win32")(
"end to end: the first server build rebuilds the Runner, the second reuses it, a change or a lost output rebuilds again",
async () => {
const { repoRoot, packageRoot } = makeRepo();
// A stand-in pnpm that behaves like the Runner build: it records how it
// was called, then brackets its (empty) step list with the same two
// record-build-inputs.mjs calls the real build makes.
const binDir = path.join(repoRoot, "bin");
const argvLog = path.join(repoRoot, "pnpm-argv.log");
mkdirSync(binDir);
const pnpm = path.join(binDir, "pnpm");
writeFileSync(
pnpm,
`${[
"#!/bin/sh",
"set -e",
`printf '%s\\n' "$*" >> "${argvLog}"`,
`"${process.execPath}" "${recordScript}" --begin --package-root "${packageRoot}" > /dev/null`,
`"${process.execPath}" "${recordScript}" --finish --package-root "${packageRoot}" > /dev/null`,
].join("\n")}\n`,
);
chmodSync(pnpm, 0o755);
const env = { ...process.env, PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}` };
delete env[FORCE_REBUILD_VARIABLE];
// The suite itself may run under pnpm; the script must then take the
// PATH fallback so the stand-in, not the real pnpm, receives the build.
delete env.npm_execpath;
const run = () => execFileAsync(process.execPath, [prepareScript, "--repo-root", repoRoot], { env });
const calls = () => readFileSync(argvLog, "utf8").split("\n").filter(Boolean);
const first = await run();
expect(calls()).toEqual(["--filter @paperclipai/paperclip-runner build"]);
expect(first.stdout).toMatch(/action:rebuild,reason:no_stamp,/);
expect(assessRunnerBuild({ repoRoot, packageRoot }).current).toBe(true);
const second = await run();
expect(calls()).toHaveLength(1);
expect(second.stdout).toMatch(/action:reuse,reason:inputs_unchanged,/);
writeFileSync(path.join(packageRoot, "src", "index.ts"), "export const one = 2;\n");
const third = await run();
expect(calls()).toHaveLength(2);
expect(third.stdout).toMatch(/action:rebuild,reason:inputs_changed,/);
rmSync(path.join(packageRoot, "dist", "cli", "acpx-runtime-sidecar.cjs"));
const fourth = await run();
expect(calls()).toHaveLength(3);
expect(fourth.stdout).toMatch(/action:rebuild,reason:outputs_changed,/);
// The stand-in build does not recreate the helper, so the new stamp's
// manifest no longer lists it and the next run reuses the build.
const fifth = await run();
expect(calls()).toHaveLength(3);
expect(fifth.stdout).toMatch(/action:reuse,reason:inputs_unchanged,/);
},
);
});
@@ -0,0 +1,89 @@
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import { removeTestHome } from "./remove-test-home.js";
const homes: string[] = [];
function makeHome(): string {
const home = fs.mkdtempSync(path.join(os.tmpdir(), "remove-test-home-"));
homes.push(home);
return home;
}
/**
* The tree a native session leaves behind: a runtime-context bundle whose
* files are read-only and whose directories are read-only too, the way
* `makeDirectoriesReadOnly` publishes it under the instance root.
*/
function leaveReadOnlyBundle(home: string): string {
const bundle = path.join(home, "instances", "default", "runtime-context-assets", "bundles", "digest");
const nested = path.join(bundle, "skills", "assigned");
fs.mkdirSync(nested, { recursive: true });
fs.writeFileSync(path.join(bundle, "AGENTS.md"), "# instructions\n", { mode: 0o444 });
fs.writeFileSync(path.join(nested, "SKILL.md"), "# skill\n", { mode: 0o444 });
fs.chmodSync(nested, 0o555);
fs.chmodSync(path.dirname(nested), 0o555);
fs.chmodSync(bundle, 0o555);
return bundle;
}
afterEach(() => {
for (const home of homes.splice(0)) {
if (fs.existsSync(home)) {
// Leave nothing behind even if an assertion failed partway.
fs.chmodSync(home, 0o700);
removeTestHome(home);
}
}
});
describe("removeTestHome", () => {
it("removes a home whose runtime-context bundle directories are read-only", () => {
const home = makeHome();
const bundle = leaveReadOnlyBundle(home);
fs.writeFileSync(path.join(home, "instances", "default", "paperclip.json"), "{}\n");
// The plain removal the setup file used to run is what fails in CI.
if (process.platform !== "win32" && process.getuid?.() !== 0) {
expect(() => fs.rmSync(home, { recursive: true, force: true })).toThrow(/EACCES|EPERM/);
expect(fs.existsSync(bundle)).toBe(true);
}
removeTestHome(home);
expect(fs.existsSync(home)).toBe(false);
});
it("removes an ordinary home and tolerates one that is already gone", () => {
const home = makeHome();
fs.mkdirSync(path.join(home, "instances", "default"), { recursive: true });
fs.writeFileSync(path.join(home, "instances", "default", "paperclip.json"), "{}\n");
removeTestHome(home);
expect(fs.existsSync(home)).toBe(false);
expect(() => removeTestHome(home)).not.toThrow();
});
// Windows keeps no Unix mode bits (chmod only toggles the read-only
// attribute, so the 0o555 assertion cannot hold) and a directory symlink
// can need elevated privileges there. POSIX-only, like the suite's other
// permission tests.
it.skipIf(process.platform === "win32")("does not follow a symlink out of the home while making directories writable", () => {
const home = makeHome();
const outside = makeHome();
const guarded = path.join(outside, "guarded");
fs.mkdirSync(guarded);
fs.writeFileSync(path.join(guarded, "keep.txt"), "keep\n");
fs.chmodSync(guarded, 0o555);
leaveReadOnlyBundle(home);
fs.symlinkSync(outside, path.join(home, "link-out"), "dir");
removeTestHome(home);
expect(fs.existsSync(home)).toBe(false);
expect(fs.existsSync(path.join(guarded, "keep.txt"))).toBe(true);
expect(fs.statSync(guarded).mode & 0o777).toBe(0o555);
fs.chmodSync(guarded, 0o700);
});
});
+57
View File
@@ -0,0 +1,57 @@
import fs from "node:fs";
import path from "node:path";
/**
* Removes a per-worker test home at worker exit.
*
* A plain recursive `rmSync` is not enough: a native session materializes its
* runtime-context bundles under the instance root and makes every directory
* in them read-only (`makeDirectoriesReadOnly` in
* services/native-runtime/runtime-context.ts), and a run that is still
* finishing when the worker exits can leave such a bundle under the worker's
* home. Removing a file inside a read-only directory fails with `EACCES`,
* Vitest reports it as an unhandled error, and the shard fails after every
* test passed. So on a permission error every directory in the tree is made
* writable first and the removal is tried once more.
*/
export function removeTestHome(home: string): void {
const options = { recursive: true, force: true, maxRetries: 8, retryDelay: 50 } as const;
try {
fs.rmSync(home, options);
} catch (error) {
if (!isPermissionError(error)) throw error;
makeDirectoriesWritable(home);
fs.rmSync(home, options);
}
}
function isPermissionError(error: unknown): boolean {
const code = (error as NodeJS.ErrnoException | null)?.code;
return code === "EACCES" || code === "EPERM";
}
function makeDirectoriesWritable(directory: string): void {
let stat: fs.Stats;
try {
stat = fs.lstatSync(directory);
} catch {
return;
}
if (!stat.isDirectory()) return;
// Owner read, write and search: enough to list and unlink the children.
// A symlink is not followed, so nothing outside the home is touched.
try {
fs.chmodSync(directory, stat.mode | 0o700);
} catch {
// A directory that cannot be changed still fails the removal below, loudly.
}
let entries: fs.Dirent[];
try {
entries = fs.readdirSync(directory, { withFileTypes: true });
} catch {
return;
}
for (const entry of entries) {
if (entry.isDirectory()) makeDirectoriesWritable(path.join(directory, entry.name));
}
}
@@ -120,7 +120,7 @@ describe.skipIf(!sentryPackage)("run failure context with the real Sentry SDK",
expect(event).toMatchObject({ exception: { values: [{ value: "OpenCode execution timed out" }] },
contexts: { run_execution: { durationMs: 100, effectiveTimeoutSec: 14400,
timeoutSource: "sandbox_default", timeoutConfigured: false, timeoutFired: true, stopReason: "timeout" } } });
expect(JSON.stringify(events)).not.toMatch(/private-timeout-|<html>|504|Timed out after/);
expect(JSON.stringify(events)).not.toMatch(/private-timeout-|<html>|504 Gateway|Timed out after/);
expect(events.find(entry => entry !== event)).not.toHaveProperty("contexts.run_execution");
} finally {
await fs.rm(root, { recursive: true, force: true });
@@ -4,6 +4,7 @@ import { tmpdir } from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { describe, expect, it } from "vitest";
import { RUNNER_VENDOR_BUILD_COMMAND } from "../../scripts/prepare-runner-vendor.mjs";
const packageJsonPath = fileURLToPath(
new URL("../../package.json", import.meta.url),
@@ -57,14 +58,39 @@ describe("server package build script", () => {
expect(packageJson.devDependencies?.["@paperclipai/paperclip-runner"]).toBe(
"workspace:*",
);
// prepare-runner-vendor.mjs reuses a Runner build whose recorded inputs
// are unchanged (the `pnpm -r build` case) and otherwise runs the Runner's
// full build, which is what a standalone server build (the Dockerfile's
// path) relies on. Pin both the indirection and the command behind it.
expect(packageJson.scripts?.["prepare:runner-vendor"]).toBe(
"node scripts/prepare-runner-vendor.mjs",
);
expect(RUNNER_VENDOR_BUILD_COMMAND).toBe(
"pnpm --filter @paperclipai/paperclip-runner build",
);
expect(packageJson.scripts?.build.startsWith("pnpm run prepare:runner-vendor && ")).toBe(true);
expect(packageJson.scripts?.build).toContain(
"cp -Rf ../packages/paperclip-runner/dist/. dist/vendor/paperclip-runner/",
);
});
it("brackets the Runner's build with its input recording", () => {
// The stamp prepare-runner-vendor.mjs trusts must only exist for a build
// that completed every step and check, so it has to be written last; and
// it must vouch for the inputs the build read, not for whatever the
// sources are when it ends, so the digest has to be captured before the
// first step reads anything (--begin) and compared again at the end
// (--finish). Nothing else in the chain may touch the stamp.
const runner = JSON.parse(readFileSync(new URL(
"../../../packages/paperclip-runner/package.json", import.meta.url,
), "utf8")) as { scripts: Record<string, string> };
const steps = runner.scripts.build.split(" && ");
expect(steps.at(0)).toBe("node scripts/record-build-inputs.mjs --begin");
expect(steps.at(-1)).toBe("node scripts/record-build-inputs.mjs --finish");
expect(steps.filter((step) => step.includes("record-build-inputs"))).toHaveLength(2);
expect(steps.length).toBeGreaterThan(2);
});
it("replaces an existing read-only image manifest with the build's copy step", () => {
const packageJson = JSON.parse(readFileSync(packageJsonPath, "utf8"));
const step = packageJson.scripts.build.split(" && ").find((command: string) =>
@@ -104,10 +130,10 @@ describe("server package build script", () => {
});
it("compiles the vendored Runner's TypeScript surface once per build", () => {
// prepare:runner-vendor runs the Runner's full build, and `pnpm -r build`
// has already run it once by the time the server builds. Each extra
// prepare:runner-vendor runs the Runner's full build whenever its recorded
// inputs are stale or missing (every standalone server build). Each extra
// build:typescript inside that chain (two tsc projects plus the eval-kernel
// rebuild) costs about 7s on the CI Build job's critical path, twice.
// rebuild) costs about 7s on the CI Build job's critical path.
// Checks that read dist/ must call their script directly, the way
// generate-replay-goldens.mjs already does, not through a `check:*`
// wrapper that rebuilds first for standalone use.
+2
View File
@@ -6,6 +6,7 @@ import os from "node:os";
import path from "node:path";
import { Server as TlsServer } from "node:tls";
type SupertestServer = NetServer & {
address(): ReturnType<NetServer["address"]>;
listen(port: number): NetServer;
@@ -30,6 +31,7 @@ const { home: paperclipTestHome, cleanup: cleanupPaperclipTestHome } = createIso
process.env.PAPERCLIP_HOME = paperclipTestHome;
// Setup-file afterAll hooks run before suites drain asynchronous heartbeats.
// Cleanup at worker exit, after all suite teardown, avoids deleting live state.
// The owned-home cleanup also handles read-only runtime-context bundles.
process.once("exit", cleanupPaperclipTestHome);
const require = createRequire(import.meta.url);