fix(ci): prepare native binaries for Runner chaos checks

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Release verification checks that Runner preserves work through
faults and shutdown.
> - A governed-wait recovery case now runs a real native Runner and a
controlled Codex process.
> - That case needs the workspace debug binaries before the chaos suite
starts.
> - The chaos workflow builds JavaScript contracts but does not build
those native binaries.
> - A clean release worker fails before it can verify recovery.
> - This pull request builds the native prerequisites with the existing
package command before the tests.

## Linked Issues or Issue Description

Refs #20336, which added the real-Runner recovery case. Refs #20322,
whose normal delivery exposed the missing prerequisite. Existing native
test preparation and related chaos PRs were checked; no duplicate
preparation repair was found.

**What happened?**

The [Cloud readiness chaos
job](https://github.com/paperclipai/paperclip/actions/runs/42686564449/job/129397518634)
failed on Core `c67909cb0f1fa973e0991ba068d9c8b2f8f64924`. The
governed-wait recovery test tried to open
`packages/paperclip-runner/runner/target/debug/paperclip-runnerd`, which
did not exist. The job had installed dependencies and built eval
contracts. It had not built native workspace binaries.

**Expected behavior**

The chaos job builds its native prerequisites before it starts the
Runner fault tests. A build failure must fail the job. The existing
recovery assertions must still execute.

**Steps to reproduce**

1. Use a source checkout with JavaScript dependencies and contracts
prepared but no Rust debug workspace binaries.
2. Run `src/native-session-runtime.test.ts` with the test name `settles
a governed wait through real Runner shutdown after a missing Codex
terminal`.
3. Observe a missing native artifact. On the hosted job it was the
daemon. In the local reproduction, canonical JavaScript preflight had
already built a release daemon, so the missing artifact was debug
`fake-codex-app-server`.
4. Build the native workspace binaries and run the unchanged case again.

**Paperclip version or commit**

Hosted failure: `c67909cb0f1fa973e0991ba068d9c8b2f8f64924`. This branch
is based on fresh master `8a6b08227484591a30a7adea3840f7e96755ed81`.

**Deployment mode**

GitHub Actions source readiness and reusable Runner chaos verification.

## What Changed

- Run the existing `pnpm --filter @paperclipai/paperclip-runner
build:rust` command before the Runner fault suites. This builds the
locked native workspace binaries, including the daemon and controlled
Codex server.
- Extend the existing release-workflow contract tests to require
unconditional native preparation after dependency installation and
before the real-Runner suite.

## Verification

- The new workflow contract failed against the previous job setup. It
passed after adding the prerequisite.
- The unchanged governed-wait case failed with a missing debug
fake-Codex binary before native preparation: 1 failed, 251 intentionally
skipped, 2.92 seconds.
- Complete canonical worktree preflight prepared the debug binaries in
10.8 seconds. The unchanged real-Runner case then passed: 1 passed, 251
intentionally skipped, 2.03 seconds. It verifies saved work, accepted
handoff, confirmed provider exit, durable state and a single admitted
turn.
- After rebasing onto refreshed master, the exact case passed again in
**1.84 seconds**. The three affected workflow contracts also passed.
- **412 existing workflow, routing and native-cache contract tests
passed** in 26.58 seconds. Their input files did not change in the
rebase.
- `actionlint -oneline -shellcheck= -pyflakes=` and `git diff --check`
passed.
- The test uses the real local Rust Runner and a controlled local
provider. It does not call a live model or modify a hosted instance.
- The full local chaos suite was not run. Hosted execution of the
changed reusable chaos job remains a postmerge check.
- Required CI for final head `dd751732bff287dbeb05ddadd25c3370c817dd7a`:
[workflow attempt
2](https://github.com/paperclipai/paperclip/actions/runs/42689605454/attempts/2)
**passed** at 20:58:20 UTC on November 9: all 53 jobs accounted for, 52
successful and 1 skipped. The fresh [Build
job](https://github.com/paperclipai/paperclip/actions/runs/42689605454/job/129419966076),
[server shard
4](https://github.com/paperclipai/paperclip/actions/runs/42689605454/job/129419970716),
and [final verify
aggregate](https://github.com/paperclipai/paperclip/actions/runs/42689605454/job/129431023346)
all passed. The exact-head check rollup and classic Snyk status are
successful, with no failures or checks still pending.
- Attempt 1 remains a nonpass: the clean public npm install requested an
AWS SDK package version before that version was published, and server
shard 4 exceeded an existing five-second completion poll before its test
retry collided with a retained fixed-name environment fixture. The
unchanged exact server case passed locally. One bounded failed-job retry
reran both failed leaves; neither the fixture nor assertions were
changed. The initial cause of the five-second completion miss has not
been established.
- Fresh exact-head Greptile review is **5/5** with no open inline
threads. Ponytail is **4/5, Ship**, with two explicitly nonblocking
suggestions: a separate native-build step deadline and tighter
workflow-job scoping in the contract test. The native build remains
bounded by the existing job deadline; the current contract proves
prerequisite ordering in the affected workflow. Both suggestions are
retained for follow-up rather than changing this final tested revision.

## Risks

- A cold native build adds compilation time and disk use to the chaos
worker. The existing job timeout remains 40 minutes on the protected AWS
route and 45 minutes on the hosted route.
- Native build failures stop the job. The test selection, assertions,
permissions and source-proof gates remain in force.
- This change addresses the observed missing artifacts. Earlier
runner-loss waves and the separate managed-credential assertion remain
nonpasses; this change does not establish their causes or repair them.

## Model Used

OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository
tools and code execution. Context-window size is not exposed by the
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-09 15:00:19 -06:00
1 parent 653ab8b18f
commit 282503b5b3
2 files changed
+15

No files matched your search

+3
View File
@@ -50,6 +50,9 @@ jobs:
pnpm --filter @paperclipai/paperclip-eval-kernel build
pnpm --filter @paperclipai/paperclip-runner report:runner-chaos-evals
- name: Build native Runner test binaries
run: pnpm --filter @paperclipai/paperclip-runner build:rust
- name: Run Runner fault and replay suites
run: |
pnpm --filter @paperclipai/paperclip-runner exec vitest run \
@@ -604,6 +604,18 @@ test("release verification builds native test binaries in a required Rust-cached
assert.doesNotMatch(cloud, /continue-on-error|always\(\)/);
});
test("chaos verification prepares native Runner binaries before fault suites", () => {
const workflow = readWorkflow("runner-chaos-evals.yml");
const steps = workflow.split(/\n(?= - )/);
const install = steps.find(step => /run:\s*pnpm install\b/.test(step));
const prepare = steps.find(step => /run:\s*pnpm --filter @paperclipai\/paperclip-runner build:rust\b/.test(step));
const faults = steps.find(step => /src\/native-session-runtime\.test\.ts/.test(step));
assert.ok(install && prepare && faults, "the real Runner fault suite needs its native build prerequisite");
assert.ok(steps.indexOf(install) < steps.indexOf(prepare));
assert.ok(steps.indexOf(prepare) < steps.indexOf(faults));
assert.doesNotMatch(prepare, /^\s+(?:if|continue-on-error):/m);
});
test("Runner eval workflows pin actions and gate paid live execution", () => {
const actionPinWorkflows = [
readWorkflow("release-verify.yml"),