mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 07:55:47 +01:00
perf(ci): repair dependency caches and parallelize transport fixtures (#18883)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Required CI checks protect changes to its native Runner and control plane. > - Recent Runner jobs restored a Rust cache but still rebuilt dependencies for 74–92 seconds. > - New workspace install hooks disabled the materialized JavaScript dependency cache. > - A single transport file also kept 326 test cases on one worker. > - This pull request corrects the cache inputs, replays known install hooks, and separates independent transport test groups. > - The same checks can finish sooner without removing test cases or widening runner access. ## Linked Issues or Issue Description Refs #18794 and #18544. Related dependency-action version update: #17698; this change fixes the cache input contract used by the currently pinned action. **What happened?** Recent PR jobs such as #18783 and #18825 restored an exact Rust dependency cache from October 16. The virtual workspace's development profile changed on October 30. Rust Cache 2.9.2 hashes member manifests but omits that virtual manifest. Rebuilding cannot replace an immutable exact cache entry. The JavaScript cache also rejected the new server and Runner Copilot postinstall hooks. **Expected behavior** Changing a Cargo profile selects a new dependency cache. An exact JavaScript dependency restore still performs required workspace install hooks. Independent transport fixtures use the existing file workers while retaining all assertions and real deadlines. **Steps to reproduce** Inspect the Rust dependency cache hit and following `build:rust` in the recent Runner package jobs. Run `node scripts/ci-dependency-cache.mjs` on the previous revision to see `cacheable: false`. Run the original transport file to measure the serial test work. ## What Changed - Add the virtual workspace manifest and ancestor Cargo configuration to all shared Rust dependency keys. Keep reader/writer identities aligned and preserve master-only cache writes and CODEOWNERS-only AWS routing. - Recognize only the exact reviewed root, server, and Runner postinstall commands. Include the server/Runner installer scripts in the dependency key. Replay workspace postinstall hooks after a materialized restore. Unknown lifecycle hooks still disable reuse. - Move attachment recovery, warm attachment, live adoption, and suspension fixtures into separate files. Keep the large event-backlog cases in the original file to avoid starting another expensive fixture group at the same time. Share five stateless test helpers outside production source. - Extend the opt-in `profile-ci` benchmark with a PR-scoped Rust cache producer and a fresh-host consumer. Require exact dependency/workspace hits and reject any Cargo unit that is rebuilt. Retain the JSON build evidence. Existing setup qualification checks real JavaScript cache restores. - Schedule the ten measured files over 50 seconds first, in approximate duration order. Package CI uses up to four workers, bounded by available CPUs; local runs retain Vitest defaults. The AWS runner has four CPUs and 16 GiB RAM; the three-worker run averaged 44% CPU and peaked at 15% memory. Preserve Vitest selection and shard membership and default ordering for other files. Job count and AWS authorization are unchanged. - Scope the optional paired server benchmark to changes in its measured inputs. Cache-only changes qualify exact restores and run the shared package tests using restored dependencies. - Update the cache and test documentation and the existing native cache contract guard. ## Verification - All 586 `.github/scripts/tests/*.test.mjs` policy tests passed, including cache trust and runner routing guards. Actionlint passed for all three changed workflows. - All 169 original top-level transport test definitions remain byte-identical. The original file passed all 326 cases in 317.99 seconds on this macOS host. A clean preliminary split passed the exact same named cases in 183.01 seconds with the same four-worker cap. - Preserve the first failed experiment: its parallel run overlapped the policy suite and timed out in the 1,024-event settled backlog case at 60 seconds; 325 cases passed. A clean repeat passed, but that does not establish the earlier failure's sole cause. The final layout keeps the backlog cases in the original file. The final layout passed the same 326 cases in 186.35 seconds (41.4% faster than the 317.99-second baseline). The original test bodies and all timeouts remain unchanged. - Fresh-host Rust qualification passed: all 308 compilation units were fresh, and Cargo completed in 1.59 seconds on a new runner ([final-head run](https://github.com/paperclipai/paperclip/actions/runs/41745082837/job/126296908816)). JavaScript setup dropped from 46 seconds cold to 23 seconds with exact dependency/build restores; all 1,052 shared-package cases passed against the restored tree ([final-head job](https://github.com/paperclipai/paperclip/actions/runs/41745082837/job/126295937797)). The previous qualification also passed (43 to 19 seconds) and confirmed all three postinstall hooks replayed. These are qualification results, not an end-to-end PR timing claim. - Initial full AWS package run passed 4,907 cases with 10 existing skips. The original-file remainder fell to 118.7 seconds, but the 100-second deadline file started last, leaving the full package at 417.43 seconds. Starting only three long files first passed 4,911 cases (including four new sequencer guards) in 422.54 seconds, so reordering alone did not improve full-package time. Summed file execution was 1,202 seconds across three busy workers. The final four-worker configuration passed **4,912 cases, with the same 10 existing skips**, in **324.99 seconds**, a **23.1% reduction** from 422.54 seconds. The five added cases guard ordering, selection, sharding, and path handling. Peak memory was 17.95% of the 16 GiB runner; average CPU was 49.14% ([final-head package job](https://github.com/paperclipai/paperclip/actions/runs/41745082974/job/126296339480)). - Local workspace typecheck and build passed. The complete local `pnpm test:run` passed all six chat partitions but reported failures in one existing OAuth redirect case and one gateway authority case in the general-server group. Both focused repetitions passed (three OAuth cases and one gateway case), and a complete repeat of both files passed all 593 cases in 42.89 seconds; that does not explain the original failures. The broad local run was stopped after 61 minutes once full final-head CI had passed; its remaining general/workspace/serialized work was not completed locally, and it is not reported as a pass. Original failure logs were retained. Full final-head CI provides the complete suite validation. Sequencer selection/shard guards and the 11-case transport suspension smoke passed. The corrected release workflow guard passed all 71 tests. - The first CI run failed only the policy guard that still expected the old native cache key; downstream aggregate checks also failed. That guard now checks the new profile/config prefix and existing host/target/protocol suffix. A second policy run found that the new sequencer test imported Vitest in the dependency-free policy lane; it now runs in the Runner package lane, where all four cases pass. Greptile reviewed final head `8011f84189` at 5/5 with no actionable findings and confirmed this issue fixed. Final-head policy, typecheck, build, and review passed. At 23:52:46 UTC on November 3, several AWS jobs were terminated together: the Runner package process received SIGTERM and browser logs recorded a runner shutdown signal. The failed/cancelled jobs passed on the same-head retry, including required `ci / verify`; no assertion or timeout was changed in response. The final concurrency adjustment at `fe5a69bb0b` also received Greptile 5/5 with no actionable findings. Final-head required CI, including `ci / verify`, passed ([aggregate](https://github.com/paperclipai/paperclip/actions/runs/41745082974/job/126300554711)). Local timings exclude install, build, cache transfer, and CI queue time. ## Risks A new Rust key needs one successful canonical master build before ordinary PRs can hit it. The fresh-host qualification uses separate PR-scoped entries. Parallel test files can increase I/O contention; case execution inside each file, timeouts, and assertions are unchanged. Two extracted suites exceed 1,000 lines because they retain existing multi-stage recovery fixtures unchanged; one parameterized fixture alone is 1,059 lines. No production execution logic, lockfile, credentials, runner trust gates, or required check is removed. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository editing, command execution, and GitHub inspection. The host does not expose a more precise model snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — targeted checks passed; the broad-run failures and repetitions are documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
925965a09b
commit
a194166017
19 files changed
+4539
-4149
No files matched your search
@@ -65,6 +65,14 @@ runs:
|
||||
pnpm install --frozen-lockfile
|
||||
fi
|
||||
|
||||
- name: Replay workspace install hooks after restore
|
||||
if: steps.installed.outputs.cache-hit == 'true'
|
||||
shell: bash
|
||||
working-directory: ${{ inputs.working-directory }}
|
||||
run: |
|
||||
pnpm -r --if-present run postinstall
|
||||
pnpm run postinstall
|
||||
|
||||
- name: Restore certified shared build outputs (read only)
|
||||
if: inputs.restore-cache == 'true' && steps.inputs.outputs.cacheable == 'true'
|
||||
id: builds
|
||||
|
||||
@@ -71,6 +71,36 @@ test('unmodelled workspace lifecycle scripts disable materialized reuse', () =>
|
||||
}
|
||||
});
|
||||
|
||||
test('known Copilot hooks permit caching, track their helpers, and still run on restore', () => {
|
||||
const files = { ...entries,
|
||||
'server/package.json': JSON.stringify({ scripts: { postinstall: 'node scripts/install-copilot-assets.mjs' } }),
|
||||
'packages/paperclip-runner/package.json': JSON.stringify({ scripts: { postinstall: 'node scripts/install-copilot-assets.mjs' } }),
|
||||
'server/scripts/install-copilot-assets.mjs': 'server bootstrap',
|
||||
'packages/paperclip-runner/scripts/install-copilot-assets.mjs': 'runner bootstrap',
|
||||
'packages/paperclip-runner/scripts/materialize-copilot-binary.mjs': 'native materializer',
|
||||
'packages/paperclip-runner/scripts/copilot-inner-distribution.mjs': 'closure pins',
|
||||
};
|
||||
const compute = (input) => cachePlan({ files: Object.keys(input),
|
||||
read: file => input[file] == null ? null : Buffer.from(input[file]), scope,
|
||||
packageDirs: ['.', 'server', 'packages/paperclip-runner'],
|
||||
});
|
||||
const original = compute(files);
|
||||
assert.equal(original.cacheable, true);
|
||||
for (const file of Object.keys(files).filter(file => /^(server|packages\/paperclip-runner)\/scripts\//.test(file))) {
|
||||
for (const replacement of ['changed', null]) {
|
||||
assert.notEqual(compute({ ...files, [file]: replacement }).dependencyKey, original.dependencyKey, file);
|
||||
}
|
||||
}
|
||||
assert.equal(cachePlan({ files: ['elsewhere/package.json'],
|
||||
read: () => Buffer.from(files['server/package.json']), scope, packageDirs: ['elsewhere'],
|
||||
}).cacheable, false, 'the command is allowed only in its reviewed owners');
|
||||
const action = readFileSync(new URL('../../actions/setup-ci-dependencies/action.yml', import.meta.url), 'utf8');
|
||||
const replay = action.split(' - name: Replay workspace install hooks after restore')[1].split(' - name:')[0];
|
||||
assert.match(replay, /if: steps\.installed\.outputs\.cache-hit == 'true'/);
|
||||
assert.match(replay, /pnpm -r --if-present run postinstall\s+pnpm run postinstall/);
|
||||
assert.ok(action.indexOf('Replay workspace install hooks') < action.indexOf('Validate workspace links'));
|
||||
});
|
||||
|
||||
test('cache restores are exact and only a frozen canonical master push can publish entries', () => {
|
||||
const action = readFileSync(new URL('../../actions/setup-ci-dependencies/action.yml', import.meta.url), 'utf8');
|
||||
const installed = action.slice(action.indexOf(' - name: Restore installed'), action.indexOf(' - name: Locate pnpm'));
|
||||
@@ -97,6 +127,15 @@ test('paired test timings remain available when lifecycle hooks disable material
|
||||
assert.match(step(name), /if: steps.qualification.outputs.cacheable == 'true'/);
|
||||
}
|
||||
assert.match(step('Explain disabled materialized reuse'), /cacheable != 'true'/);
|
||||
assert.doesNotMatch(step('Prepare unchanged baseline'), /if:/);
|
||||
assert.doesNotMatch(step('Compare complete commands and executed case inventories'), /if:/);
|
||||
for (const name of ['Prepare unchanged baseline', 'Compare complete commands and executed case inventories']) {
|
||||
assert.match(step(name), /if: steps\.test_scope\.outputs\.run == 'true'/);
|
||||
assert.doesNotMatch(step(name), /if:.*cacheable/);
|
||||
}
|
||||
const scope = step('Select changed server test work');
|
||||
assert.match(scope, /git diff --name-only HEAD\^1 HEAD --/);
|
||||
for (const file of ['scripts/benchmark-ci-work.mjs', 'scripts/run-vitest-stable.mjs',
|
||||
'server/src/__tests__/setup-supertest.ts', 'server/src/__tests__/workspace-runtime.test.ts',
|
||||
'server/src/services/workspace-git-snapshot-streaming.test.ts']) assert.ok(scope.includes(file));
|
||||
assert.doesNotMatch(scope, /scripts\/ci-dependency-cache|\.github\/actions\/setup-ci-dependencies/);
|
||||
assert.match(step('Exercise restored dependencies for cache-only changes'), /pnpm exec vitest run --project @paperclipai\/shared/);
|
||||
});
|
||||
@@ -48,11 +48,27 @@ test("every rust-cache restore in the PR workflow belongs to a guarded reader jo
|
||||
const keyInputs = [
|
||||
/uses: Swatinem\/rust-cache@([0-9a-f]{40}) # v[0-9.]+/,
|
||||
/workspaces: (\$\{\{ steps\.runner_rust_workspace\.outputs\.path \}\} -> target)/,
|
||||
/shared-key: (release-runner-v2)/,
|
||||
/shared-key: (release-runner-v2-.+)/,
|
||||
/cache-workspace-crates: (false)/,
|
||||
/cache-bin: (false)/,
|
||||
];
|
||||
|
||||
test("virtual workspace profiles and ancestor Cargo configuration partition every dependency cache", () => {
|
||||
for (const [name, workflow] of [["PR", prWorkflow], ["release", releaseWorkflow]]) {
|
||||
const keys = [...workflow.matchAll(/^\s+shared-key: (.+)$/gm)].map(([, key]) => key);
|
||||
assert.ok(keys.length > 0);
|
||||
for (const key of keys) {
|
||||
assert.ok(key.includes("hashFiles('packages/paperclip-runner/runner/Cargo.toml'"), name);
|
||||
for (const directory of ['.', 'packages', 'packages/paperclip-runner', 'packages/paperclip-runner/runner']) {
|
||||
const prefix = directory === '.' ? '' : `${directory}/`;
|
||||
for (const file of ['config', 'config.toml']) {
|
||||
assert.ok(key.includes(`'${prefix}.cargo/${file}'`), `${name}: ${prefix}.cargo/${file}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test("every PR reader restores the Rust cache under the same key the master push writes", () => {
|
||||
for (const [name, pr] of readers) {
|
||||
for (const pattern of keyInputs) {
|
||||
|
||||
@@ -7,6 +7,9 @@ on:
|
||||
- .github/actions/setup-ci-dependencies/**
|
||||
- .github/workflows/ci-work-benchmark.yml
|
||||
- scripts/ci-dependency-cache.mjs
|
||||
- .github/workflows/pr-trusted.yml
|
||||
- .github/workflows/release-verify.yml
|
||||
- scripts/runner-debug-crate-cache.mjs
|
||||
- scripts/benchmark-ci-work.mjs
|
||||
- scripts/run-vitest-stable.mjs
|
||||
- server/src/__tests__/setup-supertest.ts
|
||||
@@ -39,6 +42,21 @@ jobs:
|
||||
fetch-depth: 2
|
||||
persist-credentials: false
|
||||
|
||||
- name: Select changed server test work
|
||||
id: test_scope
|
||||
run: |
|
||||
changed=$(git diff --name-only HEAD^1 HEAD -- \
|
||||
scripts/benchmark-ci-work.mjs \
|
||||
scripts/run-vitest-stable.mjs \
|
||||
server/src/__tests__/setup-supertest.ts \
|
||||
server/src/__tests__/workspace-runtime.test.ts \
|
||||
server/src/services/workspace-git-snapshot-streaming.test.ts)
|
||||
if [ -n "$changed" ]; then
|
||||
echo "run=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "run=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
|
||||
with:
|
||||
@@ -123,6 +141,7 @@ jobs:
|
||||
echo "Materialized dependency reuse is disabled by an unmodelled install hook. Retaining cold setup and comparing test commands only." >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
- name: Prepare unchanged baseline
|
||||
if: steps.test_scope.outputs.run == 'true'
|
||||
run: |
|
||||
base_sha=$(git rev-parse HEAD^1)
|
||||
git worktree add --detach ../baseline "$base_sha"
|
||||
@@ -135,12 +154,17 @@ jobs:
|
||||
pnpm --filter @paperclipai/plugin-sdk ensure-build-deps
|
||||
|
||||
- name: Compare complete commands and executed case inventories
|
||||
if: steps.test_scope.outputs.run == 'true'
|
||||
env:
|
||||
BENCHMARK_OUTPUT: ${{ runner.temp }}/ci-work-benchmark
|
||||
run: |
|
||||
node scripts/benchmark-ci-work.mjs ../baseline "$BENCHMARK_OUTPUT"
|
||||
cat "$BENCHMARK_OUTPUT/summary.md" >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
- name: Exercise restored dependencies for cache-only changes
|
||||
if: steps.test_scope.outputs.run != 'true' && steps.qualification.outputs.cacheable == 'true'
|
||||
run: pnpm exec vitest run --project @paperclipai/shared
|
||||
|
||||
- name: Retain timings, inventories and failure logs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
|
||||
@@ -148,3 +172,120 @@ jobs:
|
||||
name: ci-work-benchmark
|
||||
path: ${{ runner.temp }}/ci-work-benchmark
|
||||
retention-days: 14
|
||||
|
||||
rust_prime:
|
||||
if: >-
|
||||
contains(github.event.pull_request.labels.*.name, 'profile-ci') &&
|
||||
(github.event.action != 'labeled' || github.event.label.name == 'profile-ci')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
|
||||
with:
|
||||
ref: ${{ github.sha }}
|
||||
persist-credentials: false
|
||||
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
|
||||
with:
|
||||
node-version: 24
|
||||
package-manager-cache: false
|
||||
- name: Select the pinned compiler and workspace path
|
||||
id: workspace
|
||||
working-directory: packages/paperclip-runner
|
||||
run: |
|
||||
toolchain=$(rustup show active-toolchain | awk '{print $1}')
|
||||
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
|
||||
rustup toolchain list | awk '{print $1}' | grep -vx "$toolchain" | xargs -r -n1 rustup toolchain uninstall || true
|
||||
pinned="$HOME/paperclip-runner-rust"
|
||||
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
|
||||
echo "path=$pinned" >> "$GITHUB_OUTPUT"
|
||||
- name: Qualify the dependency cache
|
||||
id: dependencies
|
||||
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
workspaces: ${{ steps.workspace.outputs.path }} -> target
|
||||
shared-key: qualification-${{ github.run_id }}-${{ github.run_attempt }}-${{ hashFiles('packages/paperclip-runner/runner/Cargo.toml', '.cargo/config', '.cargo/config.toml', 'packages/.cargo/config', 'packages/.cargo/config.toml', 'packages/paperclip-runner/.cargo/config', 'packages/paperclip-runner/.cargo/config.toml', 'packages/paperclip-runner/runner/.cargo/config', 'packages/paperclip-runner/runner/.cargo/config.toml') }}
|
||||
cache-workspace-crates: false
|
||||
cache-bin: false
|
||||
save-if: 'true'
|
||||
- name: Build the exact debug units
|
||||
working-directory: packages/paperclip-runner
|
||||
run: cargo build --manifest-path runner/Cargo.toml --locked --workspace --bins
|
||||
- name: Compute workspace artifact identity
|
||||
id: debug
|
||||
run: node scripts/runner-debug-crate-cache.mjs plan
|
||||
- name: Save workspace artifacts in this PR only
|
||||
id: artifacts
|
||||
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
|
||||
with:
|
||||
key: qualification-${{ github.run_id }}-${{ github.run_attempt }}-${{ steps.debug.outputs.key }}
|
||||
path: ${{ steps.debug.outputs.paths }}
|
||||
|
||||
rust_restore:
|
||||
needs: rust_prime
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
|
||||
with:
|
||||
ref: ${{ github.sha }}
|
||||
persist-credentials: false
|
||||
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
|
||||
with:
|
||||
node-version: 24
|
||||
package-manager-cache: false
|
||||
- name: Select the pinned compiler and workspace path
|
||||
id: workspace
|
||||
working-directory: packages/paperclip-runner
|
||||
run: |
|
||||
toolchain=$(rustup show active-toolchain | awk '{print $1}')
|
||||
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
|
||||
rustup toolchain list | awk '{print $1}' | grep -vx "$toolchain" | xargs -r -n1 rustup toolchain uninstall || true
|
||||
pinned="$HOME/paperclip-runner-rust"
|
||||
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
|
||||
echo "path=$pinned" >> "$GITHUB_OUTPUT"
|
||||
- name: Qualify the dependency cache
|
||||
id: dependencies
|
||||
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
workspaces: ${{ steps.workspace.outputs.path }} -> target
|
||||
shared-key: qualification-${{ github.run_id }}-${{ github.run_attempt }}-${{ hashFiles('packages/paperclip-runner/runner/Cargo.toml', '.cargo/config', '.cargo/config.toml', 'packages/.cargo/config', 'packages/.cargo/config.toml', 'packages/paperclip-runner/.cargo/config', 'packages/paperclip-runner/.cargo/config.toml', 'packages/paperclip-runner/runner/.cargo/config', 'packages/paperclip-runner/runner/.cargo/config.toml') }}
|
||||
cache-workspace-crates: false
|
||||
cache-bin: false
|
||||
save-if: 'false'
|
||||
- name: Compute workspace artifact identity
|
||||
id: debug
|
||||
run: node scripts/runner-debug-crate-cache.mjs plan
|
||||
- name: Restore workspace artifacts in this PR only
|
||||
id: artifacts
|
||||
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
|
||||
with:
|
||||
key: qualification-${{ github.run_id }}-${{ github.run_attempt }}-${{ steps.debug.outputs.key }}
|
||||
path: ${{ steps.debug.outputs.paths }}
|
||||
- name: Verify fresh-host cache reuse
|
||||
env:
|
||||
DEPENDENCIES_HIT: ${{ steps.dependencies.outputs.cache-hit }}
|
||||
ARTIFACTS_HIT: ${{ steps.artifacts.outputs.cache-hit }}
|
||||
run: |
|
||||
test "$DEPENDENCIES_HIT" = true
|
||||
test "$ARTIFACTS_HIT" = true
|
||||
node scripts/runner-debug-crate-cache.mjs mark-current
|
||||
cd packages/paperclip-runner
|
||||
started=$(date +%s)
|
||||
cargo build --manifest-path runner/Cargo.toml --locked --workspace --bins --message-format=json > "$RUNNER_TEMP/rust-restored-build.jsonl"
|
||||
echo "Restored Rust build: $(($(date +%s) - started)) seconds." >> "$GITHUB_STEP_SUMMARY"
|
||||
node --input-type=module <<'NODE'
|
||||
import { readFileSync } from 'node:fs';
|
||||
const artifacts = readFileSync(`${process.env.RUNNER_TEMP}/rust-restored-build.jsonl`, 'utf8')
|
||||
.trim().split('\n').map(line => JSON.parse(line)).filter(row => row.reason === 'compiler-artifact');
|
||||
if (!artifacts.length || artifacts.some(row => !row.fresh)) {
|
||||
throw new Error(`Cache hit rebuilt units: ${JSON.stringify(artifacts.filter(row => !row.fresh).map(row => row.package_id))}`);
|
||||
}
|
||||
console.log(`All ${artifacts.length} Cargo units were fresh on a new host.`);
|
||||
NODE
|
||||
- name: Retain Cargo reuse evidence
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
|
||||
with:
|
||||
name: rust-cache-qualification
|
||||
path: ${{ runner.temp }}/rust-restored-build.jsonl
|
||||
retention-days: 14
|
||||
@@ -370,7 +370,7 @@ jobs:
|
||||
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
|
||||
shared-key: release-runner-v2
|
||||
shared-key: release-runner-v2-${{ hashFiles('packages/paperclip-runner/runner/Cargo.toml', '.cargo/config', '.cargo/config.toml', 'packages/.cargo/config', 'packages/.cargo/config.toml', 'packages/paperclip-runner/.cargo/config', 'packages/paperclip-runner/.cargo/config.toml', 'packages/paperclip-runner/runner/.cargo/config', 'packages/paperclip-runner/runner/.cargo/config.toml') }}
|
||||
# Mirror the master writer: these also feed the cache key.
|
||||
cache-workspace-crates: false
|
||||
cache-bin: false
|
||||
@@ -784,7 +784,7 @@ jobs:
|
||||
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
|
||||
shared-key: release-runner-v2
|
||||
shared-key: release-runner-v2-${{ hashFiles('packages/paperclip-runner/runner/Cargo.toml', '.cargo/config', '.cargo/config.toml', 'packages/.cargo/config', 'packages/.cargo/config.toml', 'packages/paperclip-runner/.cargo/config', 'packages/paperclip-runner/.cargo/config.toml', 'packages/paperclip-runner/runner/.cargo/config', 'packages/paperclip-runner/runner/.cargo/config.toml') }}
|
||||
# Mirror the master writer: these also feed the cache key.
|
||||
cache-workspace-crates: false
|
||||
cache-bin: false
|
||||
@@ -1035,7 +1035,7 @@ jobs:
|
||||
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
|
||||
shared-key: release-runner-v2
|
||||
shared-key: release-runner-v2-${{ hashFiles('packages/paperclip-runner/runner/Cargo.toml', '.cargo/config', '.cargo/config.toml', 'packages/.cargo/config', 'packages/.cargo/config.toml', 'packages/paperclip-runner/.cargo/config', 'packages/paperclip-runner/.cargo/config.toml', 'packages/paperclip-runner/runner/.cargo/config', 'packages/paperclip-runner/runner/.cargo/config.toml') }}
|
||||
# Mirror the master writer: these also feed the cache key.
|
||||
cache-workspace-crates: false
|
||||
cache-bin: false
|
||||
@@ -1376,7 +1376,7 @@ jobs:
|
||||
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
|
||||
shared-key: release-runner-v2
|
||||
shared-key: release-runner-v2-${{ hashFiles('packages/paperclip-runner/runner/Cargo.toml', '.cargo/config', '.cargo/config.toml', 'packages/.cargo/config', 'packages/.cargo/config.toml', 'packages/paperclip-runner/.cargo/config', 'packages/paperclip-runner/.cargo/config.toml', 'packages/paperclip-runner/runner/.cargo/config', 'packages/paperclip-runner/runner/.cargo/config.toml') }}
|
||||
# Mirror the master writer: these also feed the cache key.
|
||||
cache-workspace-crates: false
|
||||
cache-bin: false
|
||||
|
||||
@@ -92,7 +92,7 @@ jobs:
|
||||
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
|
||||
shared-key: release-typecheck-v2
|
||||
shared-key: release-typecheck-v2-${{ hashFiles('packages/paperclip-runner/runner/Cargo.toml', '.cargo/config', '.cargo/config.toml', 'packages/.cargo/config', 'packages/.cargo/config.toml', 'packages/paperclip-runner/.cargo/config', 'packages/paperclip-runner/.cargo/config.toml', 'packages/paperclip-runner/runner/.cargo/config', 'packages/paperclip-runner/runner/.cargo/config.toml') }}
|
||||
# Rebuild workspace code and rerun every check. Cache only compiled
|
||||
# dependencies; never restore installed executables from cargo/bin.
|
||||
cache-workspace-crates: false
|
||||
@@ -398,7 +398,7 @@ jobs:
|
||||
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
|
||||
shared-key: release-runner-v2
|
||||
shared-key: release-runner-v2-${{ hashFiles('packages/paperclip-runner/runner/Cargo.toml', '.cargo/config', '.cargo/config.toml', 'packages/.cargo/config', 'packages/.cargo/config.toml', 'packages/paperclip-runner/.cargo/config', 'packages/paperclip-runner/.cargo/config.toml', 'packages/paperclip-runner/runner/.cargo/config', 'packages/paperclip-runner/runner/.cargo/config.toml') }}
|
||||
# Rebuild workspace code and rerun every check. Cache only compiled
|
||||
# dependencies; never restore installed executables from cargo/bin.
|
||||
cache-workspace-crates: false
|
||||
@@ -680,7 +680,7 @@ jobs:
|
||||
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6
|
||||
with:
|
||||
workspaces: ${{ steps.native_rust_workspace.outputs.path }} -> target
|
||||
shared-key: release-native-v2-${{ matrix.host }}-${{ matrix.target }}-${{ hashFiles('packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/src/**', 'packages/paperclip-runner/*.json') }}
|
||||
shared-key: release-native-v2-${{ hashFiles('packages/paperclip-runner/runner/Cargo.toml', '.cargo/config', '.cargo/config.toml', 'packages/.cargo/config', 'packages/.cargo/config.toml', 'packages/paperclip-runner/.cargo/config', 'packages/paperclip-runner/.cargo/config.toml', 'packages/paperclip-runner/runner/.cargo/config', 'packages/paperclip-runner/runner/.cargo/config.toml') }}-${{ matrix.host }}-${{ matrix.target }}-${{ hashFiles('packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/src/**', 'packages/paperclip-runner/*.json') }}
|
||||
cache-workspace-crates: false
|
||||
cache-bin: false
|
||||
cache-on-failure: false
|
||||
|
||||
+17
-1
@@ -1095,6 +1095,22 @@ verifying Vite/HMR behavior.
|
||||
|
||||
For normal issue work, start with the smallest targeted check that proves the change. Reserve repo-wide typecheck/build/test runs for PR-ready handoff or changes broad enough that narrow checks do not cover the risk.
|
||||
|
||||
The Runner transport suite keeps attachment recovery, warm attachment, live
|
||||
adoption, and suspension in separate test files. Vitest can schedule these
|
||||
independent fixtures on its existing file workers; cases within each file stay
|
||||
serial. The package sequencer starts the ten files measured above 50 seconds in
|
||||
the November 4, 2026 AWS run first, ordered by approximate duration. This includes
|
||||
transport deadlines, settlement, provider launch, recovery, and streaming waits,
|
||||
so fresh CI runners do not have to infer duration from file size. Package CI uses
|
||||
up to four workers, bounded by available CPUs; local runs retain Vitest defaults.
|
||||
Vitest's file selection and shard membership are unchanged. Real lease/deadline
|
||||
waits and assertions are unchanged. Run the complete
|
||||
family, including the separate deadline and settlement regressions, with:
|
||||
|
||||
```sh
|
||||
pnpm --filter @paperclipai/paperclip-runner exec vitest run runnerd-codex-transport
|
||||
```
|
||||
|
||||
#### Embedded Postgres template cache
|
||||
|
||||
To run the suite with one owned PostgreSQL server and a separate disposable
|
||||
@@ -3187,7 +3203,7 @@ Run the credential-free real-process restart suite with:
|
||||
```sh
|
||||
pnpm --filter @paperclipai/paperclip-runner build:runner-binaries
|
||||
pnpm exec vitest run server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
|
||||
pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport.test.ts -t 'adopts a live runner'
|
||||
pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport-adoption.test.ts -t 'adopts a live runner'
|
||||
```
|
||||
|
||||
The suite uses isolated PostgreSQL state, isolated `PAPERCLIP_HOME` roots, real
|
||||
|
||||
@@ -368,7 +368,14 @@ readiness can succeed. A failed lane does not cancel the other lane.
|
||||
Both lanes restore Cargo dependencies with the pinned Rust Cache action. The
|
||||
compiler comes from the Runner package's `rust-toolchain.toml` before the action
|
||||
computes its key. Compiler and Cargo metadata changes select a new cache. The
|
||||
`release-runner-v2` shared key avoids separate copies for these lanes.
|
||||
`release-runner-v2` shared key avoids separate copies for these lanes. Its suffix
|
||||
hashes the virtual workspace's `runner/Cargo.toml` and ancestor Cargo configuration.
|
||||
Rust Cache discovers member manifests through Cargo metadata, which omits this
|
||||
virtual manifest. Without the suffix, changing `[profile.dev]` can restore an
|
||||
exact but obsolete dependency cache on every run. GitHub caches are immutable,
|
||||
so the successful rebuild cannot replace that old entry. The profile/configuration
|
||||
hash also partitions fallback restores. Every reader and writer uses the same
|
||||
inputs, including the separate typecheck and cross-target caches.
|
||||
Only the Rust lane saves this cache. After verification it also runs `build:rust`
|
||||
to warm the debug dependencies used by the protocol lane; its own tests already
|
||||
warm release dependencies. The cache writer is shorter than the protocol lane.
|
||||
@@ -385,13 +392,36 @@ paths and the PR lanes can restore master's entry. The guard tests under
|
||||
Rust cache step in both workflows.
|
||||
|
||||
Workspace crates and installed Cargo binaries are excluded from this entry.
|
||||
Every run runs all assigned checks, including on a cache hit. Only an
|
||||
own-repository master-push run verifying that push's exact SHA can restore the
|
||||
cache, and only a successful Rust lane saves it. PR, tag, and manual candidate
|
||||
Every run runs all assigned checks, including on a cache hit. In release
|
||||
verification, only an own-repository master-push run verifying that push's exact
|
||||
SHA can restore the cache, and only a successful Rust lane saves it. The PR
|
||||
workflow restores master's cache read only. Tag and manual candidate release
|
||||
verification compile without this cache. A miss or eviction costs compilation
|
||||
time but does not change the checks. To discard old dependency caches, increment
|
||||
the shared-key version and let the next successful master verification warm it.
|
||||
|
||||
### Installed JavaScript dependencies
|
||||
|
||||
`setup-ci-dependencies` can restore exact `node_modules` trees and separately
|
||||
certified shared/SDK build outputs. `scripts/ci-dependency-cache.mjs` hashes
|
||||
manifests, resolution, patches, install scripts, and the runtime/checkout layout.
|
||||
The root workspace-link hook and the server/Runner Copilot asset hooks are known
|
||||
postinstall commands. An exact dependency restore replays these hooks before
|
||||
validating workspace links and build outputs. Copilot defers installation in an
|
||||
unbuilt source checkout; its normal build installs and verifies the assets.
|
||||
Provider assets are not included in the dependency cache. Any unrecognized
|
||||
workspace lifecycle hook still disables materialized reuse. Only a frozen install
|
||||
from the canonical master push can publish the shared entries.
|
||||
|
||||
For CI cache changes, the `profile-ci` label runs the paired setup benchmark
|
||||
and exercises restored dependencies with the shared-package tests. The longer
|
||||
server test comparison runs when its measured scripts or fixtures change;
|
||||
cache-only changes do not require unchanged test commands to run faster.
|
||||
It also builds Rust units on one GitHub-hosted runner and restores the dependency
|
||||
and workspace caches on a second runner with a fresh checkout. Both restores
|
||||
must be exact, and Cargo must report every unit as fresh. These qualification
|
||||
caches are scoped to the PR and run; they cannot populate master's caches.
|
||||
|
||||
### Runner release crate cache
|
||||
|
||||
The dependency entry leaves the Runner's own crate out, and a fresh checkout
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
import { BaseSequencer } from "vitest/node";
|
||||
|
||||
const slowFiles = [
|
||||
"runnerd-codex-transport.test.ts",
|
||||
"runnerd-codex-transport-settlement.test.ts",
|
||||
"cursor-cloud-runner-launch.test.ts",
|
||||
"runnerd-codex-transport-deadlines.test.ts",
|
||||
"opencode-server-driver.test.ts",
|
||||
"acpx-controller-recovery.test.ts",
|
||||
"runnerd-codex-transport-warm-attachment.test.ts",
|
||||
"runnerd-opencode-streaming-tools.test.ts",
|
||||
"live-session.test.ts",
|
||||
"runnerd-codex-transport-adoption.test.ts",
|
||||
];
|
||||
|
||||
const priority = (file) => {
|
||||
const name = file.moduleId.replaceAll("\\", "/").split("/").at(-1);
|
||||
const index = slowFiles.indexOf(name);
|
||||
return index < 0 ? slowFiles.length : index;
|
||||
};
|
||||
|
||||
export class RunnerSequencer extends BaseSequencer {
|
||||
async sort(files) {
|
||||
const sorted = await super.sort(files);
|
||||
return sorted.sort((a, b) => priority(a) - priority(b));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "vitest";
|
||||
import { BaseSequencer } from "vitest/node";
|
||||
import { RunnerSequencer } from "./vitest-sequencer.mjs";
|
||||
|
||||
const project = { name: "runner", config: { isolate: true, sequence: { groupOrder: 0 } } };
|
||||
const files = [
|
||||
"large.test.ts",
|
||||
"runnerd-codex-transport.test.ts",
|
||||
"small.test.ts",
|
||||
"runnerd-codex-transport-settlement.test.ts",
|
||||
"runnerd-codex-transport-deadlines.test.ts",
|
||||
].map((name) => ({ moduleId: `/repo/src/live/${name}`, project }));
|
||||
const ctx = {
|
||||
config: { root: "/repo", shard: { index: 1, count: 2 } },
|
||||
cache: {
|
||||
getFileTestResults: () => undefined,
|
||||
getFileStats: (key) => ({ size: 100 - files.findIndex((file) => key.endsWith(file.moduleId.slice(6))) }),
|
||||
},
|
||||
};
|
||||
|
||||
test("starts real-wait transport files first without changing test membership", async () => {
|
||||
const sorted = await new RunnerSequencer(ctx).sort(files);
|
||||
assert.deepEqual(sorted, [files[1], files[3], files[4], files[0], files[2]]);
|
||||
assert.equal(new Set(sorted).size, files.length);
|
||||
assert.deepEqual(await new BaseSequencer(ctx).sort(files), files);
|
||||
});
|
||||
|
||||
test("selected subsets stay selected and retain default ordering", async () => {
|
||||
const selected = [files[2], files[0]];
|
||||
assert.deepEqual(await new RunnerSequencer(ctx).sort(selected), await new BaseSequencer(ctx).sort(selected));
|
||||
assert.deepEqual(await new RunnerSequencer(ctx).sort([files[4]]), [files[4]]);
|
||||
assert.deepEqual(await new RunnerSequencer(ctx).sort([]), []);
|
||||
});
|
||||
|
||||
test("keeps Vitest shard membership and schedules waits within each shard", async () => {
|
||||
const all = [];
|
||||
for (const index of [1, 2]) {
|
||||
const shardCtx = { ...ctx, config: { ...ctx.config, shard: { index, count: 2 } } };
|
||||
const sequencer = new RunnerSequencer(shardCtx);
|
||||
const shard = await sequencer.shard(files);
|
||||
assert.deepEqual(shard, await new BaseSequencer(shardCtx).shard(files));
|
||||
const sorted = await sequencer.sort(shard);
|
||||
assert.deepEqual(new Set(sorted), new Set(shard));
|
||||
all.push(...sorted);
|
||||
}
|
||||
assert.equal(all.length, files.length);
|
||||
assert.deepEqual(new Set(all), new Set(files));
|
||||
});
|
||||
|
||||
test("recognizes Windows paths", async () => {
|
||||
const deadline = { ...files[4], moduleId: "C:\\repo\\src\\live\\runnerd-codex-transport-deadlines.test.ts" };
|
||||
assert.deepEqual(await new RunnerSequencer(ctx).sort([files[0], deadline]), [deadline, files[0]]);
|
||||
});
|
||||
|
||||
test("starts measured slow provider files before short files without adding work", async () => {
|
||||
const streaming = { ...files[0], moduleId: "/repo/src/live/runnerd-opencode-streaming-tools.test.ts" };
|
||||
const launch = { ...files[0], moduleId: "/repo/src/control-plane/cursor-cloud-runner-launch.test.ts" };
|
||||
assert.deepEqual(await new RunnerSequencer(ctx).sort([files[0], streaming, files[4], launch]),
|
||||
[launch, files[4], streaming, files[0]]);
|
||||
});
|
||||
@@ -0,0 +1,556 @@
|
||||
import {
|
||||
createRunnerProviderLaunch,
|
||||
runnerLaunchRequirementsDigest,
|
||||
type RunnerProviderLaunchRequirements,
|
||||
} from "./runner-provider-launch.js";
|
||||
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { spawn } from "node:child_process";
|
||||
import { createServer } from "node:http";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { expect, it, vi } from "vitest";
|
||||
import { NativeSessionCloseUnrecoverableError } from "../contracts/native-session-backend.js";
|
||||
import { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js";
|
||||
import * as durableControlPlane from "../control-plane/durable-prp-control-plane.js";
|
||||
import { codexSemanticToolSpecs } from "../drivers/codex/codex-app-server-driver.js";
|
||||
import { createCapabilityRunnerdCodexTransport, readRunnerdArtifactBinding } from "./runnerd-codex-transport.js";
|
||||
import { defaultCapabilityRunnerdBinary, fakeCodex, fakeCodexArgs } from "../../test/support/runnerd-transport.js";
|
||||
|
||||
async function verifyLiveRunnerAdoption(
|
||||
mismatchedCheckpoint: boolean,
|
||||
mismatchedArtifact = false,
|
||||
goalMidTurn = false,
|
||||
detachBeforeCleanup = false,
|
||||
startWithoutCheckpoint = false,
|
||||
preparedTask = false,
|
||||
adoptionAuthorization: "allow" | "deny" | "legacy" | "legacy-missing" | "missing-binding" = "allow",
|
||||
toolHandlerOrder?: "before" | "after" | "completion-before" | "completion-after",
|
||||
uncertainAtClose = false,
|
||||
semanticRecoveryMode?: "denied-before-cancel" | "cleanup-only",
|
||||
stopExitProofFault?: "missing" | "false",
|
||||
) {
|
||||
const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-live-adopt-"));
|
||||
const pendingCompletion = toolHandlerOrder?.startsWith("completion-") === true;
|
||||
const completionPath = join(stateDirectory, "pending-completion.json");
|
||||
if (pendingCompletion) await writeFile(completionPath, JSON.stringify({
|
||||
schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary: "Completed the synthetic work",
|
||||
completionClaim: { contractRevision: "1", objectiveSatisfied: true, criteria: [], remainingWork: [] },
|
||||
evidence: [], verification: [], attentionRequests: [], artifacts: [],
|
||||
}));
|
||||
const server = createServer();
|
||||
let authority: DurablePrpControlPlane | null = null;
|
||||
const checkpoint = vi.fn(async () => undefined);
|
||||
server.on("upgrade", (request, socket, head) => {
|
||||
if (!authority) {
|
||||
socket.destroy();
|
||||
return;
|
||||
}
|
||||
authority.handleUpgrade(request, socket, "/runner", head);
|
||||
});
|
||||
await new Promise<void>((resolveListen) =>
|
||||
server.listen(0, "127.0.0.1", resolveListen),
|
||||
);
|
||||
const address = server.address();
|
||||
if (!address || typeof address === "string")
|
||||
throw new Error("Expected adoption test listener");
|
||||
const registration = vi.fn(async (next: DurablePrpControlPlane) => {
|
||||
authority = next;
|
||||
return {
|
||||
connectUrl: `ws://127.0.0.1:${address.port}/runner`,
|
||||
...(mismatchedArtifact || uncertainAtClose || semanticRecoveryMode === "cleanup-only" ? { checkpoint } : {}),
|
||||
release: async () => {
|
||||
if (authority === next) authority = null;
|
||||
},
|
||||
};
|
||||
});
|
||||
const identity = {
|
||||
runnerInstanceId: "runner-live-adopt",
|
||||
environmentLeaseId: "lease-live-adopt",
|
||||
runId: "run-live-adopt",
|
||||
normalizedSessionId: "session-live-adopt",
|
||||
turnId: "turn-live-adopt",
|
||||
itemId: "item-live-adopt",
|
||||
};
|
||||
const sharedOptions = {
|
||||
runnerBinary: defaultCapabilityRunnerdBinary(),
|
||||
codexCommand: fakeCodex,
|
||||
codexArgs: fakeCodexArgs(stateDirectory, ...(pendingCompletion ? ["--terminal-result-file", completionPath] : toolHandlerOrder ? ["--emit-tool-call"] : []), ...(goalMidTurn ? ["--goal-autostart", "--goal-item-trigger", join(stateDirectory, "emit-goal-item")] : [])),
|
||||
stateDirectory,
|
||||
prpIdentity: identity,
|
||||
lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 },
|
||||
controlPlaneRegistration: registration,
|
||||
};
|
||||
const prepareLaunch = vi.fn(async (requirements: RunnerProviderLaunchRequirements) => ({
|
||||
configurationId: "adopted-preparation", requirementsDigest: runnerLaunchRequirementsDigest(requirements),
|
||||
provider: createRunnerProviderLaunch(requirements, { options: { codexArgs: sharedOptions.codexArgs },
|
||||
root: stateDirectory, cwd: tmpdir(), codexHome: join(stateDirectory, "codex-home"),
|
||||
codexCommand: fakeCodex, providerNodeCommand: process.execPath, opencodeProxyPath: "", acpxSidecarPath: "" }),
|
||||
}));
|
||||
const options = { ...sharedOptions, ...(preparedTask ? {
|
||||
prepareLaunch, runnerArtifact: readRunnerdArtifactBinding(defaultCapabilityRunnerdBinary()),
|
||||
runnerProcessLauncher: (spec: durableControlPlane.RunnerProcessLaunchSpec) => {
|
||||
const child = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: { ...process.env, ...spec.environment },
|
||||
detached: process.platform !== "win32", stdio: "ignore" });
|
||||
return { child, processGroupId: child.pid ?? null, completion: new Promise<{ code: number | null; signal: NodeJS.Signals | null; stdout: string; stderr: string }>((resolveExit, reject) => {
|
||||
child.once("error", reject);
|
||||
child.once("exit", (code, signal) => resolveExit({ code, signal, stdout: "", stderr: "" }));
|
||||
}) };
|
||||
},
|
||||
} : {}) };
|
||||
const first = createCapabilityRunnerdCodexTransport(options);
|
||||
const pendingTool = vi.fn(() => new Promise<Record<string, unknown>>(() => undefined));
|
||||
if (toolHandlerOrder) first.transport.setServerRequestHandler(pendingTool);
|
||||
let releaseRecovery!: () => void;
|
||||
const recoveryGate = new Promise<void>(resolve => { releaseRecovery = resolve; });
|
||||
const recoveryDenied = new Error("execution authority revoked after Stop");
|
||||
const recoveredTool = vi.fn(async () => {
|
||||
if (semanticRecoveryMode) {
|
||||
// Model server authority denying replay after Stop cleared the task lock.
|
||||
// The gate holds that denial until reattachment has returned its session,
|
||||
// before the caller invokes the eventual onSession cancellation callback.
|
||||
await recoveryGate;
|
||||
throw recoveryDenied;
|
||||
}
|
||||
return { result: { receipt: "known-outcome" }, isError: false };
|
||||
});
|
||||
const redispatchedTool = vi.fn(async () => ({ success: true, contentItems: [] }));
|
||||
let runnerPid: number | null = null;
|
||||
let runnerStartedAt: string | null = null;
|
||||
let adopted: ReturnType<typeof createCapabilityRunnerdCodexTransport> | null =
|
||||
null;
|
||||
let restoreIntervalSpies: () => void = () => undefined;
|
||||
try {
|
||||
let opened: Record<string, unknown>;
|
||||
try {
|
||||
opened = await first.transport.request("thread/start", {
|
||||
cwd: tmpdir(),
|
||||
dynamicTools: toolHandlerOrder && !pendingCompletion ? [{ name: "get_task_context", description: "Read current task", inputSchema: { type: "object", properties: {}, additionalProperties: false } }] : codexSemanticToolSpecs(),
|
||||
});
|
||||
} catch (error) {
|
||||
const stderr = await readFile(
|
||||
join(stateDirectory, "diagnostics", "runnerd.stderr.log"),
|
||||
"utf8",
|
||||
).catch(() => "");
|
||||
throw new Error(
|
||||
`${String(error)}\n${JSON.stringify(first.evidence())}${stderr ? `\n${stderr}` : ""}`,
|
||||
);
|
||||
}
|
||||
runnerPid = first.evidence().runnerPid;
|
||||
runnerStartedAt = first.transport.processInfo().startedAt;
|
||||
expect(runnerPid).toEqual(expect.any(Number));
|
||||
|
||||
if (goalMidTurn) {
|
||||
await first.transport.request("thread/goal/set", { objective: "Recover a live goal", status: "active" });
|
||||
for await (const event of first.transport.notifications()) {
|
||||
if (event.method === "turn/started") break;
|
||||
}
|
||||
}
|
||||
|
||||
if (toolHandlerOrder) {
|
||||
await first.transport.request("turn/start", { input: [{ type: "text", text: "Read current task" }] });
|
||||
await vi.waitFor(() => expect(pendingTool).toHaveBeenCalledOnce(), { timeout: 5_000 });
|
||||
}
|
||||
await first.detachControllerForRestart();
|
||||
expect(() => process.kill(runnerPid!, 0)).not.toThrow();
|
||||
if (goalMidTurn) {
|
||||
await writeFile(join(stateDirectory, "emit-goal-item"), "emit");
|
||||
// runnerd need not poll the provider into its PRP outbox while disconnected.
|
||||
// Wait for flushed provider output, not a platform-dependent final poll
|
||||
// racing the disconnect. Adoption must still bind that buffered item.
|
||||
await vi.waitFor(async () => {
|
||||
expect(await readFile(join(stateDirectory, "emit-goal-item.sent"), "utf8")).toBe("sent");
|
||||
}, { timeout: 5_000 });
|
||||
}
|
||||
|
||||
const controlPlaneStatePath = join(
|
||||
stateDirectory,
|
||||
"control-plane",
|
||||
"control-plane-state.json",
|
||||
);
|
||||
const compactProviderIdentityEvents = async () => {
|
||||
const controlPlaneState = JSON.parse(
|
||||
await readFile(controlPlaneStatePath, "utf8"),
|
||||
) as { committedEvents: Array<{ eventType: string }> };
|
||||
controlPlaneState.committedEvents =
|
||||
controlPlaneState.committedEvents.filter(
|
||||
(event) =>
|
||||
event.eventType !== "harness.ready" &&
|
||||
event.eventType !== "session.started" &&
|
||||
event.eventType !== "session.resumed",
|
||||
);
|
||||
await writeFile(
|
||||
controlPlaneStatePath,
|
||||
`${JSON.stringify(controlPlaneState, null, 2)}\n`,
|
||||
{ mode: 0o600 },
|
||||
);
|
||||
};
|
||||
if (!startWithoutCheckpoint) await compactProviderIdentityEvents();
|
||||
|
||||
if (preparedTask) {
|
||||
if (adoptionAuthorization !== "legacy") await rm(join(stateDirectory, "launch-preparation.json"));
|
||||
prepareLaunch.mockClear();
|
||||
registration.mockClear();
|
||||
if (adoptionAuthorization === "missing-binding") {
|
||||
const retained = JSON.parse(await readFile(controlPlaneStatePath, "utf8"));
|
||||
delete retained.preparedLaunchConfigurations;
|
||||
await writeFile(controlPlaneStatePath, JSON.stringify(retained), { mode: 0o600 });
|
||||
}
|
||||
}
|
||||
const authorizeRunnerAdoption = vi.fn(async () => {
|
||||
expect(registration).not.toHaveBeenCalled();
|
||||
if (adoptionAuthorization === "deny") throw new Error("project grant revoked");
|
||||
});
|
||||
const duplicateLauncher = vi.fn(() => {
|
||||
throw new Error("duplicate runner spawn attempted");
|
||||
});
|
||||
const openedThread = opened.thread as Record<string, unknown>;
|
||||
const signal = vi.fn(() => true);
|
||||
const probeError = new Error("fixture remote process identity is unavailable");
|
||||
let rejectLiveness = false;
|
||||
const intervals = uncertainAtClose ? vi.spyOn(globalThis, "setInterval") : null;
|
||||
const clearedIntervals = uncertainAtClose ? vi.spyOn(globalThis, "clearInterval") : null;
|
||||
restoreIntervalSpies = () => { intervals?.mockRestore(); clearedIntervals?.mockRestore(); };
|
||||
adopted = createCapabilityRunnerdCodexTransport({
|
||||
...options,
|
||||
...(preparedTask && !adoptionAuthorization.startsWith("legacy") ? { authorizeRunnerAdoption } : {}),
|
||||
...(uncertainAtClose ? { closeGraceMs: 2_000 } : {}),
|
||||
...(semanticRecoveryMode === "cleanup-only" ? { recoveryPurpose: "stop_cleanup" as const } : {}),
|
||||
// Hash different stable bytes without replacing the real runner artifact
|
||||
// used by concurrent tests. Adoption must never execute this path.
|
||||
...(mismatchedArtifact
|
||||
? {
|
||||
runnerBinary: resolve(import.meta.dirname, "../../package.json"),
|
||||
runnerReconnectGraceMs: 150,
|
||||
}
|
||||
: {}),
|
||||
...(toolHandlerOrder && !pendingCompletion ? { recoverSemanticToolInput: recoveredTool } : {}),
|
||||
resumeDynamicTools: pendingCompletion ? codexSemanticToolSpecs() : toolHandlerOrder ? [{ name: "get_task_context", description: "Read current task", inputSchema: { type: "object", properties: {}, additionalProperties: false } }] : [],
|
||||
resumeProviderSession: startWithoutCheckpoint ? undefined : {
|
||||
driverSessionId: String(openedThread.id),
|
||||
providerSessionId: mismatchedCheckpoint
|
||||
? "wrong-provider-session"
|
||||
: String(openedThread.sessionId),
|
||||
},
|
||||
runnerProcessLauncher: duplicateLauncher,
|
||||
adoptExistingRunner: {
|
||||
pid: runnerPid!,
|
||||
processGroupId: runnerPid,
|
||||
startedAt: runnerStartedAt,
|
||||
signal,
|
||||
isAlive: () => {
|
||||
if (rejectLiveness) return Promise.reject(probeError);
|
||||
try {
|
||||
process.kill(runnerPid!, 0);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
},
|
||||
},
|
||||
});
|
||||
if (preparedTask && ["deny", "legacy-missing", "missing-binding"].includes(adoptionAuthorization)) {
|
||||
await expect(adopted.transport.request("thread/read", {})).rejects.toThrow(
|
||||
adoptionAuthorization === "deny" ? "project grant revoked" : adoptionAuthorization === "legacy-missing"
|
||||
? "runner_launch_recovery_requirements_unavailable" : "runner_launch_recovery_configuration_unavailable");
|
||||
expect(registration).not.toHaveBeenCalled();
|
||||
expect(prepareLaunch).not.toHaveBeenCalled();
|
||||
expect(duplicateLauncher).not.toHaveBeenCalled();
|
||||
expect(signal).not.toHaveBeenCalled();
|
||||
expect(() => process.kill(runnerPid!, 0)).not.toThrow();
|
||||
expect(authorizeRunnerAdoption).toHaveBeenCalledTimes(adoptionAuthorization === "deny" ? 1 : 0);
|
||||
return;
|
||||
}
|
||||
if (mismatchedArtifact) {
|
||||
await expect(
|
||||
adopted.transport.request("thread/read", {}),
|
||||
).rejects.toThrow("native_adopted_runner_authentication_timeout");
|
||||
expect(authority?.activeRunnerConnectionCount()).toBe(0);
|
||||
await expect(
|
||||
adopted.transport.request("turn/start", {
|
||||
input: [{ type: "text", text: "must not be dispatched" }],
|
||||
}),
|
||||
).rejects.toThrow("native_adopted_runner_authentication_timeout");
|
||||
await adopted.transport.close();
|
||||
expect(signal).not.toHaveBeenCalled();
|
||||
expect(checkpoint).not.toHaveBeenCalled();
|
||||
expect(duplicateLauncher).not.toHaveBeenCalled();
|
||||
expect(() => process.kill(runnerPid!, 0)).not.toThrow();
|
||||
const retained = JSON.parse(
|
||||
await readFile(controlPlaneStatePath, "utf8"),
|
||||
) as {
|
||||
identity: unknown;
|
||||
commands: Array<{ type: string; status: string }>;
|
||||
};
|
||||
expect(retained.identity).toEqual(identity);
|
||||
expect(
|
||||
retained.commands.some(
|
||||
(command) =>
|
||||
command.type === "runner.drain" && command.status === "pending",
|
||||
),
|
||||
).toBe(true);
|
||||
expect(retained.commands.map((command) => command.type)).not.toEqual(
|
||||
expect.arrayContaining(["runner.suspend"]),
|
||||
);
|
||||
expect(retained.commands.map((command) => command.type)).not.toEqual(
|
||||
expect.arrayContaining(["turn.stop"]),
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (mismatchedCheckpoint) {
|
||||
await expect(
|
||||
adopted.transport.request("thread/read", {}),
|
||||
).rejects.toThrow("native_adopted_provider_identity_mismatch");
|
||||
expect(duplicateLauncher).not.toHaveBeenCalled();
|
||||
expect(() => process.kill(runnerPid!, 0)).not.toThrow();
|
||||
return;
|
||||
}
|
||||
if (toolHandlerOrder?.endsWith("before")) adopted.transport.setServerRequestHandler(redispatchedTool);
|
||||
await expect(adopted.transport.request(startWithoutCheckpoint ? "thread/start" : "thread/read", {})).resolves.toEqual(
|
||||
expect.objectContaining({
|
||||
thread: expect.objectContaining({ id: "codex-thread-1" }),
|
||||
}),
|
||||
);
|
||||
if (toolHandlerOrder?.endsWith("after")) adopted.transport.setServerRequestHandler(redispatchedTool);
|
||||
if (semanticRecoveryMode) {
|
||||
const retained = authority as DurablePrpControlPlane | null;
|
||||
expect(retained).not.toBeNull();
|
||||
expect(retained!.store.state.committedEvents.some(event => event.eventType === "semantic_tool.input")).toBe(true);
|
||||
expect(retained!.store.state.commands.filter(command => command.type === "turn.start")).toHaveLength(1);
|
||||
expect(retained!.store.state.commands.some(command => command.type === "turn.interrupt")).toBe(false);
|
||||
if (semanticRecoveryMode === "denied-before-cancel") {
|
||||
await vi.waitFor(() => expect(recoveredTool).toHaveBeenCalledOnce());
|
||||
// This is a real Core replay callback, not an artificial transport error.
|
||||
// Deliver its denial before giving the caller its cancellation turn.
|
||||
releaseRecovery();
|
||||
await vi.waitFor(async () => {
|
||||
await expect(adopted!.transport.request("thread/read", {})).rejects.toBe(recoveryDenied);
|
||||
});
|
||||
await expect(adopted.transport.request("turn/interrupt", {})).rejects.toBe(recoveryDenied);
|
||||
expect(retained!.store.state.commands.some(command => command.type === "turn.interrupt")).toBe(false);
|
||||
} else {
|
||||
// Let all startup/handler microtasks and a real reconciliation read run.
|
||||
// Cleanup must not enter replay even while its authority would deny it.
|
||||
await adopted.transport.request("thread/read", {});
|
||||
expect(recoveredTool).not.toHaveBeenCalled();
|
||||
expect(redispatchedTool).not.toHaveBeenCalled();
|
||||
releaseRecovery();
|
||||
await expect(adopted.transport.request("turn/interrupt", {})).resolves.toEqual({});
|
||||
expect(retained!.store.state.commands.filter(command => command.type === "turn.interrupt")).toMatchObject([{ status: "completed" }]);
|
||||
}
|
||||
expect(redispatchedTool).not.toHaveBeenCalled();
|
||||
expect(retained!.store.state.commands.some(command => command.type === "semantic_tool.result")).toBe(false);
|
||||
expect(retained!.store.state.commands.filter(command => command.type === "turn.start")).toHaveLength(1);
|
||||
expect(duplicateLauncher).not.toHaveBeenCalled();
|
||||
if (semanticRecoveryMode === "cleanup-only") {
|
||||
const faultedStopProof = vi.fn();
|
||||
if (stopExitProofFault) {
|
||||
const queue = retained!.queueCommand.bind(retained!);
|
||||
vi.spyOn(retained!, "queueCommand").mockImplementation((type, ...args) => {
|
||||
const command = queue(type, ...args);
|
||||
if (type === "turn.stop") {
|
||||
let result = command.result;
|
||||
Object.defineProperty(command, "result", {
|
||||
enumerable: true,
|
||||
configurable: true,
|
||||
get: () => result,
|
||||
set: value => {
|
||||
const receipt = value as { result?: Record<string, unknown> } | null;
|
||||
if (receipt?.result && typeof receipt.result === "object") {
|
||||
faultedStopProof(receipt.result.providerExitConfirmed);
|
||||
const stopped = { ...receipt.result };
|
||||
if (stopExitProofFault === "missing") delete stopped.providerExitConfirmed;
|
||||
else stopped.providerExitConfirmed = false;
|
||||
result = { ...receipt, result: stopped };
|
||||
} else result = value;
|
||||
},
|
||||
});
|
||||
}
|
||||
return command;
|
||||
});
|
||||
await expect(adopted.transport.close()).rejects.toBeInstanceOf(NativeSessionCloseUnrecoverableError);
|
||||
expect(faultedStopProof).toHaveBeenCalledWith(true);
|
||||
expect(checkpoint).toHaveBeenCalledExactlyOnceWith("unsettled");
|
||||
} else {
|
||||
await expect(adopted.transport.close()).resolves.toBeUndefined();
|
||||
expect(checkpoint).toHaveBeenCalledExactlyOnceWith("settled");
|
||||
expect(retained!.store.state.commands.filter(command => command.type === "turn.stop")).toMatchObject([{
|
||||
status: "completed", result: { result: { providerExitConfirmed: true } },
|
||||
}]);
|
||||
}
|
||||
expect(retained!.semanticToolResultsSettled()).toBe(false);
|
||||
expect(retained!.store.state.commands.some(command => command.type === "semantic_tool.result")).toBe(false);
|
||||
expect(retained!.store.state.committedEvents.some(event => event.eventType === "semantic_tool.input")).toBe(true);
|
||||
if (!stopExitProofFault) {
|
||||
expect(retained!.store.state.commands.filter(command => command.type === "runner.suspend")).toMatchObject([{ status: "completed" }]);
|
||||
const runnerState = JSON.parse(await readFile(join(stateDirectory, "runner", "runner-state.json"), "utf8"));
|
||||
expect(runnerState).toMatchObject({ ...identity, lifecycle: "suspended" });
|
||||
}
|
||||
const providerState = JSON.parse(await readFile(join(stateDirectory, "runner", "codex-provider-state.json"), "utf8"));
|
||||
expect(providerState).toMatchObject({ activeProviderTurnId: null, pendingEvents: [], queuedEvents: [] });
|
||||
expect(recoveredTool).not.toHaveBeenCalled();
|
||||
expect(redispatchedTool).not.toHaveBeenCalled();
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (toolHandlerOrder) {
|
||||
await vi.waitFor(() => {
|
||||
expect(pendingCompletion ? redispatchedTool : recoveredTool).toHaveBeenCalledOnce();
|
||||
const result = authority!.store.state.commands.find(command => command.type === "semantic_tool.result");
|
||||
expect(result?.status).toBe("completed");
|
||||
}, { timeout: 5_000 });
|
||||
if (pendingCompletion) {
|
||||
expect(recoveredTool).not.toHaveBeenCalled();
|
||||
expect(redispatchedTool).toHaveBeenCalledWith(expect.objectContaining({ params: expect.objectContaining({ tool: "paperclip_finish" }) }));
|
||||
} else expect(redispatchedTool).not.toHaveBeenCalled();
|
||||
}
|
||||
expect(adopted.evidence().runnerPid).toBe(runnerPid);
|
||||
expect(adopted.transport.processInfo().startedAt).toBe(runnerStartedAt);
|
||||
if (startWithoutCheckpoint) {
|
||||
const retained = JSON.parse(await readFile(controlPlaneStatePath, "utf8"));
|
||||
for (const type of ["run.prepare", "session.open"]) {
|
||||
expect(retained.commands.filter((command: { type: string }) => command.type === type)).toHaveLength(1);
|
||||
}
|
||||
}
|
||||
if (goalMidTurn) {
|
||||
const observed = await Promise.race([
|
||||
(async () => {
|
||||
for await (const notification of adopted!.transport.notifications()) {
|
||||
if (notification.method === "item/started") return notification;
|
||||
}
|
||||
throw new Error("recovered goal item was lost");
|
||||
})(),
|
||||
new Promise<never>((_, reject) => setTimeout(() => reject(new Error("recovered item timed out")), 5_000)),
|
||||
]);
|
||||
expect(observed.params).toMatchObject({ threadId: "codex-thread-1", turnId: "provider-goal-turn-1" });
|
||||
}
|
||||
expect(duplicateLauncher).not.toHaveBeenCalled();
|
||||
if (preparedTask) {
|
||||
if (adoptionAuthorization === "legacy") {
|
||||
expect(prepareLaunch).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ identity }), expect.objectContaining({ configurationId: "adopted-preparation" }));
|
||||
expect(authorizeRunnerAdoption).not.toHaveBeenCalled();
|
||||
} else {
|
||||
expect(prepareLaunch).not.toHaveBeenCalled();
|
||||
expect(authorizeRunnerAdoption).toHaveBeenCalledExactlyOnceWith("adopted-preparation", identity);
|
||||
}
|
||||
expect((await adopted.transport.request("thread/read", {})).thread).toMatchObject({ cwd: tmpdir() });
|
||||
}
|
||||
if (uncertainAtClose) {
|
||||
const before = JSON.parse(await readFile(controlPlaneStatePath, "utf8"));
|
||||
const retainedAuthority = authority as DurablePrpControlPlane | null;
|
||||
rejectLiveness = true;
|
||||
await expect(adopted.transport.close()).rejects.toBe(probeError);
|
||||
expect(retainedAuthority?.activeRunnerConnectionCount()).toBe(0);
|
||||
expect(intervals!.mock.results.length).toBeGreaterThan(0);
|
||||
for (const interval of intervals!.mock.results) if (interval.type === "return") expect(clearedIntervals).toHaveBeenCalledWith(interval.value);
|
||||
const notificationsClosed = (async () => { for await (const _event of adopted!.transport.notifications()) {} return true; })();
|
||||
let queueDeadline: ReturnType<typeof setTimeout> | undefined;
|
||||
try {
|
||||
expect(await Promise.race([notificationsClosed, new Promise<boolean>(resolve => { queueDeadline = setTimeout(() => resolve(false), 250); })])).toBe(true);
|
||||
} finally { clearTimeout(queueDeadline); }
|
||||
await expect(adopted.transport.close()).rejects.toBe(probeError);
|
||||
expect(adopted.transport.processInfo()).toMatchObject({ exited: false });
|
||||
expect(() => process.kill(runnerPid!, 0)).not.toThrow();
|
||||
expect(signal).not.toHaveBeenCalled();
|
||||
expect(checkpoint).not.toHaveBeenCalled();
|
||||
expect(duplicateLauncher).not.toHaveBeenCalled();
|
||||
const retained = JSON.parse(await readFile(controlPlaneStatePath, "utf8"));
|
||||
expect(retained.identity).toEqual(before.identity);
|
||||
expect(retained.commands).toEqual(before.commands);
|
||||
return;
|
||||
}
|
||||
expect(adopted.evidence().diagnostics).toContain(
|
||||
`adopted runner ${runnerPid} authenticated to its durable PRP authority`,
|
||||
);
|
||||
if (!startWithoutCheckpoint) {
|
||||
expect(adopted.evidence().diagnostics).toContain(
|
||||
"restored adopted provider identity from the exact durable checkpoint after PRP event compaction; awaiting live confirmation",
|
||||
);
|
||||
expect(adopted.evidence().diagnostics).toContain(
|
||||
"confirmed adopted provider identity against authenticated recovery session.snapshot",
|
||||
);
|
||||
}
|
||||
if (detachBeforeCleanup) {
|
||||
await adopted.detachControllerForRestart();
|
||||
await adopted.transport.close("old controller finalizer");
|
||||
expect(signal).not.toHaveBeenCalled();
|
||||
expect(() => process.kill(runnerPid!, 0)).not.toThrow();
|
||||
const retained = JSON.parse(await readFile(controlPlaneStatePath, "utf8"));
|
||||
const commandTypes = retained.commands.map((command: { type: string }) => command.type);
|
||||
expect(commandTypes).not.toContain("turn.stop");
|
||||
expect(commandTypes).not.toContain("runner.suspend");
|
||||
}
|
||||
} finally {
|
||||
releaseRecovery();
|
||||
await adopted?.transport.close().catch(() => undefined);
|
||||
if (uncertainAtClose) {
|
||||
const providerPid = adopted?.evidence().codexPid;
|
||||
if (providerPid) {
|
||||
try { process.kill(providerPid, "SIGKILL"); }
|
||||
catch (error) { if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; }
|
||||
}
|
||||
await (authority as DurablePrpControlPlane | null)?.stop();
|
||||
}
|
||||
restoreIntervalSpies();
|
||||
if (runnerPid) {
|
||||
try {
|
||||
process.kill(-runnerPid, "SIGKILL");
|
||||
} catch {
|
||||
// The adopted runner normally exits after its durable suspend command.
|
||||
}
|
||||
}
|
||||
server.closeAllConnections();
|
||||
if (server.listening) {
|
||||
await new Promise<void>((resolveClose) =>
|
||||
server.close(() => resolveClose()),
|
||||
);
|
||||
}
|
||||
await rm(stateDirectory, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
it(
|
||||
"adopts a live runner on the same durable authority without spawning a duplicate",
|
||||
() => verifyLiveRunnerAdoption(false),
|
||||
30_000,
|
||||
);
|
||||
|
||||
it("adopts a prepared runner after losing its controller launch cache without preparing resources again",
|
||||
() => verifyLiveRunnerAdoption(false, false, false, false, false, true), 30_000);
|
||||
|
||||
it.each(["deny", "legacy", "legacy-missing", "missing-binding"] as const)("preserves adoption authorization with %s",
|
||||
mode => verifyLiveRunnerAdoption(false, false, false, false, false, true, mode), 30_000);
|
||||
|
||||
it("retains authenticated adopted-runner ownership when a close probe rejects", () => verifyLiveRunnerAdoption(false, false, false, false, false, false, "allow", undefined, true), 30_000);
|
||||
|
||||
it(
|
||||
"blocks adopted runner artifact drift without duplicate launch, checkpoint replacement, or process signals",
|
||||
() => verifyLiveRunnerAdoption(false, true),
|
||||
15_000,
|
||||
);
|
||||
|
||||
it(
|
||||
"rejects a live runner whose provider identity mismatches the compacted checkpoint",
|
||||
() => verifyLiveRunnerAdoption(true),
|
||||
30_000,
|
||||
);
|
||||
|
||||
it("binds buffered mid-goal items only after the authenticated recovery snapshot", () => verifyLiveRunnerAdoption(false, false, true), 30_000);
|
||||
|
||||
it("keeps an adopted runner alive when the detached controller finalizer closes", () => verifyLiveRunnerAdoption(false, false, true, true), 30_000);
|
||||
|
||||
it("adopts an opening session without a checkpoint instead of bootstrapping a duplicate provider", () => verifyLiveRunnerAdoption(false, false, false, false, true), 30_000);
|
||||
|
||||
it.each(["before", "after"] as const)("settles a saved tool outcome with the handler installed %s recovery startup", order =>
|
||||
verifyLiveRunnerAdoption(false, false, false, false, false, false, "allow", order), 30_000);
|
||||
|
||||
it.each(["completion-before", "completion-after"] as const)("recovers a pending finish without an ordinary-tool lookup (%s)", order =>
|
||||
verifyLiveRunnerAdoption(false, false, false, false, false, false, "allow", order), 30_000);
|
||||
|
||||
it("exposes semantic recovery before cancellation when the authority has been revoked", () =>
|
||||
verifyLiveRunnerAdoption(false, false, false, false, false, false, "allow", "before", false, "denied-before-cancel"), 30_000);
|
||||
|
||||
it.each(["before", "after", "completion-before", "completion-after"] as const)("allows cleanup-only adoption to cancel without replaying a pending semantic input (%s)", order =>
|
||||
verifyLiveRunnerAdoption(false, false, false, false, false, false, "allow", order, false, "cleanup-only"), 30_000);
|
||||
|
||||
it.each(["missing", "false"] as const)("refuses cleanup settlement when the exact stop receipt has %s provider exit proof", fault =>
|
||||
verifyLiveRunnerAdoption(false, false, false, false, false, false, "allow", "before", false, "cleanup-only", fault), 30_000);
|
||||
+1077
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,769 @@
|
||||
import { mkdtemp, open, readFile, readdir, rm, stat, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { expect, it, vi } from "vitest";
|
||||
import type { ControlPlanePort } from "../contracts/control-plane-port.js";
|
||||
import type { NativeExecutionInputV1 } from "../contracts/native-execution.js";
|
||||
import type { NativeSession, NativeSessionBackend } from "../contracts/native-session-backend.js";
|
||||
import type { PrpEvent, PrpStructuredRunResult, PrpTerminalState } from "../protocol/replay-contract.js";
|
||||
import { executeNativeSession } from "../native-session-runtime.js";
|
||||
import { createCodexTaskEnvelope } from "../contracts/codex.js";
|
||||
import { CodexAppServerDriver } from "../drivers/codex/codex-app-server-driver.js";
|
||||
import { createCapabilityRunnerdCodexTransport } from "./runnerd-codex-transport.js";
|
||||
import { defaultCapabilityRunnerdBinary, expectTurnStarted, fakeCodex, fakeCodexArgs } from "../../test/support/runnerd-transport.js";
|
||||
|
||||
it("rejects active work and buffered tools from a resumed stopped checkpoint", async () => {
|
||||
const stateDirectory = await mkdtemp(
|
||||
join(tmpdir(), "runnerd-stopped-resume-active-"),
|
||||
);
|
||||
const identity = {
|
||||
runnerInstanceId: "runner-stopped-active",
|
||||
environmentLeaseId: "lease-stopped-active",
|
||||
runId: "run-stopped-first",
|
||||
normalizedSessionId: "session-stopped-active",
|
||||
turnId: "turn-stopped-first",
|
||||
itemId: "item-stopped-first",
|
||||
};
|
||||
const options = {
|
||||
runnerBinary: defaultCapabilityRunnerdBinary(),
|
||||
codexCommand: fakeCodex,
|
||||
codexArgs: fakeCodexArgs(
|
||||
stateDirectory,
|
||||
"--linger-after-turn-start",
|
||||
"--resume-unowned-turn-when-marked",
|
||||
"--emit-tool-call-on-resume",
|
||||
),
|
||||
stateDirectory,
|
||||
lifecyclePolicy: { mode: "per_turn" as const, idleTimeoutMs: null },
|
||||
};
|
||||
const first = createCapabilityRunnerdCodexTransport({
|
||||
...options,
|
||||
prpIdentity: identity,
|
||||
});
|
||||
let second:
|
||||
ReturnType<typeof createCapabilityRunnerdCodexTransport> | undefined;
|
||||
const semanticHandler = vi.fn(async () => ({
|
||||
success: true,
|
||||
contentItems: [],
|
||||
}));
|
||||
first.transport.setServerRequestHandler(semanticHandler);
|
||||
try {
|
||||
const opened = await first.transport.request("thread/start", {
|
||||
cwd: tmpdir(),
|
||||
dynamicTools: [
|
||||
{
|
||||
name: "get_task_context",
|
||||
description: "Read the current task.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {},
|
||||
additionalProperties: false,
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
await first.transport.request("turn/start", {
|
||||
input: [{ type: "text", text: "Wait for another instruction." }],
|
||||
});
|
||||
await expectTurnStarted(
|
||||
first.transport.notifications()[Symbol.asyncIterator](),
|
||||
);
|
||||
await first.transport.close();
|
||||
const providerPath = join(
|
||||
stateDirectory,
|
||||
"runner",
|
||||
"codex-provider-state.json",
|
||||
);
|
||||
expect(JSON.parse(await readFile(providerPath, "utf8"))).toMatchObject({
|
||||
lifecycle: "prepared",
|
||||
activeProviderTurnId: null,
|
||||
});
|
||||
await writeFile(join(stateDirectory, "resume-unowned-turn"), "armed");
|
||||
const secondIdentity = {
|
||||
...identity,
|
||||
runId: "run-stopped-second",
|
||||
turnId: "turn-stopped-second",
|
||||
itemId: "item-stopped-second",
|
||||
};
|
||||
second = createCapabilityRunnerdCodexTransport({
|
||||
...options,
|
||||
prpIdentity: secondIdentity,
|
||||
});
|
||||
second.transport.setServerRequestHandler(semanticHandler);
|
||||
await expect(second.transport.request("thread/read", {})).rejects.toThrow(
|
||||
"prepared provider checkpoint resumed unexpected active work",
|
||||
);
|
||||
// Exercise the normal durable transfer before inspecting the rejection:
|
||||
// provider pendingEvents is an acknowledged queue, not an event journal.
|
||||
await vi.waitFor(
|
||||
async () => {
|
||||
const provider = JSON.parse(await readFile(providerPath, "utf8"));
|
||||
expect(provider.pendingEvents).toEqual([]);
|
||||
const control = JSON.parse(
|
||||
await readFile(
|
||||
join(stateDirectory, "control-plane", "control-plane-state.json"),
|
||||
"utf8",
|
||||
),
|
||||
) as {
|
||||
committedEvents: Array<{
|
||||
eventType: string;
|
||||
envelope: { payload: { payload: { code?: string } } };
|
||||
}>;
|
||||
};
|
||||
const rejections = control.committedEvents.filter(
|
||||
(event) =>
|
||||
event.eventType === "harness.diagnostic" &&
|
||||
event.envelope.payload.payload.code ===
|
||||
"prepared_provider_checkpoint_has_active_work",
|
||||
);
|
||||
expect(rejections).toEqual([
|
||||
expect.objectContaining({
|
||||
logicalEffectCount: 1,
|
||||
envelope: expect.objectContaining({
|
||||
...secondIdentity,
|
||||
payload: expect.objectContaining({
|
||||
payload: expect.objectContaining({
|
||||
code: "prepared_provider_checkpoint_has_active_work",
|
||||
paperclipAccepted: false,
|
||||
providerReportedActive: true,
|
||||
}),
|
||||
}),
|
||||
}),
|
||||
}),
|
||||
]);
|
||||
},
|
||||
{ timeout: 5_000 },
|
||||
);
|
||||
const closed = JSON.parse(await readFile(providerPath, "utf8"));
|
||||
expect(closed).toMatchObject({
|
||||
lifecycle: "closed",
|
||||
threadId: (opened.thread as Record<string, unknown>).id,
|
||||
activeProviderTurnId: null,
|
||||
});
|
||||
expect(semanticHandler).not.toHaveBeenCalled();
|
||||
expect(closed.pendingEvents).toEqual([]);
|
||||
} finally {
|
||||
await Promise.allSettled([
|
||||
first.transport.close(),
|
||||
second?.transport.close(),
|
||||
]);
|
||||
await rm(stateDirectory, { recursive: true, force: true });
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
it("does not retry a real memoized transport close whose suspension proof is unavailable", async () => {
|
||||
const identity = {
|
||||
runId: "run-recovery",
|
||||
sessionId: "session-recovery",
|
||||
companyId: "company-recovery",
|
||||
issueId: "issue-recovery",
|
||||
agentId: "agent-recovery",
|
||||
};
|
||||
|
||||
const result: PrpStructuredRunResult = {
|
||||
schema: "paperclip.run_result.v1",
|
||||
reportedWorkDisposition: "done",
|
||||
summary: "Recovered native work completed.",
|
||||
completionClaim: {
|
||||
contractRevision: "1",
|
||||
objectiveSatisfied: true,
|
||||
criteria: [
|
||||
{ criterionId: "objective", status: "satisfied", evidenceRefs: [] },
|
||||
],
|
||||
remainingWork: [],
|
||||
},
|
||||
evidence: [],
|
||||
verification: [{ commandOrCheck: "recovery", status: "passed" }],
|
||||
attentionRequests: [],
|
||||
artifacts: [],
|
||||
};
|
||||
|
||||
const terminal: PrpTerminalState = {
|
||||
schema: "paperclip.prp.terminal.v1",
|
||||
turnTerminalState: "completed",
|
||||
runTerminalState: "succeeded",
|
||||
reportedWorkDisposition: "done",
|
||||
};
|
||||
|
||||
const input: NativeExecutionInputV1 = {
|
||||
schema: "paperclip.native-execution-input.v1",
|
||||
binding: {
|
||||
companyId: identity.companyId,
|
||||
runId: identity.runId,
|
||||
issueId: identity.issueId,
|
||||
agentId: identity.agentId,
|
||||
executionWorkspaceId: "workspace-recovery",
|
||||
},
|
||||
task: {
|
||||
identifier: "PAP-RECOVERY",
|
||||
title: "Recover native work",
|
||||
description: null,
|
||||
prompt: "# PAP-RECOVERY: Recover native work",
|
||||
workMode: "standard",
|
||||
},
|
||||
workspace: {
|
||||
cwd: "/workspace",
|
||||
repoUrl: null,
|
||||
repoRef: null,
|
||||
branchName: null,
|
||||
},
|
||||
session: {
|
||||
normalizedSessionId: identity.sessionId,
|
||||
driverKind: "codex_app_server",
|
||||
protocolVersion: 1,
|
||||
},
|
||||
provider: { kind: "codex", model: null },
|
||||
completionContract: {
|
||||
id: "contract-recovery",
|
||||
sha256: "contract-recovery-sha",
|
||||
schemaVersion: "paperclip.completion-contract.v1",
|
||||
contract: {
|
||||
revision: "1",
|
||||
objective: "Recover native work",
|
||||
criteria: [{ id: "objective", requirement: "Complete after recovery" }],
|
||||
},
|
||||
},
|
||||
interactionResponses: [],
|
||||
credentialBindings: [],
|
||||
};
|
||||
|
||||
function runnerEvent(
|
||||
sourceSeq: number,
|
||||
eventType: PrpEvent["eventType"],
|
||||
payload: Record<string, unknown> = {},
|
||||
): PrpEvent {
|
||||
return {
|
||||
schema: "paperclip.prp.event.v1",
|
||||
sourceEventId: `runner-recovery:${identity.runId}:${sourceSeq}`,
|
||||
sourceSeq,
|
||||
sourceInstanceId: "runner-recovery",
|
||||
sourceKind: "runner",
|
||||
runId: identity.runId,
|
||||
normalizedSessionId: identity.sessionId,
|
||||
turnId: "turn-recovery",
|
||||
eventType,
|
||||
schemaVersion: 1,
|
||||
priority: 0,
|
||||
emittedAt: "2026-08-09T00:00:00.000Z",
|
||||
payload,
|
||||
};
|
||||
}
|
||||
const stateDirectory = await mkdtemp(
|
||||
join(tmpdir(), "native-close-quarantine-"),
|
||||
);
|
||||
const readRunnerState = vi.fn(async () => ({
|
||||
schema: "paperclip.runner.durable.state.v1",
|
||||
runnerInstanceId: "runner-close-quarantine",
|
||||
environmentLeaseId: "lease-close-quarantine",
|
||||
runId: identity.runId,
|
||||
normalizedSessionId: identity.sessionId,
|
||||
turnId: "turn-close-quarantine",
|
||||
itemId: "item-close-quarantine",
|
||||
lifecycle: "ready",
|
||||
}));
|
||||
const bundle = createCapabilityRunnerdCodexTransport({
|
||||
runnerBinary: defaultCapabilityRunnerdBinary(),
|
||||
codexCommand: resolve(
|
||||
import.meta.dirname,
|
||||
"../../runner/target/debug/fake-codex-app-server",
|
||||
),
|
||||
codexArgs: ["--state-file", join(stateDirectory, "fake-codex-state.json")],
|
||||
stateDirectory,
|
||||
closeGraceMs: 400,
|
||||
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
|
||||
prpIdentity: await readRunnerState(),
|
||||
readRunnerState,
|
||||
// A checkpoint owner requires durable suspension proof. A local transport
|
||||
// without a checkpoint can simply terminate its process on close.
|
||||
controlPlaneRegistration: async (authority) => {
|
||||
await authority.start();
|
||||
return {
|
||||
connectUrl: authority.connectUrl,
|
||||
checkpoint: async () => {},
|
||||
release: async () => {},
|
||||
};
|
||||
},
|
||||
});
|
||||
try {
|
||||
await bundle.transport.request("thread/start", {
|
||||
cwd: stateDirectory,
|
||||
dynamicTools: [],
|
||||
});
|
||||
const failedClose = bundle.transport.close();
|
||||
const failure = await failedClose.catch((error: unknown) => error);
|
||||
expect(failure).toMatchObject({
|
||||
code: "native_session_close_unrecoverable",
|
||||
});
|
||||
expect(bundle.transport.close()).toBe(failedClose);
|
||||
|
||||
vi.useFakeTimers();
|
||||
const close = vi.fn(({ reason }: { reason: string }) =>
|
||||
bundle.transport.close(reason),
|
||||
);
|
||||
const capabilities = {
|
||||
resume: true,
|
||||
typedEvents: true,
|
||||
steering: false,
|
||||
interruption: true,
|
||||
structuredResult: true,
|
||||
};
|
||||
const session: NativeSession = {
|
||||
identity: () => identity,
|
||||
async capabilities() {
|
||||
return capabilities;
|
||||
},
|
||||
async *events() {
|
||||
yield runnerEvent(1, "turn.completed");
|
||||
},
|
||||
async startTurn() {
|
||||
return { turnId: "turn-recovery" };
|
||||
},
|
||||
async result() {
|
||||
return { result, terminal, turnId: "turn-recovery" };
|
||||
},
|
||||
async snapshot() {
|
||||
return {
|
||||
backendKind: "mock",
|
||||
sessionId: identity.sessionId,
|
||||
identity,
|
||||
providerSessionId: "provider-recovery",
|
||||
cursor: null,
|
||||
activeTurnId: null,
|
||||
pendingRuntimeRequests: [],
|
||||
lineage: [],
|
||||
};
|
||||
},
|
||||
close,
|
||||
};
|
||||
const openSession = vi.fn(async () => session);
|
||||
const backend: NativeSessionBackend = {
|
||||
async descriptor() {
|
||||
return {
|
||||
kind: "mock",
|
||||
name: "real-memoized-close-quarantine",
|
||||
version: "1",
|
||||
capabilities,
|
||||
};
|
||||
},
|
||||
openSession,
|
||||
};
|
||||
const port: ControlPlanePort = {
|
||||
async openRun() {},
|
||||
async checkpointSession() {},
|
||||
async appendEvent() {
|
||||
return {
|
||||
cursor: 1,
|
||||
highestContiguousSourceSeq: 1,
|
||||
disposition: "committed",
|
||||
};
|
||||
},
|
||||
async replayEvents() {
|
||||
return { events: [], highestContiguousSourceSeq: 0 };
|
||||
},
|
||||
async completeRun() {},
|
||||
};
|
||||
const execute = () =>
|
||||
executeNativeSession({
|
||||
input,
|
||||
backend,
|
||||
controlPlane: port,
|
||||
runnerInstanceId: "runner-recovery",
|
||||
controlPlaneInstanceId: "control-recovery",
|
||||
requireSessionCloseBeforeReturn: true,
|
||||
});
|
||||
await expect(execute()).rejects.toBe(failure);
|
||||
const readsAfterClose = readRunnerState.mock.calls.length;
|
||||
await expect(execute()).rejects.toMatchObject({
|
||||
code: "native_session_cleanup_quarantined",
|
||||
recovery: "operator_required",
|
||||
});
|
||||
await vi.advanceTimersByTimeAsync(600_000);
|
||||
await expect(execute()).rejects.toMatchObject({
|
||||
code: "native_session_cleanup_quarantined",
|
||||
});
|
||||
expect(openSession).toHaveBeenCalledOnce();
|
||||
expect(close).toHaveBeenCalledOnce();
|
||||
expect(readRunnerState).toHaveBeenCalledTimes(readsAfterClose);
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
await bundle.transport.close().catch(() => undefined);
|
||||
await rm(stateDirectory, { recursive: true, force: true });
|
||||
}
|
||||
}, 10_000);
|
||||
|
||||
it.each(["not_suspended", "wrong_identity"] as const)(
|
||||
"does not report local runner close healthy with %s durable evidence",
|
||||
async (mode) => {
|
||||
const stateDirectory = await mkdtemp(
|
||||
join(tmpdir(), "runnerd-local-close-unproven-"),
|
||||
);
|
||||
const identity = {
|
||||
runnerInstanceId: "runner-close-unproven",
|
||||
environmentLeaseId: "lease-close-unproven",
|
||||
runId: "run-close-unproven",
|
||||
normalizedSessionId: "session-close-unproven",
|
||||
turnId: "turn-close-unproven",
|
||||
itemId: "item-close-unproven",
|
||||
};
|
||||
const bundle = createCapabilityRunnerdCodexTransport({
|
||||
runnerBinary: defaultCapabilityRunnerdBinary(),
|
||||
codexCommand: fakeCodex,
|
||||
codexArgs: fakeCodexArgs(stateDirectory),
|
||||
stateDirectory,
|
||||
closeGraceMs: 400,
|
||||
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
|
||||
prpIdentity: identity,
|
||||
readRunnerState: async () => ({
|
||||
schema: "paperclip.runner.durable.state.v1",
|
||||
...identity,
|
||||
...(mode === "wrong_identity" ? { runId: "some-other-run" } : {}),
|
||||
lifecycle: mode === "not_suspended" ? "ready" : "suspended",
|
||||
}),
|
||||
});
|
||||
bundle.transport.setServerRequestHandler(async () => ({
|
||||
success: true,
|
||||
contentItems: [],
|
||||
}));
|
||||
try {
|
||||
await bundle.transport.request("thread/start", {
|
||||
cwd: tmpdir(),
|
||||
dynamicTools: [],
|
||||
});
|
||||
await expect(bundle.transport.close()).rejects.toThrow(
|
||||
"runner did not durably suspend before checkpoint",
|
||||
);
|
||||
const control = JSON.parse(
|
||||
await readFile(
|
||||
join(stateDirectory, "control-plane", "control-plane-state.json"),
|
||||
"utf8",
|
||||
),
|
||||
);
|
||||
expect(control.identity).toEqual(identity);
|
||||
expect(await readdir(stateDirectory)).toContain("runner");
|
||||
} finally {
|
||||
await bundle.transport.close().catch(() => undefined);
|
||||
await rm(stateDirectory, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
10_000,
|
||||
);
|
||||
|
||||
it.each([
|
||||
{
|
||||
label: "interrupted",
|
||||
args: ["--hold-turn"],
|
||||
terminal: "turn.interrupted",
|
||||
disposition: "needs_review",
|
||||
},
|
||||
{
|
||||
label: "failed",
|
||||
args: ["--fail-turn-immediately"],
|
||||
terminal: "turn.failed",
|
||||
disposition: "needs_review",
|
||||
},
|
||||
{
|
||||
label: "completed",
|
||||
args: [],
|
||||
terminal: "turn.completed",
|
||||
disposition: "done",
|
||||
},
|
||||
])("retains a $label runner's result before closing its provider turn", async ({
|
||||
label, args, terminal, disposition,
|
||||
}) => {
|
||||
const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-stop-result-"));
|
||||
const bundle = createCapabilityRunnerdCodexTransport({
|
||||
runnerBinary: defaultCapabilityRunnerdBinary(),
|
||||
codexCommand: fakeCodex,
|
||||
codexArgs: fakeCodexArgs(stateDirectory, ...args),
|
||||
stateDirectory,
|
||||
});
|
||||
const driver = new CodexAppServerDriver({
|
||||
taskEnvelope: createCodexTaskEnvelope({
|
||||
objective: "Stop and retain unfinished work.",
|
||||
}),
|
||||
environment: {
|
||||
PATH: process.env.PATH,
|
||||
HOME: join(tmpdir(), "runnerd-stop-host-home"),
|
||||
PAPERCLIP_WORKSPACE_CWD: stateDirectory,
|
||||
},
|
||||
approvalPolicy: "never",
|
||||
transportFactory: () => bundle.transport,
|
||||
});
|
||||
const session = await driver.openSession({
|
||||
runId: "run-stop-result",
|
||||
normalizedSessionId: "session-stop-result",
|
||||
workingDirectory: stateDirectory,
|
||||
});
|
||||
try {
|
||||
const turn = await session.startTurn({
|
||||
message: { role: "user", text: "Keep working until interrupted." },
|
||||
});
|
||||
if (label === "interrupted") {
|
||||
await session.interrupt({
|
||||
turnId: turn.turnId,
|
||||
reason: "Stopped by the user",
|
||||
});
|
||||
}
|
||||
const events: PrpEvent[] = [];
|
||||
for await (const event of session.events()) {
|
||||
events.push(event);
|
||||
if (event.eventType === "session.failed" || event.eventType === terminal) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
// Let already committed durable suffix events reach the facade as well.
|
||||
await bundle.transport.request("thread/read", {});
|
||||
const snapshot = await session.snapshot();
|
||||
expect(events.some((event) => event.eventType === "session.failed")).toBe(false);
|
||||
expect(events.map((event) => event.eventType)).toContain("run.result.proposed");
|
||||
expect(events.at(-1)?.eventType).toBe(terminal);
|
||||
expect(snapshot).toMatchObject({
|
||||
activeTurnId: null,
|
||||
semanticResult: {
|
||||
turnId: turn.turnId,
|
||||
result: { reportedWorkDisposition: disposition },
|
||||
},
|
||||
});
|
||||
} finally {
|
||||
await session.close().catch(() => undefined);
|
||||
await rm(stateDirectory, {
|
||||
recursive: true,
|
||||
force: true,
|
||||
maxRetries: 5,
|
||||
retryDelay: 25,
|
||||
});
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
it.each([0, 193 * 1024 * 1024])(
|
||||
"probes an exact-authority resume with %i extra journal bytes and confirms its live provider identity",
|
||||
async (extraJournalBytes) => {
|
||||
const stateDirectory = await mkdtemp(
|
||||
join(tmpdir(), "runnerd-exact-authority-resume-"),
|
||||
);
|
||||
const identity = {
|
||||
runnerInstanceId: "runner-exact-resume",
|
||||
environmentLeaseId: "lease-exact-resume",
|
||||
runId: "run-exact-resume",
|
||||
normalizedSessionId: "session-exact-resume",
|
||||
turnId: "turn-exact-resume",
|
||||
itemId: "item-exact-resume",
|
||||
};
|
||||
const options = {
|
||||
runnerBinary: defaultCapabilityRunnerdBinary(),
|
||||
codexCommand: fakeCodex,
|
||||
codexArgs: fakeCodexArgs(stateDirectory, "--durable-turn-ids"),
|
||||
stateDirectory,
|
||||
lifecyclePolicy: { mode: "per_turn" as const, idleTimeoutMs: null },
|
||||
prpIdentity: identity,
|
||||
};
|
||||
const first = createCapabilityRunnerdCodexTransport(options);
|
||||
first.transport.setServerRequestHandler(async () => ({
|
||||
success: true,
|
||||
contentItems: [],
|
||||
}));
|
||||
let providerThread: { id: string; sessionId: string } | null = null;
|
||||
try {
|
||||
const opened = await first.transport.request("thread/start", {
|
||||
cwd: tmpdir(),
|
||||
dynamicTools: [],
|
||||
});
|
||||
const thread = opened.thread as Record<string, unknown>;
|
||||
providerThread = {
|
||||
id: String(thread.id),
|
||||
sessionId: String(thread.sessionId),
|
||||
};
|
||||
} finally {
|
||||
await first.transport.close();
|
||||
}
|
||||
if (providerThread === null) {
|
||||
throw new Error(
|
||||
"exact-authority fixture did not return a provider thread",
|
||||
);
|
||||
}
|
||||
|
||||
const statePath = join(
|
||||
stateDirectory,
|
||||
"control-plane",
|
||||
"control-plane-state.json",
|
||||
);
|
||||
if (extraJournalBytes > 0) {
|
||||
const padding = Buffer.alloc(1024 * 1024, 0x20);
|
||||
const stateHandle = await open(statePath, "a");
|
||||
try {
|
||||
for (let remaining = extraJournalBytes; remaining > 0;) {
|
||||
const bytesToWrite = Math.min(remaining, padding.length);
|
||||
await stateHandle.write(padding, 0, bytesToWrite);
|
||||
remaining -= bytesToWrite;
|
||||
}
|
||||
} finally {
|
||||
await stateHandle.close();
|
||||
}
|
||||
expect((await stat(statePath)).size).toBeGreaterThan(
|
||||
64 * 1024 * 1024,
|
||||
);
|
||||
}
|
||||
const beforeResume = JSON.parse(await readFile(statePath, "utf8")) as {
|
||||
commands: Array<{ type: string }>;
|
||||
committedEvents: Array<{ eventType: string }>;
|
||||
};
|
||||
expect(
|
||||
beforeResume.commands.some((command) => command.type === "run.attach"),
|
||||
).toBe(false);
|
||||
const priorResumeEvents = beforeResume.committedEvents.filter(
|
||||
(event) => event.eventType === "session.resumed",
|
||||
).length;
|
||||
const priorSnapshots = beforeResume.commands.filter(
|
||||
(command) => command.type === "session.snapshot",
|
||||
).length;
|
||||
|
||||
const resumed = createCapabilityRunnerdCodexTransport({
|
||||
...options,
|
||||
resumeProviderSession: {
|
||||
driverSessionId: providerThread.id,
|
||||
providerSessionId: providerThread.sessionId,
|
||||
},
|
||||
});
|
||||
resumed.transport.setServerRequestHandler(async () => ({
|
||||
success: true,
|
||||
contentItems: [],
|
||||
}));
|
||||
try {
|
||||
const read = await resumed.transport.request("thread/read", {});
|
||||
expect(read.thread).toMatchObject(providerThread);
|
||||
const afterResume = JSON.parse(await readFile(statePath, "utf8")) as {
|
||||
commands: Array<{ commandId: string; type: string; status: string }>;
|
||||
committedEvents: Array<{ eventType: string }>;
|
||||
};
|
||||
expect(afterResume.commands).toContainEqual(
|
||||
expect.objectContaining({
|
||||
commandId: expect.stringMatching(/^command_resume_probe_/),
|
||||
type: "runner.drain",
|
||||
status: "completed",
|
||||
}),
|
||||
);
|
||||
expect(afterResume.commands).toContainEqual(
|
||||
expect.objectContaining({
|
||||
type: "session.snapshot",
|
||||
status: "completed",
|
||||
}),
|
||||
);
|
||||
expect(
|
||||
afterResume.commands.filter(
|
||||
(command) => command.type === "session.snapshot",
|
||||
),
|
||||
).toHaveLength(priorSnapshots + 2);
|
||||
// The authenticated snapshot above proves the live provider identity.
|
||||
// Control-first dispatch may deliver that command before the independent
|
||||
// session event is ingested. Still require exactly one durable event;
|
||||
// don't mistake an immediate file read for an event-delivery barrier.
|
||||
await vi.waitFor(
|
||||
async () => {
|
||||
const delivered = JSON.parse(await readFile(statePath, "utf8")) as {
|
||||
committedEvents: Array<{ eventType: string }>;
|
||||
};
|
||||
expect(
|
||||
delivered.committedEvents.filter(
|
||||
(event) => event.eventType === "session.resumed",
|
||||
),
|
||||
).toHaveLength(priorResumeEvents + 1);
|
||||
},
|
||||
{ timeout: 3_000, interval: 25 },
|
||||
);
|
||||
} finally {
|
||||
await resumed.transport.close();
|
||||
await rm(stateDirectory, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
30_000,
|
||||
);
|
||||
|
||||
it("still fails closed when a real close grace period cannot fit a durable suspension round trip", async () => {
|
||||
const stateDirectory = await mkdtemp(
|
||||
join(tmpdir(), "runnerd-close-grace-too-small-"),
|
||||
);
|
||||
const identity = {
|
||||
runnerInstanceId: "runner-close-grace-too-small",
|
||||
environmentLeaseId: "lease-close-grace-too-small",
|
||||
runId: "run-close-grace-too-small",
|
||||
normalizedSessionId: "session-close-grace-too-small",
|
||||
turnId: "turn-close-grace-too-small",
|
||||
itemId: "item-close-grace-too-small",
|
||||
};
|
||||
const bundle = createCapabilityRunnerdCodexTransport({
|
||||
runnerBinary: defaultCapabilityRunnerdBinary(),
|
||||
codexCommand: fakeCodex,
|
||||
codexArgs: fakeCodexArgs(stateDirectory),
|
||||
stateDirectory,
|
||||
// No real durable command round trip can complete this fast. A wider
|
||||
// budget for the provider-drain proof must not turn this barrier into
|
||||
// one that always passes; it still needs the actual proof to arrive.
|
||||
closeGraceMs: 1,
|
||||
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
|
||||
prpIdentity: identity,
|
||||
});
|
||||
bundle.transport.setServerRequestHandler(async () => ({
|
||||
success: true,
|
||||
contentItems: [],
|
||||
}));
|
||||
try {
|
||||
await bundle.transport.request("thread/start", {
|
||||
cwd: tmpdir(),
|
||||
dynamicTools: [],
|
||||
});
|
||||
const closed = bundle.transport.close();
|
||||
await expect(closed).rejects.toMatchObject({
|
||||
code: "native_session_close_unrecoverable",
|
||||
semanticToolsSettled: true,
|
||||
});
|
||||
// Retirement preserves the original rejected promise and its proof.
|
||||
expect(bundle.transport.close()).toBe(closed);
|
||||
} finally {
|
||||
await rm(stateDirectory, { recursive: true, force: true });
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
it("persists an active provider as settled before bounded suspension", async () => {
|
||||
const stateDirectory = await mkdtemp(
|
||||
join(tmpdir(), "runnerd-active-suspension-"),
|
||||
);
|
||||
const bundle = createCapabilityRunnerdCodexTransport({
|
||||
runnerBinary: defaultCapabilityRunnerdBinary(),
|
||||
codexCommand: fakeCodex,
|
||||
codexArgs: fakeCodexArgs(stateDirectory, "--linger-after-turn-start"),
|
||||
stateDirectory,
|
||||
});
|
||||
bundle.transport.setServerRequestHandler(async () => ({
|
||||
success: true,
|
||||
contentItems: [],
|
||||
}));
|
||||
try {
|
||||
await bundle.transport.request("initialize", {});
|
||||
await bundle.transport.request("thread/start", {
|
||||
cwd: tmpdir(),
|
||||
dynamicTools: [],
|
||||
});
|
||||
await bundle.transport.request("turn/start", {
|
||||
input: [{ type: "text", text: "Wait for another instruction." }],
|
||||
});
|
||||
const notifications = bundle.transport
|
||||
.notifications()
|
||||
[Symbol.asyncIterator]();
|
||||
await expectTurnStarted(notifications);
|
||||
await bundle.transport.close();
|
||||
|
||||
const providerState = JSON.parse(
|
||||
await readFile(
|
||||
join(stateDirectory, "runner", "codex-provider-state.json"),
|
||||
"utf8",
|
||||
),
|
||||
) as Record<string, unknown>;
|
||||
expect(providerState).toMatchObject({
|
||||
lifecycle: "prepared",
|
||||
activeProviderTurnId: null,
|
||||
});
|
||||
} finally {
|
||||
await bundle.transport.close();
|
||||
await rm(stateDirectory, { recursive: true, force: true });
|
||||
}
|
||||
}, 30_000);
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,88 @@
|
||||
import { join, resolve } from "node:path";
|
||||
import { expect } from "vitest";
|
||||
import {
|
||||
NATIVE_RUNTIME_ASSET_SCHEMA,
|
||||
PAPERCLIP_EXECUTION_PROMPT,
|
||||
PAPERCLIP_EXECUTION_PROMPT_REVISION,
|
||||
canonicalNativeRuntimeContextDigest,
|
||||
nativeRuntimePromptDigest,
|
||||
type NativeRuntimeContextSnapshot,
|
||||
} from "../../src/contracts/runtime-context.js";
|
||||
import { defaultCapabilityRunnerdBinary as qualifiedCapabilityRunnerdBinary } from "../../src/live/runnerd-codex-transport.js";
|
||||
|
||||
export const defaultCapabilityRunnerdBinary = () =>
|
||||
process.env.PAPERCLIP_ATTACH_TRANSITION_RUNNER ??
|
||||
qualifiedCapabilityRunnerdBinary();
|
||||
|
||||
export async function expectTurnStarted(
|
||||
notifications: AsyncIterator<{ method: string }>,
|
||||
) {
|
||||
for (let index = 0; index < 32; index += 1) {
|
||||
const next = await notifications.next();
|
||||
expect(next.done).not.toBe(true);
|
||||
if (next.value.method === "turn/started") return;
|
||||
// Startup ownership and capability facts can precede the active turn.
|
||||
expect(next.value.method).toBe("paperclip/canonicalProviderEvent");
|
||||
}
|
||||
throw new Error(
|
||||
"provider turn did not start within the bounded notification prefix",
|
||||
);
|
||||
}
|
||||
|
||||
export const fakeCodex = resolve(
|
||||
import.meta.dirname,
|
||||
"../../runner/target/debug/fake-codex-app-server",
|
||||
);
|
||||
|
||||
export function fakeCodexArgs(stateDirectory: string, ...args: string[]): string[] {
|
||||
return [
|
||||
"--state-file",
|
||||
join(stateDirectory, "fake-codex-state.json"),
|
||||
...args,
|
||||
];
|
||||
}
|
||||
|
||||
export function assignedRuntimeContext(
|
||||
skillRoot: string,
|
||||
instructionRoot: string,
|
||||
): NativeRuntimeContextSnapshot {
|
||||
const digest = "0".repeat(64);
|
||||
const value = {
|
||||
prompt: {
|
||||
revision: PAPERCLIP_EXECUTION_PROMPT_REVISION,
|
||||
text: PAPERCLIP_EXECUTION_PROMPT,
|
||||
digest: nativeRuntimePromptDigest(),
|
||||
},
|
||||
instructions: {
|
||||
entryPath: "AGENTS.md",
|
||||
bundle: {
|
||||
schema: NATIVE_RUNTIME_ASSET_SCHEMA,
|
||||
digest,
|
||||
manifestDigest: digest,
|
||||
rootPath: instructionRoot,
|
||||
fileCount: 1,
|
||||
totalBytes: 1,
|
||||
},
|
||||
},
|
||||
skills: [
|
||||
{
|
||||
key: "company/assigned",
|
||||
runtimeName: "assigned",
|
||||
versionId: "version-1",
|
||||
bundle: {
|
||||
schema: NATIVE_RUNTIME_ASSET_SCHEMA,
|
||||
digest,
|
||||
manifestDigest: digest,
|
||||
rootPath: skillRoot,
|
||||
fileCount: 1,
|
||||
totalBytes: 1,
|
||||
},
|
||||
},
|
||||
],
|
||||
mcp: { assignmentSetId: "assigned", digest, bindingId: "binding" },
|
||||
} satisfies Omit<NativeRuntimeContextSnapshot, "aggregateDigest">;
|
||||
return {
|
||||
...value,
|
||||
aggregateDigest: canonicalNativeRuntimeContextDigest(value),
|
||||
};
|
||||
}
|
||||
@@ -1,7 +1,9 @@
|
||||
import { availableParallelism } from "node:os";
|
||||
import { resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
import { configDefaults, defineConfig } from "vitest/config";
|
||||
import { RunnerSequencer } from "./scripts/vitest-sequencer.mjs";
|
||||
|
||||
const packageRoot = fileURLToPath(new URL(".", import.meta.url));
|
||||
|
||||
@@ -17,6 +19,8 @@ export default defineConfig({
|
||||
? { reporters: [...configDefaults.reporters, resolve(packageRoot, "../../scripts/ci-profile/vitest-reporter.mjs")] }
|
||||
: {}),
|
||||
environment: "node",
|
||||
...(process.env.GITHUB_ACTIONS === "true" ? { maxWorkers: Math.min(4, availableParallelism()) } : {}),
|
||||
sequence: { sequencer: RunnerSequencer },
|
||||
include: [
|
||||
"src/**/*.test.ts",
|
||||
"src/**/*.test.tsx",
|
||||
@@ -27,6 +31,7 @@ export default defineConfig({
|
||||
"examples/scenario-explorer/src/**/*.test.tsx",
|
||||
"scripts/local-runner-browser-server.test.mjs",
|
||||
"scripts/live-console-browser-server.test.mjs",
|
||||
"scripts/vitest-sequencer.test.mjs",
|
||||
],
|
||||
},
|
||||
});
|
||||
@@ -966,7 +966,9 @@ test('native dependency caches are target-bound and unmerged master callers cann
|
||||
const cache = producer.split(' - name: Cache native Rust dependencies\n')[1].split('\n - ')[0];
|
||||
assert.match(cache, /Swatinem\/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6/);
|
||||
assert.match(cache, /workspaces: \$\{\{ steps.native_rust_workspace.outputs.path \}\} -> target/);
|
||||
assert.match(cache, /shared-key: release-native-v2-\$\{\{ matrix.host \}\}-\$\{\{ matrix.target \}\}-\$\{\{ hashFiles\('packages\/paperclip-runner\/protocol\/\*\*', 'packages\/paperclip-runner\/src\/\*\*', 'packages\/paperclip-runner\/\*.json'\) \}\}/);
|
||||
const key = cache.match(/^\s*shared-key: (.+)$/m)?.[1];
|
||||
assert.ok(key?.startsWith("release-native-v2-${{ hashFiles('packages/paperclip-runner/runner/Cargo.toml', "));
|
||||
assert.ok(key?.endsWith("-${{ matrix.host }}-${{ matrix.target }}-${{ hashFiles('packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/src/**', 'packages/paperclip-runner/*.json') }}"));
|
||||
for (const option of ['cache-workspace-crates', 'cache-bin', 'cache-on-failure']) {
|
||||
assert.match(cache, new RegExp(option + ': false'));
|
||||
}
|
||||
|
||||
@@ -7,11 +7,19 @@ import { readPluginsUnder } from "./link-plugin-dev-sdk.mjs";
|
||||
|
||||
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
|
||||
const postinstallHooks = new Map([
|
||||
[".", "node scripts/link-plugin-dev-sdk.mjs"],
|
||||
["server", "node scripts/install-copilot-assets.mjs"],
|
||||
["packages/paperclip-runner", "node scripts/install-copilot-assets.mjs"],
|
||||
]);
|
||||
|
||||
export function cachePlan({ files, read, scope, packageDirs }) {
|
||||
const dependencyInputs = files.filter((file) =>
|
||||
/(^|\/)package\.json$/.test(file)
|
||||
|| ["pnpm-lock.yaml", "pnpm-workspace.yaml", ".npmrc", ".pnpmfile.cjs", "pnpmfile.cjs"].includes(file)
|
||||
|| file.startsWith("scripts/")
|
||||
|| file.startsWith("server/scripts/")
|
||||
|| file.startsWith("packages/paperclip-runner/scripts/")
|
||||
|| file.startsWith("patches/"));
|
||||
const digest = (inputs, salt) => {
|
||||
const hash = createHash("sha256").update(JSON.stringify(salt));
|
||||
@@ -28,8 +36,8 @@ export function cachePlan({ files, read, scope, packageDirs }) {
|
||||
if (manifest === null) return false;
|
||||
const scripts = JSON.parse(manifest).scripts ?? {};
|
||||
return ['preinstall', 'install', 'postinstall', 'prepare'].every((hook) =>
|
||||
scripts[hook] === undefined || (directory === '.' && hook === 'postinstall'
|
||||
&& scripts[hook] === 'node scripts/link-plugin-dev-sdk.mjs'));
|
||||
scripts[hook] === undefined || (hook === 'postinstall'
|
||||
&& scripts[hook] === postinstallHooks.get(directory)));
|
||||
});
|
||||
const buildInputs = files.filter((file) =>
|
||||
file.startsWith("packages/shared/") || file.startsWith("packages/plugins/sdk/")
|
||||
|
||||
Reference in new issue
Block a user