fix(ci): handle missing runner output and await import fixture readiness (#20339)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - CI checks must report real result failures after their setup phase.
> - Required summaries must run even when the routing gate has no runner
output.
> - Process-recovery tests must start their checkpoint deadline after
the child loads its modules.
> - An empty runner prevents summaries from starting. Slow child startup
can consume the checkpoint deadline.
> - This pull request adds a fallback for the two summaries and a ready
signal for the recovery fixture.
> - Existing routing, checkpoint, key and recovery assertions remain in
place.

## Linked Issues or Issue Description

Refs #20322. Related open PRs #12512 and #12513 propose a broader runner
contract and caller activation. This change does not adopt that contract
or change normal routing. No current repair for an empty aggregate
runner was found.

**What happened?**

The PR workflow can fail before starting any jobs when the routing gate
has no runner output. GitHub reports an empty `runs-on` value for the
`verify` and `e2e` summary jobs. Their `always()` conditions make this
path possible when a dependency cannot run. A real workflow rerun showed
both errors at lines 801 and 1784 of the trusted workflow.

The first run of this PR also failed two process-recovery cases with no
saved child checkpoints. The fixture starts its 15-second checkpoint
poll before a new Node process imports the server modules. A controlled
16-second pre-import delay reproduces the same zero-checkpoint
assertion. That evidence proves a deadline-boundary defect. The original
CI run has no phase events, so it does not prove which stage caused
those two CI failures.

**Expected behavior**

The summaries can start on `ubuntu-latest` when routing has no runner
output. They must still fail when routing or required work did not
succeed. The process fixture waits for a bounded child-ready signal
before its unchanged checkpoint deadline. It must still prove checkpoint
reuse, process termination and refusal under a changed master key.

**Steps to reproduce**

1. Start or rerun PR CI with no runner output from the routing gate.
2. Let the two `always()` summary jobs evaluate their runner
expressions.
3. Observe that the previous expressions produce an empty value and
GitHub refuses the jobs.
4. For the fixture defect, temporarily delay the child before imports
for 16 seconds. The previous fixture fails its 15-second checkpoint poll
before the child is ready. The updated fixture passes with the same
delay. The temporary delay is excluded from this PR.

**Paperclip version or commit**

Reproduced with the workflow from
`f56b097fbd25607b3b84e418085ab42c2cbb758d`. The caller and both reusable
workflow files are byte-identical between that revision and the affected
PR head. This fix is based on current master
`4a349ea44754f0b1fb8e5e04dc6cdcb05d1c3396`.

**Deployment mode**

GitHub Actions PR CI.

## What Changed

- Add `ubuntu-latest` as the fallback only for `verify` and `e2e` when
the routing runner is empty.
- Extend the existing workflow contract test to require a usable
fallback and execute each real summary script with a failed, cancelled
or skipped gate and absent routing outputs.
- Wait up to 30 seconds for the recovery child to finish imports and
construct storage before starting the existing 15-second checkpoint
poll. Keep the 30-second work abort, 60-second test timeout, and every
kill, key, state and proof assertion.
- Preserve normal runner selection, task selection, permissions and all
result assertions. No production code, public command or product
contract changes.

## Verification

- The new local workflow contract failed before the two
runner-expression changes. The existing summary scripts rejected failed
routing while the test identified the missing fallback.
- **148 focused tests passed** across CI planning, routing and E2E
sharding in 16.00 seconds. These execute the actual summary Node scripts
and preserve failure and cancellation checks.
- `actionlint -oneline -shellcheck= -pyflakes=` passed for the trusted
workflow, routing workflow and caller. `git diff --check` passed.
- The canonical worktree preflight completed with the pinned toolchain
and shared build cache.
- This test does not implement a GitHub expression evaluator. The real
GitHub startup error establishes the absent-output failure; the source
contract and actionlint validate the proposed expression. Hosted
execution of this fallback remains to be verified after the trusted
workflow reaches master.
- The unchanged two process cases passed locally before the fixture
change: 2 passed in 31.06 seconds. A controlled 16-second startup delay
then reproduced the zero-checkpoint failure in 21.81 seconds. The ready
handshake passed with the same delay in 37.11 seconds. The temporary
delay and phase logging were removed. These are measured test durations,
not a measurement of the original CI child startup.
- The final full nested-verification fixture passed: **38/38 cases in
132.30 seconds**, with disposable local PostgreSQL. This includes
partial checkpoint reuse, changed-key refusal and all existing recovery
controls.
- Fresh final head `da35ea034bdee603c021c4eba5b0f78ee6ef52d9`: the [full
PR
workflow](https://github.com/paperclipai/paperclip/actions/runs/42678538191),
[final ci /
verify](https://github.com/paperclipai/paperclip/actions/runs/42678538191/job/129374227690)
and [incremental
qualification](https://github.com/paperclipai/paperclip/actions/runs/42678537930)
passed. The [previously failing server
lane](https://github.com/paperclipai/paperclip/actions/runs/42678538191/job/129366881302)
passed. The final rollup has 61 successful checks, 6 expected skips, 2
neutral checks and no pending checks or failures. Snyk passed.
-
[Greptile](https://github.com/paperclipai/paperclip/pull/20339#issuecomment-6748154727)
and
[Ponytail](https://github.com/paperclipai/paperclip/pull/20339#issuecomment-6748242241)
both reviewed that exact final head at 5/5. There are no inline review
threads. Ponytail permits retaining the source check with actionlint,
which is the validation used here.
- The [first workflow
run](https://github.com/paperclipai/paperclip/actions/runs/42674612619)
failed the two cases described above; it is not counted as a pass. No CI
retry was used to obtain the final result. CI uses the merged trusted
workflow; its policy tests validate this proposed source file.

## Risks

- A routing failure now produces a normal failed summary instead of an
invalid runner expression. The unchanged assertions require successful
routing, policy, valid scope and selected suite results.
- Only lightweight summary jobs can fall back to GitHub-hosted capacity.
Heavy jobs and authorization rules retain their existing route.
- The new readiness allowance is bounded. A child that does not become
ready, fails its work deadline, or cannot produce the expected
checkpoint still fails. The original CI failure cause remains unproved
beyond the observed zero-checkpoint assertions.
- The fallback assertion is an exact source contract. A formatting
change can require an assertion update. The separate actionlint check
validates expression syntax; the text check does not emulate GitHub.
- This does not repair a GitHub internal error or runner interruption.
Those failures remain nonpasses.

## Model Used

OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository
tools and code execution. Context-window size is not exposed by the
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-09 13:14:16 -06:00
1 parent bce7503a82
commit 3a2f71db31
3 files changed
+22 -7

No files matched your search

+10 -2
View File
@@ -194,8 +194,8 @@ test("required aggregates reject omitted, failed and cancelled selected work", (
const resultPlan = plan(["ui/src/components/Sidebar.tsx"]);
const results = Object.fromEntries(names.map((name) => [name, { result:
name === "gate" || name === "policy" || resultPlan.jobs[name]?.run ? "success" : "skipped" }]));
const run = () => spawnSync(process.execPath, ["--input-type=module", "-e", code], { encoding: "utf8",
env: { ...process.env, FULL_CI: "true", CI_PLAN: JSON.stringify(resultPlan), CI_RESULTS: JSON.stringify(results) } });
const run = (env = {}) => spawnSync(process.execPath, ["--input-type=module", "-e", code], { encoding: "utf8",
env: { ...process.env, FULL_CI: "true", CI_PLAN: JSON.stringify(resultPlan), CI_RESULTS: JSON.stringify(results), ...env } });
assert.equal(run().status, 0);
for (const name of names) {
// Optional analysis can fail or be skipped; the unchanged task plan
@@ -208,6 +208,14 @@ test("required aggregates reject omitted, failed and cancelled selected work", (
}
results[name].result = previous;
}
for (const result of ["failure", "cancelled", "skipped"]) {
results.gate.result = result;
const failedGate = run({ FULL_CI: "", CI_PLAN: "" });
assert.notEqual(failedGate.status, 0, `${id}: an absent routing output must not pass`);
assert.match(failedGate.stderr, /CI routing must succeed/);
}
assert.match(block, /^ {4}runs-on: \$\{\{ needs\.gate\.outputs\.runner \|\| 'ubuntu-latest' \}\}$/m,
`${id}: the always-run summary must have a runner when routing produces no output`);
}
});
+2 -2
View File
@@ -798,7 +798,7 @@ jobs:
name: verify
if: ${{ always() }}
needs: [gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, sandbox_provider_tests, verify_serialized_server, canary_dry_run, verify_runner_e2e_support]
runs-on: ${{ needs.gate.outputs.runner }}
runs-on: ${{ needs.gate.outputs.runner || 'ubuntu-latest' }}
timeout-minutes: 5
steps:
- name: Validate every planned suite result
@@ -1781,7 +1781,7 @@ jobs:
name: e2e
if: ${{ always() }}
needs: [gate, policy, e2e_shards]
runs-on: ${{ needs.gate.outputs.runner }}
runs-on: ${{ needs.gate.outputs.runner || 'ubuntu-latest' }}
timeout-minutes: 5
steps:
- name: Validate every planned suite result
@@ -223,6 +223,7 @@ describe.skipIf(!support.supported)("bounded nested workspace verification", ()
if(input.pauseParentPut)await new Promise(()=>{});
return local.putFile(value);
}};
process.send?.({ready:true});
try{const result=await migrateRemoteWorkspace({db,storage,companyId:input.companyId,workspaceId:input.workspaceId,
apply:true,signal:AbortSignal.timeout(30000)});process.send?.({result});}
catch(error){process.send?.({failed:true,message:error.message});process.exitCode=1;}
@@ -230,28 +231,34 @@ describe.skipIf(!support.supported)("bounded nested workspace verification", ()
`;
const launch = (pauseKey: string | null, pauseParentPut = false, masterKey = process.env.PAPERCLIP_SECRETS_MASTER_KEY) => {
const reads: string[] = [], results: unknown[] = [], failures: string[] = [];
let ready = false;
const child = spawn(process.execPath, ["--import", path.join(repoRoot, "cli/node_modules/tsx/dist/loader.mjs"),
"--input-type=module", "--eval", body], { cwd: path.join(repoRoot, "server"), detached: true,
env: { ...process.env, PAPERCLIP_SECRETS_MASTER_KEY: masterKey, TMPDIR: temporary, DATABASE_URL: database.connectionString, IMPORT_FIXTURE: JSON.stringify({
companyId: f.companyId, workspaceId: f.parent.id, objects: path.join(home, "objects"), endpoint, keys, pauseKey, pauseParentPut }) },
stdio: ["ignore", "pipe", "pipe", "ipc"] });
let stderr = ""; child.stderr!.on("data", bytes => { stderr += bytes; });
child.stdout!.resume(); child.on("message", (message: { read?: string; result?: unknown; failed?: boolean; message?: string }) => {
child.stdout!.resume(); child.on("message", (message: { read?: string; result?: unknown; failed?: boolean; message?: string; ready?: boolean }) => {
if (message.ready) ready = true;
if (message.read) reads.push(message.read); if (message.result) results.push(message.result);
if (message.failed && message.message) failures.push(message.message);
});
const closed = new Promise<{ code: number | null; signal: string | null }>((resolve, reject) => {
child.once("error", reject); child.once("close", (code, signal) => resolve({ code, signal }));
});
return { child, reads, results, failures, closed, error: () => stderr };
return { child, reads, results, failures, closed, ready: () => ready, error: () => stderr };
};
const first = mode === "partial" ? launch(keys[1]) : launch(null, true);
try {
const prematureExit = first.closed.then(result => { throw new Error(`import_ended_before_checkpoint:${result.code}:${first.error()}`); });
await Promise.race([vi.waitFor(() => {
expect(first.ready(), `import_startup_not_ready:${first.error()}`).toBe(true);
}, { timeout: 30_000, interval: 25 }), prematureExit]);
await Promise.race([vi.waitFor(async () => {
const [snapshot] = await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, f.snapshots[0].id));
const completed = (await db.select().from(executionWorkspaceSnapshots)).filter(row => row.metadata?.nestedImportVerification);
expect(completed).toHaveLength(mode === "partial" ? 1 : keys.length); expect(snapshot).toBeDefined();
}, { timeout: 15_000, interval: 25 }), first.closed.then(result => { throw new Error(`import_ended_before_checkpoint:${result.code}:${first.error()}`); })]);
}, { timeout: 15_000, interval: 25 }), prematureExit]);
process.kill(-first.child.pid!, "SIGKILL");
expect(await first.closed).toMatchObject({ signal: "SIGKILL" });
} finally {