From 3a2f71db31aa99633610dc241f55f34ac65124de Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 9 Nov 2026 13:14:16 -0600 Subject: [PATCH] fix(ci): handle missing runner output and await import fixture readiness (#20339) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - CI checks must report real result failures after their setup phase. > - Required summaries must run even when the routing gate has no runner output. > - Process-recovery tests must start their checkpoint deadline after the child loads its modules. > - An empty runner prevents summaries from starting. Slow child startup can consume the checkpoint deadline. > - This pull request adds a fallback for the two summaries and a ready signal for the recovery fixture. > - Existing routing, checkpoint, key and recovery assertions remain in place. ## Linked Issues or Issue Description Refs #20322. Related open PRs #12512 and #12513 propose a broader runner contract and caller activation. This change does not adopt that contract or change normal routing. No current repair for an empty aggregate runner was found. **What happened?** The PR workflow can fail before starting any jobs when the routing gate has no runner output. GitHub reports an empty `runs-on` value for the `verify` and `e2e` summary jobs. Their `always()` conditions make this path possible when a dependency cannot run. A real workflow rerun showed both errors at lines 801 and 1784 of the trusted workflow. The first run of this PR also failed two process-recovery cases with no saved child checkpoints. The fixture starts its 15-second checkpoint poll before a new Node process imports the server modules. A controlled 16-second pre-import delay reproduces the same zero-checkpoint assertion. That evidence proves a deadline-boundary defect. The original CI run has no phase events, so it does not prove which stage caused those two CI failures. **Expected behavior** The summaries can start on `ubuntu-latest` when routing has no runner output. They must still fail when routing or required work did not succeed. The process fixture waits for a bounded child-ready signal before its unchanged checkpoint deadline. It must still prove checkpoint reuse, process termination and refusal under a changed master key. **Steps to reproduce** 1. Start or rerun PR CI with no runner output from the routing gate. 2. Let the two `always()` summary jobs evaluate their runner expressions. 3. Observe that the previous expressions produce an empty value and GitHub refuses the jobs. 4. For the fixture defect, temporarily delay the child before imports for 16 seconds. The previous fixture fails its 15-second checkpoint poll before the child is ready. The updated fixture passes with the same delay. The temporary delay is excluded from this PR. **Paperclip version or commit** Reproduced with the workflow from `f56b097fbd25607b3b84e418085ab42c2cbb758d`. The caller and both reusable workflow files are byte-identical between that revision and the affected PR head. This fix is based on current master `4a349ea44754f0b1fb8e5e04dc6cdcb05d1c3396`. **Deployment mode** GitHub Actions PR CI. ## What Changed - Add `ubuntu-latest` as the fallback only for `verify` and `e2e` when the routing runner is empty. - Extend the existing workflow contract test to require a usable fallback and execute each real summary script with a failed, cancelled or skipped gate and absent routing outputs. - Wait up to 30 seconds for the recovery child to finish imports and construct storage before starting the existing 15-second checkpoint poll. Keep the 30-second work abort, 60-second test timeout, and every kill, key, state and proof assertion. - Preserve normal runner selection, task selection, permissions and all result assertions. No production code, public command or product contract changes. ## Verification - The new local workflow contract failed before the two runner-expression changes. The existing summary scripts rejected failed routing while the test identified the missing fallback. - **148 focused tests passed** across CI planning, routing and E2E sharding in 16.00 seconds. These execute the actual summary Node scripts and preserve failure and cancellation checks. - `actionlint -oneline -shellcheck= -pyflakes=` passed for the trusted workflow, routing workflow and caller. `git diff --check` passed. - The canonical worktree preflight completed with the pinned toolchain and shared build cache. - This test does not implement a GitHub expression evaluator. The real GitHub startup error establishes the absent-output failure; the source contract and actionlint validate the proposed expression. Hosted execution of this fallback remains to be verified after the trusted workflow reaches master. - The unchanged two process cases passed locally before the fixture change: 2 passed in 31.06 seconds. A controlled 16-second startup delay then reproduced the zero-checkpoint failure in 21.81 seconds. The ready handshake passed with the same delay in 37.11 seconds. The temporary delay and phase logging were removed. These are measured test durations, not a measurement of the original CI child startup. - The final full nested-verification fixture passed: **38/38 cases in 132.30 seconds**, with disposable local PostgreSQL. This includes partial checkpoint reuse, changed-key refusal and all existing recovery controls. - Fresh final head `da35ea034bdee603c021c4eba5b0f78ee6ef52d9`: the [full PR workflow](https://github.com/paperclipai/paperclip/actions/runs/42678538191), [final ci / verify](https://github.com/paperclipai/paperclip/actions/runs/42678538191/job/129374227690) and [incremental qualification](https://github.com/paperclipai/paperclip/actions/runs/42678537930) passed. The [previously failing server lane](https://github.com/paperclipai/paperclip/actions/runs/42678538191/job/129366881302) passed. The final rollup has 61 successful checks, 6 expected skips, 2 neutral checks and no pending checks or failures. Snyk passed. - [Greptile](https://github.com/paperclipai/paperclip/pull/20339#issuecomment-6748154727) and [Ponytail](https://github.com/paperclipai/paperclip/pull/20339#issuecomment-6748242241) both reviewed that exact final head at 5/5. There are no inline review threads. Ponytail permits retaining the source check with actionlint, which is the validation used here. - The [first workflow run](https://github.com/paperclipai/paperclip/actions/runs/42674612619) failed the two cases described above; it is not counted as a pass. No CI retry was used to obtain the final result. CI uses the merged trusted workflow; its policy tests validate this proposed source file. ## Risks - A routing failure now produces a normal failed summary instead of an invalid runner expression. The unchanged assertions require successful routing, policy, valid scope and selected suite results. - Only lightweight summary jobs can fall back to GitHub-hosted capacity. Heavy jobs and authorization rules retain their existing route. - The new readiness allowance is bounded. A child that does not become ready, fails its work deadline, or cannot produce the expected checkpoint still fails. The original CI failure cause remains unproved beyond the observed zero-checkpoint assertions. - The fallback assertion is an exact source contract. A formatting change can require an assertion update. The separate actionlint check validates expression syntax; the text check does not emulate GitHub. - This does not repair a GitHub internal error or runner interruption. Those failures remain nonpasses. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. Context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .github/scripts/tests/ci-plan.test.mjs | 12 ++++++++++-- .github/workflows/pr-trusted.yml | 4 ++-- .../remote-workspace-nested-verification.test.ts | 13 ++++++++++--- 3 files changed, 22 insertions(+), 7 deletions(-) diff --git a/.github/scripts/tests/ci-plan.test.mjs b/.github/scripts/tests/ci-plan.test.mjs index c2d561467d8..3285cc3b072 100644 --- a/.github/scripts/tests/ci-plan.test.mjs +++ b/.github/scripts/tests/ci-plan.test.mjs @@ -194,8 +194,8 @@ test("required aggregates reject omitted, failed and cancelled selected work", ( const resultPlan = plan(["ui/src/components/Sidebar.tsx"]); const results = Object.fromEntries(names.map((name) => [name, { result: name === "gate" || name === "policy" || resultPlan.jobs[name]?.run ? "success" : "skipped" }])); - const run = () => spawnSync(process.execPath, ["--input-type=module", "-e", code], { encoding: "utf8", - env: { ...process.env, FULL_CI: "true", CI_PLAN: JSON.stringify(resultPlan), CI_RESULTS: JSON.stringify(results) } }); + const run = (env = {}) => spawnSync(process.execPath, ["--input-type=module", "-e", code], { encoding: "utf8", + env: { ...process.env, FULL_CI: "true", CI_PLAN: JSON.stringify(resultPlan), CI_RESULTS: JSON.stringify(results), ...env } }); assert.equal(run().status, 0); for (const name of names) { // Optional analysis can fail or be skipped; the unchanged task plan @@ -208,6 +208,14 @@ test("required aggregates reject omitted, failed and cancelled selected work", ( } results[name].result = previous; } + for (const result of ["failure", "cancelled", "skipped"]) { + results.gate.result = result; + const failedGate = run({ FULL_CI: "", CI_PLAN: "" }); + assert.notEqual(failedGate.status, 0, `${id}: an absent routing output must not pass`); + assert.match(failedGate.stderr, /CI routing must succeed/); + } + assert.match(block, /^ {4}runs-on: \$\{\{ needs\.gate\.outputs\.runner \|\| 'ubuntu-latest' \}\}$/m, + `${id}: the always-run summary must have a runner when routing produces no output`); } }); diff --git a/.github/workflows/pr-trusted.yml b/.github/workflows/pr-trusted.yml index 49b12390240..3a435b77553 100644 --- a/.github/workflows/pr-trusted.yml +++ b/.github/workflows/pr-trusted.yml @@ -798,7 +798,7 @@ jobs: name: verify if: ${{ always() }} needs: [gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, sandbox_provider_tests, verify_serialized_server, canary_dry_run, verify_runner_e2e_support] - runs-on: ${{ needs.gate.outputs.runner }} + runs-on: ${{ needs.gate.outputs.runner || 'ubuntu-latest' }} timeout-minutes: 5 steps: - name: Validate every planned suite result @@ -1781,7 +1781,7 @@ jobs: name: e2e if: ${{ always() }} needs: [gate, policy, e2e_shards] - runs-on: ${{ needs.gate.outputs.runner }} + runs-on: ${{ needs.gate.outputs.runner || 'ubuntu-latest' }} timeout-minutes: 5 steps: - name: Validate every planned suite result diff --git a/server/src/services/remote-workspace-nested-verification.test.ts b/server/src/services/remote-workspace-nested-verification.test.ts index 7e3494a7730..809b42f3f3e 100644 --- a/server/src/services/remote-workspace-nested-verification.test.ts +++ b/server/src/services/remote-workspace-nested-verification.test.ts @@ -223,6 +223,7 @@ describe.skipIf(!support.supported)("bounded nested workspace verification", () if(input.pauseParentPut)await new Promise(()=>{}); return local.putFile(value); }}; + process.send?.({ready:true}); try{const result=await migrateRemoteWorkspace({db,storage,companyId:input.companyId,workspaceId:input.workspaceId, apply:true,signal:AbortSignal.timeout(30000)});process.send?.({result});} catch(error){process.send?.({failed:true,message:error.message});process.exitCode=1;} @@ -230,28 +231,34 @@ describe.skipIf(!support.supported)("bounded nested workspace verification", () `; const launch = (pauseKey: string | null, pauseParentPut = false, masterKey = process.env.PAPERCLIP_SECRETS_MASTER_KEY) => { const reads: string[] = [], results: unknown[] = [], failures: string[] = []; + let ready = false; const child = spawn(process.execPath, ["--import", path.join(repoRoot, "cli/node_modules/tsx/dist/loader.mjs"), "--input-type=module", "--eval", body], { cwd: path.join(repoRoot, "server"), detached: true, env: { ...process.env, PAPERCLIP_SECRETS_MASTER_KEY: masterKey, TMPDIR: temporary, DATABASE_URL: database.connectionString, IMPORT_FIXTURE: JSON.stringify({ companyId: f.companyId, workspaceId: f.parent.id, objects: path.join(home, "objects"), endpoint, keys, pauseKey, pauseParentPut }) }, stdio: ["ignore", "pipe", "pipe", "ipc"] }); let stderr = ""; child.stderr!.on("data", bytes => { stderr += bytes; }); - child.stdout!.resume(); child.on("message", (message: { read?: string; result?: unknown; failed?: boolean; message?: string }) => { + child.stdout!.resume(); child.on("message", (message: { read?: string; result?: unknown; failed?: boolean; message?: string; ready?: boolean }) => { + if (message.ready) ready = true; if (message.read) reads.push(message.read); if (message.result) results.push(message.result); if (message.failed && message.message) failures.push(message.message); }); const closed = new Promise<{ code: number | null; signal: string | null }>((resolve, reject) => { child.once("error", reject); child.once("close", (code, signal) => resolve({ code, signal })); }); - return { child, reads, results, failures, closed, error: () => stderr }; + return { child, reads, results, failures, closed, ready: () => ready, error: () => stderr }; }; const first = mode === "partial" ? launch(keys[1]) : launch(null, true); try { + const prematureExit = first.closed.then(result => { throw new Error(`import_ended_before_checkpoint:${result.code}:${first.error()}`); }); + await Promise.race([vi.waitFor(() => { + expect(first.ready(), `import_startup_not_ready:${first.error()}`).toBe(true); + }, { timeout: 30_000, interval: 25 }), prematureExit]); await Promise.race([vi.waitFor(async () => { const [snapshot] = await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, f.snapshots[0].id)); const completed = (await db.select().from(executionWorkspaceSnapshots)).filter(row => row.metadata?.nestedImportVerification); expect(completed).toHaveLength(mode === "partial" ? 1 : keys.length); expect(snapshot).toBeDefined(); - }, { timeout: 15_000, interval: 25 }), first.closed.then(result => { throw new Error(`import_ended_before_checkpoint:${result.code}:${first.error()}`); })]); + }, { timeout: 15_000, interval: 25 }), prematureExit]); process.kill(-first.child.pid!, "SIGKILL"); expect(await first.closed).toMatchObject({ signal: "SIGKILL" }); } finally {