fix(runner): repair paid provider startup paths (#12769)

## Thinking Path

> - Paperclip manages AI agents that perform work.
> - Paperclip Runner connects durable task runs to local provider
processes.
> - The full-stack paid matrix exposed failures after the runner
integrity repair.
> - Verified JavaScript entrypoints lost their relative module graph
when Linux executed them through descriptor paths.
> - Returned provider startup errors also remained pending and became
indeterminate after recovery.
> - Sparse Codex tool lifecycle events lost the `write_document`
identity before task transcript projection.
> - This pull request repairs those three boundaries and makes the
structured-question fixture deterministic.
> - The benefit is repeatable provider startup, exact failure replay,
and correct inline Plan placement.

## Linked Issues or Issue Description

Refs #12721 and #12700.

**What happened?**

The paid runner matrix failed ACPX and OpenCode startup before provider
session creation. The runner journal then replaced the original startup
error with an indeterminate recovery result. Native Codex saved a Plan
but rendered it only as a fallback card. A legacy Claude waiting reply
could also echo the reserved terminal marker before the answer arrived.

**Expected behavior**

Verified JavaScript providers must start from immutable
descriptor-backed artifacts. Returned startup failures must persist as
terminal failed command results. Native tool lifecycle updates must
preserve the `write_document` boundary. Pre-answer fixture output must
not contain the reserved terminal marker.

**Steps to reproduce**

1. Run the local provider cells in the Runner Full-Stack E2E workflow.
2. Observe ACPX and OpenCode fail during `session.open` before provider
execution.
3. Observe recovery report `execution_indeterminate` instead of the
original startup error.
4. Run the native Codex Plan cell and observe the fallback Plan card
after the tool activity row.
5. Run the legacy Claude structured-question resume cell and observe an
early marker echo in waiting prose.

**Paperclip version or commit**

`0f9452101740835ce0b1488a204bf48acd5bafc3`

**Deployment mode**

Local development with the paid GitHub Actions acceptance workflow.

## What Changed

- Bundle the ACPX sidecar and OpenCode proxy as self-contained Node ESM
entrypoints before hashing and verified descriptor launch.
- Anchor ACPX dynamic provider package resolution at a
controller-derived provider-pack root and keep that root out of the
provider child environment.
- Persist executor-returned startup errors as redacted durable failed
command results while retaining indeterminate recovery for true process
death.
- Coalesce sparse native tool items by stable ID so a late
`write_document` name, input, and result reach the transcript boundary
once.
- Forbid the structured-question fixture from spelling or announcing its
reserved terminal marker before the user answers.

## Verification

- Rust and TypeScript regression tests cover durable failed replay, true
crash ambiguity, bundle closure, package-root derivation, environment
filtering, exact Codex tool lifecycle coalescing, and prompt
determinism.
- Local execution is intentionally limited to formatters and static diff
checks. GitHub Actions will run tests, type checks, builds, and security
checks.
- After ordinary CI is green, scoped paid cells will validate one ACPX
launch, one OpenCode launch, native Codex Plan projection, and legacy
Claude structured resume before a complete matrix rerun.
- Prior failing matrix:
https://github.com/paperclipai/paperclip/actions/runs/33682434315

## Risks

- Bundling changes the bytes covered by provider launch hashes.
Provider-pack generation already hashes the final built files.
- ACPX still loads qualified provider packages dynamically. The
controller supplies a normalized package root, while existing version,
digest, path, and descriptor checks remain active.
- Durable `failed` is terminal. Replays return the same redacted result
and do not execute the provider effect twice.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex based on GPT-5 with agentic reasoning, repository
inspection, code editing, Git, parallel subagents, and GitHub Actions
coordination. The exact deployed snapshot and context-window size are
not exposed to this task.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked related public work or described the bug in
this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket id
- [ ] I have run tests locally and they pass (intentionally deferred to
GitHub Actions)
- [x] I have added or updated tests where applicable
- [x] No documentation change is required for this runtime repair
- [x] I have considered and documented the risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

.github/workflows/runner-full-stack-e2e.yml

@@ -42,9 +42,10 @@ permissions:
   contents: read
 
 concurrency:
-  group: runner-full-stack-e2e-${{ inputs.target_branch || github.event.repository.default_branch }}
+  group: runner-full-stack-e2e-${{ github.event_name == 'workflow_dispatch' && inputs.target_branch != '' && inputs.target_branch != github.event.repository.default_branch && format('development-{0}', inputs.target_branch) || format('protected-{0}', github.run_id) }}
   # Development branch campaigns supersede older runs for the same target.
-  # Preserve every default-branch campaign for its paid audit trail.
+  # Give protected/default-branch campaigns unique groups because GitHub also
+  # replaces pending runs when cancel-in-progress is false.
   cancel-in-progress: ${{ github.event_name == 'workflow_dispatch' && inputs.target_branch != '' && inputs.target_branch != github.event.repository.default_branch }}
 
 jobs:
@@ -59,6 +60,7 @@ jobs:
       test_runner: ${{ steps.runner.outputs.runner }}
       max_parallel_default: ${{ steps.runner.outputs.max_parallel_default }}
       max_parallel_limit: ${{ steps.runner.outputs.max_parallel_limit }}
+      playwright_channel: ${{ steps.runner.outputs.playwright_channel }}
       target_sha: ${{ steps.target.outputs.sha }}
       target_ref: ${{ steps.target.outputs.ref }}
     steps:
@@ -131,6 +133,7 @@ jobs:
               echo "runner=$aws_runner"
               echo "max_parallel_default=100"
               echo "max_parallel_limit=100"
+              echo "playwright_channel=chrome"
             } >> "$GITHUB_OUTPUT"
             echo '::notice title=Paid runner routing::Using an ephemeral RunsOn Fleet runner'
           else
@@ -138,8 +141,9 @@ jobs:
               echo "runner=$github_runner"
               echo "max_parallel_default=32"
               echo "max_parallel_limit=57"
+              echo "playwright_channel="
             } >> "$GITHUB_OUTPUT"
-            echo '::notice title=Paid runner routing::RUNNER_E2E_AWS_ENABLED is not true; using the existing paid runner'
+            echo '::notice title=Paid runner routing::RUNNER_E2E_AWS_ENABLED is not true; using the proven GitHub-hosted runner'
           fi
 
   target_lock:
@@ -350,18 +354,21 @@ jobs:
       source_revision: ${{ steps.image.outputs.source_revision }}
       content_id: ${{ steps.image.outputs.content_id }}
     steps:
-      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+      - if: needs.catalog.outputs.needs_daytona == 'true'
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with:
           ref: ${{ needs.authorize.outputs.target_sha }}
           persist-credentials: false
 
       - name: Download resolved target lockfile
+        if: needs.catalog.outputs.needs_daytona == 'true'
         uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
         with:
           artifact-ids: ${{ needs.target_lock.outputs.artifact_id }}
           path: ${{ runner.temp }}/runner-e2e-target-lock
 
       - name: Restore resolved target lockfile
+        if: needs.catalog.outputs.needs_daytona == 'true'
         env:
           TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
           EXPECTED_LOCK_SHA256: ${{ needs.target_lock.outputs.lock_sha256 }}
@@ -406,9 +413,11 @@ jobs:
         run: |
           set -euo pipefail
           if [ "$NEEDS_DAYTONA" != true ]; then
-            echo "image=" >> "$GITHUB_OUTPUT"
-            echo "source_revision=" >> "$GITHUB_OUTPUT"
-            echo "content_id=" >> "$GITHUB_OUTPUT"
+            {
+              echo "image="
+              echo "source_revision="
+              echo "content_id="
+            } >> "$GITHUB_OUTPUT"
             exit 0
           fi
           [[ "$IMAGE_CONTENT_ID" =~ ^[0-9a-f]{64}$ ]]
@@ -452,9 +461,11 @@ jobs:
              .config.User == "daytona" and
              (.config.Env | any(startswith("PAPERCLIP_RUNNER_PROVIDER_PACK_ROOT=")))' \
             <<< "$image_config" >/dev/null
-          echo "image=$immutable" >> "$GITHUB_OUTPUT"
-          echo "source_revision=$source_revision" >> "$GITHUB_OUTPUT"
-          echo "content_id=$published_content_id" >> "$GITHUB_OUTPUT"
+          {
+            echo "image=$immutable"
+            echo "source_revision=$source_revision"
+            echo "content_id=$published_content_id"
+          } >> "$GITHUB_OUTPUT"
 
   build_runner_artifacts:
     name: Build reusable runner campaign artifacts
@@ -833,6 +844,7 @@ jobs:
         run: node packages/paperclip-runner/scripts/materialize-opencode-binary.mjs
 
       - name: Download immutable campaign outputs
+        if: startsWith(matrix.profileId, 'runner-') || matrix.suiteId == 'openrouter-model-breadth'
         uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
         with:
           name: ${{ needs.build_runner_artifacts.outputs.build_artifact_name }}
@@ -846,6 +858,7 @@ jobs:
           path: runner-e2e-provider-pack
 
       - name: Verify and restore campaign outputs
+        if: startsWith(matrix.profileId, 'runner-') || matrix.suiteId == 'openrouter-model-breadth'
         env:
           NEEDS_RUNNER_TYPESCRIPT: ${{ matrix.profileId == 'runner-opencode' || startsWith(matrix.profileId, 'runner-acpx-') || matrix.suiteId == 'openrouter-model-breadth' }}
           NEEDS_NATIVE_BINARY: ${{ startsWith(matrix.profileId, 'runner-') || matrix.suiteId == 'openrouter-model-breadth' }}
@@ -902,8 +915,28 @@ jobs:
         if: matrix.profileId == 'legacy-claude'
         run: npm install --global --omit=dev @anthropic-ai/claude-code@2.1.19
 
-      - name: Install Chromium
-        run: pnpm exec playwright install --with-deps chromium
+      - name: Qualify preinstalled Chrome
+        if: needs.authorize.outputs.playwright_channel == 'chrome'
+        run: |
+          set -euo pipefail
+          chrome_path="$(command -v google-chrome)"
+          test -x "$chrome_path"
+          google-chrome --version
+
+      - name: Install Chromium headless shell on GitHub-hosted fallback
+        if: needs.authorize.outputs.playwright_channel != 'chrome'
+        run: |
+          set -euo pipefail
+          for attempt in 1 2 3; do
+            if pnpm exec playwright install --with-deps --only-shell chromium; then
+              exit 0
+            fi
+            if [ "$attempt" -eq 3 ]; then
+              echo "Chromium headless shell installation failed after $attempt attempts." >&2
+              exit 1
+            fi
+            sleep "$((attempt * 10))"
+          done
 
       - name: Run paid cell
         env:
@@ -916,6 +949,7 @@ jobs:
           PAPERCLIP_E2E_CAMPAIGN_ID: gha-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.executionId }}
           PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
           PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ needs.authorize.outputs.target_ref }}
+          PAPERCLIP_PLAYWRIGHT_CHANNEL: ${{ needs.authorize.outputs.playwright_channel }}
         run: pnpm test:e2e:runner -- --id "${{ matrix.executionId }}"
 
       - name: Upload access-controlled packaged cell evidence
@@ -929,13 +963,14 @@ jobs:
 
   report:
     name: Merge and enforce campaign result
-    if: always() && needs.catalog.result == 'success'
+    if: always() && !cancelled() && needs.catalog.result == 'success'
     needs: [authorize, catalog, daytona_image, test]
     outputs:
       history_source_ready: ${{ steps.history_source_ready.outputs.ready }}
     runs-on: ubuntu-latest
     timeout-minutes: 15
     permissions:
+      actions: read
       contents: read
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
@@ -963,22 +998,52 @@ jobs:
 
       - run: pnpm install --frozen-lockfile
 
+      - name: Resolve workflow job attempts
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPOSITORY: ${{ github.repository }}
+          RUN_ID: ${{ github.run_id }}
+        run: |
+          set -euo pipefail
+          gh api --paginate --slurp \
+            "repos/$REPOSITORY/actions/runs/$RUN_ID/jobs?filter=all&per_page=100" \
+            > runner-e2e-job-pages.json
+          for attempt in $(seq 1 "${{ github.run_attempt }}"); do
+            gh api "repos/$REPOSITORY/actions/runs/$RUN_ID/attempts/$attempt" \
+              --jq '{run_attempt, run_started_at}'
+          done > runner-e2e-attempts.jsonl
+          jq -s '.' runner-e2e-attempts.jsonl > runner-e2e-attempts.json
+          jq --slurpfile attempts runner-e2e-attempts.json \
+            '{jobs: [.[].jobs[]], attempts: $attempts[0]}' \
+            runner-e2e-job-pages.json > runner-e2e-jobs.json
+
       - name: Download cell evidence
         id: download_evidence
         continue-on-error: true
         uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
         with:
-          pattern: runner-e2e-${{ github.run_id }}-${{ github.run_attempt }}-*
+          pattern: runner-e2e-${{ github.run_id }}-*-*
           path: downloaded-runner-e2e
-          merge-multiple: true
+          merge-multiple: false
 
       - name: Retry cell evidence download after transport failure
         if: steps.download_evidence.outcome == 'failure'
         uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
         with:
-          pattern: runner-e2e-${{ github.run_id }}-${{ github.run_attempt }}-*
+          pattern: runner-e2e-${{ github.run_id }}-*-*
           path: downloaded-runner-e2e
-          merge-multiple: true
+          merge-multiple: false
+
+      - name: Select latest workflow attempt per cell
+        if: always()
+        env:
+          PAPERCLIP_RUNNER_E2E_ARTIFACT_ROOT: ${{ github.workspace }}/downloaded-runner-e2e
+          PAPERCLIP_RUNNER_E2E_SELECTED_ROOT: ${{ github.workspace }}/selected-runner-e2e
+          PAPERCLIP_RUNNER_E2E_JOBS_JSON: ${{ github.workspace }}/runner-e2e-jobs.json
+          PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: ${{ needs.catalog.outputs.execution_ids }}
+          PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
+          PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ needs.authorize.outputs.target_ref }}
+        run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/select-rerun-artifacts.ts
 
       - name: Collect blob reports
         run: |
@@ -990,7 +1055,7 @@ jobs:
             if [ ! -e "$target" ]; then
               cp "$report" "$target"
             fi
-          done < <(find downloaded-runner-e2e -path '*/blob-report/*.zip' -print0)
+          done < <(find selected-runner-e2e -path '*/blob-report/*.zip' -print0)
 
       - name: Merge Playwright HTML and JUnit
         if: always()
@@ -1001,7 +1066,7 @@ jobs:
       - name: Aggregate normalized campaign results
         if: always()
         env:
-          PAPERCLIP_RUNNER_E2E_REPORT_ROOT: ${{ github.workspace }}/downloaded-runner-e2e
+          PAPERCLIP_RUNNER_E2E_REPORT_ROOT: ${{ github.workspace }}/selected-runner-e2e
           PAPERCLIP_RUNNER_E2E_REPORT_OUT: ${{ github.workspace }}/runner-e2e-merged-report/normalized
           PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: ${{ needs.catalog.outputs.execution_ids }}
           PAPERCLIP_E2E_CAMPAIGN_ID: gha-${{ github.run_id }}-${{ github.run_attempt }}
@@ -1043,6 +1108,8 @@ jobs:
     if: always() && needs.catalog.result == 'success' && needs.report.outputs.history_source_ready == 'true'
     runs-on: ubuntu-latest
     timeout-minutes: 15
+    outputs:
+      pages_artifact_name: ${{ steps.pages_artifact_name.outputs.name }}
     concurrency:
       group: runner-e2e-history-publish
       cancel-in-progress: false
@@ -1092,10 +1159,16 @@ jobs:
           RUNNER_E2E_HISTORY_PUBLIC_BASE_URL: ${{ vars.RUNNER_E2E_HISTORY_PUBLIC_BASE_URL }}
         run: pnpm test:e2e:runner:history:publish
 
+      - name: Resolve Pages artifact name
+        id: pages_artifact_name
+        if: vars.RUNNER_FULL_STACK_E2E_PUBLISH_PAGES == 'true'
+        run: echo "name=github-pages-${{ github.run_id }}-${{ github.run_attempt }}" >> "$GITHUB_OUTPUT"
+
       - name: Package pruned structured dashboard for GitHub Pages
         if: vars.RUNNER_FULL_STACK_E2E_PUBLISH_PAGES == 'true'
         uses: actions/upload-pages-artifact@7b1f4a764d45c48632c6b24a0339c27f5614fb0b # v4
         with:
+          name: ${{ steps.pages_artifact_name.outputs.name }}
           path: runner-e2e-merged-report/normalized
 
   pages:
@@ -1113,3 +1186,7 @@ jobs:
       - name: Deploy to GitHub Pages
         id: deployment
         uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4
+        with:
+          # If only this failed job is rerun, GitHub retains the successful
+          # publisher job's output from the earlier workflow attempt.
+          artifact_name: ${{ needs.publish_history.outputs.pages_artifact_name }}