7 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	db8f8fe5b7	fix(evals): select Grok subscription protocol credentials explicitly (#13901) ... ## Thinking Path > - Paperclip manages agent work through shared runner contracts. > - Direct protocol evals qualify provider behavior against a mock control plane. > - Grok supports API keys and company subscription credentials. > - The hosted protocol workflow selected an API key for every Grok cell. > - Product subscription support did not enable subscription protocol runs. > - This change adds explicit subscription selection and checks the recorded authentication mode. ## Linked Issues or Issue Description Refs #13878, #13882, #12618. The direct Grok protocol roster cannot run with subscription authentication through the trusted default-branch workflow. Add an explicit selector while keeping API-key dispatches compatible. Keep the actor allowlist, protected environment, immutable source revisions, and publication gates. ## What Changed - Add `grok_authentication` with `api_key` and `subscription` choices. Keep `api_key` as the compatibility default. - Deliver the protected `GROK_AUTH_JSON` secret only to a subscription-selected Grok cell. Do not provide an API key to that cell. - Read authentication mode from the pinned eval program's actual roster summary. Retain it in the cell, catalog, campaign roster, and result. - Reject missing or mismatched authentication evidence during aggregation. Preserve cell metadata and an allowlisted failure reason before failing a cell, so malformed evidence cannot hide the retained attempt. - Document credential setup, source separation, and temporary-secret cleanup. ## Verification - `node --test packages/paperclip-runner/scripts/runner-protocol-eval-campaign.test.mjs packages/paperclip-runner/scripts/runner-protocol-eval-workflow-security.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`: 34 tests passed. - Validated all 39 Grok cells at eval revision `3213dbec7e8ca1865ea95e6db7e7d34b095eb47a`; every selected cell requests only the subscription credential. Validation made zero provider calls. - Negative coverage rejects invalid selectors and missing or API authentication evidence in an otherwise passing subscription attempt. - `git diff --check` passed. All 53 current-head checks passed; the unchanged callback-drain timing test passed its bounded rerun, and the failed attempt is retained. Greptile reviewed `ecd3dcc0e998f07cf56fcb1f087946f50f388bec` at 5/5 with no remaining findings. - No Docker or broad builds ran on the developer machine. CI performs repository checks on the configured fleet. ## Risks Grok runs require an eval revision that records `authenticationMode` in the roster summary. Missing evidence fails closed. The credential contains account access and refresh tokens; an owner must approve its delivery to the protected environment before a live run. The change adds no PR trigger or authorization bypass. Live subscription protocol qualification remains pending this workflow reaching master. ## Model Used OpenAI GPT-6 through Codex, with tool use and code execution. The exact serving model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-23 18:43:21 -05:00
3 people	3790ca2f13	fix(runner): repair approval and Stop races and eval infrastructure (#13750) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner tasks must continue after approval and stop when the user presses Stop. > - Live evals found races at approval delivery and provider startup. > - Browser readiness and CI setup errors also hid the actual task results. > - This pull request fixes those races and the related test infrastructure. > - Regression tests and saved live reports show which cases now pass. ## Linked Issues or Issue Description Companion eval definitions PR: https://github.com/paperclipai/paperclip-evals/pull/25 (AgentCore paused and provider/environment infrastructure). Related: #13741 now supplies the late-startup Stop fence and warm-attachment recovery; this PR retains that fence and extends startup tracking and regression coverage to both native backend paths. #13539 introduced queued approvals during active runs. #13738 fixes child assignment, task replies, and warm process continuity and is already in the base. #13291 concerns automatic continuation of interrupted legacy sandbox runs; this PR fixes native startup cancellation and does not change that recovery policy. **What happened?** An accepted service approval could wait after its source run stopped. Stop could return success before the provider handle existed. Work could then start after Stop, or a cancelled run could be recorded as failed. Some E2E tests also failed on unloaded browser content or irrelevant reply wording. Runner CI could fail before model work because of dependency or sandbox setup. **Expected behavior** Deliver each settled approval once after its source run stops. Do not start work after an acknowledged Stop. Preserve the audited cancellation. Test the intended product behavior with a ready browser and verified runtime dependencies. **Steps to reproduce** 1. Approve a service request while its source run is active. Let the run finish. Check that its result starts one continuation. 2. Delay provider startup. Press Stop before its handle is available. Check cancellation, then submit `/new`. 3. Run the browser, warm-workspace, and Stop-and-redirect cases from the linked report. **Paperclip version or commit** The branch includes master at `9d19f98b5`. The report records the original source for each focused attempt. **Deployment mode** Isolated local development instances and disposable Daytona sandboxes. ## What Changed - Deliver settled tool-action results for the exact company and source run during final cleanup. Keep the existing idempotent receipt and periodic recovery sweep. - Wait for startup to hand off its provider handle before acknowledging Stop. Reject first-turn admission after cancellation. Preserve a matching audited pending or acknowledged cancellation. - Wait for mounted task history and connector controls in browser tests. Record failure evidence. Grade workspace contents and process continuity separately from exact reply wording. Require each warm-turn marker once and in order, allowing surrounding prose. - Stop-and-redirect now checks that the source file exists and work is active before Stop. - Resolve target dependency locks in an uncredentialed CI job. Verify the lock artifact hash. Keep orchestration and publication on the trusted workflow revision. - Materialize the pinned OpenCode executable and configure the exact Codex executable's user-namespace profile before provider credentials are available. - Compress Daytona directory uploads with gzip. Preserve files, executable modes, symlinks, empty directories, and confinement checks. - Classify file-transfer RPC deadlines as infrastructure. Keep unrelated runner RPC failures visible. ## Verification - [Focused live report with screenshots and original attempts](https://pages.paperclip.ing/runner-reliability-20260921/): 14 of 15 selected Product E2E cases pass across the recorded revisions. Claude and Codex Stop → `/new`, Claude service approval, delegation, both hiring/reuse cases, and native Daytona warm continuity pass. - Two credentialed Runner smoke cases pass. These are not full protocol coverage. - E2E harness after the master merge: 429 tests pass. E2E and server TypeScript checks pass. - Daytona plugin: 239 tests pass, 6 skipped. Plugin TypeScript build passes. The compression test fails against the old code and passes with the change. - Runner backend/runtime regression group: 161 tests pass. Cancellation/startup selection: 26 tests pass. Approval delivery: 34 real-database tests pass. - Workflow security: 7 tests pass. Both edited workflows pass actionlint. Runner TypeScript and Rust builds pass. - After merging master, all 389 native executor tests pass, including both native backend paths and late startup after the Stop deadline. - Post-merge `pnpm -r typecheck` and `pnpm build` pass. The monolithic local `pnpm test:run` was interrupted to integrate master and is inconclusive. The [hosted CI test partitions](https://github.com/paperclipai/paperclip/actions/runs/35620461738) pass on `50a3e43822bcba1e0d07b1b45b0be91cbf9312da`. An unchanged sandbox callback schema test initially received HTTP 503. It passed five isolated local runs, its full local test file, and one failed-job CI retry. No assertion was weakened. ## Risks - Stop can wait for the bounded startup handoff. If it cannot settle, the existing pending-recovery state remains instead of a false acknowledgement. - Immediate approval delivery must remain idempotent across cleanup and recovery sweeps. Tests cover duplicate delivery and company/run boundaries. - The workflow changes still need hosted Linux verification. They retain the trusted workflow and credential boundaries. - Gzip reduces the observed provider upload from about 1.8 GB to 663 MB. It does not yet fix the remaining Claude Daytona transfer timeout. That recovery test never reached Claude, so recovery remains unverified. Use a matching image with the verified provider package preinstalled for the next recovery test; retain cold-upload coverage separately. - The report preserves diagnostic runs with missing source metadata and marks them as such. It does not claim a new full-suite pass. - This PR adds no new prompt policy or historical status reconciliation. ## Model Used OpenAI GPT-6 through Codex performed the primary implementation and review. The exact primary backend model ID is not exposed in this session. OpenAI `gpt-5.6-luna` assisted with bounded infrastructure work and verification. The agents used repository tools, code execution, and browser tests. The exact backend revision and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: OpenAI GPT-6 <noreply@openai.com>	2026-09-21 12:50:50 -05:00
DottaandPaperclip	54a99d8840	fix(evals): make the chat viewer the default published Evalbook (#12952) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Direct Runner evals retain evidence across model configurations. > - Evalbook already has a grid and a read-only Runner Lab chat viewer. > - Public projection stripped the view and selected a second plain result page. > - This change uses the existing viewer for public and private results. > - The data access differs, but the presentation does not. ## Linked Issues or Issue Description Refs #12931, #12945. Related open runtime-contract PR #11634 does not contain this report-only change. **What happened?** The public direct-eval campaign opened plain result pages. The access-controlled artifact used the chat viewer. Users could not follow the same recorded interaction from the published grid. **Expected behavior** Every newly generated Runner Evalbook opens the existing chat viewer. The grid and durable run history remain. Public evidence has explicit redactions. **Steps to reproduce** Open campaign gha-34062394019-1 from the direct-eval history. Click a result, then compare its plain page with the corresponding Actions artifact. **Paperclip version or commit** Reproduced at 83987210d6. **Deployment mode** Static GitHub Actions artifacts and S3/CloudFront publication. Companion site-theme renderer: https://github.com/paperclipai/paperclip-evals/pull/19. This removes the Python light theme and links the same built stylesheet. ## What Changed - Add a closed public chat projection. Require mock isolation evidence before publishing recorded text. Scrub private references and withhold tool payloads, reasoning and provider state. - Validate public HTML against the exact trusted viewer shell and asset bytes. Validate the public DTO and local links. Keep CSP restrictions on outbound requests and forms. - Make the workflow render both data projections with the canonical viewer. Pass a viewer-only artifact to the trusted publisher. Reject an old renderer pin before paid execution. - Fix report-only start position, missing-state inspector, read-only controls and redaction labels. Tool evidence links select and highlight the Evidence tab even when reopened. Runner execution and the full-stack E2E workflow are unchanged. - Add a no-model report refresh command. Preserve original campaign identity, measurements and immutable history; label report revisions. - Document the single presentation and public/private evidence boundary. - Use one Runner Lab stylesheet and local fonts for the grid, Latest, test design, inventory, server gate and S3 history index. Keep static styles scoped away from live chat. - Emit exact published report/history URLs to the Actions summary and job outputs only after successful upload. Set the deployment link from that output. - Switch public Eval and Evidence panes without rendering both at once. - Add all-run history with like-for-like pass-rate and cost timelines, regression/recovery lists, exact commit links, source refs and Actions links. - Record all-attempt costs including retries. Keep provider list cost separate from estimates. Label missing coverage and historical final-only costs. - Retain all run records beyond 200, backfill a separate derived analytics projection, and exclude report refreshes from measurements. ## Verification - Focused report/publishing/projection/workflow/adapter tests: 29 passed. - Workflow Evalbook adapter tests: three passed. - Viewer unit tests: eight passed; Vite viewer build passed. - Companion renderer tests: 59 total, 57 passed, two inventory tests skipped because the expected sibling checkout is absent. - Re-rendered all 375 retained attempts from the completed campaign and validated the public bundle. Zero new model calls; 356/358 selected cells still pass. - Browser walkthrough: grid to failed chat; prompt, named tool calls, correct blocked status, visible assertions, no loading spinner or composer. Public payload redactions are explicit. - Full typecheck and build passed locally. test:run ended with 17 failed files and 19 failed tests in unrelated server/worktree areas (3772 tests passed). Latest-head CI is the final merge gate. The browser sources also typecheck with a temporary TypeScript-7-compatible path configuration; the checked-in browser config still uses removed baseUrl options and is unchanged here. - Real Chromium verification passed for passing, failing and missing-recording attempts in both the full and public bundles. It checks tool expansion, navigation, reload, read-only controls, narrow viewport visibility and the public no-network boundary. Future publications run it automatically. - Fresh live proof: three gpt-5.4-mini native Codex cases passed on the first attempt (get-task-context, create-child-task, workflow-context-document-progress); estimated total $0.00632625. Generated the full canonical report and verified all three file:// pages, all seven DevTools tabs, and evidence cross-links in Chromium. Private screenshots remain local. - Follow-up 99697c2c5: viewer build, eight unit tests, browser token gate and browser-source typecheck (existing TS7 configuration workaround) passed. Added repeated tool-to-evidence navigation to the publication browser gate. - Site-theme follow-up: 26 publishing/security tests, 59 Python tests (57 passed, two existing skips), eight viewer unit tests and the viewer build passed. Chromium verified shared colors and grid/test-design/Latest navigation. - Re-rendered all 375 retained attempts with the shared site theme, without new provider calls. - Successful hosted publication: [refreshed Evalbook](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-34062394019-1-report-site-theme-v3/index.html) and [themed history](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html). Real browser verified grid → test design → chat → All results and history → run. - Full maintained live suite completed on AWS: [Actions run 34074939112](https://github.com/paperclipai/paperclip/actions/runs/34074939112), Paperclip 856813ba3a, evals 34e1846c06a39e641182dadce5de7ea739f657f1. All 358 cells across 11 configurations ran; 355 passed (99.2%), two behavior failures and one infrastructure failure. Nine configurations are entirely green. All 360 retained attempts were rendered using the new design and published as an immutable, zero-provider-call report refresh: [full Evalbook](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-34074939112-1-report-history-v1/index.html). The trusted master workflow ran the models; this PR's viewer rendered their results afterward. - Remaining live failures: native Codex mini did not emit the expected discovery event for lazy-unauthorized-undisclosed (no mutation occurred); GLM 5.3 scheduled a wake before approval in workflow-governed-wait; GLM 5.3 timed out on create-task-document on both attempts. No scoring rules were relaxed. - Retry-inclusive recorded estimated model cost is at least $7.067692, with usage for 357/360 attempts. Provider-reported list cost is a separate alternative (at least $14.002806), not an additive cost. Missing usage is unknown, not zero; AWS compute is excluded. - [Production history](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html) now contains nine model runs and two separately labeled report refreshes. Matching full-suite comparison reports three newly failing cells and two recoveries versus the previous run. This is observed run-to-run variation, not proof of a deterministic code regression. Live HTTP checks verify the newest report link, shared theme, exact source SHAs and cost analytics. - The new full report passed Chromium checks for passing, failing and missing-recording chat pages, tool-to-Evidence links, grid/design/Latest navigation, reload, read-only controls and narrow layouts. The Mac was locked during final hosted verification; the new hosted history was checked by HTTP and generated-page browser checks, not a fresh interactive desktop walkthrough. - History follow-up: 31 publishing/security/metrics tests pass. Viewer build passes. Chromium checks desktop and narrow history pages with no document-level horizontal overflow. ### Visual verification Generated from the scrubbed completed campaign; no private provider identities or raw tool payloads are included. Full private pages were also browser-tested, but their private metadata is not published as screenshots.    Latest shared-theme proof:     ## Risks - Public chat text is newly visible, but only for the isolated mock boundary. The producer excludes raw payloads and the publisher fails closed on unknown fields, secrets, shell changes and asset substitutions. - Requires the companion canonical renderer revision and an updated RUNNER_PROTOCOL_EVALS_SHA after merge. Old pins fail before paid execution. - Existing published campaigns remain immutable. A report refresh is a separate history entry, not a new model qualification. - Successfully published the immutable site-theme refresh with the configured report-bucket SSO profile. Original run records and qualification pointers are preserved. ## Model Used OpenAI Codex, GPT-5-based coding agent with reasoning, shell and browser tools. Exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on cad99dbf04 - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups on cad99dbf04 - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-07 07:47:17 -05:00
DottaandPaperclip	fee8d8dc39	fix(runner): repair direct live provider bootstrap (#12932) ... ## Thinking Path > - Paperclip runs AI agents through qualified provider backends. > - The direct live eval workflow builds one immutable Runner runtime for every matrix cell. > - The workflow reinstalled the packed Runner with npm. > - That install discarded pnpm patches and selected provider dependencies outside the qualified lock. > - The first pnpm deployment model also placed its virtual-store marker at the wrong level; a real deployment keeps `.pnpm` beside the scoped Runner package. > - AgentCore enforced the current context-aware harness but the direct eval CLI did not supply the production v3 runtime context that harness requires. > - This pull request preserves the qualified dependency graph, resolves the real deployment layout, and makes direct evals exercise the production runtime-context contract. > - The benefit is that live eval cells reach their provider turn with the same artifacts and context contract that Paperclip qualified. ## Linked Issues or Issue Description Refs: #12931 **What happened?** The full direct live eval campaign failed every ACPX cell during `session.open`. The portable runtime had an incorrect dependency root. Its npm install also discarded the qualified ACP server patches. AgentCore cells first failed because Runner enforced `aws-agentcore-harness-v1` while the provisioned stack and eval profile use `aws-agentcore-harness-context-v2`; after aligning that revision, the direct eval CLI still omitted the required v3 runtime context. **Expected behavior** The direct eval runtime must preserve the frozen pnpm dependency graph and patched provider bytes. Runner, server validation, OpenAPI, and the deployed AgentCore stack must use one qualification revision. Direct eval attempts must supply the same immutable native runtime-context contract as production. **Steps to reproduce** 1. Dispatch `Runner Direct Live Protocol Evals` from `master`. 2. Select an ACPX Claude, ACPX Codex, or AgentCore roster. 3. Observe a pre-turn provider bootstrap failure. **Paperclip version or commit** `d96452db059338b329b458ba8fe359fef72f1363` **Deployment mode** GitHub Actions on the RunsOn Linux x64 fleet. ## What Changed - Build the reusable direct-eval runtime with `pnpm deploy --prod`. - Resolve ACPX dependencies from the actual scoped-package layout of a self-contained pnpm deployment. - Align AgentCore configuration and qualification checks on `aws-agentcore-harness-context-v2`. - Materialize a minimal immutable v3 runtime context for each isolated direct eval attempt. - Add workflow, package-authority, runtime-context, Rust, and server regression coverage. - Document the qualified packaging, runtime-context, and AgentCore revision contracts. ## Verification - `pnpm --filter @paperclipai/paperclip-runner typecheck` - `pnpm --filter @paperclipai/server typecheck` - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport.test.ts` (70 tests) - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/cli/eval-session-contract.test.ts` (14 tests) - Focused Runner contract tests (36 tests) - Focused server profile tests (47 tests) - Focused Rust managed-provider and native-selector tests (19 tests) - `node --test packages/paperclip-runner/scripts/runner-protocol-eval-workflow-security.test.mjs` - `actionlint .github/workflows/runner-protocol-live-evals.yml` - A local `pnpm deploy --prod` produced both qualified ACP server digests. - A Linux reproduction of the first follow-up smoke identified the real deployment root and the missing AgentCore runtime context. ## Risks The AgentCore revision change rejects profiles that still use the obsolete v1 value. This is intentional because the provisioned context-aware harness and current eval profile use v2. Direct eval prompts now receive the same fixed runtime-context preamble as production, so behavior scores may move; that is the intended qualification surface. The workflow package layout changes, but tests assert the new entrypoint and dependency root. This change does not modify the browser full-stack E2E workflow. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex with GPT-5.6. The context-window size is not exposed in this session. The model used extended reasoning, repository tools, code execution, Docker-based Linux reproduction, and GitHub Actions diagnostics. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (for example, `docs/...` or `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-06 15:45:07 -05:00
Dotta	d96452db05	fix(runner): restore Vite 6 viewer compatibility (#12929) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Paperclip Runner includes an issue-thread viewer for direct Evalbook reports. > - The runner package uses Vite 6.4.3. > - Dependabot changed the React plugin from version 4.7.0 to version 6.1.1. > - React plugin 6.1.1 requires Vite 7 or Vite 8 package internals. > - The live evaluation workflow found this mismatch when it built the viewer on a clean Linux worker. > - This pull request restores the compatible plugin and adds the viewer build to pull request CI. > - The benefit is that CI detects this class of report-viewer build failure before a paid evaluation campaign starts. ## Linked Issues or Issue Description **What happened?** The direct live evaluation workflow failed before model execution. The `build:issue-thread` command could not load `@vitejs/plugin-react@6.1.1` with Vite 6.4.3. The plugin imported the unavailable `vite/internal` package path. **Expected behavior** The Evalbook issue-thread viewer must build from a clean frozen-lockfile installation before the live evaluation matrix starts. **Steps to reproduce** 1. Check out commit `165ca56a22adb60e5fda56045442d9c8498116a8`. 2. Run `pnpm install --frozen-lockfile --ignore-scripts`. 3. Run `pnpm --filter @paperclipai/paperclip-runner build:issue-thread`. 4. Observe the `ERR_PACKAGE_PATH_NOT_EXPORTED` error for `vite/internal`. **Paperclip version or commit** `165ca56a22adb60e5fda56045442d9c8498116a8` **Deployment mode** Built from source in GitHub Actions on Ubuntu. ## What Changed - Restore `@vitejs/plugin-react` 4.7.0 in the Vite 6 runner package. - Follow the repository policy: trusted PR CI regenerates and verifies the lockfile artifact, and the master refresh workflow commits the lock-only update after merge. - Build the Runner Evalbook viewer in pull request CI. ## Verification - `ci / policy`: regenerated the dependency lock artifact successfully - `pnpm --filter @paperclipai/paperclip-runner build:issue-thread` - `actionlint .github/workflows/pr-trusted.yml .github/workflows/runner-protocol-live-evals.yml` - `node --test packages/paperclip-runner/scripts/runner-protocol-eval-workflow-security.test.mjs` - `git diff --check` ## Risks Low risk. This change restores the previous React plugin major version for one package. The selected version declares support for Vite 6. Pull request CI now builds the affected viewer directly. > This bug fix does not add or change a roadmap feature. ## Model Used - OpenAI Codex with GPT-5. Tool use and code execution were enabled. The Codex app managed the context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-09-06 09:16:21 -05:00
Dotta	165ca56a22	fix(runner): scope live eval tokens to eval repo (#12911) ... ## Thinking Path > - The merged direct live eval workflow must read the private `paperclip-evals` repository at an exact commit. > - Its first hosted dispatch failed before provider execution because the GitHub App token was minted from the `paperclip` repository installation. > - GitHub returned 404 while resolving the private eval commit, proving that token did not have the required repository scope. > - Minting each short-lived token from the exact private eval repository installation supplies only the cross-repository read boundary the workflow needs. > - A workflow regression now verifies every eval-token block keeps that exact scope. ## Linked Issues or Issue Description The first default-branch run of Runner Direct Live Protocol Evals failed in its immutable eval-commit verification step with HTTP 404. No provider jobs ran and no provider spend occurred. **What existing behavior does this improve?** It allows the protected direct live eval workflow to verify and check out the private `paperclipai/paperclip-evals` repository. **Current behavior** All four eval-token blocks set `GH_REPO` to `paperclipai/paperclip`, selecting a token installation that cannot read the private eval repository. **Proposed behavior** Set `GH_REPO` to the exact `paperclipai/paperclip-evals` repository in authorization, catalog, matrix, and report jobs. **Reason and benefit** The app mints a short-lived token from the correct repository installation while the main repository continues to use its ordinary read-only workflow token. **Breaking changes** None. ## What Changed - Scoped all four private-eval installation tokens to `paperclipai/paperclip-evals`. - Added a regression requiring that exact scope in every token block. ## Verification - `pnpm --filter @paperclipai/paperclip-runner test:runner-protocol-eval-publish` — 15 passed. - `node --test .github/scripts/tests/get-bot-token.test.mjs` — 3 passed. - `actionlint .github/workflows/runner-protocol-live-evals.yml` — passed. - `git diff --check` — passed. ## Risks - The workflow reads a private repository. The token is still short-lived, repository-specific, masked immediately, and used only by the protected default-branch workflow. - This changes no provider execution, Runner behavior, report content, S3 publishing, or browser E2E behavior. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex on GPT-5. The exact deployment ID and context-window size are not exposed. The model used reasoning, repository inspection, code editing, GitHub Actions diagnostics, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation where applicable - [x] I have considered and documented risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-09-06 08:25:46 -05:00
Dotta	af8439a70b	feat(runner): restore direct live eval campaigns and reports (#12909) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Runner executes agents through native and managed provider drivers. > - The direct live eval layer had drifted from the current Runner contracts. > - The old local workflow did not provide a complete parallel campaign or durable report history. > - The Runner also needed current native OpenCode and OpenRouter qualification. > - This pull request restores the direct campaign, corrects the runtime gaps that the campaign found, and adds safe hosted Evalbook history. > - The benefit is repeatable model comparison against an immutable Runner and eval source revision. ## Linked Issues or Issue Description Refs #11297 Refs #11634 **What existing behavior does this improve?** This improves the direct live `paperclip-runner` eval workflow, provider execution contract, and static Evalbook reporting path. **Current behavior** The direct evals do not have one maintained full campaign on current `master`. OpenCode has no qualified multi-model OpenRouter roster. Parallel provider bursts can compact committed events before the transport observes them. Local reports do not have a separate safe S3 history index. **Proposed behavior** Run one immutable roster-plus-case matrix. Use the shared paid AWS runner fleet. Keep raw artifacts access-controlled. Publish a sanitized canonical Evalbook report under the separate `runner-protocol-evals` S3 prefix. Keep immutable campaign directories plus root history, latest, and latest-green pointers. **Reason and benefit** Maintainers can compare native Codex, native OpenCode, ACPX, Claude Managed, and AWS AgentCore behavior over time. They can inspect failures without mixing this direct protocol layer with browser full-stack E2E. **Breaking changes** None. The new workflow and S3 prefix are additive. The existing Runner full-stack E2E workflow and report remain separate. ## What Changed - Added a trusted two-shard direct live workflow for up to 393 roster-plus-case cells. - Reused the numeric actor allowlist, protected paid environment, and RunsOn fleet controls from Runner full-stack E2E. - Added immutable Runner and eval revision resolution, exact credential boundaries, bounded retries, and cost ceilings. - Added a public report projection that removes sessions, transcripts, tool payloads, state, traces, raw failures, remote profile identities, and credential-shaped values. - Added additive S3 history under `runner-protocol-evals`, with immutable campaigns and mutable root index pointers. - Added native OpenCode model injection and current OpenRouter pricing contracts. - Fixed direct eval completion, workflow execution, semantic discovery, warm-attach state reset, executable binding, and event-burst handling. - Kept Runner browser full-stack E2E behavior and publication separate. - Documented local and hosted direct eval operation. ## Verification - `pnpm --filter @paperclipai/paperclip-runner test:runner-protocol-eval-publish` — 15 passed. - `pnpm --filter @paperclipai/paperclip-runner build:typescript` — passed. - `actionlint .github/workflows/runner-protocol-live-evals.yml .github/workflows/runner-full-stack-e2e.yml` — passed. - Local current matrix at the revision in [paperclip-evals#17](https://github.com/paperclipai/paperclip-evals/pull/17) — 323 cells across 10 enabled configurations completed. - Final local current matrix — 269 passed, 11 behavior failures, and 43 expected macOS-only ACPX platform failures. - Targeted Runner checks — 13/13 eval-session tests, 15/15 publisher/security tests, and package typecheck passed; complete PR CI is green, including all browser E2E shards. ## Risks - Paid live campaigns can consume provider budget. Actor authorization, exact per-cell ceilings, protected environments, and explicit schedule enablement bound this risk. - Public reports can leak provider data. The workflow publishes only a separately projected report and validates every file before upload. - The new workflow cannot publish until it is present on the default branch. This pull request does not change the existing `runner-full-stack-e2e` publication path. - The campaign is large. It uses two GitHub matrices and caps combined concurrency at the shared fleet limit. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex on GPT-5. The exact deployment ID and context-window size are not exposed. The model used reasoning, code editing, browser inspection, repository tools, and live provider execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-09-05 20:18:11 -05:00