PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-08 22:13:59 +01:00

Commit Graph

Author	SHA1	Message	Date
DottaandPaperclip	85e4d12856	fix(evals): report controlled protocol cases from safe records (#19070 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner protocol qualification records real provider behavior in TestOps. > - Controlled attempts keep raw records and accounting in private storage. > - A live case passed, but a later public-render command rejected its controlled context. > - The missing public report caused the final result to be stored as broken. > - This pull request creates the closed TestOps result from existing sanitized records inside the private report job. > - The benefit is correct result reporting while the privacy guard stays effective. ## Linked Issues or Issue Description Refs #18985 and #19011. The related-PR search found no separate repair for this failure. What happened? [The live canary](https://github.com/paperclipai/paperclip/actions/runs/41916513378) passed all four provider checks. Its controller verified cleanup and closed its request accounting. The private canonical report recorded one passed case. The next command used `--public-viewer` while the controlled manifest remained set. The eval program rejected it with `controlled campaign evidence must remain private`. The closed exporter then had no public payload and correctly stored a broken result. Expected behavior Report the original case grade from sanitized structured records. Keep raw controlled records private. Keep the original checks, source bindings, and missing-evidence rules. Steps to reproduce Run one controlled protocol case with TestOps reporting enabled. Let the private canonical report finish. The following public-render step fails before the closed result can read the grade. Paperclip version or commit Observed at `4740d251b7032e714d33039ba66a56948238c38f`. This repair is based on `c6cb7e2a2d42743c0c3c4ead2cbad4a299f9a2e7`. Deployment mode GitHub-hosted Linux, controlled ACPX Claude protocol qualification, first workflow attempt, zero automatic retries. ## What Changed - Add a narrow `export-runs` path to the existing TestOps helper. Bind the sanitized score and artifact to the exact attempt, case, model, provider, driver, and public-projection schema. Reuse the existing strict grade and check validation. - Keep sanitization inside the private report job. Use structured records for controlled TestOps reporting. Keep the public viewer for legacy publication. - Add record identity, privacy, grade, command, and workflow routing controls. Wire the existing helper and workflow-security files into the required Node registry. Document the reporting path. ## Verification - 43 focused campaign/sanitizer, TestOps and workflow security tests passed. The command regression writes a raw attempt, calls the production sanitizer, and runs `export-runs` on its output. It also repeats that chain after a failed-check mutation. Both new-control files now run through required `test:typescript:prep`, not only the optional publisher script. - The actual required prep ran locally with provider credentials excluded. The eval-kernel and Rust builds passed. Its Node registry executed the new producer-to-export and security controls by name: 403 passed, 2 failed, 11 existing skips. The two original package-contract failures cannot resolve this isolated clone's Codex ACP and Claude ACP package links. This is not a green local full-prep claim. The first attempt also stopped at a missing Node-types link, before an ignored clone-local link to the locked cached version restored that prerequisite. - 72 release workflow tests passed with the existing locked pnpm 9.15.4 wrapper and local loopback permission. The first sandbox run failed because loopback fixtures could not listen; the default local tool path also failed two install fixtures. No test assertions were changed to resolve these failures. - The actual repaired workflow shell ran locally against the retained canary campaign and sanitized records. It emitted one closed passed result with all four original checks. No provider request or upload occurred. - Missing or failed checks still reject a pass. Private-only markers stay absent from the closed output. Wrong model, provider, driver, attempt, case, and schemas reject. - The unmodified, pinned eval program still rejects a controlled public-viewer command. - Actionlint and `git diff --check` passed. [Full final-head CI](https://github.com/paperclipai/paperclip/actions/runs/41927878886) passed on its first attempt at `0d0c090b1b`. Required verify/e2e gates, all four Runner lanes, typecheck, build, backend and browser shards passed. Package suite: 4951 passed, 10 existing skips across 297 passed files and one existing file skip. - The required static log contains passing lines for both the production sanitizer-to-export command regression and the closed-projection security control. The prior optional-script registration did not run those tests in required CI; the one script-entry fix now enforces them. Provider/runtime and workflow bytes are unchanged by the verification-only follow-ups. - Greptile: fresh 5/5 on `0d0c090b1b`, with no actionable new issues. The separate Paperclip Review requested a producer-to-consumer regression. That regression and its required-CI wiring are complete. The addressed thread is resolved; no open review threads remain. - No fresh deployed reporting pass is claimed. The failed workflow and its stored broken result remain unchanged. ## Risks The exporter reads structured records inside the private job. It copies only the closed result fields and requires the existing sanitized marker and identity checks. The final reporting job still receives only those files and verifies the saved result bindings. The Python privacy guard, provider path, case assertions, budgets, deadlines, and retries are unchanged. ## Model Used OpenAI Codex, exact model `gpt-6.1-sol`, max reasoning effort. The inherited runtime reports a 475,000-token context window. Capabilities used: reasoning, code changes, and repository, shell, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 22:03:59 -06:00
Dotta	8002dfb06e	fix(evals): repair bounded TestOps protocol admission (#19011 ) ## Thinking Path > - Paperclip lets people manage AI agent work through a governed control plane. > - Runner protocol qualification uses real providers and records selected cases in TestOps. > - A manual registration run starts an Allure watch session before it validates the case selection. > - Watch creates the job run, but it does not fill the empty job-run input in the child environment. > - The selection helper therefore stops before it can validate the new job run. > - A configured AWS concurrency ceiling can also reject a valid smaller hosted-runner request. > - Hosted tool-cache permissions can also violate Runner qualified-executable admission before a provider starts. > - This pull request captures official session identities, keeps both concurrency ceilings, and stages the same Node bytes in an owned directory. ## Linked Issues or Issue Description Refs #18985. What happened? The [first registration attempt](https://github.com/paperclipai/paperclip/actions/runs/41877603952) created an Allure job run, then stopped with `runner_testops_jobrun_missing`. The child inherited an empty `ALLURE_JOB_RUN_ID`. The helper made no job-run API read and executed no provider case. The current configured concurrency ceiling is 100. The controlled hosted cap is 57. The original catalog guard rejects 100 before it applies a valid requested value of 2. The [first service-selected attempt](https://github.com/paperclipai/paperclip/actions/runs/41884365232) reproduced this second pre-provider failure. Its selected case was correctly reported as broken. No provider cell ran. A [second service-selected attempt](https://github.com/paperclipai/paperclip/actions/runs/41887671585) passed selection, lock and cost preflight. Runner rejected a qualified ACPX launch artifact as group- or world-writable. It recorded no model request. The verified sidecar is mode 0755. Hosted Node is the only other launch artifact, which points to Node as the cause. The original run did not retain its exact mode. The TestOps reporter retained the infrastructure failure. Expected behavior The manual path must capture the job-run and launch IDs from the active official watch session. It must validate those IDs against the same GitHub run, job, launch and standing plan before provider work. A requested concurrency value must fit the configured ceiling and the selected runner cap. The default must use the lower ceiling. Before credentials, the worker must stage the same Node bytes in an owned directory and preserve Runner permission and digest admission. Steps to reproduce 1. Dispatch the trusted Runner protocol workflow with reporting enabled and an empty job-run service input. 2. Let `allurectl watch` create the registration session. 3. Observe that the original helper rejects the empty child input before selection validation. ## What Changed - Call the supported `allurectl job-run env` command for manual registration. Use a fixed argument list without a shell and bound its time and output. - Parse only numeric job-run and launch IDs. Reject missing, duplicate, malformed or conflicting values. Discard all other command output. - Bind the captured launch ID to the authoritative job-run response. Keep the service-supplied ID path and its existing checks. - Apply requested parallelism within both configured and physical ceilings. Use their minimum as the default. Keep the selected-case capacity and provider budget checks. - Snapshot the same current Linux Node bytes into a fresh owner-only directory before credentials. Verify the copy, execute it to bind its path and version, and put only that directory first on the job-local path. Record original and copied Node modes and hashes plus the actual sidecar mode and hash. Keep shared cache permissions and Runner security checks intact. - Add focused context, identity, concurrency, permission and secret-exclusion controls. Supply the production physical cap in the existing release-workflow fixture. - Fix the existing Kimi cancellation fixture to await the complete readiness sentinel within the same 100 by 10 ms bound. Keep its cancellation and cleanup assertions. ## Verification - The original empty-ID failure reproduces locally on the actual helper with all 41 standing cases and zero job-run API reads. - All 63 Node publication, projection, metrics, permission and workflow-security tests pass. The repaired manual context flows through the actual selection helper to case 194. Wrong launch, duplicate IDs, malformed values and command failures still stop selection. - The original catalog prefix rejects configured 100 / hosted 57 / requested 2. The repaired prefix admits 2, clamps a missing request to 57, keeps legacy 100, and rejects requests above either ceiling. - The full release-policy CI command passes locally: 72 release-workflow tests and 107 companion tests. The first sandbox run could not bind its loopback fixtures; the same tests passed with local-network access. The prior-head CI fixture omission is retained and fixed by supplying the same physical cap as production. - Both full static Node file groups pass locally: 418 passed and 11 existing opt-in skips, at concurrency 2. Initial runs used stale dependency links and failed. Correcting only isolated ignored links to declared cached dependencies and pnpm 9.15.4 removed those setup failures. Local Node 26.4 differs from CI Node 24; the exact-head CI below provides canonical verification. - `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git diff --check` pass. - The new staging helper passes all four tests as a nonroot Linux user with networking disabled. An actual retained Runner comparison reproduces the qualified-artifact rejection with Node mode 0777. The same Node bytes at 0500 in a 0700 directory pass that launch boundary and reach a closed provider session. The local turn still fails without credentials or network; this is permission proof, not a protocol case pass. Local Linux Node 24.21 differs from the hosted 24.22.1 executable. - Exact final head `8e06ee5611` [repository CI](https://github.com/paperclipai/paperclip/actions/runs/41892896405) completed successfully. Typecheck, build and all four Runner lanes passed. Runner package tests passed 4950 tests in 297 files, with 10 existing skips. Fresh exact-head Greptile review is 5/5 with no actionable findings or open threads. Prior `59378f` CI Kimi empty-sentinel race and canceled chat worker remain retained. - The patched manual workflow has not run against live TestOps. A separate service-triggered probe supplies its own ID. The service probe used the original concurrency guard with a temporary configured ceiling of 57. After the probe, the original repository ceiling 100 was restored with no active campaigns. That probe does not verify this patched concurrency code. The next launch waits for the protected merge and source/lock verification. ## Risks An incompatible official CLI output fails closed. The helper never evaluates the output or records its unknown variables. The existing job, workflow URL, source, plan and provider checks remain required. An omitted concurrency request now uses the lower of the configured and physical ceilings. Explicit requests above either ceiling still fail. The worker now executes a private copy of its existing Node binary. A copy, mode, hash, path, version or sidecar mismatch stops before credential access. The copy does not make dynamically linked Homebrew Node portable; this workflow path is Linux only. This change does not alter the provider roster, policy, limits, retries or result grading. ## Model Used OpenAI Codex, GPT-6. The exact hosted model ID and context window are not exposed in this session. Used reasoning, code editing, repository inspection and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-04 15:22:59 -06:00
DottaandPaperclip	039e0dad2c	Connect TestOps protocol plans to verified GitHub attempts (#18985 ) ## Thinking Path > - Paperclip lets people coordinate agent work through a governed control plane. > - Runner protocol evals test real provider sessions against a seeded mock control plane. > - The saved TestOps protocol plan has no GitHub executor. Its dashboard shows historical results. > - A current one-case attempt also failed before any tool call or provider output. Its retained public-safe snapshot withheld the error cause. > - This pull request connects selected cases to the existing trusted workflow and retains closed operator failure facts. > - Each selected case receives a current result. Missing evidence remains broken. Existing assertions and machine grades remain unchanged. ## Linked Issues or Issue Description Current behavior The saved protocol plan cannot start a GitHub qualification attempt. Historical dashboard results can be mistaken for current execution. A failed provider turn can also lose its failure category when the live snapshot applies public evidence redaction. Proposed behavior The existing workflow accepts TestOps service inputs, validates exact case identities before provider work, runs the selected cases, and verifies their published statuses. Eval artifacts retain only validated failure phase, category, HTTP status, and retryability. Reason and benefit Operators can run the standing plan from TestOps and inspect current, source-bound results. A targeted pass cannot stand in for full-plan coverage. A failed startup can be diagnosed without publishing provider messages or request identifiers. ## What Changed - Add optional TestOps selection and protected reporting jobs to the existing Runner protocol workflow. - Keep the current actor, source, provider credential, and controlled accounting gates. TestOps retains the explicitly selected controlled or legacy mode and a separate 41-case Claude roster. Controlled mode preserves its existing reviewed policy, immutable bindings, tracing and zero-retry admission. Its paid workers explicitly use GitHub-hosted runners, matching the current manifest admission contract even when the legacy AWS fleet is enabled. - Bind the exact standing plan, workflow, job run, source revisions, selected cases, and saved statuses. Accept only verified workflow URL aliases. - Create a closed Allure projection inside the report job. Transfer only fixed case identities, check verdicts, numeric usage and statuses to the reporting environment. Controlled raw/chat/controller evidence remains private. - Publish one result for every selected case. Retain missing execution/report evidence as broken. Reject unsupported passes, model mismatch and source drift. Preserve the evidence schema literal, stop publication on cancellation, and retain closed API error codes/statuses. - Tag full selections separately from targeted attempts. Add bounded ingestion observation. Apply the exact TestOps filter before the unchanged controlled provider/cell-capacity check, so a one-cell registration fits a one-cell reviewed policy. - Retain up to eight eval-only closed provider failure records before board evidence redaction. Forward the original notifications unchanged. - Add selection, reporting, privacy, and admission regression controls. Document scope and remaining acceptance. ## Verification - Direct Node protocol publication, projection, metrics, and security tests: 56 passed. Focused controls exercise both mode admissions, missing controlled policy/retries/source rejection, closed result transfer, exact metadata, and missing-result handling. - `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git diff --check` passed. - A proposed one-cell controlled policy passed the actual hosted validator with a synthetic prepared request and zero provider calls. The actual 41-case catalog filter selected case194/get-task-context and the real hosted admission accepted it under maxCells1. Negative controls rejected the unfiltered41, unsupported provider, source, selection, origin, expiry and lock drift. This does not verify provider auth, a runtime build or TestOps publication; final merged source/lock binding is still required. - The actual diagnostic TypeScript module passed positive, negative, secret-exclusion, and unchanged-notification controls through the existing tsx runtime. A local actual-CLI startup serializer probe passed with closed facts, secret exclusion, exit status and read-only fixture cleanup. This used an isolated resolver into the existing dependency cache; canonical exact-dependency CI remains required. - Prior-head CI passed typecheck, Runner static/Rust/native-server checks and 4946 Runner package tests. One new serializer fixture failed during Linux read-only directory cleanup; this head applies the existing neighboring cleanup pattern. Unrelated chat E2E and CLI import timeouts remain retained. Latest-head required gates are pending. - Original current-source one-case attempt: https://github.com/paperclipai/paperclip/actions/runs/41860234840 . It failed with no tool calls or output. That failure remains retained and is not a passing qualification claim. - Diagnostic candidate attempt: https://github.com/paperclipai/paperclip/actions/runs/41863410602 . It retained the new closed field but still failed before output with category unknown. This disproves the claim that typed retention alone identifies the cause. - The earlier passing run https://github.com/paperclipai/paperclip/actions/runs/40627274706 used controlled Braintrust routing. Both current failures used direct legacy credentials, a different auth path. Next acceptance is one explicitly controlled case with a reviewed current policy, followed by TestOps registration and a fresh full plan. None is claimed complete by this PR body. ## Risks The TestOps contract currently binds the existing 41 cases and one recorded Claude Sonnet 5 profile. Catalog changes fail closed until reviewed. This does not qualify other providers. Admission and execution qualify only the selected accounting mode; a controlled pass does not prove direct credential health. Model-request readiness retains pending operating reconciliation. TestOps ingestion can fail independently; persisted results are checked, and missing evidence is never accepted as a pass. The new failure field contains closed facts only; raw messages, identities, credentials, and tool payloads are excluded. ## Model Used OpenAI Codex, GPT-6. The exact hosted model ID and context window are not exposed in this session. Used reasoning, repository inspection, code editing, and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the available capability details - [x] I have checked ROADMAP.md and confirmed this does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs - [x] I have described the issue with the enhancement template field labels - [x] I have not referenced internal or instance-local Paperclip issues - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run the available targeted local checks and reported the remaining checks - [x] I have added relevant positive and negative regression controls - [x] I have updated the authoritative protocol documentation - [x] I have considered and documented the risks - [ ] All required Paperclip CI gates are green on the latest head - [ ] Greptile is 5/5 with no open actionable feedback on the latest head - [x] I have addressed all current review feedback --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 11:17:56 -06:00

Author

SHA1

Message

Date

DottaandPaperclip

85e4d12856

fix(evals): report controlled protocol cases from safe records (#19070 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner protocol qualification records real provider behavior in
TestOps.
> - Controlled attempts keep raw records and accounting in private
storage.
> - A live case passed, but a later public-render command rejected its
controlled context.
> - The missing public report caused the final result to be stored as
broken.
> - This pull request creates the closed TestOps result from existing
sanitized records inside the private report job.
> - The benefit is correct result reporting while the privacy guard
stays effective.

## Linked Issues or Issue Description

Refs #18985 and #19011. The related-PR search found no separate repair
for this failure.

**What happened?**

[The live
canary](https://github.com/paperclipai/paperclip/actions/runs/41916513378)
passed all four provider checks. Its controller verified cleanup and
closed its request accounting. The private canonical report recorded one
passed case. The next command used `--public-viewer` while the
controlled manifest remained set. The eval program rejected it with
`controlled campaign evidence must remain private`. The closed exporter
then had no public payload and correctly stored a broken result.

**Expected behavior**

Report the original case grade from sanitized structured records. Keep
raw controlled records private. Keep the original checks, source
bindings, and missing-evidence rules.

**Steps to reproduce**

Run one controlled protocol case with TestOps reporting enabled. Let the
private canonical report finish. The following public-render step fails
before the closed result can read the grade.

**Paperclip version or commit**

Observed at `4740d251b7032e714d33039ba66a56948238c38f`. This repair is
based on `c6cb7e2a2d42743c0c3c4ead2cbad4a299f9a2e7`.

**Deployment mode**

GitHub-hosted Linux, controlled ACPX Claude protocol qualification,
first workflow attempt, zero automatic retries.

## What Changed

- Add a narrow `export-runs` path to the existing TestOps helper. Bind
the sanitized score and artifact to the exact attempt, case, model,
provider, driver, and public-projection schema. Reuse the existing
strict grade and check validation.
- Keep sanitization inside the private report job. Use structured
records for controlled TestOps reporting. Keep the public viewer for
legacy publication.
- Add record identity, privacy, grade, command, and workflow routing
controls. Wire the existing helper and workflow-security files into the
required Node registry. Document the reporting path.

## Verification

- 43 focused campaign/sanitizer, TestOps and workflow security tests
passed. The command regression writes a raw attempt, calls the
production sanitizer, and runs `export-runs` on its output. It also
repeats that chain after a failed-check mutation. Both new-control files
now run through required `test:typescript:prep`, not only the optional
publisher script.
- The actual required prep ran locally with provider credentials
excluded. The eval-kernel and Rust builds passed. Its Node registry
executed the new producer-to-export and security controls by name: 403
passed, 2 failed, 11 existing skips. The two original package-contract
failures cannot resolve this isolated clone's Codex ACP and Claude ACP
package links. This is not a green local full-prep claim. The first
attempt also stopped at a missing Node-types link, before an ignored
clone-local link to the locked cached version restored that
prerequisite.
- 72 release workflow tests passed with the existing locked pnpm 9.15.4
wrapper and local loopback permission. The first sandbox run failed
because loopback fixtures could not listen; the default local tool path
also failed two install fixtures. No test assertions were changed to
resolve these failures.
- The actual repaired workflow shell ran locally against the retained
canary campaign and sanitized records. It emitted one closed passed
result with all four original checks. No provider request or upload
occurred.
- Missing or failed checks still reject a pass. Private-only markers
stay absent from the closed output. Wrong model, provider, driver,
attempt, case, and schemas reject.
- The unmodified, pinned eval program still rejects a controlled
public-viewer command.
- Actionlint and `git diff --check` passed. [Full final-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41927878886)
passed on its first attempt at `0d0c090b1b`. Required verify/e2e gates,
all four Runner lanes, typecheck, build, backend and browser shards
passed. Package suite: 4951 passed, 10 existing skips across 297 passed
files and one existing file skip.
- The required static log contains passing lines for both the production
sanitizer-to-export command regression and the closed-projection
security control. The prior optional-script registration did not run
those tests in required CI; the one script-entry fix now enforces them.
Provider/runtime and workflow bytes are unchanged by the
verification-only follow-ups.
- Greptile: fresh 5/5 on `0d0c090b1b`, with no actionable new issues.
The separate Paperclip Review requested a producer-to-consumer
regression. That regression and its required-CI wiring are complete. The
addressed thread is resolved; no open review threads remain.
- No fresh deployed reporting pass is claimed. The failed workflow and
its stored broken result remain unchanged.

## Risks

The exporter reads structured records inside the private job. It copies
only the closed result fields and requires the existing sanitized marker
and identity checks. The final reporting job still receives only those
files and verifies the saved result bindings. The Python privacy guard,
provider path, case assertions, budgets, deadlines, and retries are
unchanged.

## Model Used

OpenAI Codex, exact model `gpt-6.1-sol`, max reasoning effort. The
inherited runtime reports a 475,000-token context window. Capabilities
used: reasoning, code changes, and repository, shell, and GitHub tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-11-04 22:03:59 -06:00

Dotta

8002dfb06e

fix(evals): repair bounded TestOps protocol admission (#19011 )

## Thinking Path

> - Paperclip lets people manage AI agent work through a governed
control plane.
> - Runner protocol qualification uses real providers and records
selected cases in TestOps.
> - A manual registration run starts an Allure watch session before it
validates the case selection.
> - Watch creates the job run, but it does not fill the empty job-run
input in the child environment.
> - The selection helper therefore stops before it can validate the new
job run.
> - A configured AWS concurrency ceiling can also reject a valid smaller
hosted-runner request.
> - Hosted tool-cache permissions can also violate Runner
qualified-executable admission before a provider starts.
> - This pull request captures official session identities, keeps both
concurrency ceilings, and stages the same Node bytes in an owned
directory.

## Linked Issues or Issue Description

Refs #18985.

**What happened?**

The [first registration
attempt](https://github.com/paperclipai/paperclip/actions/runs/41877603952)
created an Allure job run, then stopped with
`runner_testops_jobrun_missing`. The child inherited an empty
`ALLURE_JOB_RUN_ID`. The helper made no job-run API read and executed no
provider case.

The current configured concurrency ceiling is 100. The controlled hosted
cap is 57. The original catalog guard rejects 100 before it applies a
valid requested value of 2. The [first service-selected
attempt](https://github.com/paperclipai/paperclip/actions/runs/41884365232)
reproduced this second pre-provider failure. Its selected case was
correctly reported as broken. No provider cell ran.

A [second service-selected
attempt](https://github.com/paperclipai/paperclip/actions/runs/41887671585)
passed selection, lock and cost preflight. Runner rejected a qualified
ACPX launch artifact as group- or world-writable. It recorded no model
request. The verified sidecar is mode 0755. Hosted Node is the only
other launch artifact, which points to Node as the cause. The original
run did not retain its exact mode. The TestOps reporter retained the
infrastructure failure.

**Expected behavior**

The manual path must capture the job-run and launch IDs from the active
official watch session. It must validate those IDs against the same
GitHub run, job, launch and standing plan before provider work. A
requested concurrency value must fit the configured ceiling and the
selected runner cap. The default must use the lower ceiling. Before
credentials, the worker must stage the same Node bytes in an owned
directory and preserve Runner permission and digest admission.

**Steps to reproduce**

1. Dispatch the trusted Runner protocol workflow with reporting enabled
and an empty job-run service input.
2. Let `allurectl watch` create the registration session.
3. Observe that the original helper rejects the empty child input before
selection validation.

## What Changed

- Call the supported `allurectl job-run env` command for manual
registration. Use a fixed argument list without a shell and bound its
time and output.
- Parse only numeric job-run and launch IDs. Reject missing, duplicate,
malformed or conflicting values. Discard all other command output.
- Bind the captured launch ID to the authoritative job-run response.
Keep the service-supplied ID path and its existing checks.
- Apply requested parallelism within both configured and physical
ceilings. Use their minimum as the default. Keep the selected-case
capacity and provider budget checks.
- Snapshot the same current Linux Node bytes into a fresh owner-only
directory before credentials. Verify the copy, execute it to bind its
path and version, and put only that directory first on the job-local
path. Record original and copied Node modes and hashes plus the actual
sidecar mode and hash. Keep shared cache permissions and Runner security
checks intact.
- Add focused context, identity, concurrency, permission and
secret-exclusion controls. Supply the production physical cap in the
existing release-workflow fixture.
- Fix the existing Kimi cancellation fixture to await the complete
readiness sentinel within the same 100 by 10 ms bound. Keep its
cancellation and cleanup assertions.

## Verification

- The original empty-ID failure reproduces locally on the actual helper
with all 41 standing cases and zero job-run API reads.
- All 63 Node publication, projection, metrics, permission and
workflow-security tests pass. The repaired manual context flows through
the actual selection helper to case 194. Wrong launch, duplicate IDs,
malformed values and command failures still stop selection.
- The original catalog prefix rejects configured 100 / hosted 57 /
requested 2. The repaired prefix admits 2, clamps a missing request to
57, keeps legacy 100, and rejects requests above either ceiling.
- The full release-policy CI command passes locally: 72 release-workflow
tests and 107 companion tests. The first sandbox run could not bind its
loopback fixtures; the same tests passed with local-network access. The
prior-head CI fixture omission is retained and fixed by supplying the
same physical cap as production.
- Both full static Node file groups pass locally: 418 passed and 11
existing opt-in skips, at concurrency 2. Initial runs used stale
dependency links and failed. Correcting only isolated ignored links to
declared cached dependencies and pnpm 9.15.4 removed those setup
failures. Local Node 26.4 differs from CI Node 24; the exact-head CI
below provides canonical verification.
- `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git
diff --check` pass.
- The new staging helper passes all four tests as a nonroot Linux user
with networking disabled. An actual retained Runner comparison
reproduces the qualified-artifact rejection with Node mode 0777. The
same Node bytes at 0500 in a 0700 directory pass that launch boundary
and reach a closed provider session. The local turn still fails without
credentials or network; this is permission proof, not a protocol case
pass. Local Linux Node 24.21 differs from the hosted 24.22.1 executable.
- Exact final head `8e06ee5611` [repository
CI](https://github.com/paperclipai/paperclip/actions/runs/41892896405)
completed successfully. Typecheck, build and all four Runner lanes
passed. Runner package tests passed 4950 tests in 297 files, with 10
existing skips. Fresh exact-head Greptile review is 5/5 with no
actionable findings or open threads. Prior `59378f` CI Kimi
empty-sentinel race and canceled chat worker remain retained.
- The patched manual workflow has not run against live TestOps. A
separate service-triggered probe supplies its own ID. The service probe
used the original concurrency guard with a temporary configured ceiling
of 57. After the probe, the original repository ceiling 100 was restored
with no active campaigns. That probe does not verify this patched
concurrency code. The next launch waits for the protected merge and
source/lock verification.

## Risks

An incompatible official CLI output fails closed. The helper never
evaluates the output or records its unknown variables. The existing job,
workflow URL, source, plan and provider checks remain required. An
omitted concurrency request now uses the lower of the configured and
physical ceilings. Explicit requests above either ceiling still fail.
The worker now executes a private copy of its existing Node binary. A
copy, mode, hash, path, version or sidecar mismatch stops before
credential access. The copy does not make dynamically linked Homebrew
Node portable; this workflow path is Linux only. This change does not
alter the provider roster, policy, limits, retries or result grading.

## Model Used

OpenAI Codex, GPT-6. The exact hosted model ID and context window are
not exposed in this session. Used reasoning, code editing, repository
inspection and tool execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

2026-11-04 15:22:59 -06:00

DottaandPaperclip

039e0dad2c

Connect TestOps protocol plans to verified GitHub attempts (#18985 )

## Thinking Path

> - Paperclip lets people coordinate agent work through a governed
control plane.
> - Runner protocol evals test real provider sessions against a seeded
mock control plane.
> - The saved TestOps protocol plan has no GitHub executor. Its
dashboard shows historical results.
> - A current one-case attempt also failed before any tool call or
provider output. Its retained public-safe snapshot withheld the error
cause.
> - This pull request connects selected cases to the existing trusted
workflow and retains closed operator failure facts.
> - Each selected case receives a current result. Missing evidence
remains broken. Existing assertions and machine grades remain unchanged.

## Linked Issues or Issue Description

**Current behavior**

The saved protocol plan cannot start a GitHub qualification attempt.
Historical dashboard results can be mistaken for current execution. A
failed provider turn can also lose its failure category when the live
snapshot applies public evidence redaction.

**Proposed behavior**

The existing workflow accepts TestOps service inputs, validates exact
case identities before provider work, runs the selected cases, and
verifies their published statuses. Eval artifacts retain only validated
failure phase, category, HTTP status, and retryability.

**Reason and benefit**

Operators can run the standing plan from TestOps and inspect current,
source-bound results. A targeted pass cannot stand in for full-plan
coverage. A failed startup can be diagnosed without publishing provider
messages or request identifiers.

## What Changed

- Add optional TestOps selection and protected reporting jobs to the
existing Runner protocol workflow.
- Keep the current actor, source, provider credential, and controlled
accounting gates. TestOps retains the explicitly selected controlled or
legacy mode and a separate 41-case Claude roster. Controlled mode
preserves its existing reviewed policy, immutable bindings, tracing and
zero-retry admission. Its paid workers explicitly use GitHub-hosted
runners, matching the current manifest admission contract even when the
legacy AWS fleet is enabled.
- Bind the exact standing plan, workflow, job run, source revisions,
selected cases, and saved statuses. Accept only verified workflow URL
aliases.
- Create a closed Allure projection inside the report job. Transfer only
fixed case identities, check verdicts, numeric usage and statuses to the
reporting environment. Controlled raw/chat/controller evidence remains
private.
- Publish one result for every selected case. Retain missing
execution/report evidence as broken. Reject unsupported passes, model
mismatch and source drift. Preserve the evidence schema literal, stop
publication on cancellation, and retain closed API error codes/statuses.
- Tag full selections separately from targeted attempts. Add bounded
ingestion observation. Apply the exact TestOps filter before the
unchanged controlled provider/cell-capacity check, so a one-cell
registration fits a one-cell reviewed policy.
- Retain up to eight eval-only closed provider failure records before
board evidence redaction. Forward the original notifications unchanged.
- Add selection, reporting, privacy, and admission regression controls.
Document scope and remaining acceptance.

## Verification

- Direct Node protocol publication, projection, metrics, and security
tests: 56 passed. Focused controls exercise both mode admissions,
missing controlled policy/retries/source rejection, closed result
transfer, exact metadata, and missing-result handling.
- `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git
diff --check` passed.
- A proposed one-cell controlled policy passed the actual hosted
validator with a synthetic prepared request and zero provider calls. The
actual 41-case catalog filter selected case194/get-task-context and the
real hosted admission accepted it under maxCells1. Negative controls
rejected the unfiltered41, unsupported provider, source, selection,
origin, expiry and lock drift. This does not verify provider auth, a
runtime build or TestOps publication; final merged source/lock binding
is still required.
- The actual diagnostic TypeScript module passed positive, negative,
secret-exclusion, and unchanged-notification controls through the
existing tsx runtime. A local actual-CLI startup serializer probe passed
with closed facts, secret exclusion, exit status and read-only fixture
cleanup. This used an isolated resolver into the existing dependency
cache; canonical exact-dependency CI remains required.
- Prior-head CI passed typecheck, Runner static/Rust/native-server
checks and 4946 Runner package tests. One new serializer fixture failed
during Linux read-only directory cleanup; this head applies the existing
neighboring cleanup pattern. Unrelated chat E2E and CLI import timeouts
remain retained. Latest-head required gates are pending.
- Original current-source one-case attempt:
https://github.com/paperclipai/paperclip/actions/runs/41860234840 . It
failed with no tool calls or output. That failure remains retained and
is not a passing qualification claim.
- Diagnostic candidate attempt:
https://github.com/paperclipai/paperclip/actions/runs/41863410602 . It
retained the new closed field but still failed before output with
category unknown. This disproves the claim that typed retention alone
identifies the cause.
- The earlier passing run
https://github.com/paperclipai/paperclip/actions/runs/40627274706 used
controlled Braintrust routing. Both current failures used direct legacy
credentials, a different auth path. Next acceptance is one explicitly
controlled case with a reviewed current policy, followed by TestOps
registration and a fresh full plan. None is claimed complete by this PR
body.

## Risks

The TestOps contract currently binds the existing 41 cases and one
recorded Claude Sonnet 5 profile. Catalog changes fail closed until
reviewed. This does not qualify other providers. Admission and execution
qualify only the selected accounting mode; a controlled pass does not
prove direct credential health. Model-request readiness retains pending
operating reconciliation. TestOps ingestion can fail independently;
persisted results are checked, and missing evidence is never accepted as
a pass. The new failure field contains closed facts only; raw messages,
identities, credentials, and tool payloads are excluded.

## Model Used

OpenAI Codex, GPT-6. The exact hosted model ID and context window are
not exposed in this session. Used reasoning, repository inspection, code
editing, and tool execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and the available capability
details
- [x] I have checked ROADMAP.md and confirmed this does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs
- [x] I have described the issue with the enhancement template field
labels
- [x] I have not referenced internal or instance-local Paperclip issues
- [x] My branch name describes the change and contains no internal
ticket ID
- [x] I have run the available targeted local checks and reported the
remaining checks
- [x] I have added relevant positive and negative regression controls
- [x] I have updated the authoritative protocol documentation
- [x] I have considered and documented the risks
- [ ] All required Paperclip CI gates are green on the latest head
- [ ] Greptile is 5/5 with no open actionable feedback on the latest
head
- [x] I have addressed all current review feedback

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-11-04 11:17:56 -06:00