Preserve existing campaign measurements and initial ordinals; incorporate already-merged production behavior only. No live qualification claimed.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip manages AI agents and measures their behavior with Runner
evaluations.
> - The cost controller, native tracing, private receipts and portable
runtime are on master.
> - The maintained live workflow still launches the provider CLI
directly.
> - Independent workers need one shared selection and disjoint budget
allocations.
> - Reports must keep unresolved allocations when a worker produces no
evidence.
> - This PR connects the maintained workflow to the existing controller
and private archive.
> - Operators can inspect known costs, conservative holds and trace
evidence before qualifying paid recurrence.
## Linked Issues or Issue Description
Refs #17452, #17469, #17493, #17506, #17523.
**Current behavior**
The live Runner workflow does not use the merged cost/tracing
controller. Attempt-only aggregation cannot reconstruct a campaign when
every worker archive is absent. Ordinary Actions artifacts in this
public repository are readable by authenticated public GitHub users.
**Proposed behavior**
Controlled runs prepare one manifest, bind every selected request,
enforce disjoint allocations and retain evidence in private storage. The
report starts from the full retained manifest. Missing workers keep
their allocations. The workflow rejects retries, unsupported providers,
changed requests and absent or stale operating bounds before provider
work.
**Reason and benefit**
The maintained workflow needs auditable costs and traces, including
failures and missing receipts, before paid recurrence can be qualified.
## What Changed
- Add controlled mode as the manual default and the only scheduled mode.
Legacy mode remains explicit for other manual provider qualifications.
- Prepare the full request selection and one shared manifest. Bind
current run, first attempt, code, definitions, executable hashes and
exact profile allocations.
- Verify the copied controller and manifest before provider credentials.
Require the controlled target to equal the trusted workflow revision
before private archive access.
- Reuse private archive retention for the controller policy, cell
attempts and canonical report. AWS credentials remain transfer-step
outputs. Controlled raw evidence is never uploaded as an Actions
artifact.
- Replay a retained manifest with partial or empty worker evidence.
Count operating holds once and preserve known partial charges.
- Pin the controller to an explicitly reviewed compatible lockfile
digest. Reject resolver or byte drift before a frozen, script-free
install into a job-local store. Record the lock digest in the runtime
receipt and verify it again in private preparation.
- Build the portable controller without secrets, private environment
access or OIDC. Private preparation verifies that artifact and performs
no dependency install. OIDC jobs use no mutable dependency resolution or
shared cache.
- Add synthetic source/copied-runtime tests, executable workflow
admission tests, private archive tests and an operator runbook.
## Verification
- Controller, campaign, portable runtime and workflow tests: 104 passed.
This includes real aggregation from an isolated reporter without source
dependencies, safe diagnostics, driver drift, distinct evidence binding
and reviewed-lock drift. Real private Codex/Claude requests also pass
through an independently verified copied runtime with fake transports.
- Private transport and workflow contracts: 35 passed. Executable/static
workflow guards: 13 passed.
- Paired private report tests: 166 passed; Evalbook discovery: 7 passed.
- `pnpm -r typecheck` and `pnpm build`: passed.
- `pnpm test:e2e:runner:typecheck`, `actionlint
.github/workflows/runner-protocol-live-evals.yml` and `git diff
--check`: passed.
- `pnpm test:run` hit repeated local embedded PostgreSQL initialization
failures in unrelated DB-dependent server tests. Its owned process tree
was stopped after preserving the log. This is a local verification
limitation; current-head CI remains required.
- An earlier head hit one unchanged Rust cleanup assertion. The exact
test passed locally and the final head passed the full Rust lane; the
initial failure log is retained.
- On the final head, one unchanged chat idle-queue assertion failed once
in CI. Its single scoped retry passed. All test jobs and the final
aggregate gate passed on commit
`2e328fa53dd296174d59d19f14f068aeeb0b82d8` (CI run 40535692048, attempt
2).
- No paid model, Braintrust or cloud calls. No live variables, secrets,
schedules or resources changed.
## Risks
- Controlled mode fails closed until a reviewed
`RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON` and a compatible private
eval SHA exist. `all` does not silently become a supported subset.
- The reviewed policy must cover setup, model requests, trace/gateway
processing, archive transfer and retained storage. Hash binding does not
establish vendor billing bounds. This PR does not enable recurrence or
release historical holds.
- The controlled policy must pin `controllerLockSha256`; it never
accepts a resolver-generated digest automatically. No lockfile is
committed. A changed compatible lock needs review.
- Legacy browser verification now requires the committed frozen
lockfile. A stale lockfile fails closed; controlled preparation has no
dependency install.
- Workflow cancellation can prevent the final report. The full manifest
is retained before provider work and remains the starting point for
reconciliation.
- The workflow exceeds 1,000 lines because the shared shard anchor,
dependencies, environment expressions and credential boundaries belong
to one declarative job graph. Request preparation, accounting, archive
safety and runtime verification stay in separately tested modules.
## Model Used
OpenAI Codex, GPT-6-based coding agent with tool use and code execution.
The host does not expose an exact deployment model ID or context-window
size.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the relevant tests locally and they pass; the broad
local suite limitation is documented above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green on the final head
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups;
the secondary reviewer is also 5/5
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
Do not misclassify absent duplicate prose as a false completion claim. Require the actual visible human-only source-run-bound proposal card, exact secret/consumer/reference identity, and independent absent binding before either human decision. Contradictory prose, missing authority or wrong-card evidence still fail.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Open the exact persisted pending interaction and verify the selected synthetic app/action in the production takeover section. Scope all decisions to that review instead of waiting for a nonexistent dialog; preserve actual approval and provider-call oracles.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Prepare only company and source-agent public funding before incident prehistory. Independently preserve Morgan policy, incident and pause context and verify the canonical source has no extraneous funding gate.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Preserve the original failures while distinguishing nonretryable native billing debt and stopping before a human action can mask unauthorized budget changes. Admit only the exact declared company-preview human upload with existing independent receipt checks.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
