PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-02 16:58:52 +01:00

Commit Graph

Author	SHA1	Message	Date
DottaandPaperclip	cff331f6ca	fix(evals): retain local controller cost origin (#17990 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - Its eval journals retain model costs and operating reservations under one cap. > - The operating policy records the controller's execution origin. > - The schema currently accepts only GitHub-hosted and AWS Fleet origins. > - A local controller must therefore use an inaccurate label or fail admission. > - This PR adds the local origin and keeps the hosted-only schedule gate. ## Linked Issues or Issue Description Refs #17986. No duplicate open PR or issue was found in the operating-overhead/local search. What existing behavior does this improve? The Product E2E operating allowance and its retained journal policy. Current behavior At master `40c1ab36cd`, an unscheduled local controller cannot set `runner: "local"`. The generic allowance validator rejects it. Local qualification can use the same accounting controls but cannot record its origin correctly. Proposed behavior Accept the local origin in the generic allowance. Keep all component, price, cap, expiry, and receipt rules. The scheduled validator still requires `github-hosted` and rejects a local allowance. Reason and benefit Local qualification can retain an accurate execution origin without claiming that its reservations are billed GitHub charges. ## What Changed - Add `local` to the operating allowance's runner type and validator. - Exercise local journal creation, model admission, retained overhead, restart, and reconciliation in the existing boundary test. - Check that a local allowance fails Product scheduled admission. - Pin new hosted Runner controller admission to `github-hosted`. Preserve original labels when reading historical manifests; those reads cannot authorize a new worker. - Exercise the hosted-origin rejection and historical/local reads in both source and copied portable runtimes. - Document the local label and its accounting limits. ## Verification - Before the fix: the new local persistence case and schedule-boundary assertion failed; eight tests passed. - Focused ledger, overhead, operations, and scheduled finalization tests: 71 passed across five files. - Controlled Runner source and portable tests: 86 passed. The new hosted-origin regression failed before the fix. - Full Product support suite at the first commit: 4,524 passed, three skipped, and one retained embedded PostgreSQL bootstrap failure. The same database failure was previously reproduced on a clean baseline. - Separate Node test phase on the host: 137 passed. - Actual Linux controller: database lifecycle tests pass. After installing its missing `zip` utility, the full Product suite has 4,527 passes and one Copilot parent-version coverage failure. That same failure reproduces on clean master `40c1ab36cd` with the same Linux filesystem and dependencies (14 passes, one failure). Copilot files are unchanged. The initial seven ZIP-related failures and the watcher failure remain retained. - The final fresh Linux semantic check passes: `GOMEMLIMIT=2GiB GOGC=50 GOMAXPROCS=1 node node_modules/typescript/bin/tsc --singleThreaded -p tests/runner-e2e/tsconfig.json`. The parallel compiler OOM is retained. No semantic checks were omitted. - Both fresh reviews are 5/5, with the addressed thread resolved. Hosted Build first failed during unchanged Pi setup prefetch with an operation timeout. One job rerun recovered; the original failure remains retained. All 58 latest-head checks are now acceptable. - No paid model requests, sandbox creation, schedule changes, or historical journal edits. ## Risks - The caller supplies this origin. It must describe the actual controller. - `local` does not prove zero compute cost, provide a default allowance, or release a hold. - Both maintained hosted admission paths require the GitHub-hosted controller origin. A previously accepted hosted policy with an AWS/local label now fails new admission. Historical reads preserve the original evidence. ## Model Used - OpenAI GPT-6 through Codex, with tool use and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 02:05:36 -05:00
DottaandPaperclip	35820cf398	fix(evals): qualify Runner model receipts with separate operating reconciliation (#17929 ) ## Thinking Path > - Paperclip must account for paid eval work. > - The maintained Runner workflow distributes one immutable controller manifest. > - Its cost gate currently requires vendor proofs for every operating charge. > - Model-spend qualification must also work under an explicit policy that keeps operating charges separate. > - This PR applies the shared model-readiness policy to the hosted controller. > - It retains every operating hold and still rejects missing model receipts and incomplete workers. ## Linked Issues or Issue Description Refs #17928, #17889. What happened? Runner qualification requires full-lifecycle vendor operating proofs even when the operator selects model-request readiness. That blocks the model-token qualification path. Expected behavior An explicit `overhead.accountingReadiness: "model_requests"` policy permits absent operating cost proofs. Supplied proofs still validate. Complete final model receipts are required. Steps to reproduce Prepare a two-worker hosted manifest with the explicit policy and no vendor cost proofs. Preserve missing-worker allocations, then complete both workers and inspect the private archive summary. A second test returns a provider refusal without terminal usage and requires failure with the full hold. ## What Changed - Apply the shared explicit policy to hosted capacity and observability cost-proof admission. - Preserve pricing windows, model bounds, immutable source/request/runtime identities, trace quotas and every allocation. - Reject zero or unresolved model requests in this policy, even if a request has a provisional bound. - Retain operating amounts and pending reconciliation through campaign and private archive summaries. - Document the separate model-readiness contract. Preserve strict behavior when the field is absent. ## Verification - Regression first failed at the original vendor-bound gate. - All 80 controlled-eval Node checks pass, including source and copied portable runtime checks, archived partial/missing workers, credential exclusion and model receipt failure. - Product readiness prerequisite #17928 has its own eval unit/typecheck and full CI checks. - No paid workflow, credential change or recurring activation is part of this source PR. ## Risks This opt-in policy does not prove a full-lifecycle operating-cost ceiling. The report retains operating holds and pending reconciliation. Default policies retain the existing vendor-proof gate. Source and worker identity checks remain required. A model refusal without final usage now fails readiness even if its reservation has a reviewed bound. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 15:41:07 -05:00
DottaandPaperclip	8548173fc9	feat(evals): connect hosted Runner costs and traces (#17889 ) ## Thinking Path > - Paperclip manages AI agents and measures their behavior with Runner evaluations. > - The cost controller, native tracing, private receipts and portable runtime are on master. > - The maintained live workflow still launches the provider CLI directly. > - Independent workers need one shared selection and disjoint budget allocations. > - Reports must keep unresolved allocations when a worker produces no evidence. > - This PR connects the maintained workflow to the existing controller and private archive. > - Operators can inspect known costs, conservative holds and trace evidence before qualifying paid recurrence. ## Linked Issues or Issue Description Refs #17452, #17469, #17493, #17506, #17523. Current behavior The live Runner workflow does not use the merged cost/tracing controller. Attempt-only aggregation cannot reconstruct a campaign when every worker archive is absent. Ordinary Actions artifacts in this public repository are readable by authenticated public GitHub users. Proposed behavior Controlled runs prepare one manifest, bind every selected request, enforce disjoint allocations and retain evidence in private storage. The report starts from the full retained manifest. Missing workers keep their allocations. The workflow rejects retries, unsupported providers, changed requests and absent or stale operating bounds before provider work. Reason and benefit The maintained workflow needs auditable costs and traces, including failures and missing receipts, before paid recurrence can be qualified. ## What Changed - Add controlled mode as the manual default and the only scheduled mode. Legacy mode remains explicit for other manual provider qualifications. - Prepare the full request selection and one shared manifest. Bind current run, first attempt, code, definitions, executable hashes and exact profile allocations. - Verify the copied controller and manifest before provider credentials. Require the controlled target to equal the trusted workflow revision before private archive access. - Reuse private archive retention for the controller policy, cell attempts and canonical report. AWS credentials remain transfer-step outputs. Controlled raw evidence is never uploaded as an Actions artifact. - Replay a retained manifest with partial or empty worker evidence. Count operating holds once and preserve known partial charges. - Pin the controller to an explicitly reviewed compatible lockfile digest. Reject resolver or byte drift before a frozen, script-free install into a job-local store. Record the lock digest in the runtime receipt and verify it again in private preparation. - Build the portable controller without secrets, private environment access or OIDC. Private preparation verifies that artifact and performs no dependency install. OIDC jobs use no mutable dependency resolution or shared cache. - Add synthetic source/copied-runtime tests, executable workflow admission tests, private archive tests and an operator runbook. ## Verification - Controller, campaign, portable runtime and workflow tests: 104 passed. This includes real aggregation from an isolated reporter without source dependencies, safe diagnostics, driver drift, distinct evidence binding and reviewed-lock drift. Real private Codex/Claude requests also pass through an independently verified copied runtime with fake transports. - Private transport and workflow contracts: 35 passed. Executable/static workflow guards: 13 passed. - Paired private report tests: 166 passed; Evalbook discovery: 7 passed. - `pnpm -r typecheck` and `pnpm build`: passed. - `pnpm test:e2e:runner:typecheck`, `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git diff --check`: passed. - `pnpm test:run` hit repeated local embedded PostgreSQL initialization failures in unrelated DB-dependent server tests. Its owned process tree was stopped after preserving the log. This is a local verification limitation; current-head CI remains required. - An earlier head hit one unchanged Rust cleanup assertion. The exact test passed locally and the final head passed the full Rust lane; the initial failure log is retained. - On the final head, one unchanged chat idle-queue assertion failed once in CI. Its single scoped retry passed. All test jobs and the final aggregate gate passed on commit `2e328fa53dd296174d59d19f14f068aeeb0b82d8` (CI run 40535692048, attempt 2). - No paid model, Braintrust or cloud calls. No live variables, secrets, schedules or resources changed. ## Risks - Controlled mode fails closed until a reviewed `RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON` and a compatible private eval SHA exist. `all` does not silently become a supported subset. - The reviewed policy must cover setup, model requests, trace/gateway processing, archive transfer and retained storage. Hash binding does not establish vendor billing bounds. This PR does not enable recurrence or release historical holds. - The controlled policy must pin `controllerLockSha256`; it never accepts a resolver-generated digest automatically. No lockfile is committed. A changed compatible lock needs review. - Legacy browser verification now requires the committed frozen lockfile. A stale lockfile fails closed; controlled preparation has no dependency install. - Workflow cancellation can prevent the final report. The full manifest is retained before provider work and remains the starting point for reconciliation. - The workflow exceeds 1,000 lines because the shared shard anchor, dependencies, environment expressions and credential boundaries belong to one declarative job graph. Request preparation, accounting, archive safety and runtime verification stay in separately tested modules. ## Model Used OpenAI Codex, GPT-6-based coding agent with tool use and code execution. The host does not expose an exact deployment model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the relevant tests locally and they pass; the broad local suite limitation is documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on the final head - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups; the secondary reviewer is also 5/5 - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 11:38:16 -05:00

Author

SHA1

Message

Date

DottaandPaperclip

cff331f6ca

fix(evals): retain local controller cost origin (#17990 )

## Thinking Path

> - Paperclip manages AI agents and their work.
> - Its eval journals retain model costs and operating reservations
under one cap.
> - The operating policy records the controller's execution origin.
> - The schema currently accepts only GitHub-hosted and AWS Fleet
origins.
> - A local controller must therefore use an inaccurate label or fail
admission.
> - This PR adds the local origin and keeps the hosted-only schedule
gate.

## Linked Issues or Issue Description

Refs #17986. No duplicate open PR or issue was found in the
operating-overhead/local search.

**What existing behavior does this improve?**

The Product E2E operating allowance and its retained journal policy.

**Current behavior**

At master `40c1ab36cd`, an unscheduled local controller cannot set
`runner: "local"`. The generic allowance validator rejects it. Local
qualification can use the same accounting controls but cannot record its
origin correctly.

**Proposed behavior**

Accept the local origin in the generic allowance. Keep all component,
price, cap, expiry, and receipt rules. The scheduled validator still
requires `github-hosted` and rejects a local allowance.

**Reason and benefit**

Local qualification can retain an accurate execution origin without
claiming that its reservations are billed GitHub charges.

## What Changed

- Add `local` to the operating allowance's runner type and validator.
- Exercise local journal creation, model admission, retained overhead,
restart, and reconciliation in the existing boundary test.
- Check that a local allowance fails Product scheduled admission.
- Pin new hosted Runner controller admission to `github-hosted`.
Preserve original labels when reading historical manifests; those reads
cannot authorize a new worker.
- Exercise the hosted-origin rejection and historical/local reads in
both source and copied portable runtimes.
- Document the local label and its accounting limits.

## Verification

- Before the fix: the new local persistence case and schedule-boundary
assertion failed; eight tests passed.
- Focused ledger, overhead, operations, and scheduled finalization
tests: 71 passed across five files.
- Controlled Runner source and portable tests: 86 passed. The new
hosted-origin regression failed before the fix.
- Full Product support suite at the first commit: 4,524 passed, three
skipped, and one retained embedded PostgreSQL bootstrap failure. The
same database failure was previously reproduced on a clean baseline.
- Separate Node test phase on the host: 137 passed.
- Actual Linux controller: database lifecycle tests pass. After
installing its missing `zip` utility, the full Product suite has 4,527
passes and one Copilot parent-version coverage failure. That same
failure reproduces on clean master `40c1ab36cd` with the same Linux
filesystem and dependencies (14 passes, one failure). Copilot files are
unchanged. The initial seven ZIP-related failures and the watcher
failure remain retained.
- The final fresh Linux semantic check passes: `GOMEMLIMIT=2GiB GOGC=50
GOMAXPROCS=1 node node_modules/typescript/bin/tsc --singleThreaded -p
tests/runner-e2e/tsconfig.json`. The parallel compiler OOM is retained.
No semantic checks were omitted.
- Both fresh reviews are 5/5, with the addressed thread resolved. Hosted
Build first failed during unchanged Pi setup prefetch with an operation
timeout. One job rerun recovered; the original failure remains retained.
All 58 latest-head checks are now acceptable.
- No paid model requests, sandbox creation, schedule changes, or
historical journal edits.

## Risks

- The caller supplies this origin. It must describe the actual
controller.
- `local` does not prove zero compute cost, provide a default allowance,
or release a hold.
- Both maintained hosted admission paths require the GitHub-hosted
controller origin. A previously accepted hosted policy with an AWS/local
label now fails new admission. Historical reads preserve the original
evidence.

## Model Used

- OpenAI GPT-6 through Codex, with tool use and code execution. The
exact serving model ID and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-28 02:05:36 -05:00

DottaandPaperclip

35820cf398

fix(evals): qualify Runner model receipts with separate operating reconciliation (#17929 )

## Thinking Path

> - Paperclip must account for paid eval work.
> - The maintained Runner workflow distributes one immutable controller
manifest.
> - Its cost gate currently requires vendor proofs for every operating
charge.
> - Model-spend qualification must also work under an explicit policy
that keeps operating charges separate.
> - This PR applies the shared model-readiness policy to the hosted
controller.
> - It retains every operating hold and still rejects missing model
receipts and incomplete workers.

## Linked Issues or Issue Description

Refs #17928, #17889.

**What happened?**
Runner qualification requires full-lifecycle vendor operating proofs
even when the operator selects model-request readiness. That blocks the
model-token qualification path.

**Expected behavior**
An explicit `overhead.accountingReadiness: "model_requests"` policy
permits absent operating cost proofs. Supplied proofs still validate.
Complete final model receipts are required.

**Steps to reproduce**
Prepare a two-worker hosted manifest with the explicit policy and no
vendor cost proofs. Preserve missing-worker allocations, then complete
both workers and inspect the private archive summary. A second test
returns a provider refusal without terminal usage and requires failure
with the full hold.

## What Changed

- Apply the shared explicit policy to hosted capacity and observability
cost-proof admission.
- Preserve pricing windows, model bounds, immutable
source/request/runtime identities, trace quotas and every allocation.
- Reject zero or unresolved model requests in this policy, even if a
request has a provisional bound.
- Retain operating amounts and pending reconciliation through campaign
and private archive summaries.
- Document the separate model-readiness contract. Preserve strict
behavior when the field is absent.

## Verification

- Regression first failed at the original vendor-bound gate.
- All 80 controlled-eval Node checks pass, including source and copied
portable runtime checks, archived partial/missing workers, credential
exclusion and model receipt failure.
- Product readiness prerequisite #17928 has its own eval unit/typecheck
and full CI checks.
- No paid workflow, credential change or recurring activation is part of
this source PR.

## Risks

This opt-in policy does not prove a full-lifecycle operating-cost
ceiling. The report retains operating holds and pending reconciliation.
Default policies retain the existing vendor-proof gate. Source and
worker identity checks remain required. A model refusal without final
usage now fails readiness even if its reservation has a reviewed bound.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection and
code execution. The exact serving model ID and context window are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-27 15:41:07 -05:00

DottaandPaperclip

8548173fc9

feat(evals): connect hosted Runner costs and traces (#17889 )

## Thinking Path

> - Paperclip manages AI agents and measures their behavior with Runner
evaluations.
> - The cost controller, native tracing, private receipts and portable
runtime are on master.
> - The maintained live workflow still launches the provider CLI
directly.
> - Independent workers need one shared selection and disjoint budget
allocations.
> - Reports must keep unresolved allocations when a worker produces no
evidence.
> - This PR connects the maintained workflow to the existing controller
and private archive.
> - Operators can inspect known costs, conservative holds and trace
evidence before qualifying paid recurrence.

## Linked Issues or Issue Description

Refs #17452, #17469, #17493, #17506, #17523.

**Current behavior**

The live Runner workflow does not use the merged cost/tracing
controller. Attempt-only aggregation cannot reconstruct a campaign when
every worker archive is absent. Ordinary Actions artifacts in this
public repository are readable by authenticated public GitHub users.

**Proposed behavior**

Controlled runs prepare one manifest, bind every selected request,
enforce disjoint allocations and retain evidence in private storage. The
report starts from the full retained manifest. Missing workers keep
their allocations. The workflow rejects retries, unsupported providers,
changed requests and absent or stale operating bounds before provider
work.

**Reason and benefit**

The maintained workflow needs auditable costs and traces, including
failures and missing receipts, before paid recurrence can be qualified.

## What Changed

- Add controlled mode as the manual default and the only scheduled mode.
Legacy mode remains explicit for other manual provider qualifications.
- Prepare the full request selection and one shared manifest. Bind
current run, first attempt, code, definitions, executable hashes and
exact profile allocations.
- Verify the copied controller and manifest before provider credentials.
Require the controlled target to equal the trusted workflow revision
before private archive access.
- Reuse private archive retention for the controller policy, cell
attempts and canonical report. AWS credentials remain transfer-step
outputs. Controlled raw evidence is never uploaded as an Actions
artifact.
- Replay a retained manifest with partial or empty worker evidence.
Count operating holds once and preserve known partial charges.
- Pin the controller to an explicitly reviewed compatible lockfile
digest. Reject resolver or byte drift before a frozen, script-free
install into a job-local store. Record the lock digest in the runtime
receipt and verify it again in private preparation.
- Build the portable controller without secrets, private environment
access or OIDC. Private preparation verifies that artifact and performs
no dependency install. OIDC jobs use no mutable dependency resolution or
shared cache.
- Add synthetic source/copied-runtime tests, executable workflow
admission tests, private archive tests and an operator runbook.

## Verification

- Controller, campaign, portable runtime and workflow tests: 104 passed.
This includes real aggregation from an isolated reporter without source
dependencies, safe diagnostics, driver drift, distinct evidence binding
and reviewed-lock drift. Real private Codex/Claude requests also pass
through an independently verified copied runtime with fake transports.
- Private transport and workflow contracts: 35 passed. Executable/static
workflow guards: 13 passed.
- Paired private report tests: 166 passed; Evalbook discovery: 7 passed.
- `pnpm -r typecheck` and `pnpm build`: passed.
- `pnpm test:e2e:runner:typecheck`, `actionlint
.github/workflows/runner-protocol-live-evals.yml` and `git diff
--check`: passed.
- `pnpm test:run` hit repeated local embedded PostgreSQL initialization
failures in unrelated DB-dependent server tests. Its owned process tree
was stopped after preserving the log. This is a local verification
limitation; current-head CI remains required.
- An earlier head hit one unchanged Rust cleanup assertion. The exact
test passed locally and the final head passed the full Rust lane; the
initial failure log is retained.
- On the final head, one unchanged chat idle-queue assertion failed once
in CI. Its single scoped retry passed. All test jobs and the final
aggregate gate passed on commit
`2e328fa53dd296174d59d19f14f068aeeb0b82d8` (CI run 40535692048, attempt
2).
- No paid model, Braintrust or cloud calls. No live variables, secrets,
schedules or resources changed.

## Risks

- Controlled mode fails closed until a reviewed
`RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON` and a compatible private
eval SHA exist. `all` does not silently become a supported subset.
- The reviewed policy must cover setup, model requests, trace/gateway
processing, archive transfer and retained storage. Hash binding does not
establish vendor billing bounds. This PR does not enable recurrence or
release historical holds.
- The controlled policy must pin `controllerLockSha256`; it never
accepts a resolver-generated digest automatically. No lockfile is
committed. A changed compatible lock needs review.
- Legacy browser verification now requires the committed frozen
lockfile. A stale lockfile fails closed; controlled preparation has no
dependency install.
- Workflow cancellation can prevent the final report. The full manifest
is retained before provider work and remains the starting point for
reconciliation.
- The workflow exceeds 1,000 lines because the shared shard anchor,
dependencies, environment expressions and credential boundaries belong
to one declarative job graph. Request preparation, accounting, archive
safety and runtime verification stay in separately tested modules.

## Model Used

OpenAI Codex, GPT-6-based coding agent with tool use and code execution.
The host does not expose an exact deployment model ID or context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the relevant tests locally and they pass; the broad
local suite limitation is documented above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green on the final head
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups;
the secondary reviewer is also 5/5
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-27 11:38:16 -05:00