Commit Graph
10 Commits
Author SHA1 Message Date
170e1b0c07 feat(runner): pin Codex 0.170.0 and Agent SDK 0.3.328 so Fable 5.2 and GPT-6.2 run (#22278)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude and Codex agents from one shared
provider pack. The pack pins an exact Claude Agent SDK (which bundles
Claude Code) and an exact Codex CLI.
> - #22222 added Claude Fable 5.2 (minimum Claude Code 2.1.321) and
GPT-6.2 Sol and Astra (minimum Codex 0.169.1). The pack still bundles
Claude Code 2.1.286 and Codex 0.160.0.
> - Because of this, Paperclip Runner stops these models with a named
version error, and native remote Codex accepts only `>=0.149.0
<0.161.0`.
> - This pull request moves the provider pack to Agent SDK 0.3.328
(Claude Code 2.1.328) and Codex 0.170.0. It moves every constant that
depends on these pins, and the remote Codex window to `<0.171.0`.
> - The benefit is that Fable 5.2 and GPT-6.2 run on Paperclip Runner
after a sandbox image with this pack is promoted.

## Linked Issues or Issue Description

Refs #22222 (adds the models and the version floors; this pull request
makes them runnable on Paperclip Runner).
Refs #18488 (moves only the Codex pin to 0.166.0 and the window to
`<0.167.0`; it is not updated since November 1 and has conflicts with
`master`).
Refs #18486 (GPT-6.1 Terra needs Codex 0.165.0; Codex 0.170.0 satisfies
it).

**Overlap with #18488.** #18488 changes the same Codex pin locations.
This pull request uses a newer pin (0.170.0) and also moves the Claude
runtime. If this pull request merges first, #18488 has no remaining
change. I did not change the #18488 branch.

## What Changed

- Claude runtime: `@anthropic-ai/claude-agent-sdk` 0.3.286 → 0.3.328
(Claude Code 2.1.286 → 2.1.328). This changes the root `pnpm.overrides`,
`pnpm-workspace.yaml`, the ACPX `claude` profile, and the qualified
Claude runtime in `installation-integrity.ts`, with new executable
digests for Linux x64, Linux ARM64, macOS ARM64, and macOS x64.
- Codex runtime: `@openai/codex` 0.160.0 → 0.170.0. This changes the
runner dependency, the `codex-acp@1.6.2>@openai/codex` override, the
`codex-acp` patch dependency line, the ACPX `codex` profile, and the
qualified Linux x64 runtime with its new executable digest.
- The ACP bridges do not change: `claude-agent-acp` 0.73.0 and the
patched `codex-acp` 1.6.2. Their `commandDigest` values and profile
versions do not change.
- Generated declarations: `generated-profiles.ts` and
`generated_acpx_profiles.rs` are regenerated with
`generate:acpx-profiles`.
- Remote native Codex window: `REMOTE_CODEX_MAXIMUM_VERSION_EXCLUSIVE`
0.161.0 → 0.171.0. The minimum stays at 0.149.0. Images with Codex
0.149.0 through 0.170.x are accepted.
- Codex company observability: the qualified-version check moves to
`codex-cli 0.170.0`. The `otel` configuration keys that Paperclip writes
(`exporter`, `trace_exporter`, `metrics_exporter`) are present in the
0.170.0 configuration schema. The `OtelExporterKind`,
`OtelHttpProtocol`, and `OtelTlsConfig` definitions are identical to
0.160.0.
- Daytona runner image probes (`Dockerfile`, `Dockerfile.release`)
expect `codex-cli 0.170.0` and `2.1.328 (Claude Code)`. The remote npm
install spec in `server/src/adapters/registry.ts` is
`@openai/codex@0.170.0`.
- Tests: fixtures and assertions that mirror the pins use the new
versions. The window tests accept 0.161.0 through 0.170.99 and reject
0.171.0. The ACPX gate tests now show that Fable 5.2 and GPT-6.2 are
admitted. A new test mocks an older bundled runtime and keeps the named
version-gap errors covered. The fallback test shows that GPT-6.2 stays
on 0.170.0 and still falls back on 0.160.0.
- Opt-in qualification scripts
(`scripts/tests/native-cleanup-paginated-codex.mjs`,
`native-provider-startup-codex.mjs`) require Codex 0.170.0.
- Documentation: `docker/daytona-runner/README.md`,
`doc/architecture/paperclip-runner-compatibility.md`,
`doc/agent-observability.md`, `docs/deploy/environment-variables.md`,
and the deferred item in `doc/adapter-model-audit-2026-11-18.md`.
- `pnpm-lock.yaml` is not part of this pull request. The refresh bot
regenerates it on `master`, as for the previous pin refreshes (#14942,
#14993).

## Verification

Sources, checked on 2026-11-19:

- npm registry: `@openai/codex@0.170.0` (published 2026-11-18T03:45Z)
and `@anthropic-ai/claude-agent-sdk@0.3.328` (published
2026-11-18T23:13Z, `claudeCodeVersion` 2.1.328). Both are the `latest`
tags.
- [Codex releases](https://github.com/openai/codex/releases) 0.161.0
through 0.170.0. The notes record no app-server protocol break. 0.169.0
keeps project configuration disabled for `codex exec` in untrusted
directories; Paperclip's CLI path already uses `--skip-git-repo-check`,
and the native path uses `app-server`.
- [Claude Code
changelog](https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md)
2.1.287 through 2.1.328. See "Risks" for the headless changes that I
reviewed.

Digests and package shape:

- I downloaded each platform archive with `npm pack`, which verifies
registry integrity, and computed the SHA-256 of the executable. As a
method check, the same command on the 0.3.286 Linux x64 archive gives
the digest that is on `master` (`fe503f65…`). The Codex 0.160.0 archive
also gives the `master` digest (`12eb3e81…`).
- The extracted executables report `codex-cli 0.170.0` and `2.1.328
(Claude Code)`.
- The Codex 0.170.0 Linux x64 archive no longer contains
`codex-resources/zsh/bin/zsh`. Paperclip does not reference that path.
- The Agent SDK 0.3.328 `optionalDependencies` have the same platform
set as 0.3.286. The Codex 0.170.0 `optionalDependencies` use the same
`npm:@openai/codex@<version>-<platform>` aliases.
- Bridge link check: I installed
`@agentclientprotocol/claude-agent-acp@0.73.0` with an override to Agent
SDK 0.3.328 and imported its entry module. Node linked all of the named
SDK imports, and the import completed.

Real Codex qualification (no model turn, no credentials):

- `PAPERCLIP_TEST_CODEX_BINARY=<installed 0.170.0 Linux x64 binary> node
--import ./server/node_modules/tsx/dist/loader.mjs
scripts/tests/native-cleanup-paginated-codex.mjs` passed: `green: true`,
with the expected red control and paginated `thread/resume` through the
real 0.170.0 `app-server`. The installed binary digest matches the
pinned digest.

Focused suites (Linux x64, Node 25.9.0, local `pnpm install
--no-frozen-lockfile`):

- `cd packages/paperclip-runner && pnpm run typecheck:typescript`
passed, including `check:acpx-profiles`.
- `npx tsc --noEmit` passed in `packages/adapters/codex-local` and
`packages/adapters/claude-local`.
- `packages/adapters/codex-local`: 625 passed.
`packages/adapters/claude-local`: 419 passed, 1 skipped.
- `node --test`: `acpx-codex-package-contract` 9 passed,
`controlled-evals/controller` 77 passed, `scripts/acpx-patch-packaging`
26 passed, `scripts/__tests__/grok-public-install-sandbox` 27 passed.
- `server`: `acpx-qualification`, `codex-runtime-compatibility`, and
`codex-model-fallback` 110 passed. `native-session-executor` 1541
passed, 1 failed. The failure (`compares export process compatibility …
db.transaction is not a function`) also fails on unchanged `master` on
this host.
- `packages/paperclip-runner` vitest (`installation-integrity`,
`qualified-profiles`, `codex-security-config`, `native-execution`,
`native-backend-factory`): 283 passed, 25 failed. All 25 failures are in
`installation-integrity.test.ts` with "ACPX provider lifetime guardian
failed to start". The same 25 tests fail on unchanged `master` on this
host (master fails 30 here).

Rust (toolchain 1.97.1 from `rust-toolchain.toml`, Linux x64):

- `cargo test -p paperclip-runner-core --lib`: 658 passed, 2 failed, 4
ignored. `warm_run_attachment_reuses_the_provider_ingress_listener` also
fails on unchanged `master` on this host.
`reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job` is
flaky here: on this branch it passed 2 of 3 reruns.
- `cargo test -p paperclip-runner-core --test acpx_warm_attachment
--test native_provider_backend`: 7 and 45 passed.
- The new and changed saved-session tests pass:
`a_session_saved_by_the_previous_agent_runtime_is_refused_by_name`,
`unrestorable_saved_sessions_keep_strict_bindings_and_name_their_class`,
and `a_settled_session_saved_by_the_previous_acpx_release_continues`.
`rustfmt --check` passes.
- `server`: `native-session-resume.test.ts` and
`acpx-qualification.test.ts` 110 passed, 2 skipped. Both new resume
cases run and pass.

Not run here:

- `native-provider-startup-codex.mjs`, because it needs a compiled
`paperclip-runnerd`. The trusted CI runner workflow covers startup and
session-resume probes.
- The Docker image build and any live model turn. A live Fable 5.2 and
GPT-6.2 test follows the staging image rollout.

## Saved sessions from the previous release

This change keeps the ACPX bridge command digests and profile versions.
It changes `agentRuntimeVersion` in the `claude` and `codex` ACPX
profiles. That field is part of the saved session identity. The previous
release saved sessions with Agent SDK 0.3.286 or Codex 0.160.0. This
build refuses those saved sessions. It does not migrate them.

- **Next run (the usual case):** the saved provider descriptor names the
old runtime. `rebindNativeSessionCheckpoint` does not match it, so the
next run starts a fresh provider session in the same workspace with the
complete handoff prompt. No run fails. The earlier provider conversation
does not carry over.
- **A run in flight at the deploy (settled or mid-turn):** the runner
restore refuses the saved descriptor with the permanent class
`provider_state_incompatible`. The server reports it as
`native_provider_state_incompatible` and stops automatic retries. The
runner opens no provider and keeps the saved state bytes and the
workspace. The next run on the task starts a fresh provider session.
- **Native Codex (`codex_app_server`) and OpenCode** sessions do not
record this field, so their resume decision does not change. After a new
image is promoted, native Codex resumes a saved thread with Codex
0.170.0 instead of 0.160.0. This pull request does not test that
cross-version thread resume.

New tests:
- `acpx_provider_backend.rs`:
`a_session_saved_by_the_previous_agent_runtime_is_refused_by_name`
covers settled Claude and Codex sessions and a Claude turn interrupted
mid-run by the deploy.
`unrestorable_saved_sessions_keep_strict_bindings_and_name_their_class`
has a new `agent_runtime_release` case. Each case checks the named
class, that no provider session opens, and that the saved bytes do not
change.
- `native-session-resume.test.ts`: "a session saved by the previous
Claude or Codex ACPX runtime" checks that a same-release session
resumes, and that a session saved with Agent SDK 0.3.286 or Codex
0.160.0 starts fresh in the same workspace.

## Risks

- Medium. Every Paperclip Runner Claude and Codex run uses the new
executables. The digests, generated profiles, image probes, and window
must agree. They agree locally, and CI runs the Rust and runner suites
again.
- Claude Code 2.1.287 through 2.1.328 change some headless defaults. I
checked each one against Paperclip:
- 2.1.302: `-p` and Agent SDK sessions with no permission mode start in
`default` mode on third-party providers. Paperclip always sets a mode
(`bypassPermissions` or `default`), so this does not apply.
- 2.1.299: MCP allow rules now match only the whole server name.
Paperclip already writes `mcp__<server>__*` and exact
`mcp__paperclip__<tool>` rules.
- 2.1.302: fork subagents are on by default in `-p` and the Agent SDK,
as in the terminal. This pull request keeps the upstream default and
does not set `CLAUDE_CODE_FORK_SUBAGENT=0`. Reviewers can ask to keep
them off.
- 2.1.299 and later: stdio MCP connections negotiate MCP protocol
2026-07-28 first and fall back to the older protocol for servers that do
not answer.
- Agent SDK 0.3.328 is less than one day old. If a regression appears,
revert this pull request to return to 0.3.286 and Codex 0.160.0.
- Sandbox images built before this change still carry Codex 0.160.0 and
Claude Code 2.1.286. They stay accepted. On those images, Fable 5.2 and
GPT-6.2 keep the existing named errors and fallbacks until a new image
is promoted.
- Until the refresh bot updates `pnpm-lock.yaml` on `master`, a local
`--frozen-lockfile` install fails, as with the previous pin refreshes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude, model ID `claude-opus-5-5`, through Claude Code with
tool use (shell, file edit, `gh`) and code execution. The context window
and reasoning mode are not reported in this session.
- The registry archives, digests, release notes, configuration schemas,
and changelog were read directly. No model output was used as a source
for those facts.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 20:45:18 -08:00
DottaandPaperclip 6b9e5f3a80 fix(evals): enforce qualified ACPX profile admission before paid turns (#17964)
## Thinking Path

> - Paperclip coordinates agent work through Runner.
> - Controlled evals route model calls through a budget gateway.
> - The definitions pin each ACPX runtime identity.
> - A stale qualified profile currently reaches a paid turn before
scoring rejects it.
> - The Runner CLI already checks an exact expected profile before it
creates a provider.
> - This change forwards that existing admission argument through the
controller.
> - Profile drift can then stop before model spending.

## Linked Issues or Issue Description

Refs #17943 and #17929.

**What happened?**
The worker in run 40618022700 passed all four behavior checks and
retained three final request receipts. Its grader then rejected the old
Claude SDK pin. The controlled wrapper could not pass the existing
`--expected-acpx-profile` argument to the built CLI.

**What did you expect to happen?**
The built CLI must reject a stale expected profile before provider
creation.

**Steps to reproduce**
Run a controlled ACPX cell with a definition that pins a different SDK
from the built Runner. The old wrapper accepts only request and output
arguments. It cannot enforce the profile admission supplied by updated
definitions.

**Paperclip version**
The observed run used e4b30163bb.

**Deployment mode**
The controlled GitHub Actions Runner protocol workflow.

## What Changed

- Forward bounded expected-profile JSON through the controlled wrapper
to the same built CLI.
- Reject invalid objects, oversized arguments, and profiles on non-ACPX
requests before opening evidence.
- Exercise a real child boundary and the actual qualified Claude CLI
admission gate.
- Document the definition/controller compatibility requirement.

## Verification

- Controller suite: 77 tests passed. The new regressions fail against
the original wrapper.
- Hosted and portable controller suites: 8 tests passed.
- Actual eval-session CLI contract suite: 39 tests passed. A stale
qualified Claude SDK never creates a provider service.
- Runner package typecheck passed.
- The non-ACPX rejection guard also passed its focused regression after
review.
- No live provider request was made for this change.
- Full repository checks and current-head reviews run on this PR.

## Risks

- Updated definitions require this wrapper before they can pass the new
argument. Merge the controller first.
- Profile identity remains owned by the built Runner CLI. The controller
only forwards its existing argument.
- Existing callers that omit the argument retain their current behavior.
- Optional review suggestions to duplicate the expected-profile hash
into controller evidence and add a full built-CLI/controller integration
case are deferred. The pinned definition revision retains the expected
profile, and both controller entrypoints plus the actual CLI admission
boundary are covered. A compatibility error fails closed without a paid
fallback.

## Model Used

OpenAI Codex, GPT-6 family, with tool execution and extended reasoning.
The exact serving model ID and context window are not exposed in this
session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 21:59:20 -05:00
DottaandPaperclip 35820cf398 fix(evals): qualify Runner model receipts with separate operating reconciliation (#17929)
## Thinking Path

> - Paperclip must account for paid eval work.
> - The maintained Runner workflow distributes one immutable controller
manifest.
> - Its cost gate currently requires vendor proofs for every operating
charge.
> - Model-spend qualification must also work under an explicit policy
that keeps operating charges separate.
> - This PR applies the shared model-readiness policy to the hosted
controller.
> - It retains every operating hold and still rejects missing model
receipts and incomplete workers.

## Linked Issues or Issue Description

Refs #17928, #17889.

**What happened?**
Runner qualification requires full-lifecycle vendor operating proofs
even when the operator selects model-request readiness. That blocks the
model-token qualification path.

**Expected behavior**
An explicit `overhead.accountingReadiness: "model_requests"` policy
permits absent operating cost proofs. Supplied proofs still validate.
Complete final model receipts are required.

**Steps to reproduce**
Prepare a two-worker hosted manifest with the explicit policy and no
vendor cost proofs. Preserve missing-worker allocations, then complete
both workers and inspect the private archive summary. A second test
returns a provider refusal without terminal usage and requires failure
with the full hold.

## What Changed

- Apply the shared explicit policy to hosted capacity and observability
cost-proof admission.
- Preserve pricing windows, model bounds, immutable
source/request/runtime identities, trace quotas and every allocation.
- Reject zero or unresolved model requests in this policy, even if a
request has a provisional bound.
- Retain operating amounts and pending reconciliation through campaign
and private archive summaries.
- Document the separate model-readiness contract. Preserve strict
behavior when the field is absent.

## Verification

- Regression first failed at the original vendor-bound gate.
- All 80 controlled-eval Node checks pass, including source and copied
portable runtime checks, archived partial/missing workers, credential
exclusion and model receipt failure.
- Product readiness prerequisite #17928 has its own eval unit/typecheck
and full CI checks.
- No paid workflow, credential change or recurring activation is part of
this source PR.

## Risks

This opt-in policy does not prove a full-lifecycle operating-cost
ceiling. The report retains operating holds and pending reconciliation.
Default policies retain the existing vendor-proof gate. Source and
worker identity checks remain required. A model refusal without final
usage now fails readiness even if its reservation has a reviewed bound.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection and
code execution. The exact serving model ID and context window are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 15:41:07 -05:00
DottaandPaperclip 8548173fc9 feat(evals): connect hosted Runner costs and traces (#17889)
## Thinking Path

> - Paperclip manages AI agents and measures their behavior with Runner
evaluations.
> - The cost controller, native tracing, private receipts and portable
runtime are on master.
> - The maintained live workflow still launches the provider CLI
directly.
> - Independent workers need one shared selection and disjoint budget
allocations.
> - Reports must keep unresolved allocations when a worker produces no
evidence.
> - This PR connects the maintained workflow to the existing controller
and private archive.
> - Operators can inspect known costs, conservative holds and trace
evidence before qualifying paid recurrence.

## Linked Issues or Issue Description

Refs #17452, #17469, #17493, #17506, #17523.

**Current behavior**

The live Runner workflow does not use the merged cost/tracing
controller. Attempt-only aggregation cannot reconstruct a campaign when
every worker archive is absent. Ordinary Actions artifacts in this
public repository are readable by authenticated public GitHub users.

**Proposed behavior**

Controlled runs prepare one manifest, bind every selected request,
enforce disjoint allocations and retain evidence in private storage. The
report starts from the full retained manifest. Missing workers keep
their allocations. The workflow rejects retries, unsupported providers,
changed requests and absent or stale operating bounds before provider
work.

**Reason and benefit**

The maintained workflow needs auditable costs and traces, including
failures and missing receipts, before paid recurrence can be qualified.

## What Changed

- Add controlled mode as the manual default and the only scheduled mode.
Legacy mode remains explicit for other manual provider qualifications.
- Prepare the full request selection and one shared manifest. Bind
current run, first attempt, code, definitions, executable hashes and
exact profile allocations.
- Verify the copied controller and manifest before provider credentials.
Require the controlled target to equal the trusted workflow revision
before private archive access.
- Reuse private archive retention for the controller policy, cell
attempts and canonical report. AWS credentials remain transfer-step
outputs. Controlled raw evidence is never uploaded as an Actions
artifact.
- Replay a retained manifest with partial or empty worker evidence.
Count operating holds once and preserve known partial charges.
- Pin the controller to an explicitly reviewed compatible lockfile
digest. Reject resolver or byte drift before a frozen, script-free
install into a job-local store. Record the lock digest in the runtime
receipt and verify it again in private preparation.
- Build the portable controller without secrets, private environment
access or OIDC. Private preparation verifies that artifact and performs
no dependency install. OIDC jobs use no mutable dependency resolution or
shared cache.
- Add synthetic source/copied-runtime tests, executable workflow
admission tests, private archive tests and an operator runbook.

## Verification

- Controller, campaign, portable runtime and workflow tests: 104 passed.
This includes real aggregation from an isolated reporter without source
dependencies, safe diagnostics, driver drift, distinct evidence binding
and reviewed-lock drift. Real private Codex/Claude requests also pass
through an independently verified copied runtime with fake transports.
- Private transport and workflow contracts: 35 passed. Executable/static
workflow guards: 13 passed.
- Paired private report tests: 166 passed; Evalbook discovery: 7 passed.
- `pnpm -r typecheck` and `pnpm build`: passed.
- `pnpm test:e2e:runner:typecheck`, `actionlint
.github/workflows/runner-protocol-live-evals.yml` and `git diff
--check`: passed.
- `pnpm test:run` hit repeated local embedded PostgreSQL initialization
failures in unrelated DB-dependent server tests. Its owned process tree
was stopped after preserving the log. This is a local verification
limitation; current-head CI remains required.
- An earlier head hit one unchanged Rust cleanup assertion. The exact
test passed locally and the final head passed the full Rust lane; the
initial failure log is retained.
- On the final head, one unchanged chat idle-queue assertion failed once
in CI. Its single scoped retry passed. All test jobs and the final
aggregate gate passed on commit
`2e328fa53dd296174d59d19f14f068aeeb0b82d8` (CI run 40535692048, attempt
2).
- No paid model, Braintrust or cloud calls. No live variables, secrets,
schedules or resources changed.

## Risks

- Controlled mode fails closed until a reviewed
`RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON` and a compatible private
eval SHA exist. `all` does not silently become a supported subset.
- The reviewed policy must cover setup, model requests, trace/gateway
processing, archive transfer and retained storage. Hash binding does not
establish vendor billing bounds. This PR does not enable recurrence or
release historical holds.
- The controlled policy must pin `controllerLockSha256`; it never
accepts a resolver-generated digest automatically. No lockfile is
committed. A changed compatible lock needs review.
- Legacy browser verification now requires the committed frozen
lockfile. A stale lockfile fails closed; controlled preparation has no
dependency install.
- Workflow cancellation can prevent the final report. The full manifest
is retained before provider work and remains the starting point for
reconciliation.
- The workflow exceeds 1,000 lines because the shared shard anchor,
dependencies, environment expressions and credential boundaries belong
to one declarative job graph. Request preparation, accounting, archive
safety and runtime verification stay in separately tested modules.

## Model Used

OpenAI Codex, GPT-6-based coding agent with tool use and code execution.
The host does not expose an exact deployment model ID or context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the relevant tests locally and they pass; the broad
local suite limitation is documented above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green on the final head
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups;
the secondary reviewer is also 5/5
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 11:38:16 -05:00
DottaandPaperclip 507dc70be2 fix(evals): bind hosted controller admission to the first attempt (#17523)
Bind hosted controller admission to the repository, GitHub run, first attempt, workflow revision and selected execution before credential reads or provider launch. Keep offline evidence replay independent of the current worker environment. Preserve the single-project boundary editor and refresh project-access data with agent-detail updates.

Verified head: 1e588f75fe.
Validation: 74 synthetic source/portable controller tests, 98 focused UI tests and token gates passed before the ancestry-only final sync; four hosted/portable regressions pass again with unchanged eval and UI repair trees. Final exact-head full CI passed (40387856225), and Greptile reviewed this head at 5/5 with zero unresolved findings. The optional secondary Paperclip Review is paused and not counted as completed. No paid work or schedule changes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-26 14:43:58 -05:00
DottaandPaperclip 8c702d59db feat(evals): package a portable Runner controller runtime (#17506)
Package a verified portable Runner eval controller with complete bounded file receipts, copied dependency closure, private evidence replay and symlink-safe CLI entrypoints. Preserve fail-closed cost holds, native trace receipts and all parent security repairs.

Verified head: 1c892ff464.
Validation: 72 synthetic source/portable tests pass after current-master sync. Full current-head CI passed (40386180906); Greptile reviewed this exact head at5/5, with zero unresolved findings.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-26 14:32:46 -05:00
DottaandPaperclip 313c1db2c3 fix(evals): share bounded request policy with qualification (#17765)
## Thinking Path

> - Paperclip manages agents and their work.
> - Product E2E evaluations send model work through a gateway with cost
limits.
> - The gateway checks the request body before it reserves money or
contacts a provider.
> - A qualification probe needs that same check before it records a
spending-cap denial.
> - Namespace support already exists. Qualification must share that
policy without sending a paid request.
> - This pull request exposes the pure request validator and bounds the
supported local tool definitions.
> - The benefit is one request policy for qualification and forwarding.

## Linked Issues or Issue Description

Refs: #17452, #17469.

**What happened?**

The request validator was private to `forwardEvalRequest`. A
qualification caller could not reuse the request policy before
admission. A separate check could accept a request body that forwarding
rejects before a reservation.

**Expected behavior**

Qualification and forwarding use the same pure validator. One-level
OpenAI namespaces with local function or custom tools remain supported.
Invalid tool definitions fail before accounting or transport work.

**Steps to reproduce**

Use the synthetic native-daemon request fixture in
`eval-gateway.test.ts`. It has seven local function tools and one
namespace with five local function tools. Call the shared validator,
then forward through the fake provider. Run the malformed namespace and
tool-count cases to check rejection before side effects.

The namespace behavior follows the [OpenAI function-calling
guide](https://developers.openai.com/api/docs/guides/function-calling).
This change preserves the one-level contract introduced in #17452 and
hardened in #17469.

## What Changed

- Export `boundedEvalPayload` with credential-free route fields.
`forwardEvalRequest` calls that function.
- Keep one namespace level with function or custom leaves. Require exact
leaf types, nonblank string leaf names, and a closed namespace wrapper.
- Limit OpenAI tool definitions to 1,024 per request. Count namespace
wrappers and their children.
- Add native-daemon-shaped forwarding coverage, malformed and hosted
tool rejection, count and depth boundaries, and Anthropic compatibility
checks.
- Document the shared qualification policy in the Product E2E security
guide.
- Compile the real observability validator in the test parent. The owned
CLI fixture imports JavaScript, so its process group does not include a
test-only TypeScript compiler service. The parent resolves the compiler
from the root package, which declares that dependency. Production
cleanup checks and attempt deadlines stay intact.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
eval-gateway.test.ts cost-ledger.test.ts provider-usage.test.ts
operating-overhead.test.ts operations.test.ts`: 173 passed.
- `pnpm test:e2e:runner:typecheck`: passed.
- `pnpm -r typecheck`: passed.
- `pnpm build`: passed.
- Separate Node support checks: 137 passed.
- `pnpm test:e2e:runner:unit`: 4,271 passed, two skipped, four failed in
unchanged Daytona image and Pi catalog contracts. The same four failures
reproduce on untouched base commit
`a67a18c171816b097f46eae78fc58c7e55c9c565` (79 passed, four failed in
those files).
- `pnpm test:run`: interrupted and incomplete after required CI
completed. Initial database startup failed because this checkout omitted
the embedded PostgreSQL package postinstall aliases. The package
hydrator restored them, and 14 real database checks passed. Earlier
failures and the full output remain preserved; this local full run is
not counted as passing verification. Its owned process group exited
after SIGINT, with no owned survivors.
- An offline full native authentication probe used the real Rust daemon
and pinned Codex 0.160.0 Darwin SDK. The same captured request failed
the older operations policy before reservation and reached a mock
reservation with this policy. The OS sandbox allowed loopback only. It
used synthetic credentials, retained no raw model payload, and made no
provider calls.
- Controller fixture suite: 70 passed. Three cold comparisons each
confirmed that the old child started an owned compiler service and the
corrected child did not. On Darwin the old compiler also exited cleanly,
so this does not prove the exact historical Linux process race. Fresh
Linux CI is required.
- Independent review passed. The private qualification integration
passed 14 exact-source compatibility checks and 67 shared-source tests
without provider calls.

- Required GitHub CI checks passed on the current head. The separate
Entire approval gate still awaits an eligible reviewer; the delegated
author account's explicitly AI-assisted approval does not satisfy it.

## Risks

Requests with more than 1,024 OpenAI tool definitions or extra namespace
wrapper fields now fail closed. New provider schema fields will need
review. Anthropic tool handling, model prices, billing restrictions,
output caps, and reservation ordering remain unchanged. This offline
request-policy proof does not establish a completed model turn or Linux
or Cloud onboarding acceptance.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model identifier and
context window are not exposed in this session. The model used
reasoning, tools, and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-26 14:14:47 -05:00
DottaandPaperclip bbe1016d81 feat(evals): retain bounded native tracing in the Runner controller (#17493)
Retain bounded native tracing in the Runner controller with attempt-bound relays, private export receipts, offline cost replay, conservative holds and bounded failure diagnostics.

Verified exact head 5cb991421e: full hosted CI, fresh Greptile 5/5, zero unresolved threads, 70 local controller tests including actual Codex/Claude CLI observability validators. Secondary review has no must-fix or should-fix findings and records optional coverage follow-ups. No paid evals or schedule changes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-26 10:22:42 -05:00
DottaandPaperclip 49bd7a4ba6 feat(evals): retain portable Runner controller evidence (#17469)
Retain private Runner request evidence for offline cost reconciliation. Bound post-child reads, reject hidden paid tools, exclude configured credentials, bind incomplete results, and replace redacted proof atomically.

Verified at f5c1791623: full hosted CI, Greptile 5/5, secondary review summary 5/5, no unresolved threads. No paid evals or schedule changes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-26 09:50:58 -05:00
DottaandPaperclip 41c42895fa feat(evals): enforce fixed Runner request allocations (#17452)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner evals execute real provider harnesses against maintained
fixtures.
> - Turn-level usage arrives after requests have already spent money.
> - Parallel cells need distinct allowances before requests start.
> - Missing output must preserve the remaining allowance and any known
charges.
> - This pull request adds a controller that reuses the existing request
ledger and gateway.
> - It makes a bounded local Runner attempt reviewable before hosted
scheduling is enabled.

## Linked Issues or Issue Description

Refs #17440. That PR added the explicit Runner gateway route. This PR
owns request admission and retained campaign accounting for that route.

The existing hosted matrix is not enabled or changed here. Its global
allocation, rerun rejection, and private evidence archive remain
separate integration work.

## What Changed

- Validate a trusted manifest with exact request, CLI, package, daemon,
model, and source identities. Assign each selected cell a distinct
allowance. Count operating overhead once. Reject allocations above the
campaign limit before launch.
- Run native Codex or ACPX Claude through the shared request proxy and
append-only ledger. Pass only the proxy credential to the child. Refuse
local attempt reuse and preserve existing output files.
- Close gateway admission on cancellation. Retain the ledger, receipts,
route descriptor, child result, and cleanup evidence outside the
disposable workspace.
- Recompute campaign accounting from retained ledgers. Missing or
invalid final evidence holds the full cell allowance while preserving
known charges. Reports distinguish provisional and incomplete costs.
- Accept namespaces containing only local function/custom tools. Disable
web search in the controlled Codex profile. Continue to reject hosted
tools, including tools hidden inside a namespace.
- Add controller tests to the existing Runner static test path and
document this source-checkout entry point.

## Verification

- 20 controller tests pass. They use real child processes, local HTTP,
and the actual ledger. Cases cover reservations before dispatch,
concurrent cell isolation, caps, cancellation, replay rejection, lost
artifacts, changed receipts, and Braintrust Claude routing. Upstream
responses are synthetic.
- 52 shared gateway tests and 55 Runner CLI tests pass. The namespace
acceptance and web-search configuration regressions failed before the
repair and pass afterward.
- Runner TypeScript/Rust checks, Runner build, shared E2E typecheck, and
`git diff --check` pass.
- The first actual-harness probe found that Codex advertised both a
local tool namespace and web search. It reached no upstream transport.
That failure is retained; the compatibility repair does not allow paid
hosted tools.
- An earlier test failure showed that separate dynamic imports created
different admission-error classes. The shared runtime bridge now loads
one module graph. Another regression showed that a missing final report
hid known ledger charges; the collector now preserves those charges.
- The repaired actual-harness probe passed its control checks for native
Codex and ACPX Claude: each reached exactly one synthetic upstream
request after the ledger reservation; both retained the unresolved hold
and route evidence; both recorded clean process exit. A scan of 41
retained files found no synthetic upstream credential. Deliberate
provider rejection preserved the failed Codex turn and Claude timeout;
these are not semantic passes. Prices, operating bounds, and the package
archive identity in this probe were synthetic.
- Review regressions reproduced lost request/receipt evidence, lost
admission metadata, executable replacement after verification, and a
SIGTERM-resistant descendant surviving the CLI. The repairs now pass.
Executable bytes are staged before launch; a trusted Node resolution
hook preserves relative imports. A symlink-path regression also failed
before its repair and now passes. The final staged native probe repeated
both one-request checks successfully.
- Full current-head
[CI](https://github.com/paperclipai/paperclip/actions/runs/40115932485)
passed on `4781487215116afd10a415d4db53b1ec8e004dd5`: all 58 check runs
and the legacy status are successful, neutral, or skipped. Greptile and
Paperclip both reviewed this commit at 5/5. All five finding threads are
fixed and resolved. No paid provider request ran during this change.

## Risks

- This is a trusted source-checkout controller, not a published package
command. It needs the repository's installed shared cost dependencies.
- The manifest declares reviewed prices and full-lifecycle bounds.
Synthetic tests do not qualify real prices, invoices, storage, or
Braintrust metering.
- The attempt-directory guard is local. Hosted fanout still needs unique
global allocations, rerun rejection, and durable private retention. Do
not enable the existing matrix on this change alone.
- Forced child termination can leave cleanup unverified. That attempt
remains incomplete and holds its full allocation. These checks do not
claim a host security boundary against an adversarial target.
- The current private publisher does not archive the separate controller
directory. Native trace relay, private report integration, and live
qualification remain open.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, and code
execution. This runtime does not expose the exact model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-26 07:53:57 -05:00