Commit Graph
17 Commits
Author SHA1 Message Date
DottaandPaperclip 039e0dad2c Connect TestOps protocol plans to verified GitHub attempts (#18985)
## Thinking Path

> - Paperclip lets people coordinate agent work through a governed
control plane.
> - Runner protocol evals test real provider sessions against a seeded
mock control plane.
> - The saved TestOps protocol plan has no GitHub executor. Its
dashboard shows historical results.
> - A current one-case attempt also failed before any tool call or
provider output. Its retained public-safe snapshot withheld the error
cause.
> - This pull request connects selected cases to the existing trusted
workflow and retains closed operator failure facts.
> - Each selected case receives a current result. Missing evidence
remains broken. Existing assertions and machine grades remain unchanged.

## Linked Issues or Issue Description

**Current behavior**

The saved protocol plan cannot start a GitHub qualification attempt.
Historical dashboard results can be mistaken for current execution. A
failed provider turn can also lose its failure category when the live
snapshot applies public evidence redaction.

**Proposed behavior**

The existing workflow accepts TestOps service inputs, validates exact
case identities before provider work, runs the selected cases, and
verifies their published statuses. Eval artifacts retain only validated
failure phase, category, HTTP status, and retryability.

**Reason and benefit**

Operators can run the standing plan from TestOps and inspect current,
source-bound results. A targeted pass cannot stand in for full-plan
coverage. A failed startup can be diagnosed without publishing provider
messages or request identifiers.

## What Changed

- Add optional TestOps selection and protected reporting jobs to the
existing Runner protocol workflow.
- Keep the current actor, source, provider credential, and controlled
accounting gates. TestOps retains the explicitly selected controlled or
legacy mode and a separate 41-case Claude roster. Controlled mode
preserves its existing reviewed policy, immutable bindings, tracing and
zero-retry admission. Its paid workers explicitly use GitHub-hosted
runners, matching the current manifest admission contract even when the
legacy AWS fleet is enabled.
- Bind the exact standing plan, workflow, job run, source revisions,
selected cases, and saved statuses. Accept only verified workflow URL
aliases.
- Create a closed Allure projection inside the report job. Transfer only
fixed case identities, check verdicts, numeric usage and statuses to the
reporting environment. Controlled raw/chat/controller evidence remains
private.
- Publish one result for every selected case. Retain missing
execution/report evidence as broken. Reject unsupported passes, model
mismatch and source drift. Preserve the evidence schema literal, stop
publication on cancellation, and retain closed API error codes/statuses.
- Tag full selections separately from targeted attempts. Add bounded
ingestion observation. Apply the exact TestOps filter before the
unchanged controlled provider/cell-capacity check, so a one-cell
registration fits a one-cell reviewed policy.
- Retain up to eight eval-only closed provider failure records before
board evidence redaction. Forward the original notifications unchanged.
- Add selection, reporting, privacy, and admission regression controls.
Document scope and remaining acceptance.

## Verification

- Direct Node protocol publication, projection, metrics, and security
tests: 56 passed. Focused controls exercise both mode admissions,
missing controlled policy/retries/source rejection, closed result
transfer, exact metadata, and missing-result handling.
- `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git
diff --check` passed.
- A proposed one-cell controlled policy passed the actual hosted
validator with a synthetic prepared request and zero provider calls. The
actual 41-case catalog filter selected case194/get-task-context and the
real hosted admission accepted it under maxCells1. Negative controls
rejected the unfiltered41, unsupported provider, source, selection,
origin, expiry and lock drift. This does not verify provider auth, a
runtime build or TestOps publication; final merged source/lock binding
is still required.
- The actual diagnostic TypeScript module passed positive, negative,
secret-exclusion, and unchanged-notification controls through the
existing tsx runtime. A local actual-CLI startup serializer probe passed
with closed facts, secret exclusion, exit status and read-only fixture
cleanup. This used an isolated resolver into the existing dependency
cache; canonical exact-dependency CI remains required.
- Prior-head CI passed typecheck, Runner static/Rust/native-server
checks and 4946 Runner package tests. One new serializer fixture failed
during Linux read-only directory cleanup; this head applies the existing
neighboring cleanup pattern. Unrelated chat E2E and CLI import timeouts
remain retained. Latest-head required gates are pending.
- Original current-source one-case attempt:
https://github.com/paperclipai/paperclip/actions/runs/41860234840 . It
failed with no tool calls or output. That failure remains retained and
is not a passing qualification claim.
- Diagnostic candidate attempt:
https://github.com/paperclipai/paperclip/actions/runs/41863410602 . It
retained the new closed field but still failed before output with
category unknown. This disproves the claim that typed retention alone
identifies the cause.
- The earlier passing run
https://github.com/paperclipai/paperclip/actions/runs/40627274706 used
controlled Braintrust routing. Both current failures used direct legacy
credentials, a different auth path. Next acceptance is one explicitly
controlled case with a reviewed current policy, followed by TestOps
registration and a fresh full plan. None is claimed complete by this PR
body.

## Risks

The TestOps contract currently binds the existing 41 cases and one
recorded Claude Sonnet 5 profile. Catalog changes fail closed until
reviewed. This does not qualify other providers. Admission and execution
qualify only the selected accounting mode; a controlled pass does not
prove direct credential health. Model-request readiness retains pending
operating reconciliation. TestOps ingestion can fail independently;
persisted results are checked, and missing evidence is never accepted as
a pass. The new failure field contains closed facts only; raw messages,
identities, credentials, and tool payloads are excluded.

## Model Used

OpenAI Codex, GPT-6. The exact hosted model ID and context window are
not exposed in this session. Used reasoning, repository inspection, code
editing, and tool execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and the available capability
details
- [x] I have checked ROADMAP.md and confirmed this does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs
- [x] I have described the issue with the enhancement template field
labels
- [x] I have not referenced internal or instance-local Paperclip issues
- [x] My branch name describes the change and contains no internal
ticket ID
- [x] I have run the available targeted local checks and reported the
remaining checks
- [x] I have added relevant positive and negative regression controls
- [x] I have updated the authoritative protocol documentation
- [x] I have considered and documented the risks
- [ ] All required Paperclip CI gates are green on the latest head
- [ ] Greptile is 5/5 with no open actionable feedback on the latest
head
- [x] I have addressed all current review feedback

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 11:17:56 -06:00
DottaandPaperclip 59da15066a feat(runner): deliver native Kimi Code (#18391)
Deliver native Kimi Code setup, exact models, scoped credentials, verified packages and runtime packs. Preserve current upstream release attestations and restore verified same-run remote retry dispatch without weakening process ownership.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-30 22:26:15 -05:00
DottaandPaperclip 044628f59e Add explicit native tracing to the Runner eval CLI (#17483)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Runner Evals exercise real providers against a mock control plane.
> - The controlled gateway route creates a disposable provider home.
> - That route had no explicit native trace configuration.
> - Codex skips normal home preparation for a preconfigured home.
> - This change validates and applies the existing observability launch
contract before provider creation.
> - Operators can collect native evidence without passing vendor keys to
the child.

## Linked Issues or Issue Description

Refs #17440. Related: #17452 and #17469. This PR is based directly on
master and does not include those pending controller changes.

**Current behavior**

The eval-session CLI cannot pass an explicit native observability
launch. Passing environment alone cannot configure Codex in the
disposable gateway home.

**Proposed behavior**

Accept an opt-in descriptor bound to the gateway attempt. Validate the
existing provider launch contract, add a scoped relay token in memory,
and apply it only to the owned provider home and environment.

**Reason and benefit**

A controller can route native evidence through its bounded local relay
while the CLI keeps configuration distinct from verified ingestion. This
is the CLI contract; hosted controller integration remains separate
work.

## What Changed

- Add a strict, bounded descriptor reader for native Codex and ACPX
Claude. Bind campaign, execution, attempt, project, trace, span, and
mock authority identifiers.
- Reuse Runner observability materialization and redaction. Preserve the
model gateway route. Reject upstream destinations, extra environment
fields, credentials in headers, and unknown exporter settings.
- Add `--observability-config`, requiring an explicit gateway. Keep
direct and unconfigured paths unchanged.
- Retain secret-free configured/unverified metadata in success and
failure artifacts.
- Document the contract and controller responsibilities. No workflow,
runtime dependency, paid schedule, or public report changes.

## Verification

- Eval CLI regression selection: 81 passed before the test-only
relocation. The 17 new cross-package observability tests now pass from
`tests/runner-e2e/runner-eval-observability.test.ts`; their
integration-suite typecheck also passes. They use the actual shared
launch generator, real disposable homes, CLI success/failure paths, and
credential redaction. The first run exposed two test setup references
introduced by a broad edit; both were corrected and its failure log
remains retained.
- Runner TypeScript build and release daemon build passed. Full `pnpm -r
typecheck` passed.
- Actual native Codex and ACPX Claude reached a local receiver with
authenticated OTLP protobuf and the expected Braintrust parent. Codex
sent four log and two trace batches; Claude sent three log and one trace
batch. All recorded processes exited, disposable homes were removed, and
29 retained files contained neither synthetic credential.
- The local model endpoint deliberately returned 401. Codex retained a
failed terminal turn; Claude retained a provider timeout. These are
route/export proofs, not semantic eval passes. Native retries reached
only the fake endpoint. No paid provider or Braintrust request was made.
- The first local probe failed before provider launch because its
TypeScript loader path was wrong. It was corrected to the installed
loader; both attempts remain retained.
- Full repository build passed at initial head 7b75c0349f. The full
local test suite encountered repeated embedded PostgreSQL startup
failures before database-backed test bodies. It was stopped after five
affected suites (exit 143), so it is incomplete. Its log remains
retained. The same host had independently demonstrated PostgreSQL
semaphore exhaustion earlier in this task; this run did not collect that
lower-level error. Full hosted workflow 40131255078 passed on final head
23a172af32, including the database-backed
suites, Runner, build, typecheck and browser checks. Greptile is 5/5
with zero open threads. The secondary Paperclip reviewer ended neutral
without an assessment; it is not counted as a successful review. A
follow-up header-validation repair uses a map without a prototype and
passes the added negative regression. No exporter behavior changed.
- The CLI still reports native ingestion as unverified. No complete
capture, destination indexing, vendor billing bound, hosted admission,
or paid recurrence qualification is claimed.

- Package-boundary validation found two new imports from the standalone
Runner tests into other repository packages. Those tests now live in the
repository integration suite and use the actual launch generator there.
Its first run failed on a TOML parser resolved from the wrong package;
resolving that dependency from its Runner owner fixed the test. Three
pre-existing boundary/tracked-import findings remain in unchanged files;
this PR adds none.

## Risks

- This descriptor is intentionally narrow. Changes to the shared launch
contract must be reviewed against its validator and integration tests.
- The local relay receives native content under the selected capture
policy. Its controller must enforce ingestion quotas, shutdown, private
retention, and qualified cost bounds.
- Descriptor validation runs once before launch. It reads at most 128
KiB and refuses symlinks and non-regular files. Existing owned-home
config writing remains synchronous and small.
- Relay credentials are held in child memory. Persisted diagnostics and
artifacts use the existing redactor. The descriptor itself must remain
secret-free.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, and code
execution. This runtime does not expose the exact model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-24 14:07:28 -07:00
DottaandPaperclip d1fdfced9d feat(evals): route Runner sessions through a controller gateway (#17440)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Runner Evals test provider behavior against a mock control plane.
> - A campaign budget needs to observe and admit each provider request.
> - The existing eval CLI does not select the provider-home mode needed
for a controller-assigned route.
> - A proxy URL alone therefore cannot establish controlled Codex
routing.
> - This change adds an explicit route with an isolated home and
retained identity.
> - The controller can use that route as the basis for request
accounting and admission.

## Linked Issues or Issue Description

**What happened?**

The direct Runner eval CLI passes the ambient environment to the native
transport. The transport deliberately removes an environment-only
preconfigured-home marker unless its explicit option is also set. This
prevents a controller from assigning a Codex gateway through that marker
alone.

**Expected behavior**

A controller can explicitly route a supported eval attempt through its
local proxy. The provider receives the proxy token and isolated
configuration. The artifact identifies the assigned route on success and
failure. Ordinary direct runs keep their current path.

**Steps to reproduce**

Run the new CLI tests for `--gateway-config`. They validate the
descriptor, inspect the actual transport inputs and provider sanitizers,
and check retained artifacts and owned-home cleanup. The local probe
also ran built native Codex and ACPX Claude against a fake loopback API
using only synthetic credentials.

Related request-accounting work:
https://github.com/paperclipai/paperclip/pull/17432.

## What Changed

- Add an explicit gateway descriptor bound to the requested provider,
exact model, campaign, execution, and attempt.
- Accept only the controller's IPv4 loopback URL and a separate proxy
token.
- Create a fresh provider home for native Codex and ACPX Claude. Exclude
ambient provider credentials, subscription logins, proxies, and startup
hooks.
- Retain route identity and descriptor hash in successful and failed
artifacts. Remove the owned home in finalization.
- Document the CLI contract and its limits.

The Runner eval CLI owns this behavior. It uses the existing native
preconfigured-home contract. Production routing, permissions, and
provider protocols are unchanged.

## Verification

- Initial verification passed 56 focused gateway/CLI tests and 246 CLI
and adjacent provider-isolation tests. After the admission repair, all
247 CLI and adjacent provider-isolation tests passed, with TypeScript
checks and build passing again.
- Runner TypeScript checks and build passed.
- The native daemon rebuilt from the candidate base with locked, offline
dependencies.
- Local fake-API probe: six authenticated Responses requests from native
Codex and six authenticated Messages requests from ACPX Claude reached
the assigned endpoint. Claude also made an unauthenticated hello check.
The final commit retained both route artifacts. All 27 retained files
were scanned for the synthetic proxy token and contained none. Both
temporary homes were removed, and all four recorded provider/daemon PIDs
were absent after exit.
- The fake API intentionally rejected inference. Codex retained a failed
turn; Claude retained a timeout. These are route observations, not
passing semantic evals or live provider qualification. No paid provider
calls ran.
- The first focused command used the wrong directory and discovered no
tests. It was corrected before the successful runs above.
- Full Runner validation: 3,967 tests passed, 13 skipped, and five
failed in three unchanged installation and native-settlement files. The
failures were two installation timeouts and three retained-cleanup
settlement failures. One isolated run passed all 109 applicable tests
with six skips. This shows recovery, not a proven cause for the original
five failures; those remain retained. Hosted [CI
40111220754](https://github.com/paperclipai/paperclip/actions/runs/40111220754)
passed on final commit `a5c47b64e56c1228fca718b72a81ddc3b1d185a4`,
including repository typecheck, build, tests and Runner checks. All 57
checks passed or were neutral/skipped, and the legacy status passed.
Greptile and Paperclip reviews both scored 5/5 on that commit; no
findings remain unresolved.
- Rebased onto current master
`71c04c0bfb205cc9e7f7bc20c29fd76051b44cc4`. Its added change concerns
server cleanup; Runner source is unchanged.

- An actual ACPX Codex probe failed before reaching the fake API. Review
and source inspection confirmed that Runner deliberately rejects its
preconfigured home to protect Runner state. This route is now rejected
before service creation. Both explicit and default ACPX Codex admission
regressions failed before the repair. The unsupported lane is not
presented as qualified.

## Risks

This is an explicit new CLI route. Unsupported providers fail before
service creation. Upstream route is a controller declaration, not
independent billing evidence. The local probe saw six Codex requests
where legacy usage counted one, so the controller's request ledger must
remain the budget source. Campaign allocation, proxy lifecycle, hosted
workflow wiring, dedicated OTLP relay routing, and paid qualification
are separate remaining integration steps. This PR enables none of those
schedules.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, and code
execution. The exact serving model ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used, including the information
available in this session
- [x] I checked ROADMAP.md; this is eval infrastructure, not a competing
core feature
- [x] I searched for duplicate PRs for this exact route change
- [x] I described the issue in-PR using the bug template labels
- [x] I have not referenced internal or instance-local issues
- [x] My branch describes the change and contains no internal issue ID
- [x] Relevant focused tests pass
- [x] I added tests for the new contract and lifecycle
- [x] I updated the relevant documentation
- [x] I described the risks and unqualified behavior
- [x] All current-head Paperclip CI gates are green
- [x] Current-head Greptile is 5/5 with no unresolved findings
- [x] I will address review findings before merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-24 09:34:10 -07:00
DottaandPaperclip a3e1d915a7 fix(evals): preserve unknown Codex provider cost (#17158)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Direct Runner evals report usage and costs for each attempt.
> - Codex App Server supplies token counts without a provider USD
receipt.
> - The legacy Runner ledger fills an absent cost with zero.
> - The eval result exposed that ledger value as provider-reported cost.
> - This fix keeps provider cost unknown and preserves the estimate and
budget stop.

## Linked Issues or Issue Description

Refs #17009 and #17154. This is a separate direct Runner accounting fix
based on master.

## What Changed

- Codex direct eval usage now reports null provider USD cost with
unavailable provenance. Token counts and the pinned price estimate
remain available.
- Preserve the original ledger amount as optional
`legacyLedgerCostNanodollars`. Keep the original snapshot receipts
unchanged.
- Apply the existing budget ceiling to the retained ledger amount as
well as the token estimate. Correcting a label does not relax admission
or settlement checks.
- Use the unique operator UUID for Muse test company prefixes,
eliminating collisions from the previous 24-bit random suffix.
- Document the cost distinction in the Runner/Evals integration
contract. Other provider behavior and historical artifacts remain
unchanged.

## Verification

- Reproduced both zero and nonzero legacy amounts being labelled as
provider USD cost before the fix.
- The current eval-session contract and entrypoint tests pass (39
tests). The new regression uses the bounded usage consumer and actual
campaign cost projection. It checks unknown provider cost, the separate
estimate, unchanged raw evidence and the legacy ledger budget stop.
- All 15 Runner report, campaign and metrics Node tests pass. Four
adjacent CLI entrypoint/runtime tests also pass.
- Runner TypeScript validation passes with its standard dependency-build
prerequisite; `git diff --check` passes. Both Runner TypeScript
configurations also passed during the original accounting verification.
- Hosted CI run
[39960367674](https://github.com/paperclipai/paperclip/actions/runs/39960367674)
passes on `7d84f138044bac58cdced6baa2d0ec3d63767bb8`, including
typecheck, build, server tests and browser checks. Greptile and
Paperclip Review both report 5/5 on that commit with no actionable
findings.
- The previous CI failure was a PostgreSQL unique-index collision on
`companies_issue_prefix_idx` during Muse fixture setup. Local Muse
verification could not reach test execution because embedded PostgreSQL
timed out during macOS startup (initially reporting a missing zstd
library). The corrected fixture passed in hosted Linux CI.
- Verification used no model calls. Full repository checks were
performed by hosted CI; local verification was scoped to the changed
contracts and fixture.

## Risks

Consumers that assumed Codex always had a numeric provider cost must
handle null. The contract already permits null for other profiles, and
the report consumer retains missing-cost coverage. The legacy amount
remains available as unverified evidence and still triggers the existing
budget stop. Neither the estimate nor that ledger amount proves an
invoice charge. This change does not rewrite old artifacts or qualify
other provider routes.

## Model Used

OpenAI GPT-6 through Codex, with repository tools and code execution.
The exact deployment variant and context-window size are not exposed by
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 06:36:34 -07:00
DottaandPaperclip 57e977be72 feat: integrate Pi 1.0 into the experimental Runner (#14921)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - The experimental Runner owns provider processes and durable
sessions.
> - Pi needs working task execution and human controls.
> - The five-PR stack must preserve changes already on master.
> - Each layer now carries the complete integrated source for a safe
sequential fallback.
> - This PR belongs to native GitHub stack #15602, ending at #14956.

## Linked Issues or Issue Description

Refs #14436, #14631, #14743 and #14956.

Ship Pi 1.0 through the experimental Paperclip Runner. The five PRs are
#14921, #14922, #14923, #14924 and #14956. The user authorized the
complete merge after checks pass. Existing `pi_local` execution is
unchanged. Accounting and wider provider/platform qualification remain
deferred.

## What Changed

- Recover missing final replies after workspace finalization changes
owners, using accepted-turn evidence without rerunning work or granting
external-chat publication.
- Preserve the admitted Pi instruction root across warm runs, while
retaining changed-root rejection.
- Give Pi a bounded 15-second default shutdown grace so stop, drain
acknowledgement and durable suspension can complete. Explicit deadlines
and other providers retain their existing behavior.
- Integrate the Pi 1.0 runtime and master contracts.
- Use Pi profile 22. Preserve explicit caller-selected models and exact
native thinking levels. Keep Pi's wrapper, helper, extension and
question/control behavior unchanged from the qualified profile-19
runtime.
- Preserve master's Dot lifecycle and consent fields, configured task
environment, status guards and current Codex/Claude dependency versions.
Cursor stays qualified. Copilot stays pending; profile 17 binds the
changed shared protocol validation sources.
- Exclude general AWS IAM credentials from Pi static/custom provider
bindings and selected task projections; preserve the provider-scoped
Bedrock bearer key. Profile 21 is retained as historical provenance.
Rust and cloud install probes use the current declaration.
- Patch bundled brace-expansion 5.0.9 to the exact official 5.0.12
payload. Pin the patch and complete runtime closures. Include the patch
in normal installed setup tooling. Keep the upstream Pi shrinkwrap as
provenance and permit only this exact security correction.
- Include current attestation files in the Docker build context. Keep
the repository lockfile unchanged from master. CI and private image
builds resolve manifest changes before their frozen installation.

## Verification

- Full local `pnpm -r typecheck` passes, including Runner Rust, server
and UI. Focused integration checks pass: 194 Runner
admission/environment tests, 63 profile/credential tests with one
expected skip, 152 Dot/UI configuration tests, and Pi transcript/notice
tests.
- Full local `pnpm build` passes on the final source.
- Fresh final-source checks pass: all 698 Rust workspace tests (32
binaries), 156 credential/profile/controller tests with one expected
skip, Runner TypeScript typecheck, and 20 package/setup/sandbox tests.
- The profile-21 Pi materializer passes on the native host with the
official pinned Node 24.21.0 and its npm. It verifies all 150 locked
packages, the patched dependency and the exact closure. Setup/package
bundle tests and UI token gates pass.
- The old hashes were reproduced for all three supported targets before
calculating the patched graph. New closure hashes are darwin-arm64
`282022db10150c6632b3444df421342e7d534bdf5d5fb1097a2e79d0625a2bcf`,
darwin-x64
`64e251e19009f755c0b04f73ce2138246faab71a961b0f13d75ebfcc34bef12e`, and
linux-x64
`713b1fdff42fb56a1518bdc084f181d70bee8ebadc3e4b1d76321ed9108c8410`.
Independent native platform execution is separate from graph identity
reproduction.
- Historical cloud qualification remains unchanged: all seven core cases
pass on shipping source `10dc43c9ec65d88c2f782d62afb296d09494f215`,
harness `1a4408a48cfb5a1f094a311141c257c92cd7a893`, image
`sha256:5b3a775b383591bda1b0c1889e509acc70ce7f37c53f09733c81d59037f02280`,
and accepted Sonnet 4.6/low fixture. All 215 canonical files and all
seven cleanup checks pass independent verification. These are profile-19
results and are not relabeled as fresh profile-22 runs.
- Current Pi digest:
`sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`.
[The readiness
plan](https://github.com/paperclipai/paperclip/blob/codex/pi-production-readiness/doc/plans/2026-10-02-pi-production-readiness.md)
preserves campaign and failed-attempt provenance.
- Merge only after every PR's current-head CI and fresh review pass.
Linux CI covers the full suites, build and browser tests. The local
embedded Postgres API-authority suite cannot start on this macOS/Node 26
host, so Linux CI must confirm that suite.

### Fresh profile-22 core qualification — 2026-10-08

All seven accepted core cases pass canonically on Pi profile 22, with
`openrouter/anthropic/claude-sonnet-4.6` and native-confirmed low
thinking. This model is a fixture; production accepts the caller's
explicit Pi provider/model.

Runtime/install source: `3241a992f2a7703e59e97ed0fd3e5d6405de4401`.
Frozen accepted harness: `1a4408a48cfb5a1f094a311141c257c92cd7a893`.
Immutable cloud image:
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:506f22db7edd78f37c0c40bec1cc084af1850455026dbf467194bfbb8fcef141`.
Pi digest:
`sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`.

[Hosted Linux image and clean-install
verification](https://github.com/paperclipai/paperclip/actions/runs/37868328023)
passes, including all 20 source-bound archives, normal CLI/Pi setup,
companion import and the production pack reader. This exact installation
source includes the latest master integration and the corrected Pi warm
instruction-root fence. Full local typecheck/build and current-head
hosted CI verify the final stack. All 13 focused real-root regressions
pass. The full local executor suite passed 662 tests; 15 database tests
could not start the Mac embedded PostgreSQL service. Hosted Linux CI
passes the full required verification and E2E checks. These fresh
results keep their own source identity; profile-19 results remain
historical.

| Core path | Canonical campaign | Retained archive SHA-256 |
| --- | --- | --- |
| File edit, validation, download and Done |
`pi-core22-replyfix-0-1791511228` | 23 files;
`a473e8603a3dd4737863291f8d3d1e392391f0b16d433c3e0e0e9d8baf7a97b0` |
| Pending question and controller restart |
`pi-core22-replyfix-1-1791511376` | 33 files;
`6b829c4eb74e1f32a89c692a4ae7130dbfc1c6d3cf13915effe2103d9e242c8e` |
| Three-turn session/process/workspace continuity |
`pi-core22-replyfix-2-1791511587` | 23 files;
`7a87021f8f9a3fdd3c58bb4467f8d82c635e3ea4795d6e75f144d9aa14818df8` |
| Four typed questions and browser reconnects |
`pi-core22-replyfix-3-1791511881` | 42 files;
`9e31755252be1f4f9cb0626c984c142d4d1ae5f5bee3a7af08444db8d12c280a` |
| Plan approval and completion | `pi-core22-replyfix-4-1791512031` | 22
files;
`a0383ce1aab38e7b5a25ce0e9dd3bebea5c037ebd96ae6b29dae19015da2ae2c` |
| Same-turn steering and permission denial |
`pi-core22-replyfix-5-1791512261` | 39 files;
`c929b8c7070f0b66aedc17e65ca46e6beab1e363926ac9f7e2a75fb250f05949` |
| Stop during pending permission | `pi-core22-replyfix-6-1791512390` |
33 files;
`7f0a58ae0f4d5bfc76149435f4e322537089c5bd16e7ffe9b5ad71f10a621a07` |

All 215 canonical files (28714587 bytes) are independently
hash-verified. All seven cleanup grades pass, with no owned runtime
process or temporary root after each case. Automatic retries are zero.
The owned cloud host stopped normally after retention. The prior
profile-22 warm attempt remains failed and separately retained: archive
SHA-256
`1e54eba5ec72b50cee1534b23d1d1d4f21a090006b8a64501ba70db972abfde5`. Its
original canonical classification is preserved. Diagnosis reproduced a
product bug comparing an agent-files root against an unset
checkpoint-only field. The fix stores the admitted physical root
separately from the adopted per-run collection capability. The real-root
regression fails before the fix and passes afterward, including
rejection of a changed physical root. Fixture, grader, model and all
seven accepted case IDs are unchanged; this fresh campaign tests
final-reply publication after file registration first. The intermediate
restart attempt also remains failed and retained: archive SHA-256
`5dcaefdf1d17cf4cd54fd4cf810f45e736667392339b8ce7caf08bb4e225277f`. Its
original canonical classification is preserved. Pi resumed, wrote the
verified answer and completed its task; exact runner suspension was
proven, but idle stop consumed about 5.2s and left under 3s for the
drain acknowledgement. The Pi-only default shutdown grace is now 15s,
preserving a full 5s drain round trip and a finite suspension reserve.
Explicit caller deadlines, other provider defaults, literal drain
receipts and exact suspension identity checks remain unchanged. The
timing regression fails before this correction and passes afterward; all
18 focused settlement tests and Runner typecheck pass. The final-source
file attempt is also preserved as failed (`candidate_failure`), archive
SHA-256
`db6767b6773ea618997927ac77bdb005a5ac81492c7b9c0ffbc900449f829bc9`.
Native edit, validation, exact downloadable artifact and Done/succeeded
all passed, and the exact final reply was durably recorded. A workspace
recovery owner completed before the live heartbeat reached presentation,
leaving that reply absent from task chat. Recovery now materializes only
a completed final reply from the accepted turn of an ordinary internal
Done task, preserving issue/run/contract binding, suppression,
external-chat authorization and same-run deduplication. The database
regression covers the generated file-preparation receipt, suppression,
unapproved external continuation and replay. Server typecheck and all 49
response-selection tests pass; hosted Linux verifies the database
regression because embedded PostgreSQL cannot start on this Mac.

The delayed-final-answer database regression passes on [the final
root-source Linux server
shard](https://github.com/paperclipai/paperclip/actions/runs/37868262553/job/113628594152),
alongside 1,108 passing tests. The first root Runner shard had one
unchanged durable-resume test exceed its 5-second timeout; the identical
top-source shard and the isolated exact test passed. One rerun of that
failed job and its required aggregate passed without source or test
changes. The original failed job log and the single-rerun receipt remain
retained.

### October 9 merge verification

Current merge head: `5a8fe63512a7166aaef5cf50065a25008aa8b44b`. All
current-head checks pass, including `ci / verify` and `ci / e2e`;
exact-head Greptile review is 5/5 with no unresolved threads. Current
master conflicts are resolved. The user authorized the maintainer
override of the code-owner review gate after these checks. The seven
retained live core cases remain bound to source
`3241a992f2a7703e59e97ed0fd3e5d6405de4401` and its recorded cloud image.

## Risks

- The security correction changes the dependency closure and profile
identity. Old sessions must reopen on the new profile. Exact identities
and credential bindings fail closed.
- The runner remains experimental and requires explicit selection.
Legacy Pi Local is unchanged. Caller model IDs pass through; the E2E
model is a fixture.
- Accounting and the broad platform/provider matrix remain deferred.
This merge does not publish a release or deploy a service.

## Model Used

OpenAI GPT-6 through Codex assisted with reasoning, repository
inspection, editing and tool use. The exact serving ID and context
window are not exposed in this session. Final live qualification uses Pi
1.0.0 with `openrouter/anthropic/claude-sonnet-4.6` and native-confirmed
low thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-09 08:56:41 -05:00
DottaandPaperclip 6f9d0a56ba fix: resolve installed Codex and preserve npm host dependencies (#15555)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Installed agents need their own runtime dependencies.
> - Native Codex startup and browser login could depend on a global CLI.
> - npm bundles do not inherit workspace dependency overrides or
patches.
> - Codex native packages must come from npm for the consumer host.
> - This PR fixes executable resolution and the required npm dependency
layout.
> - Usable installed CLI versions can run without a numeric version
gate.

## Linked Issues or Issue Description

Refs: #15422. This is a prerequisite for the later Codex-default change.

**What happened?**

Packaged native Codex startup and browser login could require a global
Codex CLI. The server's vendored runner lacked its own declared Codex
bridge. A bundled wrapper could retain producer-host binaries or select
the legacy adapter's separate platform version.

**Expected behavior**

Prefer the installed Codex executable. Accept usable older or newer
versions. Use the selected host's PATH when the dependency is absent.
Keep the patched JavaScript graph. Let npm install official native
packages for the consumer platform. Grant sandbox reads only to the
selected vendor resources.

**Steps to reproduce**

1. Prepare a clean npm installation without a global Codex command.
2. Start native Codex or its browser login.
3. Inspect the selected executable, installed host package, and sandbox
resource paths.

**Paperclip version or commit**

Frozen master base: `1881894973a2b25838d8abed9bd8aeebc3af4441`.

**Deployment mode**

Source and packaged self-hosted installation.

## What Changed

- Share installed Codex command resolution across native startup, direct
evals, and browser login. Accept usable version differences. Preserve
explicit commands, recorded sessions, remote host boundaries, and the
Linux ARM64 legacy login path. Return safe errors for missing
executables and failed login terminals.
- Declare the server's Codex bridge. Retain the patched JavaScript
dependency graph in npm packages. Strip Codex native payloads. Declare
official optional host packages on the published server manifest so
native and legacy versions remain separate. Preserve consumer esbuild
platform dependencies.
- Adjust resource lookup for npm's separate platform packages. Retain
package identity, path containment, resolver ownership, and narrow
sandbox reads. Keep exact version and digest checks for explicit ACPX
artifact qualification.
- Extend existing packaging, installed-consumer, login, selection,
recovery, and integrity tests. Document the retained behavior.

The diff is now 23 files, 1,709 additions, and 53 deletions. The prior
diff had 35 files and about 3,700 changed lines. Removed work is
preserved on `codex/runner-packaging-full-snapshot` at
`6f3060beaa842ffcee21af058370d3cab5f571e9`.

Removed from this PR: release workflows and assembly, provider-pack
changes, Docker materialization, Git installer changes, extra login
HOME/working-directory isolation, and unrelated CI fixture repairs. This
PR does not change agent defaults, stored runner choices, UI, schema, or
provider qualification.

## Verification

- Final candidate: `dde37d7ed0121c10b60b8801eda80f8dc17909ad`. [All 47
ordinary CI jobs
passed](https://github.com/paperclipai/paperclip/actions/runs/37842288835)
on attempt 1, including typecheck, tests, build, E2E, runner checks, and
the installed-consumer canary. [Fresh Greptile
review](https://github.com/paperclipai/paperclip/pull/15555#issuecomment-6059581546)
is 5/5 on this head; no unresolved threads remain. Human approval is
still outstanding.
- Reused focused controls passed with Node 24: 71 initial narrowed
checks, then 56 affected Codex/selection/eval checks after the relative
PATH correction. Runner TypeScript no-emit, syntax, and diff checks
passed. The existing fixture reproduces the original relative PATH
failure and verifies working-directory selection, empty entries,
ordering, and absolute launch. One CLI entrypoint test was initially
blocked by sandbox IPC and passed with its existing local socket
allowed. Login HOME, config-directory assignments, and working directory
match master.
- [The actual clean Linux npm
consumer](https://github.com/paperclipai/paperclip/actions/runs/37842288835/job/113535523044)
passed on the final candidate's CI integration. All 17 Paperclip
tarballs omit native Codex payloads. Official npm host packages retain
their own integrity and `inBundle=false`. Native Codex selects 0.160.0;
the legacy closure retains 0.156.1. Package admission, command leases,
narrow native sandbox resources, consumer hooks, preserved lock, and
offline lifecycle controls passed. Provider calls were zero. No new
workflow is added.
- Actual official Codex 0.156.1 passed on the final committed source on
macOS ARM64: installed dependency preference, absolute and relative PATH
fallback, narrow resource lookup, and app-server initialize/initialized.
Executed module hashes match the candidate. One scripts-disabled
install, three version probes, and one handshake completed in 26
seconds; owned files and process group were removed. No login,
account/model request, or provider task ran.
- CI checked out `dfda8e708e87306d22ed735d78bdfcbe770e99b7` on base
`65b558180533039a891ee0cd1ccab9988aa79adc`. Its 13 upstream paths do not
overlap this PR's 23 paths or alter packaging/Codex inputs. The consumer
report records producer `7b8e94c08657b5ddc265946459762340f125726c`,
after the existing canary staged a generated-lock-only commit (one file,
three insertions). These identities are kept separate; raw
generated-lock bytes were not retained.
- No local Docker or Rust build, paid provider turn, merge, or
deployment. Later PRs must prove live onboarding and production cloud
packaging before changing defaults.

## Risks

- npm must install optional host dependencies. Missing dependencies
still return errors. Paperclip tarballs do not pre-bundle Codex
executables.
- The repository requires CI-owned lockfile updates. PR CI resolves the
changed manifest and stages its own producer lockfile. A raw source
Docker build with `--frozen-lockfile` must wait for the existing master
lockfile bot to merge its refresh, or use a disposable resolved
checkout. No Docker build or deployment is qualified by this PR.
- Ordinary Codex startup accepts version differences. Actual protocol or
login failures remain errors. Explicit ACPX artifact checks retain their
release pins.
- The published server delegates platform installation outside its
bundled JavaScript graph. Focused negative controls reject unsafe
package metadata and paths. The hosted consumer test passed on this
candidate’s CI integration.
- Existing agents retain their stored runner choices. There is no data
migration or automatic upgrade. Release pipeline and platform
qualification work remain separate prerequisites for later defaults.

## Model Used

OpenAI Codex, GPT-6, with reasoning, tool use, code execution, and
parallel agents. The exact serving snapshot and context window are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-09 08:19:51 -05:00
DottaandPaperclip a6306ba606 feat(runner): consolidate Cursor production integration (#15075)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native Runner keeps provider sessions under company authority,
approvals, budgets and durable recovery.
> - Cursor work was spread across candidate branches. The published
branch lacked later plan, permission and cleanup fixes.
> - Production also needs public installation and matching runtime
assets for local and Daytona execution.
> - This pull request consolidates Cursor onto current mainline recovery
behavior and completes that installation path.
> - The installed v11 release passed focused local and Daytona
qualification after the generic mode and lifecycle cleanup. The later
model-selection correction and current mainline merge produce v14
artifacts that need matching release qualification.
> - Cursor admission is enabled in source; publish only an artifact
combination with matching qualification. Native AskQuestion and complete
per-run dollar accounting remain excluded.

## Linked Issues or Issue Description

Refs: #14435, #14631, #14669, #14699, #14724.

This completes the Cursor implementation by @cryppadotta from combined
source `22c78242a4e0c2369fecf0c2dc4e7600fbad6706`. It preserves newer
mainline recovery, completion and warm-directory behavior. Pi and
Copilot remain gated.

## What Changed

- Generate named Rust and TypeScript ACPX release profiles from one
manifest. Share runtime pins with packaging and server verification.
Preserve vendor runtime versions; bind the updated ACPX patch to Cursor
profile v14 and reject stale generated declarations at build/typecheck.
- Remove ACPX model allowlists, including the former Codex and Pi
restrictions and the duplicate developer test-drive gate. Send any
explicit model ID unchanged to its provider and verify the effective
selection before prompting. The bundled ACPX package forwards unlisted
IDs, rejects mismatched acknowledgements, and restores the exact
selection after session load. It does not expand Cursor model aliases.
Provider rejection, mismatch, or missing model controls fails without a
fallback. Model examples live in evaluation fixtures, outside runtime
declarations.

- Add pinned Cursor execution, contained instructions, exact model
verification and Agent/Plan/Ask modes.
- Carry an opaque generic `mode` identifier in shared native execution,
sidecar, Rust and recovery contracts. The provider adapter owns
supported modes, defaults, native translation and acknowledgement.
- Keep native RPC recognition, accepted-plan interpretation and
permission evidence behind provider adapters. Shared settlement and
recovery verify normalized facts and their committed evidence.
- Replace the Cursor-only warm-attachment branch with a runner-owned
capability. Only Cursor opts into it. Move profile compatibility and
optional usage parsing into provider metadata and adapters.
- Write generic plan-wait receipts. Read exact historical Cursor
receipts through a separate compatibility decoder. Reject mixed formats
and preserve existing authority checks.
- Carry native plans, semantic questions, todos, child activity,
permission identities and partial usage diagnostics through the Runner.
- Preserve durable response delivery, cancellation, warm ownership and
process retirement.
- Finish accepted planning runs successfully. Keep their tasks open for
explicit direction. Acceptance does not start implementation.
- Ship `paperclipai runtime setup cursor` and its provisioner through
the public package. npm installation does not download Cursor. Setup
uses the OS account's closure-keyed cache so system-wide npm packages
can remain read-only. Run it as the Paperclip service account.
- Include Cursor in normal provider packs and Daytona images for macOS
ARM64/x64 and Linux x64.
- Reject stale release packs by source revision and current ACPX/Cursor
pins before assembly writes files. Verify current Cursor
version/profile/closure again at runtime.
- Ship all three daemon targets and the expected Linux image-pack
identity. A macOS controller uses its packaged Linux daemon for Daytona.
Image mismatches fail before provider launch.
- Use the vendored Runner boundary for installed readiness probes.
Verify the actual installed Cursor probe.
- Verify compiled public Daytona plugins and their release versions in
installed smokes.
- Record exact artifacts, the acceptance matrix, retained failures,
supported capabilities and rollback behavior in the [readiness
report](https://github.com/paperclipai/paperclip/blob/codex/cursor-production-readiness/doc/plans/2026-10-03-cursor-production-readiness.md).

## Verification

- Current head `f7ec5cc1f0e30c62a829c016ff2013a2a9d79037` merges
mainline `faa8e452c73bae5e044dd6379179a00106abb131`. It keeps Cursor
plan and cancellation guards alongside mainline historical-question
filtering. The evaluation catalog includes both Cursor and expanded
adapter accounting cases (683 total). Recursive typecheck, full build,
696 lifecycle/recovery tests, 45 fixture tests and fixture typecheck
passed. Current-head CI passed: 56 successful checks, one neutral and
four skipped. [Complete
CI](https://github.com/paperclipai/paperclip/actions/runs/37557996535).
The fresh Base Greptile review is 5/5 on this exact head, with 304 files
reviewed, zero new comments and zero unresolved threads. The user
authorized overriding the CODEOWNER review gate after checks passed; no
failing checks are overridden. Prior results below retain their own head
identities.
- Corrective head `3d2b168366258036f6b6a6fccb382c49138cc601` fixes the
post-merge Apex finding. Automatic-review and new-evidence
reconciliation preserve pending child results and recheck delivery under
the status lock before completing. Account repair now excludes unrelated
secret consumers and requires the failed agent's identity. Regression
coverage includes the commit race, delivery statuses,
current-run/current-intent exclusions, repeated reconciliation, both
database reconciliation paths, and credential consumer boundaries. All
184 affected tests, server typecheck and server build passed.
Current-head Base Greptile review is 5/5, with 304 files reviewed, zero
new comments and zero unresolved threads. Current-head CI passed: 56
successful checks, one neutral and four skipped. [Complete
CI](https://github.com/paperclipai/paperclip/actions/runs/37535994724).
This Base review is distinct from the earlier Apex review.
- Merge head `5957c257a` reconciles mainline `b508a05c4`. It preserves
both accepted-plan waits and pending-child-completion checks, current
provider selectors, task-creation response identities, and mainline ACPX
missing-file handling. The combined patch is bound to Cursor profile
v14; historical records keep their original identities.
- Merge head `5957c257a` passed recursive typecheck, full build, 43
installed ACPX/package contracts, 107 provider UI and plan/recovery
tests, 593 database-backed lifecycle tests, 49 profile/native contract
tests, 45 Product E2E fixture tests, fixture typecheck, token gates,
three provider-free browser task-creation cases, and Runner
conformance/replay checks. Its complete CI passed (55 successful checks,
one neutral and four skipped), while Apex returned 2/5 with a
child-delivery finding addressed below.
- The local full-suite attempt again failed the unchanged Git streaming
test (360-second timeout) and was stopped. The concurrent local Rust
attempt failed four unchanged Codex process/deadline tests; all four
passed serially without code changes in 7.29 seconds after removing the
competing test load. These failed commands are retained and are not
reported as full-suite passes; the fresh Linux CI runs are tracked
separately.
- The previous head `907bdb2a2778c7ffeb4a662a91460c9d1ddfc9c5` earned
Apex 5/5 with zero comments after fixing all three findings: per-user
install cache, stale release-pack rejection, and public Linux smoke
account/home handling. Its real built installer passed from read-only
public packages on macOS ARM64 and Linux x64. All 137 release-registry
checks and 64 ACPX package contracts passed. That review does not cover
this mainline reconciliation.
- Prior `beadd3654` passed the full CI matrix; its one unchanged chat
test failure and successful single retry remain in the [CI
history](https://github.com/paperclipai/paperclip/actions/runs/37521449327).
Historical results below remain attributed to their original builds.

- Fixture follow-up `dd59d7e82b103a88b7cbd7d2c38b612c0fbbff7a` removes
provider-specific model choices from generic offline ACPX tests. The
fake sidecar preserves the model and session identity selected at open
through suspension. Affected verification passed: 106 Rust tests and 73
TypeScript tests. This commit changes test code only; the
production-code checks below retain their recorded identities. Its CI
and Greptile review later passed; those results belong to that
historical head.
- Model-selection cleanup `9a070808b48960a41fdfd369ae0636b95af82459`:
252 focused Runner tests passed (six platform skips), covering all six
ACPX agents, native model acknowledgement, rejected selections,
installation integrity and recovery identity. The merged branch passed
recursive typecheck, full build, token gates, server admission (19
tests), and the Product E2E catalog (45 tests). The acceptance catalog
passed all four tests. The full Rust suite passed: 643 tests, 2 ignored.
It verifies sidecar acknowledgement of unlisted models and rejection of
model mismatches. The final commits only update Rust tests; production
sources match the verified build at
`65ec3279ac50185e3cda109b5cfd9b4f56105de0`. No new paid provider calls
were made.
- The merge preserves both Cursor and the new mainline public-MCP
fixture cases. Auto-merge remains disabled; the latest follow-up status
is recorded above. The local `pnpm test:run` attempt hit the unchanged
Git streaming test's 300-second timeout and was interrupted before
merging mainline. The broad Runner attempt found obsolete single-model
assertions plus three macOS fixture-path failures caused by a
`/private/tmp` override. The assertions are corrected; affected
TypeScript checks passed with the standard macOS temporary directory,
and the complete Rust suite passed. Neither interrupted command is a
full-suite pass.
- Earlier declaration-cleanup head `6f4a5e9e2` passed recursive
typecheck, build, Rust and focused tests. Its CI later exposed a test
expecting duplicated Grok digest literals. The current source fixes that
assertion to compare launcher bytes with the shared manifest. Historical
successes and failed attempts are retained; no new live provider
qualification is claimed.
- Previous head `e75fde6098b0ddd8cec765bfb6ecaeecb88a26a6` passed
complete CI (56 successful checks, one neutral, four skipped) and
Greptile 5/5. [Historical complete
CI](https://github.com/paperclipai/paperclip/actions/runs/37489112305).
Those results are not claimed for the cleanup head.
- Frozen live application: `d7b696f9b8f79095233e9e3d56d23e6a6018dd48`.
Public package version: `0.0.0-cursor-verify.3d0c9b7761c6`. The
declaration cleanup preserves release pins and does not relabel that
tested artifact as a build of the new source. Mainline through
`e34abee670` was reconciled while preserving accepted-plan waits,
provider-capacity handling, and both Cursor and public-MCP fixtures.
- Clean normal installation, explicit Cursor setup and daemon resolution
passed on macOS ARM64, macOS x64 under Rosetta, and Linux x64. npm
lifecycle hooks ran without silently downloading Cursor.
- Historical v11 live matrix: **18/18 passed with cleanup** (nine local,
nine Daytona) after the generic mode and lifecycle cleanup. The campaign
has 23 attempts; all five failures and their diagnoses remain recorded.
Exact case identities, hashes and limits are in the readiness report.
All provider calls are real, use the explicit Luna model and
company-bound credentials, and run without qualification or
runtime-asset overrides.
- The immutable Daytona image is
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:d6259b6bba094702c13fc2283bd85550849c1c53145b656fb2746778f9fa1747`.
The public Daytona plugin is installed independently and its version is
checked.
- Recursive typecheck, full build, token gates and Runner
contract/conformance/replay checks passed on the frozen application. Its
complete Linux CI suite passed. The duplicate local full-suite command
was incomplete after timing failures; affected repeats passed, but that
command is not reported as a clean pass.
- Qualification fixtures passed typecheck, 1,675 Vitest tests (one
skip), 128 Node checks, three provider-free browser tests, and 150
focused lifecycle tests after the final diagnostic correction. The
affected legacy Cursor command file also passed all five tests after
removing its shorter 10-second override; it now inherits the suite’s
standard 15-second timeout. Greptile is 5/5 on `e75fde609` with no
unresolved review threads. CI results above are recorded separately from
historical build results.

## Risks

- Cursor v14 includes the updated ACPX dependency patch and release
identity. The v11 live matrix and image below remain historical
evidence. They do not certify new v14 package/image artifacts.

- ACPX accepts models beyond the qualification fixtures. Availability
and entitlement depend on the provider. Successful configuration is not
a claim of live qualification for every model.
- Shared mode is an opaque identifier. Provider adapters own its
meaning. Incompatible historical sessions remain fenced; exact committed
plan waits and task history remain inspectable.
- Native AskQuestion is excluded. Paperclip semantic questions are
supported. Authoritative per-run dollar accounting is unavailable;
partial counters remain diagnostics and unknown cost is not zero.
- Image input, detailed native diffs, deeper child transcripts and
native plan-file export remain follow-ups.
- macOS x64 has clean-install and daemon-startup proof under Rosetta,
not a separate live campaign on Intel hardware.
- Release only the tested package/image combination. Merging this PR
does not publish npm packages or deploy that image. Later builds need
their own release verification. Rollback disables new Cursor admission
while preserving records and recovery inspection.
- A model can fail an exact instruction: one cancelled-plan attempt
returned the wrong summary marker despite correct cancellation. The
unchanged repeat passed; both results remain in the report.

> ROADMAP.md was checked. This completes existing native Runner/Cursor
work; it does not add an independent core feature proposal.

## Model Used

OpenAI Codex, GPT-6. The exact serving variant and context window are
not exposed in this session. The agent used reasoning, repository
inspection, code execution, protocol tests and browser-backed Product
E2E tools. Cursor acceptance uses the explicit
`gpt-5.6-luna[context=272k,reasoning=medium,fast=false]` model. That is
the evaluated provider model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — affected suites passed;
full CI and the retained local failed attempts are recorded separately
above.
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green — 56 successful checks, one
neutral and four skipped on `f7ec5cc1f0e30c62a829c016ff2013a2a9d79037`
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups —
fresh Base review passed on `f7ec5cc1f0e30c62a829c016ff2013a2a9d79037`;
zero new comments and no unresolved threads. The earlier Apex finding
remains fixed.
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-06 20:48:15 -05:00
6c36c07a4f feat(adapters): add GPT-6.1 Sol and refresh shared coding harness pins (#14942)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents run through coding-agent adapters and the native runner. Both
use the same installed provider CLIs, model catalogs, and reasoning
controls.
> - OpenAI released GPT-6.1 Sol (`gpt-6.1-sol`) in Codex. Anthropic
released Claude Sonnet 5.5. The static Codex, Bedrock, and OpenCode
catalogs do not list these IDs.
> - The shared provider pack pins Codex 0.156.0 and OpenCode 1.18.32.
The evaluation image pins older Grok, Gemini, Kimi, Cursor, and GitHub
CLI releases. Codex 0.156.0 has no bundled metadata for GPT-6.1 Sol.
> - A model entry without a current harness, or a harness pin without
its runner integrity checks, fails at run time.
> - This pull request adds the verified model IDs and moves the harness
pins, executable digests, controller checks, and image pins together.
> - The benefit is that operators can select the current models, and the
native and local adapters share one current CLI installation.

## Linked Issues or Issue Description

Refs #13829 and #13838 (the September 22, 2026 model and harness
refresh). Related pull requests: #14993 (merged October 5, 2026,
superseding #14816) added the direct Claude Sonnet 5.5 entry and
refreshed the Claude runtime to Agent SDK 0.3.286 / Claude Code 2.1.286.
This pull request does not change the Claude runtime or the direct
Claude model list; it keeps the #14993 pins and adds only the Bedrock
Sonnet 5.5 ID. After #14993 merged, this branch was rebased onto
`master` (October 5, 2026). The six overlapping pin regions
(`docker/daytona-runner/Dockerfile`, `docker/daytona-runner/README.md`,
`package.json`, `pnpm-workspace.yaml`,
`packages/adapters/claude-local/src/index.test.ts`,
`packages/paperclip-runner/src/backends/native-backend-factory.test.ts`)
were resolved by keeping this pull request's Codex 0.160.0 and OpenCode
1.18.34 pins next to #14993's Claude 0.3.286 / 2.1.286 pins, taking the
union of the Sonnet 5.5 model IDs in the Claude test, and merging both
README paragraphs. The Sonnet 5.5 effort and CLI-gate lines in the
Claude adapter were identical in both pull requests and merged without a
diff. #14917 and #14918 reordered the Claude and Codex model lists
earlier; the new entries sit where those ordering rules put them.

Sources checked on 2026-10-02:

- [OpenAI Codex models](https://learn.chatgpt.com/docs/models): GPT-6.1
Sol uses `gpt-6.1-sol`, supports reasoning efforts from Light to Ultra,
and has Standard and Fast modes at launch. The page also records that
`gpt-5.4` and `gpt-5.4-mini` retired from Codex with ChatGPT sign-in on
August 31, 2026, and that `gpt-5.5` retires on October 14, 2026. Neither
retirement applies to the OpenAI API.
- [Codex CLI releases](https://github.com/openai/codex/releases) 0.157.0
through 0.160.0. The bundled model metadata in the 0.160.0 Linux binary
contains `gpt-6.1-sol`.
- [Claude Sonnet
5.5](https://platform.claude.com/docs/en/models/sonnet-5-5/overview):
Bedrock ID `anthropic.claude-sonnet-5-5`, released September 28, 2026.
- [OpenCode releases](https://github.com/anomalyco/opencode/releases)
1.18.33 and 1.18.34 (fixes only). The OpenCode model registry lists both
added provider-qualified IDs.
- npm `latest` tags for `@xai-official/grok` 1.0.46,
`@google/gemini-cli` 0.62.0, and `@moonshot-ai/kimi-code` 2.1.1.
[xAI](https://docs.x.ai/docs/models),
[Google](https://ai.google.dev/gemini-api/docs/models), and
[Kimi](https://www.kimi.com/code/docs/en/kimi-code/models.html) list no
newer coding models.
- Cursor CLI 2026.10.01-e373342 is the version the official installer
resolves. The pinned digest is the SHA-256 of the versioned Linux x64
archive.
- [GitHub CLI 2.102.0](https://github.com/cli/cli/releases/tag/v2.102.0)
(security fixes). The pinned digest matches the release `checksums.txt`.

## What Changed

- Codex adapter: add `gpt-6.1-sol` to the model list, the Fast mode
list, and the Ultra effort set. It is the first entry: #14918 orders the
list newest version first, and its description notes the ChatGPT app
lists GPT-6.1 Sol first. Update the adapter documentation text.
- Claude adapter: add `us.anthropic.claude-sonnet-5-5` (Bedrock Sonnet
5.5) to the Bedrock catalog in the newest-Sonnet slot after Opus 5.5;
`us.anthropic.claude-sonnet-5` moves into the older-Sonnet group,
matching what `sortClaudeModels` from #14917 produces at runtime. Any
Sonnet 5.5 ID (direct or Bedrock-qualified) now gets the documented
`xhigh` and `max` efforts and requires Claude Code 2.1.284 or later on
the CLI lane (the Claude Code changelog entry for 2.1.284 adds
`claude-sonnet-5-5`). These two lines are identical to the ones #14993
merged, so the branch carries no diff for them.
- OpenCode adapter: add `openai/gpt-6.1-sol` and
`anthropic/claude-sonnet-5-5` to the static fallback catalog.
- Codex runtime pin 0.156.0 → 0.160.0 in the root and workspace
overrides, the runner package, the Codex ACP package patch, the
qualified ACPX profiles, the Linux x64 executable digest, the Rust
provider backend and its tests, the provider-pack manifest pins, the
remote controller pins, the sandbox npm install spec, and the opt-in
qualification scripts.
- Remote Codex compatibility window: upper bound 0.157.0 → 0.161.0. The
minimum stays at 0.149.0.
- OpenCode runtime pin 1.18.32 → 1.18.34 in the runner package, the
materialization script, the server and Rust qualified versions, the eval
and live-session labels, fixtures, and the configuration label.
- Evaluation image (`docker/daytona-runner/Dockerfile`): Grok CLI
1.0.46, Gemini CLI 0.62.0, Kimi Code 2.1.1, Cursor CLI
2026.10.01-e373342 with its digest, GitHub CLI 2.102.0 with its digest,
Codex and OpenCode version probes, and the refreshed lockfile digest.
The Claude Code 2.1.286 probe comes from #14993 and is unchanged here.
- `pnpm-lock.yaml` is not part of this pull request. The repository's
pull request gate rejects lockfile edits, and the refresh bot
regenerates the lockfile on master (the same flow #13838 used). The
Dockerfile `PAPERCLIP_RUNNER_LOCK_SHA256` default is the digest of the
lockfile that `pnpm install --resolution-only --ignore-scripts
--no-frozen-lockfile` (the refresh workflow's command) produces for the
combined pins on the rebased branch (`e1856797…`); that lockfile differs
from master only in the `@openai/codex` 0.160.0 platform packages, the
`@anthropic-ai/claude-agent-sdk` 0.3.286 override that #14993 introduced
(the open refresh-bot pull request #14872 carries that part),
`opencode-ai` 1.18.34 with its Linux x64 baseline, and the `codex-acp`
patch hash.
- Documentation: runner README, runner compatibility doc, environment
variable example, and a new `doc/adapter-model-audit-2026-10-02.md` with
sources and deferred items.
- Tests: Codex adapter catalog, server adapter models, Codex
compatibility window, native session executor pins, runner package
contract, OpenCode materialization, and UI effort options.

Unchanged on purpose: Claude Agent SDK 0.3.286 / Claude Code 2.1.286
(already on `master` from #14993), ACP bridges (`acpx` 0.13.1,
`claude-agent-acp` 0.73.0, `codex-acp` 1.6.2; newer upstream releases
need a separate qualification), the native Grok runtime 1.0.13, Pi
0.84.2 / 0.87.1 (the Pi 1.0 runner stack covers it), and Hermes 0.19.0
(current). `gpt-5.4` and `gpt-5.4-mini` stay in the picker because the
OpenAI API still serves them.

## Verification

Run on Linux x64 with Node 25.9.0 and pnpm 9.15.4 after `pnpm install
--no-frozen-lockfile` (the refreshed lockfile stays local; see above).
The results below were re-run on the rebased head (October 5, 2026) for
the suites the conflict resolution touches; the other rows are from the
original run and are covered by CI on every push:

- Rebased head: `packages/adapters/codex-local` 482 passed;
`packages/adapters/claude-local` 340 passed, 4 failed (`execute.remote`,
`test.probe`, `execute.acp-fallback`, `acp` spawn/env-hardening cases
that fail identically on unchanged `master` in this host environment);
`server` adapter-models + codex-runtime-compatibility +
native-session-executor + adapter-registry 607 passed, 1 failed (the
same adapter-registry override-pause case as before, also failing on
`master` here); `packages/paperclip-runner` native-backend-factory +
qualified-profiles 36 passed; `ui` codex-reasoning-effort +
config-fields + model-utils 19 passed. Rust, full typecheck, build, and
the Docker image are left to CI as before.

- `vitest run` in `packages/adapters/codex-local`: 13 passed. `vitest
run` in `packages/adapters/claude-local` (whole package, including the
new Sonnet 5.5 gate and effort tests): see the latest CI run and the
comment below. `vitest run` in `packages/adapters/opencode-local`: 48
passed, 1 failed (`runtime-config.test.ts` reads the host
`PAPERCLIP_OPENCODE_PROVIDERS` variable; it fails the same way on the
unchanged base).
- `vitest run src/__tests__/adapter-models.test.ts
src/services/native-runtime/codex-runtime-compatibility.test.ts
src/__tests__/adapter-registry.test.ts` in `server`: 84 passed, 1 failed
(`adapter-registry.test.ts` override pause test; it fails the same way
on the unchanged base).
- `vitest run` in `ui` for `codex-reasoning-effort`,
`agent-setup-fields`, `config-fields`, and `ComposerRunSettingsPicker`:
25 passed.
- `node --test test/acpx-codex-package-contract.test.mjs
scripts/materialize-opencode-binary.test.mjs
scripts/runner-protocol-eval-campaign.test.mjs` in
`packages/paperclip-runner`: 23 passed. The package contract test
verifies the installed Codex ACP executable digest and the 0.160.0 patch
pin.
- `vitest run src/drivers/acpx src/backends src/drivers/opencode
src/live/live-session.test.ts` in `packages/paperclip-runner`: 626
passed, 5 failed, 1 skipped. The 5 failures
(`installation-integrity.test.ts` `/proc/self/fd` module loading and one
OpenCode answer-selection test) also fail on the unchanged base under
Node 25; Linux CI runs Node 24.
- `pnpm run test:opencode:qualification` in `packages/paperclip-runner`
against the installed OpenCode 1.18.34 executable: passed.
- `codex --version` from the installed pack prints `codex-cli 0.160.0`.
The Linux x64 executable digest `12eb3e81…652aad` was computed from the
`@openai/codex@0.160.0-linux-x64` archive after checking its registry
`dist.integrity`.
- `pnpm check:token-gates`: all gates clean.
- `pnpm run typecheck:typescript` in `packages/paperclip-runner`:
passed. Package typechecks ran one at a time; see the comment below for
the server and UI results.

Not run here, and needed from CI:

- Rust tests and `pnpm -r typecheck` / `pnpm build` for the server (no
`cargo` in this environment; the server typecheck prepares the runner
vendor build).
- The Docker evaluation image build and the real-binary Codex startup
and session-resume probes (no Docker; the probes need the compiled
`paperclip-runnerd`). The trusted CI runner workflow covers them.
- Authenticated inference with any new model. This change is metadata
and startup validation only.

## Risks

- Codex 0.160.0 changes the bundled model catalog and app-server
behaviour (authoritative provider catalogs, incremental running-turn
tracking). The patched `codex-acp` 1.6.2 bridge is unchanged and
declares `^0.148.0`; it worked with 0.156.0 under the same override. If
CI probes show a protocol change, the pin can return to 0.156.0 by
reverting this pull request.
- The compatibility window upper bound moves to `<0.161.0`. Remote
images with Codex 0.157 to 0.160 become accepted. Older images stay
accepted down to 0.149.0.
- Until the refresh bot lands the regenerated lockfile on master, the
Dockerfile lockfile digest default does not match the committed
lockfile. The trusted CI workflow computes the digest from its own
resolution at build time, so this affects only a local build that passes
no digest.
- Existing saved model selections and effort settings are not changed.
Agents on `gpt-5.4` or `gpt-5.5` with ChatGPT sign-in need a model
change before the OpenAI retirement dates; that is documented, not
enforced.
- Rollout order: deploy the controller and runner from this change
before promoting a sandbox image that carries these pins. Older
controllers reject the new provider-pack pins.

## Model Used

- Claude Fable 5.1 (Anthropic, model ID `claude-fable-5-1`), 1M context
window, adaptive thinking, tool use. The model ran as a Paperclip agent
through the Claude Code harness, performed the web research, edited the
code, and ran the tests listed above.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Bender (Fable) <noreply@paperclip.ing>
Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-05 13:59:08 -07:00
DottaandPaperclip 24beb00575 feat(runner): add rich ACP transport and durable interaction foundation (#14430)
Add shared rich ACP transport, durable questions and permissions, verified provider packaging, and bounded activity and plan presentation. Keep Cursor, Copilot, and Pi pending their separate provider qualification.

Persist interaction settlement before publication, fence failed writes until fresh recovery, and preserve owned-process cleanup. Incorporate reviewed mainline integration with extended harness coverage.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-29 08:56:21 -05:00
DottaandPaperclip f1a394bd30 feat(runner): add Grok Build through native ACP (#13882)
## Thinking Path

> - Paperclip manages AI agents and governs their work.
> - Its native runner uses structured provider protocols for sessions
and tools.
> - Grok Build supports ACP over stdio, but the runner did not expose
it.
> - Native execution requires company-scoped credentials, verified
identities, and permission gates.
> - This change adds Grok through ACPX for local and Daytona execution.
> - Subscription login and explicit API-key execution have separate
credential paths.
> - Qualification grades real tool outcomes, durable state, and browser
workflows.

## Linked Issues or Issue Description

Refs #13845, #13847, #13850, #13878, #13901, #13973, #13977, #13979.

Add **Grok Build** to `paperclip_runner` with `provider: "acpx"`,
`acpxAgent: "grok"`, and model `grok-4.7`. Existing legacy Grok agents
keep their adapter. Merge the three companion fixes (#13973, #13977,
#13979) before treating the integrated Product qualification as deployed
behavior.

## What Changed

- Synchronize shared, TypeScript, Rust, server, validation, and UI
provider contracts.
- Run Grok native ACP stdio through ACPX and the authenticated Paperclip
MCP bridge. Verify the pinned executable and exact ACP model identity.
- Prefer company subscription login. Support an explicit company-secret
API key without automatic paid fallback. Fence refresh and copyback to
the same account and remove private runtime credentials after
containment.
- Preserve selected permissions, cancellation, durable session identity,
resume, and restart recovery. Keep unsupported steering and goals
unavailable. Preserve missing usage and cost as unknown.
- Package checksum-verified Grok Build 1.0.13 for Daytona with an
immutable, signed image built on EC2.
- Add deterministic admission, protocol, permissions, identity,
credential, failure, and cleanup checks. Add the maintained 39-case
protocol roster and separate subscription/API Product profiles.
- Fix live-test findings in reasoning events, reloads, idle-owner
retirement, credential-home cleanup, expired-login model discovery,
launcher pinning, and rerun evidence selection.
- Align control-plane state readers with the transport's 64 MiB bound
while retaining identity, ownership, lifecycle, and size rejection
checks.
- Stabilize two asynchronous CI assertions while retaining actual
outcome and filesystem-evidence checks.

## Verification

Current integration head `f114948376056fe0b6b34c1496ae8667b59daa63`
includes master `3447609d2247e75e55d91493dda91a608364f672` (2026-09-28).
Two master advances during verification overlapped the eval catalog; the
final merge preserves Grok qualification, completion updates, and
bounded API-response reading in all 348 cells. All 77 focused
catalog/eval/workflow tests pass. Both native stack layers (#14397) are
mergeable, and both exact-head Greptile reviews are 5/5 with successful
security scans and no unresolved review threads. All current-head CI is
green: 56 successful checks/statuses and four intentional skips ([CI
run](https://github.com/paperclipai/paperclip/actions/runs/36447097232)).
Trunk code-owner requirements remain enforced. The review summary’s
non-blocking saved-asset offset classification note concerns code
already merged in #14301; those runtime files are identical to master
and outside this stack’s diff. Historical live evidence below retains
its original source revisions.


Earlier integration checkpoint:
`24fc9b94ca0afb21ccdc8d26dbb2e4b258ad72cb`. Refreshed against master
`0f14d2612`, preserving Grok qualification alongside the new accounting
and lifecycle suites. All 124 focused catalog, evidence, and
service-worker checks pass. The current base workflow includes the
explicitly selected public-install verification lane; follow-up #14024
supplies its verifier script. CI at that earlier checkpoint was green
(56 successful checks/statuses, four intentional skips), and the review
is 5/5 with no unresolved findings. Prior feature CI at
`fd73f0a9b1ecdf4094685054028df71739ddc3e1` passed ([run
36148259902](https://github.com/paperclipai/paperclip/actions/runs/36148259902));
that is historical evidence, not a current-head result.

Paid Product measurements use frozen integrated source
`2d939a92b21dcaf5c77c88b54d96784d2ddd0699`, which combines the feature
with #13973, #13977, and #13979. That source passed all 52 CI checks and
clean 5/5 review. Later master syncs incorporate upstream changes. Their
checks remain separate from these pinned live measurements.

| Check | Result and source-pinned report |
| --- | --- |
| Subscription protocol roster | [39/39 first attempts; 206
assertions](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-36046839612-1/index.html),
runtime `bc6833f7`, evals `92bb4b8c` |
| API protocol roster | [39/39 first attempts; 206
assertions](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-35926577007-1/index.html),
runtime `4a1061c8`, evals `3213dbec` |
| Subscription full Product matrix | [16/16 first attempts; 144
assertions; cleanup
passed](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36096908572-1/index.html),
source `2d939a92` |
| Subscription core repetitions | 18/18: tool use, planning approval,
and Stop/resume each passed three times in local and Daytona profiles.
The full matrix contains repetition one; [repeat
two](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36104551060-1/index.html)
and [repeat
three](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36108100404-1/index.html)
each passed 6/6. Total: 28 unique subscription attempts at `2d939a92`. |
| API smoke and question continuation | [4/4 first attempts; cleanup
passed](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36147315401-1/index.html),
both environments at `2d939a92` |
| Historical API Product coverage | [16/16 full
matrix](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35875144860-1/index.html)
and 18/18 core repetitions at `4a1061c8`; retained as measurements of
that revision |
| Native Daytona proof | Three subscription and three API
MCP/permissions/resume runs passed at `bc6833f7`. Three expired-login
admission and fenced refresh checks passed without inference. All test
sandboxes were removed. |
| Inspectable artifacts and UI | Current-source screenshots verify
planning approval, direct Ask completion, question continuation after
controller restart, and two downloadable project revisions. The project
downloads pass 12 and 18 tests; all 40 independent artifact oracle
checks pass. |
| Provider-free checks | 116 eval-validator tests, 39 Grok definitions,
and 359 enabled/external campaign cells pass. Continuation regressions
above 2 MiB and 16 MiB failed before their fixes; 32 focused
recovery/ownership/size checks pass. |

The 32 unique current-source Product attempts have no failures, retries,
or skipped cells, and all cleanup checks pass. Whole-workflow timing,
model identity, image and provider-pack provenance, attempts, and
accounting coverage are retained in the canonical reports. The report
publisher's conservative `complete=false` flag is preserved; independent
audits verify the exact selected source catalog and immutable result
rows.

Pins: Grok Build `1.0.13 (5e9a58528b76)`, ACPX `0.13.1`, ACP model
`grok-4.7`. Linux binary SHA-256:
`edf79521581bb5e6b95abef848491a6a742e860da3e237ebe86a280d30dce4c1`.
Launcher SHA-256:
`f0b698395a3704ed2ffaf84ea19bdb20c36c8a0a70b7c629c7b6ffe144e59e55`.
Image:
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:76b24edfd850219e949418b19e4ceba690e84d51d199ade426e484953329b5e9`.
Image build source is `4196a4cd`, recorded separately from application
source `2d939a92`; each campaign verifies the image signature and
provider pack.

Original failed campaigns remain available: [continuation
bound](https://github.com/paperclipai/paperclip/actions/runs/36057718059),
[scheduler/event
capture](https://github.com/paperclipai/paperclip/actions/runs/36071063537),
and [startup cleanup plus EC2
interruption](https://github.com/paperclipai/paperclip/actions/runs/36080870743).
They retain their original grades. No Docker or Rust builds ran on the
developer laptop for these follow-ups.

## Risks

Merge packaging follow-up #14024 with this base before public release.
The follow-up replaces the private Grok bridge package with a built-in
launcher and makes the native binary an explicit sandbox prerequisite.

Three separate, reviewed fixes are part of the tested integrated
behavior: #13973 serializes task-run admission; #13977 captures complete
event evidence; #13979 durably reconciles failed Daytona creation. Each
has green CI and clean 5/5 review. Failed-create recovery has 277 plugin
tests, 92 SDK tests, host-runtime recovery tests, and a real Daytona
lost-deletion-receipt proof. The live proof uses a private file for
journal persistence; database durability is covered by host tests.
Worker death before delivery of a failure envelope remains outside that
recovery mechanism.

Subscription fixtures stage an authorized company login; interactive
browser sign-in is not qualified. Local Product profiles ran on EC2
Linux. The temporary subscription credential was removed from the
protected GitHub environment after all subscription audits, with absence
verified. Runtime homes and refresh copyback remain ownership-fenced.

Protocol results remain pinned to their original revisions; they are not
relabeled as tests of the latest feature commit. New binary/model
versions require qualification. Missing token usage and model cost
remain unknown; runtime estimates do not establish a full bill.
Automatic paid Grok scheduling remains disabled pending separate
reviewed enablement. The 64 MiB bound can increase memory use for
verbose sessions, and larger files still fail closed. No automatic
legacy-agent migration occurs.

## Model Used

OpenAI GPT-6 through Codex, with tool use and code execution. The exact
serving model identifier and context-window size are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-28 13:54:36 -05:00
Devin FoleyandPaperclip be6f49a425 feat(runner): refresh shared coding harness runtimes (#13838)
## Thinking Path

> - Paperclip runs agents through local adapters and the native runner.
> - Both paths must use the same installed provider CLI.
> - New models require current harness releases.
> - The runner still pins Codex 0.153.4, Claude SDK 0.3.263, and
OpenCode 1.18.29.
> - Changing the image alone would fail the runner's exact version and
executable checks.
> - This pull request updates those dependencies, integrity checks,
controller checks, and image pins together.
> - Shared installations can then run the current models without a
task-time download.

## Linked Issues or Issue Description

Refs #13829, which updates model choices and reasoning controls.
Searches found no open PR that updates these runtime pins.

**Current behavior**

The shared provider pack ships old CLIs. Claude Code 2.1.263 cannot run
Opus 5.5, which requires 2.1.280. Remote controllers reject provider
packs whose versions differ from their declared pins.

**Proposed behavior**

Use Codex 0.156.0, Claude Agent SDK 0.3.280 / Claude Code 2.1.280, and
OpenCode 1.18.32 throughout the runner. Keep the reviewed ACP bridge
patches and one shared CLI installation per provider.

**Reason and benefit**

Current harnesses support the new model IDs while preserving executable
verification and remote provider-pack compatibility checks.

## What Changed

- Update dependency overrides, the Codex ACP package patch, runtime
profiles, and remote controller pins.
- Verify the new Claude Linux x64 and macOS arm64/x64 executables and
Codex Linux x64 executable against integrity-verified npm archives.
- Refresh OpenCode version checks, fixtures, and the runner
configuration label.
- Refresh the eval image's Grok, Gemini, Kimi, Cursor, and GitHub CLI
pins and archive hashes. Hermes remains current at 0.19.0.
- Refresh the build-time lock digest from clean pnpm 9.15.4 resolution.
Leave lockfile commits to repository automation.
- Document model compatibility and the separation between CLI runtimes
and patched ACP bridges.

## Verification

- `pnpm -r typecheck` and `pnpm build` passed.
- Rust workspace release tests passed.
- Package/patch and OpenCode binary-materialization contract tests: 11
passed.
- Real Codex 0.156.0 startup-ownership and paginated session-resume
probes passed with isolated synthetic homes and no model turn.
- Codex app-server `thread/start` preserved `gpt-6-sol` and
`gpt-6-luna`; no `turn/start` was sent. An unauthenticated built-in
catalog does not include those account-served entries.
- Installed Claude integrity probes passed for `claude-opus-5-5` and
`claude-fable-5-1`.
- `pnpm --filter @paperclipai/paperclip-runner
test:opencode:qualification` passed with the actual OpenCode 1.18.32
executable under Node 24 and Node 25. The loopback provider exercise
covers health/version, session creation/read/delete, SSE, and a
completed async prompt.
- `pnpm check:token-gates` passed.
- The targeted runner suite passed 130 tests. Three macOS failures in
snapshot module lookup and OpenCode final-message selection also
reproduce on the unchanged base; Linux CI will provide the platform
check.
- [Final Linux
CI](https://github.com/paperclipai/paperclip/actions/runs/35798076399):
all gates passed. Four jobs needed one retry after their CI workers
received shutdown signals. The PR has 55 successful checks, two skipped
checks, Greptile 5/5, and no unresolved review threads.
- Changed runner configuration UI tests: 5 passed.
- Full macOS `pnpm test:run` reached 13,094 passing server tests, 84
skipped, and 18 failures before the wrapper stopped. Failures involved
skill-cache publication permissions, missing bundled connector skills in
the worktree, and a conversation-reset timing case. The 10 cache
permission failures reproduce on the unchanged base; both
conversation-reset cases passed on a targeted retry. The wrapper did not
reach its later workspace/serialized groups locally; Linux CI covers
those groups.
- The local Docker daemon did not respond, so no local Docker build was
run. No billable model requests were made.

## Risks

- Deploy the matching controller and provider pack together. Older
controllers enforce their previous exact pins.
- Current upstream CLIs can change behavior. Existing protocol tests and
isolated real Codex probes cover the integration boundaries;
authenticated model inference is not part of these checks.
- ACP bridge package versions and executable digests stay unchanged
because their executable bytes are unchanged. Only the underlying
CLI/SDK dependencies move.
- No schema migration. Revert the runtime and image pins together to
roll back.

## Model Used

OpenAI GPT-6 via Codex, with repository tools, code execution, and web
research. The exact serving model ID and context window were not exposed
by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass for the changed surfaces
and real-executable probes; full macOS-suite limitations are listed
above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-22 17:02:29 -07:00
DottaandPaperclip f6a211479f fix: share current CLI runtimes across sandbox adapters (#12994)
## Thinking Path

- Paperclip Runner needs its runtime preinstalled for fast sandbox
startup.
- Native and local adapters should launch one current CLI installation
per provider.
- An older global copy can shadow that installation, and exact native
compatibility pins must match it.
- Update the qualified releases and binary digests, expose shared CLI
entrypoints from the provider pack, and prefer the image-owned bin
directory.
- Keep dependency installation in the image build; task startup only
discovers, links, and verifies artifacts.

## Linked Issues or Issue Description

**What happened?**
Remote native startup rejected a stale global Codex, while CLI-only
images lacked runnerd entirely.

**Expected behavior**
An image-baked runtime starts without uploading binaries or installing
packages. All adapters share the same current provider CLI.

**Steps to reproduce**
Start a native remote task with the old global Codex and the updated
runtime available only under `/opt/paperclip-runner/bin`.

**Paperclip version or commit**
Discovery behavior at `54a99d884`.

**Deployment mode**
Docker with a remote sandbox.

## What Changed

- Prefer `/opt/paperclip-runner/bin`, then the user's local bin
directory, then PATH. Existing metadata and version validation remains
in force.
- Qualify Codex 0.153.4, OpenCode 1.18.29, and Claude SDK 0.3.263 / CLI
2.1.263. Update binary digests, TypeScript/Rust checks, registry
defaults, and the displayed OpenCode version together.
- Share Codex and Claude's native executable with the ACP bridges
through exact dependency overrides. Preserve the separately qualified
ACP bridge implementations and their security patches.
- Expose shared provider-pack CLI launchers; fail the pack build if
Codex ACP resolves a separate Codex installation. Update the eval
image's other agent CLIs to current stable releases and remove duplicate
global provider installs.
- Document the single-current-CLI policy in source comments and
development guidance. Latest stable releases are resolved at
review/build preparation and pinned; task startup never auto-updates.

## Verification

- Native-session and adapter-registry suites: 158 tests passed.
- Provider suites: 88 tests passed, 7 Linux-only checks skipped on
macOS. One existing macOS temporary-path alias assertion passed when
rerun with canonical `TMPDIR=/private/tmp`.
- Package-contract and OpenCode materialization tests: 11 passed.
- Full typecheck, build, and token gates passed. Rust
native-provider/recovery tests: 19 passed.
- Broad local suite: 5,974 passed, 23 failed, 41 skipped. Failures are
in unchanged macOS workspace/path/port and connection suites; focused
runtime tests pass. All latest-head Linux PR checks passed, including
the full test shards, typecheck, build, runner verification, browser
suites, and canary dry run.
- The standalone fleet image built with one current provider CLI each
and passed native Codex/Claude binary-integrity checks. A disposable
Daytona sandbox reported ready in 798 ms; its baked runner completed an
API-key `gpt-5.6-luna` turn in 2,430 ms and returned the expected marker
with a usage receipt. No runtime artifacts were uploaded or installed.
- The normal shared `codex exec` entrypoint also completed an API-key
`gpt-5.6-luna` turn in 2,321 ms.
- Both image builds verify the complete generated lockfile against a
reviewed SHA-256 before package installation or lifecycle execution.
Root lockfile changes remain CI-owned. Merge and rollout remain on hold
for operator review.

## Risks

- Updating provider CLIs changes their behavior for all adapters;
version probes and live native smoke testing are required before image
promotion.
- The image-owned directory takes precedence. Its entries must launch
the same shared CLI as the global PATH, not a private older/newer copy.
- Application qualification pins and the deployed image must move
together. No startup fallback installation is added.
- No schema or authentication-policy changes.

## Model Used

OpenAI GPT-6 (Codex). The session does not expose a more specific model
ID or context-window size. Used reasoning, repository inspection, code
execution, and browser verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-07 10:09:29 -05:00
DottaandPaperclip 83987210d6 fix(runner): align direct eval provider setup with qualified runtimes (#12945)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner direct live evals test semantic tools against a mock control
plane.
> - The first complete AWS campaign exercised 358 cells.
> - It exposed setup differences from the working full-stack harness.
> - This pull request corrects those direct-harness differences.
> - It preserves production permission defaults and the full-stack
workflow.

## Linked Issues or Issue Description

**What happened?**

Native Codex cells could not find a global codex executable. ACPX denied
unattended tool requests and lost valid provider usage receipts.
AgentCore hit a 30-second facade timeout while its worker allows 120
seconds for delivery. Several models reported a native run result
without updating the separate mock task state.

**What did you expect?**

The direct harness should use the pinned executable, explicit test
permissions, and a timeout compatible with the provider delivery
contract. Its instructions should explain which operation changes mock
task state.

**Steps to reproduce**

Run the full Runner Direct Live Protocol Evals workflow. Baseline
campaign:
https://github.com/paperclipai/paperclip/actions/runs/34059009921.

**Paperclip version**

Master at b55ce03e86. Related public
change: #12932. No duplicate fix was found.

## What Changed

- Resolve native Codex from the pinned Codex ACP dependency, as the
full-stack launcher does.
- Select explicit unattended ACPX permissions only in the isolated
direct eval harness.
- Add a configurable bounded turn-admission wait. AgentCore direct evals
use up to 125 seconds, capped by their turn budget. Other callers retain
the current default waits.
- Explain mock task-state operations separately from native run-result
reporting. Keep all scoring assertions unchanged.
- Bind the direct turn's scope to the current request, rather than stale
shared fixture notes. A provider's end-of-turn result does not authorize
an unrequested mock task completion or extra completion comment.
- Normalize the qualified Claude/Codex usage semantics without
double-counting reasoning or inventing unknown billable categories.
Forward ACPX's persisted terminal prompt-response usage for the exact
current turn; reject stale or ambiguous receipts.
- Preserve the known numeric ACPX token-counter aliases through durable
redaction. Continue redacting strings and credential-shaped values. A
regression test reproduces the previously redacted usage before
normalization.
- Add regression tests and operating documentation.

## Verification

- Passed Runner TypeScript build.
- Passed 88 focused tests across the eval request contract, provider
setup, and Runner transport.
- Passed 93 focused ACPX adapter, sidecar lifecycle, and
usage-accounting tests, including persisted receipt identity and
missing-field regressions.
- Passed 97 additional runtime-host, in-process ACPX driver, OpenCode
proxy, and MCP bridge tests.
- Reproduced the ACPX counter-redaction failure, then passed 36 Rust
decoder/normalizer and durable-redaction tests after the fix.
Credential-shaped strings and objects remain redacted.
- Native Codex get-task-context passed a real provider probe with no
global CLI dependency.
- Local ACPX probes reached a platform startup rejection on macOS;
qualification therefore used the intended AWS Linux fleet. Complete
campaign 34060573948 reached 341/358 passing with zero infrastructure
failures (up from 250/358 and 100 infrastructure failures). ACPX Claude
reached 34/35, ACPX Codex 7/8, native Codex 34/35, and AgentCore 35/35.
Remaining failures were retained in the canonical report:
https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-34060573948-1/.
- Final complete campaign 34062394019 exercises the current-request
scope clarification with all 358 cells; results pending.
- Passed git diff --check.
- Repo-wide typecheck, build, and tests are delegated to PR CI. They
were not repeated on this machine.

## Risks

- ACPX approval is scoped to the operator-requested isolated fixture
harness. Production defaults do not change.
- AgentCore admission can wait longer, but remains within the whole-turn
budget.
- The mock fixture has no production task lifecycle service. Explicit
task-state instructions describe that boundary; they do not relax
scoring.
- This change does not modify the browser full-stack workflow or its
test fixtures.
- ACPX receipt corrections apply to all native ACPX sessions. They rely
on the pinned qualified server contracts; unknown or ambiguous receipts
stay unknown, and a final accounting-read error never overwrites the
provider's terminal result.

## Model Used

- OpenAI GPT-6 (`gpt-6-astra`) through Codex, with reasoning, tool use,
and code execution. The session does not expose an exact context-window
limit.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-06 17:21:12 -05:00
DottaandPaperclip fee8d8dc39 fix(runner): repair direct live provider bootstrap (#12932)
## Thinking Path

> - Paperclip runs AI agents through qualified provider backends.
> - The direct live eval workflow builds one immutable Runner runtime
for every matrix cell.
> - The workflow reinstalled the packed Runner with npm.
> - That install discarded pnpm patches and selected provider
dependencies outside the qualified lock.
> - The first pnpm deployment model also placed its virtual-store marker
at the wrong level; a real deployment keeps `.pnpm` beside the scoped
Runner package.
> - AgentCore enforced the current context-aware harness but the direct
eval CLI did not supply the production v3 runtime context that harness
requires.
> - This pull request preserves the qualified dependency graph, resolves
the real deployment layout, and makes direct evals exercise the
production runtime-context contract.
> - The benefit is that live eval cells reach their provider turn with
the same artifacts and context contract that Paperclip qualified.

## Linked Issues or Issue Description

Refs: #12931

**What happened?**

The full direct live eval campaign failed every ACPX cell during
`session.open`. The portable runtime had an incorrect dependency root.
Its npm install also discarded the qualified ACP server patches.
AgentCore cells first failed because Runner enforced
`aws-agentcore-harness-v1` while the provisioned stack and eval profile
use `aws-agentcore-harness-context-v2`; after aligning that revision,
the direct eval CLI still omitted the required v3 runtime context.

**Expected behavior**

The direct eval runtime must preserve the frozen pnpm dependency graph
and patched provider bytes. Runner, server validation, OpenAPI, and the
deployed AgentCore stack must use one qualification revision. Direct
eval attempts must supply the same immutable native runtime-context
contract as production.

**Steps to reproduce**

1. Dispatch `Runner Direct Live Protocol Evals` from `master`.
2. Select an ACPX Claude, ACPX Codex, or AgentCore roster.
3. Observe a pre-turn provider bootstrap failure.

**Paperclip version or commit**

`d96452db059338b329b458ba8fe359fef72f1363`

**Deployment mode**

GitHub Actions on the RunsOn Linux x64 fleet.

## What Changed

- Build the reusable direct-eval runtime with `pnpm deploy --prod`.
- Resolve ACPX dependencies from the actual scoped-package layout of a
self-contained pnpm deployment.
- Align AgentCore configuration and qualification checks on
`aws-agentcore-harness-context-v2`.
- Materialize a minimal immutable v3 runtime context for each isolated
direct eval attempt.
- Add workflow, package-authority, runtime-context, Rust, and server
regression coverage.
- Document the qualified packaging, runtime-context, and AgentCore
revision contracts.

## Verification

- `pnpm --filter @paperclipai/paperclip-runner typecheck`
- `pnpm --filter @paperclipai/server typecheck`
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/live/runnerd-codex-transport.test.ts` (70 tests)
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/cli/eval-session-contract.test.ts` (14 tests)
- Focused Runner contract tests (36 tests)
- Focused server profile tests (47 tests)
- Focused Rust managed-provider and native-selector tests (19 tests)
- `node --test
packages/paperclip-runner/scripts/runner-protocol-eval-workflow-security.test.mjs`
- `actionlint .github/workflows/runner-protocol-live-evals.yml`
- A local `pnpm deploy --prod` produced both qualified ACP server
digests.
- A Linux reproduction of the first follow-up smoke identified the real
deployment root and the missing AgentCore runtime context.

## Risks

The AgentCore revision change rejects profiles that still use the
obsolete v1 value. This is intentional because the provisioned
context-aware harness and current eval profile use v2. Direct eval
prompts now receive the same fixed runtime-context preamble as
production, so behavior scores may move; that is the intended
qualification surface. The workflow package layout changes, but tests
assert the new entrypoint and dependency root. This change does not
modify the browser full-stack E2E workflow.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex with GPT-5.6. The context-window size is not exposed in
this session. The model used extended reasoning, repository tools, code
execution, Docker-based Linux reproduction, and GitHub Actions
diagnostics.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (for example, `docs/...` or
`fix/...`) and contains no internal Paperclip ticket id or
instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-06 15:45:07 -05:00
Dotta af8439a70b feat(runner): restore direct live eval campaigns and reports (#12909)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Runner executes agents through native and managed provider
drivers.
> - The direct live eval layer had drifted from the current Runner
contracts.
> - The old local workflow did not provide a complete parallel campaign
or durable report history.
> - The Runner also needed current native OpenCode and OpenRouter
qualification.
> - This pull request restores the direct campaign, corrects the runtime
gaps that the campaign found, and adds safe hosted Evalbook history.
> - The benefit is repeatable model comparison against an immutable
Runner and eval source revision.

## Linked Issues or Issue Description

Refs #11297
Refs #11634

**What existing behavior does this improve?**

This improves the direct live `paperclip-runner` eval workflow, provider
execution contract, and static Evalbook reporting path.

**Current behavior**

The direct evals do not have one maintained full campaign on current
`master`. OpenCode has no qualified multi-model OpenRouter roster.
Parallel provider bursts can compact committed events before the
transport observes them. Local reports do not have a separate safe S3
history index.

**Proposed behavior**

Run one immutable roster-plus-case matrix. Use the shared paid AWS
runner fleet. Keep raw artifacts access-controlled. Publish a sanitized
canonical Evalbook report under the separate `runner-protocol-evals` S3
prefix. Keep immutable campaign directories plus root history, latest,
and latest-green pointers.

**Reason and benefit**

Maintainers can compare native Codex, native OpenCode, ACPX, Claude
Managed, and AWS AgentCore behavior over time. They can inspect failures
without mixing this direct protocol layer with browser full-stack E2E.

**Breaking changes**

None. The new workflow and S3 prefix are additive. The existing Runner
full-stack E2E workflow and report remain separate.

## What Changed

- Added a trusted two-shard direct live workflow for up to 393
roster-plus-case cells.
- Reused the numeric actor allowlist, protected paid environment, and
RunsOn fleet controls from Runner full-stack E2E.
- Added immutable Runner and eval revision resolution, exact credential
boundaries, bounded retries, and cost ceilings.
- Added a public report projection that removes sessions, transcripts,
tool payloads, state, traces, raw failures, remote profile identities,
and credential-shaped values.
- Added additive S3 history under `runner-protocol-evals`, with
immutable campaigns and mutable root index pointers.
- Added native OpenCode model injection and current OpenRouter pricing
contracts.
- Fixed direct eval completion, workflow execution, semantic discovery,
warm-attach state reset, executable binding, and event-burst handling.
- Kept Runner browser full-stack E2E behavior and publication separate.
- Documented local and hosted direct eval operation.

## Verification

- `pnpm --filter @paperclipai/paperclip-runner
test:runner-protocol-eval-publish` — 15 passed.
- `pnpm --filter @paperclipai/paperclip-runner build:typescript` —
passed.
- `actionlint .github/workflows/runner-protocol-live-evals.yml
.github/workflows/runner-full-stack-e2e.yml` — passed.
- Local current matrix at the revision in
[paperclip-evals#17](https://github.com/paperclipai/paperclip-evals/pull/17)
— 323 cells across 10 enabled configurations completed.
- Final local current matrix — 269 passed, 11 behavior failures, and 43
expected macOS-only ACPX platform failures.
- Targeted Runner checks — 13/13 eval-session tests, 15/15
publisher/security tests, and package typecheck passed; complete PR CI
is green, including all browser E2E shards.

## Risks

- Paid live campaigns can consume provider budget. Actor authorization,
exact per-cell ceilings, protected environments, and explicit schedule
enablement bound this risk.
- Public reports can leak provider data. The workflow publishes only a
separately projected report and validates every file before upload.
- The new workflow cannot publish until it is present on the default
branch. This pull request does not change the existing
`runner-full-stack-e2e` publication path.
- The campaign is large. It uses two GitHub matrices and caps combined
concurrency at the shared fleet limit.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex on GPT-5. The exact deployment ID and context-window size
are not exposed. The model used reasoning, code editing, browser
inspection, repository tools, and live provider execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-05 20:18:11 -05:00
Dotta 5716fe907e test(runner): add full-stack acceptance and eval gates (#12700)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The runner subsystem executes agent work across local and managed
provider backends.
> - The lower pull requests restore the task runtime, provider backends,
and managed-provider control plane.
> - The restored system needs repeatable full-stack checks before it can
ship safely.
> - Paid live checks also need clear access, cost, and secret controls.
> - This pull request adds acceptance, live evaluation, chaos, and
release gates for the restored runner stack.
> - The benefit is measurable runner parity with safer release
decisions.

## Linked Issues or Issue Description

**Subsystem affected**

Cross-cutting. This change covers runner tests, release workflows,
server contracts, and evaluation tools.

**Problem or motivation**

The runner stack did not have one complete acceptance surface for native
Codex, ACPX, Claude Managed, and AWS AgentCore. Release checks could
miss provider drift, task-view regressions, cost-policy errors, and
destructive cleanup errors.

**Proposed solution**

Add a 57-cell full-stack catalog, a Daytona image, and opt-in paid
workflows. Add live evaluation, chaos, cost-limit, redaction, and
release contract checks. Add AWS AgentCore infrastructure and guarded
provisioning tools. Keep the native runner experimental flag off by
default.

**Alternatives considered**

We considered manual smoke tests only. They do not give repeatable
evidence and they do not protect release branches. We also considered
one large pull request. The stacked pull requests keep each review below
the Greptile file limit.

**Roadmap alignment**

This work supports the shipped Cloud / Sandbox agents milestone and the
shipped Agent evals & feedback milestone in `ROADMAP.md`.

Related stack:

- #12699 adds managed provider backends and lifecycle support.
- #12691 adds qualified OpenCode and ACPX provider backends.
- #12685 restores task runtime rendering and steering.

## What Changed

- Add the runner full-stack harness with 57 catalog cells and 60 unit
tests.
- Add a Daytona runner image with digest-pinned base images and
base-aware image-content checks.
- Add guarded live evaluation and chaos workflows with a fixed
40-execution matrix; live and full-stack paid schedules now run only on
Sundays or by manual dispatch.
- Add in-flight reported-usage cost stops, post-turn cost caps,
exact-threshold failure classification, secret redaction, retry
classification, and actor authorization.
- Reattach stream and hard-budget listeners before restart-recovery
continuations so restored paid sessions cannot bypass in-flight
interruption.
- Preserve OpenCode usage and cost across tool-loop messages and turns
while exposing an explicit current-run delta to durable accounting.
- Keep PNG/WebM evidence in access-controlled artifacts only, reject
SVG, and publish only pruned inert structured per-attempt evidence.
- Add AWS AgentCore infrastructure, provisioning checks, and smoke
tools; reject unsafe model identifiers, require exact stack ownership
markers, and make failed-stack replacement explicit.
- Add evaluation-session contracts and capability reports.
- Add release workflow checks for immutable action pins, frozen
dependency installs, exact weekly cron shape, paid-run guards,
provider-secret isolation, and chaos test paths.
- Reauthorize the original and triggering numeric actor IDs as the first
step of every provider-secret job, including partial reruns, before
checkout or provider access.
- Give each full-stack matrix cell only its matching provider
credential, expose Daytona only to Daytona cells, and disable shared
dependency caches anywhere paid credentials or OIDC write access are
present.
- Protect the legacy manual E2E workflow with the same default-branch,
allowlist, environment, and per-job authorization boundary.
- Rotate live-eval candidates by week and retain 120 days of compatible
history so the seven-week trend window remains viable.
- Restore the root runner-acceptance commands and reconcile reported
snapshots,
raw receipts, and terminal usage without double counting or losing late
usage.
- Mark ACPX token deltas exact only when every budget field is present,
keep
cumulative cost/request authority separate, reject non-USD cost
labeling,
  and include thought tokens in output-token budgets.
- Keep `enableNativeRunner` off by default. The acceptance harness
enables it only in its isolated test instance.

## Verification

Passed locally:

- `pnpm --filter @paperclipai/paperclip-runner typecheck`
- `pnpm test:runner-acceptance:typecheck`
- `pnpm test:runner-acceptance` (19 tests)
- focused OpenCode proxy, driver, runnerd transport, live-session, and
turn-stream tests (106 tests)
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/live/clean-room-server.test.ts` (22 tests)
- `pnpm test:e2e:runner:typecheck`
- `pnpm test:e2e:runner:unit` (62 tests)
- `node --test scripts/__tests__/release-verify-workflow.test.mjs`
- `pnpm --filter @paperclipai/paperclip-runner
test:runner-workflow-evals` (22 tests)
- `pnpm -r typecheck`
- `pnpm build`
- `node --test
packages/paperclip-runner/scripts/aws-agentcore-provisioning.test.mjs`
(6 tests)
- `git diff --check`
- `cargo test --manifest-path
packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core
--lib --locked` (161 tests)
- focused ACPX provider-event tests (10 tests)
- The rebased PR changes 92 files. `pnpm-lock.yaml` is unchanged.

I did not run paid live provider jobs or provision AWS resources. Those
checks need credentials and can create cost.

## Risks

The paid workflows can create provider cost. They require an allowlisted
original and triggering actor, the protected `runner-e2e-paid`
environment, explicit opt-in variables, and cost limits. The four
provider credentials exist only in that master-only environment, which
requires allowlisted reviewer approval and disables administrator
bypass; repository and organization Actions scopes contain no copies.

Provider usage arrives after a billable request, so the live guard
cannot prevent one request from crossing a threshold. It interrupts
immediately on the first reported threshold hit and permits no
continuation.

Visual evidence can contain secrets rendered as pixels. PNG/WebM remain
only in access-controlled workflow artifacts; SVG and per-attempt XML
are excluded, and S3/Pages receive a pruned structured dashboard.

The AWS scripts can create cloud resources. They use explicit commands,
least-privilege roles, KMS encryption, saved nonsecret metadata, and
explicit teardown.

This pull request does not enable the experimental native runner for
existing instances.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex with GPT-5. The model used extended reasoning, tool use,
code execution, and parallel subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-02 08:55:08 -05:00