Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-03 17:51:58 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,122 Commits 1,392 Branches 2,228 Tags
0748b9006273dae7ef8c303905ca0ffa0221a708
Commit Graph
8 Commits
Author SHA1 Message Date
DottaandPaperclip 774ddc7271 test(evals): add the full chat catalog to TestOps (#18542) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat feature evaluations check real user requests through the
product.
> - The YAML catalog has 244 scenarios, but the shipped TestOps
completion route selects only three.
> - The preserved catalog already supplies the execution mappings and
independent oracles.
> - A larger TestOps plan needs exact identities, bounded selection and
source-bound private evidence.
> - This pull request adds that plan route without merging the catalog
app into master or replacing its grades.
> - Users can run the full catalog or a selected subset and see the
actual outcome and prerequisites.

## Linked Issues or Issue Description

Refs: #18516 and #18515. This extends their completion pilot and reuses
its trusted reporting path.

**What existing behavior does this improve?**
The existing three-case TestOps completion workflow.

**Current behavior**
The remaining catalog scenarios cannot be selected through that TestOps
job. The one-case backlog plan has a separate strict singleton contract.

**Proposed behavior**
A separate candidate-scoped plan maps all 244 scenarios. Existing case
aliases, jobs, plans and result history stay intact. Registration is
distinct from a measured pass.

## What Changed

- Add a source-derived 244-row manifest. It binds the 242 domain
descriptors and two legacy tasks to the unchanged YAML queries,
contexts, outcomes and original bounds. The JSON snapshot avoids
importing the unmerged domain tree into master.
- Reconcile exact case identities and a dedicated full-catalog plan.
Reuse legacy aliases, including the existing backlog case, without
renaming them. Preserve history and stop on conflicting aliases or
uncertain writes.
- Extend the shared completion adapter with a separate trusted
full-catalog workflow context. Return a compact observed binding that
fits one 48 KB protected variable. Keep the three-case pilot contract
intact.
- Add a fixed-source manual workflow. Small selections run serially.
Larger selections allow eight independent workers and one serial
external-input worker. Each worker and report reader keeps a singleton
approval and exact-key AWS session. The eight-cell private authority
limit is unchanged.
- Validate and merge metadata-only singleton projections before one
protected Allure upload and complete paginated readback. Preserve failed
originals and cleanup failures beside passed results. Keep screenshots
and raw evidence private.
- Discard a changing full-catalog page-count snapshot and retry within
the same observer bounds. Malformed page fields still fail at once. A
pass still needs one consistent exact selection and all final evidence
checks.
- Keep workflow source and app source separate. The full lane builds
candidate `3b66c3c144360ddf32714a0742fd774b8ceafb16`. Its only changes
from the existing `de0941835cc8f0d01c982574ee258e110babdbeb` pin are
finite TestOps host admission and its offline test. This is not
current-master app qualification.
- Give full selections above three cases a bounded metadata-read
allowance above the existing 120-second deadline ceiling. Readback has
two upper bounds: at most twelve polls with five-second pauses, and a
size-derived deadline of at most 374 seconds, whichever exhausts first.
There is no guaranteed minimum wait or processing SLA. Small selections
and pilot deadlines stay unchanged. Final identity, grade and evidence
predicates stay intact.
- Document the real readiness limits and setup path. Five coverage gaps,
six evidence exceptions, eighteen additional-credential declarations and
thirty-seven external-input declarations overlap. They are not new pass
claims.

## Verification

- Related Node controls passed: pilot API and dispatch, 244-row
manifest, idempotent reconciliation, full selection, three-page result
readback, source guards and partition/output-size bounds. The latest ten
adapter controls include the actual registration receipt's variable-size
limit and the real bootstrap facade's closed failure codes and privacy
checks.
- A deterministic 244-result replay with one second per HTTP response
failed at the old deadline and now verifies after 127 virtual seconds. A
three-second response replay still rejects at 375 virtual seconds. The
full-lane allowance accounts for paginated reads and four-way result
batches; it is not a guaranteed network SLA. All 55 related API/adapter
controls pass. They also reproduce the changing-page-count failure
before the fix, then verify the consistent snapshot. Permanently
changing counts fail after twelve polls. A malformed numeric field still
fails before a retry.
- Owning export controls: 26 passed. They include all 244 synthetic
carriers, mixed passed/failed originals, missing and foreign
projections, privacy filtering and exact source binding.
- Candidate host owner: 31 offline controls passed. The new
finite-catalog case failed before admission changed.
- Preserve two actual local regressions: canonical scenario provenance
failed before its one-line fix; the full registration binding exceeded
the variable limit before compact output. Both now pass.
- Initial local harness setup attempts failed before test execution. The
final narrow owner configuration avoids provider-bundle setup.
- Full repository tests, build and typecheck were not run locally.
Normal required PR CI provides the remote checks.
- Owning TypeScript no-emit, actionlint, paid-workflow policy with
unsafe controls, and diff checks passed.
- Local tests used Node 26.4.0. These are synthetic source controls. No
local app, database, native build or provider qualification ran.
- CI
[41307571776](https://github.com/paperclipai/paperclip/actions/runs/41307571776)
on `23e518be` recorded one 15-second timeout in
`attention-retention-work.test.ts:153`. All other ordinary leaves
passed. The test used a fixed JavaScript date but its retention deadline
comes from PostgreSQL `updatedAt`. The exact failing await and original
SQL timestamp were not captured. The corrected pre-sync head’s server
shard (2/12) passed on AWS in CI 41309030928. Current master
independently fixed the same clock assumption in #18543; conflict
resolution adopts that whole upstream file. No local PostgreSQL run or
timeout increase was used.
- Current-head normal CI
[41463567187](https://github.com/paperclipai/paperclip/actions/runs/41463567187)
passes all required checks on
`fc6c2309cfc7d1e5a83f0a597c7fa1d2beb987e1`, with Greptile 5/5 and all
review threads resolved. The final diff has 23 catalog files. Current
master’s whole retention test is adopted; the redundant earlier test
change is absent. The valid changing-page snapshot regression passes
with all 55 related API/adapter controls. No new TestOps import or live
evaluation has run for this change. After delivery, use metadata-only
bootstrap and a three-case normal TestOps selection to verify the new
route. Do not claim a new 244-case measurement.

## Risks

- The app target remains the preserved catalog candidate. Each result
records its target and workflow revisions. No old campaign ordinal,
acceptance gate or core fix is changed.
- Some mapped cases need private account configuration or have known
source/evidence gaps. Their original checks and evidence rules can still
fail. A registration does not establish readiness or success.
- Whole-plan execution can take hours and spend provider and AWS money.
Original per-case limits and zero retries remain. Unknown setup usage is
not free. This PR does not run all 244 cases.
- The raw private `SOURCE_REF` field retains the prior
integration-lineage label. It is not a verified checked-out branch. The
actual candidate checkout and all authority checks use immutable
`3b66c3c144360ddf32714a0742fd774b8ceafb16`; public projections set
`source.ref` to null.
- TestOps has no atomic bulk upsert. Reconciliation retains partial
progress and performs no automatic resend after an uncertain response.
- The trusted inventory keeps a 12 MiB limit per public carrier file.
The larger parser ceiling does not promise admission above that limit.
The existing 244-case fixture produces a 1,359,465-byte normalized
carrier; this is not a guarantee for future payload sizes.
- Raw evidence remains under the existing private retention owner.
Public GitHub artifacts and TestOps attachments contain only allowlisted
metadata and availability receipts.
- The new workflow needs the exact master entry in the existing runner
group's selected-workflow list before a live run. No repository or
runner-group visibility is broadened by this source change.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) assisted with implementation and
independent source review, using code and tools. The host does not
report the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run the relevant local controls and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-02 08:58:44 -06:00
Dotta e122478368 test(evals): run and record the chat completion pilot through TestOps (#18516) 
## Thinking Path

> - Paperclip evals must verify real user outcomes.
> - A passing manual pilot does not make its cases runnable from
TestOps.
> - The existing TestOps adapter owns one separate backlog case.
> - The three-case completion pilot needs its own exact selection and
reporting path.
> - This workflow binds a real TestOps job run to reviewed catalog
source and AWS execution.
> - Results retain their original verdict, checks, source and private
evidence.

## Linked Issues or Issue Description

Refs #18364, #18377, #18515 and #18509. This is the main-side companion
to merged #18515. The immutable evaluated source is its actual
integration-branch squash merge,
`de0941835cc8f0d01c982574ee258e110babdbeb`; its tree matches the
reviewed catalog head. The host admission remains inert until this
main-only workflow exists. The broader catalog stays off master.

The existing TestOps connection can run only the separate backlog cell.
The project-work, project-defaults and instance-restart cases currently
require manual campaign dispatch and private retrieval. This change
creates a dedicated reusable TestOps plan and records fresh execution
outcomes there. Source checks and registration do not establish a new
live pass.

## What Changed

- Add a default-branch-only workflow with exact TestOps selection,
immutable source/build outputs, serial AWS execution and zero eval
retries. No source/model/budget selector is accepted.
- Register the three exact reusable cases and a separate plan with the
existing reviewed GitHub integration. Preserve opaque job definitions.
Validate the observed single Branch/defaultValue carrier, PATCH only
enable/integration, and use the job’s default branch without rewriting
its definition or adding plan overrides. Fence the initial operator
launch by actual GitHub run number and nonce; normal TestOps Run remains
repeatable.
- Reuse the existing private-evidence authority, scanners, bounds and
exact-key S3 policies. Authenticated selection supplies only this
original attempt's exact source and case identity. The three trusted
protocol files are unchanged.
- Skip report retrieval after an unsuccessful build or skipped
execution; retain evidence and failed grades when evaluation actually
ran.
- Check out trusted main after the target in both build and execution
jobs, and reject any target tree containing the reserved trusted-helper
path. A real temporary-Git-tree regression exercises the actual YAML
guard.
- Keep reporting credentials in separate protected jobs. Export only
closed verdict/check/accounting metadata and private-evidence receipts
to TestOps. Verify final PNGs privately and record their checksums; keep
all screenshot bytes in protected AWS storage and out of public GitHub
artifacts. Preserve failed grades and failed cleanup.
- Verify actual same-run case results and complete result-owned
attachment metadata within a fixed processing window. Missing or
inherited attachments never qualify. Keep source/result links and stable
case identities.

## Verification

- 68 dispatch, selection, registration and readback controls pass. A
checkout-boundary regression fails on the old ordering, accepts an
ordinary target and rejects a candidate-owned helper tree before trusted
checkout. They cover real workflow environment wiring, actual GitHub
metadata guards, exact subset selection, wrong sources/cases/jobs,
replays and attachment readiness.
- 38 exporter controls pass, including the existing backlog exporter and
the unchanged private-evidence authority consumer. Red/green controls
prove project work accepts the catalog’s 1–3 run range in both export
gates, while 0/4 runs are rejected; the actual pinned-source control
verifies all three cases’ run bounds. Cancelled, failed and scheduled
TestOps job stages are rejected even with complete results. The three
new cases retain all 35 independent checks; private diagnostics and all
image bytes are excluded from the public artifact. Extra media files and
directories are rejected.
- The actual checked-in workflow environment is exercised against the
actual pinned source matrix and host function for all three cases before
provider access. Local fixture controls make zero provider calls and do
not claim live AWS qualification.
- All 71 workflow-policy controls pass. Actionlint and owning typecheck
pass. Relevant TypeScript build dependencies pass.
- Full support suite: 5,827 passed, 13 failed, four skipped. A clean
unchanged-master baseline reproduces all 13 failures in the same five
files. They include fixture expectation drift, operating-receipt
fixtures, a profile pin, pnpm bootstrap counts and local database
startup. This change updates the exact TestOps workflow registries; the
two affected boundary controls now pass. Baseline failures remain
retained and are not presented as a green full support suite.
- The catalog dependency passed CI and current-head 5/5 review, has no
unresolved threads, and is merged. All current-head CI checks pass; both
Greptile and Paperclip reviews are 5/5, with no unresolved threads. No
new TestOps mutation, provider call or passing launch is claimed yet. A
fresh three-case TestOps launch and verified server-side results remain
the delivery gate.

## Risks

This is a paid execution and reporting boundary. The new workflow
accepts only trusted default-branch actor identities and a verified
project/plan/job/case selection. Its evaluated source is immutable and
separate from main reporting code. The full catalog is not promoted.
Screenshots, raw evidence and hidden provider data stay in protected
storage. Public GitHub artifacts carry only the closed metadata
projection. TestOps receives checks and private-evidence
availability/checksum receipts. Unknown setup or AWS accounting stays
unknown. One passing launch will not establish full-catalog reliability.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, code execution
and tool use. Exact hosted revision and context-window size are not
exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-11-01 06:09:47 -06:00
DottaandPaperclip 8c9dd1e444 test(runner): qualify Kimi Code lifecycle and recovery (#18029) 
Add the Kimi lifecycle and recovery qualification harness. Preserve original grades and cleanup evidence, strict native ownership audits, bounded startup and cleanup, and early rejection of unsupported local fault execution before credentials or resources.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 22:54:37 -05:00
DottaandPaperclip 9e1735ec17 test(evals): add a fixed TestOps AWS chat feature route (#18322) 
## Thinking Path

> - Paperclip lets people manage AI agents and inspect their saved work.
> - Chat feature evaluations test a real request against saved tasks,
audit records, links and reload behavior.
> - The current suite has Codex and OpenCode cells but no dedicated
runnable DeepSeek 1017 backlog plan in TestOps.
> - TestOps must select the real case before the workflow can access a
model key.
> - This PR adds one fixed, default-branch AWS route for native OpenCode
and DeepSeek 1017.
> - The route keeps original evidence private and sends only a verified
result and approved screenshots to TestOps.

## Linked Issues or Issue Description

**Subsystem affected**

Product E2E evaluation definitions and GitHub Actions delivery.

**Problem or motivation**

A dedicated TestOps plan cannot run the existing backlog chat journey
with native OpenCode. A result must prove saved work and cleanup rather
than report that a model call completed.

**Proposed solution**

Add one OpenCode backlog cell and one fixed public workflow. TestOps
selection validation runs in a token-isolated job before builds or
inference. The workflow retains private originals and reports the bound
outcome to the selected TestOps case.

**Alternatives considered**

A private-repository wrapper would need a new cross-repository GitHub
write credential. The new route uses the existing TestOps GitHub
integration instead. Existing manual candidate approval paths remain
separate.

**Roadmap alignment**

This extends the existing feature evaluation infrastructure. It changes
no production feature, global runtime default or model default.

Current main #18319 supplies the v2 grader, delegation queue accounting
and final screenshot capture; these improvements are preserved. Related
product work: #16814 and #17126 concern backlog handling. This PR
changes neither product implementation and does not claim those changes
are required.

## What Changed

- Add exactly one native OpenCode backlog cell with DeepSeek 1017 under
the separate `runner-opencode-deepseek-1017` profile. Preserve all four
existing Codex and production OpenCode cells and their models.
- Add a default-branch-only workflow with fixed evaluation, zero-result
registration and one plan-launch modes. Require GitHub actor checks and
authenticated singleton selection before inference.
- Add a separate shipped-main evidence authority. Reuse the existing
strict private storage, scanners, inventories and scoped AWS sessions.
- Bind sanitized results, generated JUnit and producer-approved
screenshots to the exact source, run, attempt and case. Keep failed
grades and cleanup failures failed.
- Add TestOps API, CLI and result adapters with pure controls and normal
unit-test wiring. Keep the reporting token out of the evaluated process.
- Verify the current server case result and attachment metadata after
upload, without masking an earlier failed upload or evaluation.
- Apply only the existing reviewed lock correction, verify unchanged
tracked dependency inputs and bind the lock receipt across the
compiled-output handoff before frozen installation.
- Bind the initial administrator launch to one approved workflow run
number and live GitHub source/actor metadata before TestOps
authentication. Use bounded REST case reconciliation and only the
reviewed existing GitHub integration.
- Make two existing CI tests observe their intended state
deterministically: a fake clock for copy-feedback lifetime and a
completed-pass barrier for retry-cancellation batches. Production
behavior and the original assertions remain unchanged.

The two test-only fixes are intentionally included because their
observation boundaries blocked this delivery’s required normal CI. They
preserve every original assertion and change no production behavior. The
original failed runs remain recorded, and the exact causes of those
observations remain unknown.

## Verification

- 157 focused profile, catalog, oracle, filesystem admission, export,
private-path and workflow controls passed on the current-main
composition. The existing stock OpenCode models and the new exact
DeepSeek 1017 selection are checked separately.
- 44 API, CLI, filesystem and post-upload readback controls passed,
including foreign selection, changed hashes, unsafe files, failed
outcomes, expired readback and no-resend behavior.
- The new exact private session and trace layout controls produced four
genuine parent failures and pass with the selected-path extension. The
frozen 244-case candidate approval registry remains unchanged.
- The lock guard has seven filesystem controls plus seven existing
canonical-owner controls. The unrestricted old workflow failed its
actual policy calibration; the corrected frozen-install route passed.
- Eval-owning TypeScript checks, workflow lint and Git whitespace checks
passed. The separate UI test correction was checked with the focused
jsdom unit suite. The server test barrier has pure wrapper and real
scheduler controls; its DB-backed route was not run locally. No
additional local typecheck/build was run for these test-only changes.
- Full native builds, provider execution, app/database behavior and
TestOps registration are reserved for GitHub Actions/AWS. They have not
run locally.
- The initial normal CI run failed the new-workflow policy allowlist and
a heartbeat scheduling assertion. The policy fix has an actual parent
red/current green control and rejects five unsafe route variants.
Heartbeat source is unchanged and its failure cause remains unknown. The
original run is retained.
- The launch-fence actual caller fails against the published parent and
passes the five own-run contradiction controls before authentication.
- The prior `a5aa359d` normal CI completed successfully. The complete
`f9e1f289` run passed all leaf jobs except an existing copy-feedback UI
assertion, plus its dependent verification gate. The four owning UI
files were unchanged from the earlier passing head and actual tested
merge; the exact scheduling cause remains unknown.
- The correction changes only that test to actual React `act` and a
scoped fake clock. All eight component tests pass; early-hide and
never-clear source variants each fail the intended lifetime assertion.
Production clipboard/card behavior is unchanged.
- The `2f27f3fc` run passed the corrected UI shard and every leaf job
except the existing retry-cancellation intermediate-count assertion,
plus its dependent verification gate. The seven relevant owners match
the earlier passing run and historical source comparison; the exact CI
scheduling cause remains unknown.
- The test-only barrier observes a persisted completed worker pass
before asserting the unchanged 20-item cap and two-company fairness,
then releases it for the original 29-item drain and 180-second idle
check. Two extracted-wrapper controls and three actual scheduler
controls pass. The real DB-backed test awaits AWS qualification.
- On `836e80ac`, both repaired test shards passed in AWS. The original
attempt retained a separate E2E cleanup HTTP 500 from a PostgreSQL lock
timeout and one runner shutdown. No production repair or precise
contention/shutdown cause is claimed.
- Exactly one targeted retry of E2E shard 3 and its aggregate gate
passed in attempt 2; exactly one recovery of the interrupted server
shard and its verification gate passed in attempt 3. Other passed jobs
were carried forward. The final normal run is successful. No
whole-workflow retry or evaluation/model retry was launched.
- Both literal required checks (`ci / verify` and `ci / e2e`) are
successful on the exact final source `836e80ac`. Greptile is 5/5 and no
review threads remain open. Actual TestOps registration and the
OpenCode/DeepSeek feature result remain unqualified until their separate
executions.

## Risks

- Actual zero-result registration, tenant API shapes and
public-repository dispatch access must be checked before the first
evaluation. Missing or ambiguous metadata stops the route.
- The real OpenCode/DeepSeek backlog result and Allure outcome remain
unqualified until that run. No passing result is seeded.
- One human query, one accounted work run, 120-second turn, 420-second
attempt, 300-second settled-state bound and zero retries remain
unchanged. Setup usage is outside work accounting and stays unknown.
- The reporting token was configured through the separately reviewed
protected credential bootstrap. It is unavailable to AWS evaluation
jobs.
- Metadata reconciliation is capped at ten pages, 1,000 cases and four
parallel detail reads. Missing or inconsistent metadata stops before
inference. The initial administrator launch counter must be created once
after registration; it does not restrict normal TestOps Run actions.
- Existing manual workflows, historical evaluation grades and candidate
approval records remain unchanged. The original CI failures are
preserved; the test corrections do not establish retrospective causes.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning and tool use. The
runtime did not expose its context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 15:08:16 -05:00
DottaandPaperclip 3bd709df8b Operationalize Product E2E accounting, tracing, and scheduling (#17009) 
## Thinking Path

> - Paperclip manages agents and the work they perform.
> - Product E2E tests exercise real agents, tools, setup and cleanup.
> - Repeated runs need retained costs and traces, including failed
attempts.
> - A passing retry must not hide earlier spending or feedback.
> - Recurring work also needs admission limits, private reports and
failure detection.
> - This change supplies those controls while keeping paid activation
behind existing gates.

## Linked Issues or Issue Description

Refs #16996 and #17158. This is the consolidated landing of the reviewed
eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033,
#17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096,
#17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127,
#17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository
permits squash merges only. A local rehearsal showed that squashing each
slice into master causes repeated conflicts in dependent slices. This PR
preserves their commits and lands their combined implementation in one
reviewed integration. Superseded slices will close after this merge.

## What Changed

- Retain every attempt's cost receipt, including setup, failures,
interruptions and retries. Separate reported amounts, estimates, missing
costs and reserved capacity in reports and dashboards.
- Add bounded request admission, exact model pricecards, gateway error
accounting, cache policy and reconciliation evidence. Refuse unsupported
routes and requests that cannot fit the allowance.
- Integrate Braintrust gateway traces and OTLP correlation, with secret
filtering and experiment imports. Record phase timing, including setup,
feedback capture and cleanup.
- Collect complaint and suggestion evidence after each attempt,
including recovery after failures. Retain explicit review dispositions
independently from the product grade.
- Add gated daily and weekly selections, measured forecasts,
setup/readiness checks, wall-time limits and retained monitor
notifications. Include declared non-model overhead in admission while
reporting the existing subscription separately.
- Keep recurring history private, bound uploads and retention, and use
isolated archive environments. Keep public publication an explicit
operation. Report and publication jobs install only the committed
lockfile, with scripts disabled and isolated stores; stale pins fail
closed.
- Integrate current master, preserve its Product E2E journey billing and
privacy checks, and test that shared setup receipts do not double-count
estimated agent usage. Preserve upstream fixture repairs.

## Verification

- Complete integration baseline
`0649a5ce7544ab20825c2183d2f421157db49f24` includes master
`a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head
as ancestors.
- On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012
Vitest tests passed, two skipped, across 193 files; all 137 Node
source-contract tests passed.
- Focused billing and Product E2E journey contract checks: 24 tests
passed. They cover estimates, unknown receipts, single-owner setup
costs, browser-only execution and private publication guards.
- Monitor and workflow wiring: all 96 Node tests passed. `node
scripts/check-token-gates.mjs`: all gates clean.
- `pnpm test:e2e:runner:typecheck` passes after the standard Runner
TypeScript build refreshed local declarations. The first attempt
rejected the new master `runner_prepared_launch` union member because
this checkout retained an older generated declaration; the source union
already contained that member.
- Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes
dependency re-resolution from archive-authorized reporting jobs after
the security scan flagged it. All 15 workflow-security tests pass after
updating the old two-install assertion.
- Hosted [CI run
39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180)
passes on the current repaired head, including typecheck, build, general
tests, Runner checks, browser shards and canary dry run. Greptile is 5/5
on this exact head with no unresolved threads. The security scan passes
after the fixed lockfile-resolution thread was resolved.
- Paperclip Review returned a terminal neutral check but an incomplete
0-file assessment on the repaired head: its platform blocked both diff
responses as prompt-injection content. This is not a successful code
review. No bypass or fabricated pass was attempted; current-head
Greptile, security and hosted CI are independent gates.
- No paid test, reservation release or recurring-run activation
accompanies this landing. Live route qualification and invoice
reconciliation remain separate evidence.

<details>
<summary>Retained earlier validation failures</summary>

Earlier broad local runs had two 15-second test timeouts (CLI poll
cadence and Retry validation), one embedded-database setup timeout, and
later PostgreSQL semaphore exhaustion. The exact timed-out tests passed
separately without larger timeouts; the broad-run causes were not
established. Those local failures remain distinct from subsequent hosted
CI results. The independent #17158 fix records its Muse fixture
collision, its UUID correction and hosted verification. Local Muse
execution was also blocked by macOS embedded-PostgreSQL startup. No
historical failure is treated as a successful paid qualification.

</details>

## Risks

- Admission is a local control, not a provider invoice cap. Missing
provider receipts or resource charges retain unknown amounts and holds;
estimates do not become billed dollars.
- The bounded path admits only documented local suites and provider
routes. Ordinary legacy or separate Runner protocol campaigns have
different controls. Merging this code does not qualify additional
routes.
- Recurrence still needs the existing enable gates and current
qualification/allowance evidence. Archive permissions and resources must
match the documented private-environment contract.
- Legacy reports may lack complete attempt or phase evidence. Reports
preserve that limitation and do not infer zero usage from missing data.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The exact deployment variant and context-window size are not
exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted checks;
full-suite host limitation disclosed above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-23 07:18:37 -07:00
DottaandPaperclip 72ff3a9f27 Measure native tool context and expand bounded workflow evals (#15218) 
## Thinking Path

> - Paperclip manages persistent agents and their assigned work.
> - Agents receive both fixed instructions and tool definitions.
> - Moving a procedure into a tool description still adds model context.
> - We need to measure the complete delivered catalog and test real
outcomes.
> - The tested reductions saved little space and introduced failing
outcomes.
> - This PR keeps measurement, bounded eval coverage and the original
evidence.
> - Production prompts, tools and runtime behavior stay unchanged.

## Linked Issues or Issue Description

Refs: #15151, #14961, #14948, #14985.

This adds the measurement and eval coverage needed to assess further
native
instruction changes. The attempted hiring and dependency reduction
failed
qualification and is excluded from the final diff.

## What Changed

- Measure the actual standard-mode tool authority, including all 39
tools and their input schemas. Capture scripted native start, resume and
continuation payloads and the OpenCode MCP declaration list.
- Add OpenCode to the two explicit-only local hiring/reuse and
delegation/feedback stories. Preserve the original task requests and
independent oracles.
- Apply one attempt per selected story and explicit company and
lead-agent budget stops.
- Select managed hiring credentials from the requested profile,
including OpenRouter.
- Run the existing Node test files under Node instead of collecting them
as Vitest suites.
- Retain sanitized comparison reports, original failed grades, source
hashes and evidence gaps.

## Verification

Final source: `2e7cef78eef7cdfe02265e0dcb03e855b8e50bd8`. Local
verification passes:

- Full repository `pnpm -r typecheck` and `pnpm build`.
- Eval-support unit tests: 1,252 Vitest checks and 128 Node checks.
- Eval TypeScript check and six final-source measurement tests. All 36
normalized
components across nine scripted deliveries and the OpenCode MCP catalog
match
the baseline exactly; the complete standing projection is 49,200 bytes.

Fresh review of this exact head is [5/5 with no remaining
findings](https://github.com/paperclipai/paperclip/pull/15218#issuecomment-5996723170);
all three review threads are resolved.
[Final-head
CI](https://github.com/paperclipai/paperclip/actions/runs/37354539126)
passes all 47 jobs, including repository typecheck, build, tests, runner
checks,
browser shards and the canary check. One initial annotation-test timeout
is
retained in attempt 1; its seven-test suite passed locally, and the
affected
CI lane passed on one targeted retry. No source or paid eval rerun was
needed.

Both ready-transition security scans passed with zero annotations. The
PR is
out of draft and conflict-free. GitHub still requires code-owner
approval for
the `package.json` test-script change; its requested reviewers are
already set.

A byte-for-byte comparison against master context
`a65ca0950834a85bb93bcc4b4042ecacdebfef53` confirms no production
changes under
`packages/` or production `server/` paths. The only server addition is a
measurement test. No new paid rerun is needed to compare unchanged
production
bytes. This does not claim that existing product defects have been
fixed.

The rejected corrected experiment had baseline **5 PASS / 1 FAIL** and
candidate
**3 PASS / 3 FAIL**, including **two newly failing pairs**. The later
readiness
experiment had candidate **3 PASS / 3 FAIL** and baseline **3 PASS / 2
FAIL / one
setup cell without a behavioral grade**. Its five comparable pairs had
two new
failures, two new passes and one unchanged pass. The missing baseline
Codex
cell never reached its provider step because Docker setup timed out.

The observed missing behaviors include parent continuation, waiting for
the
latest child revision, revised ZIP delivery and OpenCode credential
persistence.
Those failures remain failures. Source review and passing CI do not
regrade them.
The original reduction saved 460 bytes; its first repair saved only 125
bytes,
and the larger unqualified runtime repair increased the full projection.
None
of those production changes is shipped here.

Read the
[report](https://github.com/paperclipai/paperclip/blob/2e7cef78eef7cdfe02265e0dcb03e855b8e50bd8/doc/plans/2026-10-05-native-procedure-guidance.md)
and its linked sanitized receipts for exact
sources, original campaign links, pair-level results and evidence
limitations.

## Risks

- Full-catalog bytes are not model tokens, invoices, private vendor
prompts, lazy-loading behavior or truncation proof.
- The two stories are explicit-only and do not prove general coding
quality or arbitrary resume behavior. Single trials do not establish
causation or performance trends.
- The retained OpenCode candidate credential guard failed. Cleanup
removed the original provider database, so the precise persistence
mechanism remains unknown. The guard is unchanged.
- ACPX provider-execution IDs lack proven host-call mapping. No
extra-work or feedback-consumption claim is inferred by matching names,
order or counts.
- PostgreSQL cannot start locally while the host's shared-memory slots
are exhausted. Hosted CI must supply the full database checks; the full
local database suite is not claimed green.

## Model Used

OpenAI Codex, based on GPT-6, with repository tools and code execution.
The exact deployment model ID and context-window limit are not exposed
in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-05 20:03:35 -05:00
DottaandPaperclip b17019e14d fix(agents): reduce default instructions and qualify stock harnesses (#14948) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its adapters supply task context and access to Paperclip skills and
tools.
> - The default hire manual and shared prompts also repeat general work
procedures.
> - Those procedures overlap with stock provider instructions and the
Paperclip skill.
> - Existing E2E fixtures supply a QA manual, so they do not qualify the
production default.
> - This pull request reduces the generic instructions and adds real
default-hire coverage.
> - The benefit is less competing guidance, with inspectable evidence
for preserved skills and task context.

## Linked Issues or Issue Description

Refs: #14920. That merged change preserves native Codex base
instructions. This PR covers the default manual, shared legacy prompts,
operational skill guidance, and the narrowly approved ACP
skill-discovery/session-environment repair for measured delivery and
credential-persistence failures.

**What existing behavior does this improve?**

New non-CEO hires without a custom bundle and legacy task/chat startup
and continuation prompts.

**Current behavior**

The shipped default manual contains 602 words. Generic task/chat prompts
and ordinary resume deltas repeat work procedures already available
through the harness and Paperclip skill.

**Proposed behavior**

The default manual contains only the eight-word company identity. Shared
startup prompts retain identity and connection guidance. Ordinary resume
deltas retain current work context without the generic execution
contract.

**Reason and benefit**

Let the stock harness guide general work. Keep Paperclip-specific
capabilities and independently test default hires, skills, ordered
comments, and chat restart.

**Breaking changes**

New default hires receive less guidance. Existing saved manuals,
explicit custom bundles, CEO templates, and specialized wake contracts
retain their behavior. The obsolete includeExecutionContract option
remains accepted for source compatibility.

## What Changed

- Reduce the default hire manual to one sentence.
- Reduce shared task/chat defaults and remove the generic
ordinary-resume contract.
- Keep connection guidance, auth, skills, custom prompts, and
specialized wake context.
- Add credential-free instruction-boundary gates and 26 explicit Product
E2E cells across eight legacy/native profiles, including two focused
Paperclip-storage cases.
- Capture public hire receipts before providers run, then grade
delivered prompts and independent task/chat outcomes.
- Add an early legacy skill API recipe for saving a task document,
checking the saved revision receipt and linking the document. Improve
stock task/heartbeat skill-selection metadata and show a clickable
Markdown UI-link example. Keep native tool completion separate.
- Advertise bounded routing descriptions and exact successfully staged
SKILL.md paths in legacy ACP Claude; keep full bodies on demand and
preserve remote path rebasing.
- Remove only the provider environment from copied persisted ACP session
records, while loading current run credentials and preserving all other
options/conversation state.
- Regenerate both capability metadata inventories and reject stale
manifests/inventories before provider admission.
- Publish the original reduction and focused skill-repair comparisons,
preserving all failures, automatic recovery, cost coverage and
limitations.

## Verification

**Behavioral qualification remains pending.** Original legacy ACP Claude
loses the issue document only in the reduced cohort beneath an unchanged
credential failure. A source-backed diagnosis finds that neither
ordinary assignment reads the staged operational skill, while the
runtime persists provider environment in session state. The new common
repairs expose skill metadata/path and omit persisted env; strict
document and credential guards stay intact. [Inspectable diagnosis and
retained
hashes](https://github.com/paperclipai/paperclip/blob/9f654db4541d3d002769c988f6e51fc0b08dadbd/doc/plans/2026-10-03-legacy-acp-claude-readiness.md).

Current repair head `de0965984ff3edf611ae6d0e7ca5c7d5ae3947bb`
incorporates master `569c7203aa24b95440682983ce7940ba1d4247bd` (merged
#14961/#15007). All 222 affected adapter tests, adapter-utils/E2E
typechecks, and final 96 variant/grader/retry calibrations pass. The
frozen historical comparator is
`c25697f4260b6f3adfea143c3ae9932e2f42986d`: 8,280 of 8,291 paths
identical, exactly two production instruction paths plus nine declared
unit expectations differ. The operational skill/discovery/environment
repairs, selected model/profile/task/core grader/auth/permissions/retry
policy are identical. Both actual launcher prepare→verify admissions
pass with zero providers. [Immutable manifest and exact
receipts](https://github.com/paperclipai/paperclip/blob/9f654db4541d3d002769c988f6e51fc0b08dadbd/doc/plans/2026-10-03-legacy-acp-claude-evidence/manifest.json).

One original legacy ACP Claude cell per variant is authorized, with
enforced single campaign attempts, 12-minute deadlines and company/agent
1,000-cent hard stops; every product recovery run/cost is counted.
Actual live outcomes are pending. Current normal CI has one failed
server shard and failed aggregate verify under diagnosis; other normal
gates including typecheck/build/Rust/all eight browser shards pass.
Fresh review completed successfully; the valid historical startup/resume
masking finding was fixed with per-invocation task/chat checks and
strict complete-snapshot capture, calibrated and resolved. Prior heads,
failures and campaigns below remain historical evidence, not checks on
this repair head.

- Prior head `36aa4d81c49a1a8f6f04b1a068fae19aa901955f` is replayed on
merged hiring master `862a5758ba0e88a33232c1f1fa645e85c38a3113`. All 52
current-head checks pass with two intentional Storybook skips, including
repository typecheck/test/build and the browser shard. Fresh Greptile is
5/5 with zero unresolved review threads. Exact-head stock prerequisites
pass 599 assertions (598 TypeScript + 1 Rust), all six gates and
retained receipt verification, zero providers/source errors. Fingerprint
`a7f5a22d860a88fe20cce213c6d5e0004788f32c8363930729aea4fd740ad16d`.
Combined catalog/hiring calibrations pass 67 assertions, E2E typecheck
and 26-cell stock discovery pass. Canonical contract/inventory checks
and the later issue-derived reference calibration are retained; that
reference-only follow-up is not live-qualified by earlier frozen runs.
- Prior full repository typecheck/build passed. The complete local
Vitest run executed 14,956 tests: 14,870 passed, 83 skipped, three
timing failures. All three affected files passed unchanged narrow
reruns; original failures remain retained. Current-head CI now passes
the full general checks; the original local failures remain retained.
- The original 24-pair default-manual/shared-prompt comparison has two
new overall classic Claude/OpenCode document-delivery failures plus an
additional legacy ACP Claude document loss beneath an unchanged
credential-guard failure (not closed by later runs), two newly passing
OpenCode ordered cases, seven unchanged failures and 13 unchanged
passes. Equal 15/24 totals do not establish behavioral equivalence.
[Complete original
report](https://github.com/paperclipai/paperclip/blob/875f4c397d9e8c3f12f39dedd59abaf1eaf5236e/doc/plans/2026-10-02-stock-harness-live-comparison.md).
- The skill-only repair holds the eight-word manual/shared prompts and
merged #14920 fixed. All four matched profile configurations and 203
fixture/behavior files match. Candidate
`abd0b628ca642c09a54a4edc56a5227402f6686e` varies only the two skill
sources against baseline `bc83fe030234439ac51279502a28803958963e2e`.
[Candidate
workflow](https://github.com/paperclipai/paperclip/actions/runs/37060885547)
and [baseline
workflow](https://github.com/paperclipai/paperclip/actions/runs/37060888047)
each pass 571 exact-source prerequisites before providers; all eight
cells clean up successfully. Failed campaigns publish successfully and
remain failed.
- Repair pairs: Claude original Fail → Pass; Claude explicit Pass →
Pass; both OpenCode cases Fail → Fail. Explicit OpenCode's handoff
worsens beneath the unchanged failing UI-link grade: baseline gives a
clickable API URL, candidate gives a code-formatted path without an
anchor. The request's usable-link wording is narrower in the UI-only
oracle. [Complete repair report and safe
projection](https://github.com/paperclipai/paperclip/blob/875f4c397d9e8c3f12f39dedd59abaf1eaf5236e/doc/plans/2026-10-02-legacy-document-skill-repair.md).
- The subsequent narrow stock metadata/link correction has two matched
Pass → Pass cases, zero new machine failures/passes and no pending
pairs. Both original-case handoff links remain deficient: candidate uses
a wrong PAP prefix, baseline supplies a bare prefix-less slug path; the
preserved original oracle only requires a durable document. Both
explicit clickable UI-link cases pass revision/content/link grading. All
four exact-source 587-check gates, single assignment runs and cleanup
pass. This does not establish fix causality because baseline also
succeeds. [Candidate
workflow](https://github.com/paperclipai/paperclip/actions/runs/37069547401)
freezes `fe9dc1e3c518825242ed889ab9c8352986f8c2ed`; [matched
baseline](https://github.com/paperclipai/paperclip/actions/runs/37069552374)
freezes `0d7ecfa96d72fba79b7f0a25052b42c0686c0488`. This is a skill-only
comparison with reduced manuals/shared prompts held constant, not a
repeat of the historical-manual comparison. Only original and clarified
explicit classic OpenCode cases are selected, two per variant/four
expected turns. 8,242 other tracked files and both profile hashes match;
protected workflows admit each exact source before credentials.
[Complete qualification
report](https://github.com/paperclipai/paperclip/blob/74d0d3d945f4c52d0814b5a845ab5bd09f33cd6b/doc/plans/2026-10-02-opencode-skill-routing-link-qualification.md).
Candidate original loads Paperclip/reference before saving publicly;
baseline original loads it after writing locally, then saves publicly
within the same assignment. Reported cost totals are $0.0107824490
candidate / $0.0107909015 baseline, with unmetered runtime. The later
reference-only issue-derived link correction is provider-free calibrated
and **not live-qualified** by these frozen runs; no further paid runs.
- Retained tool calls show the repaired original OpenCode assignment
loads only its assigned output skill before writing locally. Operational
Paperclip is first loaded during automatic disposition recovery; its
early recipe is visible then, but it never saves the missing document.
Explicit candidate loads Paperclip and reads the new reference before
saving successfully. All nine actual runs are counted. Reported LLM
totals are $0.3802537209 baseline and $0.4918990161 candidate; local
runtime is unmetered.
- Initial setup, packaging, cancelled/missing-cell recovery, callback
test and relative-output attempts remain retained. No completed provider
failure was rerun. Frozen measurement branches are unchanged by later
canonical metadata maintenance.
- Run `pnpm test:e2e:runner:stock-harness`, `pnpm test:e2e:runner:unit`,
and `pnpm test:e2e:runner:typecheck`. Select `stock-harness` explicitly
for paid execution; it is excluded from `--all`.

Prior-head integration: `36aa4d81c49a1a8f6f04b1a068fae19aa901955f`
replays this PR on merged hiring #14985
(`862a5758ba0e88a33232c1f1fa645e85c38a3113`), preserving the four
explicit custom-CEO-bundle checks, minimal generic manual boundary, and
both suites. The combined fixture catalog and hiring calibrations pass
67 assertions; exact-head stock prerequisites pass 599 assertions (598
TypeScript + 1 Rust), all six gates and retained-receipt verification,
zero providers/source errors, fingerprint
`a7f5a22d860a88fe20cce213c6d5e0004788f32c8363930729aea4fd740ad16d`. E2E
typecheck and 26-cell stock discovery pass. Fresh current-head CI passes
all 52 checks with two intentional skips, and fresh Greptile is 5/5 with
zero unresolved review threads.

The prior source-plan browser failure is retained: a deterministic
process fixture replayed its last `fixture:plan` command on
`chat_task_completed`, writing revision 2 with identical body after the
approval handoff. This was not paid provider execution. Rebased
current-head CI passes the same assertion without an old-head retry or a
change to that browser fixture.

The merged hiring change was measured separately on immutable matched
unions, with this reduced/shared/operational context and native
completion guidance held constant. [Complete original two-profile
report](https://github.com/paperclipai/paperclip/blob/f0512647656be78e48abd8c22a3078db8bf6bcd2/doc/plans/2026-10-02-hiring-template-live-comparison.md):
[candidate](https://github.com/paperclipai/paperclip/actions/runs/37075466208)
/ [historical
baseline](https://github.com/paperclipai/paperclip/actions/runs/37075469463),
705 provider-free prerequisites each. Both pairs are unchanged Fail →
Fail on the exact-five count, with six core delivery checks passing all
four cells; 28 actual successful runs include eight automatic completion
wakes, zero retries, four successful cleanups. Source-read coverage is
uncomparable, actual model charges unknown. Separately versioned
provider-free accounting remains analytical work; original verdicts are
preserved. This does not rerun or qualify the completed default-manual
or native campaigns.

## Risks

- Legacy ACP Claude's additional delivery loss is not closed by any
later matched run and blocks the no-extra-failing-behavior merge
criterion. Legacy document delivery may have relied on the prior
manual/shared prompts. The early skill repair improves Claude in one
trial; the later OpenCode pairs pass in both variants and cannot
establish causality or robust recovery. Both original-case links remain
deficient beneath the storage-only grade. The later issue-derived
reference correction has only provider-free validation. Native
finish/block descriptions must not be supplied to legacy agents.
- The comparison holds merged native Codex fix #14920 constant; it
cannot measure that fix's before/after task performance.
- These bounded skill/context/chat workflows do not measure general
coding quality. Unrepresented providers remain unqualified.
- Saved manuals and old Codex sessions are not automatically migrated.
Codex through ACP still has a separate base-instruction follow-up.

## Model Used

OpenAI Codex, GPT-6 family as identified by this session. The exact
deployment ID and context-window size are not exposed. The assistant
used reasoning, repository tools, code execution, and delegated PR/eval
work.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (relevant suites and all
three unchanged narrow reruns pass; complete-run timing failures
retained in Verification)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green on the new repair head
(prior-head checks retained above)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
on the new repair head
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-03 12:32:42 -05:00
Dotta 5716fe907e test(runner): add full-stack acceptance and eval gates (#12700) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The runner subsystem executes agent work across local and managed
provider backends.
> - The lower pull requests restore the task runtime, provider backends,
and managed-provider control plane.
> - The restored system needs repeatable full-stack checks before it can
ship safely.
> - Paid live checks also need clear access, cost, and secret controls.
> - This pull request adds acceptance, live evaluation, chaos, and
release gates for the restored runner stack.
> - The benefit is measurable runner parity with safer release
decisions.

## Linked Issues or Issue Description

**Subsystem affected**

Cross-cutting. This change covers runner tests, release workflows,
server contracts, and evaluation tools.

**Problem or motivation**

The runner stack did not have one complete acceptance surface for native
Codex, ACPX, Claude Managed, and AWS AgentCore. Release checks could
miss provider drift, task-view regressions, cost-policy errors, and
destructive cleanup errors.

**Proposed solution**

Add a 57-cell full-stack catalog, a Daytona image, and opt-in paid
workflows. Add live evaluation, chaos, cost-limit, redaction, and
release contract checks. Add AWS AgentCore infrastructure and guarded
provisioning tools. Keep the native runner experimental flag off by
default.

**Alternatives considered**

We considered manual smoke tests only. They do not give repeatable
evidence and they do not protect release branches. We also considered
one large pull request. The stacked pull requests keep each review below
the Greptile file limit.

**Roadmap alignment**

This work supports the shipped Cloud / Sandbox agents milestone and the
shipped Agent evals & feedback milestone in `ROADMAP.md`.

Related stack:

- #12699 adds managed provider backends and lifecycle support.
- #12691 adds qualified OpenCode and ACPX provider backends.
- #12685 restores task runtime rendering and steering.

## What Changed

- Add the runner full-stack harness with 57 catalog cells and 60 unit
tests.
- Add a Daytona runner image with digest-pinned base images and
base-aware image-content checks.
- Add guarded live evaluation and chaos workflows with a fixed
40-execution matrix; live and full-stack paid schedules now run only on
Sundays or by manual dispatch.
- Add in-flight reported-usage cost stops, post-turn cost caps,
exact-threshold failure classification, secret redaction, retry
classification, and actor authorization.
- Reattach stream and hard-budget listeners before restart-recovery
continuations so restored paid sessions cannot bypass in-flight
interruption.
- Preserve OpenCode usage and cost across tool-loop messages and turns
while exposing an explicit current-run delta to durable accounting.
- Keep PNG/WebM evidence in access-controlled artifacts only, reject
SVG, and publish only pruned inert structured per-attempt evidence.
- Add AWS AgentCore infrastructure, provisioning checks, and smoke
tools; reject unsafe model identifiers, require exact stack ownership
markers, and make failed-stack replacement explicit.
- Add evaluation-session contracts and capability reports.
- Add release workflow checks for immutable action pins, frozen
dependency installs, exact weekly cron shape, paid-run guards,
provider-secret isolation, and chaos test paths.
- Reauthorize the original and triggering numeric actor IDs as the first
step of every provider-secret job, including partial reruns, before
checkout or provider access.
- Give each full-stack matrix cell only its matching provider
credential, expose Daytona only to Daytona cells, and disable shared
dependency caches anywhere paid credentials or OIDC write access are
present.
- Protect the legacy manual E2E workflow with the same default-branch,
allowlist, environment, and per-job authorization boundary.
- Rotate live-eval candidates by week and retain 120 days of compatible
history so the seven-week trend window remains viable.
- Restore the root runner-acceptance commands and reconcile reported
snapshots,
raw receipts, and terminal usage without double counting or losing late
usage.
- Mark ACPX token deltas exact only when every budget field is present,
keep
cumulative cost/request authority separate, reject non-USD cost
labeling,
  and include thought tokens in output-token budgets.
- Keep `enableNativeRunner` off by default. The acceptance harness
enables it only in its isolated test instance.

## Verification

Passed locally:

- `pnpm --filter @paperclipai/paperclip-runner typecheck`
- `pnpm test:runner-acceptance:typecheck`
- `pnpm test:runner-acceptance` (19 tests)
- focused OpenCode proxy, driver, runnerd transport, live-session, and
turn-stream tests (106 tests)
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/live/clean-room-server.test.ts` (22 tests)
- `pnpm test:e2e:runner:typecheck`
- `pnpm test:e2e:runner:unit` (62 tests)
- `node --test scripts/__tests__/release-verify-workflow.test.mjs`
- `pnpm --filter @paperclipai/paperclip-runner
test:runner-workflow-evals` (22 tests)
- `pnpm -r typecheck`
- `pnpm build`
- `node --test
packages/paperclip-runner/scripts/aws-agentcore-provisioning.test.mjs`
(6 tests)
- `git diff --check`
- `cargo test --manifest-path
packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core
--lib --locked` (161 tests)
- focused ACPX provider-event tests (10 tests)
- The rebased PR changes 92 files. `pnpm-lock.yaml` is unchanged.

I did not run paid live provider jobs or provision AWS resources. Those
checks need credentials and can create cost.

## Risks

The paid workflows can create provider cost. They require an allowlisted
original and triggering actor, the protected `runner-e2e-paid`
environment, explicit opt-in variables, and cost limits. The four
provider credentials exist only in that master-only environment, which
requires allowlisted reviewer approval and disables administrator
bypass; repository and organization Actions scopes contain no copies.

Provider usage arrives after a billable request, so the live guard
cannot prevent one request from crossing a threshold. It interrupts
immediately on the first reported threshold hit and permits no
continuation.

Visual evidence can contain secrets rendered as pixels. PNG/WebM remain
only in access-controlled workflow artifacts; SVG and per-attempt XML
are excluded, and S3/Pages receive a pruned structured dashboard.

The AWS scripts can create cloud resources. They use explicit commands,
least-privilege roles, KMS encryption, saved nonsecret metadata, and
explicit teardown.

This pull request does not enable the experimental native runner for
existing instances.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex with GPT-5. The model used extended reasoning, tool use,
code execution, and parallel subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-09-02 08:55:08 -05:00